Remove failed queue items with files.

Question

Remove failed queue items with files.

Aerglonus opened this issue 6 months ago · comments

On a previous issue i brought this up about failed tasks due to the live check going off right after the stream ends and how the only way to mitigate this was increasing the live check interval at the expense of missing X time of a live archive.

After ending stream the current queue item will finish correctly (convert and move) and immediately after another queue will start that eventually will fail (since stream no longer live).

Originally posted by @Aerglonus in #360 (comment)

The Twitch API is slow to update when the stream ends. The first archive ended as the video feed stopped. Shortly after the live channel check fires again and Twitch still says the stream is online. Eventually the video task will fail on the 2nd archived as the stream is truly offline but Twitch hasn't updated yet. A higher live check interval helps mitigate this.

I can try to make the video download task be more aggressive and clean up the archive if the process ends and there is no video file.

Originally posted by @Zibbp in #360 (comment)

After increasing the interval not by a lot I noticed that even though the task still fails the workers stop properly but the queue item remains registered and failed download thumbnail files remain (since its where it always fails), was wondering if this could be implemented to not having to remove them manually every time it fails and avoid increasing the interval a lot more.

Isaac · Answer 1 · Thu Feb 29 2024 08:16:29 GMT+0800 (China Standard Time)

This is on my ToDo list. Have you mainly noticed the thumbnail task failing when this occurs, or do you see other tasks such as save info or video download?

Caleb Contreras · Answer 2 · Mon Mar 04 2024 12:54:49 GMT+0800 (China Standard Time)

This is on my ToDo list. Have you mainly noticed the thumbnail task failing when this occurs, or do you see other tasks such as save info or video download?

The thumbnail task fails and the download video task gets triggered anyway but it stops since there's no stream and stays spinning, never switches to error even though the log says about stream not found.

When this happens files in directory are the placeholder thumbnails from twitch (grey image) and a json file(stream info), the VOD registered stays as "processing" and queue item stays as if it was downloading or processing (circle spinning).

Caleb Contreras · Answer 3 · Sun Mar 10 2024 12:21:08 GMT+0800 (China Standard Time)

This is on my ToDo list. Have you mainly noticed the thumbnail task failing when this occurs, or do you see other tasks such as save info or video download?

The thumbnail task fails and the download video task gets triggered anyway but it stops since there's no stream and stays spinning, never switches to error even though the log says about stream not found.

When this happens files in directory are the placeholder thumbnails from twitch (grey image) and a json file(stream info), the VOD registered stays as "processing" and queue item stays as if it was downloading or processing (circle spinning).

This is the task state after failing : (Adding screenshots so its more clear what i mean hehe)

Isaac · Answer 4 · Sun Mar 10 2024 12:50:49 GMT+0800 (China Standard Time)

I've made some changes in #385 which should catch if the thumbnail download task or the video download task fail in a way that indicates the stream is offline. When this happens it cleans up the workflows, queue item, and vod. I haven't added this logic to the save info task, do you have any information from the API container of how that failed?

If you want to test these fixes, build a docker image from this branch #385. Let me know how it works or if it still breaks, if you could provide the debug logs when it happens, that will help.

Caleb Contreras · Answer 5 · Sat Mar 16 2024 13:17:09 GMT+0800 (China Standard Time)

Sorry for taking so long to respond, its hard to test because for some reason sometimes it doesn't trigger the error here is the log of the API container exactly when a live archive ended and the second one (the one that fails) gets triggered.

The only things i find suspicions are those 404 and 500 errors from the proxy server that are shown right before the Download video returns the no playable stream found and this time the save info task didn't failed and the download video is marked as completed of course with no playable stream found (workers stopped as intended) attaching images at the end.

Also another thing to point out is that the vod that finished correctly and the failed one I noticed that the titles are different

VOD -> 2024-03-15-drops_and_i_have_so_many_raids_to_do__!socials_!wishlist-live-a80ab21f-e321-11ee-b860-0242ac120004
Failed VOD -> 2024-03-15-drops_''_EXTREME_VALTAN__!socials_!wishlist-live-808002e7-e350-11ee-b860-0242ac120004

THIS IS WITHOUT YOUR FIXES YET
will be testing your fixes

{\rtf1\ansi\ansicpg1252\cocoartf2709
\cocoatextscaling0\cocoaplatform0{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
{\*\expandedcolortbl;;}
\margl1440\margr1440\vieww11520\viewh8400\viewkind0
\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0

\f0\fs24 \cf0 2024-03-14 13:46:08 usermod: no changes
\
2024-03-14 13:46:08 
\
2024-03-14 13:46:08 -------------------------------------
\
2024-03-14 13:46:08 User uid:    911
\
2024-03-14 13:46:08 User gid:    911
\
2024-03-14 13:46:08 -------------------------------------
\
2024-03-14 13:46:08 
\
2024-03-14 13:46:13 Version    : 
\
2024-03-14 13:46:13 Git Hash   : 
\
2024-03-14 13:46:13 Build Time : 2024-02-19T00:46:04Z
\
2024-03-14 13:46:13 2024/03/14 15:46:13 INFO  No logger configured for temporal client. Created default one.
\
2024-03-14 13:46:13 
\
2024-03-14 13:46:13    ____    __
\
2024-03-14 13:46:13   / __/___/ /  ___
\
2024-03-14 13:46:13  / _// __/ _ \\/ _ \\
\
2024-03-14 13:46:13 /___/\\__/_//_/\\___/ v4.11.4
\
2024-03-14 13:46:13 High performance, minimalist Go web framework
\
2024-03-14 13:46:13 https://echo.labstack.com
\
2024-03-14 13:46:13 ____________________________________O/_______
\
2024-03-14 13:46:13                                     O\\
\
2024-03-14 13:46:13 \uc0\u8680  http server started on [::]:4000
\
2024-03-14 13:46:18 \{"level":"info","Namespace":"default","TaskQueue":"archive","WorkerID":"0838558eb4c6","time":"2024-03-14T15:46:18-05:00","message":"Started Worker"\}
\
2024-03-14 13:46:18 \{"level":"info","Namespace":"default","TaskQueue":"chat-download","WorkerID":"0838558eb4c6","time":"2024-03-14T15:46:18-05:00","message":"Started Worker"\}
\
2024-03-14 13:46:13 \{"level":"info","time":"2024-03-14T15:46:13-05:00","message":"config file found at /data/config.json, loading"\}
\
2024-03-14 13:46:13 \{"level":"info","time":"2024-03-14T15:46:13-05:00","message":"config file loaded: /data/config.json"\}
\
2024-03-14 13:46:13 \{"level":"debug","time":"2024-03-14T15:46:13-05:00","message":"config file loaded: /data/config.json"\}
\
2024-03-14 13:46:13 \{"level":"info","time":"2024-03-14T15:46:13-05:00","message":"Connected to temporal at 192.168.1.64:7233"\}
\
2024-03-14 13:46:14 \{"level":"info","time":"2024-03-14T15:46:14-05:00","message":"authenticated with twitch"\}
\
2024-03-14 13:46:18 \{"level":"info","time":"2024-03-14T15:46:18-05:00","message":"Starting worker with config: \{MAX_CHAT_DOWNLOAD_EXECUTIONS:5 MAX_CHAT_RENDER_EXECUTIONS:3 MAX_VIDEO_DOWNLOAD_EXECUTIONS:5 MAX_VIDEO_CONVERT_EXECUTIONS:3 TEMPORAL_URL:192.168.1.64:7233\}"\}
\
2024-03-14 13:46:18 \{"level":"info","time":"2024-03-14T15:46:18-05:00","message":"config file found at /data/config.json, loading"\}
\
2024-03-14 13:46:18 \{"level":"info","time":"2024-03-14T15:46:18-05:00","message":"config file loaded: /data/config.json"\}
\
2024-03-14 13:46:18 \{"level":"debug","time":"2024-03-14T15:46:18-05:00","message":"config file loaded: /data/config.json"\}
\
2024-03-14 13:46:18 \{"level":"debug","time":"2024-03-14T15:46:18-05:00","message":"authenticating with twitch"\}
\
2024-03-14 13:46:18 \{"level":"info","time":"2024-03-14T15:46:18-05:00","message":"setting up check watched channel videos schedule"\}
\
2024-03-14 13:46:18 \{"level":"info","time":"2024-03-14T15:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-14 13:46:18 \{"level":"info","time":"2024-03-14T15:46:18-05:00","message":"authenticated with twitch"\}
\
2024-03-14 13:46:18 \{"level":"debug","time":"2024-03-14T15:46:18-05:00","message":"setting up database connection"\}
\
2024-03-14 13:46:18 \{"level":"info","Namespace":"default","TaskQueue":"chat-render","WorkerID":"0838558eb4c6","time":"2024-03-14T15:46:18-05:00","message":"Started Worker"\}
\
2024-03-14 13:46:18 \{"level":"info","Namespace":"default","TaskQueue":"video-download","WorkerID":"0838558eb4c6","time":"2024-03-14T15:46:18-05:00","message":"Started Worker"\}
\
2024-03-14 13:46:18 \{"level":"info","Namespace":"default","TaskQueue":"video-convert","WorkerID":"0838558eb4c6","time":"2024-03-14T15:46:18-05:00","message":"Started Worker"\}
\
2024-03-15 16:13:38 \{"level":"debug","Attempt":1,"ChildWorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_5","Namespace":"default","RunID":"8b3d52f9-aade-4127-907e-a3ba716f21b7","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a80ab21f-e321-11ee-b860-0242ac120004","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-15T18:13:38-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 16:13:38 \{"level":"debug","ActivityID":"5","ActivityType":"CreateDirectory","Attempt":1,"Namespace":"default","RunID":"35c165a0-7b54-4172-a292-27411970759b","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_5","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-15T18:13:38-05:00","message":"ExecuteActivity"\}
\
2024-03-15 16:13:39 \{"level":"debug","Attempt":1,"ChildWorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_14","Namespace":"default","RunID":"8b3d52f9-aade-4127-907e-a3ba716f21b7","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a80ab21f-e321-11ee-b860-0242ac120004","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-15T18:13:39-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 16:13:39 \{"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"eb656d0b-b2f5-472f-8f10-5fd88c8858a6","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_14","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-15T18:13:39-05:00","message":"ExecuteActivity"\}
\
2024-03-15 16:13:40 \{"level":"debug","Attempt":1,"ChildWorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_23","Namespace":"default","RunID":"8b3d52f9-aade-4127-907e-a3ba716f21b7","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a80ab21f-e321-11ee-b860-0242ac120004","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-15T18:13:40-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 16:13:40 \{"level":"debug","Attempt":1,"ChildWorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_24","Namespace":"default","RunID":"8b3d52f9-aade-4127-907e-a3ba716f21b7","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a80ab21f-e321-11ee-b860-0242ac120004","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-15T18:13:40-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-14 16:46:18 \{"level":"info","time":"2024-03-14T18:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-14 19:46:18 \{"level":"info","time":"2024-03-14T21:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-14 22:46:18 \{"level":"info","time":"2024-03-15T00:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-14 23:00:00 \{"level":"info","time":"2024-03-15T01:00:00-05:00","message":"running prune videos task"\}
\
2024-03-15 01:46:18 \{"level":"info","time":"2024-03-15T03:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-15 04:46:18 \{"level":"info","time":"2024-03-15T06:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-15 07:46:18 \{"level":"info","time":"2024-03-15T09:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-15 10:46:18 \{"level":"info","time":"2024-03-15T12:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-15 13:46:18 \{"level":"info","time":"2024-03-15T15:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-15 16:13:38 \{"level":"debug","time":"2024-03-15T18:13:38-05:00","message":"creating folder: bobajenny/2024-03-15-drops_and_i_have_so_many_raids_to_do__!socials_!wishlist-live-a80ab21f-e321-11ee-b860-0242ac120004"\}
\
2024-03-15 16:13:39 \{"level":"debug","time":"2024-03-15T18:13:39-05:00","message":"getting live streams using the following query param: ?user_login=bobajenny"\}
\
2024-03-15 16:13:39 \{"level":"debug","time":"2024-03-15T18:13:39-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_bobajenny-1920x1080.jpg"\}
\
2024-03-15 16:13:40 \{"level":"debug","time":"2024-03-15T18:13:40-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_bobajenny-640x360.jpg"\}
\
2024-03-15 16:13:40 \{"level":"debug","time":"2024-03-15T18:13:40-05:00","message":"checking if tasks are done for video 43826283083"\}
\
2024-03-15 16:13:40 \{"level":"debug","Attempt":1,"Duration":600000,"Namespace":"default","RunID":"8c53010d-090a-451b-b8bb-0d931c25895b","TaskQueue":"archive","TimerID":"5","WorkerID":"0838558eb4c6","WorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-15T18:13:40-05:00","message":"NewTimer"\}
\
2024-03-15 16:13:40 \{"level":"debug","ActivityID":"5","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Namespace":"default","RunID":"a79f218f-3452-4c5f-b193-09ca3028d119","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-15T18:13:40-05:00","message":"ExecuteActivity"\}
\
2024-03-15 16:13:40 \{"level":"debug","Attempt":1,"ChildWorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_34","Namespace":"default","RunID":"8b3d52f9-aade-4127-907e-a3ba716f21b7","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a80ab21f-e321-11ee-b860-0242ac120004","WorkflowType":"ArchiveTwitchLiveVideoWorkflow","time":"2024-03-15T18:13:40-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 16:13:40 \{"level":"debug","Attempt":1,"ChildWorkflowID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af_5","Namespace":"default","RunID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_34","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-03-15T18:13:40-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 16:13:40 \{"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveVideo","Attempt":1,"Namespace":"default","RunID":"9ee1a450-ea40-4939-836b-908fd85a7a4f","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af_5","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-03-15T18:13:40-05:00","message":"ExecuteActivity"\}
\
2024-03-15 16:13:40 \{"level":"debug","time":"2024-03-15T18:13:40-05:00","message":"getting live streams using the following query param: ?user_login=bobajenny"\}
\
2024-03-15 16:13:40 \{"level":"debug","time":"2024-03-15T18:13:40-05:00","message":"checking if tasks are done for video 43826283083"\}
\
2024-03-15 16:13:41 \{"level":"debug","time":"2024-03-15T18:13:41-05:00","message":"proxy list: [\{https://eu.luminous.dev \} \{https://api.ttv.lol x-donate-to:https://ttv.lol/donate\}]"\}
\
2024-03-15 16:13:41 \{"level":"debug","time":"2024-03-15T18:13:41-05:00","message":"testing proxy server: https://eu.luminous.dev/playlist/bobajenny.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue"\}
\
2024-03-15 16:13:41 \{"level":"debug","time":"2024-03-15T18:13:41-05:00","message":"starting heartbeat download-livevideo-43826283083"\}
\
2024-03-15 16:13:41 \{"level":"debug","time":"2024-03-15T18:13:41-05:00","message":"proxy server test successful"\}
\
2024-03-15 16:13:41 \{"level":"debug","time":"2024-03-15T18:13:41-05:00","message":"proxy 0 is good"\}
\
2024-03-15 16:13:41 \{"level":"debug","time":"2024-03-15T18:13:41-05:00","message":"setting stream url to https://eu.luminous.dev/playlist/bobajenny.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue"\}
\
2024-03-15 16:13:41 \{"level":"debug","time":"2024-03-15T18:13:41-05:00","message":"streamlink live args: [--progress=force --force hls://https://eu.luminous.dev/playlist/bobajenny.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue best,best --twitch-low-latency --twitch-disable-hosting -o /tmp/43826283083_a80ab21f-e321-11ee-b860-0242ac120004-video.mp4]"\}
\
2024-03-15 16:13:41 \{"level":"debug","time":"2024-03-15T18:13:41-05:00","message":"running: streamlink --progress=force --force hls://https://eu.luminous.dev/playlist/bobajenny.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue best,best --twitch-low-latency --twitch-disable-hosting -o /tmp/43826283083_a80ab21f-e321-11ee-b860-0242ac120004-video.mp4"\}
\
2024-03-15 16:23:40 \{"level":"debug","time":"2024-03-15T18:23:40-05:00","message":"getting live streams using the following query param: ?user_login=bobajenny"\}
\
2024-03-15 16:23:40 \{"level":"debug","time":"2024-03-15T18:23:40-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_bobajenny-1920x1080.jpg"\}
\
2024-03-15 16:23:41 \{"level":"debug","time":"2024-03-15T18:23:41-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_bobajenny-640x360.jpg"\}
\
2024-03-15 16:23:42 \{"level":"debug","time":"2024-03-15T18:23:42-05:00","message":"checking if tasks are done for video 43826283083"\}
\
2024-03-15 16:46:18 \{"level":"info","time":"2024-03-15T18:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-15 19:46:18 \{"level":"info","time":"2024-03-15T21:46:18-05:00","message":"running check watched channel videos schedule"\}
\
2024-03-15 21:48:56 \{"level":"debug","time":"2024-03-15T23:48:56-05:00","message":"finished downloading live video for 43826283083"\}
\
2024-03-15 21:48:56 \{"level":"debug","time":"2024-03-15T23:48:56-05:00","message":"getting video duration"\}
\
2024-03-15 21:48:57 \{"level":"debug","time":"2024-03-15T23:48:57-05:00","message":"video duration: 20114"\}
\
2024-03-15 21:48:58 \{"level":"debug","time":"2024-03-15T23:48:58-05:00","message":"getting twitch videos for user: 61770627 with type archive and cursor eyJiIjpudWxsLCJhIjp7Ik9mZnNldCI6MjB9fQ"\}
\
2024-03-15 21:48:58 \{"level":"info","time":"2024-03-15T23:48:58-05:00","message":"found vod id 2091885635 for livestream 43826283083, updating database"\}
\
2024-03-15 21:48:58 \{"level":"debug","time":"2024-03-15T23:48:58-05:00","message":"stopping heartbeat download-livevideo-43826283083"\}
\
2024-03-15 21:48:58 \{"level":"debug","time":"2024-03-15T23:48:58-05:00","message":"checking if tasks are done for video 43826283083"\}
\
2024-03-15 16:23:40 \{"level":"debug","ActivityID":"10","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"8c53010d-090a-451b-b8bb-0d931c25895b","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-15T18:23:40-05:00","message":"ExecuteActivity"\}
\
2024-03-15 21:48:58 \{"level":"debug","Attempt":1,"ChildWorkflowID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af_14","Namespace":"default","RunID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_34","WorkflowType":"PostprocessVideoWorkflow","time":"2024-03-15T23:48:58-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 21:48:58 \{"level":"debug","ActivityID":"5","ActivityType":"PostprocessVideo","Attempt":1,"Namespace":"default","RunID":"1da8b29e-f09a-42d3-921e-0654a98ec9e6","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af_14","WorkflowType":"PostprocessVideoWorkflow","time":"2024-03-15T23:48:58-05:00","message":"ExecuteActivity"\}
\
2024-03-15 21:48:58 \{"level":"debug","Attempt":1,"ChildWorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_5","Namespace":"default","RunID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"808002e7-e350-11ee-b860-0242ac120004","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-15T23:48:58-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 21:48:58 \{"level":"debug","ActivityID":"5","ActivityType":"CreateDirectory","Attempt":1,"Namespace":"default","RunID":"adec7724-dc74-48af-8e64-4efd1023bf7d","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_5","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-15T23:48:58-05:00","message":"ExecuteActivity"\}
\
2024-03-15 21:48:58 \{"level":"debug","time":"2024-03-15T23:48:58-05:00","message":"video convert args: [-y -hide_banner -i /tmp/43826283083_a80ab21f-e321-11ee-b860-0242ac120004-video.mp4 -c:v copy -c:a copy /tmp/43826283083_a80ab21f-e321-11ee-b860-0242ac120004-video-convert.mp4]"\}
\
2024-03-15 21:48:58 \{"level":"debug","time":"2024-03-15T23:48:58-05:00","message":"starting heartbeat postprocess-video-43826283083"\}
\
2024-03-15 21:48:59 \{"level":"debug","time":"2024-03-15T23:48:59-05:00","message":"creating folder: bobajenny/2024-03-15-drops_''_EXTREME_VALTAN__!socials_!wishlist-live-808002e7-e350-11ee-b860-0242ac120004"\}
\
2024-03-15 21:48:59 \{"level":"debug","time":"2024-03-15T23:48:59-05:00","message":"getting live streams using the following query param: ?user_login=bobajenny"\}
\
2024-03-15 21:48:59 \{"level":"debug","time":"2024-03-15T23:48:59-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_bobajenny-1920x1080.jpg"\}
\
2024-03-15 21:49:00 \{"level":"debug","time":"2024-03-15T23:49:00-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_bobajenny-640x360.jpg"\}
\
2024-03-15 21:49:01 \{"level":"debug","time":"2024-03-15T23:49:01-05:00","message":"checking if tasks are done for video 43826283083"\}
\
2024-03-15 21:49:01 \{"level":"debug","time":"2024-03-15T23:49:01-05:00","message":"getting live streams using the following query param: ?user_login=bobajenny"\}
\
2024-03-15 21:49:01 \{"level":"debug","time":"2024-03-15T23:49:01-05:00","message":"checking if tasks are done for video 43826283083"\}
\
2024-03-15 21:49:01 \{"level":"debug","time":"2024-03-15T23:49:01-05:00","message":"proxy list: [\{https://eu.luminous.dev \} \{https://api.ttv.lol x-donate-to:https://ttv.lol/donate\}]"\}
\
2024-03-15 21:49:01 \{"level":"debug","time":"2024-03-15T23:49:01-05:00","message":"testing proxy server: https://eu.luminous.dev/playlist/bobajenny.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue"\}
\
2024-03-15 21:49:01 \{"level":"debug","time":"2024-03-15T23:49:01-05:00","message":"starting heartbeat download-livevideo-43826283083"\}
\
2024-03-15 21:48:59 \{"level":"debug","Attempt":1,"ChildWorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_14","Namespace":"default","RunID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"808002e7-e350-11ee-b860-0242ac120004","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-15T23:48:59-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 21:48:59 \{"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"57b1bfa1-b58c-478f-b261-c4e439bb9829","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_14","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-15T23:48:59-05:00","message":"ExecuteActivity"\}
\
2024-03-15 21:49:01 \{"level":"debug","Attempt":1,"ChildWorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_23","Namespace":"default","RunID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"808002e7-e350-11ee-b860-0242ac120004","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-15T23:49:01-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 21:49:01 \{"level":"debug","Attempt":1,"ChildWorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_24","Namespace":"default","RunID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"808002e7-e350-11ee-b860-0242ac120004","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-15T23:49:01-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 21:49:01 \{"level":"debug","ActivityID":"5","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Namespace":"default","RunID":"04a27cc9-f92d-486c-9aff-fbee6a0270ef","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-15T23:49:01-05:00","message":"ExecuteActivity"\}
\
2024-03-15 21:49:01 \{"level":"debug","Attempt":1,"Duration":600000,"Namespace":"default","RunID":"cc30f311-1352-42d0-a1b3-ac47ecf0dc36","TaskQueue":"archive","TimerID":"5","WorkerID":"0838558eb4c6","WorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-15T23:49:01-05:00","message":"NewTimer"\}
\
2024-03-15 21:49:01 \{"level":"debug","Attempt":1,"ChildWorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_34","Namespace":"default","RunID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"808002e7-e350-11ee-b860-0242ac120004","WorkflowType":"ArchiveTwitchLiveVideoWorkflow","time":"2024-03-15T23:49:01-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 21:49:01 \{"level":"debug","Attempt":1,"ChildWorkflowID":"540c06d5-5b0c-4fc5-a9c3-6b41e04154be_5","Namespace":"default","RunID":"540c06d5-5b0c-4fc5-a9c3-6b41e04154be","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a631b4b7-1d14-40b4-a1af-2ea2aa75c3b1_34","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-03-15T23:49:01-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 21:49:01 \{"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveVideo","Attempt":1,"Namespace":"default","RunID":"cf3f76a1-df64-4b8e-9e69-5fcab442df6f","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"540c06d5-5b0c-4fc5-a9c3-6b41e04154be_5","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-03-15T23:49:01-05:00","message":"ExecuteActivity"\}
\
2024-03-15 21:49:05 streamlink error: exit status 1\{"level":"error","ActivityType":"DownloadTwitchLiveVideo","Attempt":1,"Error":"exit status 1","Namespace":"default","RunID":"cf3f76a1-df64-4b8e-9e69-5fcab442df6f","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"540c06d5-5b0c-4fc5-a9c3-6b41e04154be_5","time":"2024-03-15T23:49:05-05:00","message":"Activity error."\}
\
2024-03-15 21:49:02 \{"level":"error","time":"2024-03-15T23:49:02-05:00","message":"proxy server test returned status code 404"\}
\
2024-03-15 21:49:02 \{"level":"debug","time":"2024-03-15T23:49:02-05:00","message":"testing proxy server: https://api.ttv.lol/playlist/bobajenny.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue"\}
\
2024-03-15 21:49:02 \{"level":"debug","time":"2024-03-15T23:49:02-05:00","message":"adding header x-donate-to:https://ttv.lol/donate to proxy server test"\}
\
2024-03-15 21:49:02 \{"level":"error","time":"2024-03-15T23:49:02-05:00","message":"proxy server test returned status code 500"\}
\
2024-03-15 21:49:02 \{"level":"debug","time":"2024-03-15T23:49:02-05:00","message":"streamlink live args: [--progress=force --force https://twitch.tv/bobajenny best,best --twitch-low-latency --twitch-disable-hosting -o /tmp/43826283083_808002e7-e350-11ee-b860-0242ac120004-video.mp4]"\}
\
2024-03-15 21:49:02 \{"level":"debug","time":"2024-03-15T23:49:02-05:00","message":"running: streamlink --progress=force --force https://twitch.tv/bobajenny best,best --twitch-low-latency --twitch-disable-hosting -o /tmp/43826283083_808002e7-e350-11ee-b860-0242ac120004-video.mp4"\}
\
2024-03-15 21:49:05 \{"level":"debug","time":"2024-03-15T23:49:05-05:00","message":"finished downloading live video for 43826283083 - exit status 1"\}
\
2024-03-15 21:49:05 \{"level":"debug","time":"2024-03-15T23:49:05-05:00","message":"streamlink live stdout: [cli][info] Found matching plugin twitch for URL https://twitch.tv/bobajenny\\nerror: No playable streams found on this URL: https://twitch.tv/bobajenny\\n"\}
\
2024-03-15 21:49:05 \{"level":"debug","time":"2024-03-15T23:49:05-05:00","message":"getting video duration"\}
\
2024-03-15 21:49:05 \{"level":"error","error":"exit status 1","time":"2024-03-15T23:49:05-05:00","message":"error getting video duration"\}
\
2024-03-15 21:49:05 \{"level":"debug","time":"2024-03-15T23:49:05-05:00","message":"stopping heartbeat download-livevideo-43826283083"\}
\
2024-03-15 21:50:41 \{"level":"debug","time":"2024-03-15T23:50:41-05:00","message":"finished vod video convert for 43826283083"\}
\
2024-03-15 21:50:41 \{"level":"debug","time":"2024-03-15T23:50:41-05:00","message":"stopping heartbeat postprocess-video-43826283083"\}
\
2024-03-15 21:50:41 \{"level":"debug","time":"2024-03-15T23:50:41-05:00","message":"checking if tasks are done for video 43826283083"\}
\
2024-03-15 21:50:41 \{"level":"debug","Attempt":1,"ChildWorkflowID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af_23","Namespace":"default","RunID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"8b3d52f9-aade-4127-907e-a3ba716f21b7_34","WorkflowType":"MoveVideoWorkflow","time":"2024-03-15T23:50:41-05:00","message":"ExecuteChildWorkflow"\}
\
2024-03-15 21:50:41 \{"level":"debug","ActivityID":"5","ActivityType":"MoveVideo","Attempt":1,"Namespace":"default","RunID":"3c5dc90d-a00e-4203-ab36-7df4c5fb1b40","TaskQueue":"archive","WorkerID":"0838558eb4c6","WorkflowID":"a799642e-e0ec-4be4-a8aa-d2e1c326f6af_23","WorkflowType":"MoveVideoWorkflow","time":"2024-03-15T23:50:41-05:00","message":"ExecuteActivity"\}
\
2024-03-15 21:50:41 \{"level":"debug","time":"2024-03-15T23:50:41-05:00","message":"moving file: /tmp/43826283083_a80ab21f-e321-11ee-b860-0242ac120004-video-convert.mp4 to /vods/bobajenny/2024-03-15-drops_and_i_have_so_many_raids_to_do__!socials_!wishlist-live-a80ab21f-e321-11ee-b860-0242ac120004/2024-03-15-bobajenny-live-video.mp4"\}
\
2024-03-15 21:50:41 \{"level":"debug","time":"2024-03-15T23:50:41-05:00","message":"starting heartbeat move-video-43826283083"\}
\
}

Caleb Contreras · Answer 6 · Mon Mar 18 2024 16:19:04 GMT+0800 (China Standard Time)

Update using an image with #385 branch

Happened again, the things that are different from past error are:

No proxy errors this time
VOD ID is the same on both tasks 50660554269
Even though the Workflow detected that a stream was not found it keeps querying the twitch api and makes 3 more attempts before sending the error notification to Discord.
Duplicate Live stream is archived notifications: After checking if tasks are done and verifying that all tasks indeed are finished it sends the notification but after this there's a Workflow panic and it seems that because both vods have the same external ID (like i mentioned in point 2) when it checks if the tasks on the newest item (the failed vod) are done since both external ID are the same it sends a second Livestream is archived notification.
I don't know to exactly what task this is referring to, if its for the one thats starting to convert and move the video or the other one: 2024-03-18 00:26:37 {"level":"info","time":"2024-03-18T02:26:37-05:00","message":"no vod found for livestream 50660554269, keeping live stream ID as external id"}

Workflow panic:

2024-03-18T07:39:39.544194211Z 2024-03-18T02:39:39-05:00 ERR Workflow panic | Attempt=1 Error=Potential deadlock detected: workflow goroutine "root" didn't yield for over a second Namespace=default RunID=31aa01be-3450-4a48-942d-cce3c904d666 StackTrace=process event for archive [panic]:
go.temporal.io/sdk/internal.(*coroutineState).call(0xc000f88690, 0x3b9aca00)
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1011 +0x1ab
go.temporal.io/sdk/internal.(*dispatcherImpl).ExecuteUntilAllBlocked(0xc0000a3560, 0x3b9aca00)
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1108 +0x199
go.temporal.io/sdk/internal.executeDispatcher({0x1809a20, 0xc0000a35c0}, {0x180e6a0, 0xc0000a3560}, 0x3b9aca00)
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:622 +0xcf
go.temporal.io/sdk/internal.(*syncWorkflowDefinition).OnWorkflowTaskStarted(0xc000f05980?, 0x9?)
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:592 +0x2c
go.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc000e84ae0, 0xc000f04900, 0x80?, 0x1)
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_event_handlers.go:1083 +0x229
go.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc000658900, 0xc00030b920)
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_task_handlers.go:1074 +0x153d
go.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0000baa90, 0xc00030b920, 0xc000658900, 0xc000c9c090)
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_task_handlers.go:868 +0x3bf
go.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc000655200, 0xc00030b920)
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_task_pollers.go:354 +0x3c3
go.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc000655200, {0x137e2a0, 0xc00030b920})
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_task_pollers.go:318 +0x78
go.temporal.io/sdk/internal.(*baseWorker).processTask(0xc000802c60, {0x137ede0, 0xc000e76400})
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker_base.go:505 +0x153
go.temporal.io/sdk/internal.(*baseWorker).processTaskAsync.func1()
	/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker_base.go:356 +0x45
created by go.temporal.io/sdk/internal.(*baseWorker).processTaskAsync in goroutine 122

Here is the full log

2024-03-16 21:57:13 usermod: no changes
2024-03-16 21:57:13 
2024-03-16 21:57:13 -------------------------------------
2024-03-16 21:57:13 User uid:    911
2024-03-16 21:57:13 User gid:    911
2024-03-16 21:57:13 -------------------------------------
2024-03-16 21:57:13 
2024-03-16 21:57:22 Version    : 
2024-03-16 21:57:22 Git Hash   : 
2024-03-16 21:57:22 Build Time : 2024-03-16T06:06:12Z
2024-03-16 21:57:23 2024/03/16 23:57:23 INFO  No logger configured for temporal client. Created default one.
2024-03-16 21:57:23 
2024-03-16 21:57:23    ____    __
2024-03-16 21:57:23   / __/___/ /  ___
2024-03-16 21:57:23  / _// __/ _ \/ _ \
2024-03-16 21:57:23 /___/\__/_//_/\___/ v4.11.4
2024-03-16 21:57:23 High performance, minimalist Go web framework
2024-03-16 21:57:23 https://echo.labstack.com
2024-03-16 21:57:23 ____________________________________O/_______
2024-03-16 21:57:23                                     O\
2024-03-16 21:57:23 ⇨ http server started on [::]:4000
2024-03-16 21:57:28 {"level":"info","Namespace":"default","TaskQueue":"video-download","WorkerID":"7cdee64dad4d","time":"2024-03-16T23:57:28-05:00","message":"Started Worker"}
2024-03-16 21:57:28 {"level":"info","Namespace":"default","TaskQueue":"video-convert","WorkerID":"7cdee64dad4d","time":"2024-03-16T23:57:28-05:00","message":"Started Worker"}
2024-03-16 21:57:28 {"level":"info","Namespace":"default","TaskQueue":"archive","WorkerID":"7cdee64dad4d","time":"2024-03-16T23:57:28-05:00","message":"Started Worker"}
2024-03-16 21:57:28 {"level":"info","Namespace":"default","TaskQueue":"chat-download","WorkerID":"7cdee64dad4d","time":"2024-03-16T23:57:28-05:00","message":"Started Worker"}
2024-03-16 21:57:28 {"level":"info","Namespace":"default","TaskQueue":"chat-render","WorkerID":"7cdee64dad4d","time":"2024-03-16T23:57:28-05:00","message":"Started Worker"}
2024-03-17 18:19:08 {"level":"debug","Attempt":1,"ChildWorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_5","Namespace":"default","RunID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"84fb582f-e4c5-11ee-8855-0242ac120002","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-17T20:19:08-05:00","message":"ExecuteChildWorkflow"}
2024-03-17 18:19:08 {"level":"debug","ActivityID":"5","ActivityType":"CreateDirectory","Attempt":1,"Namespace":"default","RunID":"9c9fba83-7c40-4c66-bcbc-4762da63de81","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_5","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-17T20:19:08-05:00","message":"ExecuteActivity"}
2024-03-17 18:19:09 {"level":"debug","Attempt":1,"ChildWorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_14","Namespace":"default","RunID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"84fb582f-e4c5-11ee-8855-0242ac120002","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-17T20:19:09-05:00","message":"ExecuteChildWorkflow"}
2024-03-17 18:19:09 {"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"e4fd20ac-cb87-4d15-b4f9-f0b4efd4b685","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_14","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-17T20:19:09-05:00","message":"ExecuteActivity"}
2024-03-17 18:19:11 {"level":"debug","Attempt":1,"ChildWorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_23","Namespace":"default","RunID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"84fb582f-e4c5-11ee-8855-0242ac120002","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-17T20:19:11-05:00","message":"ExecuteChildWorkflow"}
2024-03-17 18:19:11 {"level":"debug","Attempt":1,"ChildWorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_24","Namespace":"default","RunID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"84fb582f-e4c5-11ee-8855-0242ac120002","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-17T20:19:11-05:00","message":"ExecuteChildWorkflow"}
2024-03-17 18:19:11 {"level":"debug","ActivityID":"5","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Namespace":"default","RunID":"dd09bcce-5b38-491f-90d5-8fbab5fb3ef7","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-17T20:19:11-05:00","message":"ExecuteActivity"}
2024-03-17 18:19:11 {"level":"debug","Attempt":1,"Duration":600000,"Namespace":"default","RunID":"29f90370-3ee4-4987-b031-fec35d96fae3","TaskQueue":"archive","TimerID":"5","WorkerID":"7cdee64dad4d","WorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-17T20:19:11-05:00","message":"NewTimer"}
2024-03-17 18:19:11 {"level":"debug","Attempt":1,"ChildWorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_34","Namespace":"default","RunID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"84fb582f-e4c5-11ee-8855-0242ac120002","WorkflowType":"ArchiveTwitchLiveVideoWorkflow","time":"2024-03-17T20:19:11-05:00","message":"ExecuteChildWorkflow"}
2024-03-17 18:19:11 {"level":"debug","Attempt":1,"ChildWorkflowID":"6f386a12-37a7-4f7b-b295-07984f44f895_5","Namespace":"default","RunID":"6f386a12-37a7-4f7b-b295-07984f44f895","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_34","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-03-17T20:19:11-05:00","message":"ExecuteChildWorkflow"}
2024-03-17 18:19:11 {"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveVideo","Attempt":1,"Namespace":"default","RunID":"fbec0010-0079-4b3b-a5ca-3e84f16f8bc5","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"6f386a12-37a7-4f7b-b295-07984f44f895_5","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-03-17T20:19:11-05:00","message":"ExecuteActivity"}
2024-03-17 18:29:11 {"level":"debug","ActivityID":"10","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"29f90370-3ee4-4987-b031-fec35d96fae3","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-17T20:29:11-05:00","message":"ExecuteActivity"}
2024-03-16 21:57:24 {"level":"info","time":"2024-03-16T23:57:24-05:00","message":"authenticated with twitch"}
2024-03-16 21:57:27 {"level":"info","time":"2024-03-16T23:57:27-05:00","message":"Starting worker with config: {MAX_CHAT_DOWNLOAD_EXECUTIONS:5 MAX_CHAT_RENDER_EXECUTIONS:3 MAX_VIDEO_DOWNLOAD_EXECUTIONS:5 MAX_VIDEO_CONVERT_EXECUTIONS:3 TEMPORAL_URL:192.168.1.64:7233}"}
2024-03-16 21:57:27 {"level":"info","time":"2024-03-16T23:57:27-05:00","message":"config file found at /data/config.json, loading"}
2024-03-16 21:57:27 {"level":"info","time":"2024-03-16T23:57:27-05:00","message":"config file loaded: /data/config.json"}
2024-03-16 21:57:27 {"level":"debug","time":"2024-03-16T23:57:27-05:00","message":"config file loaded: /data/config.json"}
2024-03-16 21:57:27 {"level":"debug","time":"2024-03-16T23:57:27-05:00","message":"authenticating with twitch"}
2024-03-16 21:57:28 {"level":"info","time":"2024-03-16T23:57:28-05:00","message":"authenticated with twitch"}
2024-03-16 21:57:28 {"level":"debug","time":"2024-03-16T23:57:28-05:00","message":"setting up database connection"}
2024-03-16 21:57:28 {"level":"info","time":"2024-03-16T23:57:28-05:00","message":"setting up check watched channel videos schedule"}
2024-03-16 21:57:28 {"level":"info","time":"2024-03-16T23:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-16 23:00:00 {"level":"info","time":"2024-03-17T01:00:00-05:00","message":"running prune videos task"}
2024-03-17 00:57:28 {"level":"info","time":"2024-03-17T02:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-17 03:57:28 {"level":"info","time":"2024-03-17T05:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-17 06:57:28 {"level":"info","time":"2024-03-17T08:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-17 09:57:28 {"level":"info","time":"2024-03-17T11:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-17 12:57:28 {"level":"info","time":"2024-03-17T14:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-17 15:57:28 {"level":"info","time":"2024-03-17T17:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-17 18:19:08 {"level":"debug","time":"2024-03-17T20:19:08-05:00","message":"creating folder: kimchkim/2024-03-18-da_king_isa_here-live-84fb582f-e4c5-11ee-8855-0242ac120002"}
2024-03-17 18:19:09 {"level":"debug","time":"2024-03-17T20:19:09-05:00","message":"getting live streams using the following query param: ?user_login=kimchkim"}
2024-03-17 18:19:09 {"level":"debug","time":"2024-03-17T20:19:09-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_kimchkim-1920x1080.jpg"}
2024-03-17 18:19:10 {"level":"debug","time":"2024-03-17T20:19:10-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_kimchkim-640x360.jpg"}
2024-03-17 18:19:11 {"level":"debug","time":"2024-03-17T20:19:11-05:00","message":"checking if tasks are done for video 50660554269"}
2024-03-17 18:19:11 {"level":"debug","time":"2024-03-17T20:19:11-05:00","message":"getting live streams using the following query param: ?user_login=kimchkim"}
2024-03-17 18:19:11 {"level":"debug","time":"2024-03-17T20:19:11-05:00","message":"checking if tasks are done for video 50660554269"}
2024-03-17 18:19:11 {"level":"debug","time":"2024-03-17T20:19:11-05:00","message":"starting heartbeat download-livevideo-50660554269"}
2024-03-17 18:19:11 {"level":"debug","time":"2024-03-17T20:19:11-05:00","message":"proxy list: [{https://eu.luminous.dev } {https://api.ttv.lol x-donate-to:https://ttv.lol/donate}]"}
2024-03-17 18:19:11 {"level":"debug","time":"2024-03-17T20:19:11-05:00","message":"testing proxy server: https://eu.luminous.dev/playlist/kimchkim.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue"}
2024-03-17 18:19:12 {"level":"debug","time":"2024-03-17T20:19:12-05:00","message":"proxy server test successful"}
2024-03-17 18:19:12 {"level":"debug","time":"2024-03-17T20:19:12-05:00","message":"proxy 0 is good"}
2024-03-17 18:19:12 {"level":"debug","time":"2024-03-17T20:19:12-05:00","message":"setting stream url to https://eu.luminous.dev/playlist/kimchkim.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue"}
2024-03-17 18:19:12 {"level":"debug","time":"2024-03-17T20:19:12-05:00","message":"streamlink live args: [--progress=force --force hls://https://eu.luminous.dev/playlist/kimchkim.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue best,best --twitch-low-latency --twitch-disable-hosting -o /tmp/50660554269_84fb582f-e4c5-11ee-8855-0242ac120002-video.mp4]"}
2024-03-17 18:19:12 {"level":"debug","time":"2024-03-17T20:19:12-05:00","message":"running: streamlink --progress=force --force hls://https://eu.luminous.dev/playlist/kimchkim.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue best,best --twitch-low-latency --twitch-disable-hosting -o /tmp/50660554269_84fb582f-e4c5-11ee-8855-0242ac120002-video.mp4"}
2024-03-17 18:29:11 {"level":"debug","time":"2024-03-17T20:29:11-05:00","message":"getting live streams using the following query param: ?user_login=kimchkim"}
2024-03-17 18:29:11 {"level":"debug","time":"2024-03-17T20:29:11-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_kimchkim-1920x1080.jpg"}
2024-03-17 18:29:12 {"level":"debug","time":"2024-03-17T20:29:12-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_kimchkim-640x360.jpg"}
2024-03-17 18:29:13 {"level":"debug","time":"2024-03-17T20:29:13-05:00","message":"checking if tasks are done for video 50660554269"}
2024-03-17 18:57:28 {"level":"info","time":"2024-03-17T20:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-17 21:57:28 {"level":"info","time":"2024-03-17T23:57:28-05:00","message":"running check watched channel videos schedule"}
2024-03-17 23:00:00 {"level":"info","time":"2024-03-18T01:00:00-05:00","message":"running prune videos task"}
2024-03-18 00:26:35 {"level":"debug","time":"2024-03-18T02:26:35-05:00","message":"finished downloading live video for 50660554269"}
2024-03-18 00:26:35 {"level":"debug","time":"2024-03-18T02:26:35-05:00","message":"getting video duration"}
2024-03-18 00:26:37 {"level":"debug","time":"2024-03-18T02:26:37-05:00","message":"video duration: 22043"}
2024-03-18 00:26:37 {"level":"info","time":"2024-03-18T02:26:37-05:00","message":"no vod found for livestream 50660554269, keeping live stream ID as external id"}
2024-03-18 00:26:37 {"level":"debug","time":"2024-03-18T02:26:37-05:00","message":"stopping heartbeat download-livevideo-50660554269"}
2024-03-18 00:26:37 {"level":"debug","time":"2024-03-18T02:26:37-05:00","message":"checking if tasks are done for video 50660554269"}
2024-03-18 00:26:38 {"level":"debug","time":"2024-03-18T02:26:38-05:00","message":"video convert args: [-y -hide_banner -i /tmp/50660554269_84fb582f-e4c5-11ee-8855-0242ac120002-video.mp4 -c:v copy -c:a copy /tmp/50660554269_84fb582f-e4c5-11ee-8855-0242ac120002-video-convert.mp4]"}
2024-03-18 00:26:38 {"level":"debug","time":"2024-03-18T02:26:38-05:00","message":"starting heartbeat postprocess-video-50660554269"}
2024-03-18 00:26:37 {"level":"debug","Attempt":1,"ChildWorkflowID":"6f386a12-37a7-4f7b-b295-07984f44f895_14","Namespace":"default","RunID":"6f386a12-37a7-4f7b-b295-07984f44f895","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_34","WorkflowType":"PostprocessVideoWorkflow","time":"2024-03-18T02:26:37-05:00","message":"ExecuteChildWorkflow"}
2024-03-18 00:26:38 {"level":"debug","ActivityID":"5","ActivityType":"PostprocessVideo","Attempt":1,"Namespace":"default","RunID":"b57acf03-efc0-4b81-8322-ce88cf25fb24","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"6f386a12-37a7-4f7b-b295-07984f44f895_14","WorkflowType":"PostprocessVideoWorkflow","time":"2024-03-18T02:26:38-05:00","message":"ExecuteActivity"}
2024-03-18 00:26:39 {"level":"debug","Attempt":1,"ChildWorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_5","Namespace":"default","RunID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"dbf5a1fe-e4f8-11ee-8855-0242ac120002","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-18T02:26:39-05:00","message":"ExecuteChildWorkflow"}
2024-03-18 00:26:39 {"level":"debug","ActivityID":"5","ActivityType":"CreateDirectory","Attempt":1,"Namespace":"default","RunID":"d379d06f-56cd-4334-98b1-b3c46399c4c0","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_5","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-18T02:26:39-05:00","message":"ExecuteActivity"}
2024-03-18 00:26:40 {"level":"debug","Attempt":1,"ChildWorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_14","Namespace":"default","RunID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"dbf5a1fe-e4f8-11ee-8855-0242ac120002","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-18T02:26:40-05:00","message":"ExecuteChildWorkflow"}
2024-03-18 00:26:40 {"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"9c856625-1116-4ac0-84c4-fd4b398c399a","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_14","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-18T02:26:40-05:00","message":"ExecuteActivity"}
2024-03-18 00:26:42 {"level":"debug","Attempt":1,"ChildWorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_23","Namespace":"default","RunID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"dbf5a1fe-e4f8-11ee-8855-0242ac120002","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-18T02:26:42-05:00","message":"ExecuteChildWorkflow"}
2024-03-18 00:26:42 {"level":"debug","Attempt":1,"ChildWorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_24","Namespace":"default","RunID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"dbf5a1fe-e4f8-11ee-8855-0242ac120002","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-18T02:26:42-05:00","message":"ExecuteChildWorkflow"}
2024-03-18 00:26:42 {"level":"debug","ActivityID":"5","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Namespace":"default","RunID":"24d376ab-ffc1-4bea-b361-130f4fc3d229","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-18T02:26:42-05:00","message":"ExecuteActivity"}
2024-03-18 00:26:42 {"level":"debug","Attempt":1,"Duration":600000,"Namespace":"default","RunID":"3d651067-3395-4488-9e3b-fd721572a91d","TaskQueue":"archive","TimerID":"5","WorkerID":"7cdee64dad4d","WorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-18T02:26:42-05:00","message":"NewTimer"}
2024-03-18 00:26:42 {"level":"error","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Error":"no stream found for channel kimchkim","Namespace":"default","RunID":"24d376ab-ffc1-4bea-b361-130f4fc3d229","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_24","time":"2024-03-18T02:26:42-05:00","message":"Activity error."}
2024-03-18 00:27:43 {"level":"error","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":2,"Error":"no stream found for channel kimchkim","Namespace":"default","RunID":"24d376ab-ffc1-4bea-b361-130f4fc3d229","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_24","time":"2024-03-18T02:27:43-05:00","message":"Activity error."}
2024-03-18 00:26:39 {"level":"debug","time":"2024-03-18T02:26:39-05:00","message":"creating folder: kimchkim/2024-03-18-da_king_isa_here-live-dbf5a1fe-e4f8-11ee-8855-0242ac120002"}
2024-03-18 00:26:40 {"level":"debug","time":"2024-03-18T02:26:40-05:00","message":"getting live streams using the following query param: ?user_login=kimchkim"}
2024-03-18 00:26:40 {"level":"debug","time":"2024-03-18T02:26:40-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_kimchkim-1920x1080.jpg"}
2024-03-18 00:26:41 {"level":"debug","time":"2024-03-18T02:26:41-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_kimchkim-640x360.jpg"}
2024-03-18 00:26:42 {"level":"debug","time":"2024-03-18T02:26:42-05:00","message":"checking if tasks are done for video 50660554269"}
2024-03-18 00:26:42 {"level":"debug","time":"2024-03-18T02:26:42-05:00","message":"getting live streams using the following query param: ?user_login=kimchkim"}
2024-03-18 00:27:43 {"level":"debug","time":"2024-03-18T02:27:43-05:00","message":"getting live streams using the following query param: ?user_login=kimchkim"}
2024-03-18 00:28:58 {"level":"debug","time":"2024-03-18T02:28:58-05:00","message":"finished vod video convert for 50660554269"}
2024-03-18 00:28:58 {"level":"debug","time":"2024-03-18T02:28:58-05:00","message":"stopping heartbeat postprocess-video-50660554269"}
2024-03-18 00:28:58 {"level":"debug","time":"2024-03-18T02:28:58-05:00","message":"checking if tasks are done for video 50660554269"}
2024-03-18 00:28:58 {"level":"debug","Attempt":1,"ChildWorkflowID":"6f386a12-37a7-4f7b-b295-07984f44f895_23","Namespace":"default","RunID":"6f386a12-37a7-4f7b-b295-07984f44f895","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"f5dfc6da-d0a4-44d0-a205-9c86438a02f6_34","WorkflowType":"MoveVideoWorkflow","time":"2024-03-18T02:28:58-05:00","message":"ExecuteChildWorkflow"}
2024-03-18 00:28:58 {"level":"debug","ActivityID":"5","ActivityType":"MoveVideo","Attempt":1,"Namespace":"default","RunID":"31aa01be-3450-4a48-942d-cce3c904d666","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"6f386a12-37a7-4f7b-b295-07984f44f895_23","WorkflowType":"MoveVideoWorkflow","time":"2024-03-18T02:28:58-05:00","message":"ExecuteActivity"}
2024-03-18 00:28:58 {"level":"debug","time":"2024-03-18T02:28:58-05:00","message":"moving file: /tmp/50660554269_84fb582f-e4c5-11ee-8855-0242ac120002-video-convert.mp4 to /vods/kimchkim/2024-03-18-da_king_isa_here-live-84fb582f-e4c5-11ee-8855-0242ac120002/2024-03-18-kimchkim-live-video.mp4"}
2024-03-18 00:28:58 {"level":"debug","time":"2024-03-18T02:28:58-05:00","message":"starting heartbeat move-video-50660554269"}
2024-03-18 00:29:43 {"level":"debug","time":"2024-03-18T02:29:43-05:00","message":"getting live streams using the following query param: ?user_login=kimchkim"}
2024-03-18 00:29:43 {"level":"error","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":3,"Error":"no stream found for channel kimchkim","Namespace":"default","RunID":"24d376ab-ffc1-4bea-b361-130f4fc3d229","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"a9db421b-cdd8-4c50-9318-dbb6ab8d25c4_24","time":"2024-03-18T02:29:43-05:00","message":"Activity error."}
2024-03-18 00:39:38 {"level":"debug","time":"2024-03-18T02:39:38-05:00","message":"deleting file: /tmp/50660554269_84fb582f-e4c5-11ee-8855-0242ac120002-video.mp4"}
2024-03-18 00:39:38 {"level":"debug","time":"2024-03-18T02:39:38-05:00","message":"stopping heartbeat move-video-50660554269"}
2024-03-18 00:39:38 {"level":"debug","time":"2024-03-18T02:39:38-05:00","message":"checking if tasks are done for video 50660554269"}
2024-03-18 00:39:38 {"level":"debug","time":"2024-03-18T02:39:38-05:00","message":"all tasks for video 50660554269 are done"}
2024-03-18 00:39:39 {"level":"error","Attempt":1,"Error":"Potential deadlock detected: workflow goroutine \"root\" didn't yield for over a second","Namespace":"default","RunID":"31aa01be-3450-4a48-942d-cce3c904d666","StackTrace":"process event for archive [panic]:\ngo.temporal.io/sdk/internal.(*coroutineState).call(0xc000f88690, 0x3b9aca00)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1011 +0x1ab\ngo.temporal.io/sdk/internal.(*dispatcherImpl).ExecuteUntilAllBlocked(0xc0000a3560, 0x3b9aca00)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1108 +0x199\ngo.temporal.io/sdk/internal.executeDispatcher({0x1809a20, 0xc0000a35c0}, {0x180e6a0, 0xc0000a3560}, 0x3b9aca00)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:622 +0xcf\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).OnWorkflowTaskStarted(0xc000f05980?, 0x9?)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:592 +0x2c\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc000e84ae0, 0xc000f04900, 0x80?, 0x1)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_event_handlers.go:1083 +0x229\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc000658900, 0xc00030b920)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_task_handlers.go:1074 +0x153d\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc0000baa90, 0xc00030b920, 0xc000658900, 0xc000c9c090)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_task_handlers.go:868 +0x3bf\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc000655200, 0xc00030b920)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_task_pollers.go:354 +0x3c3\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc000655200, {0x137e2a0, 0xc00030b920})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_task_pollers.go:318 +0x78\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc000802c60, {0x137ede0, 0xc000e76400})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker_base.go:505 +0x153\ngo.temporal.io/sdk/internal.(*baseWorker).processTaskAsync.func1()\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker_base.go:356 +0x45\ncreated by go.temporal.io/sdk/internal.(*baseWorker).processTaskAsync in goroutine 122\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker_base.go:352 +0xa5","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"6f386a12-37a7-4f7b-b295-07984f44f895_23","WorkflowType":"MoveVideoWorkflow","time":"2024-03-18T02:39:39-05:00","message":"Workflow panic"}
2024-03-18 00:39:39 {"level":"warn","Attempt":1,"Error":"Potential deadlock detected: workflow goroutine \"root\" didn't yield for over a second","Namespace":"default","RunID":"31aa01be-3450-4a48-942d-cce3c904d666","TaskQueue":"archive","WorkerID":"7cdee64dad4d","WorkflowID":"6f386a12-37a7-4f7b-b295-07984f44f895_23","WorkflowType":"MoveVideoWorkflow","time":"2024-03-18T02:39:39-05:00","message":"Failed to process workflow task."}
2024-03-18 00:39:42 {"level":"debug","time":"2024-03-18T02:39:42-05:00","message":"checking if tasks are done for video 50660554269"}
2024-03-18 00:39:42 {"level":"debug","time":"2024-03-18T02:39:42-05:00","message":"all tasks for video 50660554269 are done"}

Isaac · Answer 7 · Tue Mar 19 2024 07:10:20 GMT+0800 (China Standard Time)

I did not add the cleanup logic to the save info workflow as I was wanting to test if this worked before implementing further. I've updated #385 to cleanup the archive if the task fails in the save info activity. Pull and build the latest version of that branch and give it another try.

If/when this error occurs again:

Do multiple webhook notifications get sent?

I'll try to recreate this with a low check interval using the same streams in this issue to get some more data.

Caleb Contreras · Answer 8 · Wed Mar 20 2024 10:42:07 GMT+0800 (China Standard Time)

With last update in #385 :
1- No multiple webhook notification this time
2 - Workflows b5153ab0-e65b-11ee-8e4d-0242ac120004 and cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24 it seems like both got "no stream found" but failed to be cancelled (kept running after that)

Both tried to cancelled but got the following error (full log at the end):

2024-03-19 18:53:15 {"level":"error","Attempt":9,"Error":"runtime error: invalid memory address or nil pointer dereference",....

3 - Queue item has Download Thumbnail and Save info both as failed and item status as ERROR didn't seem to clean up directory/queue items for given VOD ID b5153ab0-e65b-11ee-8e4d-0242ac120004

Workflows

Workflow 1 - cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24

Workflow 2 - b5153ab0-e65b-11ee-8e4d-0242ac120004

Full log:

2024-03-18 22:43:28 usermod: no changes
2024-03-18 22:43:28 
2024-03-18 22:43:28 -------------------------------------
2024-03-18 22:43:28 User uid:    911
2024-03-18 22:43:28 User gid:    911
2024-03-18 22:43:28 -------------------------------------
2024-03-18 22:43:28 
2024-03-18 22:43:39 Version    : 
2024-03-18 22:43:39 Git Hash   : 
2024-03-18 22:43:39 Build Time : 2024-03-19T05:35:03Z
2024-03-18 22:43:39 2024/03/19 00:43:39 INFO  No logger configured for temporal client. Created default one.
2024-03-18 22:43:39 
2024-03-18 22:43:39    ____    __
2024-03-18 22:43:39   / __/___/ /  ___
2024-03-18 22:43:39  / _// __/ _ \/ _ \
2024-03-18 22:43:39 /___/\__/_//_/\___/ v4.11.4
2024-03-18 22:43:39 High performance, minimalist Go web framework
2024-03-18 22:43:39 https://echo.labstack.com
2024-03-18 22:43:39 ____________________________________O/_______
2024-03-18 22:43:39                                     O\
2024-03-18 22:43:39 ⇨ http server started on [::]:4000
2024-03-18 22:43:45 {"level":"info","Namespace":"default","TaskQueue":"archive","WorkerID":"615cb419001f","time":"2024-03-19T00:43:45-05:00","message":"Started Worker"}
2024-03-18 22:43:45 {"level":"info","Namespace":"default","TaskQueue":"chat-download","WorkerID":"615cb419001f","time":"2024-03-19T00:43:45-05:00","message":"Started Worker"}
2024-03-18 22:43:45 {"level":"info","Namespace":"default","TaskQueue":"chat-render","WorkerID":"615cb419001f","time":"2024-03-19T00:43:45-05:00","message":"Started Worker"}
2024-03-18 22:43:45 {"level":"info","Namespace":"default","TaskQueue":"video-download","WorkerID":"615cb419001f","time":"2024-03-19T00:43:45-05:00","message":"Started Worker"}
2024-03-18 22:43:45 {"level":"info","Namespace":"default","TaskQueue":"video-convert","WorkerID":"615cb419001f","time":"2024-03-19T00:43:45-05:00","message":"Started Worker"}
2024-03-19 16:37:25 {"level":"debug","Attempt":1,"ChildWorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_5","Namespace":"default","RunID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"a3c69867-e649-11ee-8e4d-0242ac120004","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-19T18:37:25-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 16:37:25 {"level":"debug","ActivityID":"5","ActivityType":"CreateDirectory","Attempt":1,"Namespace":"default","RunID":"d3a062c8-c91c-4087-9d71-14b0c6b531b1","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_5","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-19T18:37:25-05:00","message":"ExecuteActivity"}
2024-03-19 16:37:25 {"level":"debug","Attempt":1,"ChildWorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_14","Namespace":"default","RunID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"a3c69867-e649-11ee-8e4d-0242ac120004","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-19T18:37:25-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 16:37:25 {"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"c63005cc-bff5-4762-924a-cb3a89f37510","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_14","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-19T18:37:25-05:00","message":"ExecuteActivity"}
2024-03-19 16:37:27 {"level":"debug","Attempt":1,"ChildWorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_23","Namespace":"default","RunID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"a3c69867-e649-11ee-8e4d-0242ac120004","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-19T18:37:27-05:00","message":"ExecuteChildWorkflow"}
2024-03-18 22:43:39 {"level":"info","time":"2024-03-19T00:43:39-05:00","message":"config file found at /data/config.json, loading"}
2024-03-18 22:43:39 {"level":"info","time":"2024-03-19T00:43:39-05:00","message":"config file loaded: /data/config.json"}
2024-03-18 22:43:39 {"level":"debug","time":"2024-03-19T00:43:39-05:00","message":"config file loaded: /data/config.json"}
2024-03-18 22:43:39 {"level":"info","time":"2024-03-19T00:43:39-05:00","message":"Connected to temporal at 192.168.1.64:7233"}
2024-03-18 22:43:40 {"level":"info","time":"2024-03-19T00:43:40-05:00","message":"authenticated with twitch"}
2024-03-18 22:43:44 {"level":"info","time":"2024-03-19T00:43:44-05:00","message":"Starting worker with config: {MAX_CHAT_DOWNLOAD_EXECUTIONS:5 MAX_CHAT_RENDER_EXECUTIONS:3 MAX_VIDEO_DOWNLOAD_EXECUTIONS:5 MAX_VIDEO_CONVERT_EXECUTIONS:3 TEMPORAL_URL:192.168.1.64:7233}"}
2024-03-18 22:43:44 {"level":"info","time":"2024-03-19T00:43:44-05:00","message":"config file found at /data/config.json, loading"}
2024-03-18 22:43:44 {"level":"info","time":"2024-03-19T00:43:44-05:00","message":"config file loaded: /data/config.json"}
2024-03-18 22:43:44 {"level":"debug","time":"2024-03-19T00:43:44-05:00","message":"config file loaded: /data/config.json"}
2024-03-18 22:43:44 {"level":"debug","time":"2024-03-19T00:43:44-05:00","message":"authenticating with twitch"}
2024-03-18 22:43:44 {"level":"info","time":"2024-03-19T00:43:44-05:00","message":"setting up check watched channel videos schedule"}
2024-03-18 22:43:44 {"level":"info","time":"2024-03-19T00:43:44-05:00","message":"running check watched channel videos schedule"}
2024-03-18 22:43:45 {"level":"info","time":"2024-03-19T00:43:45-05:00","message":"authenticated with twitch"}
2024-03-18 22:43:45 {"level":"debug","time":"2024-03-19T00:43:45-05:00","message":"setting up database connection"}
2024-03-18 23:00:00 {"level":"info","time":"2024-03-19T01:00:00-05:00","message":"running prune videos task"}
2024-03-19 01:43:44 {"level":"info","time":"2024-03-19T03:43:44-05:00","message":"running check watched channel videos schedule"}
2024-03-19 04:43:44 {"level":"info","time":"2024-03-19T06:43:44-05:00","message":"running check watched channel videos schedule"}
2024-03-19 07:43:44 {"level":"info","time":"2024-03-19T09:43:44-05:00","message":"running check watched channel videos schedule"}
2024-03-19 10:43:44 {"level":"info","time":"2024-03-19T12:43:44-05:00","message":"running check watched channel videos schedule"}
2024-03-19 13:43:44 {"level":"info","time":"2024-03-19T15:43:44-05:00","message":"running check watched channel videos schedule"}
2024-03-19 16:37:25 {"level":"debug","time":"2024-03-19T18:37:25-05:00","message":"creating folder: chominnnn/2024-03-19-let's_get_distracted_on_our_work_together~-live-a3c69867-e649-11ee-8e4d-0242ac120004"}
2024-03-19 16:37:25 {"level":"debug","time":"2024-03-19T18:37:25-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 16:37:25 {"level":"debug","time":"2024-03-19T18:37:25-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_chominnnn-1920x1080.jpg"}
2024-03-19 16:37:26 {"level":"debug","time":"2024-03-19T18:37:26-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_chominnnn-640x360.jpg"}
2024-03-19 16:37:27 {"level":"debug","time":"2024-03-19T18:37:27-05:00","message":"checking if tasks are done for video 42097172920"}
2024-03-19 16:37:27 {"level":"debug","Attempt":1,"ChildWorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_24","Namespace":"default","RunID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"a3c69867-e649-11ee-8e4d-0242ac120004","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T18:37:27-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 16:37:27 {"level":"debug","Attempt":1,"Duration":600000,"Namespace":"default","RunID":"6eed408e-59a6-43a3-a44d-a0a6b4fd0614","TaskQueue":"archive","TimerID":"5","WorkerID":"615cb419001f","WorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-19T18:37:27-05:00","message":"NewTimer"}
2024-03-19 16:37:27 {"level":"debug","ActivityID":"5","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Namespace":"default","RunID":"acecf5e8-1d74-4150-b76f-dcedfe7a47da","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T18:37:27-05:00","message":"ExecuteActivity"}
2024-03-19 16:37:27 {"level":"debug","time":"2024-03-19T18:37:27-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 16:37:28 {"level":"debug","time":"2024-03-19T18:37:28-05:00","message":"checking if tasks are done for video 42097172920"}
2024-03-19 16:37:28 {"level":"debug","time":"2024-03-19T18:37:28-05:00","message":"proxy list: [{https://eu.luminous.dev } {https://api.ttv.lol x-donate-to:https://ttv.lol/donate}]"}
2024-03-19 16:37:28 {"level":"debug","Attempt":1,"ChildWorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_34","Namespace":"default","RunID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"a3c69867-e649-11ee-8e4d-0242ac120004","WorkflowType":"ArchiveTwitchLiveVideoWorkflow","time":"2024-03-19T18:37:28-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 16:37:28 {"level":"debug","Attempt":1,"ChildWorkflowID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7_5","Namespace":"default","RunID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_34","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-03-19T18:37:28-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 16:37:28 {"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveVideo","Attempt":1,"Namespace":"default","RunID":"1c2b8b54-fee4-47da-b019-73b10321d973","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7_5","WorkflowType":"DownloadTwitchLiveVideoWorkflow","time":"2024-03-19T18:37:28-05:00","message":"ExecuteActivity"}
2024-03-19 16:47:27 {"level":"debug","ActivityID":"10","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"6eed408e-59a6-43a3-a44d-a0a6b4fd0614","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-19T18:47:27-05:00","message":"ExecuteActivity"}
2024-03-19 18:46:39 {"level":"debug","Attempt":1,"ChildWorkflowID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7_14","Namespace":"default","RunID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_34","WorkflowType":"PostprocessVideoWorkflow","time":"2024-03-19T20:46:39-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 18:46:39 {"level":"debug","ActivityID":"5","ActivityType":"PostprocessVideo","Attempt":1,"Namespace":"default","RunID":"d294d92e-7bb7-458c-8cf2-b31c949dae70","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7_14","WorkflowType":"PostprocessVideoWorkflow","time":"2024-03-19T20:46:39-05:00","message":"ExecuteActivity"}
2024-03-19 16:37:28 {"level":"debug","time":"2024-03-19T18:37:28-05:00","message":"testing proxy server: https://eu.luminous.dev/playlist/chominnnn.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue"}
2024-03-19 16:37:28 {"level":"debug","time":"2024-03-19T18:37:28-05:00","message":"starting heartbeat download-livevideo-42097172920"}
2024-03-19 16:37:29 {"level":"debug","time":"2024-03-19T18:37:29-05:00","message":"proxy server test successful"}
2024-03-19 16:37:29 {"level":"debug","time":"2024-03-19T18:37:29-05:00","message":"proxy 0 is good"}
2024-03-19 16:37:29 {"level":"debug","time":"2024-03-19T18:37:29-05:00","message":"setting stream url to https://eu.luminous.dev/playlist/chominnnn.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue"}
2024-03-19 16:37:29 {"level":"debug","time":"2024-03-19T18:37:29-05:00","message":"streamlink live args: [--progress=force --force hls://https://eu.luminous.dev/playlist/chominnnn.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue best,best --twitch-low-latency --twitch-disable-hosting -o /tmp/42097172920_a3c69867-e649-11ee-8e4d-0242ac120004-video.mp4]"}
2024-03-19 16:37:29 {"level":"debug","time":"2024-03-19T18:37:29-05:00","message":"running: streamlink --progress=force --force hls://https://eu.luminous.dev/playlist/chominnnn.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue best,best --twitch-low-latency --twitch-disable-hosting -o /tmp/42097172920_a3c69867-e649-11ee-8e4d-0242ac120004-video.mp4"}
2024-03-19 16:43:44 {"level":"info","time":"2024-03-19T18:43:44-05:00","message":"running check watched channel videos schedule"}
2024-03-19 16:47:27 {"level":"debug","time":"2024-03-19T18:47:27-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 16:47:28 {"level":"debug","time":"2024-03-19T18:47:28-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_chominnnn-1920x1080.jpg"}
2024-03-19 16:47:29 {"level":"debug","time":"2024-03-19T18:47:29-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_chominnnn-640x360.jpg"}
2024-03-19 16:47:30 {"level":"debug","time":"2024-03-19T18:47:30-05:00","message":"checking if tasks are done for video 42097172920"}
2024-03-19 18:46:38 {"level":"debug","time":"2024-03-19T20:46:38-05:00","message":"finished downloading live video for 42097172920"}
2024-03-19 18:46:38 {"level":"debug","time":"2024-03-19T20:46:38-05:00","message":"getting video duration"}
2024-03-19 18:46:39 {"level":"debug","time":"2024-03-19T20:46:39-05:00","message":"video duration: 7751"}
2024-03-19 18:46:39 {"level":"debug","time":"2024-03-19T20:46:39-05:00","message":"getting twitch videos for user: 515887757 with type archive and cursor eyJiIjpudWxsLCJhIjp7Ik9mZnNldCI6OH19"}
2024-03-19 18:46:39 {"level":"info","time":"2024-03-19T20:46:39-05:00","message":"no vod found for livestream 42097172920, keeping live stream ID as external id"}
2024-03-19 18:46:39 {"level":"debug","time":"2024-03-19T20:46:39-05:00","message":"stopping heartbeat download-livevideo-42097172920"}
2024-03-19 18:46:39 {"level":"debug","time":"2024-03-19T20:46:39-05:00","message":"checking if tasks are done for video 42097172920"}
2024-03-19 18:46:39 {"level":"debug","time":"2024-03-19T20:46:39-05:00","message":"video convert args: [-y -hide_banner -i /tmp/42097172920_a3c69867-e649-11ee-8e4d-0242ac120004-video.mp4 -c:v copy -c:a copy /tmp/42097172920_a3c69867-e649-11ee-8e4d-0242ac120004-video-convert.mp4]"}
2024-03-19 18:46:39 {"level":"debug","time":"2024-03-19T20:46:39-05:00","message":"starting heartbeat postprocess-video-42097172920"}
2024-03-19 18:46:45 {"level":"debug","time":"2024-03-19T20:46:45-05:00","message":"creating folder: chominnnn/2024-03-19-let's_get_distracted_on_our_work_together~-live-b5153ab0-e65b-11ee-8e4d-0242ac120004"}
2024-03-19 18:46:45 {"level":"debug","time":"2024-03-19T20:46:45-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 18:46:45 {"level":"debug","time":"2024-03-19T20:46:45-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_chominnnn-1920x1080.jpg"}
2024-03-19 18:46:46 {"level":"debug","time":"2024-03-19T20:46:46-05:00","message":"downloading file: https://static-cdn.jtvnw.net/previews-ttv/live_user_chominnnn-640x360.jpg"}
2024-03-19 18:46:47 {"level":"debug","time":"2024-03-19T20:46:47-05:00","message":"checking if tasks are done for video 42097172920"}
2024-03-19 18:46:47 {"level":"debug","time":"2024-03-19T20:46:47-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 18:47:39 {"level":"debug","time":"2024-03-19T20:47:39-05:00","message":"finished vod video convert for 42097172920"}
2024-03-19 18:47:39 {"level":"debug","time":"2024-03-19T20:47:39-05:00","message":"stopping heartbeat postprocess-video-42097172920"}
2024-03-19 18:47:39 {"level":"debug","time":"2024-03-19T20:47:39-05:00","message":"checking if tasks are done for video 42097172920"}
2024-03-19 18:47:40 {"level":"debug","time":"2024-03-19T20:47:40-05:00","message":"moving file: /tmp/42097172920_a3c69867-e649-11ee-8e4d-0242ac120004-video-convert.mp4 to /vods/chominnnn/2024-03-19-let's_get_distracted_on_our_work_together~-live-a3c69867-e649-11ee-8e4d-0242ac120004/2024-03-19-chominnnn-live-video.mp4"}
2024-03-19 18:47:40 {"level":"debug","time":"2024-03-19T20:47:40-05:00","message":"starting heartbeat move-video-42097172920"}
2024-03-19 18:47:47 {"level":"debug","time":"2024-03-19T20:47:47-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 18:49:47 {"level":"debug","time":"2024-03-19T20:49:47-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 18:46:45 {"level":"debug","Attempt":1,"ChildWorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_5","Namespace":"default","RunID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"b5153ab0-e65b-11ee-8e4d-0242ac120004","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-19T20:46:45-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 18:46:45 {"level":"debug","ActivityID":"5","ActivityType":"CreateDirectory","Attempt":1,"Namespace":"default","RunID":"a656d16e-8ef5-436e-bfc3-973f64100926","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_5","WorkflowType":"CreateDirectoryWorkflow","time":"2024-03-19T20:46:45-05:00","message":"ExecuteActivity"}
2024-03-19 18:46:45 {"level":"debug","Attempt":1,"ChildWorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_14","Namespace":"default","RunID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"b5153ab0-e65b-11ee-8e4d-0242ac120004","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-19T20:46:45-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 18:46:45 {"level":"debug","ActivityID":"5","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"e836913f-cefb-42ac-8ee4-a5cb69b2180e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_14","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflow","time":"2024-03-19T20:46:45-05:00","message":"ExecuteActivity"}
2024-03-19 18:46:47 {"level":"debug","Attempt":1,"ChildWorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_23","Namespace":"default","RunID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"b5153ab0-e65b-11ee-8e4d-0242ac120004","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-19T20:46:47-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 18:46:47 {"level":"debug","Attempt":1,"ChildWorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","Namespace":"default","RunID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"b5153ab0-e65b-11ee-8e4d-0242ac120004","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:46:47-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 18:46:47 {"level":"debug","ActivityID":"5","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:46:47-05:00","message":"ExecuteActivity"}
2024-03-19 18:46:47 {"level":"debug","Attempt":1,"Duration":600000,"Namespace":"default","RunID":"2f4241ca-99be-4125-9ba3-df75bb9ab84e","TaskQueue":"archive","TimerID":"5","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-19T20:46:47-05:00","message":"NewTimer"}
2024-03-19 18:46:47 {"level":"error","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":1,"Error":"no stream found for channel chominnnn","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","time":"2024-03-19T20:46:47-05:00","message":"Activity error."}
2024-03-19 18:47:39 {"level":"debug","Attempt":1,"ChildWorkflowID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7_23","Namespace":"default","RunID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"dace69f8-1ca8-4c61-80ec-d98fcc77e9fd_34","WorkflowType":"MoveVideoWorkflow","time":"2024-03-19T20:47:39-05:00","message":"ExecuteChildWorkflow"}
2024-03-19 18:47:40 {"level":"debug","ActivityID":"5","ActivityType":"MoveVideo","Attempt":1,"Namespace":"default","RunID":"a509d293-131b-4592-a837-518dbe09888c","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"e379eb83-4e7b-42fe-928e-ad446d3cd7b7_23","WorkflowType":"MoveVideoWorkflow","time":"2024-03-19T20:47:40-05:00","message":"ExecuteActivity"}
2024-03-19 18:47:47 {"level":"error","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":2,"Error":"no stream found for channel chominnnn","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","time":"2024-03-19T20:47:47-05:00","message":"Activity error."}
2024-03-19 18:49:47 {"level":"error","ActivityType":"SaveTwitchLiveVideoInfo","Attempt":3,"Error":"no stream found for channel chominnnn","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","time":"2024-03-19T20:49:47-05:00","message":"Activity error."}
2024-03-19 18:49:47 {"level":"error","Attempt":1,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc000de6d50, 0xb}, {0xc000de6d38, 0x4}, {0xc000de6d60, 0x6}, {0xc000de6d68, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc0006d6390}, {{0xc000de6d50, 0xb}, {0xc000de6d38, 0x4}, {0xc000de6d60, 0x6}, {0xc000de6d68, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc0006d63c0, 0x2, 0xc0006d63c0?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc000631d18?}, {0xc0006d63c0?, 0xf9e07e48?, 0x7fcd40921408?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc000133140, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc0003054a0, {0x1809a40, 0xc00078db00}, 0xc000d47e18)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000526e00, {0x1809a40, 0xc00078db00}, 0xc000d47bd8)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc0006d6240})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:49:47-05:00","message":"Workflow panic"}
2024-03-19 18:49:47 {"level":"warn","Attempt":1,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:49:47-05:00","message":"Failed to process workflow task."}
2024-03-19 18:49:47 {"level":"info","time":"2024-03-19T20:49:47-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:49:47 {"level":"debug","time":"2024-03-19T20:49:47-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:49:47 {"level":"info","time":"2024-03-19T20:49:47-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:49:47 {"level":"debug","time":"2024-03-19T20:49:47-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:49:47 {"level":"info","time":"2024-03-19T20:49:47-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:49:47 {"level":"debug","time":"2024-03-19T20:49:47-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:49:57 {"level":"info","time":"2024-03-19T20:49:57-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:49:57 {"level":"debug","time":"2024-03-19T20:49:57-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:50:07 {"level":"info","time":"2024-03-19T20:50:07-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:50:07 {"level":"debug","time":"2024-03-19T20:50:07-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:49:47 {"level":"error","Attempt":1,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc000d366c0, 0xb}, {0xc000d366cc, 0x4}, {0xc000d366e0, 0x6}, {0xc000d366e8, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000792990}, {{0xc000d366c0, 0xb}, {0xc000d366cc, 0x4}, {0xc000d366e0, 0x6}, {0xc000d366e8, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc0007929c0, 0x2, 0xc0007929c0?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc0006f1d18?}, {0xc0007929c0?, 0xf9ae7948?, 0x7fccf9d75468?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc000524b60, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000d51e50, {0x1809a40, 0xc000a0d020}, 0xc000701200)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000dfd800, {0x1809a40, 0xc000a0d020}, 0xc00069f710)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000792840})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:49:47-05:00","message":"Workflow panic"}
2024-03-19 18:49:47 {"level":"warn","Attempt":1,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:49:47-05:00","message":"Failed to process workflow task."}
2024-03-19 18:49:47 {"level":"error","Attempt":2,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc000401880, 0xb}, {0xc0004016b8, 0x4}, {0xc000401890, 0x6}, {0xc000401898, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc00047f5f0}, {{0xc000401880, 0xb}, {0xc0004016b8, 0x4}, {0xc000401890, 0x6}, {0xc000401898, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc00047f620, 0x2, 0xc00047f620?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc000d7bd18?}, {0xc00047f620?, 0xf9cbb3e8?, 0x7fccf9e88028?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc0001f3b20, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000f88f00, {0x1809a40, 0xc000e25320}, 0xc0010009d8)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc0009d43c0, {0x1809a40, 0xc000e25320}, 0xc001000618)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc00047f4a0})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:49:47-05:00","message":"Workflow panic"}
2024-03-19 18:49:47 {"level":"warn","Attempt":2,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:49:47-05:00","message":"Failed to process workflow task."}
2024-03-19 18:49:57 {"level":"error","Attempt":3,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc000400560, 0xb}, {0xc00040056c, 0x4}, {0xc000400570, 0x6}, {0xc000400578, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000702cc0}, {{0xc000400560, 0xb}, {0xc00040056c, 0x4}, {0xc000400570, 0x6}, {0xc000400578, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000702cf0, 0x2, 0xc000702cf0?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc0004f7d18?}, {0xc000702cf0?, 0xf9cbb3e8?, 0x7fccf9c760e8?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc000a4e0c0, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000f880a0, {0x1809a40, 0xc000e24180}, 0xc001000168)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc0009d4140, {0x1809a40, 0xc000e24180}, 0xc000700048)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000702b70})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:49:57-05:00","message":"Workflow panic"}
2024-03-19 18:49:57 {"level":"warn","Attempt":3,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:49:57-05:00","message":"Failed to process workflow task."}
2024-03-19 18:50:22 {"level":"info","time":"2024-03-19T20:50:22-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:50:22 {"level":"debug","time":"2024-03-19T20:50:22-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:50:40 {"level":"info","time":"2024-03-19T20:50:40-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:50:40 {"level":"debug","time":"2024-03-19T20:50:40-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:50:59 {"level":"debug","time":"2024-03-19T20:50:59-05:00","message":"deleting file: /tmp/42097172920_a3c69867-e649-11ee-8e4d-0242ac120004-video.mp4"}
2024-03-19 18:50:59 {"level":"debug","time":"2024-03-19T20:50:59-05:00","message":"stopping heartbeat move-video-42097172920"}
2024-03-19 18:50:59 {"level":"debug","time":"2024-03-19T20:50:59-05:00","message":"checking if tasks are done for video 42097172920"}
2024-03-19 18:50:59 {"level":"debug","time":"2024-03-19T20:50:59-05:00","message":"all tasks for video 42097172920 are done"}
2024-03-19 18:51:06 {"level":"info","time":"2024-03-19T20:51:06-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:51:06 {"level":"debug","time":"2024-03-19T20:51:06-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:50:07 {"level":"error","Attempt":4,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc000d362c0, 0xb}, {0xc000d362cc, 0x4}, {0xc000d362d0, 0x6}, {0xc000d362d8, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc0007938c0}, {{0xc000d362c0, 0xb}, {0xc000d362cc, 0x4}, {0xc000d362d0, 0x6}, {0xc000d362d8, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000793920, 0x2, 0xc000793920?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc000647d18?}, {0xc000793920?, 0xf9ae7948?, 0x7fccf9d75468?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc0005243e0, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000462960, {0x1809a40, 0xc000a0c240}, 0xc0007004b0)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000dfc880, {0x1809a40, 0xc000a0c240}, 0xc00062c300)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000793650})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:50:07-05:00","message":"Workflow panic"}
2024-03-19 18:50:07 {"level":"warn","Attempt":4,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:50:07-05:00","message":"Failed to process workflow task."}
2024-03-19 18:50:22 {"level":"error","Attempt":5,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc000d36a30, 0xb}, {0xc000d36a3c, 0x4}, {0xc000d36a40, 0x6}, {0xc000d36a48, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc0009f2b10}, {{0xc000d36a30, 0xb}, {0xc000d36a3c, 0x4}, {0xc000d36a40, 0x6}, {0xc000d36a48, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc0009f2b40, 0x2, 0xc0009f2b40?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc000647d18?}, {0xc0009f2b40?, 0xf9ae7948?, 0x7fccf9d75468?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc000524b80, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000462d20, {0x1809a40, 0xc000a0c600}, 0xc000700840)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000dfcc80, {0x1809a40, 0xc000a0c600}, 0xc00062c960)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc0009f29c0})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:50:22-05:00","message":"Workflow panic"}
2024-03-19 18:50:22 {"level":"warn","Attempt":5,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:50:22-05:00","message":"Failed to process workflow task."}
2024-03-19 18:50:40 {"level":"error","Attempt":6,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc0001fd750, 0xb}, {0xc0001fd738, 0x4}, {0xc0001fd760, 0x6}, {0xc0001fd768, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000ef6360}, {{0xc0001fd750, 0xb}, {0xc0001fd738, 0x4}, {0xc0001fd760, 0x6}, {0xc0001fd768, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000ef6390, 0x2, 0xc000ef6390?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc0009c7d18?}, {0xc000ef6390?, 0xf9e055a8?, 0x7fccf9cbcc48?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc00063e800, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000a47680, {0x1809a40, 0xc000a24cc0}, 0xc00062d140)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000a35c00, {0x1809a40, 0xc000a24cc0}, 0xc00069e8e8)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000ef6210})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:50:40-05:00","message":"Workflow panic"}
2024-03-19 18:50:40 {"level":"warn","Attempt":6,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:50:40-05:00","message":"Failed to process workflow task."}
2024-03-19 18:51:51 {"level":"info","time":"2024-03-19T20:51:51-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:51:51 {"level":"debug","time":"2024-03-19T20:51:51-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:53:15 {"level":"info","time":"2024-03-19T20:53:15-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:53:15 {"level":"debug","time":"2024-03-19T20:53:15-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:55:35 {"level":"info","time":"2024-03-19T20:55:35-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 18:55:35 {"level":"debug","time":"2024-03-19T20:55:35-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:51:06 {"level":"error","Attempt":7,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc0001ea160, 0xb}, {0xc0001ea148, 0x4}, {0xc0001ea170, 0x6}, {0xc0001ea178, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000615ad0}, {{0xc0001ea160, 0xb}, {0xc0001ea148, 0x4}, {0xc0001ea170, 0x6}, {0xc0001ea178, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000615b00, 0x2, 0xc000615b00?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc000d7bd18?}, {0xc000615b00?, 0xf9dc8ca8?, 0x7fcd40921408?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc0001336a0, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000416e60, {0x1809a40, 0xc00080bec0}, 0xc000a86960)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000186940, {0x1809a40, 0xc00080bec0}, 0xc00069fa10)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000615530})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:51:06-05:00","message":"Workflow panic"}
2024-03-19 18:51:06 {"level":"warn","Attempt":7,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:51:06-05:00","message":"Failed to process workflow task."}
2024-03-19 18:51:51 {"level":"error","Attempt":8,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc0005186a0, 0xb}, {0xc000518688, 0x4}, {0xc0005186b0, 0x6}, {0xc0005186b8, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000a8fef0}, {{0xc0005186a0, 0xb}, {0xc000518688, 0x4}, {0xc0005186b0, 0x6}, {0xc0005186b8, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000a8ff20, 0x2, 0xc000a8ff20?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc0006f5d18?}, {0xc000a8ff20?, 0xf9d7cf48?, 0x7fccf9e86908?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc0000b86c0, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc0000bc640, {0x1809a40, 0xc000e921e0}, 0xc000d464f8)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000e14940, {0x1809a40, 0xc000e921e0}, 0xc000d46048)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000a8fda0})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:51:51-05:00","message":"Workflow panic"}
2024-03-19 18:51:51 {"level":"warn","Attempt":8,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:51:51-05:00","message":"Failed to process workflow task."}
2024-03-19 18:53:15 {"level":"error","Attempt":9,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc000a505f0, 0xb}, {0xc000a505d8, 0x4}, {0xc000a50600, 0x6}, {0xc000a50608, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000cf2a50}, {{0xc000a505f0, 0xb}, {0xc000a505d8, 0x4}, {0xc000a50600, 0x6}, {0xc000a50608, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000cf2a80, 0x2, 0xc000cf2a80?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc0009c5d18?}, {0xc000cf2a80?, 0xf9e055a8?, 0x7fccf9cbcc48?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc00063e4e0, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000ceef00, {0x1809a40, 0xc000a245a0}, 0xc00062c9c0)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000a27280, {0x1809a40, 0xc000a245a0}, 0xc00062c600)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000cf2900})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:53:15-05:00","message":"Workflow panic"}
2024-03-19 18:53:15 {"level":"warn","Attempt":9,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:53:15-05:00","message":"Failed to process workflow task."}
2024-03-19 18:56:47 {"level":"debug","time":"2024-03-19T20:56:47-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 18:57:47 {"level":"debug","time":"2024-03-19T20:57:47-05:00","message":"getting live streams using the following query param: ?user_login=chominnnn"}
2024-03-19 19:00:43 {"level":"info","time":"2024-03-19T21:00:43-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 19:00:43 {"level":"debug","time":"2024-03-19T21:00:43-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 19:10:19 {"level":"info","time":"2024-03-19T21:10:19-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 19:10:19 {"level":"debug","time":"2024-03-19T21:10:19-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 18:55:35 {"level":"error","Attempt":10,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc000680780, 0xb}, {0xc00068078c, 0x4}, {0xc0006807a0, 0x6}, {0xc0006807a8, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000a32300}, {{0xc000680780, 0xb}, {0xc00068078c, 0x4}, {0xc0006807a0, 0x6}, {0xc0006807a8, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000a32330, 0x2, 0xc000a32330?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc0004f7d18?}, {0xc000a32330?, 0xf9d75508?, 0x7fccf9e143b8?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc000456540, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000ebb8b0, {0x1809a40, 0xc000e25020}, 0xc001001050)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc0009d5640, {0x1809a40, 0xc000e25020}, 0xc000d470b0)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000a321b0})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:55:35-05:00","message":"Workflow panic"}
2024-03-19 18:55:35 {"level":"warn","Attempt":10,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T20:55:35-05:00","message":"Failed to process workflow task."}
2024-03-19 18:56:47 {"level":"debug","ActivityID":"10","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Namespace":"default","RunID":"2f4241ca-99be-4125-9ba3-df75bb9ab84e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_23","WorkflowType":"DownloadTwitchLiveThumbnailsWorkflowWait","time":"2024-03-19T20:56:47-05:00","message":"ExecuteActivity"}
2024-03-19 18:56:47 {"level":"error","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":1,"Error":"no stream found for channel chominnnn","Namespace":"default","RunID":"2f4241ca-99be-4125-9ba3-df75bb9ab84e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_23","time":"2024-03-19T20:56:47-05:00","message":"Activity error."}
2024-03-19 18:57:47 {"level":"error","ActivityType":"DownloadTwitchLiveThumbnails","Attempt":2,"Error":"no stream found for channel chominnnn","Namespace":"default","RunID":"2f4241ca-99be-4125-9ba3-df75bb9ab84e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_23","time":"2024-03-19T20:57:47-05:00","message":"Activity error."}
2024-03-19 19:00:43 {"level":"error","Attempt":11,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc0006809c0, 0xb}, {0xc000680998, 0x4}, {0xc0006809e0, 0x6}, {0xc0006809e8, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000a62060}, {{0xc0006809c0, 0xb}, {0xc000680998, 0x4}, {0xc0006809e0, 0x6}, {0xc0006809e8, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000a62090, 0x2, 0xc000a62090?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc000637d18?}, {0xc000a62090?, 0xf9ae7948?, 0x7fccf9e86908?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc0000b8a80, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000ef5310, {0x1809a40, 0xc000709500}, 0xc001000eb8)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000187440, {0x1809a40, 0xc000709500}, 0xc001000b10)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000a8fe90})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T21:00:43-05:00","message":"Workflow panic"}
2024-03-19 19:00:43 {"level":"warn","Attempt":11,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T21:00:43-05:00","message":"Failed to process workflow task."}
2024-03-19 19:10:19 {"level":"error","Attempt":12,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc0001ea310, 0xb}, {0xc0001ea31c, 0x4}, {0xc0001ea320, 0x6}, {0xc0001ea328, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000a63590}, {{0xc0001ea310, 0xb}, {0xc0001ea31c, 0x4}, {0xc0001ea320, 0x6}, {0xc0001ea328, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000a635c0, 0x2, 0xc000a635c0?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc000631d18?}, {0xc000a635c0?, 0xf9d75508?, 0x7fccf9c93108?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc000d381a0, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc0006905f0, {0x1809a40, 0xc000e24300}, 0xc00069e7f8)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000e144c0, {0x1809a40, 0xc000e24300}, 0xc000798048)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000a63440})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T21:10:19-05:00","message":"Workflow panic"}
2024-03-19 19:19:46 {"level":"info","time":"2024-03-19T21:19:46-05:00","message":"no stream found for channel - cancelling workflow"}
2024-03-19 19:19:46 {"level":"debug","time":"2024-03-19T21:19:46-05:00","message":"cancelling workflow: b5153ab0-e65b-11ee-8e4d-0242ac120004 run: cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b"}
2024-03-19 19:10:19 {"level":"warn","Attempt":12,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T21:10:19-05:00","message":"Failed to process workflow task."}
2024-03-19 19:19:46 {"level":"error","Attempt":13,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","StackTrace":"coroutine root [panic]:\ngithub.com/zibbp/ganymede/internal/workflows.cancelWorkflowAndCleanup({0x1808620, 0x22b1100}, {{0xc0001fdcc0, 0xb}, {0xc0001fdccc, 0x4}, {0xc0001fdd10, 0x6}, {0xc0001fdd18, 0x4}, ...})\n\t/app/internal/workflows/video.go:87 +0x44c\ngithub.com/zibbp/ganymede/internal/workflows.SaveTwitchLiveVideoInfoWorkflow({0x1809308, 0xc000de1110}, {{0xc0001fdcc0, 0xb}, {0xc0001fdccc, 0x4}, {0xc0001fdd10, 0x6}, {0xc0001fdd18, 0x4}, ...})\n\t/app/internal/workflows/video.go:363 +0x226\nreflect.Value.call({0x13f27e0?, 0x1678d68?, 0x30?}, {0x15cee60, 0x4}, {0xc000de1140, 0x2, 0xc000de1140?})\n\t/usr/local/go/src/reflect/value.go:596 +0xce5\nreflect.Value.Call({0x13f27e0?, 0x1678d68?, 0xc000635d18?}, {0xc000de1140?, 0xf9cbb3e8?, 0x7fccf9e143b8?})\n\t/usr/local/go/src/reflect/value.go:380 +0xb9\ngo.temporal.io/sdk/internal.executeFunction({0x13f27e0, 0x1678d68}, {0xc000456a60, 0x2, 0x13f53c0?})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:1812 +0x26b\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc000134e60, {0x1809a40, 0xc000759860}, 0xc001001878)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:529 +0x150\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute(0xc000a35d00, {0x1809a40, 0xc000759860}, 0xc000799530)\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_worker.go:805 +0x291\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1({0x1809308, 0xc000de0fc0})\n\t/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:512 +0xc2","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T21:19:46-05:00","message":"Workflow panic"}
2024-03-19 19:19:46 {"level":"warn","Attempt":13,"Error":"runtime error: invalid memory address or nil pointer dereference","Namespace":"default","RunID":"754df74b-1c77-4b4f-a50c-939be267716e","TaskQueue":"archive","WorkerID":"615cb419001f","WorkflowID":"cb15d1ea-2cc4-4fda-9184-ae1fb0e1cf5b_24","WorkflowType":"SaveTwitchLiveVideoInfoWorkflow","time":"2024-03-19T21:19:46-05:00","message":"Failed to process workflow task."}

Isaac · Answer 9 · Thu Mar 21 2024 08:02:56 GMT+0800 (China Standard Time)

I setup my demo instance with a channel you had issues with. Everything seems to have worked correctly in my instance. The streamlink download of the video exited with "no playable streams found on this url..." and it went through the cancellation process.

{"log":"{\"level\":\"debug\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"finished downloading live video for 42090393528 - exit status 1\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.10426593Z"}
{"log":"{\"level\":\"debug\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"streamlink live stdout: [cli][info] Found matching plugin twitch for URL https://twitch.tv/bobajenny\\n[plugins.twitch][error] twirp error not_found: transcode does not exist\\nerror: No playable streams found on this URL: https://twitch.tv/bobajenny\\n\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.104289141Z"}
{"log":"{\"level\":\"error\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"no playable streams found on this URL for 42090393528\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.104292541Z"}
{"log":"{\"level\":\"debug\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"stopping heartbeat download-livevideo-42090393528\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.108400683Z"}
{"log":"{\"level\":\"error\",\"ActivityType\":\"DownloadTwitchLiveVideo\",\"Attempt\":1,\"Error\":\"no playable streams found on this URL\",\"Namespace\":\"default\",\"RunID\":\"9b36d42e-8240-4b56-b584-902b6411fe70\",\"TaskQueue\":\"archive\",\"WorkerID\":\"1e0ba293468e\",\"WorkflowID\":\"7b67a0b4-0872-411b-9782-5169f2f33c87_5\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"Activity error.\"}\n","stream":"stdout","time":"2024-03-19T04:40:14.108422513Z"}
{"log":"{\"level\":\"debug\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"moving file: /tmp/42090393528_1ba64200-e583-11ee-a144-0242ac130006-video-convert.mp4 to /vods/bobajenny/2024-03-18-42090393528-live-1ba64200-e583-11ee-a144-0242ac130006/42090393528-video.mp4\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.114996979Z"}
{"log":"{\"level\":\"debug\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"starting heartbeat move-video-42090393528\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.115020089Z"}
{"log":"{\"level\":\"error\",\"error\":\"activity error (type: DownloadTwitchLiveVideo, scheduledEventID: 5, startedEventID: 6, identity: 1e0ba293468e): no playable streams found on this URL\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"no stream found for channel\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.127699797Z"}
{"log":"{\"level\":\"info\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"no stream found for channel - cancelling workflow\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.127718387Z"}
{"log":"{\"level\":\"debug\",\"time\":\"2024-03-18T23:40:14-05:00\",\"message\":\"cancelling workflow: c54c80a1-e5aa-11ee-a144-0242ac130006 run: 8d0a1de4-997b-496c-ab0c-a7b5fa004f09\"}\n","stream":"stderr","time":"2024-03-19T04:40:14.128256929Z"}

The main temporal ArchiveLiveVideoWorkflow was correctly terminated with a reason.

Regarding the invalid memory address panic in your logs. I believe that is happen because the temporal client isn't being initialized in the worker. I've updated #385 to setup this up. Can you pull, rebuild, and test again? Also, when you post docker logs can you run docker logs ganymede-api >& /tmp/api.log to dump all the logs to a file? I noticed that some of the timestamps are out of order, which I think is due to some logs being sent to stderr. Just make sure to clean up your access token in these.

Caleb Contreras · Answer 10 · Tue Mar 26 2024 11:46:09 GMT+0800 (China Standard Time)

Can confirm latest #385 commit #f491e16 is working properly and fixes this issue.

Just some notes about this:

I had the system running using this image, during which it archived three streams of varying lengths (7 hours, 5 hours, and 2 hours). Only the 7-hour and 2-hour streams resulted in the 'stream not found' live archive issue, which this update resolves. Interestingly, the 5-hour stream concluded without any problems (no second live archive nor discord notification).
There's always a second "streamer is live!" webhook notification being sent when the duplicate task starts.

Isaac · Answer 11 · Wed Mar 27 2024 09:14:53 GMT+0800 (China Standard Time)

Glad to hear the fixes are working. I'll get this cleaned up and merged sometime this week.

I had the system running using this image, during which it archived three streams of varying lengths (7 hours, 5 hours, and 2 hours). Only the 7-hour and 2-hour streams resulted in the 'stream not found' live archive issue, which this update resolves. Interestingly, the 5-hour stream concluded without any problems (no second live archive nor discord notification).

The Twitch API is very unreliable when it updates if the stream is live or not

There's always a second "streamer is live!" webhook notification being sent when the duplicate task starts.

That makes sense, I send the webhook as soon as the stream is detected as live (which technically it still is according to the API I check). I'd prefer to not have the webhook logic in the workflow and activities as it would then be sent via the worker which I don't feel this right. I don't have a good solution for the double webhook notification right now.

Isaac · Answer 12 · Mon Apr 01 2024 04:02:49 GMT+0800 (China Standard Time)

Fixes released in v2.2.0.