"Chat Download" never stops

[CappiSteijns]

I'm not sure what is happening, but the "chat download" never stop, resulting in errors. Some downloads are successfully recorded and moved, while others are not. See the pictures and logs below.

The logs in ganymede itself;
"Video Download" log:

"Chat Download" log:

ganymede-api-2024-06-13T15-46-06.log
ganymede-temporal-2024-06-13T15-47-47.log

If you need more logs let me know please :)

EDIT: added docker compose

services:
  ganymede-api:
    container_name: ganymede-api
    image: ghcr.io/zibbp/ganymede:latest
    restart: unless-stopped
    environment:
      - TZ=Europe/Amsterdam # Set to your timezone
      - DB_HOST=192.168.68.61
      - DB_PORT=4803
      - DB_USER=ganymede
      - DB_PASS=####
      - DB_NAME=ganymede-prd
      - DB_SSL=disable
      - JWT_SECRET=SECRET
      - JWT_REFRESH_SECRET=SECRET
      - TWITCH_CLIENT_ID=############xjm7o
      - TWITCH_CLIENT_SECRET=6################or6j
      - FRONTEND_HOST=https://###########.nl
      - COOKIE_DOMAIN=########.nl
      # OPTIONAL
      # - OAUTH_PROVIDER_URL=
      # - OAUTH_CLIENT_ID=
      # - OAUTH_CLIENT_SECRET=
      # - OAUTH_REDIRECT_URL=http://IP:PORT/api/v1/auth/oauth/callback # Points to the API service
      # Volgende heb ik toegevoegd voor UPDATE 2.0.0
      - TEMPORAL_URL=ganymede-temporal:7233
      # WORKER
      - MAX_CHAT_DOWNLOAD_EXECUTIONS=5
      - MAX_CHAT_RENDER_EXECUTIONS=3
      - MAX_VIDEO_DOWNLOAD_EXECUTIONS=5
      - MAX_VIDEO_CONVERT_EXECUTIONS=3      
    volumes:
      - /mnt/user/Twitch/ganymede:/vods
      - /mnt/user/appdata/ganymede-api/logs:/logs
      - /mnt/user/appdata/ganymede-api:/data
      # Uncomment below to persist temp files
      - /mnt/user/Twitch/ganymede/temp:/tmp
    ports:
      - 4800:4000
  ganymede-frontend:
    container_name: ganymede-frontend
    image: ghcr.io/zibbp/ganymede-frontend:latest
    restart: unless-stopped
    environment:
      - REQUIRE_LOGIN=true
      - API_URL=https://apiganymede.cappiflex.nl # Points to the API service
      - CDN_URL=https://cdnganymede.cappiflex.nl # Points to the CDN service
      - SHOW_SSO_LOGIN_BUTTON=true # show/hide SSO login button on login page
      - FORCE_SSO_AUTH=false # force SSO auth for all users (bypasses login page and redirects to SSO)
    ports:
      - 4801:3000
  ganymede-db:
    container_name: ganymede-db
    image: postgres:14
    volumes:
      - /mnt/user/appdata/ganymede-db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=#######
      - POSTGRES_USER=ganymede
      - POSTGRES_DB=ganymede-prd
    ports:
      - 4803:5432
  ganymede-nginx:
    container_name: ganymede-nginx
    image: nginx
    volumes:
      - /mnt/user/appdata/ganymedenginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /mnt/user/Twitch/ganymede:/mnt/vods
    ports:
      - 4802:8080
  ganymede-temporal:
    image: temporalio/auto-setup:1.23
    container_name: ganymede-temporal
    depends_on:
      - ganymede-db
    environment:
      - DB=postgres12 # this tells temporal to use postgres (not the db name)
      - DB_PORT=5432
      - POSTGRES_USER=ganymede
      - POSTGRES_PWD=#####
      - POSTGRES_SEEDS=ganymede-db # name of the db service
    ports:
      - 7233:7233

[Isaac]

It looks like a similar issue to #437 which I've pushed a fix for to the :main image. When there is nothing being archived, can you switch the API container to use the :main image and try again?

I plan to publish a release this weekend with some fixes that have been building up the past few weeks.

[CappiSteijns]

It looks like a similar issue to #437 which I've pushed a fix for to the :main image. When there is nothing being archived, can you switch the API container to use the :main image and try again?

I plan to publish a release this weekend with some fixes that have been building up the past few weeks.

Allrighty, changed to :main for the api container.

Any chance the vods in the queue will finish?
When going to Admin - Tasks - Start stuck queue items I get the error: Request Failed Key: 'StartTaskRequest.Task' Error:Field validation for 'Task' failed on the 'oneof' tag

[Isaac]

If those are livestreams, there's not much you can do to recover the chats. If you have /tmp mounted locally that chats will likely still be there. For those queue items, you'll have to delete them from Admin > Queue, and set the vod to "processed" in Admin > VODs.
The admin task is deprecated and doesn't work. It is removed, waiting on a release to push it out.

[CappiSteijns]

If those are livestreams, there's not much you can do to recover the chats. If you have /tmp mounted locally that chats will likely still be there. For those queue items, you'll have to delete them from Admin > Queue, and set the vod to "processed" in Admin > VODs. The admin task is deprecated and doesn't work. It is removed, waiting on a release to push it out.

So I have like all these files in the /tmp folder:

and set the vod to "processed" in Admin > VODs.

When I do that the vod does become clickable in Ganymede but it won't load the vod because it's still in the /tmp folder and does not get moved to the correct folder. What would be the steps to manual move the vod's and chat files from /tmp? If that would even be possible.

[Isaac]

You can try to manually restart the workflow items for each archive. Start by navigating to Workflows > Closed. Then you'll need to find the workflow you want to run and confirm that it corresponds with your video ID. Start clicking into workflows and finding that one that matches your video ID.

Once you've found the group of workflows for your video ID, locate the one that you want to "jumpstart". We'll start with the task after the chat download, ConvertTwitchLiveChatWorkflow.

Clicking the restart button on the workflow page will restart the workflow. Hopefully this will take the *-live-chat.json file and convert it into the standard format that can be used.

After the live chat has been converted, you'll need to find the MoveTwitchChatWorkflow for your video ID and run that as well. Repeat the same for the incomplete video workflows as well.

The logs of the ganymede-api container will tell you if this process is going smooth or if it's run into issues.

It's a complex process to get these going again so let me know if you have any questions/issues.

[Blaiz0]

Hi, i seem to have to same issue. Recently migrated from 1.43 on a synology, to 2.3.0, now 2.3.1 on a windows machine.
I would really like to get at least one memorable vod archived.
A lot of them seem to fail on KillTwitchLiveChatDownload.

The problem is i cant seem to find ConvertTwitchLiveChatWorkflow, to "jumpstart" it.
All the -live-chat and -video files are in the tmp directory.

Also notice that start time and close time is "invalid date" on all closed workflows. Active workflows have invalid date on start time, but the current time on close time.

Have temporal UI too, if that helps.

[Isaac]

The problem is i cant seem to find ConvertTwitchLiveChatWorkflow, to "jumpstart" it.
All the -live-chat and -video files are in the tmp directory.

Ya that's right because the workflow didn't progress past the download. I can likely recover everything with some manual effort. First, make a copy of all the data in the Ganymede container to somewhere on your host to make sure it doesn't get deleted.

docker cp ganymede-api:/tmp ./tmp-backup

I'll walk through the manual archive steps below.

Video

To perform the manual steps of the video archive process, follow the below instructions.

1. Exec into the ganymede-api container with docker exec -it ganymede-api /bin/bash.
2. Identify the video in /tmp that you want to go through the archive process with. For example, 44359154267_d1283376-2b5c-11ef-b9de-0242ac1a0007-video.mp4
3. Run the following ffmpeg command to perform some video post-processing, updating with your correct video path. Note the -convert appended to the output file name.

ffmpeg -i /tmp/44359154267_d1283376-2b5c-11ef-b9de-0242ac1a0007-video.mp4 -c:v copy -c:a copy /tmp/44359154267_d1283376-2b5c-11ef-b9de-0242ac1a0007-video-convert.mp4

3. Edit the VOD in Ganymede (Admin > VODs) and copy the "Video Path".
   
4. Copy the -convert file from the FFmpeg command to the path you copied

cp /tmp/44359154267_d1283376-2b5c-11ef-b9de-0242ac1a0007-video-convert.mp4 /vods/staysafetv/2024-06-15-44359154267-live-d1283376-2b5c-11ef-b9de-0242ac1a0007/44359154267-video.mp4

5. Go back to editing the VOD in Ganymede and disable the "Is Processing" checkmark at the top and save. Now if you view the VOD you should be able to play the video.

Chat

Converting the chat is more difficult. If you send me the following information/files I can do this for you. I will need the following:

• The -live-chat.json files from the /tmp directory
• The chat_start time for each chat/VOD. You can retrieve this by running the below command. Update the ID with the Ganymede ID of the VOD you are recovering

docker exec ganymede-db psql -U ganymede ganymede-prd -c "select chat_start from queues where vod_queue = 'd1283376-2b5c-11ef-b9de-0242ac1a0007'"

[Isaac]

v2.3.2 has been released that includes the fix to this chat download issue. If you're attempting to recover failed archives, I advise to hold off on updating until you get those recovered.

[Blaiz0]

Hi, thank you so much!
Updated to v2.3.1:main tag on api and fronted just before you replied. I see that fixed the invalid dates.

I was able to convert and transfer the video files to the archive.

Here are the two chats:
Chat 1:
40658934565_49e946eb-1493-11ef-8dc9-0242ac150007-live-chat.json
chat_start

2024-05-17 21:20:32.790546+00

Chat 2:
42362888808_18a5b85a-1759-11ef-8dc9-0242ac150007-live-chat.json
chat_start

2024-05-21 10:01:33.420771+00

[Isaac]

Here are the converted chat files. One of them is likely longer than the video as it failed to stop downloading. This shouldn't affect the playback though as the start time is what really matters. You'll need to move these files to the correct path found by editing the VOD in Ganymede and copying to the chat_path.
40658934565_49e946eb-1493-11ef-8dc9-0242ac150007-chat-convert.json
42362888808_18a5b85a-1759-11ef-8dc9-0242ac150007-chat-convert.json

If you want to render these chats into videos for long-term archival reasons. Follow the below steps

1. Exec into the ganymede-api container with docker exec -it ganymede-api /bin/bash.
2. Copy the chat path for each video from Ganymede
3. Run the following command

TwitchDownloaderCLI chatrender -i <path_to_chat_json> <arguments> -o /tmp/chat.mp4

• path_to_chat_json is the path to the chat json you just copied
• arguments can be retrieved from Admin > Settings and copying the arguments from "Chat Render Args"

4. Once the chat render is complete move the video to the correct location

If you're not trying to recover any more failed archives I recommend updated to the images to the latest version now.

[CappiSteijns]

As for me, I decided the failed VOD's were not important enough to go to all the troubles haha :)

@Blaiz0 If you can let us know you succesfully saved the vod I can close this issue.

Thanks @Zibbp for the continues good help :D

[Blaiz0]

Yeah i was able to successfully archive the chats!
The issue was only intermittent for me, and i haven't been able to test out v2.3.2 yet. Im sure you can close it, ill report back if it still a problem after some testing.