Version 2 Releasing Soon

Question

Version 2 Releasing Soon

Zibbp opened this issue 8 months ago · comments

I'm working on a version 2 release and hope to have it out by Christmas🎄.

Features

New Queue System

I'm overhauling the queue system to use Temporal workflows. This allows me to easily create tasks and put them in workflows for an easier and more robust queue system. This new system includes automated retries and hopefully less one-off queue issues. Using Temporal also introduces the idea of distributed worker nodes so if you have multiple systems you can balance load between them. Distributed workers will not be available in the next release, maybe in the future.

The v2 release will include some changes to the docker-compose file to accommodate the Temporal server that will need to run. To not break existing installs, I'm planning on running an ephemeral temporal server in the main API container until users can update their compose file. This ephemeral server will not persist data between restarts of the API container so I advice everyone to update their compose file with the proper service (available when v2 is released).

Frontend Updates

I gave the frontend a new coating of paint for the dark theme. I've also updated all of the packages to run on the latest versions and fixed any bugs that came with that.

Other Features

Chapters/Categories

I'm hoping to get #317 in this release utilizing the new queue system

Planned features

Thumbnails

In a future release I'm planning on adding support to generate video thumbnails, similar to when scrubbing a Youtube video, using FFmpeg. Generating the thumbnails is slow and resource intensive so it will be opt-in. Initially it will probably be manual, requiring users to click a button to generate thumbnails for a video to test the functionality out.

jayjay181818 · Answer 1 · Sun Dec 17 2023 23:25:17 GMT+0800 (China Standard Time)

Very exciting and a lovely Christmas surprise! will be interesting to see how it runs on my Orange Pi 5 plus and Raspberry pi 5 that arrived this week. I imagine performance will likely be the same because it relies on other project tools for a lot of the functionality. I guess that's a good excuse to create a fresh instance, that being said, I have seemed to struggle getting the chats to download on the RPI5 but not the OPI5+ which is odd (I've tinkered a lot to try get it working).

I imagine this has already been requested before, but any chance we could get progress bars or at least logs for the video move? Less of an issue when using an nvme drive, but when I've ran ganymede on slow boards via micro SD cards, it can be a long process with the larger vods.

Thanks for your work dude, and Merry Christmas!

jayjay181818 · Answer 2 · Mon Dec 18 2023 00:52:31 GMT+0800 (China Standard Time)

Very exciting and a lovely Christmas surprise! will be interesting to see how it runs on my Orange Pi 5 plus and Raspberry pi 5 that arrived this week. I imagine performance will likely be the same because it relies on other project tools for a lot of the functionality. I guess that's a good excuse to create a fresh instance, that being said, I have seemed to struggle getting the chats to download on the RPI5 but not the OPI5+ which is odd (I've tinkered a lot to try get it working).

I imagine this has already been requested before, but any chance we could get progress bars or at least logs for the video move? Less of an issue when using an nvme drive, but when I've ran ganymede on slow boards via micro SD cards, it can be a long process with the larger vods.

Thanks for your work dude, and Merry Christmas!

Further update to the issues I mentioned on my raspberry pi 5.. So it seems to be fine in Ubuntu, but not Raspberry pi OS, I was scratching my head, reinstalled the OS multiple times, toyed with the compose file for hours (replicated 3 different working yaml's from other machines I've tested etc). Very weird indeed, I wonder if either it's a temporary bug in raspberry pi OS that will get patched later down the line, or if there is some dependency missing from a clean install.

Isaac · Answer 3 · Tue Dec 19 2023 07:05:04 GMT+0800 (China Standard Time)

Strange that it's not working on Raspberry PI OS. Is that OS 32bit? I know you can install Ubuntu on the newer PIs using 64 bit, that maybe that is why it's working? Any errors in the logs? You should be able to exec into the container docker exec -it ganymede-api /bin/bash and run the chat download manually to debug TwitchDownloaderCLI chatdownload --id <id> --embed-images -o /tmp/debug.json

I imagine this has already been requested before, but any chance we could get progress bars or at least logs for the video move? Less of an issue when using an nvme drive, but when I've ran ganymede on slow boards via micro SD cards, it can be a long process with the larger vods.

I'm surprised downloads work fine on the SD card, that must be slow! I'll look into implementing a progress bar and transfer speed rate on the queue page.

jayjay181818 · Answer 4 · Tue Dec 19 2023 09:10:46 GMT+0800 (China Standard Time)

Strange that it's not working on Raspberry PI OS. Is that OS 32bit? I know you can install Ubuntu on the newer PIs using 64 bit, that maybe that is why it's working? Any errors in the logs? You should be able to exec into the container docker exec -it ganymede-api /bin/bash and run the chat download manually to debug TwitchDownloaderCLI chatdownload --id <id> --embed-images -o /tmp/debug.json

I imagine this has already been requested before, but any chance we could get progress bars or at least logs for the video move? Less of an issue when using an nvme drive, but when I've ran ganymede on slow boards via micro SD cards, it can be a long process with the larger vods.

I'm surprised downloads work fine on the SD card, that must be slow! I'll look into implementing a progress bar and transfer speed rate on the queue page.

Yeah, very odd indeed! 64bit Rasberry pi OS, i tried the server image, recommended and full version with all the same result that it would error out with the chat download but the video would process fine. I would of suspected it might be as RPOS is Debian based instead, but I have it running fine on other SBC's which are running dietpi which is also debian based as far as I'm aware. I will say that all of these instances have been deployed via docker, would be interesting to see how it responds to a local deployments.

They can be a tad slow via an SD card that's for sure! At one point i was running it on an Orange Pi Zero 3, a 10 hour VOD could take 22 hours to render the chat! It was fine for my use case at the time of just archiving for a friend that wanted their streams backed up. Pretty impressive that a $15 SBC can manage it at all not gonna lie.

I did want to ask, is there a way to allow Ganymede to utilise more resources? I understand a fresh out of the box deployment can run two jobs, but it would be nice to let it rip through a single job quickly when resources allow for it. On my orange pi 5 plus it can only end up using between 35-45% when of the CPU running from an NVME (a tad of a performance boost from the SD on the OPIZ3, a 2 hour vod takes about 45-50 mins). Im not sure if this is how Ganymede is coded to work or if there is some fiddling required with my docker and that a local deployment would use all the resources possible.

One last suggestion as i have been rambling, is it worth having a user submitted performance spreadsheet where people can submit what render speeds people manage on different hardware? I just have a hunch people might be curios what experience to expect from different hardware and it could guide their choice if they are buying something dedicated to this task (a twitch friend saw my instance and thought it was super cool and i had to do a bit of manual benchmarking for them to make their decision). Could be a google docs spreadsheet on the main page.

Isaac · Answer 5 · Tue Dec 19 2023 11:23:29 GMT+0800 (China Standard Time)

Let me know if you find out any more regarding the RPI OS issue. I can help out if you're able to manually run the chat download.

I did want to ask, is there a way to allow Ganymede to utilise more resources? I understand a fresh out of the box deployment can run two jobs, but it would be nice to let it rip through a single job quickly when resources allow for it. On my orange pi 5 plus it can only end up using between 35-45% when of the CPU running from an NVME (a tad of a performance boost from the SD on the OPIZ3, a 2 hour vod takes about 45-50 mins). Im not sure if this is how Ganymede is coded to work or if there is some fiddling required with my docker and that a local deployment would use all the resources possible.

I've tried to get the chat render to go faster but had no luck. I've even tried using hardware accelerating while encoding and could not get it to go any faster. You may want to post an issue in the upstream repository https://github.com/lay295/TwitchDownloader about if the ffmpeg process can be any faster (consume more resources if possible).

One last suggestion as i have been rambling, is it worth having a user submitted performance spreadsheet where people can submit what render speeds people manage on different hardware? I just have a hunch people might be curios what experience to expect from different hardware and it could guide their choice if they are buying something dedicated to this task (a twitch friend saw my instance and thought it was super cool and i had to do a bit of manual benchmarking for them to make their decision). Could be a google docs spreadsheet on the main page.

Sounds great, feel free to post an issue or discussion to get it started and I can pin it for others to see and contribute.

jayjay181818 · Answer 6 · Wed Dec 20 2023 02:25:52 GMT+0800 (China Standard Time)

Let me know if you find out any more regarding the RPI OS issue. I can help out if you're able to manually run the chat download.

I did want to ask, is there a way to allow Ganymede to utilise more resources? I understand a fresh out of the box deployment can run two jobs, but it would be nice to let it rip through a single job quickly when resources allow for it. On my orange pi 5 plus it can only end up using between 35-45% when of the CPU running from an NVME (a tad of a performance boost from the SD on the OPIZ3, a 2 hour vod takes about 45-50 mins). Im not sure if this is how Ganymede is coded to work or if there is some fiddling required with my docker and that a local deployment would use all the resources possible.

I've tried to get the chat render to go faster but had no luck. I've even tried using hardware accelerating while encoding and could not get it to go any faster. You may want to post an issue in the upstream repository https://github.com/lay295/TwitchDownloader about if the ffmpeg process can be any faster (consume more resources if possible).

One last suggestion as i have been rambling, is it worth having a user submitted performance spreadsheet where people can submit what render speeds people manage on different hardware? I just have a hunch people might be curios what experience to expect from different hardware and it could guide their choice if they are buying something dedicated to this task (a twitch friend saw my instance and thought it was super cool and i had to do a bit of manual benchmarking for them to make their decision). Could be a google docs spreadsheet on the main page.

Sounds great, feel free to post an issue or discussion to get it started and I can pin it for others to see and contribute.

Cheers dude, I'll let you know once I get a chance to do more testing. I've actually got the last RPI OS on the first SSD that I tested with lying about, so I will just need to dig it out and swap the drive/complete a manual install.

Interesting that you tried to fiddle about with it and didn't manage to speed it up yourself, presume that was a local deployment to? I'll try get the testing done first and then potentially I'll look into posting an issue in the upstream repository.

Once I've got some benchmarks complete I'll go about it! Hopefully, I can find a non sub walled VOD with a nicely timed length to use (such as 1, 2 or 4 hours and not 3:23:26 for example).

Isaac · Answer 7 · Mon Dec 25 2023 23:35:57 GMT+0800 (China Standard Time)

Version 2 has been released. If you encounter any issues please open a new issue or post here.

soleny894 · Answer 8 · Tue Dec 26 2023 01:05:00 GMT+0800 (China Standard Time)

Hi @Zibbp,

Thanks for the new release. I tried to upgrade to it but I'm facing an issue with the config file.
There is an error "While parsing config: unexpected end of JSON input". At first I thought it was my config, so I deleted it and I restarted the api container to let it generate a new one. But it still says the same thing again and again:

{"level":"info","time":"2023-12-25T17:54:28+01:00","message":"config file found at /data/config.json, loading"}
Version    : 
Git Hash   : 35cad5d66e771e6c79d38cf13757f4c28aa7b78a
Build Time : 2023-12-24T23:33:48Z
{"level":"info","time":"2023-12-25T17:54:28+01:00","message":"config file found at /data/config.json, loading"}
{"level":"debug","time":"2023-12-25T17:54:28+01:00","message":"config file loaded: /data/config.json"}
{"level":"debug","time":"2023-12-25T17:54:28+01:00","message":"authenticating with twitch"}
{"level":"debug","time":"2023-12-25T17:54:28+01:00","message":"config file loaded: /data/config.json"}
{"level":"panic","error":"While parsing config: unexpected end of JSON input","time":"2023-12-25T17:54:28+01:00","message":"error reading config file"}
panic: error reading config file

goroutine 1 [running]:
github.com/rs/zerolog/log.Panic.(*Logger).Panic.func1({0x188ddb0?, 0x0?})
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/log.go:396 +0x27
github.com/rs/zerolog.(*Event).msg(0xc0001c44d0, {0x188ddb0, 0x19})
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/event.go:158 +0x2c2
github.com/rs/zerolog.(*Event).Msg(...)
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/event.go:110
github.com/zibbp/ganymede/internal/config.NewConfig()
	/app/internal/config/config.go:148 +0x8ca
main.Run()
	/app/cmd/server/main.go:65 +0x25
main.main()
	/app/cmd/server/main.go:124 +0x4d4
{"level":"info","time":"2023-12-25T17:54:29+01:00","message":"authenticated with twitch"}
{"level":"debug","time":"2023-12-25T17:54:29+01:00","message":"setting up database connection"}

After this, it shows a few logs about workers but nothing more and the container is not ready.
Any idea of what I could do to fix it?

Isaac · Answer 9 · Tue Dec 26 2023 01:11:58 GMT+0800 (China Standard Time)

Does the JSON file look normal? Are the permissions to the data/config.json file and directory correct so that the container and read and write it? If the API container is running a PUID=1000 and PGID=1000 then the data directory and config.json need to be owned by the 1000 user (you).

soleny894 · Answer 10 · Tue Dec 26 2023 01:22:41 GMT+0800 (China Standard Time)

The JSON file looks normal. I generated it multiple times and the error always comes back.
The data directory and the .json file both have the correct permissions (1000:1000 with rw-r-r), container have PUID=1000 and PGID=1000.

Current config.json that fails:

{
  "active_queue_items": 2,
  "archive": {
    "save_as_hls": false
  },
  "db_seeded": false,
  "debug": false,
  "live_check_interval_seconds": 300,
  "livestream": {
    "proxies": [
      {
        "header": "",
        "url": "https://eu.luminous.dev"
      },
      {
        "header": "x-donate-to:https://ttv.lol/donate",
        "url": "https://api.ttv.lol"
      }
    ],
    "proxy_enabled": false,
    "proxy_parameters": "%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue",
    "proxy_whitelist": [
      ""
    ]
  },
  "notifications": {
    "error_enabled": true,
    "error_template": "⚠️ Error: Queue ID {{queue_id}} for {{channel_display_name}} failed at task {{failed_task}}.",
    "error_webhook_url": "",
    "is_live_enabled": true,
    "is_live_template": "🔴 {{channel_display_name}} is live!",
    "is_live_webhook_url": "",
    "live_success_enabled": true,
    "live_success_template": "✅ Live Stream Archived: {{vod_title}} by {{channel_display_name}}.",
    "live_success_webhook_url": "",
    "video_success_enabled": true,
    "video_success_template": "✅ Video Archived: {{vod_title}} by {{channel_display_name}}.",
    "video_success_webhook_url": ""
  },
  "oauth_enabled": false,
  "parameters": {
    "chat_render": "-h 1440 -w 340 --framerate 30 --font Inter --font-size 13",
    "streamlink_live": "--twitch-low-latency,--twitch-disable-hosting",
    "twitch_token": "",
    "video_convert": "-c:v copy -c:a copy"
  },
  "registration_enabled": true,
  "storage_templates": {
    "file_template": "{{id}}",
    "folder_template": "{{date}}-{{id}}-{{type}}-{{uuid}}"
  },
  "video_check_interval_minutes": 180
}

Isaac · Answer 11 · Tue Dec 26 2023 01:38:27 GMT+0800 (China Standard Time)

Hmm I was able to start the container with the provided JSON config. Can you add this to your API service in the docker-compose.yml file and bring the API container back up? It should print out what the container sees as the config.

entrypoint: "cat /data/config.json"

      - TEMPORAL_URL=temporal:7233
    volumes:
      - ./data:/data
      - ./logs:/logs
      - ./vods:/vods
    ports:
      - 4800:4000
+   entrypoint: "cat /data/config.json"

As the JSON is valid, I'm thinking that it's trying to read an empty file.

soleny894 · Answer 12 · Tue Dec 26 2023 01:46:54 GMT+0800 (China Standard Time)

I use Kubernetes, so I used command & args instead. Here's what I got from inside the API container logs:

{
  "active_queue_items": 2,
  "archive": {
    "save_as_hls": false
  },
  "db_seeded": false,
  "debug": false,
  "live_check_interval_seconds": 300,
  "livestream": {
    "proxies": [
      {
        "header": "",
        "url": "https://eu.luminous.dev"
      },
      {
        "header": "x-donate-to:https://ttv.lol/donate",
        "url": "https://api.ttv.lol"
      }
    ],
    "proxy_enabled": false,
    "proxy_parameters": "%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue",
    "proxy_whitelist": [
      ""
    ]
  },
  "notifications": {
    "error_enabled": true,
    "error_template": "⚠️ Error: Queue ID {{queue_id}} for {{channel_display_name}} failed at task {{failed_task}}.",
    "error_webhook_url": "",
    "is_live_enabled": true,
    "is_live_template": "🔴 {{channel_display_name}} is live!",
    "is_live_webhook_url": "",
    "live_success_enabled": true,
    "live_success_template": "✅ Live Stream Archived: {{vod_title}} by {{channel_display_name}}.",
    "live_success_webhook_url": "",
    "video_success_enabled": true,
    "video_success_template": "✅ Video Archived: {{vod_title}} by {{channel_display_name}}.",
    "video_success_webhook_url": ""
  },
  "oauth_enabled": false,
  "parameters": {
    "chat_render": "-h 1440 -w 340 --framerate 30 --font Inter --font-size 13",
    "streamlink_live": "--twitch-low-latency,--twitch-disable-hosting",
    "twitch_token": "",
    "video_convert": "-c:v copy -c:a copy"
  },
  "registration_enabled": true,
  "storage_templates": {
    "file_template": "{{id}}",
    "folder_template": "{{date}}-{{id}}-{{type}}-{{uuid}}"
  },
  "video_check_interval_minutes": 180
}

soleny894 · Answer 13 · Tue Dec 26 2023 02:23:09 GMT+0800 (China Standard Time)

I went back to v1.4.3 only for the API server to see if I could get the same thing. I got an error related to the database:

{"level":"fatal","error":"error creating user: ent: constraint failed: pq: duplicate key value violates unique constraint \"users_username_key\"","time":"2023-12-25T19:14:44+01:00","message":"error seeding database"}

Turned "db_seeded" from false to true in the config, restarted the container and it was up and running.
Reverted back to v2.0.0 for the API server, same config error as before even with the working configuration from 1.4.3.

Isaac · Answer 14 · Wed Dec 27 2023 01:01:25 GMT+0800 (China Standard Time)

I'm starting to think it may be related to the package used for the config. Still not sure why it's only effecting you. I've created a new branch that drops the version to an older version. This is slightly newer than the one running in v1.4.3 so not 100% if it will work.

Can you pull #333 build the image locally and give it a try?

soleny894 · Answer 15 · Wed Dec 27 2023 01:56:47 GMT+0800 (China Standard Time)

Can you pull #333 build the image locally and give it a try?

Just tried with the image I built and unfortunately I still face the same config error.
I noticed something when doing more tests. It replaces the config file every time I start the container. For example, if I set db_seeded to true and if I change some other values without breaking the json format, the whole config is replaced by a new one when the container is started. The logs says it loaded the config multiple times but never said it created a new one, but it was still replaced.

Isaac · Answer 16 · Wed Dec 27 2023 03:07:28 GMT+0800 (China Standard Time)

Can you share your Kubernetes manifests? It must be something in there then

soleny894 · Answer 17 · Wed Dec 27 2023 03:43:20 GMT+0800 (China Standard Time)

Can you share your Kubernetes manifests?

Here is my manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ganymede-app-api
  name: ganymede-app-api-deployment
  namespace: ganymede
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ganymede-app-api
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: ganymede-app-api
    spec:
      containers:
        - name: ganymede-api
          env:
            - name: PUID
              value: "1000"
            - name: PGID
              value: "1000"
            - name: DB_HOST
              value: ganymede-db-svc
            - name: DB_NAME
              value: ganymede
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  name: ganymede-app-secret
                  key: DB_PASS
            - name: DB_PORT
              value: "5432"
            - name: DB_SSL
              value: disable
            - name: DB_USER
              value: ganymede
            - name: FRONTEND_HOST
              value: https://vods.domain.com
            - name: JWT_REFRESH_SECRET
              valueFrom:
                secretKeyRef:
                  name: ganymede-app-secret
                  key: JWT_REFRESH_SECRET
            - name: JWT_SECRET
              valueFrom:
                secretKeyRef:
                  name: ganymede-app-secret
                  key: JWT_SECRET
            - name: TWITCH_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: ganymede-app-secret
                  key: TWITCH_CLIENT_ID
            - name: TWITCH_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: ganymede-app-secret
                  key: TWITCH_CLIENT_SECRET
            - name: TEMPORAL_URL
              value: "ganymede-temporal-svc:7233"
            - name: MAX_CHAT_DOWNLOAD_EXECUTIONS
              value: "5"
            - name: MAX_CHAT_RENDER_EXECUTIONS
              value: "3"
            - name: MAX_VIDEO_DOWNLOAD_EXECUTIONS
              value: "5"
            - name: MAX_VIDEO_CONVERT_EXECUTIONS
              value: "3"
            - name: TZ
              value: Europe/Amsterdam
          resources: {}
          image: ghcr.io/zibbp/ganymede:v2.0.0
          ports:
            - containerPort: 4000
          volumeMounts:
            - mountPath: /vods
              name: nfs-ganymede-app
              subPath: vods
            - mountPath: /logs
              name: nfs-ganymede-app
              subPath: logs
            - mountPath: /data
              name: nfs-ganymede-app
              subPath: data
            - mountPath: /tmp
              name: vods-tmp-dir
      volumes:
        - name: nfs-ganymede-app
          persistentVolumeClaim:
            claimName: nfs-ganymede-app
        - name: vods-tmp-dir
          emptyDir: {}

I use a nfs mount that keeps the data. The config file is also there and is always mounted correctly in the container (the data is also saved when I shutdown the container as is).

I've been able to start the API server by going manually in the container, turning db_seeded to true in the config and then starting the API with /opt/app/ganymede-api. It then works and I can access my vods from the frontend.

{"level":"info","time":"2023-12-26T20:25:38+01:00","message":"config file found at /data/config.json, loading"}
{"level":"debug","time":"2023-12-26T20:25:38+01:00","message":"config file loaded: /data/config.json"}
2023/12/26 20:25:38 INFO  No logger configured for temporal client. Created default one.
{"level":"info","time":"2023-12-26T20:25:38+01:00","message":"Connected to temporal at ganymede-temporal-svc:7233"}

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.11.3
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:4000
{"level":"info","time":"2023-12-26T20:25:39+01:00","message":"authenticated with twitch"}
{"level":"info","time":"2023-12-26T20:25:43+01:00","message":"setting up check watched channel videos schedule"}
{"level":"info","time":"2023-12-26T20:25:43+01:00","message":"running check watched channel videos schedule"}

I'm not sure it's coming from my setup. The issue seems to happen when it initialize, it loads the config but fails and it also replace the config file even if it loads it.

soleny894 · Answer 18 · Wed Dec 27 2023 05:59:32 GMT+0800 (China Standard Time)

I've been finally able to fix my issue. I commented out this part of the code to stop the refreshConfig function to do anything to my configuration (that was already newly created anyway). Now I can start my container 100% of the time. I went back to v2.0.0 to make sure that was the real fix and not something else and I got the config error again. Switched back to my image and the error was gone.

} else {
		log.Info().Msgf("config file found at %s, loading", configPath)
		err := viper.ReadInConfig()
		// Rewrite config file to apply new variables and remove old values
+		//refreshConfig(configPath)
		log.Debug().Msgf("config file loaded: %s", viper.ConfigFileUsed())
		if err != nil {
			log.Panic().Err(err).Msg("error reading config file")
		}
	}

StephenC · Answer 19 · Wed Dec 27 2023 07:32:19 GMT+0800 (China Standard Time)

quick question, I had a queue built up since it got stuck while I was sick. After upgrading I still see the queue of 70 some items but I don't see any workflows running. Is there a way to import the old queue to have it process these with the new temporal workflow?

Isaac · Answer 20 · Wed Dec 27 2023 08:40:31 GMT+0800 (China Standard Time)

quick question, I had a queue built up since it got stuck while I was sick. After upgrading I still see the queue of 70 some items but I don't see any workflows running. Is there a way to import the old queue to have it process these with the new temporal workflow?

Unfortunately not, you will need to re-archive the VODs to get them to process using the new system. If you have the list of IDs you can create a simple bash loop to archive the vods.

curl --request POST \
  --url http://IP:4800/api/v1/archive/vod \
  --header 'Content-Type: application/json' \
  --cookie 'access-token=<ACCESS TOKEN COOKIE VALUE>' \
  --data '{
	"vod_id": "twitch_vod_id",
	"quality": "best",
	"chat": true,
	"render_chat": true
}'

I've been finally able to fix my issue. I commented out this part of the code to stop the refreshConfig function to do anything to my configuration (that was already newly created anyway). Now I can start my container 100% of the time. I went back to v2.0.0 to make sure that was the real fix and not something else and I got the config error again. Switched back to my image and the error was gone.

Not sure how commenting out the refreshConfig allows it to properly boot. The original error stated it had issues loading the config which runs before the refreshConfig code. As this is all over NFS maybe there's a weird issue there? I've added some retry logic to the config just to see if it works if you want to build an image from #334.

soleny894 · Answer 21 · Wed Dec 27 2023 17:27:49 GMT+0800 (China Standard Time)

Not sure how commenting out the refreshConfig allows it to properly boot. The original error stated it had issues loading the config which runs before the refreshConfig code.

For some reason, if you check the logs I provided in my first post, it loads the config 2 times based on the "config file loaded" log. In the code, the debug message is placed after the function and since my config was being edited all the time the container started, I commented it to see if it would change anything. Not sure what's running it 2 times but commenting the function did fix my issue.

I've added some retry logic to the config just to see if it works if you want to build an image from #334.

It works for me:

{"level":"info","time":"2023-12-27T09:50:49+01:00","message":"Starting worker with config: {MAX_CHAT_DOWNLOAD_EXECUTIONS:3 MAX_CHAT_RENDER_EXECUTIONS:3 MAX_VIDEO_DOWNLOAD_EXECUTIONS:3 MAX_VIDEO_CONVERT_EXECUTIONS:3 TEMPORAL_URL:ganymede-temporal-svc:7233}"}
{"level":"info","time":"2023-12-27T09:50:49+01:00","message":"config file found at /data/config.json, loading"}
{"level":"info","time":"2023-12-27T09:50:49+01:00","message":"config file loaded: /data/config.json"}
Version    : 
Git Hash   : b7cf72469b4df6b71401fe6b9ff3b4d61f993840
Build Time : 2023-12-27T08:38:32Z
{"level":"info","time":"2023-12-27T09:50:49+01:00","message":"config file found at /data/config.json, loading"}
{"level":"error","error":"While parsing config: unexpected end of JSON input","time":"2023-12-27T09:50:49+01:00","message":"error loading config (attempt 1/10)"}
{"level":"info","time":"2023-12-27T09:50:49+01:00","message":"retrying in 1 second"}
{"level":"debug","time":"2023-12-27T09:50:49+01:00","message":"config file loaded: /data/config.json"}
{"level":"debug","time":"2023-12-27T09:50:49+01:00","message":"authenticating with twitch"}
{"level":"info","time":"2023-12-27T09:50:50+01:00","message":"config file loaded: /data/config.json"}
{"level":"info","time":"2023-12-27T09:50:50+01:00","message":"authenticated with twitch"}
{"level":"debug","time":"2023-12-27T09:50:50+01:00","message":"setting up database connection"}
{"level":"info","Namespace":"default","TaskQueue":"archive","WorkerID":"ganymede-app-api-deploy-5557cf8644-2gf62","time":"2023-12-27T09:50:50+01:00","message":"Started Worker"}
{"level":"debug","time":"2023-12-27T09:50:50+01:00","message":"config file loaded: /data/config.json"}
2023/12/27 09:50:50 INFO  No logger configured for temporal client. Created default one.
{"level":"info","time":"2023-12-27T09:50:50+01:00","message":"Connected to temporal at ganymede-temporal-svc:7233"}

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.11.3
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:4000
{"level":"info","Namespace":"default","TaskQueue":"chat-download","WorkerID":"ganymede-app-api-deploy-5557cf8644-2gf62","time":"2023-12-27T09:50:50+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"chat-render","WorkerID":"ganymede-app-api-deploy-5557cf8644-2gf62","time":"2023-12-27T09:50:50+01:00","message":"Started Worker"}
{"level":"info","time":"2023-12-27T09:50:51+01:00","message":"authenticated with twitch"}
{"level":"info","Namespace":"default","TaskQueue":"video-download","WorkerID":"ganymede-app-api-deploy-5557cf8644-2gf62","time":"2023-12-27T09:50:51+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"video-convert","WorkerID":"ganymede-app-api-deploy-5557cf8644-2gf62","time":"2023-12-27T09:50:51+01:00","message":"Started Worker"}
{"level":"info","time":"2023-12-27T09:50:55+01:00","message":"setting up check watched channel videos schedule"}
{"level":"info","time":"2023-12-27T09:50:55+01:00","message":"running check watched channel videos schedule"}

It does error out (every single time I start the container), but the new attempt allows the API server to start. Timing issue maybe?
The same logic need to be used for the config creation. I tested without having a config.json file and I got the same error (it creates it even if it says it can't find it):

{"level":"info","time":"2023-12-27T10:15:19+01:00","message":"Starting worker with config: {MAX_CHAT_DOWNLOAD_EXECUTIONS:3 MAX_CHAT_RENDER_EXECUTIONS:3 MAX_VIDEO_DOWNLOAD_EXECUTIONS:3 MAX_VIDEO_CONVERT_EXECUTIONS:3 TEMPORAL_URL:ganymede-temporal-svc:7233}"}
{"level":"info","time":"2023-12-27T10:15:19+01:00","message":"config file not found at /data/config.json, creating new one"}
Version    : 
Git Hash   : b7cf72469b4df6b71401fe6b9ff3b4d61f993840
Build Time : 2023-12-27T08:38:32Z
{"level":"info","time":"2023-12-27T10:15:19+01:00","message":"config file not found at /data/config.json, creating new one"}
{"level":"panic","error":"open /data/config.json: file exists","time":"2023-12-27T10:15:19+01:00","message":"error creating config file"}
panic: error creating config file

goroutine 1 [running]:
github.com/rs/zerolog/log.Panic.(*Logger).Panic.func1({0x188ff21?, 0x0?})
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/log.go:396 +0x27
github.com/rs/zerolog.(*Event).msg(0xc00031f0a0, {0x188ff21, 0x1a})
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/event.go:158 +0x2c2
github.com/rs/zerolog.(*Event).Msg(...)
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/event.go:110
github.com/zibbp/ganymede/internal/config.NewConfig()
	/app/internal/config/config.go:140 +0x81e
main.Run()
	/app/cmd/server/main.go:65 +0x25
main.main()
	/app/cmd/server/main.go:124 +0x4d4
{"level":"debug","time":"2023-12-27T10:15:19+01:00","message":"authenticating with twitch"}
{"level":"info","time":"2023-12-27T10:15:20+01:00","message":"authenticated with twitch"}
{"level":"debug","time":"2023-12-27T10:15:20+01:00","message":"setting up database connection"}

Restarting the container make it works, but I get the database error instead (I have to set db_seeded to true manually):

{"level":"info","time":"2023-12-27T10:16:12+01:00","message":"Starting worker with config: {MAX_CHAT_DOWNLOAD_EXECUTIONS:3 MAX_CHAT_RENDER_EXECUTIONS:3 MAX_VIDEO_DOWNLOAD_EXECUTIONS:3 MAX_VIDEO_CONVERT_EXECUTIONS:3 TEMPORAL_URL:ganymede-temporal-svc:7233}"}
{"level":"info","time":"2023-12-27T10:16:12+01:00","message":"config file found at /data/config.json, loading"}
{"level":"info","time":"2023-12-27T10:16:12+01:00","message":"config file loaded: /data/config.json"}
Version    : 
Git Hash   : b7cf72469b4df6b71401fe6b9ff3b4d61f993840
Build Time : 2023-12-27T08:38:32Z
{"level":"info","time":"2023-12-27T10:16:12+01:00","message":"config file found at /data/config.json, loading"}
{"level":"error","error":"While parsing config: unexpected end of JSON input","time":"2023-12-27T10:16:12+01:00","message":"error loading config (attempt 1/10)"}
{"level":"info","time":"2023-12-27T10:16:12+01:00","message":"retrying in 1 second"}
{"level":"debug","time":"2023-12-27T10:16:12+01:00","message":"config file loaded: /data/config.json"}
{"level":"debug","time":"2023-12-27T10:16:12+01:00","message":"authenticating with twitch"}
{"level":"info","time":"2023-12-27T10:16:13+01:00","message":"config file loaded: /data/config.json"}
{"level":"debug","time":"2023-12-27T10:16:13+01:00","message":"config file loaded: /data/config.json"}
{"level":"info","time":"2023-12-27T10:16:13+01:00","message":"authenticated with twitch"}
{"level":"debug","time":"2023-12-27T10:16:13+01:00","message":"setting up database connection"}
{"level":"info","Namespace":"default","TaskQueue":"chat-download","WorkerID":"ganymede-app-api-deploy-7b745c88b5-vnp9k","time":"2023-12-27T10:16:13+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"chat-render","WorkerID":"ganymede-app-api-deploy-7b745c88b5-vnp9k","time":"2023-12-27T10:16:14+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"video-download","WorkerID":"ganymede-app-api-deploy-7b745c88b5-vnp9k","time":"2023-12-27T10:16:14+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"video-convert","WorkerID":"ganymede-app-api-deploy-7b745c88b5-vnp9k","time":"2023-12-27T10:16:14+01:00","message":"Started Worker"}
{"level":"fatal","error":"error creating user: ent: constraint failed: pq: duplicate key value violates unique constraint \"users_username_key\"","time":"2023-12-27T10:16:14+01:00","message":"error seeding database"}
{"level":"info","Namespace":"default","TaskQueue":"archive","WorkerID":"ganymede-app-api-deploy-7b745c88b5-vnp9k","time":"2023-12-27T10:16:14+01:00","message":"Started Worker"}

Isaac · Answer 22 · Thu Dec 28 2023 00:25:02 GMT+0800 (China Standard Time)

The config being loaded twice is expected. The second is the worker process loading the config. I'm assuming this is causing some issues as the API and worker are trying to load and perform modifications to the config file at the same time. I'm working on an update to the branch now.

Isaac · Answer 23 · Thu Dec 28 2023 00:53:23 GMT+0800 (China Standard Time)

I've pushed some more changes to #334 if you want to give it another try. I've added a delay starting the worker process to hopefully resolve the config file conflicts. I've also updated the worker to not refresh the config which should fix the db_seeded being reset.

soleny894 · Answer 24 · Thu Dec 28 2023 01:48:33 GMT+0800 (China Standard Time)

I've pushed some more changes to #334 if you want to give it another try. I've added a delay starting the worker process to hopefully resolve the config file conflicts. I've also updated the worker to not refresh the config which should fix the db_seeded being reset.

The config error is completely gone now, that's good. Tested both with and without config and it works. The only thing remaining is the db error when creating a new config. It should set db_seeded to true if it detects that the user is already present in the database. It would fix the other github issue I've seen too.

Version    : 
Git Hash   : 29e8ded192b024d87fb5fead69921ce60d672388
Build Time : 2023-12-27T17:24:17Z
{"level":"info","time":"2023-12-27T18:40:04+01:00","message":"config file not found at /data/config.json, creating new one"}
{"level":"info","time":"2023-12-27T18:40:05+01:00","message":"config file created"}
{"level":"fatal","error":"error creating user: ent: constraint failed: pq: duplicate key value violates unique constraint \"users_username_key\"","time":"2023-12-27T18:40:05+01:00","message":"error seeding database"}
{"level":"info","time":"2023-12-27T18:40:09+01:00","message":"Starting worker with config: {MAX_CHAT_DOWNLOAD_EXECUTIONS:3 MAX_CHAT_RENDER_EXECUTIONS:3 MAX_VIDEO_DOWNLOAD_EXECUTIONS:3 MAX_VIDEO_CONVERT_EXECUTIONS:3 TEMPORAL_URL:ganymede-temporal-svc:7233}"}
{"level":"info","time":"2023-12-27T18:40:09+01:00","message":"config file found at /data/config.json, loading"}
{"level":"info","time":"2023-12-27T18:40:09+01:00","message":"config file loaded: /data/config.json"}
{"level":"debug","time":"2023-12-27T18:40:09+01:00","message":"config file loaded: /data/config.json"}
{"level":"debug","time":"2023-12-27T18:40:09+01:00","message":"authenticating with twitch"}
{"level":"info","time":"2023-12-27T18:40:10+01:00","message":"authenticated with twitch"}
{"level":"debug","time":"2023-12-27T18:40:10+01:00","message":"setting up database connection"}
{"level":"info","Namespace":"default","TaskQueue":"archive","WorkerID":"ganymede-app-api-deploy-864bdbf998-s5d6q","time":"2023-12-27T18:40:10+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"chat-download","WorkerID":"ganymede-app-api-deploy-864bdbf998-s5d6q","time":"2023-12-27T18:40:11+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"chat-render","WorkerID":"ganymede-app-api-deploy-864bdbf998-s5d6q","time":"2023-12-27T18:40:11+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"video-download","WorkerID":"ganymede-app-api-deploy-864bdbf998-s5d6q","time":"2023-12-27T18:40:11+01:00","message":"Started Worker"}
{"level":"info","Namespace":"default","TaskQueue":"video-convert","WorkerID":"ganymede-app-api-deploy-864bdbf998-s5d6q","time":"2023-12-27T18:40:12+01:00","message":"Started Worker"}

Isaac · Answer 25 · Thu Dec 28 2023 04:14:12 GMT+0800 (China Standard Time)

Pushed a new commit to the branch that detects if the DB should be seeded if there are no users. I've also removed the config options as that will no longer be needed. Testing this on my side now.

soleny894 · Answer 26 · Thu Dec 28 2023 04:30:56 GMT+0800 (China Standard Time)

All good for me it seems. All the issues are now fixed. Thank you!

Isaac · Answer 27 · Thu Dec 28 2023 04:32:08 GMT+0800 (China Standard Time)

Sweet, thanks for helping troubleshoot! I'll publish a new release shortly.

c-hri-s · Answer 28 · Sun Dec 31 2023 05:33:12 GMT+0800 (China Standard Time)

Not entirely sure if this is connected, but not sure the 'to not break existing installs' part is working, at least not for me.
I use watchtower to keep things updated to :latest, and for the past week or so ganymede-api crashes on launch as follows:

Running on a Synology NAS. Haven't changed anything in the config/docker-compose for months.

-------------------------------------
User uid:    1024
User gid:    101
-------------------------------------
Version    : 
Git Hash   : 2a9ebe83aad91771a4e9e46da43ecc8e2ed54c1b
Build Time : 2023-12-27T21:07:21Z
2023-12-30T21:23:53Z 2023-12-30T21:23:53Z INF 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z INF 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z INF 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z INF config file found at /data/config.json, loading 
2023-12-30T21:23:53Z 2023-12-30T21:23:53Z INF 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z INF 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z INF 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z INF config file loaded: /data/config.json 
2023-12-30T21:23:53Z 2023-12-30T21:23:53Z DBG 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z DBG 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z DBG 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z DBG config file loaded: /data/config.json 
2023/12/30 21:23:53 INFO  No logger configured for temporal client. Created default one.
2023-12-30T21:23:53Z 2023-12-30T21:23:53Z panic 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z panic 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z panic 2023-12-30T21:23:53Z 2023-12-30T21:23:53Z panic Unable to create client: failed reaching server: last connection error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:7233: connect: connection refused" 
panic: Unable to create client: failed reaching server: last connection error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:7233: connect: connection refused"
goroutine 1 [running]:
github.com/rs/zerolog/log.Panic.(*Logger).Panic.func1({0xc0005ee180?, 0x0?})
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/log.go:396 +0x27
github.com/rs/zerolog.(*Event).msg(0xc0003c2850, {0xc0005ee180, 0xb7})
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/event.go:158 +0x2c2
github.com/rs/zerolog.(*Event).Msgf(0xc0003c2850, {0x1891ce4?, 0x0?}, {0xc0006499b0?, 0x0?, 0x0?})
	/go/pkg/mod/github.com/rs/zerolog@v1.31.0/event.go:131 +0x46
github.com/zibbp/ganymede/internal/temporal.InitializeTemporalClient()
	/app/internal/temporal/client.go:26 +0x145
main.Run()
	/app/cmd/server/main.go:85 +0xd9
main.main()
	/app/cmd/server/main.go:124 +0x4d4
2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF Starting worker with config: {MAX_CHAT_DOWNLOAD_EXECUTIONS:5 MAX_CHAT_RENDER_EXECUTIONS:3 MAX_VIDEO_DOWNLOAD_EXECUTIONS:5 MAX_VIDEO_CONVERT_EXECUTIONS:3 TEMPORAL_URL:temporal:7233} 
2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF config file found at /data/config.json, loading 
2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z INF config file loaded: /data/config.json 
2023-12-30T21:23:58Z 2023-12-30T21:23:58Z DBG 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z DBG 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z DBG 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z DBG config file loaded: /data/config.json 
2023-12-30T21:23:58Z 2023-12-30T21:23:58Z fatal 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z fatal 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z fatal 2023-12-30T21:23:58Z 2023-12-30T21:23:58Z fatal Unable to create client: failed reaching server: last connection error: connection error: desc = "transport: Error while dialing: dial tcp: lookup temporal on 127.0.0.11:53: no such host" 
usermod: no changes

Edit: Adding the new sections to the docker-compose sorted things out, but the fallback didn't seem to work for me at least

Isaac · Answer 29 · Sun Dec 31 2023 06:02:33 GMT+0800 (China Standard Time)

Ya I ended up not bundling it in the API container to make it easier on myself and keep everyone running the same setup.

Entrivax · Answer 30 · Mon Jan 01 2024 01:45:56 GMT+0800 (China Standard Time)

I have a question with this update: how can I restart part of the process of an item in the queue, since the "restart" button in not there anymore? I had to restart Ganymede at some point, and some of the items in the queue does not have a workflow anymore, some of them still need to convert the video, others need to move the converted video and some need to start the download.

Isaac · Answer 31 · Mon Jan 01 2024 22:59:46 GMT+0800 (China Standard Time)

I have a question with this update: how can I restart part of the process of an item in the queue, since the "restart" button in not there anymore? I had to restart Ganymede at some point, and some of the items in the queue does not have a workflow anymore, some of them still need to convert the video, others need to move the converted video and some need to start the download.

You can restart individual tasks, or the "parent" task of that individual workflow. To accomplish this, find the workflow on the Workflows page. As these haven't completed yet they might still be on the "Active" page, or check the "Closed" page.

Then click it to see more information and click the restart button at the top.

The UI/UX of the workflows isn't great and I'm still figuring out how to improve on it. You'll also only be able to see workflows from the past day. I'm still working on fixing that.

Entrivax · Answer 32 · Tue Jan 02 2024 00:14:30 GMT+0800 (China Standard Time)

I have a question with this update: how can I restart part of the process of an item in the queue, since the "restart" button in not there anymore? I had to restart Ganymede at some point, and some of the items in the queue does not have a workflow anymore, some of them still need to convert the video, others need to move the converted video and some need to start the download.

You can restart individual tasks, or the "parent" task of that individual workflow. To accomplish this, find the workflow on the Workflows page. As these haven't completed yet they might still be on the "Active" page, or check the "Closed" page. Then click it to see more information and click the restart button at the top.

The UI/UX of the workflows isn't great and I'm still figuring out how to improve on it. You'll also only be able to see workflows from the past day. I'm still working on fixing that.

That's what I guessed, but the items I still have in the queue do not have a workflow anymore for an unknown reason.
Here I have 15 items remaining in the queue, but I have the workflow of one item, that is already finished in this case:

Isaac · Answer 33 · Tue Jan 02 2024 01:52:35 GMT+0800 (China Standard Time)

That's because the queue items are over a day old and the workflow items currently get removed after a day. I've got a fix ready to keep workflows around for significantly longer for the temporal image but need to make some changes for the API and frontend. You'll need to re-archive the queue items.

Entrivax · Answer 34 · Tue Jan 02 2024 04:16:54 GMT+0800 (China Standard Time)

Alright, I will do the remaining of the tasks by hand and manually set them as completed for the ones with the download completed (I don't want to redownload 20+GB per VOD, thank you for logging the ffmpeg commands executed btw).
For the rest, I will re-archive.