Unhandled exception during the Chat Download step

[Entrivax]

During the Chat Download step, it fails around the end. I managed to have this error on one stream as of now,

Step logs:

[STATUS] - Downloading 68%
[STATUS] - Downloading 73%
[STATUS] - Downloading 78%
[STATUS] - Downloading 82%
[STATUS] - Downloading 87%
[STATUS] - Downloading 90%
[STATUS] - Downloading + Embedding ImagesUnhandled exception. System.AggregateException: One or more errors occurred. (Response status code does not indicate success: 404 (Not Found).)
---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at TwitchDownloaderCore.TwitchHelper.GetStvEmoteData(Int32 streamerId, List`1 stvResponse, Boolean allowUnlistedEmotes)
at TwitchDownloaderCore.TwitchHelper.GetThirdPartyEmoteData(Int32 streamerId, Boolean getBttv, Boolean getFfz, Boolean getStv, Boolean allowUnlistedEmotes, CancellationToken cancellationToken)
at TwitchDownloaderCore.TwitchHelper.GetThirdPartyEmotes(List`1 comments, Int32 streamerId, String cacheFolder, EmbeddedData embeddedData, Boolean bttv, Boolean ffz, Boolean stv, Boolean allowUnlistedEmotes, Boolean offline, CancellationToken cancellationToken)
at TwitchDownloaderCore.ChatDownloader.DownloadAsync(IProgress`1 progress, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean )
at System.Threading.Tasks.Task.Wait(Int32 , CancellationToken )
at System.Threading.Tasks.Task.Wait()
at TwitchDownloaderCLI.Modes.DownloadChat.Download(ChatDownloadArgs inputOptions)
at CommandLine.ParserResultExtensions.WithParsed[T](ParserResult`1 result, Action`1 action)
at TwitchDownloaderCLI.Program.Main(String[] args)

API logs:

{"level":"error","error":"signal: aborted","time":"2023-11-20T20:31:44+01:00","message":"error running TwitchDownloaderCLI for vod chat download"}
{"level":"error","error":"signal: aborted","time":"2023-11-20T20:31:44+01:00","message":"error downloading chat"}

Ganymede info:

Ganymede was deployed with the latest tag from the registry.
Also, I managed to download successfully the chat from the faulty stream with the CLI directly on Windows, with TwitchDownloaderCLI 1.53.2 (the same version as for Ganymede) and the same with the 1.53.4.

> ./TwitchDownloaderCLI.exe chatdownload --id 1980026722 -o chat.json -E
TwitchDownloaderCLI 1.53.2 Copyright (c) 2019 lay295 and contributors
[STATUS] - Downloading 90%
[STATUS] - Downloading + Embedding Images
[STATUS] - Backfilling commenter info
[STATUS] - Writing output file

> ./TwitchDownloaderCLI.exe chatdownload --id 1980026722 -o chat.json -E
TwitchDownloaderCLI 1.53.4 Copyright (c) lay295 and contributors
[STATUS] - Downloading 90%
[STATUS] - Downloading + Embedding Images
[STATUS] - Backfilling commenter info
[STATUS] - Writing output file

[Isaac]

I was able to download the chat of 1980026722 on my test instance, running version TwitchDownloaderCLI 1.53.2. Can you retry the chat download task again? Are you running the ARM image or x86?

[Entrivax]

I retried now and this time it worked.
It should be the x86 image.

{"level":"error","error":"signal: aborted","time":"2023-11-20T20:30:50+01:00","message":"error running TwitchDownloaderCLI for vod chat download"}
{"level":"error","error":"signal: aborted","time":"2023-11-20T20:30:50+01:00","message":"error downloading chat"}
{"level":"error","error":"signal: aborted","time":"2023-11-20T20:31:44+01:00","message":"error running TwitchDownloaderCLI for vod chat download"}
{"level":"error","error":"signal: aborted","time":"2023-11-20T20:31:44+01:00","message":"error downloading chat"}
{"level":"info","time":"2023-11-20T22:28:09+01:00","message":"running check watched channel videos schedule"}
{"level":"error","error":"signal: aborted","time":"2023-11-20T22:51:16+01:00","message":"error running TwitchDownloaderCLI for vod chat download"}
{"level":"error","error":"signal: aborted","time":"2023-11-20T22:51:16+01:00","message":"error downloading chat"}

The first 3 times (with even 2 hours between the 2nd and 3rd tries), it didn't, and retried just now and it works.
The JSON of the chat is even bigger now than it was from when I downloaded it directly with the CLI. Must have been a server error from 7TV and the CLI just skipping it, I guess?

I suppose there is nothing to do then, and just close the issue.

And for the record, if someone happens to have the same problem, maybe retry (a few hours) later?

[Isaac]

Ya there's not much to do here, other than retry at another time. I'm in the early stages of re-doing the queue system with retries included for tasks, though it may not help much if you waited two hours in this case.

[Entrivax]

Yeah, in this case, two hours was not even enough, but 5+ hours later, it was fine. So except if you allow to auto-retry on multiple hours, there is not much to do.

I'm closing the issue, then.