Note
UNOFFICIAL HELPER SCRIPT
Migration helper for TubeArchivist from the legacy filenaming convention (v0.3.6 and earlier) to the current naming convention (v0.4.0 and later).
- Detects videos in filesystem that are not in ElasticSearch.
- Detects videos that are in ElasticSearch and not in the filesystem.
- Provides and output of what those videos are.
- Allow automatic migration for those files that are detected.
- Allow automatic updates to Elasticsearch.
- Allow migration of subtitles/other files found.
- This is expected to run within a running TubeArchivist container instance that is running v0.4.0 and had issues with migrating a larger library of files.
- This is only expecting to move
.mp4and.vttfiles that align with TubeArchivist downloads. Will not move other files at this time. - If a file exists on the Filesystem in the
SOURCE_DIRand not within ElasticSearch, you will need to run a manual Filesystem Rescan or Manual Import to pickup those files. This will be alerted if an automated migration is attempted. - If a file exists within ElasticSearch and not on the Filesystem in the
SOURCE_DIR, you will need to locate the files before attempting to perform a migration. Alternatively, if you perform a manual Filesystem Rescan, TubeArchivist will remove those entries for you. - This makes several assumptions during the runtime process. If those are incorrect or different from how TubeArchivist expects them to be performed, those specific items may not work properly and will need to be updated manually.
- If a channel no longer exists for
yt-dlpto find or within ElasticSearch, you will need to include achannel.idfile within the channel's directory that includes its ID.
Warning
Using the PERFORM_MIGRATION action is a destructive process and could cause issues with files. It is recommended to not use it unless advised or after you have reviewed an initial output of what is expected to happen.
| Argument | Flag | Default | Purpose |
|---|---|---|---|
SOURCE_DIR |
-d | /youtube |
The source directory that will be searched for videos that need to be migrated. This can be used to specify an individual folder instead of the entire /youtube directory1. |
USE_YTDLP |
-Y | True |
Allows the user to disable calls to YouTube via yt-dlp. This will not allow any calls to YouTube and will instead only search ElasticSearch. |
YTDLP_SLEEP |
-s | 3 |
Number of seconds to wait between each call to YouTube when using yt-dlp. Value will not be used if USE_YTDLP is set to False. |
PERFORM_MIGRATION |
-M | False |
If set to False, this will perform a review of what files need to be migrated and why. If set to True, this will attempt to migrate all files2. |
DEBUG |
-B | False |
If set to True, this will show debugging outputs. |
DRY_RUN |
-r | False |
If set to True and PERFORM_MIGRATION is True, then it will only show what it expects to change. All details are preceeded with a DRY_RUN statement. |
GUESS_TYPES |
-g | False |
If set to True, will attempt to guess the type of the files by looking at the file itself. Decreases chances of false positives based on file extension, but does access the file and can slow down the analysis. |
This is expected to run from within the TubeArchivist container, at the /app directory. This allows it to see the TubeArchivist helper functions.
You can run this script with the optional flags. For example:
python ta_migration_helper.py -s /path/to/custom/directory -Y -S 5 -M
This would set the source directory to /path/to/custom/directory, disable YouTube calls via yt-dlp, set the sleep time to 5 seconds between YouTube calls, and enable migration.
Footnotes
-
This could cause issues with the migration portion, as it will be relative to the
SOURCE_DIR. ↩ -
This is a destructive process and could cause issues with files. There is a ten second barrier after analysis to allow cancellation of the script before starting the migration process - once it is started, it should not be interrupted. ↩