Move video assets outside of repo
agg23 opened this issue · comments
I was surprised to find that cloning the repo took a significant amount of time. The current on disk size of the repo is 508MB, which is quite substantial, and seems unnecessary for a collection of documentation.
Better for the clone to take long than for video assets to get lost when hosted on external servers.
For the reasons Natrox stated I won't be moving the video assets outside of the repo. This is this way because of how both GitHub Pages and MkDocs work. The site is hosted by GitHub by using the page as generated here --> https://github.com/MiSTer-devel/MkDocs_MiSTer/tree/gh-pages - MkDocs is a static site generator.
You can save slightly more time using git clone --depth 1 https://github.com/MiSTer-devel/MkDocs_MiSTer.git. After that initial clone, it's only deltas, so it won't be a major issue any longer.
May be can make a submodule with video and put it into separate repository. So the one who wants just text pages can simply use git clone. Others who will like with video can use git clone --recursive or use submodule command.
Probably it will be more convenient for PR when you don't need to clone with submodules.
@sorgelig's suggestion is exactly what I was thinking. I'm not sure if it would interfere with the site generation (it shouldn't, you can still clone like normal), but it would be nicer for users that just want to update docs, and never will update videos.
I'll look into how this may affect the static site generation using the workflows. I may have to add some environment variables in the github workflow to accomplish this, there might be discussions about this on the MkDocs for Material page.
I could make a separate branch on the same repo that has all of the videos, that way git clone --depth 1 should only clone the main branch.
This would be simpler as it would not need a separate repo. During the checkout step of the workflow to generate the static site I could potentially add a git merge step and have it merge with this other branch, putting the videos files in separate folders. Then just update all of the relative links in all the documents with a suffix to that different top level folder... Need to test the workflow stuff out here...
If that works, I like that solution way more since it's self contained.
My idea to use git merge is wrong I'm realizing... that would bork it all up. I'm seeing ways to copy a folder from a separate branch as part of a workflow online, might be able to adapt something like that, going to test it out later probably. It's a fun thing to learn to do at the very least.
I have a spin on the idea of a sub-module that may be easier to work with.
Two separate projects:
- MkDocs_MiSTer_text
- MkDocs_MiSTer_media
These can be used to divvy up the different media. Then, to tie them together:
- MkDocs_MiSTer (this project) which consists of two sub-modules (MkDocs_MiSTer_text and MkDocs_MiSTer_media).
This way you can keep them both separate and people can choose which one to clone. The main MkDocs_MiSTer branch would just be kept up-to-date by syncing the latest text and media projects as sub-modules.
It provides a clear hierarchy, if ever you need to add more elements, for example IMG files for the OS, you could create an additional MkDocs_MiSTer_data project or whatever you'd like to call it and amend the main project with a new sub-module to it.
As for how to go about this, I would recommend forking this repository and isolating all of the data. This keeps your revision history in place.
Videos make up almost half of the total asset size. As a first step I'll make sure to reduce the size of the repo as cloned to 1 depth because it's not just videos causing this :P I must've missed some.
Tip if you're on Linux:
tree -h -C --du --sort size . | less
:)
Oh I'm an idiot, it's because of the pack file, which is always going to be relative to the full file size of all the files in your repo.
Originally when I started making this I experimented with using embedded youtube videos instead, but I really didn't like how that flowed, there were problems embedding external videos with Material for MkDocs for some reason. I had tried multiple video embed plugins that were available. The quickest loading and smoothest option in terms of aspect ratio, border, etc... was to do the current method. I also just checked the file sizes and tested some compression, it's about as efficient as it can get without sacrificing too much quality, so I'll just continue exploring the workflow to copy a folder from a branch before the documentation build step, just wanted to make sure I explored that avenue first.
If you will decide to move video and other large media into separate repository (inside MiSTer-devel of course) as a submodule, then i suggest to re-setup this repository with current code as an initial commit, so it will start with new clean history and cloning won't include past already removed/moved media.
Yup, I definitely agree with doing that :)
Don't do it quite like that unless you really don't care about the history. Instead, you can rewrite the history to remove certain files/directories (i.e. all of the assets), so you can keep the history of your changes (the commit IDs won't be the same, but that's id).
True. Would you use something like git filter-branch (or git filter-repo) to all .mp4 files to do this in your experience?
It looks like filter-repo is the recommendation now, but yes, I believe I've used filter-branch in the past to accomplish this. Luckily this is a small repo (by number of files), so it's easy to verify things haven't gone horribly wrong. Just make sure you have a local copy of the repo, and ideally validate all of the changes locally before you force push.
Try a git clone --depth 1 https://github.com/MiSTer-devel/MkDocs_MiSTer.git for me now and tell me what the filesize is for you ;)
Much more reasonable, thanks :P
Sweet!