'Composition' of versions?
mastercoms opened this issue · comments
Hi, this is a great tool! We're moving over from our current archive setup to this tool, and we have a flow which doesn't seem to be supported thus far. We split out our engine installs into 4 sections:
- editor only
- editor+uegame targets
- editor symbols
- editor+uegame symbols
This is so users can save bandwidth by choosing if they want to download symbols or not, or choose which feature set they want in their engine. Obviously, longtail seems to manage a store folder strictly, so it will delete content in the target not found from the source. Would there be a possibility of basically being able to compose a combination of different "versions", perhaps through layers so that one version could have priority in case of conflicts? This would allow us to store editor, editor+uegame, and symbols as versions, and then get a benefit from the chunking as well instead of requiring a redownload when switching between editor only and editor+uegame. So for example, we would upload these versions:
- editor-v1
- full-engine-v1
- editor-symbols-v1
- full-engine-symbols-v1
And then, a user could select to install from both sources:
get --source-paths gs://bucket/store/editor-v1.json gs://bucket/store/editor-symbols-v1.json --target-path v1
(--target-path required when specifying --source-paths?) (editor-symbols-v1 would override editor-v1 if there the same file exists in both (even though in this example there wouldn't be))
We could do a workaround on this for now by copying out the engine to different stores and then manually combining them but then we wouldn't get the full shared chunking benefit that we would get if longtail was aware of this. I do understand that this might be a hard problem though so I would understand if this doesn't fit within the design.
Also, this approach would allow us to be much more flexible on our install options in our team, similar to EGS:
No, there is no real composition function as you say.
To clarify, you upload the different sections separately to one and the same store? And then you want to make a combination of this sections depending on user choice when downloading?
Perhaps you could try the --no-scan-target option?
Won't be fully what you need but perhaps the core components you download with scan-target to get initial state and for the other components you do --no-scan-target which will only add/overwrite changed files.
It won't be as optimal since the first step would remove any components not in the initial pack...
I guess it would be possible to add a feature where you specify more than one version file - it should not be to complicated, but I want to understand how your setup looks in detail before doing any such work.
To clarify, we haven't used longtail for this yet. Right now we're just using longtail to ship the editor without symbols to artists and also forgoing build machines from using longtail/archives for full engines with game targets (they now build the engine from source repo themselves).
How the setup worked in the past is we would have 4 7z archives per engine version (4.27-1, for example): editor-4.27-1.7z, editor-symbols-4.27-1.7z, engine-4.27-1.7z, and engine-symbols-4.27-1.7z. Then the user would select if they would want an editor or engine "bundle" and if they wanted to download symbols. Then, our download script would download and extract the appropriate archives into the 4.27-1/ folder. For example, if a user selected editor and no symbols, it would only download editor-4.27-1.7z. If a user selected engine and symbols, it would download engine-4.27-1.7z and engine-symbols-4.27-1.7z.
We moved to longtail because fast version deltas are very important to us, so that we can release the engine on a faster cadence than before to artists, so we are ok with dropping symbols support for now.
--no-scan-target sounds like it could hackily work, but I think we'd hold off on that kind of solution.
To clarify, you upload the different sections separately to one and the same store? And then you want to make a combination of this sections depending on user choice when downloading?
To answer this question theoretically, yes I want to upload different sections to the same store. And then allow the user to select some of these sections.
To clarify, you upload the different sections separately to one and the same store? And then you want to make a combination of this sections depending on user choice when downloading?
To answer this question theoretically, yes I want to upload different sections to the same store. And then allow the user to select some of these sections.
Alright. Do you expect to do the partial upload using the regex filtering support or do you envision uploading separate folders?
I ask since uploading separate folders might cause some logistical issues with where the root of a version is for the different sections.
For symbols support, we exclude *.pdb from Engine, and upload another Engine with only *.pdb. However, for the editor vs. editor with game targets, we just upload the Engine folder from both, since it's not exactly clear all the differences, I would expect longtail would be able to see the difference between the two on its own without us having to enumerate it.
Working on support here #235 hopefully it will cover your needs once complete.