Filesystem store blocks extension differ from other stores
peterlindgren opened this issue · comments
Hi,
I noticed that upsync'ing to a filesystem store writes blocks with an .lrb extension. I noticed this while experimeting with adding a http(s)-Store for downsync'ing, and upsync via filesystem path.
I was able to confirm that the extensions are interchangeable, at least in my limited test, by switching the .lsb extension to .lrb in getBlockPath (in remotestore.go). It also appears that the longtailstore/fsstore.go code is not used much since Read, Write, etc is never called?
Is this expected behavior? Legacy code or something I'm mistaken about?
I'm using v0.3.0.
Thanks!
-Peter
Yes, this is expected. Sadly due to legacy reasons I don’t want to change the extension even though they are the same format.
In theory an implementor can store a block in any for it wants and how it wants, in practice they all use the same format so far.
The filesystem block store implementation takes an argument where you can change the extension but it is not exposed out to the command line tool.
Might consider changing that is the future but don’t want to break anybodys large store or caches so haven’t done that so far.
As for the fsstore.go not being used, it was added for testing purposes so the remotestore.go logic could be tested without having to access S3 or GCS.
You can force usage of it by prefixing your filesystem path with fsblob:// but it will not be as efficient as the built in filesystem store.
The upsync speed is not super important in my particular use-case so fsblob:// might work out. Thank you!
But why do you care, not following really?
If you upload to a regular file system (or network share) you would use the default file system store by specifying a regular path which works just fine for downloading if you specify a regular file pat. Or you use S3/GCS as a store and would use that path.
I can only see this making any sense if you upload to a file share and the copy the data to a S3/GCS bucket and download from there?
I want to upsync via file system (network shared) from build agents and then users/clients download via a http(s)- or sftp-protocol store. We host the builds on-site so I don't want to use S3/GCS/AZ.
I will change so block stores that target the file system (not caches) to use the ".lsb" extension.
This will likley not affect any number of users, the big issue was caches and those stay the same.
Should be fixed in 0.3.3 @peterlindgren
Thank you Dan!