apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store

Home Page:https://apple.github.io/foundationdb/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support opaque marker in S3BlobStore

larshagencognite opened this issue · comments

When paging through objects from S3, the S3BlobStore actor uses the last object name or prefix from the previous response as a marker the next page of results

lastFile = listResult.objects.back().name;

This is fine for S3 and many compatible backends, but makes it harder to use some backends like Azure blob store. This is because Azure blob store returns opaque NextMarkers, which are intended to be used as Marker for the next page.
By preferring the NextMarker field when present, we would be able to handle more backends.
S3 docs
Azure docs
I realize that there is a separate feature being developed for running backups directly against Azure blob store. This would be a much smaller change, and would allow us to stop relying on workarounds in compatibility layers like S3Proxy and Minio