Support opaque marker in S3BlobStore
larshagencognite opened this issue · comments
Lars Hagen commented
When paging through objects from S3, the S3BlobStore
actor uses the last object name or prefix from the previous response as a marker the next page of results
foundationdb/fdbclient/S3BlobStore.actor.cpp
Line 1221 in 21da0f2
This is fine for S3 and many compatible backends, but makes it harder to use some backends like Azure blob store. This is because Azure blob store returns opaque
NextMarker
s, which are intended to be used as Marker
for the next page.By preferring the
NextMarker
field when present, we would be able to handle more backends.S3 docs
Azure docs
I realize that there is a separate feature being developed for running backups directly against Azure blob store. This would be a much smaller change, and would allow us to stop relying on workarounds in compatibility layers like S3Proxy and Minio