dstore is a simple abstraction on top of Local storage and Cloud
storage. It handles commonly used functions to store things (locally,
or on cloud storage providers), list files, delete, etc..
It is used by StreamingFast.
It currently supports:
- AWS S3 (
s3://[bucket]/path?region=us-east-1, with AWS-specific env vars)- Minio (through the S3 interface)
- Google Storage (
gs://[bucket]/path, withGOOGLE_APPLICATION_CREDENTIALSenv var set) - Azure Blob Storage (
az://[account].[container]/path, withAZURE_STORAGE_KEYenv var set) - Local file systems (including virtual of fused-based) (
file:///prefix)
The storetests package contains all our integration tests we perform on our store implementation.
Some of the store implementations can be tested directly while few others, from Cloud Providers
essentially, requires some extra environment variables to run. They are skip if the correct
environment variables for the provider is not set.
To run the full test suite, you will need to perform the following steps.
First, you will need to have locally a few dependencies:
Then, start minio server:
mkdir -p /tmp/minio-tests/store-tests
cd /tmp/minio-tests
minio server .Ensure you have access to GCP Storage Bucket, S3 Bucket, then run the full test suite:
STORETESTS_GS_STORE_URL="gs://streamingfast-developement-random/store-tests"\
STORETESTS_S3_STORE_URL="s3://streamingfast-customer-outbox/store-tests?region=us-east-2"\
STORETESTS_S3_MINIO_STORE_URL="s3://localhost:9000/store-tests?region=none&insecure=true&access_key_id=minioadmin&secret_access_key=minioadmin"\
go test ./...Note S3 bucket is not existing anymore, you need to provide your own.
Issues and PR in this repo related strictly to the dstore library.
Report any protocol-specific issues in their respective repositories
Please first refer to the general StreamingFast contribution guide, if you wish to contribute to this code base.