seqsense / s3sync

♻️ Golang utility for syncing between s3 and local, similar to `aws s3 sync`

s3sync

Golang utility for syncing between s3 and local

Usage

Use New to create a manager, and Sync function syncs between s3 and local filesystem.

import (
  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/seqsense/s3sync"
)

func main() {
  // Creates an AWS session
  sess, _ := session.NewSession(&aws.Config{
    Region: aws.String("us-east-1"),
  })

  syncManager := s3sync.New(sess)

  // Sync from s3 to local
  syncManager.Sync("s3://yourbucket/path/to/dir", "local/path/to/dir")

  // Sync from local to s3
  syncManager.Sync("local/path/to/dir", "s3://yourbucket/path/to/dir")

  // Sync from s3 to s3
  syncManager.Sync("s3://yourbucket/path/to/dir", "s3://anotherbucket/path/to/dir")
}

Sets the custom logger

You can set your custom logger.

import "github.com/seqsense/s3sync"

...
s3sync.SetLogger(&CustomLogger{})
...

The logger needs to implement Log and Logf methods. See the godoc for details.

Sets up the parallelism

You can configure the number of parallel jobs for sync. Default is 16.

s3sync.new(sess, s3sync.WithParallel(16)) // This is the same as default.
s3sync.new(sess, s3sync.WithParallel(1)) // You can sync one by one.

License

Apache 2.0 License. See LICENSE.