AWS CLI s3 sync
for Node.js provides a modern client to perform S3 sync operations between file systems and S3 buckets in the spirit of the official AWS CLI command.
AWS CLI installation is NOT required by this module.
- Sync a local file system with a remote Amazon S3 bucket
- Sync a remote Amazon S3 bucket with a local file system
- Sync two remote Amazon S3 buckets
- Sync only new and updated objects
- Support AWS CLI options
--delete
and--dryrun
- Track object sync progress
- Sync any number of objects (no 1000 objects limit)
- Transfer objects concurrently
- Manage differences in folder structures easily through relocation
- There is no way to achieve S3 sync using the AWS SDK for JavaScript v3 alone.
- AWS CLI installation is NOT required.
- The package contains no external dependency besides AWS SDK.
- The AWS SDK dependency is up-to-date (AWS SDK for JavaScript v3).
- The module overcomes a set of common limitations listed at the bottom of this README.
npm install s3-sync-client
S3SyncClient
extends the AWS SDK S3Client
class and should be instantiated the same way.
const S3SyncClient = require('s3-sync-client');
const sync = new S3SyncClient({
region: 'eu-west-3',
credentials: {
accessKeyId: process.env.ACCESS_KEY_ID,
secretAccessKey: process.env.SECRET_ACCESS_KEY,
},
});
const S3SyncClient = require('s3-sync-client');
const sync = new S3SyncClient({ /* credentials */ });
// aws s3 sync /path/to/local/dir s3://mybucket2
await sync.bucketWithLocal('/path/to/local/dir', 'mybucket2');
// aws s3 sync /path/to/local/dir s3://mybucket2/zzz --delete
await sync.bucketWithLocal('/path/to/local/dir', 'mybucket2/zzz', { del: true });
const S3SyncClient = require('s3-sync-client');
const sync = new S3SyncClient({ /* credentials */ });
// aws s3 sync s3://mybucket /path/to/some/local --delete
await sync.localWithBucket('mybucket', '/path/to/some/local', { del: true });
// aws s3 sync s3://mybucket2 /path/to/local/dir --dryrun
const syncOps = await sync.localWithBucket('mybucket2', '/path/to/local/dir', { dryRun: true });
console.log(syncOps); // log download and delete operations to perform
const S3SyncClient = require('s3-sync-client');
const sync = new S3SyncClient({ /* credentials */ });
// aws s3 sync s3://my-source-bucket s3://my-target-bucket --delete
await sync.bucketWithBucket('my-source-bucket', 'my-target-bucket', { del: true });
const EventEmitter = require('events');
const S3SyncClient = require('s3-sync-client');
const sync = new S3SyncClient({ /* credentials */ });
const monitor = new EventEmitter();
monitor.on('progress', (progress) => console.log(progress));
setTimeout(() => monitor.emit('abort'), 30000); // optional abort
await sync.localWithBucket('mybucket', '/path/to/local/dir', { monitor });
/* output:
...
{
size: { current: 11925, total: 35688 },
count: { current: 3974, total: 10000 }
}
...
and abort unfinished sync after 30s (promise rejected with an AbortError)
*/
const S3SyncClient = require('s3-sync-client');
const sync = new S3SyncClient({ /* credentials */ });
// sync s3://my-source-bucket/a/b/c.txt to s3://my-target-bucket/zzz/c.txt
await sync.bucketWithBucket('my-source-bucket/a/b/c.txt', 'my-target-bucket', {
relocations: [ // multiple relocations can be applied
['a/b', 'zzz'],
],
});
// sync s3://mybucket/flowers/red/rose.png to /path/to/local/dir/rose.png
await sync.localWithBucket('mybucket/flowers/red/rose.png', '/path/to/local/dir', {
relocations: [
['flowers/red', ''], // folder flowers/red will be flattened during sync
],
});
Additional code examples are available in the test folder.
configuration
<Object> Configuration as in the AWS SDK S3Client. See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html.
localDir
<string> Local directorybucketPrefix
<string> Remote bucket name which may contain a prefix appended with a/
separatoroptions
<Object>del
<boolean> Equivalent to CLI--delete
optiondryRun
<boolean> Equivalent to CLI--dryrun
optionmonitor
<EventEmitter>- Attach
progress
event to receive upload progress notifications - Emit
abort
event to stop object uploads immediately
- Attach
maxConcurrentTransfers
<number> Each upload generates a Promise which is resolved when a local object is written to the S3 bucket. This parameter sets the maximum number of upload promises that might be running concurrently.relocations
<Array> Allows uploading objects to remote folders without mirroring the source directory structure. Each relocation should be specified as an <Array> of[sourcePrefix, targetPrefix]
.
- Returns: <Promise> Fulfills with an <Object> of sync operations upon success.
Sync a remote S3 bucket with the local file system.
Similar to AWS CLI aws s3 sync localDir s3://bucketPrefix [options]
.
bucketPrefix
<string> Remote bucket name which may contain a prefix appended with a/
separatorlocalDir
<string> Local directoryoptions
<Object>del
<boolean> Equivalent to CLI--delete
optiondryRun
<boolean> Equivalent to CLI--dryrun
optionmonitor
<EventEmitter>- Attach
progress
event to receive download progress notifications - Emit
abort
event to stop object downloads immediately
- Attach
maxConcurrentTransfers
<number> Each download generates a Promise which is resolved when a remote object is written to the local file system. This parameter sets the maximum number of download promises that might be running concurrently.relocations
<Array> Allows downloading objects to local directories without mirroring the source folder structure. Each relocation should be specified as an <Array> of[sourcePrefix, targetPrefix]
.
- Returns: <Promise> Fulfills with an <Object> of sync operations upon success.
Sync the local file system with a remote S3 bucket.
Similar to AWS CLI aws s3 sync s3://bucketPrefix localDir [options]
.
sourceBucketPrefix
<string> Remote reference bucket name which may contain a prefix appended with a/
separatortargetBucketPrefix
<string> Remote bucket name to sync which may contain a prefix appended with a/
separatoroptions
<Object>del
<boolean> Equivalent to CLI--delete
optiondryRun
<boolean> Equivalent to CLI--dryrun
optionmonitor
<EventEmitter>- Attach
progress
event to receive copy progress notifications - Emit
abort
event to stop object copy operations immediately
- Attach
maxConcurrentTransfers
<number> Each copy generates a Promise which is resolved after the object has been copied. This parameter sets the maximum number of copy promises that might be running concurrently.relocations
<Array> Allows copying objects to remote folders without mirroring the source folder structure. Each relocation should be specified as an <Array> of[sourcePrefix, targetPrefix]
.
- Returns: <Promise> Fulfills with an <Object> of sync operations upon success.
Sync two remote S3 buckets.
Similar to AWS CLI aws s3 sync s3://sourceBucketPrefix s3://targetBucketPrefix [options]
.
See CHANGELOG.md.
AWS CLI s3 sync
for Node.js has been developed to solve the S3 syncing limitations of the existing GitHub repo and NPM modules.
Most of the existing repo and NPM modules encounter one or more of the following limitations:
- requires AWS CLI to be installed
- uses Etag to perform file comparison (Etag should be considered an opaque field, and should not be used)
- limits S3 bucket object listing to 1000 objects
- supports syncing bucket with local, but doesn't support syncing local with bucket
- uses outdated dependencies
- is unmaintained
The following JavaScript modules suffer at least one of the limitations:
- https://github.com/guerrerocarlos/aws-cli-s3-sync
- https://github.com/thousandxyz/s3-lambo
- https://github.com/Quobject/aws-cli-js
- https://github.com/auth0/node-s3-client
- https://github.com/andrewrk/node-s3-client
- https://github.com/hughsk/s3-sync
- https://github.com/issacg/s3sync
AWS CLI s3 sync
for Node.js has some limitations too:
- does not support multipart transfers