Tool to sync contents between local file system and remote object storage.
This tool can achieve synchronizations as:
- directory A → container B
- container B → directory A
- container B → container C
Here, directory is located in local file system and made up of files and sub directories, while container (also called bucket according to AWS S3) is vessel in remote CEPH storage where objects saved.
In CLI:
# Command "ceph-sync", "cehsync" and "ossync" will be generated.
npm install -g ceph-sync
# Show help info.
ceph-sync -h
# Run sync task.
ceph-sync --source /path/of/container --target /path/of/conn.json
As API:
const fs2ceph = require('ceph-sync/fs2ceph');
const progress = fs2ceph(
/* source */ '/path/of/container',
/* target */ connConfig );
progress.on('error', (err) => {
// ...
});
progress.on('end', (meta) => {
// Sychronization successfully finished.
});
The connection configuration is a JSON object required by the dependent package ceph. To describe an accessible (readable and writable) CEPH container, following properties are required:
- endPoint
- subuser
- key
- container
Here is a dummy example:
{
"endPoint" : "http://storage.example.com/",
"subuser" : "userName:subUserName",
"key" : "380289ba59473a368c593c1f1de6efb0380289ba5",
"container" : "containerName"
}
For CLI usage, CEPH connection config should be stored in a JSON file.
When installed globally, ceph-sync will create a homonymous global command. Run ceph-sync -h
in terminal to print the man page.
ceph-sync will occupy a hidden directory named .ceph-sync
in home directory of current user.
ceph-sync offers three functions to achieve different tasks:
- jinang/Progress ceph2ceph(object sourceConn, object targetConn, object options)
- jinang/Progress ceph2fs(object sourceConn, string targetDir, object options)
- jinang/Progress fs2ceph(string sourceDir, object targetConn, object options)
- Here "2" is a homophone of "to".
sourceConn
andtargetConn
may be an object containing CEPH storage connection configuration, or an instance of swift Connection.- The functions accept similar
options
argument, see section Parameteroptions
for details. - The functions are all asynchronous and will return an instance of jinang/Progress. Via the returned value, we may learn about and control the sync progress. See section Get Into Sync Progress for details.
Each function may be required solely:
const cephSync = require('ceph-sync');
const ceph2ceph = require('ceph-sync/ceph2ceph');
const ceph2fs = require('ceph-sync/ceph2fs');
const fs2ceph = require('ceph-sync/fs2ceph');
// E.g., next two functions are equivalent.
cephSync.ceph2ceph
ceph2ceph
-
string[] options.names
Object names to be synchronised. -
Function options.mapper
Object name mapper. -
Function options.filter
Object name filter. -
Function options.dualMetaFilter
Filter with paramenter(stat, meta)
.
Only acceptable infs2ceph()
. -
boolean options.ifNoneMatch
Check etag firstly. If target object / file already exists and has same etag with source object / file, keep it instead of doing replacement.
This option is only effective inceph2ceph()
. -
number options.maxCreated
Maximum creation allowed (then the progress will be terminated). -
number options.maxCreating
Maximum cocurrent creating operation allowed. -
number options.maxErrors
Maximum exceptions allowed (then the progress will be terminated). -
number options.retry
Maximum retry times on exception for each object or object list.
Via the returned instance of jinang/Progress
, we may learn about what happened and then control the sync progress.
-
progress.on(string eventName, Function listener)
See section Events During Sync Progress for aviable events and their accompanied arguments. -
progress.abort()
Terminate the progress as soon as possible. -
progress.quit()
Quit the progress gracefully.
- Object meta
- string mark
- Object meta
- Object meta
- Error error
- Error error
- Object meta
{ errors /* number */, created /* number */, ignored /* number */, }