English | 简体中文
A cross-platform file synchronization tool out of the box based on golang.
The first need Go installed (version 1.18+ is required), then you can use the below
command to install gofs.
go install github.com/no-src/gofs/...@latestIf you want to run in a docker, you should install or build with the -tags netgo flag or set the
environment CGO_ENABLED=0, otherwise you may get an error that the gofs not found, when the docker container is
running.
go install -tags netgo github.com/no-src/gofs/...@latestYou can install a program run in the background using the following command on Windows.
go install -ldflags="-H windowsgui" github.com/no-src/gofs/...@latestPlease ensure the source directory and dest directory exists first, replace the following path with your real path.
$ mkdir source destGenerate the TLS cert file and key file for testing purposes.
The TLS cert and key files are just used by File Server and Remote Disk Server.
$ go run $GOROOT/src/crypto/tls/generate_cert.go --host 127.0.0.1
2021/12/30 17:21:54 wrote cert.pem
2021/12/30 17:21:54 wrote key.pemLook up our workspace.
$ ls
cert.pem key.pem source destMonitor source directory and sync change files to dest directory.
You can use the logically_delete flag to enable the logically delete and avoid deleting files by mistake.
Set the checkpoint_count flag to use the checkpoint in the file to reduce transfer unmodified file chunks, by
default checkpoint_count=10, which means it has 10+2 checkpoints at most. There are two additional checkpoints at
the head and tail. The first checkpoint is equal to the chunk_size, it is optional. The last checkpoint is equal to
the file size, it is required. The checkpoint offset set by the checkpoint_count is always more than chunk_size,
unless the file size is less than or equal to chunk_size, then the checkpoint_count will be zero, so it is optional.
$ gofs -source=./source -dest=./destSync the whole path immediately from source directory to dest directory.
$ gofs -source=./source -dest=./dest -sync_onceSync the whole path from source directory to dest directory with cron.
# Per 30 seconds sync the whole path from source directory to dest directory
$ gofs -source=./source -dest=./dest -sync_cron="*/30 * * * * *"Start a daemon to create subprocess to work, and record pid info to pid file.
$ gofs -source=./source -dest=./dest -daemon -daemon_pidStart a file server for source directory and dest directory.
The file server is use HTTPS default, set the tls_cert_file and tls_key_file flags to customize the cert file and
key file.
You can disable the HTTPS by set the tls flag to false if you don't need it.
If you set the tls to true, the file server default port is 443, otherwise it is 80, and you can customize the
default port with the server_addr flag, like -server_addr=":443".
You should set the rand_user_count flag to auto generate some random users or set the users flag to customize server
users for security reasons.
The server users will output to log if you set the rand_user_count flag greater than zero.
If you need to compress the files, add the server_compress flag to enable gzip compression for response, but it is not
fast now, and may reduce transmission efficiency in the LAN.
# Start a file server and create three random users
# Replace the `tls_cert_file` and `tls_key_file` flags with your real cert files in the production environment
$ gofs -source=./source -dest=./dest -server -tls_cert_file=cert.pem -tls_key_file=key.pem -rand_user_count=3Start a remote disk server as a remote file source.
The source flag detail see Remote Server Source Protocol.
Pay attention to that remote disk server users must have read permission at least, for
example, -users="gofs|password|r".
You can use the checkpoint_count flag like the Local Disk.
# Start a remote disk server
# Replace the `tls_cert_file` and `tls_key_file` flags with your real cert files in the production environment
# Replace the `users` flag with complex username and password for security
$ gofs -source="rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1" -dest=./dest -users="gofs|password|r" -tls_cert_file=cert.pem -tls_key_file=key.pemStart a remote disk client to sync change files from remote disk server.
Use the sync_once flag to sync the whole path immediately from remote disk server to local dest directory,
like Sync Once.
Use the sync_cron flag to sync the whole path from remote disk server to local dest directory with cron,
like Sync Cron.
The source flag detail see Remote Server Source Protocol.
# Start a remote disk client
# Replace the `users` flag with your real username and password
$ gofs -source="rs://127.0.0.1:8105" -dest=./dest -users="gofs|password"Start a Remote Disk Server as a remote file source, then enable the remote push server with
the push_server flag.
Pay attention to that remote push server users must have read and write permission at least, for
example, -users="gofs|password|rw".
# Start a remote disk server and enable the remote push server
# Replace the `tls_cert_file` and `tls_key_file` flags with your real cert files in the production environment
# Replace the `users` flag with complex username and password for security
$ gofs -source="rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1" -dest=./dest -users="gofs|password|rw" -tls_cert_file=cert.pem -tls_key_file=key.pem -push_serverStart a remote push client to sync change files to the Remote Push Server.
Use the chunk_size flag to set the chunk size of the big file to upload. The default value of chunk_size
is 1048576, which means 1MB.
You can use the checkpoint_count flag like the Local Disk.
More flag usage see Remote Disk Client.
# Start a remote push client and enable local disk sync, sync the file changes from source path to the local dest path and the remote push server
# Replace the `users` flag with your real username and password
$ gofs -source="./source" -dest="rs://127.0.0.1:8105?local_sync_disabled=false&path=./dest" -users="gofs|password"The remote server source protocol is based on URI, see RFC 3986.
The scheme name is rs.
The remote server source uses 0.0.0.0 or other local ip address as host in Remote Disk Server
mode, and use ip address or domain name as host in Remote Disk Client mode.
The remote server source port, default is 8105.
Use the following parameters in Remote Disk Server mode only.
paththe Remote Disk Server actual local source directorymoderunning mode, in Remote Disk Server mode isserver, default is running in Remote Disk Client modefs_serverFile Server address, likehttps://127.0.0.1local_sync_disableddisabled Remote Disk Server sync changes to its local dest path,trueorfalse, default isfalse
For example, in Remote Disk Server mode.
rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1
\_/ \_______/ \__/ \____________________________________________________________________________/
| | | |
scheme host port parameter
Enable manage api base on File Server by using the manage flag.
By default, allow to access manage api by private address and loopback address only.
You can disable it by setting the manage_private flag to false.
$ gofs -source=./source -dest=./dest -server -tls_cert_file=cert.pem -tls_key_file=key.pem -rand_user_count=3 -manageThe pprof url address like this
https://127.0.0.1/manage/pprof/
Reading the program config, default return the config with json format, and support json and yaml format
currently.
https://127.0.0.1/manage/config
Or use the format parameter to specific the config format.
https://127.0.0.1/manage/config?format=yaml
Use the report flag to enable report api route, and start to collect the report data, need to enable the manage flag
first.
The details of the report api see Report API.
https://127.0.0.1/manage/report
Enable the file logger and console logger by default, and you can disable the file logger by setting the log_file flag
to false.
Use the log_level flag to set the log level, default is INFO, (DEBUG=0 INFO=1 WARN=2 ERROR=3).
Use the log_dir flag to set the directory of the log file, default is ./logs/.
Use the log_flush flag to enable auto flush log with interval, default is true.
Use the log_flush_interval flag to set the log flush interval duration, default is 3s.
Use the log_event flag to enable the event log, write to file, default is false.
# set the logger config in "Local Disk" mode
$ gofs -source=./source -dest=./dest -log_file -log_level=0 -log_dir="./logs/" -log_flush -log_flush_interval=3s -log_eventIf you want, you can use a configuration file to replace all the flags.It supports json and yaml format currently.
All the configuration fields are the same as the flags, you can refer to the Configuration Example or the response of Config API.
$ gofs -conf=./gofs.yamlYou can use the checksum flag to calculate the file checksum and print the result.
The chunk_size and checkpoint_count flags are effective here the same as in the Local Disk.
$ gofs -source=./gofs -checksum$ gofs -h$ gofs -v$ gofs -about