fd0 / split

Split large files into smaller ones using deterministic Content Defined Chunking

Split large files into smaller ones using the same Content Defined Chunking algorithm the restic backup program uses.

Build (using Go >= 1.11):

$ go build

Sample usage:

$ ./split --verbose --output /tmp --input /tmp/data
next chunk offset 0, 814244 bytes written to /tmp/split-000
next chunk offset 814244, 1649886 bytes written to /tmp/split-001
next chunk offset 2464130, 3332485 bytes written to /tmp/split-002
next chunk offset 5796615, 1996103 bytes written to /tmp/split-003
[...]
next chunk offset 101940538, 700441 bytes written to /tmp/split-069
next chunk offset 102640979, 533829 bytes written to /tmp/split-070
next chunk offset 103174808, 537761 bytes written to /tmp/split-071
next chunk offset 103712569, 1145031 bytes written to /tmp/split-072
wrote 104857600 bytes to 73 files

Using cat, we can put the file back together (and use sh256sum to verify it's the same data):

$ cat /tmp/split-* | sha256sum
66b9d2c5de34170b93f387988a80fb600717da5e437dcc4da1025343fb9019a1  -

$ sha256sum /tmp/data
66b9d2c5de34170b93f387988a80fb600717da5e437dcc4da1025343fb9019a1  /tmp/data

Check out the help for other options:

$ ./split -h
Usage of split:
  -i, --input file     Read from file instead of stdin
  -u, --max-size n     Set maximal chunk size to n bytes (default 8388608)
  -l, --min-size n     Set minimal chunk size to n bytes (default 524288)
  -o, --output dir     Write files to directory dir instead of the current directory (default ".")
  -p, --polynomial p   Use polynomial p for splitting (hex notation, no prefix) (default "3DA3358B4DC173")
  -t, --template s     Use s as the (printf-style) template for output files (default "split-%03d")
  -v, --verbose        Be verbose

The library used for this program is https://github.com/restic/chunker

If you're interested in the mathematical foundation for Content Defined Chunking with Rabin Fingerprints, head over to the restic blog which has an introductory article.