A system for efficiently streaming bundles of files over a connection.
Features:
- Compression - supports gzip and lz4
- Chunked - can stream files without knowing the size in advance (e.g. generated files/downloads)
This package was developed based on poor experiences with docker's usage of tar as a part of their API. The tar file format requires lengths to be written before the contents of the file. This created problems, as I had to buffer huge files in memory as they were collected over HTTP or generated on the fly.
I created this for the purpose of streaming files to test machines in a CI system. This is also potentially useful for data import/export.
A command line interface has been created as a debugging tool and a demo. To install it, run:
go get github.com/jaddr2line/filestream/cmd/filestream/...
To encode files to stdout:
# Encode directories "example" and "test" to stdout
filestream example test | somecommand
# Encode to stdout with gzip compression
filestream -z gzip -l 9 example | somecommand
Decoding from stdin:
# Decode stream to current directory.
curl https://example.com/something | filestream -d
# Decode stream relative to directory example
curl https://example.com/something | filestream -d -C example
Listing files in stream:
curl https://example.com/something | filestream -t
The tool can also read/write the stream to different places:
# Store directory example to a file called "output.dat"
filestream example -s output.dat
# Decode from a file
filestream -d -s output.dat
# Decode data from an HTTP GET request
filestream -d -s https://example.com/something