freezer-truck allows users to synchronize files on a cron schedule from a remote server over SSH. It was primarily
created to download hot copies of files to a staging area before they are tagged and sorted for cold storage. It
downloads files over SSH with SFTP and provides the ability to concurrently do so.
Most existing solutions use the contents of local and remote folders to determine if synchronization is necessary. Since
freezer-truck was designed with the expectation that another process would move files out of the directory it
downloads to, it instead uses a local DB (badger) to keep track of which files
have been loaded. This prevents the synchronization process from re-downloading files that have already been loaded, but simply
moved.
Configuration is done through the following environment variables.
TRUCK_SCHEDULECRONRequired Crontab-format schedule thatfreezer-truckshould follow to download files.TRUCK_HOSTRequired SSH host to load files fromTRUCK_USERNAMERequired Username to use when connecting to host over SSH.TRUCK_PRIVATEKEYOptional if TRUCK_PASSWORD is specified SSH privatekey to use.TRUCK_PASSWORDOptional if TRUCK_PRIVATEKEY is specified Either the user's login password for SSH access, OR the password to the specified private key file ifTRUCK_PRIVATEKEYis also setTRUCK_INSECUREKNOWNHOSTSOptional, default false Ignore known hosts file. NOT RECOMMENDED FOR PRODUCTIONTRUCK_BADGERPATHRequired Path where badger database should be created.TRUCK_CONCURRENCYLIMITRequired How many files should be concurrently downloadedTRUCK_REMOTEFILEROOTRequired Remote folder path where files should be loaded fromTRUCK_LOCALFILEROOTRequired Local folder path where complete downloads should be placed. In-progress downloads will be located under the path specified byTRUCK_TEMPFILEROOTTRUCK_TEMPFILEROOTRequired Local folder path where in-progress downloads should be placed. After completion, files will be moved to the path specified byTRUCK_LOCALFILEROOT