download-sam
According to SAM: "The System for Award Management (SAM) is an official website of the U.S. government. There is no cost to use SAM."
This utility polls the https://www.sam.gov website and looks for a downloadable file named according to the historical filename pattern for SAM downloads: SAM_PUBLIC_MONTHLY_YYYYMMnn.ZIP where YYYY is the year, MM is the month, and nn is the day between 1 and 10 that the file is generated by the website.
The SAM monthly file is historically posted in the first week of the month. But sometimes it has been late. This utility - typically scheduled to run on the 10th of the month - incrementally scans the website for day 1, and progressing to day 10, for a file matching the naming convention. Once a match is found, it is downloaded. and copied to the specified ingest folder, as well as to an S3 bucket as a backup. Before beginning any processing, the utility examines the specified S3 bucket to see if the file has already been downloaded and - if it has - then it exits without doing anything (unless the -force argument is supplied on the command line.)
The functionality of the utility is controlled by the command line options:
-yyyymm n
Mandatory. Specifies the year and month of the SAM file to download. Specify the literal default to download a file for the year and month corresponding to the current system time. Otherwise specify the exact year and month to download. E.g. "202001" for January 2020.
-accesskey keyval
Mandatory unless -downloadonly is specified. Your AWS Access Key for S3 bucket access.
-secretkey keyval
Mandatory unless -downloadonly is specified. Your AWS Secret Access Key for S3 bucket access.
-bucket s3bucket
Mandatory unless -downloadonly is specified. The name of the S3 bucket that the SAM download data should be uploaded into. Do not prefix with any S3 URL prefixes such as S3: or S3://. Provide only the bare bucket name.
-prefix prefix
Optional. The name of the S3 bucket prefix that will be prefixed to the file name when the file is uploaded to S3. For example, if your bucket is named mybucket and the file to download is SAM_PUBLIC_MONTHLY_20200301.ZIP, and you specify sam as the bucket prefix then the file will be copied to the following S3 url: s3://mybucket/sam/SAM_PUBLIC_MONTHLY_20200301.ZIP. If no prefix is specified then the file will be uploaded into the "root" of the bucket. Note: S3 keys and prefixes are case sensitive.
-ingest fqpn
Optional. The fully-qualified path name of the ingest folder on the local filesystem that the downloaded file should be placed in. The utility will download the file from the sam.gov website into the current working directory, and then move it to this location for ingest, and also copy it to S3 as an archive version. The specified ingest directory must exist, and it must be a directory (not a file) otherwise the utility will abort.
-encrypted
Optional. If provided, then the two key values are taken to be encrypted values. The utility will decrypt them when accessing the AWS bucket. If not provided, then the two key values are interpreted as plain text values.
-encrypt
Optional. Encrypt the provided keys. Display the encrypted values to the console, and immediately exit with no further processing. Example:
download-sam -accesskey foo -secretkey bar -encrypt -bucket ignored -yyyymm 202001 -ingest .
Access Key:
0x01000000D08C9DDF0115D1118C7A00C04FC297EB0100000084B3BA3DD1E11440896B8643B8CD670E04000000020000000000106600000001000020000000A1D7780D9B8D79F7843DFD4FEFDCA34FA8EB91CDC32DDAF7D9D5397C9F5EAEED000000000E80000000020000200000009E3B64A3426D6EED8D65211134E057C39F7E009B300C41648ECA60E8DAD7B71E10000000573B78FF7EB505EE3C9ADBB9F1797B8940000000659A0980D83948B02E0EC2E0A7F32F03D600E2D69C9C4C4344F652B164619EF5FF8AD354D38A282E27A2603F4F79222A24F3F446973F4123E53289A962323A64
Secret Access Key:
0x01000000D08C9DDF0115D1118C7A00C04FC297EB0100000084B3BA3DD1E11440896B8643B8CD670E04000000020000000000106600000001000020000000207BA17696DA4E92170A5E994F5D86E1D16F270CE10F76567C00EC3395BF2518000000000E8000000002000020000000C056487DC2EF4DC9D82338FF5A0DC2B3B5CB645B00444D8D566088CF8D499CB410000000D2FA5C1ED2F58AA87F633564FED543CA40000000EBF0B946DB79DE15A066FA9B5AB7E0BA36B069E39B06E8E2BBEA13329749A4311EAE1F6EA62C3253BB0A129DE715486EABB5F0C041D381FB89124116A1D80CBC
Then subsequently:
download-sam -accesskey 0x01000000D08C9D... etc.
The objective is provide some reasonable protection of secrets in plain text. Note: the C# utilizing this option requires the inclusion of my encryption/decryption DLL which is also in GitHub: https://github.com/aceeric/crypto.
-force
Optional. Specify this option to force the utility to download the monthly file even if the S3 bucket already has a file for the current month. The default behavior, if this option is omitted, is for the utility to check the S3 bucket on startup and if a file for the year and month specified by the -yyyymm option is present there, the utility exits without doing anything.
-keyname name
Optional. The key name to assign to the file when it is uploaded to S3. By default, the downloaded filename is used as the key name when the object is copied to S3. This option enables you to override that and explicitly specify a key name. Note - if you specify this option, as well as the -prefix option, both are used.
-onlydownload
Optional. If provided, the the utility stops after downloading the file, and does not copy the file to S3. If specified, then the AWS credentials are optional and ignored. The utility also skips the check for an existing copy of the file in S3. Useful for just grabbing down the file from the website.
-jobid guid
Optional. Defines a job ID for the logging subsystem. A GUID value is supplied in the canonical 8-4-4-4-12 form. If provided, then the logging subsystem is initialized with the provided GUID. The default behavior is for the logging subsystem to generate its own GUID.
-log file|db|con
Optional. Determines how the application communicates errors, status, etc. If not supplied, then all output goes to the console. If file is specified, then the application logs to a log file in the application directory called download-sam.log. If db is specified, then logging occurs to the database. If con is specified, then output goes to the console (same as if the option were omitted.) If logging to file or db is specified then the application runs silently with no console output.
Note: the C# utilizing this option requires the inclusion of my logging DLL which is also in GitHub: https://github.com/aceeric/logger.
-loglevel err|warn|info
Optional. Defines the logging level. err specifies that only errors will be reported. warn means errors and warnings, and info means all messages. The default is info.
Note: this utility - along with most of the C# in by repo - was created as part of a private project. The project completed and the sponsor was kind enough to grant me the right to host the code in my repo. Hence no commit history.