foxsec-pipeline
Apache Beam pipelines for analyzing log data.
Tests
Tests can be executed locally using Docker.
Run all tests
docker build -t foxsec-pipeline:latest .
bin/m testRun a specific test
docker build -t foxsec-pipeline:latest .
bin/m test -Dtest=ParserTestUsage
Pipeline Runtime Secrets
Generating locally for use in PipelineOptions
Pipeline runtime secrets can be generated locally using the main method in the RuntimeSecrets class.
bin/m compile exec:java -Dexec.mainClass=com.mozilla.secops.crypto.RuntimeSecrets -Dexec.args='-i testkey -k dataflow -p my-gcp-dataflow-project -r dataflow'Run the class with no options to see usage information. Note that in this case, the key ring name and key name
are being specified as dataflow. The existing RuntimeSecrets class requires the keys to be accessible
using these identifiers when the pipeline is executing.
The output of the command can be prefixed with cloudkms:// in an option to enable runtime decryption of the secrets
during pipeline execution.
Documentation
javadoc documentation is currently updated manually and although should be up to date, may not be current with master.
Introduction to Beam
To get familiar with developing pipelines in Beam, this repository also contains a small workshop that provides some guidance on building basic pipelines. The introduction document can be found here.
Contributing
See the contributing guidelines.