tiny in--{command}-->out runner
you probably should use something else:
sdflow fits in a project that does small-scale, standardized, repeated data load and transformation.
I like make, but want an easier way of doing input/output hash validation.
just also doesn't have built-in validation.
task syntax is too verbose for these low-complexity targets.
dvc is too cumbersome for repeat runs.
sdflow just puts these features together:
- suitable for small number of targets, tiny syntax, simple dependency chains
- built-in sha256 validation or generation for inputs/outputs, so the Sdflow.yaml file serves as the reference for data integrity.
- built-in support for HTTP(S)/S3 sources as inputs
Anything else, you should use a more mature tool.