HazardDede / pnp

Pull 'n' Push - Pulls data from sources and pushes it to sinks

Pull 'n' Push

Pulls data from sources and pushes it to sinks with optional transformations in between.

Installation

pip install pnp

Installation with extras:

pip install pnp[fswatcher,faceR]

Please consult the component documentation to see if a component requires an extra or not.

Getting started

Define pulls to fetch / pull data from source systems. Define one push or multiple pushes per pull to transfer the pulled data anywhere else (you only need a plugin that knows how to handle the target). You configure your pipeline in yaml:

tasks:
  - name: hello-world
    pull:
      plugin: pnp.plugins.pull.simple.Repeat
      args:
        interval: 1s
        repeat: "Hello World"
    push:
      - plugin: pnp.plugins.push.simple.Echo

Copy this configuration and create the file helloworld.yaml. Run it:

pnp helloworld.yaml

This example yields the string Hello World every second.

Hint: You can validate your config without actually executing it with

   pnp --check helloworld.yaml

If you want to learn more please see the documentation at Read the Docs.