tap-outreach

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

Pulls raw data from Outreach API
Outputs the schema for each resource
Incrementally pulls data based on the input state

Streams

Quick Start

Install

Clone this repository, and then install using setup.py. We recommend using a virtualenv:

> virtualenv -p python3 venv
> source venv/bin/activate
> python setup.py install
OR
> cd .../tap-outreach
> pip install .

Here is what the config should look like

{
    "start_date": "2019-01-01T00:00:00Z",
    "client_id": <CLIENT_ID>,
    "client_secret": <CLIENT_SECRET>,
    "redirect_uri": <REDIRECT_URI>,
    "refresh_token": <REFRESH_TOKEN>,
    "quota_limit": <QUOTA_LIMIT>
}

The following are required values: start_date, client_id, client_secret, redirect_uri, refresh_token The following are optional values: quota_limit

Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

  {
    "bookmarks": {
      "stream_1": "2020-01-04T00:00:00.000000Z",
      "stream_2": 1576266090,
      ...
    }
  }

Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:
```
tap-outreach --config config.json --discover > catalog.json
```
See the Singer docs on discovery mode here.

Run the Tap in Sync Mode (with catalog) and write out to state file

For Sync mode:

> tap-outreach --config tap_config.json --catalog catalog.json > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

To load to json files to verify outputs:

> tap-outreach --config tap_config.json --catalog catalog.json | target-json > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

To pseudo-load to Stitch Import API with dry run:

> tap-outreach --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

Test the Tap

While developing the Outreach tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

> pylint tap_outreach -d broad-except,chained-comparison,empty-docstring,fixme,invalid-name,line-too-long,missing-class-docstring,missing-function-docstring,missing-module-docstring,no-else-raise,no-else-return,too-few-public-methods,too-many-arguments,too-many-branches,too-many-lines,too-many-locals,ungrouped-imports,wrong-spelling-in-comment,wrong-spelling-in-docstring

The pylint disables that Stitch uses are:

broad-except,chained-comparison,empty-docstring,fixme,invalid-name,line-too-long,missing-class-docstring,missing-function-docstring,missing-module-docstring,no-else-raise,no-else-return,too-few-public-methods,too-many-arguments,too-many-branches,too-many-lines,too-many-locals,ungrouped-imports,wrong-spelling-in-comment,wrong-spelling-in-docstring

To check the tap and verify working:

> tap-outreach --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

Check tap resulted in the following:

The output is valid.
It contained 8240 messages for 16 streams.

    16 schema messages
8108 record messages
    116 state messages

Details by stream:
+-----------------------------+---------+---------+
| stream                      | records | schemas |
+-----------------------------+---------+---------+
| **ENDPOINT_A**              | 23      | 1       |
+-----------------------------+---------+---------+

betoesquivel / tap-outreach

tap-outreach

Streams

Quick Start

About

Languages