prudhvi85 / airbyte-cli

This is the next generation Go based Airbyte CLI.

Airbyte CLI

This is the next generation Go based Airbyte CLI.

Contents

• Setup
• Configuration as YAML
• Usage
  • get
  • search
  • check
  • logs
  • export
  • create

Setup

1. Clone this repo to your working environment git clone https://github.com/harshithmullapudi/airbyte-cli.git
2. Install go (if not installed already). brew install go (for mac users)
3. Jump to the airbyte-cli directory and run go install .
4. You're all set to fly

Configuration as YAML

Before we interact with Airbyte API, we need to set Airbyte API URL and workspace, for which you can use the following command

$ airbyte set-config

Usage

An Airbyte CLI command has the following structure:

$ airbyte <command> <subcommand> [options and parameters]

<command>
          get - Get configuration of Sources/Destinations/Connections
          search - Search in sources
          check - Check connection to Source/Destination
          logs - Fetch logs for a job
          set-config - Set your airbyte url and workspaceID
          help - Help about any command

To view help documentation, use one of the following:

$ airbyte -h
$ airbyte <command> -h
$ airbyte <command> <subcommand> -h

get

Return all

• workspaces (/v1/workspaces/list)
• sources (/v1/sources/list)
• destinations (/v1/destinations/list)
• connections (/v1/web_backend/connections/list)

You can use page(p) and offset(o) to fetch sources respectively.

To list workspaces, the command would be:

$ airbyte get workspaces

To get a workspace, the command would be:

$ airbyte get workspace [workspaceId]

To list sources, the command would be:

$ airbyte get sources

To get a source, the command would be:

$ airbyte get source [sourceId]

To get jobs for a connection, the command would be:

$ airbyte get jobs [connectionId]

search

• sources
• connections

To search in sources, the command would be:

$ airbyte search sources [string]

check

Validate

• source (/v1/sources/check_connection)
• destination (/v1/destinations/check_connection)

To validate a source, the command would be:

$ airbyte check source [sourceId]

logs

Get logs for a job

$ airbyte logs [jobId]

export

Export sources, connections and destinations to a target folder

$ airbyte export -t [absolute path to target folder]

create

You can create and validation sources, destinations and connections through a yaml file.

1. Create a folder in your home example: /Users/name/home/load
2. Create a yaml file SOURCE_CONNECTION.yaml for sources, DESTINATION_CONNECTION.yaml for destinations and STANDARD_SYNC.yaml for connections
3. Hurray

$ airbyte create -f /Users/name/home/load -c

Validate sources, destinations and connections before creation

$ airbyte create -f [path to config folder]

You can pass -c as a flag to create the sources which will validate the source and skip it either if validation failes or if it finds sourceId and a source for that respective Id.

$ airbyte create -f [path to config folder]