Giters

vanus-labs / vanus-connect

Vanus Connect allows you to skip the complex integration with external services by offering out-of-the-box connectors.

Home Page:https://www.vanus.ai/connectors

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

🎯New Source Connector Request: test Source Connector

Michaelg22 opened this issue · comments

commented

Source Connector Twitter

What is a Vanus Connector

Vanus Connect allows you to skip the complex integration with external services by offering out-of-the-box connectors. Each connector acts as a proxy between outside services and your system, enabling users to have faster integrations with no codes. All data produced or processed by our connectors conform to the CloudEvents specification, which helps enterprises leverage event-driven architecture to do business in the age of events.

What is a Source Connector

A Source connector obtains data from an underlying data producer and delivers it to the target after the original data has been transformed into a CloudEvents. For example:

  • MySQL Source: extracts data from the Binlog, transforms it into CloudEvents and sends it to the target.

  • GitHub Source: uses a Webhook server to receive the event from GitHub, transform them to CloudEvents, and send them to the target.

  • AWS S3 Source: Uses the AWS SDK to get the data, transforms it into CloudEvents, and sends them to the target.

There are two types of connectors push or pull connectors.

  • A push connector is passive and only does its logic when it receives an event
  • A pull connector must pull the data in an interval. We use a push method when it’s possible

Features of this Source Connector

Receive events from TikTok when a user’s post is liked, shared, or commented on, and when a user is followed and put the data into a CloudEvent to send it to the target

Specific data needed

When someone likes the user's post

  • Post ID: The unique identifier for the post that was liked. The post could be a video.
  • User ID: The unique identifier for the user who liked the post.
  • User screen name: The screen name (i.e., username) of the user who liked the post.
  • Like creation time: The timestamp indicates when the like was created.
  • Post URL: The URL of the post that was liked.
  • Number of likes: The total number of likes the post has received.

When a post is commented on

  • Post ID: The unique identifier for the post that was commented on. The post could be a video.
  • User ID: The unique identifier for the user who commented on the post.
  • User screen name: The screen name (i.e., username) of the user who commented on the post.
  • Number of Comments: The total number of replies the post has received.
  • Post URL: The URL of the post that was commented on.
  • Comment creation time: The timestamp indicates when the comment was created.

When a user is followed

  • User ID: The unique identifier for the user who followed.
  • Number of Followers: The total Number of Followers the user.
  • Follow creation time: The timestamp indicates when the user was followed.

How to create a new connector

  • Fork the vanus-connect repo.
  • Create a source_twitter_proposal.md based on the source_proposal_example.md under the vanus-connect/proposals directory. The proposal will introduce how a developer tends to implement the connector in detail. The proposal PR must be approved and merged before going to the next step.
  • Copy one of the source templates (Golang template, Java template) to vanus-connect/connectors/ and rename it as source-twitter.
  • Implement your connector and submit a PR.
  • Wait for the connector to be reviewed and approved.

How to claim to implement this connector

If you want to solve this issue, please leave a comment on this issue like:

I'd like to implement the connector, please assign it to me.

Vanus community will assign the issue to you on time.