More flexible and cheaper forest database population.

[Daniel Kelleher]

Summary

Replace the existing Forest database population pipeline with a more flexible one that we control.

Details

The forest feature in app.sunrisestake.com is driven by on-chain data tracking gSOL minting, and transfers of gSOL between wallets.

If you have staked SOL in the app, you have a tree. This tree is based on the age of your stake as well as the size of the stake.

If you have sent gSOL to someone else, or received gSOL, their tree also appears in your forest.

This feature allows the creation of communities of individuals interested in regeneration on Solana, allows discovery of projects to donate to, and organisations that support gSOL, as well as add a sense of community and humanity to the staking feature.

It is currently driven by a somewhat limited and opaque chain of systems, that were put together for the Grizzlython hackathon.

Current Flow

The flow at present is as follows:

Solana -> Helius.xyz -> Triggr (Intersect) -> Zapier -> MongoDB

1. A transfer occurs on Solana
2. This is picked up by Helius, which calls a webhook in Intersect
3. Intersect parses the data and calls Zapier
4. Zapier extracts specific fields and makes a POST to MongoDB via the MongoDB http API.

The mongodb database is then read by the app and used to show the forest on screen.

Problems

The main problems with the above, which we want to fix in this ticket are:

1. Intersect is an offshoot of Triggr which is being deprecated. It is essentially a pipe from helius to Zapier, it is hosted on a machine by the Triggr founder and cannot be easily changed or maintained.
2. Intersect only accepts certain transaction types at present. Mint gSOL and SPL token transfer. We have a concrete requirement to support other transaction types. Specifically, we would like metadata nft mints using gsol as payment (such as for this item on coinable) to result in an entry in the forest db.
3. Zapier is expensive at scale. If we are replacing Intersect, we do not need to keep it. Rather, we can make DB inserts directly in the code.

The Proposal

Replace the above flow with the following one:

1. A transfer occurs on Solana
2. This is picked up by Helius, which calls a Google Cloud function
3. The Google Cloud function makes a POST to MongoDB via the MongoDB http API.

Why Google Cloud function?

It does not have to be. AWS Lambda is also ok. I think a lambda/cloud function is appropriate as opposed to running a server as it reduces maintenance, and scales well with price as activity scales. We have some functions in google already, which is why I would prefer GCP, however, these functions are also somewhat ad-hoc, there are no tests or CI/CD for them. Part of this work would be to set up a proper repo including tests and CI/CD, and then later, in another ticket we can move all the functions into there.

We have an account in GCP already, and @dankelleher will provide access to whoever picks up this ticket.

How do we set up Helius (step 2)?

Once a cloud function is set up, @dankelleher will update Helius to post to it. At the moment it is posting to webhook.site for debugging. You can use that site to see what the transactions will look like.

Why POST via http rather than using the mongodb client.

This is a suggestion more than a requirement, but the reasons are:

• smaller function payload if the mongo client is excluded = faster cold start time for the function
• simple - just http
• less work - we can copy the function already in the Zapier:

POST
https://data.mongodb-api.com/app/data-vhksk/endpoint/data/v1/action/insertOne
Headers
  Content-type: application/json
  api-key: <<SECRET>>
Body:

{
      "dataSource": "Cluster0",
      "database": "gsol-tracker",
      "collection": "transfers",
      "document": {
        "timestamp": { "$date": {"$numberLong": "{{timestamp}}000" }},
        "sender": "{{tokenTransfers[]fromUserAccount}}",
        "recipient": "{{tokenTransfers[]toUserAccount}}",
				"amount": {{tokenTransfers[]tokenAmount}}
      }
  }

Acceptance Criteria

1. The cloud function is deployed to production and publishing to new collections in mongodb
2. The app is switched to pointing to these new collections
3. CI/CD is set up and tests, linting are passing
4. gSOL mints, transfers and metaplex mints are all populating the database
5. It is easy to add new transaction types as needed, and this feature is documented in the repo