RFC: Data source connectors

[Max Stoiber]

Summary

Allow quick adding of nodes that are backed by REST, gRPC, and/or other data sources. (unsure about dbs)

Proposed Solution

TODO

[Bouba]

Hi there,

I've come across this issue about allowing quick adding of nodes backed by various data sources like REST, gRPC, etc., and I find it incredibly interesting and aligned with some of the challenges I'm facing.

I'm particularly interested in the potential to create specific connectors for common services like CMS or e-commerce platforms. The idea would be to develop generic wrappers around the specific SDKs of these services, making them open-source for broader use and adaptation.

A key feature I believe would be invaluable is the composability of these nodes. It would be great if we could import a node and then have the flexibility to redefine or add new properties according to our specific needs. This approach would significantly streamline the process of integrating multiple APIs by reducing the complexity involved in mapping and transforming data for the frontend.

One practical use case I envision is in scenarios where there's a need to integrate a multitude of APIs. A composable node system would simplify this process, making it more manageable and efficient for frontend teams.

The idea of open-source nodes that can be customized and extended fits perfectly with the needs of many developers looking to simplify API integrations.

[Jovi De Croock]

Hey @b-barry

Thank you for the thoughtful reply! I agree with you on datasources in a very early draft of this library I actually used the datasource concept.

I find the idea of creating an ecosystem of datasources quite appealing, I however personally am not a big fan of exporting node as part of it 😅 the main thing you highlight there is my reason why, the properties should need to be altered which is going to be very hard from both a TypeScript and GraphQL.js point of view as the type is stored and "just works".

That being said, I still like the idea and would encourage a slightly different paradigm where the SDK-Ecosystem exports an interface which can be used in node({ interfaces: [] }) that can already give you a minimal working interface of what's possible with the type and additionally to that the TypeScript type that is exported (or used with the datasource). The reason behind the above is that in t.expose() we'll give you all the options that are present in the type passed to the datasource or node.

How does that resonate with you?

[Bouba]

Hey @JoviDeCroock

If I get it, we will have datasource that will be a wrapper around the SDK like you do in your example then you will have also exported interface for common field. Hovewer, I don't think the interface will be needed if we have a good typing from the datasource.
Like you said, we have technically everything to make the mapping. The complexity is mostly in the mapping.

What do you think?

[Jovi De Croock]

I agree, so basically the package exports

type ShopifyProduct = { id: string; name: string; price: number }
export type ShopifyProductDatasource = Datasource<ShopifyProduct>;

Now when that is used in a node it automatically does datasource.getOne or datasource.getMany for the load function. The type is automatically derived as it's part of the generic passed to the Datasource type. That means that when you do

node({
  name: 'Product',
  datasource: new ShopifyDatasource(yourEnv),
  // load becomes optional as it behind the scenes takes getOne or getMany
  fields: t => ({
    title: t.exposeString('name') // automatically suggests fields from the ShopifyProduct type
  })
})

to put the API-surface in practice

[Bouba]

I like it, it is interesting ! Another suggestion below

node({
  name: 'Product',
  datasource: new ShopifyDatasource(yourEnv),
  fields: t => ({
    title: t.exposeString('price.value') // could we have multi level mapping ?
  })
})

[Jovi De Croock]

Concluded in #103 (comment)