firebase / genkit

An open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to integrate, test, and deploy sophisticated AI features to Firebase or Google Cloud.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Go] allow actions to be registered at any time

jba opened this issue · comments

Currently, all models, embedders, etc. must be registered before the dev server/reflection API starts. This is awkward in Go, because we'd like to be able to create things whenever we want, typically shortly before we use them. For example, if only one route of a server does indexing, we might want to create the indexer when that route is handled.

There are two reasons for registering everything at startup:

  1. To support a "getting started" CUJ where a user can run genkit init, specify a plugin or two and get a menu of all available actions in the UI.
  2. To support the "run in playground" feature of the trace viewer, where one can start a program in dev mode, view an action trace from a previous run, then execute that action with the same inputs from the playground.

The first reason does indeed require everything to be registered up front, and the templates used by genkit init, being code under our control, can make that happen.

The second reason doesn't actually require everything to be registered at the beginning. It only requires that there is enough information in the trace to register an action if it isn't already registered. At startup, anything that can't be saved in the trace, such as auth-related values like clients, must be created and kept around. But other inputs to registering an action can live in the trace and be passed as part of the runAction request.

This would require traces to be extended with the inputs needed to define the action being traced, and having those inputs passed in the runAction request. If the dev server's runAction handler did not find the key in the registry, it would invoke the plugin with the name and inputs to the action, and the plugin would then create and register the action.

As a simple example, if a googleai model wasn't registered, all the googleai plugin would need is the model name (already part of the key) to register the model. In most cases, other inputs would be needed too. For instance, a retriever needs an embedder, and that embedder might need to be registered recursively.

The downsides are added complexity for the runAction request, and added space in every trace, or the ability to store action definition inputs in another way to avoid duplicating it in every trace. This would be a lot of work, and it wouldn't help JS at all, because everything is already registered at the beginning. So to be worth it, it would have to make the Go programming experience significantly better.