Support for work queues
ameyapathare opened this issue · comments
It would be very helpful to be able to setup/configure our work queues via terraform. Right now, we create an 'on-demand', 'priority', and 'default' work queue for each work pool. We are also contemplating creating a separate work queue per data source, which would involve setting up hundreds of work queues.
Work queues are the last remaining resource before we switch over to using this terraform provider to provision our prefect infrastructure!
@jamiezieziula i've assigned this to you to take a look
at a high level, here are some pointers that should you get started:
-
Think of this provider as a jazzed-up API client, so there will be code to add in the
/internal/api
section of this repo to set up a new section that interacts with the/work_pools/{pool_name}/queues
API. This is the best starting point - we're setting up the interface between the provider and the Prefect Cloud API- In
/internal/api
, we'll set newstruct
data models for the new client (eg.WorkQueuesClient
), the representation of the resource (eg.WorkQueue
), and optionally the create/update payloads (eg.WorkQueueCreate
orWorkQueueUpdate
). See this existing example forWorkPool
- In
/internal/client
, we'll set the actual API methods that correspond withGET
POST
DELETE
etc. See this existing example forWorkPool
- customarily, we'll create theList
,Get
,Create
,Update
, andDelete
methods -- essentially, CRUD + a List method (which hitsPOST /filter
) - Be sure to add your new client interface into the root level
PrefectClient
interface
- In
-
when looking at our API schema, you might see 2 different "groups" of
work-queue
API endpoints. for this provider, we'll want to use the one that is a part of the/work_pools/{pool_name}/queues
API (so the one always associated with a particular work pool) and NOT the one inside of the/work_queue
API (this is for the old agent workloads, which is on its way out) -
As far as the actual Terraform resources, we'll want to support
datasource
,resource
, andimport
functionality for this newWork Queue
object type. See the matrix here.datasource
allows querying existing resources, eg.data "some_object" "my_object_name" {}
, so you can read down attributes (like theid
or others)resource
allows fully managing the resource in Terraformimport
allows linking existing resources into the TF lifecycle
-
The way we structured this repo, you'll have to set up a separate:
datasource
in the /internal/provider/datasources directory, in awork_queue.go
fileresource
in the /internal/provider/resources directory, in awork_queue.go
fileimport
support goes into the associated resources file, inside of a reservedImportState
method on the resource class/struct. See this example for aservice_account
import, via a commonly used pattern where we allow passing a resource ID or a resource name via the prefixname/<your resource name here>
-
For
datasource
andresource
configurations in the code - the top level function methods are special + reserved, so they represent the logic that the Terraform provider framework will expect + call. You can pull those over from an existing, correspondingdatasources/<name>.go
/resources/<name>.go
. The logic to update will predominantly be:- The
Schema
methods + the*Model
structs - The
Read
method, as we'll be calling the client methods you defined first + have the flexibility to perform certain kinds of checks
- The
-
Finally, there are root-level configurations where you'll need to add your newly created
datasources.NewWorkQueueDataSource
andresources.NewWorkQueueResource