Feedback on mediaTypes for storing OPA bundles as OCI images

Question

Feedback on mediaTypes for storing OPA bundles as OCI images

garethr opened this issue 5 years ago · comments

As discussed briefly with @tsandall. Posting for visibility and to widen the discussion.

I've been hacking on https://github.com/instrumenta/conftest, which uses OPA/rego but presents an interface for local unit testing of configuration. I'll be talking at KubeCon in a few weeks about this and why I think it's useful.

One thing I've added recently is the ability to share rego files on OCI registries. Basically you can do the following and download existing rules or other bits.

conftest pull instrumenta.azurecr.io/kubernetes-helpers

That spun off work I was doing with @SteveLasker (product manager for Azure Container Registry at Microsoft) before I left Docker. Basically better support in registries for other types of content than just Docker images. Steve has a proposal up at:

https://github.com/SteveLasker/RegistryArtifactTypes/blob/master/mediaTypes.md

The rationale for sharing things via OCI images is described in this blog post https://stevelasker.blog/2019/01/25/cloud-native-artifact-stores-evolve-from-container-registries/. But in short, everyone already has one (whether cloud provider, public/private, self-hosted, geo-replicated, etc.)

As part of the proposal content can have a mimetype. This allows for more intelligent clients and interesting possibilities on the registries. As a very simple example, the registry could show the OPA icon when you view an OCI image which contains a rego bundle.

I'm proposing the following mimetypes for OPA content:

Usage	mediaType
OPA Bundle	`application/vnd.cncf.openpolicyagent.config.v1`
Bundle .manifest	`application/vnd.cncf.openpolicyagent.manifest.v1`
Rego files	`application/vnd.cncf.openpolicyagent.rego.v1`
Data files	`application/vnd.cncf.openpolicyagent.data.v1`

This broadly follows the current bundle format, just packaged as an OCI image rather than a flat tar file. https://www.openpolicyagent.org/docs/latest/bundles/

It should be possible to unpack an OCI bundle and get back the folder structure. The different mediaTypes simply mean the content is separated into individual layers in to the OCI image.

Sharing OPA bundles as OCI images may be useful outside conftest, at which point I'm happy to rip out that code to somewhere else.

Do the above media types look sensible to folks?

Torin Sandall · Answer 1 · Thu May 09 2019 10:45:53 GMT+0800 (China Standard Time)

This looks good to me. For the data file mimetypes, would it make sense to distinguish between JSON and YAML (OPA supports both)?

There shouldn't be any issue using the OPA logo/icon in the registry. /cc @caniszczyk just to double check.

Gareth Rushgrove · Answer 2 · Thu May 09 2019 19:48:34 GMT+0800 (China Standard Time)

I think the convention here is use subtypes. SO you can say "this is data" and "this is formatted in YAML". eg.

application/vnd.cncf.openpolicyagent.data.v1+json
application/vnd.cncf.openpolicyagent.data.v1+yaml

Chris Aniszczyk · Answer 3 · Thu May 09 2019 20:47:53 GMT+0800 (China Standard Time)

it’s fine

On Thu, May 9, 2019 at 6:48 AM Gareth Rushgrove ***@***.***> wrote: I think the conversion here is use subtypes. SO you can say "this is data" and "this is formatted in YAML". eg. application/vnd.cncf.openpolicyagent.data.v1+json application/vnd.cncf.openpolicyagent.data.v1+yaml — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1413 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAPSIJJD2XOCALVZK3TNULPUQFRLANCNFSM4HLICY4A> .

-- Cheers, Chris Aniszczyk http://aniszczyk.org +1 512 961 6719

Steve Lasker · Answer 4 · Sat May 11 2019 01:07:12 GMT+0800 (China Standard Time)

Hey Gareth, et all. Great to see new scenarios.
I’m departing on a flight, so a quick response.
on the bundle, i’m assuming this is how you want registries to identify your artifact type using manifest.config.mediaType.
here’s what i’d suggest: application/vnd.cncf.openpolicyagent.config.v1+json
This declares the type, version, provides for a config and says the config is in json format. If you don’t need a config, you can send an empty json config file or should be able to specify dev/null, but i saw an error when using ORAS yesterday
for the rego and data files, are these layers
If so, i’d suggest application/vnd.cncf.openpolicyagent.data.layer.v1.json (or yaml)
the important part is to make it obvious it’s a layer type. If there are multiple files in the layer, using tar. If there are individual files, you can save on the compression and decompression and just reference txt, yaml, json, html, xml, whatEverNewFileFormat comes.
Here’s the bare bones we’re using for this week, prepping for rejekts and kubecon: https://github.com/Azure/acr/blob/master/docs/artifact-media-types.json
Are you planning to use OCI index to reference multiple artifacts, or just one OCI manifest?
I have more clarity on how we want to describe this in the distribution spec, so i’ll work on clearing up the inferences form all the discussion.
I’ll review more tonight as we’re taking off.

Gareth Rushgrove · Answer 5 · Sat May 11 2019 23:49:25 GMT+0800 (China Standard Time)

Thanks for the feedback Steve. Looks like we had the same thing for the top level config. Modified versions of the layer suggestions below:

Usage	mediaType
OPA Bundle	`application/vnd.cncf.openpolicyagent.config.v1+json`
Bundle .manifest layer	`application/vnd.cncf.openpolicyagent.manifest.layer.v1.json`
Rego files layer	`application/vnd.cncf.openpolicyagent.policy.layer.v1.rego`
Data files layer	`application/vnd.cncf.openpolicyagent.data.layer.v1.json`

This assumes a single layer per file, mainly to facilitate easy uploading with oras like so:

oras push localhost:5000/my-bundle-bundle:latest \
  .manifest:application/vnd.cncf.openpolicyagent.manifest.layer.v1.json \
  helpers.rego:application/vnd.cncf.openpolicyagent.rego.layer.v1.rego \
  deny.rego:application/vnd.cncf.openpolicyagent.policy.layer.v1.rego \
  samples.json:application/vnd.cncf.openpolicyagent.data.layer.v1.json \

I assume that later on also using application/vnd.cncf.openpolicyagent.policy.layer.v1.tar and application/vnd.cncf.openpolicyagent.data.layer.v1.tar for multiple data or policy files per layer would also make sense.

@jdolitsky I'd love your feedback on whether that makes sense?

I'd imagine writing a custom uploader with some domain knowledge using oras as a library, but looking to punt that to a little later.

Steve Lasker · Answer 6 · Sun May 12 2019 09:15:07 GMT+0800 (China Standard Time)

Ahh, I see the confusion. You don't actually push a manifest by referencing a file. By pushing the --manifest-config with a mediaType, the OCI manifest is constructed for you. See the modified version below. Also, note the +rego vs. .rego

oras push localhost:5000/my-bundle-bundle:latest \
  --manifest-config ./config.json:application/vnd.cncf.openpolicyagent.config.v1+json \
  helpers.rego:application/vnd.cncf.openpolicyagent.rego.layer.v1+rego \
  deny.rego:application/vnd.cncf.openpolicyagent.policy.layer.v1+rego \
  samples.json:application/vnd.cncf.openpolicyagent.data.layer.v1+json

If you don't need a config.json file, you should be able to pass in dev/null. I believe this is not currently supported in the ORAS cli, but we'll need to investigate and fix if not supported.

So, you don't actually need the

Usage	mediaType
Bundle .manifest layer	application/vnd.cncf.openpolicyagent.manifest.layer.v1.json

Gareth Rushgrove · Answer 7 · Sun May 12 2019 14:35:10 GMT+0800 (China Standard Time)

You don't actually push a manifest by referencing a file

There are two manifests :) One is the OCI manifest for the bundle as a whole, which I'd missed out from the oras command example. The other is the .manifest file (formatted in JSON) that is specific to OPA and defined the OPA bundle docs. The intention is to support storing that on it's own layer.

Also, note the +rego vs. .rego

Ah, the first draft of your blog post had eg. application/vnd.cncf.helm.values.layer.v3.yaml so I thought this had changed. But it's the right way round in the final published post. Swapping back.

Updates for both of those below:

Usage	mediaType
OPA Bundle	`application/vnd.cncf.openpolicyagent.config.v1+json`
Bundle .manifest layer	`application/vnd.cncf.openpolicyagent.manifest.layer.v1+json`
Rego files layer	`application/vnd.cncf.openpolicyagent.policy.layer.v1+rego`
Data files layer	`application/vnd.cncf.openpolicyagent.data.layer.v1+json`

And usage with Oras:

oras push localhost:5000/my-bundle-bundle:latest \
  --manifest-config ./config.json:application/vnd.cncf.openpolicyagent.config.v1+json \
  .manifest:application/vnd.cncf.openpolicyagent.manifest.layer.v1+json \
  helpers.rego:application/vnd.cncf.openpolicyagent.policy.layer.v1+rego \
  deny.rego:application/vnd.cncf.openpolicyagent.policy.layer.v1+rego \
  samples.json:application/vnd.cncf.openpolicyagent.data.layer.v1+json \

Steve Lasker · Answer 8 · Sun May 12 2019 18:52:14 GMT+0800 (China Standard Time)

The world of manifests. This makes sense.
If you can do a PR to https://github.com/Azure/acr/blob/master/docs/artifact-media-types.json for the OCI manifest.config.mediaType and the short name, we can get this loaded up for next week.

Josh Dolitsky · Answer 9 · Mon May 13 2019 03:59:09 GMT+0800 (China Standard Time)

@garethr your table of mediatypes + ORAS command in your last comment looks good to me, although seems the command has one extra media type than the table? It could maybe be the case that some bundle contain files/layers that others don't, and that's ok if the client knows how to handle it.

For some hints of using ORAS as a Go lib, you can take a look at the docs. We are still working on putting together a bunch of examples for copy-pasta goodness.

The following is how you might push an OPA bundle based on your spec:

package main

import (
	"context"
	"fmt"
	"io/ioutil"

	"github.com/containerd/containerd/remotes/docker"
	"github.com/deislabs/oras/pkg/content"
	"github.com/deislabs/oras/pkg/oras"
	ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)

const (
	// Remote reference
	// Command used to run a local test registry on port 5000:
	//   docker run --rm -it -p 5000:5000 registry
	RemoteRef = "localhost:5000/opa/mybundle:1.0.0"

	// OPA-specific file names
	// Command used to create dummy files created for test purposes:
	//   for f in ".manifest" "helpers.rego" "deny.rego" "samples.json"; do echo "hello world" > $f; done
	OpenPolicyAgentManifestFileName = ".manifest"
	OpenPolicyAgentRegoFileName     = "helpers.rego"
	OpenPolicyAgentPolicyFileName   = "deny.rego"
	OpenPolicyAgentDataFileName     = "samples.json"

	// OPA-specific media types
	OpenPolicyAgentConfigMediaType        = "application/vnd.cncf.openpolicyagent.config.v1+json"
	OpenPolicyAgentManifestLayerMediaType = "application/vnd.cncf.openpolicyagent.manifest.layer.v1+json"
	OpenPolicyAgentRegoLayerMediaType     = "application/vnd.cncf.openpolicyagent.rego.layer.v1+rego"
	OpenPolicyAgentPolicyLayerMediaType   = "application/vnd.cncf.openpolicyagent.policy.layer.v1+rego"
	OpenPolicyAgentDataLayerMediaType     = "application/vnd.cncf.openpolicyagent.data.layer.v1+json"
)

func check(e error) {
	if e != nil {
		panic(e)
	}
}

func getFileContents(fileName string) []byte {
	b, err := ioutil.ReadFile(fileName)
	check(err)
	return b
}

func main() {
	// Setup
	var contents []byte
	var layer ocispec.Descriptor
	var layers []ocispec.Descriptor
	resolver := docker.NewResolver(docker.ResolverOptions{})
	memoryStore := content.NewMemoryStore()

	// Add layers
	contents = getFileContents(OpenPolicyAgentManifestFileName)
	layer = memoryStore.Add(OpenPolicyAgentManifestFileName, OpenPolicyAgentManifestLayerMediaType, contents)
	layers = append(layers, layer)

	contents = getFileContents(OpenPolicyAgentRegoFileName)
	layer = memoryStore.Add(OpenPolicyAgentRegoFileName, OpenPolicyAgentRegoLayerMediaType, contents)
	layers = append(layers, layer)

	contents = getFileContents(OpenPolicyAgentPolicyFileName)
	layer = memoryStore.Add(OpenPolicyAgentPolicyFileName, OpenPolicyAgentPolicyLayerMediaType, contents)
	layers = append(layers, layer)

	contents = getFileContents(OpenPolicyAgentDataFileName)
	layer = memoryStore.Add(OpenPolicyAgentDataFileName, OpenPolicyAgentDataLayerMediaType, contents)
	layers = append(layers, layer)

	// Push
	fmt.Printf("Pushing OPA bundle to %s...\n", RemoteRef)
	extraOpts := []oras.PushOpt{oras.WithConfigMediaType(OpenPolicyAgentConfigMediaType)}
	manifest, err := oras.Push(context.Background(), resolver, RemoteRef, memoryStore, layers, extraOpts...)
	check(err)
	fmt.Printf("Pushed OPA bundle to %s with digest %s\n", RemoteRef, manifest.Digest)
}

and the resulting manifest:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.openpolicyagent.config.v1+json",
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "size": 2
  },
  "layers": [
    {
      "mediaType": "application/vnd.cncf.openpolicyagent.manifest.layer.v1+json",
      "digest": "sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447",
      "size": 12,
      "annotations": {
        "org.opencontainers.image.title": ".manifest"
      }
    },
    {
      "mediaType": "application/vnd.cncf.openpolicyagent.rego.layer.v1+rego",
      "digest": "sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447",
      "size": 12,
      "annotations": {
        "org.opencontainers.image.title": "helpers.rego"
      }
    },
    {
      "mediaType": "application/vnd.cncf.openpolicyagent.policy.layer.v1+rego",
      "digest": "sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447",
      "size": 12,
      "annotations": {
        "org.opencontainers.image.title": "deny.rego"
      }
    },
    {
      "mediaType": "application/vnd.cncf.openpolicyagent.data.layer.v1+json",
      "digest": "sha256:a948904f2f0f479b8f8197694b30184b0d2ed1c1cd2a1ec0fb85d299a192a447",
      "size": 12,
      "annotations": {
        "org.opencontainers.image.title": "samples.json"
      }
    }
  ]
}

Hope this helps! It's interesting to see this concept being brought to other projects.

Gareth Rushgrove · Answer 10 · Mon May 13 2019 05:30:40 GMT+0800 (China Standard Time)

@jdolitsky thanks, the extra type was a typo from a previous version. Fixed now.

Thanks for the code, I'll use that as the starting point. I have the pull side already implemented.

Gareth Rushgrove · Answer 11 · Tue May 14 2019 14:33:39 GMT+0800 (China Standard Time)

I've submitted the top level media type to the ACR registry Azure/acr#225. I'll close this issue when that gets merged. Thanks everyone.

Josh Dolitsky · Answer 12 · Tue May 14 2019 20:21:26 GMT+0800 (China Standard Time)

This topic needs to be brought to another forum, but there is probably a more appropriate long term place to store these “well known” media types (OCI?)

…

On Tue, May 14, 2019 at 1:33 AM Gareth Rushgrove ***@***.***> wrote: I've submitted the top level media type to the ACR registry Azure/acr#225 <Azure/acr#225>. I'll close this issue when that gets merged. Thanks everyone. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1413?email_source=notifications&email_token=AADACFRF5YME7XDX6R6L6M3PVJMMPA5CNFSM4HLICY4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVKNBXQ#issuecomment-492097758>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADACFQVE6T57IHD4VYBJILPVJMMPANCNFSM4HLICY4A> .

Torin Sandall · Answer 13 · Sat May 25 2019 22:47:39 GMT+0800 (China Standard Time)

I'm going to close this since everyone is on board. If you think there is anything we should do in OPA around this, feel free to file more issues. Cheers!

Steve Lasker · Answer 14 · Fri Aug 09 2019 03:20:24 GMT+0800 (China Standard Time)

This topic needs to be brought to another forum, but there is probably a more appropriate long term place to store these “well known” media types (OCI?)
…
On Tue, May 14, 2019 at 1:33 AM Gareth Rushgrove @.***> wrote: I've submitted the top level media type to the ACR registry Azure/acr#225 <Azure/acr#225>. I'll close this issue when that gets merged. Thanks everyone. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1413?email_source=notifications&email_token=AADACFRF5YME7XDX6R6L6M3PVJMMPA5CNFSM4HLICY4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVKNBXQ#issuecomment-492097758>, or mute the thread https://github.com/notifications/unsubscribe-auth/AADACFQVE6T57IHD4VYBJILPVJMMPANCNFSM4HLICY4A .

Just commenting on the location for mediaTypes.
As we continue to work through the OCI TOB, we're proposing having well known artifacts submit their mediaTypes for an artifact using this type of format: artifactTypes
A description for how to author these is here: Registering well known artifact types

Steve