New resource interface (+ add versioning)

Question

New resource interface (+ add versioning)

vito opened this issue 8 years ago · comments

Alex Suraci commented 8 years ago

Proposed changes

Rename in and out scripts to get and put. These are confusingly named something different because a long time ago jobs just had _in_puts and _out_puts.
Run ./check, ./get, and ./put, and ./info (see next point) rather than /opt/resource/X. Having them in /opt is a bit Linux-specific and assumes containers have their own chroot, which is not the case on Darwin or Windows.
Add a info script which prints the resource's API version, e.g. {"version":"2.0"}. This will start at 2.0. If /info does not exist we'll execute today's resource interface behavior.
Change get to not emit the version or metadata. This was from when in didn't know what it was fetching until it was done. It should now be an error if it can't fetch the requested version. We'll also move all metadata collection to check (see point 8).
Add a delete action, which looks like put (can be given files and params to determine what to delete) but returns the set of versions that it deleted. This is to support bulk deletes, e.g. to garbage collect intermediate artifacts after a final build is shipped.
Remove the source and destination arguments passed to get and put - just make it the working directory.
Give check an official scratch space, which is the current working directory. No real semantics change here, just consistency with get and put, and potential for future behavior like reusing the scratch space but not reusing check containers for extended periods. Note: we've introduced /scratch recently so this change may just mean making that the work-dir.
Move all metadata emitting to check, so that it's always present. The original thought was that metadata collection may be expensive, but so far we haven't seen that to be the case.
Change put to emit an array of versions, rather than just one, and without metadata. Technically the git resource may push many commits, so this is necessary to track them all as outputs of a build. This could also support batch creation. Notably, the versions emitted by put are ordered chronologically, so that the dependent get knows to fetch the latest version. We would save them internally, along with an association with the build, and rely on check to determine the final ordering and metadata, being careful to not leave gaps in the version history (i.e. other commits pushed slightly before the put registered the build's).
Change put to write to a specified file, rather than stdout, so that we don't have to be attached to process its response. This is one of the few ways a build can error after the ATC reattaches (unexpected end of JSON). With it written to a file, we can just try to read the file when we re-attach after seeing that the process exited. This also frees up stdout/stderr for normal logging, which has been an occasional pitfall during resource development/debugging.
Remove the distinction between source and params; resources will receive a single config. The distinction will remain in the pipeline. This makes it easier to implement a resource without planning ahead for interesting dynamic vs. static usage patterns, and will get more powerful with #684.

Alex Suraci commented 8 years ago

woops

Alex Suraci · Answer 1 · Fri Jul 15 2016 01:55:06 GMT+0800 (China Standard Time)

Tagging a bunch of resource authors who would probably be interested in where the new resource interface goes. Pardon the spam if you're not interested.

@jtarchie @jchesterpivotal @robdimsdale @xoebus @cppforlife @drnic @thewoolleyman @xtreme-gavin-enns @zachgersh @ljfranklin @drnic

Dmitriy Kalinin · Answer 2 · Fri Jul 15 2016 02:32:22 GMT+0800 (China Standard Time)

info instead of version; version as a key in resp?

Sent from my iPhone

On Jul 14, 2016, at 10:55 AM, Alex Suraci notifications@github.com wrote:

Tagging a bunch of resource authors who would probably be interested in where the new resource interface goes. Pardon the spam if you're not interested.

@jtarchie @jchesterpivotal @robdimsdale @xoebus @cppforlife @drnic @thewoolleyman @xtreme-gavin-enns @zachgersh @ljfranklin @drnic

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Alex Suraci · Answer 3 · Fri Jul 15 2016 02:34:52 GMT+0800 (China Standard Time)

Yeah that's probably better long-term.

./info:

{ "api_version": "1.0" }

Not sure if we'd bother with actual versioning or just have it contain a number.

Jacques Chester · Answer 4 · Fri Jul 15 2016 05:05:59 GMT+0800 (China Standard Time)

Does api_version refer to the API of the resource or of Concourse?

Alex Suraci · Answer 5 · Fri Jul 15 2016 05:16:15 GMT+0800 (China Standard Time)

@jchesterpivotal the version of the resource interface that the resource conforms to. it'd be its own thing and probably not change much (this is our first big resource interface change since its inception)

Chad Woolley · Answer 6 · Sat Jul 16 2016 04:25:54 GMT+0800 (China Standard Time)

One thought on returning arrays of versions from different containers - I assume the concern about running on different containers is really about the array of versions being different or non-deterministic across multiple invocations to generate the array, whether or not it was run on the same container or not?

If so, the git-branches resource is a great example of something that is hard or maybe even impossible to make deterministic. It is just fetching a list of git refs at a given point in time, but git doesn't version refs. So, you just get whatever refs exist at the time you generated the version array, and a one may have been deleted then re-added since the last time it was generated, resulting in an version which is identical to a previous one, even though the ref may contain completely different code. That's why my implementation of git-branches resource has to include an independent increment value as part of the version, to ensure a new version is emitted even in these cases.

This may be what you are intending to solve with the git-branch-heads resource?

But nevertheless, it's still a problem that could exist with other types of resources people may come up with.

Also, when a resource version does not have an inherent monotonically increasing version component, it causes problems with sorting in the resources UI. For example, the current s3 resource with regex version has very bad sorting behavior in the UI. First of all, it forces the use of a semantic version, which we don't care about because we are tagging our "versions" with specifically-generated checksum SHAs (separate issue, but relevant. There's a good reason for doing this and not relying on concourse-generated versions, because the s3 resources are build artifacts which are looked up by the locally-generated checksum SHA outside of our Concourse system). So, we had to force it into a 0.0.0.abcdef format, where the 0.0.0. part never changes. Now that we have many hundreds of these, the latest resource emitted can be pages down in the resource view, e.g. for 0.0.0.aa... vs 0.0.0.ff....

So, given these concerns, would it be of value to require that every resource includes an monotonically increasing increment component to its version, to solve these types of issues? Just the existence of it would solve the first scenario of "identical looking but different" versions, e.g. deleted then re-added git branches. And If it were required to be an integer (which could either just be an increment or a timestamp), then the Concourse resource UI could default to using it as the sort field, avoiding the types of sorting issues we are seeing with only-alphanumeric-prerelease-component semantic versions with the S3 resource.

Thoughts?

Thanks,
-- Chad

Alex Suraci · Answer 7 · Sat Jul 16 2016 04:51:22 GMT+0800 (China Standard Time)

@thewoolleyman The only concern is that some resources (e.g. docker-image) need to spin up daemons and may not be coded today to support multiple scripts running in the same container. For example they both may try to spin up the daemon, and one will fail. We don't document the expected isolation of each script relative to each other.

The non-determinism is another valid concern: even in the git case, we'd need to at least know which final SHA it pushed (n), so that we know if we run check and find n-3, n-2, n-1, n, and n+1 (because someone else pushed immediately after), only n-3..n should be considered outputs of the build. This pushes me towards put returning the array of versions, as that's something it can know e.g. from the git push output that it just ran.

The git-branch-heads resource is not trying to solve anything about the git-branches resource, it's just a different use case, affected by the same non-linearity issue. That doesn't sound like something we can necessarily solve with the resource interface, though. Arguably if a ref is pushed and then un-pushed, it shouldn't be a new version anyway (in the case the same version was collected before the push that was later reverted).

The S3 example sounds like a misuse of the S3 resource. You should probably just use versioned_file or something instead, or at least put something increasing in there (unix timestamp?). There's literally no way to sort those in a reasonable way without coupling it to git. They're just strings.

I don't think any of that justifies requiring a monotonically increasing value. Idempotent updates are common and should not yield new versions. Resources are representing the content with the version, not just change over time; if something has gone forward and then back to the previous version, and everything is the same as it was, then it should be the same version.

Chad Woolley · Answer 8 · Sat Jul 16 2016 06:24:08 GMT+0800 (China Standard Time)

@vito I think that all makes sense, except for the claim of our misuse of the S3 resource. I probably didn't explain the use case well enough.

We cannot use versioned_file because the SHA component of the version is a _calculated checksum_ which uniquely identifies the gzipped artifact on S3. This checksum must be able to be calculated and used to download the artifact OUTSIDE of the Concourse system.

An example is our Rails fixture files generated via the fixture_builder gem. These take several minutes to build, but only need to ever be built once for every git SHA, because they are completely derived based off of the current schema.rb and test object mothers. So, the checksum (and name of the S3 published file) is based off these files from which the fixtures are derived

So we logically want to push this delay onto CI, and avoid incurring it locally every time a developer pulls a new SHA which needs a new version of fixtures.

Therefore, we have concourse build and publish them in a zip file named after the checksum, via the S3 resource (because that's what it's for, publishing files to S3). Then, on developer boxes, we only need to calculate the checksum based on the currently-checked-out files (which takes milliseconds), then download from s3 the file which matches that checksum (which takes only a second or two).

And we can't put a timestamp or other increasing component into the version, because there's no way for the non-concourse clients to know what that was in order to download the file matching their locally-generated checksum.

So, this isn't a misuse of the S3 artifact, because we are using it for the intended purpose: uploading concourse-generated files to S3.

The problem is (specific to the s3 resource case):

It is unnecessarily forcing us to use a semver format for the filename, when we really just want to use a calculated SHA. This in itself isn't a big problem, we worked around by using the 0.0.0.<calculated SHA pretending to be a prerelease version> approach. But...
This makes the sorting in the resource view unusable, because even though 0.0.0.aa... might be the most-recently-generated version, it may be buried several pages deep, while all the older 0.0.0.ff... versions are at the top of the list.
(update) and the lack of a URL to directly navigate to a specific resource version (and avoid going through pagination) only compounds the pain.

So, that brings us back to a the more general problem related to this issue, which I think does justify requiring a monotonically increasing value to force sorting in the case of resources which do not otherwise have an inherent sortable version component, e.g. our use of the S3 resource.

Alex Suraci · Answer 9 · Sat Jul 16 2016 06:57:00 GMT+0800 (China Standard Time)

@thewoolleyman If you're using the S3 resource without versioning (semver or versioned_file) you're misusing the S3 resource and should probably use or write a different resource. It does not make sense to add a monotonically increasing number to the resource, as it already has an established versioning scheme when used correctly.

Chad Woolley · Answer 10 · Sat Jul 16 2016 08:04:49 GMT+0800 (China Standard Time)

@vito OK, if you say wanting the S3 resource to support naming its published file based on an externally-derivable unique checksum is a misuse of it, and we shouldn't expect it to be chronologically sorted, then we clearly have to roll our own if we want sorting. Good to know.

But, how do the standard resources versions know how to properly sort themselves? E.g. the git resource, and an S3 resource used with versioned_file, which are both just random SHAs, just like our usage of the S3 resource with 0.0.0.SHA semvers. I assume this is somehow determined by some value in the internal Concourse database?

If so, is there some way we could ensure that this internal version is always used to provide chronological sorting, or a way to allow it to be used if the default sorting isn't useful (as is the case with our S3 usage)?

And it also seems like exposing this would be the natural unique ID for a URL pointing directly to resource versions (which would be useful in debugging issues in a pipeline, e.g. clicking a resource version to see what upstream job builds generated it, without having to manually search for it in the resource view, possibly through many paginations if there is no way to chronologically sort them).

-- Chad

Alex Suraci · Answer 11 · Sat Jul 16 2016 08:34:00 GMT+0800 (China Standard Time)

@thewoolleyman The git resource knows their order because in git, SHAs have an order to them (i.e. git log). This is how every resource works; the s3 resource knows their ordering because either it's via versioned_file (inherently ordered) or semver (order determined by filenames).

Chad Woolley · Answer 12 · Sat Jul 16 2016 09:32:18 GMT+0800 (China Standard Time)

@vito But the resources web page (/pipelines/PIPELINE_NAME/resources/RESOURCE_NAME) knows this "order" based off a value stored in the concourse database, but it isn't always "inherent" by my definition of "inherent".

I believe the ordering is based on "check_order" in the DB, as shown here:

https://github.com/concourse/atc/blob/master/db/pipeline_db_test.go#L590
https://github.com/concourse/atc/blob/master/db/pipeline_db.go#L480
https://github.com/concourse/atc/blob/master/db/pipeline_db.go#L921

Looking at the last link, incrementCheckOrderWhenNewerVersion, I'm guessing the meaning of "Newer" or "CheckOrder" is implemented differently for different resources, or different usages of different resources? It can't just always be the chronological order in which they were emitted, otherwise our S3 semver resources wouldn't sort the way they do (which is by semantic version order).

In the case of an S3 resource with Semver ordering, there IS obviously an "inherent" ordering (Semver), which is observed by the Resources UI.

But in the case of a Git resource SHA or versioned_file JSON version object, there IS NOT an inherent ordering, at least not to anything displayed in the Resources UI (it's just a random SHA). Yet, the latest one is always at the top in chronological order. Presumably this is because the original API request ordered it based off the check_order in the database, which was chronological.

So, relevant to this issue, there's other resource "versions" which also do not have an "inherent" ordering in their actual JSON version object, e.g. the git-branches resource, the proposed get-branch-heads resource.

But specific to the S3 resource, looking at it that way, our requirement is not a "misuse" of the S3 resource. It just a feature request for a combination of different existing behaviors. Specifically:

We want a random SHA as the name of the S3 file (obviously possible since that's exactly what the name of a versioned_file S3 resource file is),
...but where we control the filename (obviously possible because that's what we're doing with the regexp S3 resource option, except we are unnecessarily required to make it conform to a semver format, even though we don't care about semantic versioning of the artifact)
...and we want it to be sorted in chronological order even though it has a random SHA as the name of the S3 file (obviously possible since that's exactly what the versioned_file S3 resource option does.)

So, you can state that you don't want the official S3 resource to support that particular combination of obviously-possible existing behaviors, and I'm glad to fork and implement it if our team cares enough. But it's definitely not a "misuse", it's a legitimate use case for what we want to do.

Anyway, I'm glad to take this to a separate issue on the s3 resource, if you don't think this discussion is relevant to this issue about changes to the resource/version interface.

Alex Suraci · Answer 13 · Sat Jul 16 2016 10:05:40 GMT+0800 (China Standard Time)

@thewoolleyman The order in the database comes from check. It's not arbitrary. Git resource shas and versioned_files absolutely do have an inherent ordering.

Your current usage is a misuse of the S3 resource as-is. What you're requesting is a very different resource. What you define as 'chronological order' is impossible to determine after-the-fact; looking at the bucket and ordering the filenames is impossible for the S3 resource, as it depends on knowledge from the git repo, which the S3 resource knows nothing about. Being able to discover older versions of something is supported by resources that properly implement the full interface, which would be impossible in this case.

Resources that do not have an inherent ordering should decide, at their own discretion, whether they need an increment value. It is certainly not a property of all resources.

At this point the topic is getting derailed and it feels like this should be taken elsewhere. There are many more parties that should be able to contribute to this discussion.

Chad Woolley · Answer 14 · Mon Jul 18 2016 23:16:59 GMT+0800 (China Standard Time)

@vito The specifics to the S3 resource may be unrelated to this issue, as I admitted.

However, the concerns I have raised about how to properly order resource versions in the UI, and address them directly via URI, even if they do not conform to the current implementations of resources or versioning in concourse, is in my opinion, very relevant to an issue which is intended to discusses proposed changes to how resources and versioning are handled.

And our use case (leveraging Concourse to publish a resource externally while using externally determined versions) is completely valid and necessary, as I described in detail, and it's reasonable to expect a decent UX, and as I said, our problems are relevant to other use cases people may come up with (proven by our discovery of the need to manually add an "increment" value to the git-branches, which also does not have an inherent sorting).

Therefore, we may be "misusing" the existing S3 resource, that's a valid statement.

But dismissing and ignoring our concerns over an unusable UX which could potentially be addressed by changes in how versioning is handled is not valid. Isn't an issue discussing breaking changes to how versioning is handled is exactly the place to discuss that? These concerns don't have to be addressed in this issue's proposed changes if you deem them out of scope, but they definitely merit consideration for a future potential feature request, rather than being dismissed outright as a "misuse".

Anyway, having said that, I'll drop the point.

Alex Suraci · Answer 15 · Mon Jul 18 2016 23:46:13 GMT+0800 (China Standard Time)

@thewoolleyman I've not dismissed or ignored your concerns. From the first comment I've been addressing them, while trying to push unrelated things that cloud the discussion aside. You've been leaving very long comments that make a lot of assumptions which makes the discussion more difficult than it would be if we both got to where we want in smaller steps. It makes it frustrating for both of us.

One of those assumptions was the behavior of the S3 resource, and then later the git resource and others. As I've said previously, the ordering of versions is determined entirely by the resource, and if a resource is correctly implemented, it returns versions in the correct order, and the UI and everything else respects that. If there is no inherent order, then the resource may want to rely on an increment value, but even that is unrelated to ordering - a resource would only do that to force the presence of a new version in order to trigger new builds (as is the case for the git-branches resource). In some cases you won't actually want that, e.g. if I do a rollback deploy with BOSH to something I've deployed previously, I'd expect to just get the previous version again. So, it doesn't feel like something common to all resources, and resources that need it already have a way to do it.

Alex Suraci · Answer 16 · Thu Jul 28 2016 01:01:01 GMT+0800 (China Standard Time)

Another thing to think about is whether the distinction between source and params is worth keeping. There are many times where a resource has its config hardcoded in source, but then people want to do things dynamically, e.g.:

vmware-archive/bosh-deployment-resource#13 - dynamic deployment name
vmware-archive/bosh-deployment-resource#6 - dynamic target address
cloudfoundry-community/cf-resource#20 - dynamic space name
https://github.com/robdimsdale/concourse-pipeline-resource - which supports making basically the entire config dynamic
https://github.com/concourse/docker-image-resource#out-push-an-image-or-build-and-push-a-dockerfile - which supports loading a tag from a file, effectively making the tag in the source useless

I've pushed back on some of these because it feels wrong to have every resource support both static and dynamic configuration, and have every resource implement it themselves, differently each time.

I think that the current source and params terminology is meaningful for the pipeline narrative. Each resource should have a source so the pipeline knows where it's coming from. And a put or a get should configure params if they want to configure how the operation is performed. One subtle distinction is that put params can be dynamic (i.e. read from a file) but get params cannot.

My end goal is that each resource simply receives a bag of static JSON config, with no indirection to deal with (no credential fetching, no resolving values to file contents, etc), and does its thing.

So, here's one idea:

For the resource interface, merge source and params into just config.

In the pipeline, support explicitly loading config from files in previous steps, e.g.:

- get: some-file
- put: my-image
  params: {tag: {load: some-file/version}}

We could even support loading JSON values like so:

- get: branches
- task: generate-pipelines
  file: # ...
- put: my-pipelines
  params: {pipelines: {load: generated-pipelines/pipelines.json, format: json}}

Interestingly, this allows get to also be dynamically configured, as the file loading occurs in Concourse, bringing the cache identifier to the forefront.

@robdimsdale @cppforlife @xoebus thoughts?

Sam Bryant · Answer 17 · Thu Jul 28 2016 01:41:38 GMT+0800 (China Standard Time)

I like the sound of merging source and params, one query I have as a user of custom resources.

How would backwards compatibility be manged?
It wouldn't be the cleanest user experience to have to declare one resource_type version for new concourse releases and different ones for the older concourse releases? We don't have the luxury of being packaged with the concourse releases.

Alex Suraci · Answer 18 · Thu Jul 28 2016 01:56:48 GMT+0800 (China Standard Time)

@srbry Sounds like you're asking the general question of how do resources migrate to this new interface we're planning?

I think it's just a matter of using semver for your custom resource types via tags. If you're supporting both for an interim period, then that can just be a minor bump. Once you cut over to the new resource interface (which I would recommend we do aggressively), just do a major bump. Folks using custom resource types should probably be pinning the version down in their pipelines already. (Though it would be nice to have versioning semantics, i.e. tag: 1.x. - which we could maybe add to the docker image resource.)

Alex Suraci · Answer 19 · Thu Jul 28 2016 01:57:51 GMT+0800 (China Standard Time)

@srbry We could also potentially support having a generic shim that translates the old interface usage to the new interface.

Sam Bryant · Answer 20 · Thu Jul 28 2016 02:05:14 GMT+0800 (China Standard Time)

@vito thanks, sounds simple enough

Rob Dimsdale-Zucker · Answer 21 · Thu Jul 28 2016 12:37:32 GMT+0800 (China Standard Time)

@vito I like the idea of pushing up the dynamic config parsing and just returning a single bag of config. Especially if it allows dynamic configuration on get which works with caching. That's delightful. Digging into the implementation a little, would you only support one file, or multiple? If multiple, what's the precedence for the same value in multiple files? Last-most wins?

Alex Suraci · Answer 22 · Thu Jul 28 2016 12:47:55 GMT+0800 (China Standard Time)

@robdimsdale Multiple; there wouldn't really be a precedence concern as the load syntax (which may look like (( )) come to think of it) would just refer to the step + filename, same as file: on a task. There could be an arbitrary number of those arbitrarily deep in params (or one at the toplevel).

Rob Dimsdale-Zucker · Answer 23 · Thu Jul 28 2016 22:34:54 GMT+0800 (China Standard Time)

Ok thanks @vito. I want to make sure I understand what you're saying, so taking your earlier example:

- get: branches
- task: generate-pipelines
  file: # ...
- put: my-pipelines
  params: {pipelines: {load: generated-pipelines/pipelines.json, format: json}}

instead it might look like:

- get: branches
- task: generate-pipelines
  file: # ...
- put: my-pipelines
  params: {pipelines: (( load: generated-pipelines/pipelines.json, format: json} ))

Independent of where the (( )) go, your point about precedence is that load: will only ever take one file for one key. So there won't be a way to merge multiple load: files into a single bag of params, e.g.:

- get: branches
- task: generate-pipelines
  file: # ...
- put: my-pipelines
  params: {load: somefile, load: someotherfile}}

where somefile provides keys a: foo,b: [bar] and someotherfile provides b: [baz], c: moo with the resultant params being: a: foo, b: [bar,baz], c: foo.

To be clear, I think your proposal is fine, and my clarifications are overcomplicated - I just want to make sure I understand your thinking.

Alex Suraci · Answer 24 · Thu Jul 28 2016 23:06:44 GMT+0800 (China Standard Time)

@robdimsdale Yep. It'd behave very similarly to how we're planning to use (( )) for dynamically loading credentials, which is why I'm tempted to unify the syntax. The format key makes it slightly more complicated though, and I don't want to invent another YAML grammar, so I'm on the fence.

No merging would be supported either way, at least initially.

Alex Suraci · Answer 25 · Thu Jul 28 2016 23:08:21 GMT+0800 (China Standard Time)

Update on the "Should we change how we pass the source/destination directory to resources?" question: I think we'll just make it the working directory, and also formalize this for check so that it can know to use the current working directory as a "scratch space" (e.g. the persistent repo in the git resource so it can optimally pull instead of doing a fresh clone every time).

Rob Dimsdale-Zucker · Answer 26 · Thu Jul 28 2016 23:15:02 GMT+0800 (China Standard Time)

On the subject of passing source/destination to resources, will caching apply to in and out as well as check? How will this work with the output being the overwritten input?

Alex Suraci · Answer 27 · Thu Jul 28 2016 23:16:02 GMT+0800 (China Standard Time)

No, that won't change from how things are today, we're only changing how they're told the directory.

Alex Suraci · Answer 28 · Sat Jul 30 2016 02:23:32 GMT+0800 (China Standard Time)

Cross-referencing #292 and #300 regarding #534 (comment) - I'm now starting to think of it as a feature enabled for all aspects of a build plan; not resource-specific.

There are some super interesting implications to that.

Example for #292:

- task: generate-matrix
- aggregate: {load: generated-matrix/steps.yml}

Example for #300:

- put: workers
  params: {acquire: true}
- task: some-task
  tags: [{load: workers/name}]

Alex Suraci · Answer 29 · Sat Jul 30 2016 09:35:22 GMT+0800 (China Standard Time)

Issue #210 could also be doable with that generic load feature.

Now that we have version pinning on the get step, you could do something like:

- get: valid-versions
  trigger: true
- aggregate:
  - get: concourse
    version: {load: valid-versions/concourse.json}
  - get: garden-runc
    version: {load: valid-versions/garden-runc.json}

Alex Suraci · Answer 30 · Thu Sep 29 2016 08:56:31 GMT+0800 (China Standard Time)

Meta comment: we should probably have a different workflow for proposals like this. It's awkward editing the original issue without any history of the changes. I'm thinking GitHub's new PR review system might actually work out pretty well; I could PR to a docs/design notes repo and y'all could review them (and even block merging until things are resolved...though that could backfire if people forget to go back and re-approve. hm.).

Anyway.

Latest decisions

Force all metadata to come from `check`.

This way we have all metadata as soon as the version is found. No one brought up any cases where collecting the metadata is too expensive, so this should be OK.

Change `put` to return the set of versions (without metadata).

These will be saved internally and correlated to the build, and we'll rely on check to actually produce the metadata and ordering.

This raises a couple questions however:

When to run check. Run it immediately? In a separate container (reusing the existing check container)?
What does this mean for "every put implies a get"? Just fetch the latest version of the set? In which case we actually will have to make sure it's ordered, so it's not really a set.

New things to consider

Should we have one executable with 3 subcommands instead of 3 executables?

A few resources employ a strategy of having a single executable that's symlinked 3 times and switches on ARGV[0] to know what to do.

This raises a couple questions:

What about discovering the version? That should probably remain a separate command; I don't want everyone implementing option parsing.
Risk of creeping towards full-blown option parsing.
What are the benefits to this? Smaller images thanks to not shipping 3 binaries? Shared setup code? Any others?

JT A. · Answer 31 · Thu Sep 29 2016 10:26:25 GMT+0800 (China Standard Time)

Force all metadata to come from check.

@vito the meta data used in the PR resource is the current state of PR -- failure, pending, etc. That is done on the put. Maybe it is not useful meta information at that point.

Should we have one executable with 3 subcommands instead of 3 executables?

Yes, please. This would make it possible to have a single binary that does all the things.

Rob Dimsdale-Zucker · Answer 32 · Thu Sep 29 2016 12:56:11 GMT+0800 (China Standard Time)

Force metadata to come from check

For the resources I maintain, this is almost trivial and doesn't also provide a lot of value. I'm slightly in favor as it will reduce some complexity about where in the lifecycle metadata is available.

Should we have one executable with 3 subcommands instead of 3 executables?

I have no strong opinion on this. The resources I maintain already do somewhat complex operations like validation, so option parsing doesn't add a lot of complexity. On the other hand, what does this improve? The images might be ~10MB instead of ~30MB, but that doesn't seem particularly valuable.

Put creating many versions

This doesn't impact me a lot - for the resources I maintain it's not a lot of effort to implement, but also doesn't provide a lot of value. I do see the potential to simplify the case where teams have multiple pipelines in parallel for different versions of a resource, especially if the put for each individual resource is idempotent (i.e. if I re-rerun with a new input I might create one new resource version, update one existing resource version, and leave others as they are).

Alex Suraci · Answer 33 · Fri Sep 30 2016 07:47:20 GMT+0800 (China Standard Time)

@robdimsdale ~10MB instead of ~30MB adds up pretty quickly when you multiply it by the number of core resources Concourse ships with. The other benefits are shared setup code and potentially simpler distribution and (manual) testing.

@jtarchie Since your resource already emits everything available from check (with every: true), that would make it so that the metadata is always up-to-date, rather than relying on get and put.

Alex Suraci · Answer 34 · Sun Oct 02 2016 02:27:53 GMT+0800 (China Standard Time)

One interesting thing about moving all the metadata to check is that it will break any resources that don't have an external "source of truth" for their versions.

Here's how get after put would probably have to work:

put returns versions, in any order, without metadata
after put, we run check, which will fill in the metadata and ordering
save each of these versions as outputs of the build
then we get the latest version that check returned that was present in the set returned by put

This guarantees that all resource versions come from one place: check. I like this because it removes the weirdness of put directly messing with the version history, and potentially causing versions to be skipped. But it will break some resources. Granted, the resources that are broken by this would be ones that kind of "cheat" the resource model anyway. If your check returns no versions, there's no source of truth, so it's not really a resource.

For example, notification-style resources like the Slack resource tend to have a no-op check that yields no versions at all. The Slack resource could probably be fleshed out a bit, though; in principle check could check the message history and return messages emitted by the resource. This could also be useful for triggering things from Slack.

Another casualty of this change would be the Time resource's put action, which generates a new timestamp on-the-fly. But this has always been weird (you still have to configure an interval), so I don't feel too bad about it.

John Locke · Answer 35 · Sun Oct 02 2016 05:12:16 GMT+0800 (China Standard Time)

This is interesting; my custom Matrix Notification Resource is modeled on Slack, but Matrix supports storing arbitrary state in rooms. I've been trying to think of how this might be an effective place to store very brief, arbitrary strings or state identifiers, for example, that might better fit into what a resource represents. And possibly act as a trigger...

The question is, what makes sense to return as the version?

Instead of using the Matrix resource to store/retrieve state directly, I've ended up passing a lot of metadata into Matrix using an event rather than a state -- and then I have a separate bot that moves the data from the event into room state for later retrieval. And then the same bot triggers Concourse jobs using fly. I needed to use this approach anyway, because we have so many pipelines we turn them off when not in use -- we were having trouble with more than 5 or 6 pipelines active (and we have ~40 of them in all) so having the bot enable/disable pipelines as various projects needed them has worked very well -- but it would be fairly easy to configure a Matrix notification resource to store state in a room and retrieve it in a version check -- though it does require escalating the power level of the account in Matrix to be able to set state.

Long-winded way of arriving at the question -- what should something like the Slack resource return from a check, if there isn't some state being preserved in the resource? Is a notification resource really a "resource" at all, or just an event listener? If a time resource is really just a trigger, and not something you would ever put, should it really be a resource? If an event listener doesn't really have any state for a get, should it really be a resource?

You're right, these don't fit your model of a resource -- so it seems to me, either the conception of the resource needs to change to allow listen-only or trigger-only resources, or we need some other structure to support those...

Chad Woolley · Answer 36 · Tue Oct 04 2016 00:54:07 GMT+0800 (China Standard Time)

And to add on to @freelock 's point - as I explained at length above, there's also a need for "resources" which can generate artifact "versions" based on an algorithm such as checksumming which 1) can be externally generated and referenced based on the same algorithm 2) are not forced to fit into constraints of being monotonically increasing or a semantic version format.

Another casualty of this change would be the Time resource's put action, which generates a new timestamp on-the-fly. But this has always been weird (you still have to configure an interval), so I don't feel too bad about it.

I'm not sure what you mean by "weird", and I personally would feel bad and concerned about breaking existing resources and use cases without full understanding and careful consideration.

So, based on my examples and @FreeLocks : I think there's several valid existing use cases that don't fit into the current paradigm of what a "resource" and "version" should be, and I'm not sure they are fully addressed by the proposals as they currently stand (maybe they are, but at least it has not explicitly explained how they are addressed/not broken).

Alex Suraci · Answer 37 · Tue Oct 04 2016 01:15:38 GMT+0800 (China Standard Time)

@freelock Yeah, the same thought has occurred to me in the past. Usually though, after thinking about each resource that appears to only partially implement the interface, there is natural and even useful behavior to be attained by filling out the whole thing.

Notification resources sound like they fit the model just fine: it's perfectly reasonable to want to both listen for (check) and emit (put) events, and collect metadata about the event (get), which could be used for parameterized events or messages (e.g. "destroy [environment-name]"). This applies to Slack, DataDog, etc.

The Slack resource could emit a message timestamp of some sort from check, and collect metadata from the message from get. This could be used to drive a pipeline from the channel with parameterized messages. See https://api.slack.com/events/message and https://api.slack.com/methods/channels.history for more info. This would also become more useful once web hooks (#331) land.

Alex Suraci · Answer 38 · Sat Oct 15 2016 06:11:04 GMT+0800 (China Standard Time)

Another potential amendment: we may want to just run ./check, ./in, and ./out rather than /opt/resource/X. This would better support Garden backends that don't support per-container rootfses; they could just be set of scripts. It's also oddly platform specific (/opt being a Linuxy thing).

Lyle Franklin · Answer 39 · Thu May 25 2017 22:58:00 GMT+0800 (China Standard Time)

@vito any rough estimates on timeline for this change? I've got some breaking changes I was thinking of making to Terraform resource down the line, would be nice to lump it in with the API changes described above.

Alex Suraci · Answer 40 · Sat Jun 10 2017 22:10:55 GMT+0800 (China Standard Time)

@ljfranklin It's currently scheduled for v3.3.0, but I'd place a conservative estimate as a month, as we're taking things a bit slower now and doing more refactoring now that Life and a couple big features have shipped.

Being scheduled for v3.3.0 is a big improvement over its priority before, at least.

Ethan Edwards · Answer 41 · Mon Oct 02 2017 21:48:10 GMT+0800 (China Standard Time)

Is this still going to be targeted in 3.x?

Evan Farrar · Answer 42 · Thu Nov 30 2017 14:52:08 GMT+0800 (China Standard Time)

👍 for delete (can't add reactions on mobile GitHub)

John Wells · Answer 43 · Sat Feb 03 2018 08:33:05 GMT+0800 (China Standard Time)

Is this still on the roadmap?

Alex Suraci · Answer 44 · Sat Feb 03 2018 09:57:38 GMT+0800 (China Standard Time)

@madmod There's a chance we tackle it soon (maybe in part) as we're working on spaces which also implies a change to the resource interface.

Benjamin P. Jung · Answer 45 · Fri Mar 02 2018 20:40:40 GMT+0800 (China Standard Time)

I was thinking about a new resource interface for quite some time now and I wanted to throw in a bunch ideas that came into my mind:

Docker tags

Since all resource_types are (right now?) basically docker images, it might make sense to use Docker labels to attach some metadata to docker images that can be used as Concourse resources. This would open up a few possibilities, e.g. having a "Concourse Resource Registry" that can be used to automatically list resources

Possible tags:

org.concourse-ci.resource.version=2
org.concourse-ci.resource.actions=check,in,out (or check, get, put delete, whatever... in the future...)
org.concourse-ci.resource.description="A dummy resource that does nothing"
org.concourse-ci.resource.documentation=[link to documentation]
org.concourse-ci.resource.author="John Doe <j.doe@example.com>

JSON schema

An embedded schema that is part of any resource that can be used to validate the source- and params- configuration would be really neat. I think that JSON schema might be used to do that job quite nicely (http://json-schema.org/)

Advantages:

JSON schema can be used to document the resource configuration parameters
JSON schema can be used by Editor/IDE plugins to perform validation (mandatory vs. optional fields, regex and field type constraints, etc., etc...) and show useful hints while editing (the current VSCode Concourse plugin only shows hints and tips for well-known resources)
JSON schema is a pretty solid tech and implementations for all major programming languages (including Go) seem to exist.
JSON schema can be used to extract the documentation for a resource

Since Docker label values can contain any arbitrary text of any length, it might be a possibility to embed the schema(s) into the resource docker images by associating them with well known tag values (but I think that might be a bit complicated)

Just my 2 cents.

Jacques Chester · Answer 46 · Fri Mar 23 2018 00:11:02 GMT+0800 (China Standard Time)

I'm very strongly in favour of schema support as a way to improve validations and tooling.

eve n.u. · Answer 47 · Fri Mar 30 2018 07:03:13 GMT+0800 (China Standard Time)

@vito @robdimsdale you could use the YAML native << append syntax to get multiple top-level objects loaded, right? (but you would be subject to the native merging rules, which yous probably know more intimately than I):

- put: my-pipelines
  params:
    <<: ((load: somefile))
    <<: ((load: someotherfile))

and if needed can define your own merging logic by adding a shim task:

- task: merge-my-params
  file: merge/the/env.sh
  env:
    FILE_1: "((load: somefile))"
    ...

limitation would be now on variable number of files to load.

eve n.u. · Answer 48 · Fri Mar 30 2018 07:52:23 GMT+0800 (China Standard Time)

@vito @robdimsdale regarding the use of source vs params, I like the idea of where/how you specify them being decoupled from the API, so that you can put a param in the resource definition and always inherit. however that issue seems separate from the issue of what a resource author can do with an opaque config style API.

to that effect, as far as caching goes two things continue to bother me:

specific keys on some resource type configs would still be useful but not correspond to whether bits are cache-worthy. In particular, there are all kinds of reasons why a get with different params could have identical bits, such as auth info, or its contents be inferable from another cache, such as network optimization params.
Similarly, put params have often been used to relate the put to previous steps in the build, which we will still need to do but which would change the config, but conceivably not its cache identifier.

as such, the concept of static vs dynamic configuration seems very important to the resource type author, because some of the keys may substantially change what versions appear in the history.

concretely: if you have a resource for a git repo, it would be intuitive under this new design to push paths and ignore_paths into the get params rather than defining a new resource for every directory that only relates to a few of your builds. But now I have a problem: the resource is producing hella versions, but only some of them relate to the different builds; I therefore have to provide some suspect filtering power for the ignore_paths to be able to be respected as far as triggering is concerned. How do we save the resource author from having to distinguish between static & dynamic in this case? @headcr4sh 's schema plan seems like a viable option but i haven't got a solution in mind that I like.

Alex Suraci · Answer 49 · Tue Apr 03 2018 03:21:24 GMT+0800 (China Standard Time)

@jraqula The caching question is extremely tricky. Changing credentials may not affect the contents of the cache, but having the credentials be verified should be a precursor to anyone's ability to use the cache. If they aren't part of the key, and we don't perform any additional verification (there's no resource interface for this), we've got a very serious information leak on our hands (credentials effectively become optional after the first person fetches it)!

Re: paths/ignore_paths, that whole approach is pretty flawed at the moment. It would probably be better to have some way to determine trigger: true dynamically based on some sort of metadata attached to each version. That's quite a complicated topic on its own. In any case, the current properties only have meaning to check, since they influence what versions it collects in the first place. Specifying them on get steps would be too late. Remember: the only thing we check is what they've configured under resources:, so the only config available will be whatever's under source.

eve n.u. · Answer 50 · Wed Apr 04 2018 09:12:48 GMT+0800 (China Standard Time)

@vito secondary concern then while I stew over that: params: {force: true} on a put: my-git-resource. That seems like an important thing to be able to do idempotently, but i feel like genuinely would constitute a delete as well -- there may now be spurious SHAs in the version history.

Easy to imagine this sort of idempotent "change" that is neither append-only nor redact-only applying to other resources generally.

As such what would we feel about instead of adding delete and instead (or having delete be a synonym in some way) allow put to specify a delta which includes removed versions in the history?

very basic strategy would be use a context-sensitive diffpatch like diff but without requiring line numbers or their equivalent (as currently put does not have to be aware of the entire history of versions to report the change), which would be possible as long as version strings are required to be unique?

Jacques Chester · Answer 51 · Wed Apr 04 2018 22:14:11 GMT+0800 (China Standard Time)

A put with a delta might be better called patch, or am I missing something obvious?

Alex Suraci · Answer 52 · Wed Apr 04 2018 22:31:34 GMT+0800 (China Standard Time)

@jraqula I can see the appeal of having put and delete be one interface since they both have just about the same signature:

put    : Config -> Bits -> Set Version
delete : Config -> Bits -> Set Version

We could change it to explicitly return a response object saying "here are the versions I created", "here are the versions I deleted".

Devil's advocate, though - version removal could be discovered by check, maybe even more easily. If check is given a version it doesn't recognize (in the event of a push -f), we could have it return the new latest version and the version preceding it. Concourse would then consider any versions in-between as deleted. We already converge on the return value of check for the version ordering, so it kind of follows that we would have to deal with this type of response value anyway. I suspect this would be easier than teaching the git resource's put action to also know how to detect the versions that it destroyed.

Side note: I'm going to close this issue and make a PR to https://github.com/concourse/rfcs (new repo) instead. This will allow more structure for the discussion. It'll be much easier to follow than a long, gigantic linear discussion with no history as to when the initial issue's proposal has been changed. I'm stealing this pattern from Rust and others.

Alex Suraci · Answer 53 · Wed Apr 11 2018 04:47:52 GMT+0800 (China Standard Time)

Closing and moving the discussion to concourse/rfcs#1. Thanks everyone who has helped out so far!

New resource interface (+ add versioning)

Proposed changes

Latest decisions

Force all metadata to come from check.

Change put to return the set of versions (without metadata).

New things to consider

Should we have one executable with 3 subcommands instead of 3 executables?

Docker tags

JSON schema

Force all metadata to come from `check`.

Change `put` to return the set of versions (without metadata).