Applying CSVW transformations to parameters

[Rick Moynihan]

Lifting this issue out of slack, for more general discussion/clarification. It may or may not be a problem.

Generally is CSVW perhaps best understood as a method for describing things after the fact, and therefore it’s almost purely declarative. It’s not really about processing, so there’s no notion of process or any real notion of inputs/outputs or parameterisable arguments etc.

It seems like it might be best thought of as a markup format for google to understand CSV files rather than a method to transform data. Obviously at some level they're the same thing, but specific areas where this distinction is perhaps important are that processes can be reused with different inputs.

Yet CSVW appears to have no notion of how you might parameterise a CSVW transformation process.

How do "process ideas" such as parameters/arguments/environments/contexts apply here?

[Robin Gower]

So the particular problem is that source documents (and transformation definition) are specified as URLs and implementers are instructed to retrieve e.g. the data.csv from that URL, or to find the metadata alongside the csv file.

We operate in a different context, we want to feed csv/ json files into a pipeline - e.g. by passing filenames at the cli or contents in an HTTP body. In other words the csvw authors want to put csv on the web, we want to put csv into pipelines!

I believe we could follow the spirit of the spec by hosting the prepared csv and json metadata at a given URL, then instructing the csv2rdf step to retrieve it (note that you're supposed to be able to find one from the other by convention and (?) because the json metadata contains a url reference to the csv).

In practice we might want to ignore this part of the spec, such that the url parameter is just an identifier for the source (that mightn't resolve) and make our csv2rdf implementation expect csv and json as input parameters as we have with other grafter pipelines.

[Rick Moynihan]

Thanks for expanding on the specific problem @Robsteranium, that is indeed what I was referring to.

I should also point out that searching the CSVW working groups motivating use cases none of them come from the perspective of transformation.

I think you're right that we're operating in a different context, and there is stuff we should do to reconcile it. Some strategies are:

A: Build the declarative CSVW world

As I think @Robsteranium is suggesting an option is to have a prep phase which creates real world side effects to ensure the world is in a state that agrees with the view of the tabular-metadata file. When run in a remote context this might involve:

1. In a real world context this involves upload the input files to a URL (we should probably also use a hash of file contents in the URL string to ensure idempotency e.g. https://csvinputs.s3.amazonaws.com/0d599f0ec05c3bda8c3b8a68c32a1b47.csv (or in a test context this just creates the file locally on your filesystem possibly with a hash and sets url to the file path).
2. Execute the transformation and build out all the target URIs, observations etc inside your draftset. Essentially we materialise the implications of the csvw jsonld metadata document into the RDF world.
3. For completeness we could store the jsonld somewhere too and link it in with Link headers on CSV downloads etc and whatever the spec suggests. So CSVW readers can build an RDF view of our CSV). (Though personally I think this is probably pointless as you can more easily just consume the RDF.)

How would such an approach handle updates / revisions and appends of new data? It's not clear to me how this should be represented. Do you use a csvw:TableGroup and append new tables for each new quarter?

Additional difficulties with this approach are considerations about how this might work with draftsets. In a perfect world the csv input file would be inside the draftset and downloadable too. Except we probably want to store it on S3, so will need to munge permissions on draftset state changes / publication etc.

B: We identify paramaterisable things in the spec and handle them as custom extensions

There are many ways we could do this, some of which might keep within the spec. One way that strays out of the spec might be to replace "tables" with a new extension called "arguments" which is a set of named input arguments providable via environment/command-line/web-form.

[Ric Roberts]

Yeah, there's a 1:1 (lossless?) correspondence between the csvw and the rdf. If we have rdf, you should be able to offer it for download as csvw. And if you have csv we get get the rdf triples that represent it.

So a client-pipeline would go from our agreed interchange format (for that client), to the csvw input format, and then run our csvw.clj implementation