vmware-archive / kubecfg

It would be great if the jsonnet lib search path supported locating remote resources via HTTP. My use case involves keeping some jsonnet libraries in separate repositories in GitLab (where raw files can be accessed via HTTP).

While it is possible to download the libraries to disk and adjusting the search path, it requires wrapping kubecfg with custom code. It would be nice if kubecfg supported this directly.

Thoughts:

if there are any concerns about this breaking existing jsonnet files out there, it might be turned on only via a cmdline flag
could be implemented by overriding ksonnet VM's ImportCallback
- the callback would first delegate to the original callback, searching the filesystem
- then it could try GETing a jsonnet document from the webserver; url would be constructed by joining the search path url (beginning with http://) and the import path. If a non-success code is returned, nothing happens (except logging this), otherwise the response contents is converted to string and imported as jsonnet.

@anguslees Does this make sense to you? Would you accept a PR implementing this functionality?

I've been planning to add import-by-url for ages, and would ❤️ a PR that already implements it :)

I think we should just switch to always import by URL, with URLs relative to the current file's location. Then the initial import (from command line) should be a fully qualified file:///... path - and the rest should just fall out with no need to mix searching on the local filesystem as a special case. In a similar vein, don't try to intuit http vs https, just ask the user to provide correct URL schemes everywhere.

ie: In a file loaded as file:///path/bar.jsonnet:

import "foo.libsonnet";  // Will load file:///path/foo.libsonnet as per old behaviour
import "https://github.com/foo/bar/mylib.libsonnet"; // Will load https://.../mylib.libsonnet

If https://github.com/foo/bar/mylib.libsonnet contains import otherlib.libsonnet then that becomes https://github.com/foo/bar/otherlib.libsonnet, etc.

Keep it simple initially: just raise errors on HTTP errors. Eventually I think we will want to add an aggressive on-disk cache, so we can avoid network lookups (and work offline) for repeated invocations.

How does that sound?

I like the idea of unifying access by treating all import paths as URLs. This however means that relative filesystem paths given among search paths will need to be expanded to full paths, since you can't express relative paths using the file:/// scheme, but I guess that's not a problem.

Following your example of import otherlib.libsonnet being expanded to https://github.com/foo/bar/otherlib.libsonnet when imported from https://github.com/foo/bar/mylib.libsonnet - is this how imports work at the moment? Are they really relative to the file where these import statements appear?

I'd imagine the import behavior to be:

If the import statement path/url is absolute, then try loading it directly. If loading the resource fails, the import fails too.
If the import statement path/url is relative:
- first, try importing from URL derived from where the current file was found + import statement path.
- then iterate over all search "path" URLs and append the import statement path, taking the first one found.

@anguslees What are your thoughts?

I'm experiencing some issues with the jsonnet_cgos ImportCallback function (getting cgo argument has Go pointer to Go pointer error). Related to this - has switching to https://github.com/google/go-jsonnet ever been an option you've considered?

yes, cgo is fun :) go-jsonnet has only very recently become a viable replacement - and yes, I'd be happy to switch to it at this point.

I haven't used the jsonnet_cgo ImportCallback personally, but a colleague added the feature and uses it extensively without issues. If you have some wip code pushed somewhere, I'd be happy to take a look..

Wishlist: support http/s jsonnet search path