robertlemmen / svc-shelve6

An Artifact Repository for Raku and Friends

Shelve6

... A artifact repository for the raku language and friends

The goal is to build a artifact repository service that raku modules, but also other stuff, can be pushed to and pulled via the usual means (i.e. pushed from CI or manual workflows, pulled via zef). This would be useful for an organisation doing raku development, does not want all their code publicly available, yet want to use a regular module-centric, tarball/release-based development flow.

Essentially this is a "content storage" service as described in S22

Features

• Upload Raku modules
• Fetch with zef
• Authentication
• Multiple configurable logical repos

Upcoming and ToDo

• cucumis sextus tests

• adress already in use error handling/reporting

• UI to browse and manage
• API to manage and automate
• Local cache/proxy for other repositories, like CPAN. Could be just a cache, or a fetch-ahead full copy. Perhaps both, configurably.
• Rarification/expiry of artifacts in configured repositories
• Web hooks for automation
• Verification and other plugins
• Shared file store, multiple shelve6 instances. Or a database as a store.
• More auth types
• Full-blown monitoring, resilience etc
• More metadata, like when uploaded and by whom. "on-behalf" in upload script so that a CI or automation job can say on whose command they uploaded

Also grep for the XXX fixmes in the code!

Usage

Shelve6 comes as a web service that you can just start e.g. directly from the checked-out source repository via RAKUDOLIB=lib bin/shelve6, or if it is properly installed just via shelve6. It reads a config.yaml file, a simple sample is included, and might look like this:

    server:
        port: 8080
        base-url: "http://localhost:8080"
    store:
        basedir: store
    repositories:
        - name: p6repo

• base-url is where you want the service to be found externally
• port is of course the port the service listens on, note that it currently only binds to the first localhost interface, let me know if that gives you grief.
• basedir is a directory where shleve6 will store the artifacts
• and the repositories is a list of logical artifact repositories in which you can store modules

With the service running, you can use the supplied shelve6-upload script to put artifacts into shelve6:

    bin/shelve6-upload raku-foo-bar-0.1.tar.gz http://localhost:8080/repos/p6repo

This script is just a thin wrapper around curl, you just need a multipart form post really.

In order to fetch artifacts, you need to configure your zef to recognise the repository. In my case I have a ~/.config/zef/config.json, where in the Repository section I added:

    {
        "short-name" : "shelve6",
        "enabled" : 1,
        "module" : "Zef::Repository::Ecosystems",
        "options" : {
            "name" : "shelve6",
            "auto-update" : 1,
            "mirrors" : [
                "http://localhost:8080/repos/p6repo/packages.json"
            ]
        }
    },

After that, zef happily pulls from shelve6!

Authentication and Authorization

If you want to use the repository for private code, it may be a good idea to enable some security on it. Shelve6 can use credentials in the request and map them to a set of roles associated with that credential. Currently the only credential type supported are 'opaque' tokens, these are just striings that are not looked into (so not JWT or so). These come in a HTTP header like Authorization: Bearer supersecret, where 'supersecret' is the credential. In the future more credential types can be supported. To configure the mapping of credentials to roles, extend the 'server' part of the configuration:

server:
    port: 8080
    base-url: "http://localhost:8080"
    authentication:
        opaque-tokens:
            - token: supersecret
              roles: [CI]
              owner: raku-ci-1
            - token: eng8ahquia2kungeitaequie
              roles: [DEV, ADMIN]
              owner: Max Mustermann <[e-mail]>

In order to actually require any roles, you need to configure which roles allow what operation, on the repository:

repositories:
    - name: p6repo
      authorization:
        upload: [CI, DEV, ADMIN]
        download: [CI, DEV, ADMIN]
        delete: [ADMIN]

Note that the credential is associated with all the roles from the server config, but any role in the repository section is sufficient for access to be granted. For example the credential 'eng8ahquia2kungeitaequie' above gives both the 'DEV' and the 'ADMIN' roles, any of which would be enough to upload and download artifacts.

The shelve6-upload script supports setting these tokens through a commandline argument or an environment variable.

In order to enable zef to provide credentials during module fetching, you need to install the Zef::Service::AuthenticatedDownload plugin:

    zef install Zef::Service::AuthenticatedDownload

And configure it. The zef README explains where you can find the zef config, where you can add the plugin and configure it, which is best done before the other web fetchers to avoid them trying to download and failing due to auth:

        {
            "short-name" : "authenticated-download",
            "module" : "Zef::Service::AuthenticatedDownload",
            "options" : { 
                "configured-hosts" : [
                    {
                        "hostname" : "localhost",
                        "auth-type" : "opaque-token",
                        "token" :  "supersecret"
                    }
                ]
            }
        },

This will make zef use the configured credential for the host in question. If you do not want to put the credential into the config file, you can also leave it out and supply it via the ZEF_AUTH_OPAQUE_TOKEN environment variable.

License

Shelve6 is licensed under the Artistic License 2.0.

Feedback and Contact

Please let me know what you think: Robert Lemmen [e-mail]