Giters

saehun / elisp-repo-kit

Emacs Lisp package for creating an Emacs Lisp Github repo with CI

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Elisp Repo Kit (ERK)

melpa package melpa stable package CI workflow status DCO Check

This repository is a kit to start a new elisp package repository on GitHub. The package contained has commands to streamline elisp development.

Quickly set up an Emacs Lisp repository on GitHub with:

  • An elisp package
  • CI with GitHub Actions, configured for Darwin (MacOS) and Linux
  • Nix environment for obtaining dependencies or reproducibly developing CI locally
  • Licensing, DCO, DCO sign-off checks, PR template and CONTRIBUTING instructions
  • MELPA publishing compatible

To get started:

Install the package and run erk-new, provide a directory, and answer all the questions.

Install ERK

(use-package erk) ; vanilla

;; using elpaca
(elpaca-use-package
 (erk :host github
      :repo "positron-solutions/elisp-repo-kit"))

;; straight without `straight-use-package-by-default'
(straight-use-package '(erk :type git :host github
                            :repo "positron-solutions/elisp-repo-kit"))

;; straight with `straight-use-package-by-default' t
(use-package erk
  :straight
  (erk :type git :host github :repo "positron-solutions/elisp-repo-kit"))

;; or use melpa, manual load-path & require, you brave yak shaver

erk-new will ask for:

  • Root directory you want to clone to
  • Package name
  • Package prefix
  • Author name
  • GitHub user or organization
  • Email address

The erk-new calls erk-rename-relicense to rename all of the files, string replace names, and re-license to GPL3. Now just follow the steps in finish setting up Have fun!

Manual cloning

The standalone command, erk-clone will clone without renaming.

This repo is also a template repository, so you can fork without forking (Bruce Lee).

If you create via template or clone manually, it’s presumed you know what you’re doing at that point. Call erk-rename on its own to rename in these cases.

There are some customize options that cause the renaming to be transitively consistent.

Manually Add Just CI

Copy the .github folder and the contributing guide to your package. Set up your secrets for Cachix. Read the CI customization section.

Contents

Using ERK for development

Elisp repo kit contains some convenience functions to reload your package and to discover and run ert tests. These shortcuts just make common cases faster.

Run tests

Run erk-ert-project within your project. The tests will be discovered, rebuilt & reloaded if necessary, and run. There are a few other commands to augment the ert package.

Loading and re-loading your package

Run erk-reload-project-package in one of your project files. The features will be recompiled and reloaded.

The erk-reload-project-tests is the complementary command for tests.

Duplicating CI Locally

The CI configuration is all stored in .github. Usually you will want development instructions in your new repository. The CONTRIBUTING guide contains instructions to reproduce the CI behavior.

Finish setting up your new GitHub repo

You can copy this checklist to your org agenda files:

  • [X] Create a repository (from install instructions)
  • [ ] Create an empty GitHub repository and check the git remote configuration
  • [ ] Set up your git commit signing (and verification so that it’s obvious) and sign-off so that it will be straightforward for for FSF to pull in your changes if they later change to DCO instead of copyright assignment.
  • [ ] Sign up for cachix and, create a binary cache with API tokens and public read access
  • [ ] Add repository secrets necessary for your GitHub actions CACHIX_AUTH_TOKEN and CACHIX_CACHE_NAME (settings -> secrets -> new repository secret)
  • [ ] Enable actions and add the following actions to your allowed actions list: 
    actions/checkout@v3.2.0,
cachix/cachix-action@v12,
cachix/install-nix-action@v18,
actions/setup-python@v4,

    Note, Python is used to run a DCO check script, nothing more.

  • [ ] Get your package working, pushed, actions run, and CI badges all green
  • [ ] Publish to MELPA
  • [ ] Make a post on reddit and mastodon about your new package

Optional Steps

  • [ ] Install org-make-toc for the automatic TOC generation in this org document to work.
  • [ ] Branch protect and enable check requirements for your default branch (usually master). Merge commits, verified only, and no force push are recommended settings.
  • [ ] Enable requiring contributors to sign-off on web-based commits
  • [ ] For security of your Cachix secrets and any other secrets you may someday add, require Actions approval for all outside contributors and leave repository permissions at read-only

Cachix is somewhat optional. It’s free for open-source projects. It’s about as easy to sign up and generate the token as to remove the step from the GitHub actions, so you won’t save much time by avoiding it.

*If you opt out of cachix or any other binary cache, you will definitely want to turn off tests for emacsGit etc because the build times are about 30min-1hr per run when a fresh Emacs must be built.*

Overview of file contents and structure

/After cloning and renaming,/ you will have a file tree like this:

├── .gitignore                        # ignores for byte compiles, autoloads etc
│
├── README.org                        # this file
├── CONTRIBUTING.org                  # typical instructions for development
├── COPYING                           # a GPL3 license
├── DCO                               # Developer Certificate of Origin
│
├── .github
│   ├── .envrc                        # direnv integration with `nix develop`
│   ├── flake.nix                     # dependencies for this project
│   ├── flake.lock                    # version controlled lock of flake.nix input versions
│   ├── run-shim.el                   # elisp script with test & lint routines
│   ├── pull_request_template.md      # reminders for PR contributors
│   └── workflows
│       ├── ci.yml                    # workflow for lints and tests
│       └── dco.yml                   # workflow to check DCO sign-offs
│
├── lisp
│   └── erk.el                        # the package
│
└── test
    └── erk-test.el                   # ERT unit tests

You can use either a multi-file or flat layout for lisp. Just name test files something-test.el and keep all lisp files in root, /lisp or /test directories.

Customizing CI

The run-shim.el script is just provides a CLI interface for adding commands in the ci.yml CI declaration. Each action step just loads the shell, declared in the flake.nix and then runs the shim in Emacs. The shim consumes the CLI command arguments, so you can parameterize the invocations that way.

  • If you need extra elisp dependencies during CI, add them to the epkgs list in the flake.nix.
  • If you need extra 3rd party dependencies, add them to packages in the call to mkShell.
  • To invoke different elisp operations, add / modify the commands in run-shim.el.

There’s more information in CONTRIBUTING about running commands locally. You will want this information in your new repository.

Licensing, Developer Certificate of Origin

This template project is distributed with the MIT license. erk-new will also run erk-rename-relicense, which will automatically switch to the GPL3 license. The MIT license allows re-licensing, and so this change is compatible. If you accept non-trivial changes to your project, it will be very hard to change to the GPL3 later, so consider this choice.

The new repository will also come with DCO sign-off checking on PR’s. The instructions are in the CONTRIBUTING guide. A DCO sign-off policy will give your project a clear attestation of sufficient direct or transitive authority from each contributor to submit changes under the terms of your project’s license. This can only improve your legal protection from incidentally handling copyrighted code.

The DCO choice in this repository is also meant to encourage & push stodgy organizations whose responsibility it was to invent better processes towards lower friction paths to contribute code. If you fail to implement the DCO sign-off scheme, there is less hope that the FSF will someday independently merge changes that accumulate in your package because there will not be a .

Publishing to MELPA

If you have green CI, you have already passed many requirements of publishing a MELPA package. You still need to build your package and verify your recipe. You are going to clone melpa in order to make your PR. You can use the clone to verify the recipe.

Creating the recipe

Fork MELPA personally (not for organization) and clone it to wherever you keep your upstreams. It’s a good idea to separate upstreams from projects you actively maintain so you can see and delete upstreams when not in use.

mkdir -p upstream
cd upstream
git clone git@github.com:$GITHUB_USER/melpa.git  # replace $GITHUB_USER

Install package-build

(use-package package-build)

package-build-create-recipe will give you something like:

(erk :fetcher github :repo "positron-solutions/elisp-repo-kit")

The following template can be filled in and pull-requested to MELPA to publish. You don’t need to touch :files. The commit and branch are optional depending on how you version / develop / tag your releases.

Copy the recipe into recipes/erk inside your MELPA clone.

Testing package build

Inside the MELPA clone root:

# Builds the package
make recipes/erk
# Test sandbox installation (will affect ~/.emacs.d/elpa  So much for sandbox ¯\_(ツ)_/¯
EMACS_COMMAND=$(which emacs) make sandbox INSTALL=erk

Testing stable package build

You need a tag on your default (usually master) branch of your repo, positron-solutions/elisp-repo-kit. Use git tag -S v0.1.0 and git push origin v0.1.0. You can also just create a release in the GitHub interface.

# Test stable builds against your tags
STABLE=t make recipes/erk

MELPA Lints

Lastly, install melpazoid and call melpazoid on your main feature. It does some additional lints. You may need to install package-lint if you don’t have it. It’s not declared in melpazoid’s requirements. Getting the package in Nix is not easy yet since melpazoid is not yet on Melpa.

(straight-use-package
 '(melpazoid :type git :host github :repo "riscy/melpazoid" :files ("melpazoid/melpazoid.el")))

If everything works, you are ready to make a pull request to MELPA. Push your changes and check all the boxes in the PR template except the one that requires you to read the instructions.

Maintaining nixpkgs versions

Nixpkgs has a new release about every six months. You can check their branches and tags to see what’s current. To get updated dependencies from MELPA, it’s necessary to update the emacs-overlay with nix flake lock --update-input emacs-overlay. You can also specify revs and branches if you need to roll back. There is a make shortcut: make flake-update MacOS tends to get a little less test emphasis, and so nixpkgs-darwin-<version> branches exist and are required to pass more Darwin tests before merging. This is more stable if you are on MacOS. nixpkgs-unstable or master are your other less common options.

Package scope and relation to other work

There are two functional goals of this repository:

  • Automate the annoying work necessary to set up a new repository
  • Streamline common elisp development workflows

Commands within this package will focus on cleaner integration of the tests and lints with Emacs. There has been a lot of work in this area, but much of it is tangled with dependency management and sandbox creation. Much of it is done in languages other than elisp and focused on non-interactive workflows with no interactive integration on top.

Providing close to out-of-box CI is a big focus. By making it easier to qualify changes from other users, it becomes less burdonsome to maintain software, and by extension, less burdensom to publish and create software. The effect is to drive creation of elisp in a way that can accelerate the flow of elisp into Emacs itself.

Dependency Management

This repository uses pure dependency management and then levarages it to provide dependencies for development and CI environments. The resulting user experience is built around CI for reproducibility and interactive testing for development speed.

Because most elisp dependencies can be obtained without extensive system dependency management, many tools for testing Emacs packages provide dependency management and loading those dependencies into a fresh Emacs instance. This aligns well with ad-hoc sandboxed local testing. This was fine in the old days of impure dependency management and dirty environments.

The Emacs Nix Overlay and Emacs support within nixpkgs make it possible to stating and obtaining elisp dependencies in a completely pure way. Non-elisp dependencies are trivially provided form nixpkgs. Nix is extremely reliable at dependency management, and it is no surprise that much complexity is normalized away by just the basic behavior model of Nix. In addition, if your project needs or includes additional binary dependencies or modules, Nix is an excellent way to provide them to CI and users.

Discovering and Running Tests & Lints

During development, the commands provided under the erk- prefix make it more convenient to reload your package and test features. You can run the ert tests for a project while working on multiple packages.

During CI, this repository uses an elisp shim for discovering and running tests. The commands within the package for convenience during development are not depeneded upon during CI.

The CI actions obtain an environment with dependencies using Nix, so this can also be done locally using Nix, meaning re-creating environments is available to the user without leaning on CI.

Comparisons

There are many comparisons available to understand the roles of similar tools and how they relate to each other.

nix-emacs-ci capture the work needed to provide a running Emacs to CI. Tools like eldev and makem.sh have support for providing dependencies to that Emacs. The Nix flake in this project describes both of these tasks. Makem and Eldev etc document Gihub workflows, but the workflows in this repository are meant to be used out-of-the-box after cloning, although to be fair, there’s more decisions than actual work.

Nix-emacs-ci provides a lot of backwards-compatibility versions of Emacs. The nix-overlay is more forward looking, providing emacsGit and sometimes other upstream branches when a big feature like native compilation is in the pipeline. Nix-emacs-ci is also still using legacy Nix, without flakes. Flakes are just nicer and the way Nix is going.

Contributing

For turn-key contribution to the software ecosystem that keeps you moving, see the funding links.

For code-based contribution, first decide if you want to work on this repository or fork it to something entirely different.

The CONTRIBUTING guide in this repo contains development instructions, including singing & sign-off configuration. You will usually want this file in your own repositories.

Non-exhaustive list of changes that are very welcome:

  • More interactive integrations with high-value elisp development workflows
  • Running additional or better kinds of tests & lints
  • Fix bugs
  • Expose trivial options where a structural choice has limited them unnecessarily
  • Behave the same, but with a less complicated code
  • Guix or other pure dependency management support

Changes will likely be rejected if it is aimed at:

  • Non-elisp interfaces meant for invocation outside of Emacs or with scripting implemented in a language besides elisp.
  • Managing dependencies outside of Nix (or other pure dependency management) expressions
  • CI infrastructure support for non-Actions infrastructure (which cannot be tested in this repo)
  • Backwards compatibility for Emacs two versions behind next release. Master, current stable release, and release - 1 are the only versions being supported
  • pre-flake Nix support
  • Guix support that interferes with Nix support

Footnote on FSF and Emacs Core Licensing

Free Software Foundation (FSF) frequently requires copyright assignment on all code that goes into Emacs core. Many free software projects formerly requiring copyright assignment have since switched to using a Developer Certificate of Origin. DCO sign-off is a practice accepted by git, GCC, and the Linux Kernel.

Doing DCO sign-off is not the same as copyright assignment, and serves a slightly different purpose. DCO sign-off is an attestation from the submitter stating that they have sufficient direct or transitive authority make their submission under the terms of the license of the recieving project. Copyright assignment serves a more offensive role in the case of GPL non-compliance, giving FSF alone legal standing. If you don’t care about FSF being able to sue people, the DCO should suffice.

Using the DCO may make it easier for code in your project to be included in Emacs core later. It is the intent of this choice to steer FSF towards DCO-like solutions in order to accelerate code flow into Emacs. Regardless of FSF’s ongoing position on use of DCO’s, by requiring DCO sign-off and GPG signature, you can be assured that changes submitted to a code base you control are strongly attested to be covered by the license you chose.

Shout-outs

  • alphapapa for being super prolific at everything, including package writing, documentation, and activity on various social platforms
  • adisbladis for the Nix overlay that makes the CI and local development so nice
  • NobbZ for being all over the Nix & Emacs interwebs
  • FSF and all contributors to Emacs & packages for the Yak shaving club

About

Emacs Lisp package for creating an Emacs Lisp Github repo with CI

License:MIT License

Languages

Language:Emacs Lisp 100.0%