Emacs infrastructure tracking issue

[adisbladis]

I'm making an effort to clean up our Emacs packaging and use autogenerated packages whenever possible.
This is an incomplete list of things I want to achieve:

• Clean up redundant hand-written packages #65874
• Make emacs attrsets easily overrideable (while retaining the order of the composition) #65874
• Drop deprecated emacsPackages & rename emacsPackagesNg -> emacsPackages #66301
• Create an auto-updating emacs overlay - (well underway https://github.com/adisbladis/exwm-overlay/tree/melpa ), most changes required will go into nixpkgs
  • Move overlay to https://github.com/nix-community/

Strectch goals (I'm unlikely to do these myself)

- [ ] Package spacemacs
- [ ] Package doom-emacs

These have proven to be difficult. See https://github.com/vlaci/nix-doom-emacs/.

[Tad Fisher]

Here's a weird one for you: I added org-jira to my config using emacsWithPackages, including org (which should pull from the Org repo). It is byte-compiled using melpaBuild, as are all packages in melpa-packages.nix.

Unfortunately, at runtime, org-jira breaks, because it's byte-compiled against the version of Org that is shipped with Emacs, and not the one I am bundling in the generated site-lisp. This is the issue showing the error, and the resolution (delete .elc files so it is recompiled when loaded): ahungry/org-jira#163

So, to work around this, I override org-jira and explicitly add org to packageRequires in order to get it to load a recent Org and not the builtin version.

My questions, then, are:

1. Should these packages be declaring org in their Package-Requires? It is built-in, after all, but I suspect the Emacs packaging infrastructure assumes byte-compilation at runtime, and not the per-package byte-compilation stage performed by Nix.
2. Do we need to manually add builtin libraries to packageRequires in case they are also published to an ELPA repo, in order to avoid errors as a result of byte-compilation?
3. Could we implicitly add certain builtin libraries from the resolved package set to all packages, in case they are loaded during byte-compilation?
4. Is what we are doing correct at all? Should we defer byte-compilation to the emacsWithPackages builder itself, so it has a full site-lisp to resolve references during compilation?

[Jan Malakhovski]

IMHO:

1. Should these packages be declaring `org` in their `Package-Requires`? It is built-in, after all, but I suspect the Emacs packaging infrastructure assumes byte-compilation at runtime, and not the per-package byte-compilation stage performed by Nix.

Ideally, yes. (E.g. note that Haskell's Cabal packages do explicitly list versions of GHC builtin libraries they use.) In practice, making all of them obey this would be a lot of effort.

2. Do we need to manually add builtin libraries to `packageRequires` in case they are also published to an ELPA repo, in order to avoid errors as a result of byte-compilation?

I would rather not, too much effort.

3. Could we implicitly add certain builtin libraries from the resolved package set to all packages, in case they are loaded during byte-compilation?

Yes, ideally, there should be a `emacsBuiltin` attrset in `emacsPackages` (of a lowest priority for fixpointing) all pointing to current `emacs`, all of which can then be added as requisites to most packages by default. In such a setup we will, at most, need to manually override `org` and the rest of builtin packages present in other package sets so that they won't depend on themselves when fixpointing.

4. Is what we are doing correct at all? Should we defer byte-compilation to the `emacsWithPackages` builder itself, so it has a full `site-lisp` to resolve references during compilation?

That's an option that will solve your issue, but it will not solve the underlying issue. IMHO, broken byte-compilation is just a tip of an iceberg here. E.g., say one package hardcodes a path to binary/script from its dependency while being `make`ed.

[László Vaskó]

My two cents:
I am fairly happy with my nix-doom-emacs derivation.
There are two challenges packaging doom in its current state:

1. It uses straight.el for package management. I had to create a wrapper to present an Emacs package set as a straight repository. IMHO upstream straight.el could be extended with a "package lock" functionality (like npm, cargo, etc): it could write a json file containing packages needed. This information then could be used to
   1. populate the straight repository from the nix store
   2. or a more fine-grained solution would be to have something like pip2nix which generates derivations on the fly from Python dependency specifications.
2. Unfortunately doom uses its installation directory for persisting run-time state. It'd need an upstream fix to package without hacks

[Tad Fisher]

Ran into an issue similar to #66303 (comment) while testing Emacs 27.0.90. In this case, some package is pulling in seq from ELPA and byte-compiling against it instead of the built-in seq feature. This results in an infinite recursion at startup, because seq-subseq used to delegate to cl-subseq, but now cl-subseq calls the former.

In this case, I don't want seq to be pulled in from ELPA, so there would need to be some mechanism to prefer built-in packages if available.

[Akira Komamura]

Hello,

I am currently developing a Nix-based framework for testing Emacs packages, which uses Steve Purcell's nix-emacs-ci for backported previous versions of Emacs. It also uses emacs-overlay for a fresh set of packages required by a package under test.

In some situations, however, it fails to run tests on older versions of Emacs, and this seems to be about built-in packages of Emacs. For example, lsp-dart fails on Emacs 25.2 with the following error due to missing define-symbol-prop function:

Debugger entered--Lisp error: (void-function define-symbol-prop)
define-symbol-prop(lsp-range--pcase-macroexpander edebug-form-spec nil)
byte-code("\300\301\302\303#\300\304\305\301#\303\207" [define-symbol-prop lsp-range--pcase-macroexpander edebug-form-spec nil lsp-range pcase-macroexpander] 5)
require(lsp-mode)
eval-buffer(#<buffer load-31078> nil "/run/user/1000/lsp-dart-ert5uy/lsp-dart-utils.el" nil t) ; Reading at buffer position 822
load-with-code-conversion("/run/user/1000/lsp-dart-ert5uy/lsp-dart-utils.el" "/run/user/1000/lsp-dart-ert5uy/lsp-dart-utils.el" nil t)
require(lsp-dart-utils)
eval-buffer(#<buffer load> nil "/run/user/1000/lsp-dart-ert5uy/test/lsp-dart-utils-test.el" nil t) ; Reading at buffer position 784

lsp-mode uses pcase-defmacro macro, which had already existed in Emacs 25.2 but changed to call define-symbol-prop function in Emacs 26, which had not existed before. I suppose Emacs packages in the Nix repo are built with builtin packages contained in the latest version of Emacs (26.3 at present), which can cause issues in old versions of Emacs. Is this view correct?

What would be a proper solution to run latest packages on older versions of Emacs? Is it necessary to override every package depending on such built-in packages, as suggested in #66303 (comment)? Or should I even create a different variant of package set for each version of Emacs? Would wrapping macro expansions with eval-and-compile or something would prevent the missing function error? Thank you in advance.

[Matthew Piziak]

@tadfisher I just ran into the same problem. Did you ever resolve your case with cl-subseq and seq-subseq calling each other?

[adisbladis]

Since all the big items are checked off I'm going to close this.
Any smaller issues should be reported separately.