orgstrap allows Org files to describe their own requirements and
define their own functionality, making them self-contained standalone
computational artifacts dependent only on Emacs or other
implementations of the Org Babel protocol in the future.
This file bootstraps itself to provide the tools to use orgstrap in
any Org file.
orgstrap has a formal specification and this
file contains 3 implementations that support 3 slightly different use
cases along with tooling for common orgstrap workflows.
orgstrap works with all versions of Emacs since 24.4 and Org since 8.2.10.
Please see the changelog for the latest updates.
- Getting started
- Hello orgstrap
- Inspiration
- Use cases
- Details
- Specification
- Local Variables
- Code
- Changelog
- Contributing
- Guides
- Best practices
- Bootstrapping to Emacs, bootstrapping to Org
- Examples
- Background, file local variables, and checksums
- Experience reports
- Future work
orgstrap is easy.
If you already have orgstrap installed you can enable it for any
Org file by running M-x orgstrap-init which will add the basic
orgstrap machinery to the current buffer.
orgstrap is available on melpa. You can install it via
M-x package-install orgstrap or (use-package orgstrap).
You can also try out orgstrap without installing just by opening
this file in Emacs!
- Obtain the org mode source for this file. (e.g. from GitHub).
- Open the source in Emacs*.
(e.g.
M-xurl-handler-modethenC-x C-fhttps://raw.githubusercontent.com/tgbugs/orgstrap/master/README.org). - Decline the file local variables.
- Inspect the orgstrap block and file local variables.
- Reload the file and accept the file local variables.
- Congratulations you can now use
orgstrapwith your own files!
If you install orgstrap in this way you have to open the file again
every time you open a new Emacs, so installing orgstrap.el
via package.el or by some other means is recommended.
A minor mode for editing orgstrapped files is included as orgstrap-edit-mode.
It is activated by orgstrap-init. When enabled it automatically updates
the orgstrap-block-checksum prop line local variable whenever the
orgstrap block changes.
If you do not use orgstrap-edit-mode then the easiest way to add the
orgstrap checksum to a file is to invoke M-x orgstrap-add-block-checksum.
orgstrap also includes orgstrap-mode, which is a regional minor mode
for org-mode. When enabled, orgstrap-mode detects, checks, and runs
orgstrap blocks when visiting Org files, superseding any embedded eval:
local variables.
The rest of this file is an overview of the use cases for orgstrap and
the implementation of orgstrap along with discussion and commentary.
If you are looking for examples of how to use orgstrap this files is a good place to start.
org-mode file executable (with a bit of safety).
# -*- orgstrap-cypher: sha256; orgstrap-block-checksum: 66ba9b040e22cc1d30b6f1d428b2641758ce1e5f6ff9ac8afd32ce7d2f4a1bae; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; -*-
# [[orgstrap][jump to orgstrap block for this file]]
#+name: orgstrap
#+begin_src elisp :results none
(message "orgstrap successful!") ; (ref:im-a-coderef-and-thats-ok)
#+end_src
=orgstrap= a plain-text executable format. Powered by Org mode and Emacs.
# Local Variables:
# eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap--confirm-eval-minimal (lang body) (not (and (member lang '("elisp" "emacs-lisp")) (eq orgstrap-block-checksum (intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))
# End:
By default org-mode source block headers only take existing elisp functions as arguments.
This means that header arguments can become extremely verbose.
Wouldn’t it be great if you could use the magical mystical power of defun
inside an org file itself to provide simple, reusable functionality rather
than copying and pasting yanking and putting killing and yanking raw
elisp around the buffer?
With orgstrap you can.
orgstrap makes sure that the functionality that you need is available when you need it.
Whether it is (defun dir-tramp-sudo (host) (format "/ssh:%s|sudo:%s:" host host)) to
simplify a pattern for remote execution when using the :dir header argument, or a
function to detect and set the right environment variables, orgstrap is there for you.
orgstrap specifies what is essentially a plain-text executable file format.
Thus, it can be used for nearly everything[fn::Now, whether it should be….].
While many (including the author) might find this to be totally radically awesome, there are much better, saner, and safer ways to execute arbitrary code than to hash some elisp blocks and use Emacs file local variables to automatically eval a specially named source block only when it matches the hash.
| Use case | Good idea | Alternative |
|---|---|---|
| Always run defuns used in file | ✅ Yes | init.el, C-c C-c |
| Install elisp code directly | ❌ No | Use package.el, straight, etc. |
| Self tangling files | ✅ I do it | C-c C-v C-t |
| Install packages required by file | Probably | System package manager |
| Create an Emacs based botnet | ✅ ✅ Definitely | ??? |
| Create Orgware for non-technical users | ✅ Yes | Web server and the unholy trinity. |
| Replace hard to follow instructions | ✅ Yes | Hard to follow instructions |
| Tangle git hook files for publishing | ✅ Yes | Manually tangle |
| System specific behavior without edits | ✅ Yes | #+name: literal blocks via : |
| Version control for source blocks | ❌ ❌ Please no | git, hg, svn, anything please |
| Detect and set environment variables | ✅ Yes |
orgstrap in an org file is
automatically run using an eval: file local variable. Users can
review and add the file local variables to their known safe list
so that the code can be run in the future without the need to bother
them again.
When opening a file for the first time, users should decline the local
variables, review the eval: local variable and the orgstrap block
directly, and then reload, revisit, or M-x org-mode and only then
accept the local variables. This only needs to be done once for the
eval: local variables (unless they are updated).
orgstrap block that is used for this file.
;; This is an example that also nowebs in the source for
;; `orgstrap-init' and `orgstrap-add-block-checksum' along
;; with the rest of the orgstrap machinery so it is easy to
;; use orgstrap to create and update orgstrap blocks
<<orgstrap-run-helper-defuns>>
<<orgstrap-dev-helper-defuns>>
<<orgstrap-edit-helper-defuns>>
<<orgstrap-init-helper-defuns>>
<<orgstrap-extra-helper-defuns>>
;; tangle helpers
(defun ow---strip-empty-lines-and-refs ()
(save-excursion
(goto-char (point-min))
(while (re-search-forward "^ +$\\|[ ;]*(ref:.+)$" nil t)
;; FIXME stripping the refs here can cause a divergence for the checksums
;; FIXME this incorrectly strips refs from orgstrap-minimal.org due to wrong mode
(replace-match ""))))
;; XXX reminder that this cannot be a buffer local hook because it
;; doesn't run in this buffer this is likely a bug
(add-hook 'org-babel-tangle-body-hook #'ow---strip-empty-lines-and-refs)
;; helper functions to update examples
(defun orgstrap--update-examples ()
"Use with `orgstrap-on-change-hook' to automatically keep the contents
of the example blocks in sync."
;; XXX WARNING if you update the orgstrap-file-local-variables-common block
;; you MUST re`eval-defun' for `orgstrap--local-variables--eval-common' and
;; `orgstrap--lv-common-with-block-name' otherwise the changes will not take
(let ((pairs `(("local-variables-prop-line-example" ,(orgstrap--local-variables-prop-line-string))
("local-variables-portable-example" ,(orgstrap--file-local-variables-string))
("local-variables-minimal-example" ,(let ((orgstrap-use-minimal-local-variables t))
(orgstrap--file-local-variables-string))))))
(mapcar (lambda (name-content) (apply #'orgstrap-update-src-block name-content)) pairs)))
(defun orgstrap--local-variables-prop-line-string ()
"Copy the first logical line of the file since it is easier and faster
than trying to sort out which variables were or were not in the prop line."
;; XXX NOTE There are some cases involving bootstrapping to emacs where the first line of
;;an org-mode file is a shebang, but we will deal with those if and when they arrise
(buffer-substring-no-properties 1 (save-excursion (goto-char 0) (next-logical-line) (point))))
(defun orgstrap--file-local-variables-string ()
(let (print-length)
(with-temp-buffer
(org-mode)
(goto-char 0)
(insert "#+name: orgstrap\n#+begin_src elisp :lexical yes\n#+end_src\n")
(orgstrap--add-file-local-variables orgstrap-use-minimal-local-variables)
(goto-char 0)
(kill-whole-line 4)
(buffer-string))))
;; tangle blocks and update examples on change
(add-hook 'orgstrap-on-change-hook #'org-babel-tangle nil t) ;; FIXME should fire on non-semantic changes
(add-hook 'orgstrap-on-change-hook #'orgstrap--update-examples nil t)
;; enable orgstrap mode locally for this file when this block runs
(orgstrap-edit-mode 1)
(message "orgstrap complete!")The headers for the block above look like this.
#+name: orgstrap #+begin_src elisp :results none :noweb no-export :lexical yes <<orgstrap>> #+end_src
Additional machinery is provided as part of this file to update the local
variable value of orgstrap-block-checksum so that only known blocks can
be run. Note that this DOES NOT PROTECT against someone changing the block
and the checksum at the same time and sending you a malicious file! You need
an alternate and trusted source against which to verify the checksum of the
orgstrap block.
A couple of notes on portability and backward compatibility with older
versions of Emacs. I have tried to get orgstrap running on emacs-23,
however the differences between org 6.33x and org 8.2.10 are too
large to be overcome without significant additional code. First, all
uses of (setq-local var "value") have to be changed to
(set (make-local-variable 'var) "value") so that the local variable
eval code can run. However once that is done, you discover that all of
the org-babel functions are missing, and then you will discover that
emacs-23 doesn’t support lexical binding. Therefore, we don’t support
emacs-23 and older versions.
There is a major usability issue for orgstrap when running Emacs
< 27. Specifically, prior to Emacs 27 it is not possible to view the
file whose local variables are about to be set because it is
impossible to switch out of the file local variables confirmation
buffer. Starting in Emacs 27 it is possible to change buffer to view
the file that is about to have its file local variables set.
The specification for orgstrap makes extensive use of terminology derived from the Emacs manual section on Specifying File Variables and the Org manual section on the Structure of Code Blocks.
What the Emacs manual calls the first line or prop-line is referred
to in this document as the prop line and the variables specified in
it are referred to as prop line local variables. What the Emacs
manual explicitly calls the local variables list we refer to in the
same way[fn::In other sections of the readme that contains this
specification the nomenclature is inconsistent, and refers to these
variously as end local variables or simply as local variables or file
local variables.].
What the Org manual refers to as a source code block we refer to in the
same way.
In order for an Org mode file to support the use of orgstrap it must
contain the following.
The prop line of the Org file must include three local variables:
orgstrap-cypher, orgstrap-norm-func-name, and orgstrap-block-checksum.
Anywhere in the rest of the file there must be an Org source code block
that has the <name> orgstrap with whitespace preceding the o and only
whitespace following the p until a newline. Newline and whitespace are as
defined by Org mode syntax.
This source code block is henceforth referred to as the orgstrap block.
If there is more than one source code block with the <name> orgstrap
then the source code block that starts closest to the beginning of the file
is the orgstrap block.
The <language> for the orgstrap block must be elisp or emacs-lisp. [fn::
It is possible that other languages might be supported in the future. However,
that is somewhat challenging given that Org and Orb-babel only implicitly
specify that a conforming implementation that can execute source code blocks
must support Emacs lisp source code blocks and the use of Emacs lisp in
header arguments. There is an infinitesimal possibility that Org-babel will
support the use of other languages for inline header arguments since it
already supports them via blocks and it is not trivial to allow additional
languages to be used inline without some additional way to indicate the language
in use for a particular block. On the other hand, there is a small possibility
that other languages could be supported in the orgstrap block by specifying
them as part of the local variables list. However it is not clear that this
is needed, because it is possible to specify a small orgstrap block that can
ensure that the required Org-babel language implementations are installed and
then securely run those blocks. This block can probably be stripped down
sufficiently to make it possible to implement only the subset of elisp
required to run that block.]
Everything else about the orgstrap block is delegated to Org mode, including
header arguments, and noweb expansion.
When provided with the same file whose orgstrap block was originally hashed
(where “the same file” means a file with the same checksum when hashed using
the algorithm specified by the orgstrap-cypher variable), a conforming
implementation must be able to do the following.
A conforming implementation must be able to reproduce the orgstrap-block-checksum
using only the information contained in the orgstrap-cypher and
orgstrap-norm-func-name prop line local varaibles, and information
contained in the rest of the file explicitly excluding the contents of
the orgstrap-block-checksum prop line local varaible. The most obvious
additional information required being the contents of the orgstrap block [fn::
The reference implementation provided in the readme containing this specification
uses an Emacs eval: local variable (elv) in the local variables list. Embedding
an elv is not required by this specification. However, such an implementation allows
files to depend only on the core Emacs implementation.
In the future an optional extension may be added to this document that specifies the
behavior for files using an elv in the local variables list.
A minimal implementation that works without elvs is also provided.
Files that contain only the prop line local variables are dependent on an implementation of orgstrap already being present on the system running the file.
There is a fine balance between portability and compactness since a minimal implementation has to make more assumptions about the systems it will run on.
Multi-stage orgstrap or other means of bootstrapping a working runtime for an Org file such as the process implemented in the Bootstrapping to Emacs, bootstrapping to Org section of this readme are ongoing areas of exploration.].
One implementation detail is that conforming implementations must implement noweb expansion and coderef removal prior to passing the contents of the
orgstrap blockto a normalization function.
Normalization functions that produce different output given the same input for at least one input must have different names. One way this can be achieved is by suffixing a name with a version number.
In order for an orgstrap normalization function name to be considered official it must have an implementation bearing that name in the Normalization functions section of the readme that contains this specification. Once a function has been named, no other function shall ever bear the same name unless for all inputs it produces output that is byte-identical to the output of all other previous implementations of the function bearing that name.
A key point about
orgstrap-norm-func-nameis that the implementation of these functions must be agreed upon by various implementations, if a user inserts a fake hash, implementations should deal with it by running the normalization and hashing process again using a known-conforming implementation on a system that they control.
- Overview
- Org version support
- Normalization
- Definitions
- Note on noweb support
- Note on coderefs
- How local variables appear in the file
orgstrap (minimal and
portable) that are small enough to fit in the local variables list at
the end of a file. The local variables list must start less than 3000
chars from the end of the file.
We use setq-local in eval: to set org-confirm-babel-evaluate
because it is a safe-local-variable only when the value is t and
cannot be set directly as a file local variable. In this context this
workaround seems reasonable and not malicious because the use of
eval: should alert users that some arbitrary stuff is going on and
that they should be on high alert to check it.
Below in Definitions there is a more readable version of what the compacted local variables code at the end of the file is doing. Always check that the =eval:= local variables in unknown orgstrapped files match a known set when reviewing and accepting local variables.
orgstrap eval local variables, or elvs for short, are little
helpers at the end of the file that make everything work in a portable
manner when orgstrap.el is not present on a system.
While elvs are not required by the specification, they greatly reduce the complexity of implementation. They also simplify the instructions to two steps: 1. install Emacs, 2. open the file.
Different versions of theorgstrap local variables work with
different versions of org-mode. We include an explicit version
check and fail so that strange partial successes can be avoided
and so that newer versions of the local variables can be simplified
when backward compatibility is not needed. For example one might
imagine a future where no local variables are needed in the file
at all, only the cypher and the checksum because we managed to
get support for the convention built into org-mode directly.
This will also allow us to streamline which block to use based on whether noweb is being used. If it is not then we can decide automatically.
If orgstrap is installed, we use the installed version of orgstrap anyway so don’t bother.
(let ((a (org-version)) ; actual
(n orgstrap-min-org-version)) ; need
(or (fboundp #'orgstrap--confirm-eval) ; orgstrap with portable is already present on the system
(not n)
(string< n a)
(string= n a)
(error "Your Org is too old! %s < %s" a n)))
string<must be used in order to support emacs-24
Shared normalization code embedded as elvs.
(unless (boundp 'orgstrap-norm-func)
(defvar-local orgstrap-norm-func orgstrap-norm-func-name))
(defun orgstrap-norm-embd (body)
"Normalize BODY."
(funcall orgstrap-norm-func body))
(unless (fboundp #'orgstrap-norm)
(defalias 'orgstrap-norm #'orgstrap-norm-embd))Normalization functions for orgstrap.el.
(defun orgstrap-norm (body)
"Normalize BODY."
(if orgstrap--debug
(orgstrap-norm-debug body)
(funcall orgstrap-norm-func body)))
(defun orgstrap-norm-debug (body)
"Insert BODY normalized with NORM-FUNC into a buffer for easier debug."
(let* ((print-quoted nil)
(bname (format "body-norm-%s" emacs-major-version))
(buffer (let ((existing (get-buffer bname)))
(if existing existing
(create-file-buffer bname))))
(body-normalized (funcall orgstrap-norm-func body)))
(with-current-buffer buffer
(erase-buffer)
(insert body-normalized))
body-normalized))
;; orgstrap normalization functions
<<block-orgstrap-norm-func--dprp-1-0>>
<<block-orgstrap-norm-func--prp-1-1>>
<<block-orgstrap-norm-func--prp-1-0>>
For emacs < 26 (org < 9) either lowercase
#+caption:must be placed BEFORE#+name:, OR#+CAPTION:must be uppercase and can come after#+name:, otherwise#+name:will not be associated with the block. What a fun bug.Addendum. Apparently in the older version of Org
:nowebis always yes. As a result, testing against Emacs 24 or 25 will alert you if you forget to set:nowebon a block.
This normalization function is obsolete
(let ((print-quoted nil))
(prin1-to-string (read (concat "(progn\n" body "\n)"))))(defun orgstrap-norm-func--prp-1-0 (body)
"Normalize BODY using prp-1-0."
<<orgstrap-code-normalization--prin1-read-progn-1-0>>)
(make-obsolete #'orgstrap-norm-func--prp-1-0 #'orgstrap-norm-func--prp-1-1 "1.2")Normalize BODY by wrapping in progn, calling read, and then prin1-to-string.
There are still unresolved issues if tabs are present in the orgstrap block which
is why 1.0 is included. print-quoted is critical for consistent hashing.
prin1-to-string is used to normalize the code in the orgstrap block,
removing any comments and formatting irregularities. This is important
for two reasons.
First it helps prevent denial of service attacks against human auditors who have low bandwidth for detecting fiddly changes.
Second, normalization that ignores comments makes it possible to improve the documentation of code without changing the checksum. Hopefully this will reduce one of the obstacles to enhancing the documentation of orgstrap code and blocks over time since rehashing will not be required when the meaningful code itself has not changed.
(print-quoted nil) is needed for backward compatibility due to a change
to the default from nil to t in emacs-27 (sigh). See
emacs commit 72ee93d68daea00e2ee69417afd4e31b3145a9fa.
(let (print-quoted print-length print-level)
(prin1-to-string (read (concat "(progn\n" body "\n)"))))(defun orgstrap-norm-func--prp-1-1 (body)
"Normalize BODY using prp-1-1."
<<orgstrap-code-normalization--prin1-read-progn-1-1>>)I learned that print-length and print-level exist in the usual
way, which is that somehow they got set to something other than nil
and as a result checksums started failing left and right because the
number of expressions in the body of the progn eval was greater than
the value of print-length, resulting in truncation and replacement
with .... This can also happens inside add-file-local-variable and
possibly even inside format!? Therefore I’m updating to version to
1.1 of the normalization procedure so that I can defensively bind
those variables to nil.
A normalization function that is invariant to changes in docstrings.
Walk the tree and setcdr out docstrings.
This normalization function must be portable between versions, which means that the forms that get spliced must be from a static set that does not change between versions.
Since this normalization is mostly a quality of life improvement to
allow docstrings to be changed without rehashing, limiting to a
specific set of forms is ok. If you use something like cl-defun and
change the docstring, then you will have to rehash. The 90% use case
is covered here in a compact manner.
(let ((p (read (concat "(progn\n" body "\n)")))
(m '(defun defun-local defmacro defvar defvar-local defconst defcustom))
print-quoted print-length print-level)
(cl-labels
((f
(b)
(cl-loop
for e in b when (listp e) do ; for expression in body when the expression is a list
(or
(and
(memq (car e) m) ; is a form with docstrings
(let ((n (nthcdr 4 e))) ; body after docstring
(and
(stringp (nth 3 e)) ; has a docstring
(or (cl-subseq m 3) n) ; var or doc not last
(f n) ; recurse for nested
;; splice out the docstring and return t to avoid the other branch
(or (setcdr (cddr e) n) t))))
;; recurse e.g. for (when x (defvar y t))
(f e)))
p))
(prin1-to-string (f p))))(defun orgstrap-norm-func--dprp-1-0 (body)
"Normalize BODY using dprp-1-0."
<<orgstrap-code-normalization--dedoc-prin1-read-progn-1-0>>)(orgstrap--with-block "orgstrap"
(prog1 (read (orgstrap-norm-func--dprp-1-0 body)) nil))orgstrap-init to populate file local variables.
The portable confirm eval is extracted to its own block so that we can
include it as a backstop for users who have orgstrap installed but are
running an older version of org-mode than is supported by the file
that they are trying to load.
;;;###autoload
(defun orgstrap--confirm-eval-portable (lang _body)
"A backwards compatible, portable implementation for confirm-eval.
This should be called by `org-confirm-babel-evaluate'. As implemented
the only LANG that is supported is emacs-lisp or elisp. The argument
_BODY is rederived for portability and thus not used."
;; `org-confirm-babel-evaluate' will prompt the user when the value
;; that is returned is non-nil, therefore we negate positive matchs
(not (and (member lang '("elisp" "emacs-lisp"))
(let* ((body (orgstrap--expand-body (org-babel-get-src-block-info)))
(body-normalized (orgstrap-norm body))
(content-checksum
(intern
(secure-hash
orgstrap-cypher
body-normalized))))
;;(message "%s %s" orgstrap-block-checksum content-checksum)
;;(message "%s" body-normalized)
(eq orgstrap-block-checksum content-checksum)))))
;; portable eval is used as the default implementation in orgstrap.el
;;;###autoload
(unless (fboundp #'orgstrap--confirm-eval)
(defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-portable))(defun orgstrap--confirm-eval-minimal (lang body)
(not (and (member lang '("elisp" "emacs-lisp"))
(eq orgstrap-block-checksum
(intern
(secure-hash
orgstrap-cypher
(orgstrap-norm body)))))))
(unless (fboundp #'orgstrap--confirm-eval)
;; if `orgstrap--confirm-eval' is bound use it since it is
;; is the portable version XXX NOTE the minimal version will
;; not be installed as local variables if it detects that there
;; are unescaped coderefs since those will cause portable and minimal
;; to produce different hashes
(defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal))Once orgstrap--confirm-eval is defined the rest of the eval: local variables are the same.
(let (enable-local-eval) (vc-find-file-hook)) ; use the obsolete alias since it works in 24
(let ((ocbe org-confirm-babel-evaluate)
(obs (org-babel-find-named-block ,orgstrap-orgstrap-block-name))) ; quasiquoted when nowebbed
(if obs
(unwind-protect
(save-excursion
(setq-local orgstrap-norm-func orgstrap-norm-func-name)
(setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval)
(goto-char obs) ; FIXME `org-save-outline-visibility' but that is not portable
(org-babel-execute-src-block))
(when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval)
;; XXX allow orgstrap blocks to set ocbe so audit for that
(setq-local org-confirm-babel-evaluate ocbe))
(ignore-errors
(org-set-visibility-according-to-property)))
;; FIXME warn or error here?
(warn "No orgstrap block.")))Since orgstrap-norm-func is a dynamic variable it simplifies the
potential future case where we don’t embed the normalization function,
still not sure if we really want to do that though
(let ((orgstrap-norm-func orgstrap-norm-func-name)
(org-confirm-babel-evaluate #'orgstrap--confirm-eval)
(obs (org-babel-find-named-block ,orgstrap-orgstrap-block-name))) ; quasiquoted when nowebbed
(if obs
(unwind-protect
(save-excursion
(goto-char obs)
(org-babel-execute-src-block))
(org-set-startup-visibility))
;; FIXME warn or error here?
(warn "No orgstrap block.")))>= 9.3.8.
The portable set of local variables described below works with versions of
Org as far back as 8.2.10 (the version bundled with emacs-24.5).
org-mode do not know what to do with coderefs.
The simplest solution is to hide them in comments as ;(ref:coderef)
if you need them. See (clrin) and (oab) for examples in this file.
Here is the prop line from the first line of this file that
includes the cypher and checksum of the orgstrap block.
# -*- org-adapt-indentation: nil; org-edit-src-content-indentation: 0; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: d33bdc8924478fedbd92ff73836c43e136d90e4c18393ff7c5e0aeda37f892d2; -*-
Here are the portable local variables from the end of the file.
# Local Variables:
# eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap-org-src-coderef-regexp (_fmt &optional label) (let ((fmt org-coderef-label-format)) (format "\\([:blank:]*\\(%s\\)[:blank:]*\\)$" (replace-regexp-in-string "%s" (if label (regexp-quote label) "\\([-a-zA-Z0-9_][-a-zA-Z0-9_ ]*\\)") (regexp-quote fmt) nil t)))) (unless (fboundp #'org-src-coderef-regexp) (defalias 'org-src-coderef-regexp #'orgstrap-org-src-coderef-regexp)) (defun orgstrap--expand-body (info) (let ((coderef (nth 6 info)) (expand (if (org-babel-noweb-p (nth 2 info) :eval) (org-babel-expand-noweb-references info) (nth 1 info)))) (if (not coderef) expand (replace-regexp-in-string (org-src-coderef-regexp coderef) "" expand nil nil 1)))) (defun orgstrap--confirm-eval-portable (lang _body) (not (and (member lang '("elisp" "emacs-lisp")) (let* ((body (orgstrap--expand-body (org-babel-get-src-block-info))) (body-normalized (orgstrap-norm body)) (content-checksum (intern (secure-hash orgstrap-cypher body-normalized)))) (eq orgstrap-block-checksum content-checksum))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-portable)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))
# End:
Here are the minimal local variables from the end of the file.
# Local Variables:
# eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap--confirm-eval-minimal (lang body) (not (and (member lang '("elisp" "emacs-lisp")) (eq orgstrap-block-checksum (intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))
# End:
This section contains the implementation of functions to calculate
orgstrap-block-checksum and set it as a prop line local variable.
It also contains functions to embed the bootstrapping code as an
eval: local variable in the local variables list, along with other
quality of life functionality for the user such as orgstrap-mode,
orgstrap-edit-mode, and orgstrap-init.
Testing org-src-coderef-regexp with fboundp in ref:orgstrap-expand-body
is needed due to changes in the behavior of org-babel-get-src-block-info
roughly around the 9.0 release.
The changes in behavior for org-babel-get-src-block-info are commits
orgit-rev:~/git/NOFORK/org-mode::88659208793dca18b7672428175e9a712af7b5ad and
orgit-rev:~/git/NOFORK/org-mode::9738da473277712804e0d004899388ad71c6b791. They
both occur before the introduction of org-src-coderef-regexp in
orgit-rev:~/git/NOFORK/org-mode::9f47b37231b3c45afcd604a191e346200bd76e98.
All of this happend before orgit-rev:~/git/NOFORK/org-mode::release_9.0. By
testing org-src-coderef-regexp with fboundp there are only a tiny number
of versions where there might be some inconsistent behavior, e.g.
orgit-rev:~/git/NOFORK/org-mode::release_8.3.6, but I suspect that the probability
that anyone anywhere is running one of those versions is approximately zero.
(defun orgstrap-org-src-coderef-regexp (_fmt &optional label)
"Backport `org-src-coderef-regexp' for 24 and 25.
See the upstream docstring for info on LABEL.
_FMT has the wrong meaning in 24 and 25."
(let ((fmt org-coderef-label-format))
(format "\\([:blank:]*\\(%s\\)[:blank:]*\\)$"
(replace-regexp-in-string
"%s"
(if label
(regexp-quote label)
"\\([-a-zA-Z0-9_][-a-zA-Z0-9_ ]*\\)")
(regexp-quote fmt)
nil t))))
(unless (fboundp #'org-src-coderef-regexp)
(defalias 'org-src-coderef-regexp #'orgstrap-org-src-coderef-regexp))
(defun orgstrap--expand-body (info)
"Expand noweb references in INFO body and remove any coderefs."
;; this is a backport of `org-babel--expand-body'
(let ((coderef (nth 6 info))
(expand
(if (org-babel-noweb-p (nth 2 info) :eval)
(org-babel-expand-noweb-references info)
(nth 1 info))))
(if (not coderef)
expand
(replace-regexp-in-string
(org-src-coderef-regexp coderef) "" expand nil nil 1))))In order for orgstrap to be maximally portable and not depend on already being installed, the implementation needs to work with the local variables list eval variable without complicating the situation when orgstrap is installed as a package.
While ideally this would be done using only the standard hooks around
hack-local-variables such an approach does not work because the
variables are filtered before those hooks can run. Therefore, we have
to advise hack-local-variables-confirm in order to capture and
remove any orgstrap elvs that we find. For maximum safety this
minimally requires mutation of the all-vars list passed to
hack-local-variables-confirm.
This is a fairly deep tampering with the way that hack-local-variables works, so special attention should be given when reviewing the security implications of any changes.
(require 'cl-lib)
;;;###autoload
(defvar orgstrap-mode nil
"Variable to track whether `orgstrap-mode' is enabled.")
(cl-eval-when (eval compile load)
;; prevent warnings since this is used as a variable in a macro
(defvar orgstrap-orgstrap-block-name "orgstrap"
"Set the default blockname to orgstrap by convention.
This makes it easier to search for orgstrap if someone encounters
an orgstrapped file and wants to know what is going on."))
(defvar orgstrap-default-cypher 'sha256
"The default cypher passed to `secure-hash' when hashing blocks.")
(defvar-local orgstrap-cypher orgstrap-default-cypher
"Local variable for the cypher for the current buffer.
If you change `orgstrap-default-cypher' you should update this as well
using `setq-default' since it will not change automatically.")
(put 'orgstrap-cypher 'safe-local-variable (lambda (v) (ignore v) t))
(defvar-local orgstrap-block-checksum nil
"Local variable for the expected checksum for the current orgstrap block.")
;; `orgstrap-block-checksum' is not a safe local variable, if it is set
;; as safe then there will be no check and code will execute without a check
;; it is also not risky, so we leave it unmarked
(defconst orgstrap--internal-norm-funcs
'(orgstrap-norm-func--prp-1-0
orgstrap-norm-func--prp-1-1
orgstrap-norm-func--dprp-1-0)
"List internally implemented normalization functions.
Used to determine which norm func names are safe local variables.")
(defvar-local orgstrap-norm-func-name nil
"Local variable for the name of the current orgstrap-norm-func.")
(put 'orgstrap-norm-func-name 'safe-local-variable
(lambda (value) (and orgstrap-mode (memq value orgstrap--internal-norm-funcs))))
;; Unless `orgstrap-mode' is enabled and the name is in the list of
;; functions that are implemented internally this is not safe
(defvar-local orgstrap-norm-func #'orgstrap-norm-func--dprp-1-0
"Dynamic variable to simplify calling normalizaiton functions.
Defaults to `orgstrap-norm-func--dprp-1-0'.")
(defvar orgstrap--debug nil
"If non-nil run `orgstrap-norm' in debug mode.")
(defgroup orgstrap nil
"Tools for bootstraping Org mode files using Org Babel."
:tag "orgstrap"
:group 'org
:link '(url-link :tag "README on GitHub"
"https://github.com/tgbugs/orgstrap/blob/master/README.org"))
(defcustom orgstrap-always-edit nil
"If non-nil command `orgstrap-mode' activates command `orgstrap-edit-mode'."
:type 'boolean
:group 'orgstrap)
(defcustom orgstrap-always-eval nil
"Always try to run orgstrap blocks even when populating `org-agenda'."
:type 'boolean
:group 'orgstrap)
(defcustom orgstrap-always-eval-whitelist nil
"List of files that should always try to run orgstrap blocks."
:type 'list
:group 'orgstrap)
(defcustom orgstrap-file-blacklist nil
"List of files that should never run orgstrap blocks.
For files on the blacklist `orgstrap-block-checksum' is removed from
the local variables list so that the checksum will not be added to
the `safe-local-variable-values' list. If it were added it would then
be impossible to prevent execution of the source block when `orgstrap-mode'
is disabled.
This is useful when developing a block that modifies Emacs' configuration.
NOTE this variable only works if `orgstrap-mode' is enabled."
:type 'list
:group 'orgstrap)
;; orgstrap blacklist
(defun orgstrap-blacklist-current-file (&optional universal-argument)
"Add the current file to `orgstrap-file-blacklist'.
If UNIVERSAL-ARGUMENT is provided do not run `orgstrap-revoke-current-buffer'."
;; It is usually better to revoke a checksum when its file is blacklisted since
;; it is easier for the user to add the checksum again when needed than it is
;; for them to revoke manually. The prefix argument allows users who know that
;; they only want to blacklist the file and not revoke to do so though such
;; cases are expected to be fairly rare.
;; FIXME blacklisting a bad file that has already been approved is painful
;; right now, you have to manually set `enable-local-eval' to nil, load the
;; file, run this function, and then reset `enable-local-eval'.
(interactive "P")
(unless universal-argument
(orgstrap-revoke-current-buffer))
(add-to-list 'orgstrap-file-blacklist (buffer-file-name))
(customize-save-variable 'orgstrap-file-blacklist orgstrap-file-blacklist))
(defun orgstrap-unblacklist-current-file ()
"Remove the current file from `orgstrap-file-blacklist'."
(interactive)
(setq orgstrap-file-blacklist (delete (buffer-file-name) orgstrap-file-blacklist))
(customize-save-variable 'orgstrap-file-blacklist orgstrap-file-blacklist))
;; orgstrap revoke
(defun orgstrap-revoke-checksums (&rest checksums)
"Delete CHECKSUMS or all checksums if nil from `safe-local-variables-values'."
(interactive)
(cl-delete-if (lambda (pair)
(cl-destructuring-bind (key . value)
pair
(and
(eq key 'orgstrap-block-checksum)
(or (null checksums) (memq value checksums)))))
safe-local-variable-values)
(customize-save-variable 'safe-local-variable-values safe-local-variable-values))
(defun orgstrap-revoke-current-buffer ()
"Delete checksum(s) for current buffer from `safe-local-variable-values'.
Deletes embedded and current values of `orgstrap-block-checksum'."
(interactive)
(let* ((elv (orgstrap--read-current-local-variables))
(cpair (assoc 'orgstrap-block-checksum elv))
(checksum-existing (and cpair (cdr cpair))))
(orgstrap-revoke-checksums orgstrap-block-checksum checksum-existing)))
(defun orgstrap-revoke-elvs ()
"Delete all approved orgstrap elvs from `safe-local-variable-values'."
(interactive)
(cl-delete-if #'orgstrap--match-elvs safe-local-variable-values)
(customize-save-variable 'safe-local-variable-values safe-local-variable-values))
(define-obsolete-function-alias
'orgstrap-revoke-eval-local-variables
#'orgstrap-revoke-elvs
"1.2.4"
"Replaced by the more compact `orgstrap-revoke-elvs'.")
;; orgstrap run helpers
<<orgstrap-portable-confirm-eval>>
;; orgstrap-mode implementation
(defun orgstrap--org-buffer ()
"Only run when in `org-mode' and command `orgstrap-mode' is enabled.
Sets further hooks."
(when enable-local-eval
;; if `enable-local-eval' is nil we honor it and will not run
;; orgstrap blocks natively, this matches the behavior of the
;; embedded elvs and simplifies logic for cases
;; where orgstrap should not run (e.g. when populating `org-agenda')
(advice-add #'hack-local-variables-confirm :around #'orgstrap--hack-lv-confirm)
(unless (member (buffer-file-name) orgstrap-file-blacklist)
(add-hook 'before-hack-local-variables-hook #'orgstrap--before-hack-lv nil t))))
(defun orgstrap--hack-lv-confirm (command &rest args)
"Advise `hack-local-variables-confirm' to remove orgstrap eval variables.
COMMAND should be `hack-local-variables-confirm' with ARGS (all-vars
unsafe-vars risky-vars dir-name)."
(advice-remove #'hack-local-variables-confirm #'orgstrap--hack-lv-confirm)
(cl-destructuring-bind (all-vars unsafe-vars risky-vars dir-name)
(cl-loop
for arg in
(if (member (buffer-file-name) orgstrap-file-blacklist)
(cl-loop ; zap checksums for blacklisted
for arg in args collect
(if (listp arg)
(cl-delete-if
(lambda (pair) (eq (car pair) 'orgstrap-block-checksum))
arg)
arg))
args)
collect ; use `cl-delete-if' to mutate the lists in calling scope
(if (listp arg) (cl-delete-if #'orgstrap--match-elvs arg) arg))
;; After removal we have to recheck to see if unsafe-vars and
;; risky-vars are empty so we can skip the confirm dialogue. If we
;; do not, then the dialogue breaks the flow.
(or (and (null unsafe-vars)
(null risky-vars))
(funcall command all-vars unsafe-vars risky-vars dir-name))))
(defun orgstrap--before-hack-lv ()
"If `orgstrap' is in the current buffer, add hook to run the orgstrap block."
;; This approach is safer than trying to introspect some of the implementation
;; internals. This hook will only run if there are actually local variables to
;; hack, so there is little to no chance of lingering hooks if an error occures
(remove-hook 'before-hack-local-variables-hook #'orgstrap--before-hack-lv t)
;; XXX we have to remove elvs here since `hack-local-variables-confirm' is not called
;; if all variables are marked as safe, e.g. via `orgstrap-whitelist-file'
;; FIXME other interactions between blacklist and whitelist may need to be handled here
(setq file-local-variables-alist (cl-delete-if #'orgstrap--match-elvs file-local-variables-alist))
(add-hook 'hack-local-variables-hook #'orgstrap--hack-lv nil t))
(defun orgstrap--used-in-current-buffer-p ()
"Return t if all the required orgstrap prop line local variables are present."
(and (boundp 'orgstrap-cypher) orgstrap-cypher
(boundp 'orgstrap-block-checksum) orgstrap-block-checksum
(boundp 'orgstrap-norm-func-name) orgstrap-norm-func-name))
(defmacro orgstrap--lv-common-with-block-name ()
"Helper macro to allow use of same code between core and lv impls."
`(progn
<<orgstrap-file-local-variables-common>>))
(defun orgstrap--hack-lv ()
"If orgstrap is present, run the orgstrap block for the current buffer."
;; we remove this hook here and we do not have to worry because
;; it is always added by `orgstrap--before-hack-lv'
(remove-hook 'hack-local-variables-hook #'orgstrap--hack-lv t)
(when (orgstrap--used-in-current-buffer-p)
(orgstrap--lv-common-with-block-name)
(when orgstrap-always-edit
(orgstrap-edit-mode 1))))
(defun orgstrap--match-elvs (pair)
"Return nil if PAIR matchs any elv used by orgstrap.
Avoid false positives if possible if at all possible."
(and (eq (car pair) 'eval)
;;(message "%s" (cdr pair))
;; keep the detection simple for now, any eval lv that
;; so much as mentions orgstrap is nuked, and in the future
;; if orgstrap-nb is used we may need to nuke that too
(string-match "orgstrap" (prin1-to-string (cdr pair)))))
;;;###autoload
(defun orgstrap-mode (&optional arg)
"A regional minor mode for `org-mode' that automatically runs orgstrap blocks.
When visiting an Org file or activating `org-mode', if orgstrap prop line local
variables are detect then use the installed orgstrap implementation to run the
orgstrap block. If orgstrap embedded local variables are present, they will not
be executed. `orgstrap-mode' is not a normal minor mode since it does not run
any hooks and when enabled only adds a function to `org-mode-hook'. ARG is the
universal prefix argument."
(interactive "P")
(ignore arg)
(let ((turn-on (not orgstrap-mode)))
(cond (turn-on
(add-hook 'org-mode-hook #'orgstrap--org-buffer)
(setq orgstrap-mode t)
(message "orgstrap-mode enabled"))
(arg) ; orgstrap-mode already enabled so don't disable it
(t
(remove-hook 'org-mode-hook #'orgstrap--org-buffer)
(setq orgstrap-mode nil)
(message "orgstrap-mode disabled")))))
;; orgstrap do not run aka `org-agenda' eval protection
(defun orgstrap--advise-no-eval-lv (command &rest args)
"Advise COMMAND to disable elvs for files loaded inside it.
ARGS vary by COMMAND.
If the elvs are disabled then `orgstrap-block-checksum' is added
to the `ignored-local-variables' list for files loaded inside
COMMAND. This makes it possible to open orgstrapped files where
the elvs will not run without having to accept the irrelevant
variable for `orgstrap-block-checksum'."
;; continually prompting users to accept a local variable when they
;; cannot inspect the file and when accidentally accepting could
;; allow unchecked execution at some point in the future is bad
;; better to simply pretend that the elvs and the block checksum
;; do not even exist unless the file is explicitly on a whitelist
;; orgstrapped files are just plain old org files in this context
;; since agenda doesn't use any babel functionality ... of course
;; I can totally imagine using orgstrap to automatically populate
;; an org file or update an org file using orgstrap to keep the
;; agenda in sync with some external source ... so need a variable
;; to control this
(if orgstrap-always-eval
(apply command args)
(let* ((enable-local-eval
(and args
orgstrap-always-eval-whitelist
(member (car args)
orgstrap-always-eval-whitelist)
enable-local-eval))
(ignored-local-variables
(if enable-local-eval ignored-local-variables
(cons 'orgstrap-block-checksum ignored-local-variables))))
(apply command args))))
(advice-add #'org-get-agenda-file-buffer :around #'orgstrap--advise-no-eval-lv);;; edit helpers
(defvar orgstrap--clone-stamp-source-buffer-block nil
"Source code buffer and block for `orgstrap-stamp'.")
(defcustom orgstrap-on-change-hook nil
"Hook run via `before-save-hook' when command `orgstrap-edit-mode' is enabled.
Only runs when the contents of the orgstrap block have changed."
:type 'hook
:group 'orgstrap)
(defcustom orgstrap-use-minimal-local-variables nil
"Set whether minimal, smaller but less portable variables are used.
If nil then backward compatible local variables are used instead.
If the value is customized to be non-nil then compact local variables
are used and `orgstrap-min-org-version' is set accordingly. If the
current version of org mode does not support the features required to
use the minimal variables then the portable variables are used instead."
:type 'boolean
:group 'orgstrap)
;; edit utility functions
(defun orgstrap--current-buffer-cypher ()
"Return the cypher used for the current buffer.
The value is `orgstrap-cypher' if it is bound otherwise
`orgstrap-default-cypher' is returned."
(if (boundp 'orgstrap-cypher) orgstrap-cypher orgstrap-default-cypher))
<<orgstrap-expand-body>>
<<orgstrap-code-normalization-functions>>
(defun orgstrap--goto-named-src-block (blockname)
"Goto org block named BLOCKNAME.
Like `org-babel-goto-named-src-block' but non-interactive, does
not use the mark ring, and errors if the block is not found."
(let ((obs (org-babel-find-named-block blockname)))
(if obs (goto-char obs)
(error "No block named %s" blockname))))
(defmacro orgstrap--with-block (blockname &rest macro-body)
"Go to the source block named BLOCKNAME and execute MACRO-BODY.
The macro provides local bindings for four names:
`info', `params', `body-unexpanded', and `body'."
(declare (indent defun))
`(save-excursion
(let* ((info
(org-save-outline-visibility 'use-markers
(orgstrap--goto-named-src-block ,blockname)
(org-babel-get-src-block-info)))
(params (nth 2 info))
(body-unexpanded (nth 1 info))
(body (orgstrap--expand-body info)))
,@macro-body)))
(defun orgstrap--update-on-change ()
"Run via the `before-save-hook' local variable.
Test if the checksum of the orgstrap block has changed,
if so update the `orgstrap-block-checksum' local variable
and then run `orgstrap-on-change-hook'."
(let* ((elv (orgstrap--read-current-local-variables))
(cpair (assoc 'orgstrap-block-checksum elv))
(checksum-existing (and cpair (cdr cpair)))
(checksum (orgstrap-get-block-checksum)))
(unless (eq checksum-existing (intern checksum))
(remove-hook 'before-save-hook #'orgstrap--update-on-change t)
;; for some reason tangling from a buffer counts as saving from that buffer
;; so have to remove the hook to avoid infinite loop
(unwind-protect
(save-excursion
(undo)
(undo-boundary) ; insert an undo boundary so that the
;; changes to the checksum are transparent to the user
(undo) ; undo the undo above
(orgstrap-add-block-checksum nil checksum)
(run-hooks 'orgstrap-on-change-hook))
(add-hook 'before-save-hook #'orgstrap--update-on-change nil t)))))
(defun orgstrap--get-actual-params (params)
"Filter defaults, nulls, and junk from src block PARAMS."
(let ((defaults (append org-babel-default-header-args
org-babel-default-header-args:emacs-lisp)))
(cl-remove-if (lambda (pair)
(or (member pair defaults)
(memq (car pair) '(:result-params :result-type))
(null (cdr pair))))
params)))
(defun orgstrap-header-source-element (header-name &optional block-name &rest more-names)
"Given HEADER-NAME find the element that provides its value.
If BLOCK-NAME is non-nil then search for headers for that block,
otherwise search for headers associated with the current block.
If MORE-NAMES are provided return the value for each (or nil)."
;; get the current headers, see if the value is set anywhere
;; or if it is default, search for default anyway just to be sure
;; return nil if not found
;; when searching for any header go to the end of the src line
;; `re-search-backward' from that point for :header-arg but not
;; going beyond the affiliated keywords for the current element
;; (if you can get affiliated keywords for the current element
;; that might simplify the search as well? check the impl for how
;; the actual values are obtained during execution etc)
;; when found use `org-element-at-point' to obtain the element
;; in another function the operates on the element
;; the element will give start, end, value, etc.
;; find bounds of value from element or sub element
;; delete the value, replace with new value
(ignore header-name block-name more-names)
(error "Not implemented TODO"))
(defun orgstrap-update-src-block-header (name new-params &optional update)
"Add header arguments to block NAME from NEW-PARAMS from some other block.
Existing header arguments will NOT be removed if they are not included in
NEW-PARAMS. If UPDATE is non-nil existing header arguments are updated."
(let ((new-act-params (orgstrap--get-actual-params new-params)))
(orgstrap--with-block name
(ignore body body-unexpanded)
(let ((existing-act-params (orgstrap--get-actual-params params)))
(dolist (pair new-act-params)
(cl-destructuring-bind (key . value)
pair
(let ((header-arg (substring (symbol-name key) 1)))
(if (assq key existing-act-params)
(if update
(unless (member pair existing-act-params)
;; TODO remove existing
;; `org-babel-insert-header-arg' does not remove
;; and it is not trivial to find the actual location
;; of an existing header argument there are 4 places
;; that we will have to look and then in some cases
;; we will have to append even if we do find them
(org-babel-insert-header-arg header-arg value)
;; This message works around the fact that we don't
;; have replace here, only append TODO consider
;; changing the way update works to be nil, replace,
;; or append once an in-place replace is implemented
(message "%s superseded for block %s." key name))
(warn "%s already defined for block %s!" key name))
(org-babel-insert-header-arg header-arg value)))))))))
(unless (fboundp #'flatten-tree)
;; backwards compatibility for Emacs < 27
(defun flatten-tree (tree)
(let (elems)
(while (consp tree)
(let ((elem (pop tree)))
(while (consp elem)
(push (cdr elem) tree)
(setq elem (car elem)))
(if elem (push elem elems))))
(if tree (push tree elems))
(nreverse elems))))
(defun orgstrap--check-portable-subset (body)
"Ensure that BODY uses only symbols that are portable for `prin1-to-string'."
;; XXX Note that [.] may diverge again because `.asdf' can be read without
;; escaping the leading . whereas `\?asdf' cannot. This is an artifact of
;; the c implementation of `prin1' being reused to handle both chars.
(let* ((l (flatten-tree (read (concat "(progn\n" body "\n)"))))
(symbols (cl-remove-duplicates (cl-remove-if-not #'symbolp l)))
(bads (cl-remove-if-not
(lambda (s) (string-match "[.?]" (cl-subseq (symbol-name s) 1)))
symbols)))
(when bads
(error "checksum failed: non-portable symbols detected: %S" bads))))
;; edit user facing functions
(defun orgstrap-get-block-checksum (&optional cypher)
"Calculate the `orgstrap-block-checksum' for the current buffer using CYPHER."
(interactive)
(orgstrap--with-block orgstrap-orgstrap-block-name
(ignore params body-unexpanded)
(orgstrap--check-portable-subset body)
(let ((cypher (or cypher (orgstrap--current-buffer-cypher)))
(body-normalized (orgstrap-norm body)))
(secure-hash cypher body-normalized))))
(defun orgstrap-add-block-checksum (&optional cypher checksum)
"Add `orgstrap-block-checksum' to file local variables of `current-buffer'.
The optional CYPHER argument should almost never be used,
instead change the value of `orgstrap-default-cypher' or manually
change the file property line variable. CHECKSUM can be passed
directly if it has been calculated before and only needs to be set.
If `orgstrap-save-developer-checksums' is non-nil then add the checksum to
`orsgrap-developer-checksums'."
(interactive)
(let* ((cypher (or cypher (orgstrap--current-buffer-cypher)))
(orgstrap-block-checksum (or checksum (orgstrap-get-block-checksum cypher))))
(when orgstrap-block-checksum
(save-excursion
(add-file-local-variable-prop-line 'orgstrap-cypher cypher)
(add-file-local-variable-prop-line 'orgstrap-norm-func-name orgstrap-norm-func)
(add-file-local-variable-prop-line 'orgstrap-block-checksum (intern orgstrap-block-checksum)))
(when orgstrap-save-developer-checksums
(add-to-list 'orgstrap-developer-checksums (intern orgstrap-block-checksum))))
orgstrap-block-checksum))
(defun orgstrap-run-block ()
"Evaluate the orgstrap block for the current buffer."
;; bind to :orb or something like that
(interactive)
(save-excursion
(orgstrap--goto-named-src-block orgstrap-orgstrap-block-name)
(org-babel-execute-src-block)))
(defun orgstrap-clone (&optional universal-argument)
"Set current block or orgstrap block as the source for `orgstrap-stamp'.
If a UNIVERSAL-ARGUMENT is supplied then the orgstrap block is always used."
;; TODO consider whether to avoid the inversion of behavior around C-u
;; namely that nil -> always from orgstrap block, C-u -> current block
;; this would avoid confusion where unprefixed could produce both
;; behaviors and only switch when already on a src block
(interactive "P")
(let ((current-element (org-element-at-point))
(current-buffer (current-buffer)))
(if (and (eq (org-element-type current-element) 'src-block)
(not universal-argument))
(let ((block-name (org-element-property :name current-element)))
(if block-name
(setq orgstrap--clone-stamp-source-buffer-block
(cons current-buffer block-name))
(warn "The current block has no name, it cannot be a clone source!")))
(if (orgstrap--used-in-current-buffer-p)
(setq orgstrap--clone-stamp-source-buffer-block
(cons current-buffer orgstrap-orgstrap-block-name))
(warn "orgstrap is not used in the current buffer!")))))
(defun orgstrap-stamp (&optional universal-argument overwrite)
"Stamp orgstrap block via `orgstrap-clone' to current buffer.
If UNIVERSAL-ARGUMENT is \\='(16) aka (C-u C-u) this will OVERWRITE any existing
block. If you are not calling this interactively all as (orgstrap-stamp nil t)
for calirty. You cannot stamp an orgstrap block into its own buffer."
(interactive "P")
(unless (eq major-mode 'org-mode)
(user-error "`orgstrap-stamp' only works in org-mode buffers"))
(unless orgstrap--clone-stamp-source-buffer-block
(user-error "No value to clone! Use `orgstrap-clone' first"))
(let ((overwrite (or overwrite (equal universal-argument '(16))))
(source-buffer (car orgstrap--clone-stamp-source-buffer-block))
(source-block-name (cdr orgstrap--clone-stamp-source-buffer-block))
(target-buffer (current-buffer)))
(when (eq source-buffer target-buffer)
(error "Source and target are the same buffer. Not stamping!"))
(cl-destructuring-bind (source-body
source-params
org-adapt-indentation
org-edit-src-content-indentation)
(save-window-excursion
(with-current-buffer source-buffer
(orgstrap--with-block source-block-name
(ignore body-unexpanded)
(list body
params
org-adapt-indentation
org-edit-src-content-indentation))))
(if (and (not overwrite)
(member orgstrap-orgstrap-block-name
(org-babel-src-block-names)))
(warn "orgstrap block already exists not stamping!")
(orgstrap--add-orgstrap-block source-body) ; FIXME somehow the hash is different !?!??!
(orgstrap-update-src-block-header orgstrap-orgstrap-block-name source-params t)
(orgstrap-add-block-checksum) ; I think it is correct to add the checksum here
(message "Stamped orgsrap block from %s" (buffer-file-name source-buffer))))))
;;;###autoload
(define-minor-mode orgstrap-edit-mode
"Minor mode for editing with orgstrapped files."
:init-value nil :lighter "" :keymap nil
(unless (eq major-mode 'org-mode)
(setq orgstrap-edit-mode 0)
(user-error "`orgstrap-edit-mode' only works with org-mode buffers"))
(cond (orgstrap-edit-mode
(add-hook 'before-save-hook #'orgstrap--update-on-change nil t))
(t
(remove-hook 'before-save-hook #'orgstrap--update-on-change t))));;; dev helpers
(defcustom orgstrap-developer-checksums-file (concat user-emacs-directory "orgstrap-developer-checksums.el")
"Path to developer checksums file."
:type 'path
:group 'orgstrap)
(defcustom orgstrap-save-developer-checksums nil ; FIXME naming
"Whether or not to save checksums of orgstrap blocks under development."
:type 'boolean
:group 'orgstrap
:set (lambda (variable value)
(set-default variable value)
(if value
(add-hook 'orgstrap-on-change-hook #'orgstrap-save-developer-checksums)
(remove-hook 'orgstrap-on-change-hook #'orgstrap-save-developer-checksums))))
(defvar orgstrap-developer-checksums nil ; not custom because it is saved elsewhere
"List of checksums for orgstrap blocks created or modified by the user.")
(defun orgstrap--pp-to-string (value)
"Ensure that we actually print the whole VALUE not just the summarized subset."
(let (print-level print-length)
(pp-to-string value)))
(defun orgstrap-revoke-developer-checksums (&optional universal-argument)
"Remove all saved developer checksums. UNIVERSAL-ARGUMENT is a placeholder."
(interactive "P") (ignore universal-argument)
(setq orgstrap-developer-checksums nil)
(orgstrap-save-developer-checksums t))
(defun orgstrap-save-developer-checksums (&optional overwrite)
"Function to update `orgstrap-developer-checksums-file'.
If OVERWRITE is non-nil then overwrite the existing checksums."
(interactive "P")
(if orgstrap-save-developer-checksums
(let* ((checksums orgstrap-developer-checksums)
(buffer (find-file-noselect orgstrap-developer-checksums-file)))
(with-current-buffer buffer
(unwind-protect
(progn
(lock-buffer)
(let* ((saved (and (not (= (buffer-size) 0)) (cadr (nth 2 (read (buffer-string))))))
;; XXX NOTE saved is not used to updated `orgstrap-developer-checksums' here
;; FIXME massively inefficient
(combined (or (and (not overwrite)
(cl-remove-duplicates (append checksums saved)))
checksums)))
;; TODO do we need to check whether combined and saved are different?
;; (message "checksums: %s\nsaved: %s\ncombined: %s" checksums saved combined)
(erase-buffer)
(insert ";;; -*- mode: emacs-lisp; lexical-binding: t -*-\n")
(insert ";;; DO NOT EDIT THIS FILE IT IS AUTOGENERATED AND WILL BE OVERWRITTEN!\n\n")
(insert (string-replace
" " "\n"
(orgstrap--pp-to-string `(setq orgstrap-developer-checksums ',combined))))
(insert "\n;;; set developer checksums as safe local variables\n\n")
(insert
(orgstrap--pp-to-string
'(mapcar (lambda (checksum-value)
(add-to-list 'safe-local-variable-values
(cons 'orgstrap-block-checksum checksum-value)))
orgstrap-developer-checksums)))
(pp-buffer)
(indent-region (point-min) (point-max))
(save-buffer)))
(unlock-buffer)
(kill-buffer))))
(warn "No checksums were saved because `orgstrap-save-developer-checksums' is not set.")))
A note on filter aka cl-remove-if-not in orgstrap--add-file-local-variables at (clrin).
| emacs version | require |
|---|---|
| < 24 | ‘cl |
| < 25 | ‘cl-lib |
| < 27 | ‘seq |
The most portable thing to do for now is (require 'cl-lib) since we
don’t currently support anything below 23. Then use cl-remove-if-not.
There is a similar issue with pcase, which is that in emacs-24 the
syntax was closer to cl-case when dealing with symbols. Since cl-lib
is already in use, cl-case is the logical solution for portability.
Not all functionality works in older versions of Org. For example see
update block issue which is caused by the fact that
org-babel-update-block-body is broken prior to revision
orgit-rev:~/git/NOFORK/org-mode::7d6b8f51ec1993a66a385b98b2df42d0853fe289
which is not present in the versions of Org released with Emacs < 26.
"set -e \"-C\" \"-e\" \"-e\"\n{ null=/dev/null;} > \"${null:=/dev/null}\"\n{ args=;file=;MyInvocation=;__p=$(mktemp -d);touch ${__p}/=;chmod +x ${__p}/=;__op=$PATH;PATH=${__p}:$PATH;} > \"${null}\"\n$file = $MyInvocation.MyCommand.Source\n{ file=$0;PATH=$__op;rm ${__p}/=;rmdir ${__p};} > \"${null}\"\nemacs -batch -no-site-file -eval \"(let (vc-follow-symlinks) (defun orgstrap--confirm-eval (l _) (not (memq (intern l) '(elisp emacs-lisp)))) (let ((file (pop argv)) enable-local-variables) (find-file-literally file) (end-of-line) (when (eq (char-before) ?\\^m) (let ((coding-system-for-read 'utf-8)) (revert-buffer nil t t)))) (let ((enable-local-eval t) (enable-local-variables :all) (major-mode 'org-mode) find-file-literally) (require 'org) (org-set-regexps-and-options) (hack-local-variables)))\" \"${file}\" -- ${args} \"${@}\"\nexit\n<# powershell open"
;;; init helpers
(defvar orgstrap-link-message "jump to the orgstrap block for this file"
"Default message for file internal links.")
(defvar-local orgstrap--local-variables nil
"Variable to capture local variables from `hack-local-variables'.")
;; local variable generation functions
(defun orgstrap--get-min-org-version (info minimal)
"Get minimum org mode version needed by the orgstrap block for this file.
INFO is the source block info. MINIMAL sets whether to use minimal local vars."
(if minimal
(let ((coderef (or (nth 6 info) org-coderef-label-format))
(noweb (org-babel-noweb-p (nth 2 info) :eval)))
(if noweb
"9.3.8"
(let* ((body (or (nth 1 info) ""))
(crrx (org-src-coderef-regexp coderef))
(pos (string-match crrx body))
(commented
(and pos (string-match
(concat (rx ";" (zero-or-more whitespace)) crrx) body))))
;; FIXME the right way to do this is similar to what is done in
;; `org-export-resolve-coderef' but for now we know we are in elisp
(if (or (not pos) commented)
"8.2.10"
"9.3.8"))))
"8.2.10"))
(defun orgstrap--have-min-org-version (info minimal)
"See if current version of org meets minimum requirements for orgstrap block.
INFO is the source block info.
MINIMAL is passed to `orgstrap--get-min-org-version'."
(let ((actual (org-version))
(need (orgstrap--get-min-org-version info minimal)))
(or (not need)
(string< need actual)
(string= need actual))))
(defun orgstrap--dedoc (sexp)
"Remove docstrings from SEXP. WARNING mutates sexp!"
(let ((m '(defun defun-local defmacro defvar defvar-local defconst defcustom)))
(cl-loop
for e in sexp when (listp e) do ; for expression in sexp when the expression is a list
(or
(and
(memq (car e) m) ; is a form with docstrings
(let ((n (nthcdr 4 e))) ; body after docstring
(and
(stringp (nth 3 e)) ; has a docstring
(or (cl-subseq m 3) n) ; var or doc not last
(orgstrap--dedoc n) ; recurse for nested
;; splice out the docstring and return t to avoid the other branch
(or (setcdr (cddr e) n) t))))
;; recurse e.g. for (when x (defvar y t))
(orgstrap--dedoc e))))
sexp)
(defun orgstrap--local-variables--check-version (info &optional minimal)
"Return the version check local variables given INFO and MINIMAL."
`(
(setq-local orgstrap-min-org-version ,(orgstrap--get-min-org-version info minimal))
<<orgstrap-check-org-version>>))
(defun orgstrap--local-variables--norm (&optional norm-func-name)
"Return the normalization function for local variables given NORM-FUNC-NAME."
(let ((norm-func-name (or norm-func-name (default-value 'orgstrap-norm-func))))
(cl-case norm-func-name
(orgstrap-norm-func--dprp-1-0
'(
<<block-orgstrap-norm-func--dprp-1-0>>))
(orgstrap-norm-func--prp-1-1
'(
<<block-orgstrap-norm-func--prp-1-1>>))
(orgstrap-norm-func--prp-1-0
(error "`orgstrap-norm-func--prp-1-0' is deprecated.
Please update `orgstrap-norm-func-name' to `orgstrap-norm-func--prp-1-1'"))
(otherwise (error "Don't know that normalization function %s" norm-func-name)))))
(defun orgstrap--local-variables--norm-common ()
"Return the common normalization functions for local variables."
'(
<<orgstrap-normalization-common-embed>>))
(defun orgstrap--local-variables--eval (info &optional minimal)
"Return the portable or MINIMAL elvs given INFO."
(let* ((minimal (or minimal orgstrap-use-minimal-local-variables))
(minimal (and minimal (orgstrap--have-min-org-version info minimal))))
(if minimal
'(
<<orgstrap-minimal-confirm-eval>>)
'( ;(ref:elv-noweb-issue)
;; if you automatically reindent it will break these two
<<orgstrap-expand-body>>
<<orgstrap-portable-confirm-eval>>))))
(defun orgstrap--local-variables--eval-common ()
"Return the common eval check functions for local variables."
`( ; quasiquote to fill in `orgstrap-orgstrap-block-name'
<<orgstrap-file-local-variables-common>>))
;; init utility functions
(defun orgstrap--new-heading-elisp-block (heading block-name &optional header-args noexport)
"Create a new elisp block named BLOCK-NAME in a new heading titled HEADING.
The heading is inserted at the top of the current file.
HEADER-ARGS is an alist of symbols that are converted to strings.
If NOEXPORT is non-nil then the :noexport: tag is added to the heading."
(declare (indent 1))
(save-excursion
(goto-char (point-min))
(outline-next-heading) ;; alternately outline-next-heading
(org-meta-return)
(insert (format "%s%s\n" heading (if noexport " :noexport:" "")))
;;(org-edit-headline heading)
;;(when noexport (org-set-tags "noexport"))
(move-end-of-line 1)
(insert "\n#+name: " block-name "\n")
(insert "#+begin_src elisp")
(mapc (lambda (header-arg-value)
(insert " :" (symbol-name (car header-arg-value))
" " (symbol-name (cdr header-arg-value))))
header-args)
(insert "\n#+end_src\n")))
(defun orgstrap--trap-hack-locals (command &rest args)
"Advice for `hack-local-variables-filter' to do nothing except the following.
Set `orgstrap--local-variables' to the reversed list of read variables which
are the first argument in the lambda list ARGS.
COMMAND is unused since we don't actually want to hack the local variables,
just get their current values."
(ignore command)
(setq-local orgstrap--local-variables (reverse (car args)))
nil)
(defun orgstrap--read-current-local-variables ()
"Return the local variables for the current file without applying them."
(interactive)
;; orgstrap--local-variables is a temporary local variable that is used to
;; capture the input to `hack-local-variables-filter' it is unset at the end
;; of this function so that it cannot accidentally be used when it might be stale
(setq-local orgstrap--local-variables nil)
(let ((enable-local-variables t))
(advice-add #'hack-local-variables-filter :around #'orgstrap--trap-hack-locals)
(unwind-protect
(hack-local-variables nil)
(advice-remove #'hack-local-variables-filter #'orgstrap--trap-hack-locals))
(let ((local-variables orgstrap--local-variables))
(makunbound 'orgstrap--local-variables)
local-variables)))
(defun orgstrap--add-link-to-orgstrap-block (&optional link-message)
"Add an `org-mode' link pointing to the orgstrap block for the current file.
The link is placed in comment on the second line of the file. LINK-MESSAGE
can be used to override the default value set via `orgstrap-link-message'"
(interactive) ; TODO prompt for message with C-u ?
(goto-char (point-min))
(next-logical-line) ; required to get correct behavior?
(let ((link-message (or link-message orgstrap-link-message)))
(unless (save-excursion
(re-search-forward
(format "^# \\[\\[%s\\]\\[.+\\]\\]$"
orgstrap-orgstrap-block-name)
nil t)) ; XXX for some reason save-excursion fails so we have to reset
(goto-char (point-min))
(next-logical-line) ; use logical-line to avoid issues with visual line mode
(insert (format "# [[%s][%s]]\n"
orgstrap-orgstrap-block-name
(or link-message orgstrap-link-message))))))
(defun orgstrap--add-orgstrap-block (&optional block-contents)
"Add a new elisp source block with #+name: orgstrap to the current buffer.
If a block with that name already exists raise an error.
Insert BLOCK-CONTENTS if they are supplied."
(interactive)
(let ((all-block-names (org-babel-src-block-names)))
(if (member orgstrap-orgstrap-block-name all-block-names)
(warn "orgstrap block already exists not adding!")
(goto-char (point-max))
(insert "\n")
(orgstrap--new-heading-elisp-block "Bootstrap"
orgstrap-orgstrap-block-name
'((results . none)
(exports . none)
(lexical . yes))
'noexport)
(goto-char (point-max))
(insert "\n** Local Variables :ARCHIVE:\n")
(orgstrap--with-block orgstrap-orgstrap-block-name
(ignore params body-unexpanded body)
(when block-contents
;; FIXME `org-babel-update-block-body' is broken in < 26 (ref:obubb-issue)
;; for now warn and fail if the version is known bad NOTE trying to backport
;; is not simple because there are changes to the function signatures
(if (string< org-version "8.3.4")
(warn "Your version of Org is too old to use this feature! %s < 8.3.4"
org-version)
(org-babel-update-block-body block-contents)))
nil))))
(defun orgstrap--lv-command (info &optional minimal norm-func-name)
"Create the elvs for an orgstrapped file.
INFO is the output of `org-babel-get-src-block-info' for the orgstrap block.
MINIMAL determines whether a non-portable block has been requested.
NORM-FUNC-NAME names the function used to normalize orgstrap blocks."
(let ((lv-cver (orgstrap--local-variables--check-version
info
minimal))
(lv-norm (orgstrap--local-variables--norm
norm-func-name))
(lv-ncom (orgstrap--local-variables--norm-common))
(lv-eval (orgstrap--local-variables--eval
info
minimal))
(lv-ecom (orgstrap--local-variables--eval-common)))
(cons 'progn (orgstrap--dedoc (append lv-cver lv-norm lv-ncom lv-eval lv-ecom)))))
(defun orgstrap--add-file-local-variables (&optional minimal norm-func-name)
"Add the file local variables needed to make orgstrap work.
MINIMAL is used to control whether the portable or minimal block is used.
If MINIMAL is set but the orgstrap block uses features like noweb and
uncommented coderefs and function `org-version' is too old, then the portable
block will be used. NORM-FUNC-NAME is an optional argument that can be provided
to determine which normalization function is used independent of the current
buffer or global setting for `orgstrap-norm-func'.
When run, this function replaces any existing orgstrap elv with the latest
implementation available according to the preferences for the current buffer
and configuration. Other elvs are retained if they are present, and the
orgstrap elv is always added first."
;; switching comments probably wont work ? we can try
;; Use a prefix argument (i.e. C-u) to add file local variables comments instead of in a :noexport:
(interactive)
(let ((info (save-excursion
(orgstrap--goto-named-src-block orgstrap-orgstrap-block-name)
(org-babel-get-src-block-info)))
(elv (orgstrap--read-current-local-variables)))
(let ((lv-command (orgstrap--lv-command info minimal norm-func-name))
(commands-existing (mapcar #'cdr (cl-remove-if-not (lambda (l) (eq (car l) 'eval)) elv)))) ;(ref:clrin)
(let* ((stripped
(cl-remove-if
(lambda (cmd) (orgstrap--match-elvs (cons 'eval cmd)))
commands-existing))
(eval-commands (cons lv-command stripped)))
(when commands-existing
(delete-file-local-variable 'eval))
(let ((print-escape-newlines t) ; needed to preserve the escaped newlines
;; if `print-length' or `print-level' is accidentally set
;; `add-file-local-variable' will truncate the sexp with and elispsis
;; this is clearly a bug in `add-file-local-variable' and possibly in
;; something deeper, `print-length' is the only one that has actually
;; caused issues, but better safe than sorry
print-length print-level)
(mapcar (lambda (sexp) (add-file-local-variable 'eval sexp)) eval-commands))))))
(defun orgstrap--before-first-dull ()
"Goto the first non-empty line not starting with a sharp sign."
(goto-char (point-min))
(re-search-forward "\n[^#\n \t]")
(beginning-of-line))
(defun orgstrap--goto-elvs ()
"Goto the start of the elvs for the current buffer.
If no elvs are found goto `point-max' instead."
(widen)
(goto-char (point-max))
(search-backward "\n\^L" (max (- (point-max) 3000) (point-min)) 'move)
(when (let ((case-fold-search t))
(search-forward "Local Variables:" nil t))
(beginning-of-line)))
(defconst orgstrap--shebang-body
<<orgstrap-shebang-body()>>
"Shebang block body content.")
(defun orgstrap--add-shebang-block (&optional update)
"Add a shebang block to the current buffer."
;; goto correct location
;; create empty bash block
;; fill block
;; go to start of elvs
;; add powershell closing line
(let ((block-name "orgstrap-shebang")
(header-args '((eval . never) (results . none) (exports . none))))
(if (org-babel-find-named-block block-name)
(if update
(orgstrap-update-src-block "orgstrap-shebang" orgstrap--shebang-body)
(warn "A shebang block already exists. Not adding."))
(if update
(warn "A shebang block does not exist. Not updating.")
(save-excursion
(orgstrap--before-first-dull)
(insert "\n#+name: " block-name "\n")
(insert "#+begin_src bash")
(mapc (lambda (header-arg-value)
(insert " :" (symbol-name (car header-arg-value))
" " (symbol-name (cdr header-arg-value))))
header-args)
(insert "\n#+end_src\n")
(orgstrap-update-src-block "orgstrap-shebang" orgstrap--shebang-body)
(orgstrap--goto-elvs)
(insert (format "# close powershell comment %s>\n" "#")))))))
(defun orgstrap-update-shebang-block (&optional universal-argument)
"Update an existing shebang block. UNIVERSAL-ARGUMENT is ignored."
(interactive "P")
(ignore universal-argument)
(orgstrap--add-shebang-block 'update))
;; init user facing functions
;;;###autoload
(defun orgstrap-init (&optional prefix-argument shebang)
"Initialize orgstrap in a buffer and enable command `orgstrap-edit-mode'.
If PREFIX-ARGUMENT is non-nil and has a value of 4 or 64 init will attempt
to use the minimal local variables if possible.
If SHEBANG is non-nil or PREFIX-ARGUMENT is greater than or equal to 16
then a shebang block will also be added to the file.
Example usage.
M-x orgstrap-init -> portable elvs
C-u M-x orgstrap-init -> minimal elvs
C-u C-u M-x orgstrap-init -> portable elvs + shebang
C-u C-u C-u M-x orgstrap-init -> minimal elvs + shebang"
(interactive "P")
(unless (eq major-mode 'org-mode)
(error "Cannot orgstrap, buffer not in `org-mode' it is in %s!" major-mode))
;; TODO option for no link?
;; TODO option for local variables in comments vs noexport
(let (onf)
(let ((shebang (or shebang (and prefix-argument (>= (car prefix-argument) 16))))
(orgstrap-norm-func
(or (cdr (assoc 'orgstrap-norm-func-name (orgstrap--read-current-local-variables)))
(default-value 'orgstrap-norm-func))))
(save-excursion
(orgstrap--add-orgstrap-block)
(orgstrap-add-block-checksum)
(orgstrap--add-link-to-orgstrap-block)
;; FIXME sometimes local variables don't populate due to an out of range error
(orgstrap--add-file-local-variables
(or (and prefix-argument (memq (car prefix-argument) '(4 64))) orgstrap-use-minimal-local-variables))
(when shebang (orgstrap--add-shebang-block))
(orgstrap-edit-mode 1)
(setq onf orgstrap-norm-func)))
;; reset to ensure that a stale value is not inserted on next save
(setq-local orgstrap-norm-func onf)));;; extra helpers
(defun orgstrap-update-src-block (name content)
"Set the content of source block named NAME to string CONTENT.
XXX NOTE THAT THIS CANNOT BE USED WITH #+BEGIN_EXAMPLE BLOCKS."
;; FIXME this seems to fail if the existing block is empty?
;; or at least adding file local variables fails?
(let ((block (org-babel-find-named-block name)))
(if block
(save-excursion
(orgstrap--goto-named-src-block name)
(org-babel-update-block-body content))
(error "No block with name %s" name))))
(defun orgstrap-get-src-block-checksum (&optional cypher)
"Calculate of the checksum of the current source block using CYPHER."
(interactive)
(let* ((info (org-babel-get-src-block-info))
(params (nth 2 info))
(body-unexpanded (nth 1 info))
(body (orgstrap--expand-body info))
(body-normalized
(orgstrap-norm body))
(cypher (or cypher (orgstrap--current-buffer-cypher))))
(ignore params body-unexpanded)
(secure-hash cypher body-normalized)))
(defun orgstrap-get-named-src-block-checksum (name &optional cypher)
"Calculate the checksum of the first sourc block named NAME using CYPHER."
(interactive)
(orgstrap--with-block name
(ignore params body-unexpanded)
(let ((cypher (or cypher (orgstrap--current-buffer-cypher)))
(body-normalized
(orgstrap-norm body)))
(secure-hash cypher body-normalized))))
(defun orgstrap-run-additional-blocks (&rest name-checksum) ;(ref:oab)
"Securely run additional blocks in languages other than elisp.
Do this by providing the name of the block and the checksum to be embedded
in the orgstrap block as NAME-CHECKSUM pairs."
(ignore name-checksum)
(error "TODO"))
(defun orgstrap--get-elvs (&optional from-flv-alist)
"Return the elvs as they are written in the current buffer.
If FROM-FLV-ALIST is not null display the elvs that are in
`file-local-variables-alist'."
(cl-loop
for var in
(if from-flv-alist
file-local-variables-alist
(orgstrap--read-current-local-variables))
when (orgstrap--match-elvs var)
return (cdr var)))
(defun orgstrap-inspect-elvs (&optional from-flv-alist)
"Display the elvs for the current buffer.
If FROM-FLV-ALIST is not null display the elvs that are in
`file-local-variables-alist'."
(interactive "P")
(let ((buffer (get-buffer-create (format "%s orgstrap elvs" (buffer-file-name))))
(elvs (orgstrap--get-elvs from-flv-alist))
(cypher orgstrap-cypher)
print-length print-level)
(with-current-buffer buffer
(emacs-lisp-mode)
(read-only-mode)
(let ((inhibit-read-only t))
(erase-buffer)
;; TODO insert checksum in comment
(insert ";; -*- elvs-checksum: "
(secure-hash cypher (orgstrap-norm (pp-to-string elvs)))
"; -*-\n")
(insert (pp-to-string elvs))
(goto-char (point-min))
(while (re-search-forward "(\\(let\\|defun\\|when\\|unless\\|if\\|read\\)" nil t)
(join-line 1))
(indent-region (point-min) (point-max)))
(local-set-key (kbd "q") #'quit-window)
(goto-char (point-min)))
(display-buffer buffer)))
(defun orgstrap--whitelist-current-buffer ()
"Mark local variable values in the current buffer as safe."
(let ((lvs (orgstrap--read-current-local-variables)))
(customize-push-and-save 'safe-local-variable-values lvs)))
;; extra user facing functions
;;;###autoload
(defun orgstrap-whitelist-file (path)
"Add local variables in PATH as safe custom variable values.
This is useful when distributing orgstrapped files.
Use with a command similar to the following.
Since -batch implies -q, `user-init-file' must be passed explicitly.
emacs -batch -eval \\
\"(let ((user-init-file (pop argv)) (file (pop argv))) (package-initialize) (orgstrap-whitelist-file file))\" \\
~/.emacs.d/init.el /path/to/whitelist.org"
(let (enable-local-variables) ; < 28 don't run orgstrap block
;; `find-file-literally' is broken on 27 so regularize behavior
(with-current-buffer (find-file-literally path)
(orgstrap--whitelist-current-buffer)
(kill-buffer))))Ideally we want to call orgstrap-run-additional-blocks as
(orgstrap-run-additional-blocks "additional-block-name" "checksum-value-hash-thing" "ab2" "cs2")
It probably makes sense to house this in its own orgstrap-aux block or something.
I want to keep the file local variables as minimal as possible, so having another
aux block that could be automatically updated with the names and hashes of additional
blocks would be nice … probably via something like orgstrap-add-additional-block
but it will not go in the local variables because we want there to be some hope of
orgstrap being portable to other platforms outside of Emacs at some point in the
very distant future, so keeping the machinery outside of the org file itself as
minimal as possible is critical.
;;; orgstrap.el --- Bootstrap an Org file using file local variables -*- lexical-binding: t -*-
;; Author: Tom Gillespie
;; URL: https://github.com/tgbugs/orgstrap
;; Keywords: lisp org org-mode bootstrap
;; Version: 1.5.5 (ref:orgstrap.el-version)
;; Package-Requires: ((emacs "24.4"))
;;;; License and Commentary
;; License:
;; SPDX-License-Identifier: GPL-3.0-or-later
;;; Commentary:
;; orgstrap is a specification and tooling for bootstrapping Org files.
;; It allows Org files to describe their own requirements, and
;; define their own functionality, making them self-contained,
;; standalone computational artifacts, dependent only on Emacs,
;; or other implementations of the Org-babel protocol in the future.
;; orgstrap.el is an elisp implementation of the orgstrap conventions.
;; It defines a regional minor mode for `org-mode' that runs orgstrap
;; blocks. It also provides `orgstrap-init' and `orgstrap-edit-mode'
;; to simplify authoring of orgstrapped files. For more details see
;; README.org which is also the literate source for this orgstrap.el
;; file in the git repo at
;; https://github.com/tgbugs/orgstrap/blob/master/README.org
;; or wherever you can find git:c1b28526ef9931654b72dff559da2205feb87f75
;; Code in an orgstrap block is usually meant to be executed directly by its
;; containing Org file. However, if the code is something that will be reused
;; over time outside the defining Org file, then it may be better to tangle and
;; load the file so that it is easier to debug/xref functions. The code in
;; this orgstrap.el file in particular is tangled for inclusion in one of the
;; *elpas so as to protect the orgstrap namespace and to make it eaiser to
;; use orgstrap in Emacs.
;; The license for the orgstrap.el code reflects the fact that the
;; code for expanding and hashing blocks reuses code from ob-core.el,
;; which at the time of writing is licensed as part of Emacs.
;;; Code:
(require 'org)
(require 'org-element)
<<orgstrap-run-helper-defuns>>
<<orgstrap-dev-helper-defuns>>
<<orgstrap-edit-helper-defuns>>
<<orgstrap-init-helper-defuns>>
<<orgstrap-extra-helper-defuns>>
(provide 'orgstrap)
;;; orgstrap.el ends here
emacs-24 -Q $THIS_FILE
emacs-25 -Q $THIS_FILE
emacs-26 -Q $THIS_FILE
emacs-27 -Q $THIS_FILE
emacs-28 -Q $THIS_FILE
emacs-29-vcs -Q $THIS_FILEemacs-24 -Q orgstrap-minimal.org
emacs-25 -Q orgstrap-minimal.org
emacs-26 -Q orgstrap-minimal.org
emacs-27 -Q orgstrap-minimal.org
emacs-28 -Q orgstrap-minimal.org
emacs-29-vcs -Q orgstrap-minimal.orgBefore running the tests below you need to generate ./orgstrap-autoloads.el.
Newer version of autoload-generate-file-autoloads add functions that may not be
supported by older versions of Emacs. Thus you should run this on the oldest version
of Emacs you will be testing against.
(require 'autoload)
(with-current-buffer (find-file-noselect "orgstrap-autoloads.el")
(erase-buffer)
(let* ((cb (current-buffer))
(fn (buffer-file-name cb))
(generated-autoload-file fn))
(autoload-generate-file-autoloads "orgstrap.el" cb fn))
(save-buffer)
(kill-buffer))versions=( 24 25 26 27 28 29-vcs )
test_files=( test-no-lv-list.org test-lv-list-portable test-lv-list-minimal )
for v in ${versions[@]}; do
[ -d test/emacs-$v ] || mkdir -p test/emacs-$v
for f in ${test_files[@]};do
# uncomment and reorder to debug tests
#f=test-lv-list-minimal
#emacs -Q \
emacs-$v -Q -batch \
-eval "(setq user-init-file (concat default-directory \"test/emacs-${v}/init.el\"))" \
-l orgstrap-autoloads.el \
-eval "(message \"\n%s\"(emacs-version))" \
-f toggle-debug-on-error \
-eval "(add-to-list 'load-path \"$(pwd)/\")" \
-eval "(orgstrap-mode)" \
-eval "(defun orgstrap-test () (error \"test failed.\"))" \
-eval "(orgstrap-whitelist-file \"${f}\")" \
-visit $f \
-eval "(orgstrap-test)" 2>&1
done
done;; create buffer
;; fill buffer
;; set code
;; hash
;; save buffer
;; open in all the other impls having set the checksum as accepted
;; with orgstrap-mode enabled
;; without orgstrap-mode enabled
;; with minimal
;; with portable
;; with noweb
;; without noweb
;; iterate over norm funcs
;; orgstrap-norm-func-name mismatch
;; orgstrap-norm-func-name not in internal list
;; .org extension and mode: org local variable
(defconst orgstrap--test-matrix
`(((orgstrap-mode (nil t))
(lv-type (nil minimal portable))
(noweb (nil t))
;; (comments (nil link noweb)) ; comments aren't actually relevant here I think?
(norm-func ,orgstrap--internal-norm-funcs)
(path-suffix-lv ((".org" . (mode . org))
(".org" nil)
("" . (mode . org))
("" . (mode . nil))))))
"The dimensions of the test files that need to be generated."
)- Blacklist and open, then unblacklist and open. This requires an actual file since we need buffer file name.
- Need a way to test revoke as well.
# -*- orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 8d941e14e89664b834f5b28c070f9f7b0ec55b092b55cc23dd903c010fdaeda5; -*-
# [[orgstrap][jump to the orgstrap block for this file]]
* Bootstrap :noexport:
#+name: orgstrap
#+begin_src elisp :results none :lexical yes
(defun orgstrap-test ()
(if (cl-remove-if-not #'orgstrap--match-elvs
file-local-variables-alist)
(error "elv is still present!")
(message "No local variables here!")))
(message "orgstrap successful")
#+end_src
# -*- mode: org; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 14e85d1213ef7a6739ca6ca7361a227b0a55346d4c7c6457bdd5f7ba91ff5dff; -*-
# [[orgstrap][jump to the orgstrap block for this file]]
* Bootstrap :noexport:
#+name: orgstrap
#+begin_src elisp :results none :lexical yes
(defun orgstrap-test ()
(if (cl-remove-if-not #'orgstrap--match-elvs
file-local-variables-alist)
(error "elv is still present!")
(message "Portable local variables here!")))
(message "orgstrap successful")
#+end_src
** Local Variables :ARCHIVE:
<<local-variables-portable-example>>
# -*- mode: org; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 3008580fd616cdfca904c7508ae023f782585229a87adab33fb8c2d391f89561; -*-
# [[orgstrap][jump to the orgstrap block for this file]]
* Bootstrap :noexport:
#+name: orgstrap
#+begin_src elisp :results none :lexical yes
(defun orgstrap-test ()
(if (cl-remove-if-not #'orgstrap--match-elvs
file-local-variables-alist)
(error "elv is still present!")
(message "Minimal local variables here!")))
(message "orgstrap successful")
#+end_src
** Local Variables :ARCHIVE:
<<local-variables-minimal-example>>
Use flycheck-mode on ./orgstrap.el to checkdoc for melpa.
Don’t forget to run flycheck-package-setup to get better reports.
Before a release run the following block and fix any byte compile errors and warnings. Using Emacs 26 ensures that bytecode is forward and backward compatible.
(ow-run-command "emacs" "-batch" "-f" "batch-byte-compile" "orgstrap.el")
(ow-run-command "emacs-26" "-batch" "-f" "batch-byte-compile" "orgstrap.el")
(delete-file "orgstrap.elc")Run ref:test-matrix-run.
These are not automated at the moment. Run ref:test-portable and do the following for each version of Emacs.
- Accept lvs.
- For Emacs >= 26
orgstrap-clone. - switch to
*scratch* - enable
org-mode orgstrap-init- optional edit block
- optional save to file and test reload the saved file
- For Emacs >= 26 in
*scratch*buffer undo - For Emacs >= 26
orgstrap-stamp - For Emacs >= 26 check that the checksum matches the checksum for this file
- quit
Things that need to be done for a release.
- Bump the version number in the orgstrap.el header comment. You will need to manually retangle after this step.
- Update the changelog.
- Convert the changelog entry to markdown for the GitHub release.
C-c C-e C-s m M.
- Fix missing paren.
Missed due to a duplicate orgstrap block for debug in the same file.
- Update
orgstrap--shebang-bodyto setfind-file-literallyto nil.If you use shebang blocks you should update them so that a call to
find-fileonbuffer-file-nameof the orgstrapped file will continue without prompting the user.
- Ignore errors from
org-set-visibility-according-to-property.Recent changes to
org-set-visibility-according-to-propertyresult in errors if it is called when a file is not inorg-modeand/or when a buffer is inorg-modebut Emacs isnoninteractive.This requires and update to the elvs which wraps the call in
ignore-errorswhich is more space efficient than e.g. testing whether we are non-interactive and also mostlyfoolfuture proof.If you use orgstrap shebang blocks you should updated your elvs.
- Update
orgstrap--shebang-bodyto simplify and improve safety.If you use shebang blocks you should update them so that shebang blocks are not affected by leaking environment variables. See ./shebang.org for a full explication of the operation of each line of shell/powershell code.
- Add ability to update shebang blocks to latest version.
New interactive function
orgstrap-update-shebang-blockupdates an existing shebang block to the latestorgstrap--shebang-body. Internallyorgstrap--add-shebang-blockwas updated to accept an optionalupdateargument which is used to control whether to add a new or update an existing block.
- Update
orgstrap--shebang-bodyso that shebang blocks pass args correctly.If you use shebang blocks you should update them so that they pass the correct args to emacs. See ./shebang.org for more details on the changes.
- Regularize behavior of
orgstrap-whitelist-file.An internal call to
find-file-literallyhas different behavior in 27 vs 28 (see notes about that in ./shebang.org). - Rename all normalization functions to remove use of
..See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=55645 for details.
You will need to update any orgstrapped files to use the new names. The simplest way to do this is to update to to 1.5 and then delete the prop line local variables and then run
orgstrap-init. - Add a check to ensure that only portable symbol names are used.
Symbols with
[.?]appearing anywhere other than at the start of the symbol are banned since their prin1 representation diverges across the 28/29 boundary.If you have orgstrap blocks that contain such symbols you will need to change the symbol names. An error will be raised when calling
orgstrap-add-block-checksumwith a list of symbols that need to be changed.
- Update
orgstrap-initso that it can insert a shebang block.Use 2 or more prefix arguments to add a shebang block when using
orgsgrap-init. For exampleC-u C-u C-u M-x orgstrap-init.
- Remove all
orgstrap-dovariables.The core of orgstrap is not the right place to maintain these. They will reappear under
ow-doin the future. - Add
orgstrap-norm-func--dprp-1.0and make it the default norm func.The default normalization function for orgstrap is now invariant to changes in the docstring for
defun,defun-local,defmacro,defvar,defvar-local,defconst, anddefcustom. This allows improved documentation without requiring the user to re-audit.Note that
orgstrap-norm-func--prp-1.1has NOT been deprecated, but is no longer the default. It is still useful if for whatever reason you want to minimize the elvs. - Add support for batch execution.
The preferred method is to use an org shebang block (see ./shebang.org) It is also possible to maintain an automatically updating list of developer checksums. This approach was deemed to be silly given shebang blocks, however the functionality is retained.
- Add
orgstrap-whitelist-fileto make it easier to mark known safe files in batch.See the docstring for example usage.
- Add
orgstrap-inspect-elvsto inspect the elvs for the current buffer.The command also calculates the elvs checksum for comparison.
Known elv checksums for this release are below. The order is minimal, minimal-noweb, and portable (aka minimal-noweb-eval).
For prp-1.1
446d0c80d72bb89dd149181e6a24eafa011d12d6dc99fad958a03ddebd9a95ad\b294539a74f2a1932d39790d6377a4229bd3d5e84df64d968baa8ff3f85349cc\543e3400c80e2cc7b9bf94b1799d1460b240776c2c7415a2f6c0c7a9507978ef\For dprp-1.0
aa080a6469c22dfe960c43fa3bff3b92a6bc3da9383ec8fcd7d0a019192e7aa0\72c52a3483905aff6b83c6cd2c36899a2a8d1cbc603b6e5ce8c9e98f0dd7b099\9e33bc67b8850147962edcece4ea1193bb6cf3711264a26bde91b1f838912ffc\ - Fix
org-edit-modeso that it now activates correctly. - Fix
orgstrap-initso that it no longer misplaces the link to the orgstrap block if there is already content in the buffer. - Fix
orgstrap-initso that invocation in files with existing elvs updates only the existing orgstrap elv and preserves other elvs. - Fix
orgstrap-initto read and pass the current value oforgstrap-norm-func-namewhen creating local variables.If
orgstrap-norm-func-nameis missing, the default value oforgstrap-norm-funcis used.This prevents klobbering while also providing an easy way to update the normalization function — just update the variable value and run
orgstrap-init. - Fix
orgstrap-norm-funcby always declaring it withdefvar-local. - Update the elvs to handle issues with symlinks and vc mode. The core functionality remains compatible.
- Update the elvs so that
org-confirm-babel-evaluatecan be set by an orgstrap block without having to modify the elvs. - Update the elvs so that they only restore visibility set via
property drawers.
This makes it possible to use the orgstrap block to control initial visibility and narrowing to simplify the presentation of orgstrapped files (and avoid distracting users with the orgstrap machinery).
- Update
orgstrap-initto put the Bootstrap section at the end of the file and to put the elvs in an archived heading inside that.This pattern has been found to be quite effective for a number of different use cases.
- Fix bad defaults on
orgstrap-do-*custom variables.If these are not set to t by default then it is impossible individual blocks to know whether a nil value was intentional on the part of the user or not. Users must set values to nil in their config if they do not want certain sections to run.
- Ignore
orgstrap-block-checksumwhen loadingorg-agenda.If
enable-local-evalis t (following the behavior described in the 1.2.2 changelog), thenorgstrap-block-checksumis not ignored and if the local variable value has not been added to the safe list then the user will be prompted. - Add
orgstrap-do-*variables.Boolean control variables that can be used enable/disable standard functionality/steps needed by org files. See ./do.org for more details.
- Improve behavior of
orgstrap-blacklist-current-file.Now revokes the current buffer checksum by default, this can be overridden by providing a universal argument. The blacklist is immediately saved via
customize-save-variable. - Fix
orgstrap-modeto use the universal argument.Behavior is now correct when
(orgstrap-mode t)is called.
- Add
orgstrap-cloneandorgstrap-stampcommands.orgstrap-clonestores the current buffer and current or orgstrap blockorgstrap-stampcopies the expanded contents and headers of that block to a new orgstrap block in a new file. Useful in cases where users want to duplicate functionality in a new file.NOTE
orgstrap-stamponly works for org version >= 8.3.4 which means that it does not work for versions of Emacs < 26. orgstrap--add-orgstrap-blockaddblock-contentsargument.This simplifies the implementation of stamp, and makes it possible to set the initial contents of the orgstrap block programmatically. There are some lingering issues with indentation that may need to be resolved for this to work seamlessly.
- Fix byte compile bug from destructuring-bind not being aliased.
- Fix for issues with noweb blocks containing multi-line docstrings.
We need this to test
orgstrap-clonewith this readme file. Without it the checksum will not match in the stamped file due to differences in the leading whitespace in docstrings. - Make
orgstrap-revoke-eval-local-variablesobsolete.Replaced by the more compact
orgstrap-revoke-elvs.
- Add
orgstrap-file-blacklistto block eval of specific files.The functionality only works if
orgstrap-modeis enabled.
- Do not run eval local variables when loading org-agenda.
orgstrap-always-evalcan be set totby users who want to evaluate orgstrap blocks in all situations. More granular control is provided by adding the full path of files that should always try to run their orgstrap block toorgstrap-always-eval-whitelist.In all cases, if the global setting for
enable-local-evalis more restrictive then it is honored (i.e., nil will block any execution and the default ~’maybe~ will continue to prompt). - Add ability to revoke previously approved orgstrap-block-checksums.
Rapid revocation of permissions is an important part of any security system. Therefore we now provide a way to revoke all previously approved values for
orgstrap-block-checksumin a single commandorgstrap-revoke-checksums. This command can also be provided with a specific list of checksums to revoke. Another convenience functionorgstrap-revoke-current-bufferis provided that revokes the checksum of the orgstrap block for the current buffer. - Add ability to revoke previously approved eval local variables.
As with revocations for
orgstrap-block-checksumvalues, we also need a way to revoke eval local variables. There is no granular control. Useorgstrap-revoke-eval-local-variablesto nuke all orgstrap eval local variables from orbit.
- Fix bad startup visibility when using orgstrap.
You should run =M-:= =(orgstrap–add-file-local-variables)= to update embedded eval local variables.
Running
org-babel-execute-src-blockchanges the visibility of the tree holding the orgstrap block. As a result the startup visibility of any org file using orgstrap was incorrect. Adding a call toorg-set-startup-visibilityin the unwind forms ensures that startup visibility is correct.
- Add
orgstrap-norm-func--prp-1.1and make it the default norm func.This change does not effect the default behavior of
orgstrap. The reason for the change is to defensively shadowprint-lengthandprint-leveltonilso that if they are somehow non-nil, Emacs will not truncate the contents of the src block prior to hashing. - Mark
orgstrap-norm-func--prp-1.0as obsolete.You should update any files using prp-1.0 to use prp-1.1.
Whenever there is a case where a change in the environment can cause a change in the output of a normalization function there is a risk that it could be exploited.
- Fix
orgstrap--hack-lvto remove itself from the localhack-local-variables-hook.
- Renamed existing
orgstrap-modetoorgstrap-edit-mode.This is a BREAKING CHANGE. Please update your workflows.
- Added the new
orgstrap-modeimplementation.This is a regional minor mode for
org-modewhich makes it possible to use orgstrap without the embedded local variables. This allows for greater security at the expense of portability, depending on the exact use case. By a stroke of good fortune it is possible to use thehack-local-variableshooks to trap and remove the embedded local variables if they are present so that the orgstrap block is not evaluated twice. - Added
orgstrap-always-editas a custom variable.If non-nil then
orgstrap-edit-modewill be automatically activated byorgstrap-mode. orgstrap--add-file-local-variablesupdate existingeval:vars.This change makes it vastly easier to switch between portable and minimal implementations, and should make it easier to switch the normalization function once we get that implemented.
If an existing orgstrap eval file local variable is detected it is removed and the latest version is added. Other
eval:variables are not modified. Note however that the orgstrap eval variable will always be placed first.
orgstrap repository lives at https://github.com/tgbugs/orgstrap.
There are a number of ways to contribute to orgstrap.
- Have an Org file that use
orgstrap? Create an issue or a pull request to add it to the list of examples from around the web. - Encounter a bug? Please submit an issue!
- Feel like writing some elisp? Check out future work for a list of potential projects (TODO status not visible on GitHub).
This guide is for users of Org files that have orgstrap blocks.
This includes Emacs users who have their own configs as well as users who might be encountering Emacs for the first time.
This guide is for developers who want to use orgstrap in their own
org files.
It covers workflows for development, distribution, and maintenance of orgstrap files.
It also covers best practices and effective strategies for making Org files accessible to users.
A key issue that orgstrap must contend with is the fact that there is only one global namespace for all elisp functions. Variables are not an issue for orgstrap because buffer local variables provide sufficient separation.
To this end there are two conventions that orgstrap and orgware follow.
The ersatz namespaces orgstrap--- and ow--- may be used in any
orgstrap block. Developers may assume that any such definitions will
remain unchanged at least until another orgstrap block runs. Clearly it
is impossible to know for sure that no one else will use a function
with the same name ow---you-re-standing-on-my-toe-! that has different
behavior. We can make an attempt to keep a record of all known ow---
function names, but ultimately it is up to the user to run a check.
This guide is for developers who want to contribute to the core
orgstrap implementation or documentation.
In short. If you use orgstrap.el don’t accept the elvs.
Suggestions for users. If you run an orgstrapped file via -Q or similar, then you have to accept the file. The elvs probably aren’t going to include package initialize and require orgstrap, but maybe they could. Essentially, in -Q mode the user should know that they have to initialize it all themselves (orgware could do it for them).
When you aren’t running in -Q mode and you DO have orgstrap.el installed on your system then I strongly suggest that you should NOT accept, or even actively remove, any and all approved elvs and use orgstrap-mode since it has a number of features that can enhance the security of execution such as blacklists etc.
There is a big difference between using a script to install a program directly from the internet and using a script to ask the host system to install a program.
Even if you audit a random script from the internet it is unlikely that you will be able to do due diligence. On the other hand, if you ask your system package manager to install something for you, there is a much better chance that it has at least been somewhat audited, and there is usually an existing process for getting a package into the system which helps to mitigate certain types of attacks.
To give a military example it is the difference between inspecting and accepting a package from a random person because they say you asked for it yesterday (maybe you did!) versus only every allowing packages to come through procurement. You are much less likely to get a bomb or a packaged rigged to exfil data if you go through procurement because there is an established process for how to do things and that process enshrines generations experience about how to not get blown up by the pizza guy.
So, if you are writing instructions that require a certain tool, it is better to tell whoever is following them to ask procurement to get the tool for them than to tell them to going out to the hardware store and get it themselves, or worse, give them the address of a random tool delivery man who happens to be a good buddy of yours. Even if everyone involved is trustworthy those kinds of relationships are much easier for some third party to compromise and use for their own purposes.
The obvious corollary when you are the user rather than the author, is that if you encounter instructions that ask you to directly install software from a random place you should be suspicious, even, perhaps especially, if that random place is housed within a larger reputable site. If you’re not in a hurry, ask for the software to be packaged, or package it yourself so that it can go through the process.
One problem that orgstrap has is that an orgstrap block can modify the Emacs configuration. Modifying the config is often critical to get the desired behavior for the particular target user population. This is not an issue if the users do not use Emacs for anything else. However, if the user opening the file is an Emacs user then blocks that modify the configuration are a serious problem.
There are three ways around this issue: one a standard convention for
naming a variable to control evaluation of config related code or two
always use emacs -q and of course three a combination of both.
In the first case a set of standard conventions for variable names can be used to control whether some, or all configuration variables are set. However, this can only attain the status of a best practice because orgstrap blocks run arbitrary code and there is no way to enforce the convention (thus why it is in this section).
One convention that I have been testing is to …
In the second case we suggest that users always open orgstrapped files
with emacs -q. This is as close as we can get to having a sandboxed
execution environment. If the user also wants to load their config
then they would have to use -l as well. This is a pain, but without
converting all of the variables into buffer local variables this is
unlikely to work.
Another major drawback of using emacs -q is that there are many
more advanced features enabled by orgstrap.el that cannot be used
without running emacs -q -l orgstrap.el or some equivalent. This
adds significant complexity to the command line invocation. There
is no easy way to work around this since the features of interest
are ones that must be available before the orgstrap block is run.
Another approach is to use emacs -q -f package-initialize.
There is also the issue of how to handle potential name collisions.
In summary, Emacs and orgstrap are very sharp tools. I have tried to provide a bit of protection via the checksum mechanism, however if you are an Emacs user, you should probably always check the orgstrap block to make sure that it won’t completely klobber your config.
If you are authoring a file that uses orgstrap, it is friendly to put
any config related code in its own block so that it can be controlled
globally via the-orgstrap-variable-name-to-be-determined.
- apinatomy.org
An executable Org file for running pipelines that build computational models of anatomy. - queries.org
An org file set up as an interface to query knowledge bases that is configured to provide a familiar (cua+) interface for users who may be unfamiliar with Emacs. Has examples for loading packages, and of one way to hide configuration to avoid information overload. A simpler version of the file (which leverages the same orgstrap block because the files are distributed together) is at scratch.org. - welcome.org
An org file that acts as a static welcome page for a docker image, with links to other interactive org files. Has an example of how to use narrowing to hide the orgstrap machinery from the user.
Configurations for services that can be reused across orgstrap files.
Include these in part or whole to simplify common orgstrap workflows.
As mentioned above, the primary use case fororgstrap was that I was sick of having
to work around the limitation that I had to do one of four things. I either one, had
to remember to eval the source block containing defuns used later before I could
eval other source blocks that used those functions in headers, or two, had to put those
functions in init.el, destroying the ability to use org files as standalone self describing
portable and reusable computational artifacts, three, had to copy and paste verbose
elisp bits around to achieve what I wanted, or four, had to double tangle a file so that
the results of the first tangle could be loaded before calling the second tangle so that
the functionality would be available (this also produces the situation described in three).
Furthermore, it is hard for humans to follow all the steps needed to get everything
working – even when ‘everything’ is just invoking C-c C-c on a single source block
I still forget. This can lead to bad things if some of those source blocks were
interdependent, or proceeded with a nil, etc.
File local variables to the rescue!
I’m slightly embarrassed to say how long it took me to arrive at the current solution.
I had known for quite a while that file local variables are a pathway to abilities that
the evils of arbitrary code execution, but it didn’t click that all I was looking for was
the ability to just run some arbitrary elisp code every time a particular file was loaded,
which of course is exactly what file local variables are for.
The only question then was how to avoid the very real dangers of enabling arbitrary code
execution of plain text. Actually it was more along the lines of “How can I keep org-babel
happy without also pwning myself?” Fortunately org-confirm-babel-evaluate can be customized
to be a function that accepts the body of the code to be evaluated. Therefore we can do the
following.
When creating a file.
- Hash the block to be run before distributing the file.
Make sure to test if there are any changes to the header.
For example I have a bad habit of accidentally setting
:noweb no-exportincorrectly without the dash and that will prevent the checksum from updating if a nowebbed block changes. - Embed the checksum in the file local variable property line. The property line is highly visible as the first line of the file. This makes it easy for users to verify that the embedded checksum matches a known independent checksum (running step 2). Thus if the embedded checksum does not match a known checksum the user will notice, and if the code to be executed does not match the embedded checksum then the user will at least be prompted by org-mode to run the block even in the case where they accepted the file local variables. Emacs also prompts for verification of the property line value which is another opportunity for the user to check.
- Publish the checksum independent of the file itself.
It is trivial for someone to change the contents of the orgstrap block
and rerun
M-xorgstrap-add-block-checksum. Therefore known checksums need to be published independent of the files themselves.
When running a file.
- Audit, accept, and store permanently the eval file local variables. Storing audited variables permanently is critical for improving signal to noise so that unexpected mismatches retain their salience and can elicit the correct response (i.e., suspicion).
- Audit the orgstrap block
I assume most people are not going to do this. However, one of the advantages
of the current approach is that the same orgstrap blocks can be reused across
multiple files which reduces the audit load such that one only needs to review
unique orgstrap blocks, not all files. [fn::NOTE there are certain patterns inside
blocks that are NOT safe to accept because they introduce a level of indirection
that orgstrap cannot verify. Examples of these kinds of dangerous blocks are ones
that make any reference to other blocks in the file via some means other than noweb.
This isn’t really surprising, and for use cases where
org-babel-execute-src-blockis called multiple times on different blocks, the default execution protection will work. In addition, any blocks which want to run automatically without prompting should use theorgstrap--confirm-evalfunction (see Future work).] - Verify that the embedded checksum matches the independent checksum. A known embedded checksum matching the content checksum only means that the content matches the content observed by the provider of the independent checksum (assuming no hash collisions).
- Observe whether org-mode complains that the orgstrap block has changed.
The first use case for orgstrap beyond personal use was to create a
stand alone application that would allow a user to run, modify, and
create SPARQL queries running from a local server[fn::The file itself
is in a private git repo at the moment, but the functionality will be
extracted and made public, and the repo itself will be as well.]
The primary users had no prior Emacs experience. Under supervision via a video call they were able to follow the instructions to download Emacs, download a zip of the file plus data and additional software, unzip, and click on the file (which opened in Emacs by default), and accept the local variables.
There were a couple of hiccups. In one case Java for macos was missing. In the other case the built-in version of Org mode was not correctly replaced so Emacs had to be restarted. In the second case there was also an issue with multiple windows being opened and the confirmation window for the local variables thus being hard to find.
Even with the slowdown they were able to get up and running within about 20 minutes. This is an enormous improvement over previous attempts which involved many hard to follow instructions that could take nearly 3 hours to complete and debug.
Emacs 29 introduces the long desired command line option--init-directory=DIR. This solves a number of issues related to
sandboxing the impact on existing Emacs configurations of using
orgstrap for various things. That is somewhat secondary however.
More importantly, after a significant amount of experience working
with this setup, it seems fairly clear to me that the default behavior
when using orgstrap shebang blocks is that orgstrap configuration
files and more importantly packages should be kept separate from the
default user-emacs-directory to avoid a wide variety of issues.
Probably add this to ow-enable-use-packages with options to use
something like ~~/.config/orgstrap/~ or ~~/.orgstrap.d/~ and
additionally to sandbox packages for the whole file. I’m thinking that
probably won’t be necessary for most use cases.
This was part of the issue, though it wasn’t that it was not being
reset, it was that orgstrap-init did not source the default value
because orgstrap-norm-func was incorrectly marked as a global
dynamic variable instead of as defvar-local.
The other part of the issue was the we were not using the current
orgstrap-norm-func-name for the buffer during orgstrap-init.
Even all that wasn’t quite right. orgstrap-norm-func has to be
overwritten before the checksum is added for existing files, otherwise
the stale value will persist for files where local variables were
actually accepted instead of ignored.
base64 (or whatever) encode a compressed data blob, stick it in an
archived heading and then add --pack and --unpack and --repack
or something equivalent. This is probably the most reasonable way
to manage distributing org files that have dynamic data associated
with them, such as an sqlite database or something.
xz -zk file
base64 -w 127 file.xz > file.xz.base64 # 127 is a nicely sized prime
xz -d file.xz
xz -dc file.lz > /dev/null(math-prime-test 109 9999)
(math-prime-test 113 9999)
(math-prime-test 127 9999) ; this oneThese are likely to be of interest since they can also be used to tar a whole
directory, at which point it is possible to deal with the dissociation issue.
dired-compress-files-alist
dired-do-compress
dired-compress-file
base64-encode-region
base64-decode-region
Well, that was productive for figuring out what was going wrong.
Turns out that :ARCHIVE: sections are still seen by flyspell and by
auto-complete. narrow-to-region seems to have the behavior we want
but it doesn’t do multiple. For some reason archived text is still
being searched by auto-complete-mode which causes massive slowdowns.
I have looked into modifying org-hide-archived-subtrees so that
isearch does not open and search inside, however it does not seem to
make any difference.
apparently flyspell doesn’t honor read-only so we have to use something else and it also seems to ignore the first regexp I list here, so no good solutions thus far, I still think that narrowing to before and after are the best solution …
(add-to-list 'ispell-skip-region-alist '("^\\* data :ARCHIVE:$" . "=$"))
(add-to-list 'ispell-skip-region-alist '("^#+begin_data$" . "^#+end_data$"))
(auto-complete-mode 0)
(rainbow-delimiters-org-mode 0)
(flyspell-mode 0)(use-package zones) ; not part of the core so hard to make use of#+startup: showall
* data :ARCHIVE:
:PROPERTIES:
:visibility: folded
:END:
put the big stuff here
* Bootstrap :noexport:
#+begin_src elisp
(let ((inhibit-read-only t))
(add-text-properties 21 44543976
'(read-only t)))
#+end_src
vc is called via find-file-hook which explicitly runs after
hack-local-variables which explains how we are getting in so early
with the orgstrap blocks, a solution has been found
orgstrap-block-checksum-sources alist as a custom variable so that it is
easier for people to know what came from where in a summary even though
we have the revoke functionality.
Record a series of short screen casts to illustrate common orgstrap authoring and consumption workflows.
One major usability feature would be to figure out how to display the full body of the orgstrap block in the other window when the confirm local variables dialogue was presented. It seems easy enough when orgstrap.el is installed, however it seems like it might be hard to implement a minimal version, but maybe not, it is basically just save excursion and rearrange so that the local variables confirm buffer and the orgstrap block are the only two windows visible.Another issue is whether it is possible to do this in Emacs < 27, since it is not possible to switch out of the confirm dialogue.
Probably use org-babel-expand-src-block via call-interactively.
This will eat into the elvs budget. This also addresses some of the
security concerns.
The answer is that orgstrap isn’t the place to handle these issues beyond providing documentation on best practices.
The best practice for this is to use the orgstrap block in a sane manner. For interactive use orgstrap blocks should include variable settings and defuns at most. Little to no actual computation should be done at that stage.
Longer running processes, such as tangling or building etc, should be masked
in (when noninteractive body ...). That allows orgstrap blocks to make
the functionality defined in the file available via command line arguments
(including editing via ./orgstrapped.org –edit).
In this context the evolution of orgstrap do will be to provide a library to make command line interaction with orgstrapped files discoverable. We are most of the way there because there is already an implementation of docopt for elisp.
Since all the conventions for how this is done are defined locally by each file, you could
in principle rename the special block as you see fit, perhaps from orgstrap to main if
you need to pretend that the file is actually c source code with some special syntax.
However, this is not advisable if you care about portability since it depends on an
implementation detail of orgstrap.el which is not required by the specification. Namely
that orgstrap-orgstrap-block-name is not required as one of the prop line local
variables. Given the desire for the orgstrap machinery to be as unobtrusive as possible,
it is unlikely that support for an arbitrary name for the block will be added to the spec.
That said, it is worth considering how and whether to update the spec so that a conforming implementation could do this if it wanted to. All the change does is move a convention to an optional variable. Maybe a compact variable such as orgstrap-bn could be specified as an optional prop line variable, and if absent the orgstrap block name defaults to orgstrap, otherwise the block name searched is the value of the variable.
Still not 100% sure about this. It would increase the complexity of the implementation for sure. It will require updating how and when we populate the link to orgstrap block, and makes auditing more difficult. It also opens up a way to trick the user, namely by having a link to an innocent looking orgstrap block, and no convention set in the prop line, or maybe even having it in the prop line (people are habitual and assume things are not present when they are), and then using setq-local, or some other means to change the block name to a malicious block elsewhere in the file. Implementations would have to know to check for this and fail if it was detected. Basically we would have to specify that if a block named orgstrap is present in a file when orgstrap-bn is present and not set to orgstrap, then it is a fatal error and orgstrap will not continue. Sticking this in the 900 or so chars we have left for further features seems like it would be a stretch, but might be possible.
orgstrap currently does not check all the headers or vars properties that materialized
onto a source block we probably need to do this. For the time being users need to check
for any hidden header properties that might be attached if the source block is buried
within a tree somewhere. See org-babel-one-header-arg-safe-p for one way that this
might be implemented in the elvs.
This is more effectively implemented in ./shebang.org because it bypasses the orgstrap checksum entirely. It is still secure because the user has to intentionally run the org file as a script. The overall complexity for the user is lower as well since they do not have to maintain or worry about the batch helper file, and the churn in the batch helper file is also eliminated. This section is retained for the record.
There are a number of use cases for being able to process orgstrapped files in batch mode.
For example being able to load a file and have it automatically tangle itself vastly simplifies
a number of different workflows. emacs -q --batch -l orgstrap-known-safe.el my-file.org seems
like a reasonable approach. Essentially orgstrap-known-safe.el needs to contain the safe eval
blocks and the audited hashes so that local variable prompts will not be triggered since they
always return no when in batch mode. One additional feature is be to able to pass the checksum
on the command line. The eval variables would still have to be loaded in some way, but avoiding
the need to open and edit orgstrap-known-safe.el for each new file, and possibly edit it again
to remove the approval in the future.
(let ((buffer (find-file-noselect "orgstrap-batch-helper.el.example")))
(with-current-buffer buffer
(erase-buffer)
(let (print-length print-level (print-escape-newlines t))
(insert ";;; -*- mode: emacs-lisp; lexical-binding: t -*-\n\n")
(insert ";;; add audited checksums here\n\n")
(insert "(setq-local\n orgstrap-audited-checksums\n '(\n\n ))\n\n")
;; TODO insert test file block checksums
(insert ";;; set audiated checksums as safe local variables\n\n")
(insert
(pp-to-string
'(mapcar (lambda (checksum-value)
(add-to-list 'safe-local-variable-values (cons 'orgstrap-block-checksum checksum-value)))
orgstrap-audited-checksums))))
(insert "\n;;; helper local variables\n\n")
(cl-loop
for local-variable in '((orgstrap-cypher . sha256)
(orgstrap-norm-func-name . orgstrap-norm-func--prp-1-1)
(orgstrap-norm-func-name . orgstrap-norm-func--dprp-1-0))
do
(let (print-length print-level (print-escape-newlines t))
(insert (prin1-to-string
`(add-to-list 'safe-local-variable-values ',local-variable)))
(insert "\n")))
(insert "\n;;; known eval local variables\n\n")
)
(cl-loop
for (eval-local-variable elv-checksum) in
(cl-remove-duplicates
(cl-loop
for block-name in '("example-noweb-no" "example-noweb-yes" "example-noweb-eval")
append
(cl-loop
for minimal in '(nil t)
append
(cl-loop
for norm-func-name in
'(;; orgstrap-norm-func--prp-1-0 ; deprecated do not include
orgstrap-norm-func--prp-1-1
orgstrap-norm-func--dprp-1-0)
collect
(let ((info (save-excursion
(orgstrap--goto-named-src-block block-name)
(org-babel-get-src-block-info))))
(setf (nth 6 info) "hrm")
(list (orgstrap--lv-command info minimal norm-func-name)
(secure-hash
orgstrap-cypher
(orgstrap-norm
(let (print-quoted print-length print-level)
(prin1-to-string (orgstrap--lv-command info minimal norm-func-name)))))
)))))
:test #'equal)
do
(with-current-buffer buffer
(let (print-length print-level (print-escape-newlines t))
(insert (prin1-to-string `(add-to-list 'orgstrap-known-elvs ,elv-checksum)))
(insert "\n")
(insert (prin1-to-string
`(add-to-list 'safe-local-eval-forms ',eval-local-variable)))
(insert "\n"))))
(with-current-buffer buffer
(save-buffer)))emacs -q --batch \
-l orgstrap-batch-helper.el \
--eval "(message \"\n%s\"(emacs-version))" \
--eval "(defun orgstrap-test () (error \"Test failed.\"))" \
-f toggle-debug-on-error \
--visit test-lv-list-minimal \
--eval "(message \"done\")" 2>&1
emacs -q --batch \
-l orgstrap-batch-helper.el \
--eval "(message \"\n%s\"(emacs-version))" \
--eval "(defun orgstrap-test () (error \"Test failed.\"))" \
-f toggle-debug-on-error \
--visit test-lv-list-portable \
--eval "(message \"done\")" 2>&1In principle the simplest way to do this is to use the :cache yes header on a block.
However, unless the state is persisted into a users init.el file or equivalent, then
the file would need a way to know that it had not been run when opened again in a new
Emacs session. Similar issue with opening the same file in multiple Emacs sessions at
the same time. The block simply will not run again if the cached result is present.
Therefore, since :cache yes by itself is a dead end for ensuring that functionality
is always available any time a file is loaded there are a couple of options.
- Persist to
init.el. This is evil. - Request to tangle and install as package. A variant of this is simply to use package.el to install the desired functionality in a persistent way in combination with accept klobbering.
- Figure out how to transparently wrap an elisp block in
unless. - Advise
defun(say what!?)? @@comment: TERROR@@ - Figure out how to un-cache a block when Emacs exits. This will fail in nasty, unpredictable, and hard to debug ways.
- Set
:cache (if (boundp 'orgstrap-already-run) "yes" "no"). This ALMOST works. If:cache noembedded the sha1 sum then we would be golden. This seems like the best bet. - Accept klobbering.
- Advise org-babel-eval to run with org-babel-sha1-sum even when cache is not set to yes
Another possibility would be
- put checksums of orgstrap blocks that have been run in a list
- use a special header arg
:run-once yesto mark blocks that should not run if their checksum is already on the list.
If used with multiple blocks/multiple revals this would make it possible to run only a subset
When bootstrapping a new system there are many times when want to create a
file only if it does not already exist. The :tangle header does not support
this use case, but we can implement it anyway using the example below.
#+name: orgstrap
#+begin_src elisp
(defun tangle-once (path) (if (file-exists-p path) "no" path))
#+end_src
#+begin_src bash :tangle (tangle-once "./path-to-tangle")
echo lol
#+end_src
# I think I've seen this before but you apparently can't have ,#+end_src on the line before #+end_src ... fun bug
There must be only a single one of those blocks so that the rest of the blocks can safely use the functions defined in the orgstrap block.
A single elisp block is sufficient to enable nearly all use cases involving tangling source blocks to file without having to fight the prompts. However, it is very much not sufficient for any use cases that involve other languages. This is particularly an issue for org files that want to bootstrap whole systems.
The simplest solution to me seems to be to add a second prompt variable which is
an alist of source block checksums and names[fn::the names are not technically required
but are for human readability]. As soon as the orgstrap block is run
orgstrap--confirm-eval is no longer needed and can be replace with a function
that validates the other blocks from the prompt variable.
This seems like a tractable approach, but also over complicated because it is surely
easier in a case like this where blocks are very unlikely to be reused across org files
to simply (setq-local org-confirm-babel-evaluate nil) and tell people to audit the
whole file. The alternative in that case might be to hash all the source blocks and
validate all of them at once at the start of the orgstrap block. This might need some
additional machinery, not entirely sure, maybe just have orgstrap-all-blocks-checksum
that can be used in cases like that. The advantage here is that the core of the process
can be verified once and then the documentation around it can change and grow as needed.
See https://github.com/tgbugs/laundry for the start of work implementing org mode in Racket that would make it possible to have a practical discussion about how to approach Org babel beyond Emacs.
The spec may need to be extended to allow multiple orgstrap blocks
with the same name. We might need to make a provision for the
implementation specific language blocks to allow multiple names
with the block for the main implementation language getting priority,
however that may add significantly more complexity than e.g. just
adding orgstrap-elisp, orgstrap-racket, orgstrap-{lang} as
block names that are also searched.
This is unlikely to be useful any time soon given that there aren’t full implementations of org mode outside of Emacs. If some setup is written in another language the best approach is to call the other block using the orgstrap block.
If at some point it becomes possible to use another language in the top level of org, then I imagine that such functionality will go in as a property or a file local variable or something like that. Early candidates for other languages that might be supportable are other lisp dialects. With such mechanisms in place, it would be relatively straight forward to lift the restriction on the language type, or rather, to include the normalized language name as part of the hash, or possibly to include it as a prop line local variable. More exploration would be required, but until there is some other implementation that has an extension language that is not elisp, this is a moot point.
One additional source of noise in addition to comments are defun and
defmacro docstrings. These should be dropped from the tree if they are
present. This is now partially implemented via orgstrap--dedoc.
The issues in 1.2.4 with inconsistent noweb block indentation. Is yet another reason for this. There is some lingering inconsistency in exactly how many leading spaces are included when nowebbing in elisp forms that include a multi-line string. This is one of those extremely annoying but hard to fix issues related to noweb and leading whitespace.
Reorder the expressions used in the orgstrap block alphabetically (or something like that) according to a deterministic rule, but not in a way that changes program semantics. For example a function definition cannot be moved after a top level invocation of that function.
- defuns with different names can be reordered
- defuns with the same name can be reordered as a block but cannot internally be reordered because the order of shadowing matters
- While it might be nice to completely erase the names of functions as well
as internal variable names, this would make it trivial to shadow existing
function names in ways that are malicious. The exact names matter, so we
have to preserve them. Also the cost of not being able to tell that
(lambda (a) (+ a a))and(lambda (b) (+ b b))are the same seems fairly small. - One potential approach is to lift all defuns to the top, and then function calls or whatever the more generic procedure invocation means. The simple local rule is that all definitions must occur before usage except in the case where there is a shadowing event that happens after a first invocation. This is annoying, but if a call to a function happens before that function is defined we have to assume that the call is calling some other function and those statements cannot be reordered. So the ordering is calls to functions with names matching any later defuns or any later assignment. Then defuns and assignments, finally procedure invocations which might also include assignments. I get the sense that this is covered under some part of compiler theory but can’t quite put my finger on it.
(use-package org-ref) ; for ref:
(use-package ox-gfm) ; simplify export of changelog for release
(use-package flycheck-package) ; linting for melpaOne potential way to simplify command line execution of orgstrapped files is to write a wrapper that is aware of the internal variables to control various aspects of orgstrap behavior, such as config, testing, tangling, setup, build/make, dependencies, exporting, publishing, etc. Essentially we wind up with a set of functionalities that are commonly needed and set conventions for how to run only the desired subset.
Dealing with dependencies between functionalities is probably out of scope. The assumption is that Emacs and system packages are required for all further functionality, but beyond that it is not clear.
Ideally would like to move toward using ert for testing in this file.
Generally it would be good to have an established workflow for testing
orgstrap files beyond just nowebbing into when orgstrap-do-test.
As those workflows become established it would be nice to have a set of minimal wrappers for the various CI systems that will be sufficient to kick off the testing. See https://github.com/purcell/nix-emacs-ci for one way we might approach this.
This saves about 300 chars over the current minimal version.
(cl-defun orgstrap--length-of-sexp-at-point (&aux print-level print-length (print-escape-newlines t))
(interactive)
(message
"%s"
(length
(prin1-to-string
(read
;; (org-babel-get-src-block-info)
(thing-at-point 'sexp))))))
(defalias 'length-of-sexp-at-point #'orgstrap--length-of-sexp-at-point)(let ((n "8.2.10") ; need
(a (org-version)) ; actual
(org-confirm-babel-evaluate #'orgstrap--confirm-eval)
(obs (org-babel-find-named-block "orgstrap")))
(or
(fboundp #'orgstrap--confirm-eval)
(not n)
(string< n a)
(string= n a)
(error "Org too old! %s < %s" a n))
(defun orgstrap-norm-func--prp-1-1 (body)
(let (print-quoted print-length print-level)
(prin1-to-string (read (concat "(progn\n" body "\n)")))))
(unless (fboundp #'orgstrap-norm)
(defun orgstrap-norm (body) (funcall orgstrap-norm-func-name body)))
(unless (fboundp 'orgstrap--confirm-eval)
(defun orgstrap--confirm-eval (lang body)
(not (and (member lang '("elisp" "emacs-lisp"))
(eq orgstrap-block-checksum
(intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))))
(unwind-protect
(save-excursion
(goto-char obs)
(org-babel-execute-src-block))
(org-set-startup-visibility)))See ./defl.org for an awful hack.
Not sure exactly which function is causing this but getting the checksum of the block seems like it might be the one, unfolding and not restoring the state.The issue was in orgstrap--with-block because goto-char will cause
outlines to open and we have to use org-save-outline-visibility with
use-markers set to ensure that folding is restored.
save-restriction was not what we needed.
https://stackoverflow.com/a/44158824 put me on the right track.
(defcustom orgstrap-blacklist-files nil "List of files that should not be orgstrapped." ;; TODO but that should still have orgstrap-edit-mode enabled? :type 'list :group 'orgstrap)
or rather not breaks, but tries to co-exist with it which duplicates the line which seems to break the ability to detangle among other things
What to do if we are in a buffer that has an orgstrap block? I think the answer is to check for the local variables, and if they are present, do nothing, if they are absent run the block?
prin1-to-string has inconsistent behavior when the string in question contains
an escaped tab character \t. This can make checksums inconsistent.
This is now done as part of orgstrap-inspect-elvs.
orgstrap-mode. When the block
checksum has not been confirmed orgstrap--hack-lv-confirm might
filter/skip the checksum. I don’t think this is actually what I
observed (see below).
I can’t seem to reproduce this issue using the following.
emacs -q -l orgstrap-autoloads.el \
-f toggle-debug-on-error \
--eval "(add-to-list 'load-path \"$(pwd)/\")" \
--eval "(orgstrap-mode)" orgstrap-minimal.orgI’m fairly certain that the issue that I was actually observing was the fact that when there is a checksum mismatch, then nothing is displayed and the block asks you to eval it – you should decline and inspect the block.
The issue it seems is actually that the block that we are being asked to evaluate is not visible in the other window. So I have added a todo item for that.
This is much easier now that I have put all the normalization and hashing machinery in a single elv, and it no longer needs to rely on the version of the block, just the version of the normalization function, which is currently embedded along with everything else.
If orgstrap is installed, then having a orgstrap-mode which could add itself to org-mode-hook when enabled would supersede the local variables list implementation, or possibly just do it when the local variables version was absent. Disabling the local variables when orgstrap-mode is detect to be on might be possible, but without it the user would have to explicitly decline the variables or risk having the orgstrap block run twice. Going to have to get this resolved before doing any announcements since it could leave users with quite a bit of pain if we don’t get that check in before people start using it.
Before save hook and/or before commit hook to automatically update the block checksum.
Woo! https://melpa.org/#/orgstrap gh:melpa/melpa/pull/7111
This is essentially what is implemented via orgstrap--have-min-org-version.
The choice to prefer the minimal local variables is left up to the user and
is controlled via the orgstrap-use-minimal-local-variables custom variable.
The only time when minimal is not used is if the version of org that is drafting
the file is too old (i.e. < 9.3.8).