Papis.el

Introduction

Motivation

The main motivation of this package is to make it easy to interact with emacs and papis, with a focus to org-mode.

If possible, we don’t want to reinvent the wheel, so this project should try to play well with the very good packages like org-citation, org-ref, org-noter, citar…

Disclaimer

If you’re an emacs lisp hacker, feel free to chip in. Otherwise this project should be treated as β software and it might just completely change in the future.

What is implemented

I have implemented some functions that I use in my day-to-day research and worfklow.

At some point they will get better documented…

org-noter compatibility

We have found that papis works quite well with org-noter since org-noter is quite simply designed. You can get away with the following configuration

(defmacro @papis-query ()
  `(interactive ,papis--query-prompt))

When opening a pdf file of a papis document you can just type M-x org-noter and if there is notes.org file in there then org-noter will open a session for you.

Of course your notes file should be an org file, which you can do by setting notes-name equal to notes.org in your papis configuration.

org-transclusion compatibility

You can put this into your configuration file in order to let org-transclusion know about papis

(require 'cl-lib)
(defun papis-org-transclusion-add-papis-id (link plist)
  (when (string= "papis" (org-element-property :type link))
    (let* ((id (org-element-property :path link))
           (doc (papis--from-id id))
           (notes-path (papis--ensured-notes-path doc))
           (new-link (with-temp-buffer
                       (insert "file:")
                       (insert notes-path)
                       (beginning-of-buffer)
                       (org-element-link-parser))))
      (org-transclusion-add-org-file new-link plist))))
(cl-pushnew 'papis-org-transclusion-add-papis-id
            org-transclusion-add-functions)

and now you can have a document with many notes of a topic and then

* Important papers
#+transclude: [[papis:d27eac97f9dab8e63d1ceeddc41bb8ff][This and that paper]] :expand-links 
#+transclude: [[papis:0858c8b2885089446f1647ad9bb80a41][A fifth-order perturbation comparison of electron correlation theories]] :expand-links

And to add these kind of lines you can use this function

(defun papis-transclusion-insert (doc)
  (interactive (list (papis--read-doc)))
  (insert "#+transclude: [[papis:")
  (insert (papis-doc-id doc))
  (insert "][")
  (insert (papis-doc-get doc "title"))
  (insert "]]")
  (insert " :expand-links")
  (org-transclusion-add))

Implementation

papis.el is written as a literate program.

Generalities

We interact with papis through the papis’ json exporter.
We use org-links to get information directly from papis.

The libraries that we will need are therefore:

(require 'ol)
(require 'json)
(require 'f)

(defgroup papis nil
  "Official papis package for emacs"
  :group 'papis
  :prefix "papis-"
  :link '(url-link :tag "Github"
          "https://github.com/papis/papis.el"))

Variables

(defvar papis--temp-output-file nil
  "This variable holds the papis temporary output file where the json
  output is dumped")

(defcustom papis-binary-path
  "papis"
  "The binary path for papis.
   You might have papis installed for instance in some
   virtual environment"
  :type 'string
  :group 'papis)

(defcustom papis-read-format-function
  #'papis-default-read-format-function
  "Function taking a papis document (hashmap) and outputing a
   string representation of it to be fed into the reader."
  :group 'papis)

(defcustom papis--query-prompt
  "Papis Query: "
  "The prompt to show users in order to accept a query
  "
  :type 'string
  :group 'papis)

You can set the

`papis-library`

You can set the main library used in papis by setting

(setq papis-library "my-other-library")

(defcustom papis-library
  nil
  "papis library to be used in commands.
   If it is set to nil then the default library of your system will
   be used.
  "
  :type 'string
  :group 'papis)

Document

(defun papis-doc-get-folder (doc)
  (papis-doc-get doc "_papis_local_folder"))

(defun papis-doc-id (doc)
  (let ((id (papis-doc-get doc "papis_id")))
    (unless id
      (error "Document '%s' does not have an id!"
             doc))
    id))

(defun papis-id-query (doc)
  (format "papis_id:%s" (papis-doc-id doc)))

(defun papis--get-file-paths (doc)
  (mapcar (lambda (f) (concat (papis-doc-get-folder doc) "/" f))
          (papis-doc-get doc "files")))

(defun papis-doc-get (doc key &optional default)
  (gethash key doc default))

(defun papis--get-ref (doc)
  (papis-doc-get doc "ref"))

(defun papis--doc-update (doc)
  (let ((folder (papis-doc-get-folder doc)))
    (papis--cmd (concat "update --doc-folder " folder))))

Commands

Introduction

Most papis commands will need a query, the macro @papis-query will take care of having the same query prompt in all commands.

(defmacro @papis-query ()
  `(interactive ,papis--query-prompt))

Issuing commands to the shell

The main interface with papis commands will be papis--cmd which is a function intended for library writers.

(cl-defun papis--cmd (cmd &optional with-stdout)
  "Helping function to run papis commands"
  (let* ((lib-flags (if papis-library
                        (concat "-l " papis-library)
                      ""))
         (sys (if with-stdout
                  #'shell-command-to-string
                #'shell-command))
         (full-cmd (format "%s %s %s" papis-binary-path lib-flags cmd)))
    (message full-cmd)
    (funcall sys
             full-cmd)))

`papis-query`

A papis document object is represented in papis.el as a hashtable, and the command that turns a query into a list of hashtables is papis-query. This is done via the papis’ json exporter, i.e., we query python and get a json document with the documents that emacs reads in.

(defun papis--json-string-to-documents (json-file)
  (let ((json-object-type 'hash-table)
        (json-array-type 'list)
        (json-key-type 'string))
    (json-read-from-string json-file)))

(cl-defun papis-query (&key query id doc-folder)
  "Make a general papis query:
   it returns a list of hashtables where every hashtable is a papis document"
  (when id
    (setq query (papis-id-query id)))
  (papis--json-string-to-documents (papis-json :query query
                                               :doc-folder doc-folder)))

`papis-open`

The cornerstone of papis is opening documents, in emacs the command is also available:

(cl-defun papis--update (&key id doc-folder alist)
  (let (sets)
    (dolist (pair alist)
      (push (format "--set %s %S" (car pair) (cdr pair))
            sets))
    (papis--cmd (format "update %s %s"
                        (string-join sets " ")
                        (if doc-folder
                            (format "--doc-folder %S" doc-folder)
                          (format "papis_id:%s" id))))))
(defun papis-browse (doc)
  (interactive (list (papis--read-doc)))
  (let ((url
         (cond
           ((papis-doc-get doc "url" nil))
           ((when-let ((doi (papis-doc-get doc "doi" nil)))
              (format "https://doi.org/%s" doi))
            (t (error "Neither url nor doi found in this document."))))))
    (browse-url url)))

(defun papis-open (doc)
  (interactive (list (papis--read-doc)))
  (let* ((files (papis--get-file-paths doc))
         (file (pcase (length files)
                 (1 (car files))
                 (0 (error "Doc has no files"))
                 (_ (completing-read "file: " files)))))
    (split-window-horizontally)
    (find-file file)))

Notes

(defcustom papis-edit-new-notes-hook nil
  "Hook for when a new note file is being edited.

   The argument of the hook is the respective document."
  :type 'hook)

(defun papis--default-notes-name ()
  (string-replace "\n" "" (papis--cmd "config notes-name" t)))

(defun papis--notes-path (doc)
  "Return the notes path to the given document.
   This does not make sure that the notes file exists,
   it just gets a path that hsould be there."
  (let ((query (papis-id-query doc)))
    (papis--cmd (format "list --notes %s"
                        query)
                t)))

(defun papis--ensured-notes-path (doc)
  (let ((maybe-notes (papis-doc-get doc "notes"))
        (id-query (papis-id-query doc)))
    (unless maybe-notes
      (setq maybe-notes (papis--default-notes-name))
      ;; will this work on windows? someone cares?
      (papis--cmd (format "edit --notes --editor echo %s" id-query)))
    (string-replace "\n" ""
                    (papis--cmd (format "list --notes %s" id-query)
                                t))))

(defun papis-notes (doc &optional run-hook)
  """
  Create notes for a document or open the note
  DOC is the papis document.
  Whenever RUN-HOOK is non-nil, the hook for the notes
  will be ran.
  """
  (interactive (list (papis--read-doc)
                     current-prefix-arg))
  (let ((has-notes-p (papis-doc-get doc "notes")))
    (let ((notes-path (papis--ensured-notes-path doc)))
      (when (or (not has-notes-p) run-hook)
        (with-current-buffer (find-file notes-path)
          (run-hook-with-args 'papis-edit-new-notes-hook
                              doc)))
      (find-file notes-path))))

`papis-edit`

You can edit the info files using papis-edit, notice that commiting the Implement waiting after editing the file like

(define-minor-mode papis-edit-mode
    "General mode for editing papis files"
  :keymap `((,(kbd "C-c C-c") .
              ,(defun papis-edit-update-cache (folder)
                 (interactive (list default-directory))
                 (message "Updating the cache for %s" folder)
                 (papis--cmd (format "cache update --doc-folder %s"
                                     folder)))))

  (defvar-local papis-edit-mode-id nil))

(defun papis-edit (doc)
  (interactive (list (papis--read-doc)))
  (let* ((folder (papis-doc-get-folder doc))
         (info (concat folder "/" "info.yaml")))
    (find-file info)
    (papis-edit-mode)))

`papis-exec`

(defun papis-exec (python-file &optional arguments)
  (let ((fmt "exec %s %s"))
    (papis--cmd (format fmt
                        python-file
                        (or arguments ""))
                t)))

`papis-export`

(progn
  (defmacro papis--make-exporter (format-name)
    `(cl-defun ,(intern (format "papis-%s" format-name))
         (&key query doc-folder)
       (let ((outfile (make-temp-file "papis-")))
         (papis--cmd (format "export --all --format %s %s -o %s"
                             ,(symbol-name format-name)
                             (if doc-folder (format "--doc-folder %S" doc-folder)
                               (format "%S" query))
                             outfile))
         (with-current-buffer (find-file-noselect outfile)
           (prog1 (buffer-string)
             (kill-buffer))))))

  (papis--make-exporter bibtex)
  (papis--make-exporter yaml)
  (papis--make-exporter typist)
  (papis--make-exporter json))

Document reader

The main dynamic searcher used in papis.el uses the function papis-default-read-format-function.

(defun papis-default-read-format-function (doc)
  `(
    ,(format "%s\n\t%s\n\t«%s» +%s %s"
             (papis-doc-get doc "title")
             (papis-doc-get doc "author")
             (papis-doc-get doc "year")
             (or (papis-doc-get doc "tags") "")
             (let ((n (papis-doc-get doc "_note"))) (if n (concat ":note " n) "")))
    .
    ,doc))


(defun papis--org-looking-at-link ()
  (when (eq major-mode 'org-mode)
    (let* ((context (org-element-lineage (org-element-context)
                                         '(link)
                                         t))
           ;; (type (org-element-type context))
           (papis-id (org-element-property :path context)))
      papis-id)))

the papis reader should be rather understwood as a dwim (do what i mean) reader. if you are in org-mode and you have the point above a papis link, then it will get that paper to do whatever papis action you’re planning to do.

(defun papis-from-id (papis-id)
  (let* ((query (format "papis_id:%s" papis-id))
         (results (papis-query :query query)))
    (pcase (length results)
      (0 (error "No documents found with papis_id '%s'"
                papis-id))
      (1 (car results))
      (_ (error "Too many documents (%d) found with papis_id '%s'"
                (length results) papis-id)))))

(defun papis--read-doc (&optional force-query)
  (cond
    ;; if in org mode and in org link, return it
    ((and (not force-query)
          (papis--org-looking-at-link))
     (papis-from-id (papis--org-looking-at-link)))
    ((and (not force-query)
          (let* ((filename (buffer-file-name (current-buffer)))
                 (dirname (f-dirname filename))
                 (yaml.info (f-join dirname "info.yaml")))
            (when (file-exists-p yaml.info)
              (car (papis-query :doc-folder dirname))))))
    ((and (not force-query)
          (let* ((results (papis-query :query (read-string papis--query-prompt
                                                           nil 'papis)))
                 (formatted-results (mapcar papis-read-format-function results)))
            (cdr (assoc
                  (completing-read "Select an entry: " formatted-results)
                  formatted-results)))))))

Org-links

`papis`

(require 'ol-doi)
(org-link-set-parameters "papis"
                         :follow (lambda (papis-id)
                                   (papis-open (papis-from-id papis-id)))
                         :export #'ol-papis-export
                         :complete (lambda (&optional arg)
                                     (format "papis:%s"
                                             (papis-doc-get (papis--read-doc)
                                                            "papis_id")))
                         :insert-description
                         (lambda (link desc)
                           (let* ((papis-id (string-replace "papis:"  "" link))
                                  (doc (papis-from-id papis-id)))
                             (papis-doc-get doc "title"))))

(defun ol-papis-export (papis-id description format info)
  (let* ((doc (papis-from-id papis-id))
         (doi (papis-doc-get doc "doi"))
         (url (papis-doc-get doc "url")))
    (cond
      (doi (org-link-doi-export doi description format info)))))

Paper sections

When doing research, often you would like to create some notes on every paper and write some sections with the section titles being links to the papers with some properties so that you can use org-mode’s colum mode.

You can use the following function to create a link with properties

(defun papis-org-insert-heading (doc)
  (interactive (list (papis--read-doc)))
  (let ((title (papis-doc-get doc "title"))
        (author (papis-doc-get doc "author"))
        (year (papis-doc-get doc "year"))
        (doi (papis-doc-get doc "doi"))
        (papis-id (papis-doc-get doc "papis_id")))
    (org-insert-heading)
    (insert (format "[[papis:%s][%s]]" papis-id title))
    (org-set-property "PAPIS_ID" papis-id)
    (org-set-property "AUTHOR" author)
    (org-set-property "TITLE" title)
    (org-set-property "YEAR" (format "%s" year))
    (org-set-property "DOI" doi)))

A recommendation can be to write as the COLUMNS variable and the PROPERTIES like so

#+COLUMNS: %7TODO %5YEAR %10AUTHOR %45TITLE %TAGS
#+PROPERTIES: TITLE AUTHOR YEAR

and then you can turn on the org-columns mode.

`org-ref` compatibility

Open pdfs

org-ref can open the pdf of a publicaction from the cite:my-reference link, but in the case of papis this pdf lives in an isolated folder of its own.

However in org-ref you can customize how you get the pdf from the cite link through the elisp:org-ref-get-pdf-filename-function. Therefore, in order to use papis to open the pdf of the referenced documents you can set

(setq org-ref-get-pdf-filename-function
      #'papis-org-ref-get-pdf-filename)

Its implementation is given below:

(defun papis-org-ref-get-pdf-filename (key)
    (interactive)
    (let* ((docs (papis-query (format "ref:'%s'" key)))
           (doc (car docs))
           (files (papis--get-file-paths doc)))
      (pcase (length files)
        (1 (car files))
        (_ (completing-read "" files)))))

Citations

In general it is recommended to use the citation mechanisms of org-ref, however, if for some reason you would like to cite directly from papis, you can use the function

(defun papis-insert-citation (doc)
  (interactive (list (papis--read-doc)))
  (let* ((ref (papis--get-ref doc)))
    (if (fboundp 'citar-insert-citation)
        (citar-insert-citation (list ref))
      (insert (format "[cite:@%s]" ref)))))

and we will need also a way of listing all the keys of the document for further functions. I took this from the good citar package

(defun papis-org-list-keys ()
  "List citation keys in the org buffer."
  (let ((org-tree (org-element-parse-buffer)))
    (delete-dups
     (org-element-map org-tree 'citation-reference
       (lambda (r) (org-element-property :key r))
       org-tree))))

Bibtex entries

In this section we want to develop a way to generate a bibtex bibliography from references appearing in the document currently being edited.

Convert references into bibtex entries

First we need a script that accepts a list of

import argparse
import papis.api
from papis.bibtex import to_bibtex

parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter,
                                 description='')
parser.add_argument('refs', help='References', action='store', nargs='*')
args = parser.parse_args()

docs = []

for ref in args.refs:
    docs.extend(papis.api.get_documents_in_lib(library=None, search=ref))

for d in docs:
    print(to_bibtex(d))

(defvar papis--refs-to-bibtex-script
"
<<references-to-bibtex-python-script>>
")

(defun papis--refs-to-bibtex (refs)
  (let ((py-script (make-temp-file "papis-bibtex-script" nil ".py")))
    (with-temp-buffer
      (insert papis--refs-to-bibtex-script)
      (write-file py-script))
    (papis-exec py-script (s-join " " refs))))

The `papis-bibtex-refs` dynamic block

(defun papis-create-papis-bibtex-refs-dblock (bibfile)
  (insert (format "#+begin: papis-bibtex-refs :tangle %s" bibfile))
  (insert "\n")
  (insert "#+end:"))

(defun papis-extract-citations-into-dblock (&optional bibfile)
  (interactive)
  (if (org-find-dblock "papis-bibtex-refs")
      (progn
        (org-show-entry)
        (org-update-dblock))
    (papis-create-papis-bibtex-refs-dblock
     (or bibfile (read-file-name "Bib file: " nil "main.bib")))))

(defun org-dblock-write:papis-bibtex-refs (params)
  (let ((tangle-file (or (plist-get params :tangle)
                         (buffer-file-name)))
        (exports ":exports none"))
    (insert
     (format "#+begin_src bibtex %s :tangle %s\n"
             exports
             tangle-file)))
  (let* ((refs (papis-org-list-keys))
         (queries (mapcar (lambda (r) (format "ref:\"%s\"" r))
                          refs)))
    (insert (papis--refs-to-bibtex queries)))
  (insert "#+end_src\n"))

End

(provide 'papis)

papis / papis.el

Papis.el

Introduction

Motivation

Disclaimer

What is implemented

org-noter compatibility

org-transclusion compatibility

Implementation

Generalities

Variables

`papis-library`

Document

Commands

Introduction

Issuing commands to the shell

`papis-query`

`papis-open`

Notes

`papis-edit`

`papis-exec`

`papis-export`

Document reader

Org-links

`papis`

Paper sections

`org-ref` compatibility

Open pdfs

Citations

Bibtex entries

Convert references into bibtex entries

The `papis-bibtex-refs` dynamic block

End

About

Languages

Papis.el

Introduction

Motivation

Disclaimer

What is implemented

org-noter compatibility

org-transclusion compatibility

Implementation

Generalities

Variables

papis-library

Document

Commands

Introduction

Issuing commands to the shell

papis-query

papis-open

Notes

papis-edit

papis-exec

papis-export

Document reader

Org-links

papis

Paper sections

org-ref compatibility

Open pdfs

Citations

Bibtex entries

Convert references into bibtex entries

The papis-bibtex-refs dynamic block

End

About

Languages

`papis-library`

`papis-query`

`papis-open`

`papis-edit`

`papis-exec`

`papis-export`

`papis`

`org-ref` compatibility

The `papis-bibtex-refs` dynamic block