SHRFACE

Description
Installation
Configuration
News/Logs

Description

This package extends shr / eww with org features and analysis capability. It can be used in dash-docs, eww, nov.el, mu/mu4e, anki.el, etc. It is also able to export HTML buffer to Org buffer/file, or download/archive web pages.

Configurable org-like heading faces, headline bullets, item bullets,paragraph indentation, fill-column, item bullet, versatile hyper links(http/https/file/mailto/etc) face and so on.
Browse the internet or local html file with eww just like org mode.
Read dash docsets with dash-docs and the beauty of org faces.
Read epub files with nov.el , just like org mode.
Read html email with mu/mu4e , the same reading experience just like org mode without formatting html to org file.
Switch/jump the headlines just like org-mode in eww and nov.el with imenu
Toggle/cycle the headlines just like org-mode in eww and nov.el with outline minor mode , org-cycle/org-shifttab, or shrface-outline-cycle / shrface-outline-cycle-buffer.
Analysis capability:
- Headline analysis: List all headlines with clickable texts, or jump around with ivy.
- URL analysis: List all classified URL with clickable texts in a buffer, or jump around with ivy.
Export HTML buffer to org buffer/file using shr engine (no Pandoc is needed).

The project target is to apply org features and analysis capability to shr, and all libraries that render HTML with shr (Simple HTML Renderer).

As EmacsWiki says:

shr.el is an HTML renderer in Emacs as of version 24.4 (based on libxml2, it was originally a part of Gnus). It’s the basis of the web browser eww.

Installation

It’s available on Melpa :

M-x package-install shrface

Configuration

Quick Start

use-package

I recommend configure with use-package, it is more elegant:

(use-package shrface
  :defer t
  :config
  (shrface-basic)
  (shrface-trial)
  (shrface-default-keybindings) ; setup default keybindings
  (setq shrface-href-versatile t))

(use-package eww
  :defer t
  :init
  (add-hook 'eww-after-render-hook #'shrface-mode)
  :config
  (require 'shrface))

(use-package nov
  :defer t
  :init
  (add-hook 'nov-mode-hook #'shrface-mode)
  :config
  (require 'shrface)
  (setq nov-shr-rendering-functions '((img . nov-render-img) (title . nov-render-title)))
  (setq nov-shr-rendering-functions (append nov-shr-rendering-functions shr-external-rendering-functions)))

(use-package anki
  :defer t
  :load-path "~/.emacs.d/lisp/anki/"
  :init
  (add-hook 'anki-mode-hook #'shrface-mode)
  (autoload 'anki "anki")
  (autoload 'anki-browser "anki")
  (autoload 'anki-list-decks "anki")
  :config
  (require 'shrface)
  (setq anki-shr-rendering-functions (append anki-shr-rendering-functions shr-external-rendering-functions))
  (setq sql-sqlite-program "/usr/bin/sqlite3")
  (setq anki-collection-dir "/Users/chandamon/Library/Application Support/Anki2/User 1"))

Check this section to see the supported faces: Supported Faces
Check the experimental faces notes here: Experimental faces
There is another library inherit-org to inherit org faces to even more non-org buffers, and it is no dependency/relationships with shrface, check here https://github.com/chenyanming/inherit-org

Font setup

The default font is variable-pitch.

(set-face-attribute 'variable-pitch nil :font (format "%s:pixelsize=%d" "iA Writer Quattro S" 15))

Keybinding Settings

You can to call shrface-default-keybindings to enable the recommended keybindings.

(defun shrface-default-keybindings ()
  (interactive)
  "Sets up the default keybindings for `shrface-mode'."
  (define-key shrface-mode-map (kbd "TAB") 'shrface-outline-cycle)
  (define-key shrface-mode-map (kbd "<backtab>") 'shrface-outline-cycle-buffer)
  (define-key shrface-mode-map (kbd "C-t") 'shrface-toggle-bullets)
  (define-key shrface-mode-map (kbd "C-j") 'shrface-next-headline)
  (define-key shrface-mode-map (kbd "C-k") 'shrface-previous-headline)
  (define-key shrface-mode-map (kbd "M-l") 'shrface-links-counsel) ; or 'shrface-links-helm or 'shrface-links-consult
  (define-key shrface-mode-map (kbd "M-h") 'shrface-headline-counsel)) ; or 'shrface-headline-helm or 'shrface-headline-consult

Since the keybindings may conflict with other modes, such as nov-mode-map, eww-mode-map, mu4e-mode-map. If you want to enable shrface’s keybindings on these modes, you have to bind the functions to those maps as well.

Here is the keybinding example:

(with-eval-after-load 'nov
  (define-key nov-mode-map (kbd "<tab>") 'shrface-outline-cycle)
  (define-key nov-mode-map (kbd "S-<tab>") 'shrface-outline-cycle-buffer)
  (define-key nov-mode-map (kbd "C-t") 'shrface-toggle-bullets)
  (define-key nov-mode-map (kbd "C-j") 'shrface-next-headline)
  (define-key nov-mode-map (kbd "C-k") 'shrface-previous-headline)
  (define-key nov-mode-map (kbd "M-l") 'shrface-links-counsel) ; or 'shrface-links-helm or 'shrface-links-consult
  (define-key nov-mode-map (kbd "M-h") 'shrface-headline-counsel)) ; or 'shrface-headline-helm or 'shrface-headline-consult

(with-eval-after-load 'eww
  (define-key eww-mode-map (kbd "<tab>") 'shrface-outline-cycle)
  (define-key eww-mode-map (kbd "S-<tab>") 'shrface-outline-cycle-buffer)
  (define-key eww-mode-map (kbd "C-t") 'shrface-toggle-bullets)
  (define-key eww-mode-map (kbd "C-j") 'shrface-next-headline)
  (define-key eww-mode-map (kbd "C-k") 'shrface-previous-headline)
  (define-key eww-mode-map (kbd "M-l") 'shrface-links-counsel) ; or 'shrface-links-helm or 'shrface-links-consult
  (define-key eww-mode-map (kbd "M-h") 'shrface-headline-counsel)) ; or 'shrface-headline-helm or 'shrface-headline-consult

(with-eval-after-load 'mu4e
  (define-key mu4e-view-mode-map (kbd "<tab>") 'shrface-outline-cycle)
  (define-key mu4e-view-mode-map (kbd "S-<tab>") 'shrface-outline-cycle-buffer)
  (define-key mu4e-view-mode-map (kbd "C-t") 'shrface-toggle-bullets)
  (define-key mu4e-view-mode-map (kbd "C-j") 'shrface-next-headline)
  (define-key mu4e-view-mode-map (kbd "C-k") 'shrface-previous-headline)
  (define-key mu4e-view-mode-map (kbd "M-l") 'shrface-links-counsel) ; or 'shrface-links-helm or 'shrface-links-consult
  (define-key mu4e-view-mode-map (kbd "M-h") 'shrface-headline-counsel)) ; or 'shrface-headline-helm or 'shrface-headline-consult

In additional to the keys provided by shrface-mode, the following features should also work, you can test and find which fits your requirement:

`org-mode`

org-cycle,
org-shifttab
org-content
org-overview

etc.

`outline-minor-mode`

`imenu-mode`

Features Anatomy

shrface-mode

shrface-mode is a minor mode that help you enable/disable the following additional features:

imenu
outline-minior-mode
Setup shr faces

Please be careful, every time you run shrface-mode, faces will be enable/disable globally for all shr rendered buffers. Since I have not found a good way to enable/disable the faces per buffer yet.

If you want to enable shrface just when you need to, you should avoid to use add-hook like the recommended settings above, and just enable/disable shrface-mode via M-x.

shrface-occur

List all headlines, and you can easily jump between them by mouse click.

shrface-links

List all URLs and classify them. You can easily go to the occurrence (Left Click/Enter), copy the URL (Middle Click) or browse the URL (Right Click). Besides, if all-the-icons is available, web icon for each link will be shown.

shrface-links-counsel

List all URLs with counsel.

The URLs are listed in order based the position on the buffer.
The first candidate to select is the next url counting from the current cursor position. In this case, you can use C-M-m (ivy-call), C-M-n (ivy-next-line-and-call), and C-M-p (ivy-previous-line-and-call) to jump around all URLs without losing your position.
C-o to fire the action menu on the selected candidate.
C-c C-o to fire the ivy-occur

shrface-headline-counsel

List all headlines with counsel.

It is a better solution than imenu and shrface-occur.
It can work without the headline bullets. You can disable the bullets via (setq shrface-toggle-bullets t) and still be able to jump around the headlines.
The first candidate to select is the current context headline of the current cursor position. In this case, you can use C-M-m (ivy-call), C-M-n (ivy-next-line-and-call), and C-M-p (ivy-previous-line-and-call) to jump around all headlines without losing your position.
C-o to fire the action menu on the selected candidate.
C-c C-o to fire the ivy-occur

shrface-links-helm

List all URLs with helm.

TAB to preview the link
RET to goto the link

shrface-headline-helm

List all headlines with helm.

TAB to preview the headline
RET to goto the headline

shrface-links-consult

List all links with consult.

shrface-headline-consult

List all headlines with consult.

shrface-next-headline, shrface-previous-headline

These two headline functions are designed to replace outline-next-headline and outline-previous-headline. They scan the headline number text properties and jump to the headlines which means it can work under no bullets circumstance if (setq shrface-toggle-bullets t).

Headline bullets (h1 to h6)

Customize the headline bullets

You can configure your favorite bullets up to 6 levels of headings (cycled through if less than 6 bullets in setting).

You can set it with:

(setq shrface-bullets-bullet-list ("◉" "○" "✸" "✿"))

PS: The bullets setting can be derived from org-bullets-bullet-list or org-superstar-headline-bullets-list, if org-bullets or org-superstar is available.

Toggle headline bullets locally/temporary

The quick way to toggle(disable/enable) headline bullets locally/temporary:

M-x shrface-toggle-bullets

Please notice: In mu4e-view-mode, using shrface-toggle-bullets will toggle bullets globally.

Disable headline bullets globally

If you do not like headline bullets, disable them globally by:

(setq shrface-toggle-bullets t)

Please notice, the following features are also disabled:

function shrface-occur
variable shrface-mode

However, the following features are still be able to use:

function shrface-links
function shrface-links-counsel
function shrface-headline-counsel
function shrface-previous-headline
function shrface-next-headline
function shrface-links-helm
function shrface-headline-helm
function shrface-links-consult
function shrface-headline-consult

Item bullet

You can configure your favorite item bullet for shrface

You can set it with:

(setq shrface-item-bullet "-")

PS: Only one type of item bullet is supported, prettified by shrface-item-bullet-face

Paragraph indentation and fill column

Both shrface-paragraph-indentation and shrface-paragraph-fill-column are obsolete.

Pleas use shr-indentation to control the indentation, and shr-width to adjust the text width (fill-column).

Versatile URL

You can enable versatile URL faces support simply by:

(setq shrface-href-versatile t)

The following types of URL can be customized.

http
https
ftp
file
mailto
other

Supported faces

Here are the faces supported:

;;; Faces for `shrface-basic

(defcustom shrface-bullets-bullet-list
  (or (bound-and-true-p org-bullets-bullet-list)
      (bound-and-true-p org-superstar-headline-bullets-list)
      '("◉"
        "○"
        "✸"
        "✿"))
  "Bullets for headings"
  :group 'shrface
  :type '(repeat (string :tag "Bullet character")))

(defface shrface-href-face '((t :inherit org-link))
  "Default <href> face if `shrface-href-versatile' is nil"
  :group 'shrface-faces)

(defface shrface-href-other-face '((t :inherit org-link :foreground "#87cefa"))
  "Face used for <href> other than http:// https:// ftp://
file:// mailto:// if `shrface-href-versatile' is NON-nil. For
example, it can be used for fontifying charter links with epub
files when using nov.el."
  :group 'shrface-faces)

(defface shrface-href-http-face '((t :inherit org-link :foreground "#39CCCC"))
  "Face used for <href>, http:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-href-https-face '((t :inherit org-link :foreground "#7FDBFF"))
  "Face used for <href>, https:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-href-ftp-face '((t :inherit org-link :foreground "#3D9970"))
  "Face used for <href>, ftp:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-href-file-face '((t :inherit org-link :foreground "#2ECC40"))
  "Face used for <href>, file:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-href-mailto-face '((t :inherit org-link :foreground "#FF851B"))
  "Face used for <href>, mailto:// if `shrface-href-versatile' is
NON-nil"
  :group 'shrface-faces)

(defface shrface-h1-face '((t :inherit org-level-1))
  "Face used for <h1> headlines."
  :group 'shrface-faces)

(defface shrface-h2-face '((t :inherit org-level-2))
  "Face used for <h2> headlines."
  :group 'shrface-faces)

(defface shrface-h3-face '((t :inherit org-level-3))
  "Face used for <h3> headlines."
  :group 'shrface-faces)

(defface shrface-h4-face  '((t :inherit org-level-4))
  "Face used for <h4> headlines."
  :group 'shrface-faces)

(defface shrface-h5-face  '((t :inherit org-level-5))
  "Face used for <h5> headlines."
  :group 'shrface-faces)

(defface shrface-h6-face '((t :inherit org-level-6))
  "Face used for <h6> headlines."
  :group 'shrface-faces)

(defface shrface-verbatim '((t :inherit org-verbatim))
  "Face used for verbatim/emphasis - <em>."
  :group 'shrface-faces)

(defface shrface-item-bullet-face '((t :inherit org-list-dt))
  "Face used for unordered list bullet"
  :group 'shrface-faces)

(defface shrface-item-number-face '((t :inherit org-list-dt))
  "Face used for ordered list numbers"
  :group 'shrface-faces)

(defface shrface-description-list-term-face '((t :inherit org-list-dt))
  "Face used for description list terms <dt>"
  :group 'shrface-faces)

(defface shrface-figure '((t :inherit org-table))
  "Face used for figure <figure>, e.g. figure captions."
  :group 'shrface-faces)

;;; Faces for `shrface-analysis' realted buffers

(defface shrface-links-title-face '((t :inherit default))
  "Face used for *shrface-links* title"
  :group 'shrface-analysis-faces)

(defface shrface-links-url-face '((t :inherit font-lock-comment-face))
  "Face used for *shrface-links* url"
  :group 'shrface-analysis-faces)

(defface shrface-links-mouse-face '((t :inherit mode-line-highlight))
  "Face used for *shrface-links* mouse face"
  :group 'shrface-analysis-faces)

Experimental face(s)

;;; Faces for `shrface-trail' realted buffers

(defface shrface-code '((t :inherit org-code))
  "TODO Face used for inline code"
  :group 'shrface-faces)

Export Html to Org

Exporting HTML to Org doesn’t need to enable shrface-mode, or run shrface-basic / shrface-trail, you just make sure you (require 'shrface) before using the exporting features:

shrface-html-export-as-org: Export current html buffer to an org buffer.
shrface-html-export-to-org: Export current html buffer to an org file.
shrface-org-title: Dynamically bound and overwrite the org title.
shrface-request-url: Dynamically bound the request url, so that shrface can fix the relative/absolute urls based on the request url.

If you use org-web-tools to download the http page into org, you can also override its pandoc engine with shr engine as following:

(advice-add 'org-web-tools--html-to-org-with-pandoc :override 'shrface-html-convert-as-org-string)

Here are two simple examples to use shrface-html-export-as-org, and shrface-html-export-to-org, using Request.

Download an URL and output an Org buffer/file:

(defun request-url-as-org (url)
  (interactive "sRequest url: ")
  (require 'shrface)
  (require 'request)
  (request url
    :parser 'buffer-string
    :headers '(("User-Agent" . "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36"))
    :sync nil
    :success (cl-function
              (lambda (&key data &allow-other-keys)
                (let ((shrface-request-url url))
                  (shrface-html-export-as-org data))))))

(defun request-url-to-org (url)
  (interactive "sRequest url: ")
  (require 'shrface)
  (require 'request)
  (request url
    :parser 'buffer-string
    :headers '(("User-Agent" . "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36"))
    :sync nil
    :success (cl-function
              (lambda (&key data &allow-other-keys)
                (let ((shrface-request-url url))
                  (shrface-html-export-to-org data "request.org"))))))

If you use org-web-tools, it is very easy to capture the web as a readable format:

(defun request-url-readable (url)
  (interactive "sRequest url: ")
  (require 'shrface)
  (require 'request)
  (require 'org-web-tools)
  (request url
    :parser 'buffer-string
    :headers '(("User-Agent" . "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36"))
    :sync nil
    :success (cl-function
              (lambda (&key data &allow-other-keys)
                (let* ((web (org-web-tools--eww-readable data))
                       (title (car web))
                       (html (cdr web))
                       (shrface-org-title title)
                       (shrface-request-url url))
                  (shrface-html-export-as-org html))))))

Enable org-like ellipsis

To enable shrface-ellipsis the “three-dots” after each headline when it is hidden just like org mode, please manually add the predefined function: shrface-outline-visibility-changed to outline-view-change-hook,

(add-hook 'outline-view-change-hook 'shrface-outline-visibility-changed)

(Optional) Enable source codes highlight

You can install shr-tag-pre-highlight.el to enable source codes highlight and background color.

(use-package shr-tag-pre-highlight
  :ensure t
  :after shr
  :config
  (add-to-list 'shr-external-rendering-functions
               '(pre . shr-tag-pre-highlight))
  (when (version< emacs-version "26")
    (with-eval-after-load 'eww
      (advice-add 'eww-display-html :around
                  'eww-display-html--override-shr-external-rendering-functions))))

Hacking the `shr-tag-pre-highlight.el`

If you want to add indentation, background color, #+BEGIN_SRC lang, and #+END_SRC for, you can overwrite the function as following:

(require 'shr-tag-pre-highlight)
(add-to-list 'shr-external-rendering-functions '(pre . shrface-shr-tag-pre-highlight))
(defun shrface-shr-tag-pre-highlight (pre)
    "Highlighting code in PRE."
    (let* ((shr-folding-mode 'none)
           (shr-current-font 'default)
           (code (with-temp-buffer
                   (shr-generic pre)
                   ;; (indent-rigidly (point-min) (point-max) 2)
                   (buffer-string)))
           (lang (or (shr-tag-pre-highlight-guess-language-attr pre)
                     (let ((sym (language-detection-string code)))
                       (and sym (symbol-name sym)))))
           (mode (and lang
                      (shr-tag-pre-highlight--get-lang-mode lang))))
      (shr-ensure-newline)
      (shr-ensure-newline)
      (setq start (point))
      (insert
       (propertize (concat "#+BEGIN_SRC " lang "\n") 'face 'org-block-begin-line)
       (or (and (fboundp mode)
                (with-demoted-errors "Error while fontifying: %S"
                  (shr-tag-pre-highlight-fontify code mode)))
           code)
       (propertize "#+END_SRC" 'face 'org-block-end-line ))
      (shr-ensure-newline)
      (setq end (point))
      (if light
          (add-face-text-property start end '(:background "#D8DEE9" :extend t))
        (add-face-text-property start end '(:background "#292b2e" :extend t)))
      (shr-ensure-newline)
      (insert "\n")))

Sometimes a wrong language is detected, but it is still great for highlight, even just for eye pleasing.

Screenshots when both enable `shrface` and the code highlights

News/Logs

TBD

Remove org-indent-mode support (Because org-indent-mode can not work on non-org mode at this moment)

`2024-03-31`

Version 2.6.4

Add shrface-links-consult
Add shrface-headline-consult
Add shrface-ellipsis and shrface-outline-visibility-changed function for outline-view-change-hook

`2021-01-23`

Version 2.6.3:

Add shrface-html-export-as-org
Add shrface-html-export-to-org
Add shrface-tag-u
Add shrface-tag-strong

`2021-01-20`

Version 2.6.2:

Add keys for shrface-mode.
Move settings to shrface-mode. So that we can use shrface-mode on other shr rendered buffers via M-x.

`2020-11-08`

Version 2.6.1:

Add: shrface-outline-cycle
Add: shrface-outline-cycle-buffer

`2020-05-07`

Version 2.6:

New Function: shrface-headline-helm and shrface-links-helm

`2020-05-02`

Version 2.5:

New Function: shrface-headline-counsel: better solution than imenu to list all headlines
New Function: shrface-next-headline: better solution than outline-next-headline, it can work without headline bullets
New Function: shrface-previous-headline: better solution than outline-previous-headline, it can work without headline bullets

`2020-05-01`

Version 2.4:

Improved: Improve the user experience for shrface-links-counsel
New face: shrface-figure

`2020-04-29`

Version 2.3:

New customization: shrface-toggle-bullets Quick way to toggle the headline bullets.

`2020-04-26`

Version 2.2:

New customization: shrface-imenu-depth The maximum level for Imenu access to shrface headlines.
New Feature: shrface-links and shrface-links-counsel The first shrface-analysis feature, to list all possible URL in a new buffer.

`2020-04-23`

Version 2.1:

New Feature: shrface-occur
New faces:
- shrface-href-http-face
- shrface-href-https-face
- shrface-href-ftp-face
- shrface-href-file-face
- shrface-href-mailto-face
- shrface-href-other-face

`2020-04-20`

Version 2.0:

New face: shrface-description-list-term-face

`2020-04-19`

Version 1.9:

New Functions: shrface-basic and shrface-trail

`2020-04-18`

Version 1.8:

New face: shrface-item-number-face
New Minor Mode: shrface-mode

`2020-04-17`

Version 1.7:

New feature: shrface-item-bullet
New face: shrface-item-bullet-face

Version 1.6:

New feature: org-indent-mode support (Enabled by default)

`2020-04-16`

Version 1.5:

New feature: outline minior mode support (Enabled by default, but not the keybindings)

`2020-04-15`

Version 1.4:

New feature: imenu support

`2020-04-13`

Version 1.3:

New face: shrface-code (Experimental face, disabled by default)

`2020-04-12`

Version 1.2:

New face: shrface-verbatim

`2020-04-11`

Version 1.1:

Fixed bug: Wrong indentation handling make some items in paragraph disappear (such as images)

`2020-04-10`

Version 1.0:

New face: shrface-bullets-bullet-list
New face: shrface-h1-face
New face: shrface-h2-face
New face: shrface-h3-face
New face: shrface-h4-face
New face: shrface-h5-face
New face: shrface-h6-face
New face: shrface-h6-face
New face: shrface-href-face
New customizable variable: shrface-paragraph-indentation
New customizable variable: shrface-paragraph-fill-column

SHRFACE

Table of Contents

Description

Installation

Configuration

Quick Start

use-package

Font setup

Keybinding Settings

org-mode

outline-minor-mode

imenu-mode

Features Anatomy

shrface-mode

shrface-occur

shrface-links

shrface-links-counsel

shrface-headline-counsel

shrface-links-helm

shrface-headline-helm

shrface-links-consult

shrface-headline-consult

shrface-next-headline, shrface-previous-headline

Headline bullets (h1 to h6)

Customize the headline bullets

Toggle headline bullets locally/temporary

Disable headline bullets globally

Item bullet

Paragraph indentation and fill column

Versatile URL

Supported faces

Experimental face(s)

Export Html to Org

Enable org-like ellipsis

(Optional) Enable source codes highlight

Hacking the shr-tag-pre-highlight.el

Screenshots when both enable shrface and the code highlights

News/Logs

TBD

2024-03-31

2021-01-23

2021-01-20

2020-11-08

2020-05-07

2020-05-02

2020-05-01

2020-04-29

2020-04-26

2020-04-23

2020-04-20

2020-04-19

2020-04-18

2020-04-17

2020-04-16

2020-04-15

2020-04-13

2020-04-12

2020-04-11

2020-04-10

About

Languages