maljub01 / organice

An implementation of Org mode without the dependency of Emacs - built for mobile and desktop browsers

organice documentation

organice - /’ɔ:gənaɪz/

organice organizes Org files nicely!

General

Tests:

Documentation: https://organice.200ok.ch/documentation.html

Community chat: #organice on IRC Freenode, or #organice:matrix.org on Matrix

What does this project do?

organice is an implementation of Org mode without the dependency of Emacs. It is built for mobile and desktop browsers and syncs with Dropbox, Google Drive and WebDAV.

At 200ok, we run an instance of organice at https://organice.200ok.ch, which is open for anyone to use! organice does not have a back-end (it’s just a front-end application, which uses either Dropbox, Google Drive or WebDAV as back-end storage). We don’t store any kind of data on our servers - we also don’t use analytics on organice.200ok.ch.

Why is this project useful

Emacs is great, but it’s desktop software. For users who want to access or edit their Org mode files whilst on the go, organice is a great choice.

Introduction

If you prefer a video to some text, we’ve got you covered! For EmacsConf 2019, we’ve created a 10 minute introductory video into the rationale and usability of organice.

You can watch it on:

• Youtube
• emacsconf.org

Installation

organice is a web application. You can use it from any browser. On iOS and Android, you can install organice to your homescreen. When started from there, it will run in full-screen as a Progressive Web Application (PWA) which will add some offline capabilities.

First, go to the page that you want to use as your start screen when using organice as an app from your homescreen. This could be the “files” overview or your main Org file. Then to install to the homescreen follow these platform specific instructions:

• iOS:

  Open organice in Mobile Safari. Tap the “share” button and select “Add to Home Screen”.

• Android:

  The exact procedure may differ depending on your browser and Android version. If you discover improvements to the following procedure, please let us know!

  First open the organice web page in your mobile browser.

  • On Chrome, tap the “menu” button (three vertically stacked dots) and select “Add to homescreen”.
  • On Firefox, tap the home icon with the plus sign inside it which is immediately to the right of the URL in the address bar.
  • Other browsers may have a similar procedure to one of these.

  At this point, most browsers will present a popup banner with the option to “Add to homescreen” or “Install”.

  If you want the organice app icon on your home screen to jump to a specific .org file, and/or if you want multiple app icons on the home screen to jump to different files, things can get a bit more tricky. For instance, Chrome only offers “Add to homescreen” the first time; once it has been added, the option changes to “Open organice”. Firefox allows creation of multiple home screen icons, but when opening any of these, organice does not jump directly to the specific file which was open when the icon was created.

  Therefore if you want multiple organice bookmark icons on your homescreen which jump straight to particular files, you may have better luck using another technique. For example, you can use a third-party program called ”Open Link With…”. Once that’s installed, from Chrome, visit the page in organice you want to make a homescreen icon for, then tap the “menu” button again, select “Share…”, and then “Add To Home”, then “organice”. Once it’s added, your home launcher may allow you to change the visible icon to the organice icon, or whatever other icon you choose.

Usage

Current restrictions/expectations of organice

“Current” means we’re working hard on removing the following restrictions and expectations.

• organice understands only a few in-buffer settings (see Supported in-buffer configuration)
  • Other in-buffer settings are imported and re-exported but are not editable with organice.
• Other content before the first headline is imported and re-exported, but invisible and currently not editable with organice.
• After potential in-buffer settings, your Org file has to begin with a headline.

Apart from these restrictions, organice is very robust in reading and editing your Org file and not breaking any of it. We’re having users with 10’000 lines in their files including all kinds of native Org functionality - and even these files work just fine in organice!

Generally, when working with distributed Org files, we’re recommending to put them under version control and to check for bugs and racing conditions between clients.

Please file an issue if you find additional restrictions, expectations or bugs that you you wouldn’t have expected.

Background information

organice has a custom parser for Org files. It works quite fine and has unit tests to prove it. One of the quality goals for the parser is that when it parses and re-exports an Org file, it should not change the original file. Not seeing unrelated diffs is important for the productivity of the user. It sounds trivial, but lots of alternative products do not live up to this expectation.

Writing a parser for a complex syntax like Org mode in custom code is hard. Therefore, we are in the process of implementing a proper EBNF based parser and a set of tests behind that. If you’re interested, please check it out: https://github.com/200ok-ch/org-parser

The strategy we’re using with regards to the parser is this:

• Keep improving the existing custom parser for new features and make bug fixes as long as the new one isn’t ready.
• In parallel, work on the new one until there is feature parity between both parsers.
• When the new one is finished, integrate it into organice.

Progressive Web App / Offline Support

organice can run as a PWA (Progressive Web App) - see the installation instructions and does have offline support. From your home screen, organice will start up in full screen and it will use a Service Worker to cache the application. On a desktop browser, the Service Worker will be used automatically. This is implemented using the Create React App Progressive Web App functionality which enables the following features:

• All static assets are cached so that organice loads fast on subsequent visits, regardless of network connectivity.
• Updates are downloaded in the background.
• organice works regardless of network state, even if offline.
• On mobile devices, organice can be added directly to the user’s home screen, app icon and all.

Following that, if you start modifying your Org file when offline, organice will recognize that you are offline and queue up the synchronization until you are online again.

organice also understands when it’s local Org file is outdated compared to the upstream file and will ask you want you want to do - pull the one from the synchronization back-end, push the one from organice or cancel. This happens when you made changes to your file on at least two machines at the same time without synchronizing them in the meantime. For this, we recommend to put your Org file under version control which is the idiomatic solution for changing text based files on multiple machines in parallel.

Customization

General

Since organice implements Org mode, one might wonder if we plan to duplicate the Emacs configuration strategy. In Emacs Org mode, there’s more than 500 variables for customization - and on top of that, there’s often two ways to configure things:

1. Using elisp
2. Using in-buffer settings

Modifying Org behavior using elisp (variables) is certainly mighty and powerful. However, the goal of organice is not to clone Emacs in full. In fact, it could be argued that this is not possible. Emacs being a LISP machine has inherent power that cannot be brought to a web application. Instead, the goal is to make Org mode accessible on smartphones and for non-Emacs users. For both use-cases, elisp variable configuration is not an idiomatic or ergonomic option.

organice implements this customization strategy:

• Use in-buffer settings where appropriate
• Build custom and mobile friendly user interfaces where appropriate
  • For example capture templates

Supported in-buffer configuration

In-buffer settings

• #+TODO
• #+TYP_TODO

#+STARTUP: options

• nologrepeat: Do not record when reinstating repeating item

Drawer properties

• logrepeat and nologrepeat: Whether to record when reinstating repeating item

:PROPERTIES:
:LOGGING:  logrepeat
:END:

Other customizations

For some customizations, organice exposes a mobile friendly user interface. Please find them in the ‘settings’ view (cogs icon in the header on the right).

Development

organice is built with React and Redux. It was bootstrapped with Create React App. The tests are written with React Testing Library. The internal data structures are written as immutable persistent data collections with the Immutable library.

Prerequisites

You will need a version of the Node.js engine installed which fulfills the requirement stated in package.json. If you don’t already have this installed, it is recommended to install it via nvm. The organice repository already contains an .nvmrc file, so once you have nvm installed, the following commands should be sufficient:

nvm install
nvm use

Setup

To install the necessary packages, run:

yarn install

To test against your own Dropbox account, you’ll need to create a .env file by copying .env.sample to just .env.

cp .env.sample .env

Running the application:

yarn start

Running the tests:

yarn test

For searching the Org file, there’s a grammar for the search clause. It’s written in pegjs. Generating the parser code happens automatically on yarn start|build|test. When working on the parser, you can manually generate it with:

./bin/compile_search_parser.sh

Testing

When you’re developing a new feature and you want to manually test it, it’s best to check it out in a Desktop browser and on your smartphone. This is how you do that:

Desktop

Run the application with yarn start which will open organice in your configured default browser. Alternatively, visit http://localhost:3000 in the browser of your choice.

Smartphone

There are multiple options on how you can connect from your smartphone to your computer running organice.

When running organice with yarn start, it will show you all the IPs that the application server is bound to. One will be local to your computer, one will be on your network (if you’re connected to a LAN or Wifi, that is).

If your smartphone has access to the same network, you can access it with the given IP address and port number.

If your new feature doesn’t require a synchronization back-end, just open the sample.org file which doesn’t require a login. You’re good to go.

Synchronizing with Dropbox or Google Drive

If your new feature does require the Dropbox or Google Drive synchronization back-end, there’s an extra step you need to perform.

Both Dropbox and Google Drive require a whitelist of domains that they can be synchronized from. The whitelist for local domains is exclusively short: http://localhost:3000.

Hence, to be able to login from your phone to your dev instance of organice, you’ll need to set up port forwarding. If you have a shell on your phone and an ssh client, you can do that with the following command:

ssh -L 3000:localhost:3000 user-dev-machine

If you don’t have a shell on your phone, you can use a dedicated SSH application (like Terminus).

Debugging Tests

Apart from the popular choice of console.log-debugging, it’s easy to use Chrome or Chromium for debugging tests.

Place a debugger; statement in any test, then run:

yarn test:dbg

This will start running your Jest tests, but pause before executing to allow a debugger to attach to the process.

Open the following in Chrome:

about:inspect

After opening that link, the Chrome Developer Tools will be displayed. Select inspect on your process and a breakpoint will be set at the first line of the react script (this is done to give you time to open the developer tools and to prevent Jest from executing before you have time to do so). Click the button that looks like a “play” button in the upper right hand side of the screen to continue execution. When Jest executes the test that contains the debugger statement, execution will pause and you can examine the current scope and call stack.

The “Create React App” upstream docs for this feature are here: https://create-react-app.dev/docs/debugging-tests/

Contributions

Please see our contributor guidelines and our code of conduct.

Deployment

Since organice is a front-end only application, it can easily be deployed to any server capable of serving a static application.

Please note: If you want the hosted application to connect to Dropbox or Google Drive, please read the section on Synchronization back-ends.

FTP

First create the production build locally: yarn run build Note: Creating a build will actually make your REACT_APP_* variables from the .env file available under process.env even though it’ll be a front-end application.

And then upload to your web-server. Here’s a script for your convenience:

HOST='your_ftp_server_host'
USER='ftp_user'
PASSWD='ftp_password'

lftp $HOST <<END_SCRIPT
user $USER $PASSWD
mirror -R build/
quit
END_SCRIPT
exit 0

Docker

organice is also available as a Docker image.

With docker-compose

If docker-compose is installed, the following command downloads and runs the latest image automatically.

docker-compose up -d

The webserver is listening on port 5000 and can be reached here: http://localhost:5000

If you want to build the image yourself, use the docker-compose-dev.yaml file:

docker-compose -f docker-compose-dev.yaml up

Without docker-compose

If docker-compose is not installed the command looks like this:

docker run -p 5000:5000 --name organice twohundredok/organice:latest

Again the webserver is listening on port 5000 and can be reached here: http://localhost:5000

Heroku

Assuming, you have an account and have installed the command line tools, deployment is as easy as:

heroku create
heroku config:set ON_HEROKU=1
git push heroku master

Synchronization back-ends

Note that logging in to Dropbox will only work if you’re running the app on http://localhost:3000, because all redirect URIs must be specified ahead of time on the Dropbox developer console.

To configure your own application on Dropbox, please go here and then configure this app key in the .env file. Make sure to add your own URL as Redirect URI.

To configure your own application on Google Drive, please generate an API key as described on this page.

WebDAV

General

With WebDAV support, organice can potentially be used with a multitude of synchronization backends: Client/Server services ownCloud, Nextcloud and Seafile, but also self hosted dedicated WebDAV servers like Apache or Nginx.

More information

In the WebDAV FAQ, you’ll find lots more information regarding WebDAV:

• A screencast of how organice works when logging in to a WebDAV server
• Documentation how on to setup your own WebDAV Server with Apache2 on Debian
• Documentation how to configure Nextcloud behind haproxy to allow WebDAV
• Documentation on Nextcloud sharing

Routing

Whilst organice is a true Single Page Application (SPA) and therefore has no back-end whatsoever, this does have an implication for deployment with regard to routing. For routes like example.com/foo to work, we need a little something extra. Within the context of a running SPA, /foo would be matched by the React Router and the proper page would be rendered by JavaScript. When initially requesting a route like that from the web server itself, the SPA is not running yet and the web server itself wouldn’t find a file called /foo. It would return a 404. The whole topic is explained in depth in this SO answer: https://stackoverflow.com/a/36623117

For https://organice.200ok.ch we’ve opted to:

• Use the modern HTML5 history API with BrowserRouter
• Not configure a back-end for isomorphic routing, because it would complicate application and deployment unnecessarily (SEO is a non-issue for organice)
• Use good old Apache Webserver for hosting the compiled static assets

Therefore configuring a catchall is as easy as setting up a .htaccess file in the root of the organice folder containing:

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
RewriteRule ^ - [L]

RewriteRule ^ /index.html [L]

N.B.: If you’re using WebDAV as a sync back-end, and the RewriteRule is allowed to apply to a WebDAV directory, it will break PUT requests to upload new files! Here’s documentation on how to configure both features together correctly.

Contrib

organice aims to follow the ‘batteries included’ philosophy. For example, the documentation is rather extensive and includes wider topics than just its own functionality - for example it includes documentation on various tested deployment strategies.

However, the community regularly comes up with a whole range of options on how to use organice more effectively for specific use-cases. Sometimes, these options are generic enough so that the maintainers take the functionality into core. Sometimes, it’s not that well suited to be added into core, but still is potentially very well suited to a wider range of users. For that, organice follows the contrib model which many bigger projects use (i.e. Org mode) for such contributions.

The please see the contrib folder for details.

Capture templates

organice supports capture templates by implementing a flexible mechanism using URL parameters. These three of the following parameters are required and must be URL encoded:

• captureTemplateName: the name of the capture template to use. This capture template must already exist in Settings > Capture templates.
• captureFile: the path (for Dropbox) or id (for Google Drive) of the file in which to execute the capture template.
• captureContent: the content you’d like to capture. This content will be placed at the cursor position if specified in the capture template (with %?), or at the end of the template if its not specified.

You can also specify additional custom variables for use in your templates. They should be in the format captureVariable_<your custom variable>, and should also be URL encoded. In your capture template they’d show up as %<your custom variable>.

organice allows you to specify where the captured content will be inserted, via a “header path” which is a list of headers to match. If the list is empty, the content will be inserted at the end of the file, or the beginning if the prepend option is selected.

Examples

Simple: Capture a string

Say, you want to capture thoughts/todos as they occur to you. You might want to have a capture template to just get these things out of your head.

This makes for a good “Inbox” capture template:

Capture Template

* TODO %?
%U

Example URL

https://organice.200ok.ch?captureTemplateName=Inbox&captureContent=Read+up+on+capture+templates&captureFile=/org/things.org

Result

* TODO Read up on capture templates
[2019-09-08 Sun 20:54]

With custom variable

If you want to add web pages to a reading queue (with a title, a capture date and a URL), this would be a good starting point:

Capture Template

* %?
%u

- URL: %mediaURL

Example URL

https://organice.200ok.ch?captureTemplateName=Media&captureContent=Play+Emacs+like+an+instrument&captureFile=/org/media.org&captureVariable_mediaURL=https://200ok.ch/posts/2018-04-27_Play_Emacs_like_an_Instrument.html

Result

* Play Emacs like an instrument
[2019-09-08 Sun]

- URL: https://200ok.ch/posts/2018-04-27_Play_Emacs_like_an_Instrument.html

Bookmarklets

Since organice is a web application, you can use the capture templates feature to create bookmarklets, of course! For example, if you want a bookmarklet to add the current page (title, capture date and URL) to your reading queue using this capture template, all you need is a little bit of JavaScript:

javascript:(function() {
  const {title} = document;
  const url = `https://organice.200ok.ch?captureTemplateName=Media&captureContent=${title}&captureFile=/org/media.org&captureVariable_mediaURL=${
  window.location.href
}`;
  window.open(url, "_blank");
})()

Bookmarklets Demo

iOS

This is what using a bookmarklet to capture a website looks like in iOS:

Siri integration

The organice capture mechanism integrates very nicely with the Siri Shortcuts feature in iOS, allowing you to use Siri to execute capture templates.

You can use this sample Shortcut to get started with this right away in iOS 12 or newer. Open the link on your iOS device and click “Get Shortcut”. Then open up the Shortcuts app and edit the template by following the directions in the comments. Then record a Siri trigger and you’re good to go!

Comparison

Beorg

Before starting work on organice, @munen (the original maintainer) used Beorg and donated to it multiple times, because he was very happy to have a good option to access Org files on my phone with it.

The important differences to him were:

• organice is FOSS which is very much in the spirit of Org whilst Beorg is proprietary
• organice is web based, so there is no lock-in to a specific device or OS
• Beorg currently has better offline support

org-web

organice has a shared history with org-web. In fact, it is a friendly fork. organice differs from org-web in that:

• organice is a community driven project. See our
  • Code of conduct
  • Contributing guidelines
  • Community chat: #organice on IRC Freenode, or #organice:matrix.org on Matrix
• organice has the commitment of a Swiss company (200ok llc: https://200ok.ch/) behind it to continually work on it.
  • 200ok has a strong track record in fostering Free and Open Source Software (https://200ok.ch/floss.html) and has co-organized EmacsConf 2019.
  • That’s also why organice is Free Software (with the strong AGPL-3.0 license) whereas org-web is Open Source (with The Unlicense).
  • The continuous effort yields a certain power over time. At the time of writing this, organice has ~3 times as many commits (~1700 vs 600) and contributors (29 vs. 8). Of course, quantity doesn’t trump quality. However, many of the new contributors brought significant features and improvements, not just tiny patches.
• organice initially focused on becoming bug free - for example on parsing and exporting org files correctly.
• organice continues to evolve independently with its own feature set. For example, it has WebDAV support.
• organice is a project with equal focus on mobile as desktop browsers.
• org-web tracks users with Google Analytics. organice does not.
• organice has great documentation: https://organice.200ok.ch/documentation.html

What’s new?

To see how organice differs from org-web, please consult the changelog which contains the user visible changes since forking.

Acknowledgment

We are extraordinarily grateful to DanielDe, the original creator!

We forked the project, because we have different visions on how to go forward. He envisions a mobile only solution, we think it’s great to have organice be available to any browser to enable anyone on the go or any non-Emacs user easy access to Org files. Also, DanielDe thinks of org-web as his pet project whereas organice has the full power of 200ok llc behind it whilst building a strong self-sufficient community around it.

Thank you for all, DanielDe!🙏

Attributions

Logo

Illustration credit: Vecteezy.com