Save a GMail query as a local maildir
Overview
A simple CLI tool to save a GMail query as a local maildir.
There are two main use-cases:
- Archive a conversation for offline access or backup.
- Perform incremental backups of an open-ended query (e.g. all non-inbox emails)
Gmail-archive uses Google Auth 2.0 for authentication. On the first run, it will open your browser and ask for permission to give your gmail-archive instance read-only access to to your Gmail account. Access and refresh tokens are stored on disk, so this step isn't required on subsequent runs.
Install
npm install -g gmail-archive
This will make the gmail-archive
command available on your PATH
. Use gmail-archive --help
to see the available actions and options.
Usage
Archive a conversation
For example, let's say you want to archive the thread with funny gifs in April 2017 under the ~/mail/gifs2017
directory.
gmail-archive init \
--dir ~/mail/gifs2017 \
--query "has:attachment subject:gif" \
--before 2017-05-01 \
--after 2017-04-01 \
--client-id <google_client_id> \
--client-secret <google_client_secret>
See below for how to obtain Google API credentials. This will open an OAUTH window in your default browser and cache your OAUTH tokens to disk
Then sync the maildir:
gmail-archive fetch --dir ~/mail/gifs2017 --verbose
The sync uses only atomic writes to the maildir and can be safely interrupted and resumed at any
time. Subsequent calls to gmail-archive fetch
will only download messages that haven't already been
downloaded.
Incremental backups of the GMail's "All Mail" folder
Let's use the gmail-archive cli to create a monthly maildirs that syncs all archived (i.e., non-inbox emails) from your Gmail. This will result in, e.g., the following maildir hierarchy.
<root>/
2014.01/
cur/
new/
tmp/
2014.02/
cur/
new/
tmp/
...
2018.02/
cur/
new/
tmp/p
First configure the archive.
gmail-archive init \
--dir ~/mail/archive \
--query "-label:inbox" \
--period month \
--client-id <google_client_id> \
--client-secret <google_client_secret>
We are now ready to sync the archive. The first run will fetch all matching messages, and subsequent runs will efficiently fetch only the new messages.
gmail-archive fetch --dir ~/mail/archive --verbose
Obtaining Google API credentials
You need to register gcal-to-org as an application yourself to obtain a Client ID and Client secret.
Go to the Google API Manager and create a new project under any name.
Within that project, enable the "GMail" API. There should be a searchbox where you can just enter those terms.
In the sidebar, select "Credentials" and then create a new "OAuth Client ID". The application type is "Other".
You’ll be prompted to create a OAuth consent screen first. Fill out that form however you like.
Finally you should have a Client ID and a Client secret. Provide these in the config below.
Notes
This tool is in the early stages and addresses my particular needs. However, it should be easy to adapt it to your use-case, and I would be happy to iterate on the design.
The fetch command's --concurrency
option controls how many emails to download in parallel. The default
value, 10, provides an order-of-magnitude speedup over IMAP sync tools like offlineimap.
Due to limitations of Gmail's API, entire emails must be stored in memory before writing them to disk. Large attachments can therefore cause the Node VM to run out of memory. Should you experience this, consider increasing the size of Node's old space.
export NODE_OPTIONS=--max_old_space_size=4096
gmail-archive [OPTIONS]
License
Copyright © 2017-2018 John Whitbeck
Distributed under the Apache License Version 2.0.