This is a Python script to mimic the functionality of Pinboard's Twitter integration. It reads the latest toots from a Mastodon account and bookmarks them in a Pinboard.in account. It is meant to be run repeatedly as a crontab job to continuosly update your bookmarks in the background.
usage: python3 masto-pinb.py [-h] [--toots] [--log_json] [--favs] [--bmarks]
[--dry_run] [--verbose] [--get_last GET_LAST]
Arguments for Mastodon-Pinboard bookmarker app
optional arguments:
-h, --help show this help message and exit
--toots Bookmark user toots
--log_json Log toots in JSON format to local file
--favs Bookmark user favorites
--bmarks Bookmark user Mastodon bookmarks
--dry_run Dry run: don't actually bookmark anything
--verbose Print actions as they occur
--get_last GET_LAST retrieve only last n toots (default=20)
It is designed to be run periodically. The period
between runs depends on your profligacy as a tooter and how much
latency you care about between when you post and when a toot is
bookmarked. Running this often will retrieve (but not bookmark)
redundant toots, but decrease latency; running it less often will
increase latency, but decrease API usage and possibly miss toots. You
may increase the GET_LAST value if you think you will generate more
than the default 20 toots returned by the API between runs. I am not a
heavy tooter so I use the @hourly crontab shortcut to run it hourly,
with the default 20 toot recall.
This does not use the Mastodon pagination API so at most the 40 last toots (--get_last 40) can be retrieved on any run,
including the first run. To backup more of your history, use the masto-backup.py script described below.
The Mastodon.social instance is rate-limited to something like 300 calls per 5 minutes. This script uses 3 calls at most per run; if the rate limit is exceeded it will exit with an exception -- this is fine as the script should pick up the unbookmarked posts on its next invocation. The Pinboard rate limit is not documented that I could find.
By default the script will bookmark all toots associated with the
account authorized in the usercred.secret file, as well as all toots
favorited and bookmarked by that user. This can be changed by any
combination of the --toots, --bmarks, and --favs command line
options; for example using only--favs will bookmark only favorites,
while --toots --bmarks wil bookmark user toots and bookmarks but not
favorites.
pip3 install Mastodon.py will install the required Mastodon Python libraries,
pip3 install "pinboard>=2.0" will install thre required Pinboard libriaries.
pip3 install html2txt will install the required html2text library by Aaron Swartz.
You will also need API tokens from both Pinboard and Mastodon. These are stored as text files with a .secret extension. (The .secret extension are included in the .gitignore so they are not checked into the repository). They should be kept in the same directory as the masto-pinb.py Python script.
The Pinboard credentials can be obtained here https://pinboard.in/settings/password and should be stored as a single line of text in the file pinboard_auth.secret
The Mastodon user credential file is stored as masto_pinb_usercred.secret and can be generated by following the instructions at https://mastodonpy.readthedocs.io/en/stable. You will need to register your app, but this need only be done once -- you can use register_app.py as a template to do this if you are not using 2FA.
Several files are generated in the same directory as the masto-pinb.py Python script. In order to avoid duplicating Pinboard API calls, the IDs of recently-bookmarked toots are stored in the local text files cached_bmarks.secret, cached_toots.secret, and cached_favs.secret. (Though not particularly sensitive, these have the .secret extension so that git will ignore them via .gitignore.) These files are truncated to the 120 most recent toots on every run so they will not grow large.
If the --log_json command line argument is specified, every bookmarked toot is also stored locally, appended to the appropriate json file named toots.json, favs.json, and bmarks.json. The size of these files is not managed and they may grow large.
The script masto-backup.py will archive your posts, favorites,
and/or bookmarks as text json files. It needs a Mastodon credential
file as above (masto_pinb_usercred.secret) but does not need
Pinboard credentials. It uses the paging functions of the API and will
pace requests to not exceed API rate limits.
This is likely not as useful as the native Mastodon exporter, but might be handy as a scaffold for your own exploration. (Yes there's a lot of code reuse here with the bookmarking script, but they are just different enough that I duplicated instead of modularized.)
usage: python3 masto-backup.py [-h] [--toots] [--favs] [--bmarks] [--verbose]
[--all_pages] [--get_last GET_LAST]
[--get_n_pages GET_NPAGES] [--page_wait PAGE_WAIT]
Arguments for Mastodon backup app
optional arguments:
-h, --help show this help message and exit
--toots Bookmark user toots
--favs Bookmark user favorites
--bmarks Bookmark user Mastodon bookmarks
--verbose Print actions as they occur
--all_pages Tries to get all pages available. May take some time!
--get_last GET_LAST retrieve only GET_LAST toots per page (default=40)
--get_n_pages GET_NPAGES
retrieve N_PAGES pages (default = 1). Ignored with --all_pages
--page_wait PAGE_WAIT
wait WAIT_SECONDS between page requests to stay inside API limits (default= 1.0
If none of --toots, --favs, and/or --bmarks is specified, all
are assumend and each will be archived in seperate files. Json output files
are named MODE-backupYYYY-MM-DD.json where MODE is one of toots,
favs, and/or bmarks depending on command line arguments.