Many of us are using yahoo-group-archiver to back up Yahoo Groups API results. This script takes the output of that tool, and converts it into individual email files, mbox mail folders, and optionally, PDF files.
Mail folders stored as mbox can be imported by a wide range of desktop and server-side email clients, including Thunderbird (Linux, Mac, Windows), Apple Mail.app (Mac), Microsoft Outlook (Windows and Mac).
Many non-technical users won't know what to do with an mbox file, but will really appreciate getting a PDF file containing all the emails in the list. You can enable experimental PDF support by installing Andrew Ferrier's email2pdf script. This process is known to be buggy, and your bug reports would be appreciated.
- Perl 5.14 or higher
- several Perl modules installed via CPAN:
- CAM::PDF
- Email::MIME
- Email::Sender
- HTML::Entities
- HTML::FormatText::WithLinks::AndTables
- IO::All
- IPC::Cmd
- JSON
- List::AllUtils
- Log::Dispatch
- MCE
- Pandoc
- Sort::Naturally
- Text::Levenshtein::XS
- autodie
- Optional: if you're creating PDF files for lists with more than say 10,000 emails, you'll probably need to install
qpdfto avoid running out of memory (there are packages for Yum/RPM, Debian/Ubuntu, and MacOS brew)
mkdir output-dir
yahoo-group-archive-tools.pl --source <archived-input-dir> --destination <output-dir>
Start by installing Andrew Ferrier's email2pdf script. It can be a little complicated to install, but giving someone a PDF file of their list can elicit delight. This is experimental, so bug reports are appreciated.
mkdir output-dir
yahoo-group-archive-tools.pl --source <archived-input-dir> --destination <output-dir> --pdf --email2pdf <path to email2pdf Python script>
The output directory will contain:
- An
emailfolder containing standalone email files for every email in the archive, e.g.email/1.eml,email/2.eml. The emails won't be pristine, because Yahoo redacts email addresses (see that and other caveats below). The email IDs reflect those downloaded by yahoo-group-archiver, and it's normal to see some gaps in keeping with the original numbering. - A consolidated mailbox file,
mbox/list.mbox, for the entire history of the list - With PDF support enabled, a
pdf-individualdirectory containing individual PDFs for every email - With PDF support enabled, a
pdf-combineddirectory with a single PDF file containing every email
- This tool builds on output from IgnoredAmbiance's Yahoo Group Archiver, and has also been tested with the ArchiveTeam fork
- Read more about the Yahoo Groups archiving process, the tools people are using, and the community of people doing the work at ArchiveTeam Yahoo Groups project
The Yahoo Groups API redacts emails found in message headers. For
example, they'll rewrite ceo@ford.com as ceo@....
Why is this bad?
- Deleting hostnames from headers could cause the emails to be unparseable by client software expecting valid hostnames.
- It's hard for people to tell the difference between users. For example,
ceo@ford.comandceo@toyota.comlook the same if both are truncated toceo@....
Because the API tells us the submitting Yahoo user's username, we can make a fake email domain that preserves the part before the @ in redacted emails, while being unique per user.
- Imagine the CEO of Ford,
ceo@ford.com(Yahoo IDfordfan), emails the list:- Yahoo Groups redacts the hostname, and saves that as
ceo@... - We turn that it into
ceo@fordfan.yahoo.invalid
- Yahoo Groups redacts the hostname, and saves that as
- Then the CEO of Toyota,
ceo@toyota.com(Yahoo IDtoyotalover123), emails the list:- Yahoo Groups also saves that as
ceo@...even though this is a totally different person - But we turn that email it into
ceo@toyotalover123.yahoo.invalid, which is different fromceo@fordfan.yahoo.invalid
- Yahoo Groups also saves that as
We make this change in several headers that include the original sender's email, including From and Message-Id. We save the original redacted version as an X- header. For example, if Yahoo says an email is From: ceo@..., we modify that to From: ceo@ceo123.yahoo.invalid, and save the original as X-Original-Yahoo-Groups-Redacted-From: ceo@.... If we don't have a Yahoo profile name (e.g. "ceo123"), we use the numeric Yahoo user ID (e.g. "123456789") instead.
The Yahoo Groups API detaches all attachments, and saves them in a separate place.
We try to stitch the emails back together, navigating through the MIME structure to attach the right attachment at the right place. In some cases, we're not able to identify where in the email MIME structure an attachment goes, so we reattach orphaned attachments to the whole email. In some cases, Yahoo doesn't give us the attachment, so we replace the attachment with a text part containing an error message, with original attachment-related headers added (X-Yahoo-Groups-Attachment-Not-Found, X-Original-Content-Type, X-Original-Content-Disposition, X-Original-Content-Id).
The Yahoo Groups API forcibly truncates email messages with over 64 KB in text, and places a truncation message right in the middle of encoded content, e.g. Base64.
Whenever we see an email body that end with (Message over 64 KB, truncated), we remove that string from the broken message part, and pray that downstream parsers will be able to deal with truncated HTML, Base64, etc. We mark these message parts with a X-Yahoo-Groups-Content-Truncated header.
The Yahoo Groups API appears to be decoding and recoding textual message bodies, because we see Unicode "U+FFFD" replacement characters in the raw RFC822 text that should be 7-bit clean. We're also seeing ^M linefeeds at the end of every header line and MIME body part.
We remove invalid linefeeds and 8-bit characters from 7-bit RFC822 text.
Installing this script requires installing many CPAN dependencies. If you're confused, feel free to search for things like "installing CPAN ". In some cases, an environment's package manager may have all these scripts. At least one Ubuntu user was able to install all the dependencies using the package manager (all but one: they could install the Text::LevenshteinXS module rather than Text::Levenshtein::XS, so they just changed the dependency in the Perl script to match.)
This tool directly executes the email2pdf script specified by the --email2pdf option. Make sure the #! shebang line is set to the Python interpreter of your choice. You can test email2pdf execution by manually running something like email2pdf --headers -i <a .eml file> --output-file <the-filename-to-write.pdf> on a single email/[number].eml file generated by this script.
Significant changes:
- 2020-12-09: Saves list description metadata in several formats
- 2020-04-07: Can process only specific emails, helpful for spam filtering
- 2020-02-11: One more strategy to turn ornery emails into PDF
- 2020-01-23: Fixes bug in some emails with X-eGroups-Approved-By: header
- 2020-01-21: Fixes bug where PDF Date/Subject sometimes listed as 'ARRAY'
- 2020-01-14: Uses qpdf to create combined PDFs for lists with many emails
- 2020-01-05: Faster PDF generation, with fewer conversion errors
- 2019-12-17: Checks /topics/ for attachments and email descriptions
- 2019-12-11: Checks /attachments/ for attachments
- 2019-12-10: Support for PDF generation
- 2019-12-07: First solid release
- Catch and solve some of the most common email2pdf errors
- Maybe fix redacted headers in sub-parts so the message is valid
- Need to verify that attached files round trip correctly
This software is copyright Anirvan Chatterjee, and licensed under the MIT License.
Have questions, bug reports, or suggestions? Feel free to use GitHub's issue tracker. If you need to contact me privately, DM me @anirvan on Twitter.