demuxfb - demultiplex exported facebook json chat data!

demuxfb is a Python package to reframe conversations from Facebook 'Download Your Information' json dumps into a more exact form, accounting for the different categorizations of messages that the json metadata itself does not distinguish.

Motivation

Facebook chats are very feature-rich. All within a chat, there may be polls, events, video and audio calls, embedded games, and more. But the Facebook 'Download Your Data' exportation format fails to reflect these features; many distinct categories of messages are structurally indistinguishable from ordinary text messages.

When analysing the dataset, should an automated event reminder, 'Reminder, 30 minutes until 10 PM.', be treated the same as a human-written message? This should be up to the analyst to decide, not the exportation format.

Scope

This package defines a method to 'parse' Facebook JSON data for single chats into a more useful and complete representative form. This form is intended to be used an intermediary for analysis on the dataset.

We achieve this by string-matching message content to known schemas for the special feature-driven messages. The JSON messages in the original time series are promoted into appropriate Python classes according to these rules.

This process is limited in several ways.

It is difficult to establish a formal spec of what these schemas are -- we are on the outside, so it has been up to human inference on a limited dataset. So, given this difficulty in verification, the rules may be wrong or incomplete in the first place, or may silently become obsolete in future Facebook updates.
Facebook features are region-specific. Currently, this package assumes an Australian dataset, so the inbuilt rules will fail to detect things like sending money through messenger (a U.S. feature).
The Facebook exporter compresses the natural data. For example, Alice typing 'Alice waved hello to the group.' is impossible to distinguish from Alice using the wave feature. We take a parsimonious approach in these kinds of situations where there is no single correct answer, even though different or more complex procedures may emperically be better in the relevant context.

Expect misclassification.

Installation

The recommended approach is to download the repository and copy the src/demuxfb folder to wherever you're using it, and just put import demuxfb as with any other source package. This allows you to easily modify the source to suit your exact needs, if necessary (see the Modification section).

Alternatively, if you're happy to use the code as is, you can download binaries from the Releases page. The date components of the version numbers signify that the release was built against a Facebook archive dataset generated on that date. It is recommended that you use fresh Facebook archives and the most recent release.

After choosing a version, the binary can be installed with

pip install demuxfb-VER.tar.gz

This should work for Python 3.8+, and may incidentally be compatible with earlier Python 3 releases.

Usage

You can download your Facebook data at https://www.facebook.com/dyi/. Before beginning the download, you must set the file format to 'JSON' and confirm that the 'Messages' option is checked.

Once the archive has been generated and you have downloaded it, unzip it. Your chats should then be located in the messages/inbox/ and messages/archived_threads/ subfolders. From here, a simple usage example would be as follows.

from pathlib import Path
import demuxfb

# 1. Point to where the conversation will be fed in from the disk.
path = Path('C:/users/nicho/downloads/facebook-nicholaskilleen/'
            'messages/inbox/ourchat_95kldfjg4')
feed = demuxfb.ChatFolderFeed(path)

# 2. Create the chat (takes a while, reports progress to stdout).
chat = demuxfb.build_chat(feed, 'Nicholas Killeen')

# 3. Do some stub analysis with it.
print('Number of text messages in the conversation:',
      len([message for message in chat.messages
           if isinstance(message, demuxfb.message.TextMessage)]))

Documentation

The documentation is available online at https://nick-killeen.github.io/demuxfb/. You can also read it in source or with help(demuxfb) in Python, or can compile it with Sphinx by using the Makefile in the docs_src/ directory.

Modification

You may want to extend or modify demuxfb's vocabulary to work better with your dataset. This is not possible to do through the package interface, so you will need to modify the source. This section can help get you started.

demuxfb works by iterating through all JSON messages, for each consulting an ordered list of rules until one of them 'matches' the current JSON iterate and produces an appropriate message object. It is these rules that define the parsing logic, so they will generally be the focus of any modifications or additions you would like to make.

All rules are defined in the src/demuxfb/_rules.py file. For example, consider the scenario of wanting to recognize JSON snippets similar to the following as PlanDeletionMessages:

{
  "sender_name": "Jacob Smith",
  "timestamp_ms": 1478855024210,
  "content": "Jake deleted the plan Saturday Hangout for Sat, Aug 5 at 12 PM.",
  "type": "Generic"
}

Then, in the src/demuxfb/_rules.py file, we might have the following rule:

@_register_rule()
def _match_plan_deletion_message(cf: '_ChatFactory'
                                 ) -> Optional[msg.PlanDeletionMessage]:
    if cf.match([_Tok.SENDER_ALIAS, ' deleted the plan ', _Tok.PLAN_TITLE,
                 ' for ',  _Tok.PLAN_DATE_TIME]):
        message = cf.make_common(msg.PlanDeletionMessage)
        message.plan_title = cf.captures[_Tok.PLAN_TITLE]
        message.plan_date_time = cf.captures[_Tok.PLAN_DATE_TIME]
        return message
    
    return None

All rules are registered with the @_register_rule decorator. The order in which these registrations appear in source will correspond to the precedence of rules, more general rules being defined at the bottom of the file.

Rules have two possible return types: they return None to indicate their non-applicability to the current state, causing demuxfb to move on to the next rule; or they instantiate and return a specialized message object which will be copied to the output structure (here PlanDeletionMessage). These types are defined in the src/demuxfb/message.py file, where you are free to add your own or change the existing ones.

Rules take in a single _ChatFactory argument that encompasses the world state. This state persists throughout the sequential parsing process, and includes:

A message_json property that contains the current JSON that is meant to be turned into a message.
An identity manager for the participants in the conversation.
Token definitions (which can be updated dynamically, though no rule currently makes use of this).
Scoping variables, such as plan_is_active -- one might expect that a genuine plan deletion event will not occur if there does not exist an ongoing plan, so, we can set and unset this variable when we recognize plan deletion and creation events and use it accordingly (see the actual _match_plan_deletion_message in source for an example).

The relevant documentation for this starts at the definition of _ChatFactory in the src/demuixfb/_chat.py file, but an explicit visit there is not immediately necessary, since the existing rules in src/demuxfb/_rules.py exemplify the repetitive use of small collection of patterns.

nick-killeen / demuxfb