reshinthadithyan / squeakily

A library for squeakily cleaning and filtering language datasets.

squeakily

This repository is heavily inspired by BigScience’s ROOTs project and EleutherAI’s The Pile.

The overall pipeline is as follows:

In this library, we define filtering as data instances being removed from the dataset based on some criteria and cleaning as data instances being modified in some way.

Install

pip install squeakily

How to use

Using the API

First, we need to define a datasource. squeakily accepts any Dataset object from the HuggingFace Datasets library. For example, we can use the wikitext dataset:

from datasets import load_dataset

ds = load_dataset("wikitext", "wikitext-103-v1", split="train[:1%]")

We simply need to wrap the Dataset object in a dictionary, with the key being the name of the datasource and the value being the Dataset object, the filter and cleaners. For example:

from squeakily.filter import check_char_repetition, check_flagged_words
from squeakily.clean import remove_empty_lines, normalize_whitespace

datasources = [
    {
        "dataset": ds,
        "columns": ["text"],
        "filters": [check_char_repetition, check_flagged_words],
        "cleaners": [remove_empty_lines, normalize_whitespace],
    },
    # ...
]

Warning

Note: The order of the filters and cleaning functions matter. Filters and cleaners are applied in the order they are defined.

Important

Note: As of now, we only use the first column of the given column names. This is because the squeakily library is designed to work with language datasets, which usually have a single column of text. Future versions will support multiple columns.

Finally, we can apply the filters and cleaners to the datasouces using a Pipeline object:

from squeakily.core import Pipeline

pipeline = Pipeline(datasources)
pipeline.run()

[11/16/22 04:32:57] INFO     Running datasource: wikitext                                                core.py:41

                    INFO     Running filter: check_char_repetition on text                               core.py:54

                    INFO     Running filter: check_flagged_words on text                                 core.py:54

                    INFO     Running cleaner: remove_empty_lines on text                                 core.py:57

[11/16/22 04:32:59] INFO     Running cleaner: normalize_whitespace on text                               core.py:57

Note

Note: If you want to run cleaners first, you can pass cleaning_first=True to the run function.

pipeline.run(cleaning_first=True)

If you need to run a filter or cleaner at the dataset level rather than the example level, you can pass global_filters or global_cleaners to the Pipeline.run function. For example:

from squeakily.filter import minhash_dedup

pipeline.run(global_filters=[minhash_dedup])

Note

Note: If you use global filters or cleaners, all datasets must have a common column name in order to properly concatenate them.

Note

Note: You can also specifiy if you want a specific dataset to be skipped by setting the skip_global parameter to True when defining the datasource.

datasources = [
    {
        "dataset": ds,
        "columns": ["text"],
        "filters": [check_char_repetition, check_flagged_words],
        "cleaners": [remove_empty_lines, normalize_whitespace],
        "skip_global": True,
    },
    # ...
]