chris-ha458/dolma

Dolma is an open dataset of 3 trillion tokens from a diverse mix of web content, academic publications, code, books, and encyclopedic materials. It was created as a training corpus for OLMo, AI2 language model.

Dolma is available for download on the HuggingFace 🤗 Hub: huggingface.co/datasets/allenai/dolma. To access Dolma, users must agree to the terms of the terms of AI2 ImpACT License for Medium Risk Artifacts. You can also read more about Dolma in our announcement, as well as by consulting its data sheet.

This repository contains tools for generating and inspecting Dolma. To get started, install the Dolma Python library from PyPI.

pip install dolma

Usage

The dolma CLI can be access using the dolma command. To see the available commands, use the --help flag.

dolma --help

At the moment, the CLI supports three commands: tag, dedupe, and mix.

For all commands, configurations can be specified from command line, or by passing a YAML or JSON file using the -c flag. For example:

dolma -c config.yaml dedupe --dedupe.name "test"

The `tag` command

The tag command is used to run any of the built-in taggers on a set of documents. For example:

dolma tag \
    --experiment sample \
    --documents \
        's3://ai2-llm/pretraining-data/sources/common-crawl/test/v0/documents/**/*.json.gz' \
        's3://ai2-llm/pretraining-data/sources/common-crawl/test/v1/documents/*.json.gz' \
    --taggers random_number_v1 \
    --processes 2

This command will run the random_number_v1 tagger on all documents in the specified S3 paths. The results will be written to the s3://ai2-llm/pretraining-data/sources/common-crawl/test/v0/attributes/sample and s3://ai2-llm/pretraining-data/sources/common-crawl/test/v1/attributes/sample paths.

The `dedupe` command

The dedupe command is used to deduplicate a set of documents at the attribute level using a bloom filter. For example configurations, see directory tests/config. For example:

dolma dedupe -c tests/config/dedupe-paragraphs.json

The `mix` command

The mix command is used to mix documents from multiple sources, optionally filtering by attributes and/or performing string replacement. For example configurations, see directory tests/config. For example:

dolma mix -c tests/config/mixer.json

Development

Create a conda environment with Python >= 3.8. In this case, we use Python 3.10 and use Anaconda to create the environment.

conda create -n dolma python=3.10

After creating the environment, activate it and install necessary tools using the included makefile.

conda activate dolma
make setup

and restart your shell. Finally, to begin development, install the repository in editable mode using maturin.

make develop

To run tests, use the following command.

make test

You can choose to run just the Python or Rust tests by calling make test-python or make test-rust respectively.

You can skip S3 related tests by exporting DOLMA_TESTS_SKIP_AWS=True

DOLMA_TESTS_SKIP_AWS=True make test

Contributing

Before committing, use the following command

make style

Citation

If you use this repository, please cite it as:

@software{dolma,
    author = {{Soldaini, Luca and Lo, Kyle and Kinney, Rodney and Naik, Aakanksha and Ravichander, Abhilasha and Bhagia, Akshita and Groeneveld, Dirk and Schwenk, Dustin and Magnusson, Ian and Chandu, Khyathi}},
    license = {{Apache-2.0}},
    title = {{Dolma}},
    url = {https://github.com/allenai/dolma}
}

chris-ha458 / dolma