Giters

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.

Home Page:https://allenai.github.io/dolma/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How does Exact paragraph deduplication performed?

silverriver opened this issue · comments

commented

hi, thank you for your great work.

I am wondering how exactly does the "Exact paragraph deduplication" operation is carried out?

For my understanding, "Exact paragraph deduplication" follows these steps:

  1. split each document to paragraphs
  2. detect dupliacted paragraphs (using the bloom filter)
  3. remove dupliacted paragraphs.

However, there are a few questions:

  1. for step 3, assume you have a paragraph that is dupliacted for N times. Then it is reasonable to remove N-1 dupliacted ones. I am wondering which one of the N paragraph should be retrained and which N-1 ones should be removed?
  2. removing a paragraph from a document will usually hurt the original document. Will it down grade the data quality?
commented

hello!

regarding your questions:

  1. We keep the first one seen by the deduper. the process is non-deterministic.
  2. Paragraph deduplication is fairly standard practice when training LM models, so shouldn't be an issue.