This package provides code for producing training data for optical character recognition and handwritten text recogntion (OCR and HTR) by aligning the output of an initial model on a collections of images with a collection of digital editions of similar texts.
First, install passim
. Then install kraken
. If you want to start with PDF files of books rather than page images, use the pdf
option:
pip install --user kraken[pdf]
After this is complete, the programs seriatim
, kraken
, and ketos
should be in your PATH and available on the command line.
Install the scripts in this package with:
pip install --user .
We use make
to manage OCR of a potentially large number of input pages. Create a directory for your work, go into that directory, and link to the Makefile
in this package:
ln -s <path to src>/acdc_train/etc/Makefile
If you're starting with PDF files, put them in a subdirectory named pdf
. If you're starting with individual page image files instead, create a directory named images
with subdirectories each containing the page image files for a book.
If you put plain text files in a directory named electronic_texts
, they will be interpreted with OpenITI markdown. If you prefer, you could put JSONL-formatted input in electronic_texts.json
. This uses the passim
conventions of an id
field for a unique document identifier and text
field, potentially with escaped newlines, for the text.
In the paper, we bootstrapped training starting from page segmentation and transcription models trained on printed Arabic-script books. You can change the segment
and ocr
variables in the Makefile
to train from a different starting model.
You should then be able to run experiments with three rounds of OCR'ing the books in pdf
or images
and retraining by running this make
command:
make gen2-print-n7m5.out/alto-union/ft_best.mlmodel
If any of the steps in the pipeline complain about running out of memory, try setting the following environment variable, e.g., for 20G:
export SPARK_SUBMIT_ARGS='--executor-memory 20G --driver-memory 20G'