Digital Pen Path Recovery From Handwriting
The system approaches the problem of extracting pen paths from images of handwritten text using a hybrid approach consisting of both classical image processing techniques, and Deep Neural Networks.
The result of this is a system that can take an image of a set of or a single character, and output a directed list of points representing the path the pen likely took to draw the character,
This system built with Python 3.7, and has only been tested with Python 3.7. Newer versions should work but have not been tested. It is recommended to use a venv when running this script for ease of use.
The following packages are dependencies for this project (non-exhaustive):
- NumPy
- MatPlotLib
- SciKit-Image
- TensorFlow 2.x
- opencv-python (OpenCV2)
- imutils
For the purposes of testing the system and the models, there is a collection of images supplied in the /demo/
directory which can be copied into the /test/image_input/
directory to allow you to run the path_recovery.py
script. Detailed instructions are below.
-
Empty the
/test/
directory to ensure there are no extraneous results from previous runs. Create this directory if it does not exist already -
Choose a set of input images from the
/demo/*
directory, and copy them into the a/test/image_input
directory -
Run the
/path_recovery.py
script with the parametersdirectory test/image_input
python3 path_recovery.py directory test/image_input
This will create many new directories in the
/test/
directory, showing each step the system took to produce the output. -
To help with visualising the results, you may run the
/evaluation.py
script to produce a board of images to allow for easier direct comparisonpython3 evaluation.py
This will separately compile all images in the
/test/prediction_image
,/test/adopted_image
, and/test/image_input
directories, and create an image with the corresponding name for each directory in the/test/
directory
For more information on these scripts, see the Directory and Script Structure section below
To evaluate the system against your own handwriting or examples of text, you can simply place an image in the /test/multi_char/
directory, then run the /image_processing/seperate_chars.py
script to extract each character out
from the image, and create individual character image files in the /test/image_input/
directory. Then you can simply
follow the instructions above to extract the pen paths.
-
Create the
/test/multi_char
directory and place your image containing multiple characters in it. -
Run the
/image_processing/seperate_chars.py
script to generate the individual character images.python3 image_processing/seperate_chars.py <input_image_path> <output_directory>
e.g.
python3 image_processing/seperate_chars.py test/multi_char/all-chars.jpeg test/image_input
the
<input_image_path>
andoutput_directory
parameters can be omitted, in which case the default valuestest/multi_char/all-chars.jpeg
andtest/image_input
will be used respectively -
Follow steps 3 & 4 in the Using the Demo Images section to extract edges from these images you have generated
This project includes a script in /neural_network/uji_encoder.py
that can generate training data from the UJIpenchars
dataset included in the /original_data/
directory. Follow the instructions in the
Directory and Script Structure section below to generate this data which you can
then use for testing the system.
This system makes use of modules that can be run individually if needed, for example to train the network. In order to
do this from the command-line, you must run the script as a module to ensure the relative imports still work, for
example to train you should run python3 -m neural_network.network
instead of the path directly.
The main entry point for the whole system. This script will take a single or a directory of input images, extract the
undirected edges from them using the image_processing/main.py
script, then run those edges through the trained model
with the neural_network/network.py
script to direct the edges.
All of this is done in the /test/
directory, where many additional directories will be created documenting each
step in detail. The script requires at least one image in the /test/image_input/
directory to run.
Syntax: python3 path_recovery.py <mode:single|directory> <input_image_path>
This will separately compile all images in the /test/prediction_image
, /test/adopted_image
, and
/test/image_input
directories, and create an image with the corresponding name for each directory in the
/test/
directory. These paths are all hardcoded.
Syntax: python3 evaluation.py
This directory is where any files/directories related to the manual running of the system are held. The
path_recovery.py
script will create many directories in here documenting the various steps it takes to produce an
output sequence from the input image. These are described below:
/test/image_input/
This directory contains all the 64x64 single-character images/test/adopted_path/
This directory contains the directed output sequences for each input image, after they have been run through the neural network and the input shape adoption algorithm./test/adopted_image/
Theadopted_path
sequences are plotted onto a 64x64px coordinate grid for visualisation purposes/test/image_output/
The undirected list of edges is output here from theimage_processing
portion of the system/test/prediction_path/
The directed pen path output of the neural network, before running through the shape adoption algorithm/test/prediction_image/
Every path in theprediction_path
directory is plotted on a 64x64px coordinate grid to allow for easy visual analysis of the networks performance/test/analysis_image/
Theimage_input
,prediction_image
, andadopted_image
images for each character are plotted next to each other to allow for visualisation of the systems output
Contains all scripts required to extract undirected edges from an input image. Aside from the main.py
and
seperate_chars.py
scripts, all the scripts are not intended to be run directly and are simply used to divide up the
structure of the program, and are instead called from the main.py
script.
The main entry point for the image processing portion of the system. The script can be run in 2 different modes,
single
where the script only processes a specified file, or directory
where the script process a whole directory of
files.
Run with three parameters depending on situation
python3 -m image_processing/main <mode:single|directory> <input_path> <output_path>
Where the input_path and output_path are paths to a file or a directory depending on what mode the script is run in.
This script uses OpenCVs contours and bounding boxes to create a new 64x64px image for every character in the image supplied to it.
Syntax: python3 -m image_processing.seperate_chars <input_file_path> <output_directory>
The <input_image_path>
and output_directory
parameters can be omitted, in which case the default values
test/multi_char/all-chars.jpeg
and test/image_input
will be used respectively
This script contains just two variables which are used in various image_processing
scripts. If SHOW_STEPS
is set to
True, throughout the various processing steps a OpenCV window will open showing the progress of the script, and the
various steps it is taking. Each step progresses after the WAIT_TIME
time has passed (in milliseconds). If
WAIT_TIME
is set to zero, the script only processes onto the next step after any keyboard input.
This directory contains all the scripts required to both train and use the neural network including creating the training data.
The main entry point for using the neural network. Intended to be used from another script if predicting but can be
called directly for training. All training data must be in the /test.nosync/
directory. This directory should contain
3 different directories: ground_truth
containing the ground truth pen paths; image_output
containing the undirected
edge output from the image_processing scripts
Syntax: python3 -m neural_network.network
Using the UJIpenchars dataset in the /orignal_data/
directory, this script creates the training data for the neural
network. It also employs some data augmentation techniques to create up to 200x the samples from the original dataset,
by using various scaling, offset and rotation factors. It will populate the /test.nosync/image_input
and
/test.nosync/ground_truth
directories. The image_processing/main.py
script should be run on the
/test.nosync/image_input
directory to populate the /test.nosync/image_output
directory required for training.
To change the number of samples generates by the script, simply change the SCALING_FACTOR_LIST
, OFFSET_LIST
, and
ROTATION_LIST
constants at the top of the script.
This script is run when the /image_processing/main.py
script is not able to extract edges for every file in the
/test.nosync/image_input
directory. It will remove any character from the /test.nosync/ground_truth
directory that
does not have a corresponding file in the /test.nosync/image_output
directory. This ensures the x and y datasets for
the neural networks training is of the same size and match with eachother