myint / untokenize

Transforms tokens into original source code (while preserving whitespace)

untokenize

untokenize transforms tokens into source code. Unlike the standard library's tokenize.untokenize(), it preserves the original whitespace between tokens.

Usage

import untokenize
source_code = untokenize.untokenize(tokens)

Tests

To run the unit tests:

$ ./test_untokenize.py

There is also an acid test. It tokenizes Python code and confirms that the code generated by untokenize exactly matches the original source code from before tokenization:

$ ./test_acid.py