preprocess.py -- a portable multi-language file preprocessor
============================================================
Download the latest preprocess.py packages from here:
(archive) http://preprocess.googlecode.com/files/preprocess-1.1.0.zip
Home : http://trentm.com/projects/preprocess/
License : MIT (see LICENSE.txt)
Platforms : Windows, Linux, Mac OS X, Unix
Current Version : 1.1
Dev Status : Fairly mature, has been used in the Komodo build system for
over 7 years.
Requirements : Python >= 2.3 (http://www.activestate.com/ActivePython/)
What's new?
-----------
Support has been added for preprocessing TeX, Fortran, C#, Java, Shell
script and PHP files. See the [Change Log](#changelog) below for more.
Why preprocess.py?
------------------
There are millions of templating systems out there (most of them
developed for the web). This isn't one of those, though it does share
some basics: a markup syntax for templates that are processed to give
resultant text output. The main difference with `preprocess.py` is
that its syntax is hidden in comments (whatever the syntax for comments
maybe in the target filetype) so that the file can still have valid
syntax. A comparison with the C preprocessor is more apt.
`preprocess.py` is targetted at build systems that deal with many
types of files. Languages for which it works include: C++, Python,
Perl, Tcl, XML, JavaScript, CSS, IDL, TeX, Fortran, PHP, Java, Shell
scripts (Bash, CSH, etc.) and C#. Preprocess is usable both as a
command line app and as a Python module.
Here is how is works: All preprocessor statements are on their own
line. A preprocessor statement is a comment (as appropriate for the
language of the file being preprocessed). This way the preprocessor
statements do not make an unpreprocessed file syntactically incorrect.
For example:
preprocess -D FEATURES=macros,scc myapp.py
will yield this transformation:
... ...
# #if "macros" in FEATURES
def do_work_with_macros(): def do_work_with_macros():
pass pass
# #else
def do_work_without_macros():
pass
# #endif
... ...
or, with a JavaScript file:
... ...
// #if "macros" in FEATURES
function do_work_with_macros() { function do_work_with_macros() {
} }
// #else
function do_work_without_macros() {
}
// #endif
... ...
Despite these contrived examples preprocess has proved useful for
build-time code differentiation in the
[Komodo](http://www.activestate.com/Komodo) build system -- which
includes source code in Python, JavaScript, XML, CSS, Perl, and C/C++.
The #if expression (`"macros" in FEATURES` in the example) is Python
code, so has Python's full comparison richness. A number of
preprocessor statements are implemented:
#define VAR [VALUE]
#undef VAR
#ifdef VAR
#ifndef VAR
#if EXPRESSION
#elif EXPRESSION
#else
#endif
#error ERROR_STRING
#include "FILE"
As well, preprocess will do in-line substitution of defined variables.
Although this is currently off by default because substitution will occur
in program strings, which is not ideal. When a future version of
preprocess can lex languages being preprocessed it will NOT substitute
into program strings and substitution will be turned ON by default.
Please send any feedback to [Trent Mick](mailto:trentm@google's mail
thing.com).
Install Notes
-------------
Download the latest `preprocess` source package, unzip it, and run `python
setup.py install`:
unzip preprocess-1.1.0.zip
cd preprocess-1.1.0
python setup.py install
If your install fails then please visit [the Troubleshooting
FAQ](http://trentm.com/faq.html#troubleshooting-python-package-installation).
This will install `preprocess.py` into your Python `site-packages` and
also into your Python bin directory. If you can now run `preprocess`
and get a response then you are good to go, otherwise read on.
The *problem* is that the Python bin directory is not always on your
PATH on some operating systems -- notably Mac OS X. To finish the
install on OS X either manually move 'preprocess' to somewhere on your
PATH:
cp preprocess.py /usr/local/bin/preprocess
or create a symlink to it (try one of these depending on your Python
version):
ln -s /System/Library/Frameworks/Python.framework/Versions/2.3/bin/preprocess /usr/local/bin/preprocess
ln -s /Library/Frameworks/Python.framework/Versions/2.4/bin/preprocess /usr/local/bin/preprocess
(Note: You'll probably need to prefix those commands with `sudo` and
the exact paths may differ on your system.)
Getting Started
---------------
Once you have it install, run `preprocess --help` for full usage
information:
$ preprocess --help
Preprocess a file.
Command Line Usage:
preprocess [<options>...] <infile>
Options:
-h, --help Print this help and exit.
-V, --version Print the version info and exit.
-v, --verbose Give verbose output for errors.
-o <outfile> Write output to the given file instead of to stdout.
-f, --force Overwrite given output file. (Otherwise an IOError
will be raised if <outfile> already exists.
-D <define> Define a variable for preprocessing. <define>
can simply be a variable name (in which case it
will be true) or it can be of the form
<var>=<val>. An attempt will be made to convert
<val> to an integer so "-D FOO=0" will create a
false value.
-I <dir> Add an directory to the include path for
#include directives.
-k, --keep-lines Emit empty lines for preprocessor statement
lines and skipped output lines. This allows line
numbers to stay constant.
-s, --substitute Substitute defines into emitted lines. By
default substitution is NOT done because it
currently will substitute into program strings.
Module Usage:
from preprocess import preprocess
preprocess(infile, outfile=sys.stdout, defines={}, force=0,
keepLines=0, includePath=[], substitute=0)
The <infile> can be marked up with special preprocessor statement lines
of the form:
<comment-prefix> <preprocessor-statement> <comment-suffix>
where the <comment-prefix/suffix> are the native comment delimiters for
that file type.
Examples
--------
HTML (*.htm, *.html) or XML (*.xml, *.kpf, *.xul) files:
<!-- #if FOO -->
...
<!-- #endif -->
Python (*.py), Perl (*.pl), Tcl (*.tcl), Ruby (*.rb), Bash (*.sh),
or make ([Mm]akefile*) files:
# #if defined('FAV_COLOR') and FAV_COLOR == "blue"
...
# #elif FAV_COLOR == "red"
...
# #else
...
# #endif
C (*.c, *.h), C++ (*.cpp, *.cxx, *.cc, *.h, *.hpp, *.hxx, *.hh),
Java (*.java), PHP (*.php) or C# (*.cs) files:
// #define FAV_COLOR 'blue'
...
/* #ifndef FAV_COLOR */
...
// #endif
Fortran 77 (*.f) or 90/95 (*.f90) files:
C #if COEFF == 'var'
...
C #endif
Preprocessor Syntax
-------------------
- Valid statements:
#define <var> [<value>]
#undef <var>
#ifdef <var>
#ifndef <var>
#if <expr>
#elif <expr>
#else
#endif
#error <error string>
#include "<file>"
where <expr> is any valid Python expression.
- The expression after #if/elif may be a Python statement. It is an
error to refer to a variable that has not been defined by a -D
option or by an in-content #define.
- Special built-in methods for expressions:
defined(varName) Return true if given variable is defined.
Tips
----
A suggested file naming convention is to let input files to
preprocess be of the form <basename>.p.<ext> and direct the output
of preprocess to <basename>.<ext>, e.g.:
preprocess -o foo.py foo.p.py
The advantage is that other tools (esp. editors) will still
recognize the unpreprocessed file as the original language.
And, for module usage, read the preprocess.preprocess() docstring:
pydoc preprocess.preprocess
Change Log
----------
### v1.1.0
- Move to code.google.com/p/preprocess for code hosting.
- Re-org directory structure to assist with deployment to pypi and
better installation with setup.py.
- Pulled the "content.types" file that assists with filetype
determination into "preprocess.py". This makes "preprocess.py" fully
independent and also makes the "setup.py" simpler. The
"-c|--content-types-path" option can be used to specify
addition content types information.
### v1.0.9
- Fix the 'contentType' optional arg for #include's.
- Add cheap XML content sniffing.
### v1.0.8
- Allow for JS and CSS-style comment delims in XML/HTML. Ideally this
would deal with full lexing but that isn't going to happen soon.
### v1.0.7
- Allow explicit specification of content type.
### v1.0.6
- Add ability to include a filename mentioned in a define: '#include
VAR'.
### v1.0.5
- Make sure to use the *longest* define names first when doing
substitutions. This ensure that substitution in this line:
FOO and BAR are FOOBAR
will do the right thing if there are "FOO" and "FOOBAR" defines.
### v1.0.4
- Add WiX XML file extensions.
- Add XSLT file extensions.
### v1.0.3
- TeX support (from Hans Petter Langtangen)
### v1.0.2
- Fix a bug with -k|--keep-lines and preprocessor some directives in
ignored if blocks (undef, define, error, include): those lines were
not kept. (bug noted by Eric Promislow)
### v1.0.1
- Fix documentation error for '#define' statement. The correct syntax
is '#define VAR [VALUE]' while the docs used to say
'#define VAR[=VALUE]'. (from Hans Petter Langtangen)
- Correct '! ...' comment-style for Fortran -- the '!' can be on any
column in Fortran 90. (from Hans Petter Langtangen)
- Return a non-zero exit code on error.
### v1.0.0
- Update the test suite (it had been broken for quite a while) and add
a Fortran test case.
- Improve Fortran support to support any char in the first column to
indicate a comment. (Idea from Hans Petter Langtangen)
- Recognize '.f90' files as Fortran. (from Hans Petter Langtangen)
- Add Java, C#, Shell script and PHP support. (from Hans Petter
Langtangen)
### v0.9.2
- Add Fortran support (from Hans Petter Langtangen)
- Ensure content.types gets written to "bindir" next to preprocess.py
there so it can be picked up (from Hans Petter Langtangen).
### v0.9.1
- Fully read in the input file before processing. This allows
preprocessing of a file onto itself.
### v0.9.0
- Change version attributes and semantics. Before: had a _version_
tuple. After: __version__ is a string, __version_info__ is a tuple.
### v0.8.1
- Mentioned #ifdef and #ifndef in documentation (these have been there
for a while).
- Add preprocess.exe to source package (should fix installation on
Windows).
- Incorporate Komodo changes:
- change 171050: add Ruby support
- change 160914: Only attempt to convert define strings from the
command-line to *int* instead of eval'ing as any Python
expression: which could surprise with strings that work as
floats.
- change 67962: Fix '#include' directives in preprocessed files.
### v0.8.0
- Move hosting to trentm.com. Improve the starter docs a little bit.
### 0.7.0:
- Fix bug 1: Nested #if-blocks will not be properly handled.
- Add 'Text' type for .txt files and default (with a warn) unknown
filetypes to 'Text'. Text files are defined to use to '#...'-style
comments to allow if/else/.../endif directives as in
Perl/Python/Tcl files.
### 0.6.1:
- Fix a bug where preprocessor statements were not ignored when not
emitting. For example the following should _not_ cause an error:
# #if 0
# #error womba womba womba
# #endif
- Fix a bug where multiple uses of preprocess.preprocess() in the
same interpreter would erroneously re-use the same list of
__preprocessedFiles. This could cause false detection of recursive
#include's.
- Fix #include, broken in 0.6.0.
### 0.6.0:
- substitution: Variables can now replaced with their defined value
in preprocessed file content. This is turned OFF by default
because, IMO, substitution should not be done in program strings.
I need to add lexing for all supported languages before I can do
*that* properly. Substitution can be turned on with the
--substitute command-line option or the subst=1 module interface
option.
- Add support for preprocessing HTML files.
### 0.5.0:
- Add #error, #define, #undef, #ifdef and #ifndef statements.
- #include statement, -I command line option and 'includePath'
module interface option to specify an include path
- Add __FILE__ and __LINE__ default defines.
- More strict and more helpful error messages:
- Lines of the form "#else <expr>" and "#endif <expr>" no longer
match.
- error messages for illegal #if-block constructs
- error messages for use of defined(BAR) instead of
defined('BAR') in expressions
- New "keep lines" option to output blank lines for skipped content
lines and preprocessor statement lines (to preserve line numbers
in the processed file).
### 0.4.0:
- Add #elif preprocessor statement.
- Add defined() built-in, e.g. #if defined('FOO')
### 0.3.2:
- Make #if expressions Python code.
- Change "defines" attribute of preprocess.preprocess().
- Add -f|--force option to overwrite given output file.
### 0.2.0:
- Add content types for C/C++.
- Better module documentation.
- You can define *false* vars on the command line now.
- 'python setup.py install' works.
### 0.1.0:
- First release.