The Java code in this repository implements a set of tools for working with MIL programs, including code for generating MIL from LC or MIL source files, optimizing MIL code, and translating MIL to LLVM.
These tools are primarily intended for use as a backend for the alb compiler for Habit. However, they can also be used independently, as suggested by the following quick demo. (Note that this command sequence must run in the top level directory; we need to do a better job with search paths for source files in future releases!)
A draft paper with more details about MIL is available from https://web.cecs.pdx.edu/~mpj/pubs/mil.pdf.
The following items are required to use the code in this repository:
- Java Development Kit (version >= 1.8 should be sufficient)
- Apache Ant (version >= 1.9.2 should be sufficient)
First, we need to build the mil-tools, and the simplest way to do this (assuming you already have a suitable JDK and copy of Apache Ant installed) uses the following command sequence:
# Ensure a clean starting point:
ant clean
# Build mil-tools:
ant
[Use of ant clean is recommended after every update to ensure
a consistent build.]
Next, copy the milc shell script (or the milc.bat file on
Windows) into a suitable folder on your path, and then edit that
file so that it includes a full path to the location of the
mil-tools.jar file on your system. After these steps, you should
be able to run milc (without arguments) in any directory to
produce output that starts as follows (run the command yourself
to see the full list of milc options):
$ milc
usage: milc [inputs|options]
inputs: filename.milc Load options from specified file
filename.mil Load MIL source from specified file (.lmil for literate)
filename.lc Load LC source from specified file (.llc for literate)
options: -v verbose on
-d display debug messages
-ipathlist append items to input search path
...
$
The process for compiling an LC source file (in this case,
demo/fib.lc) to MIL and then generating LLVM is illustrated by
the following commands (this time assuming that you have a
suitable installation of LLVM):
# Use mil-tools to compile fib.lc into LLVM:
milc -ilib --standalone demo/fib.lc -ltmp/fib.ll
# Build an executable from the generated LLVM code:
clang -o tmp/fib tmp/fib.ll demo/runtime.c
A brief explanation of the milc command line options used here:
-ilibsets the path for input files to include thelibdirectory, which is where the defaulprelude.lcand other library files are stored.--standaloneconfiguresmilcto generate a program with an entrypoint calledmain.demo/fib.lcspecifies the name of an LC source program.-ltmp/fib.llindicates thatmilcshould generate an LLVM version of the program and write a copy of the code in the filetmp/fib.ll.
And a similarly brief explanation of the clang command line options:
-o tmp/fibsets the name (fib) and folder location (tmp) of the executable program thatclangproduces.tmp/fib.llspecifies the name of the input LLVM code; of course, this is the name of the file that was generated by the preceedingmilccommand.demo/runtime.cis the name of a small C source file that provides an implementation of theprintWordfunction that is used in the originalfib.lcprogram. Some additional examples that combine LC and C code in a single program are described later in this document.
Depending on details of your LLVM installation/platform (for
example, on macOS), you may need to substitute gcc for clang
in the second command here. You may also need to substitute java -jar mil-tools.jar in place of milc, depending on your platform (Windows
users can substitute milc.bat).
Other methods for compiling the generated LLVM code may be useful
in other settings. For example, you can use the following two
commands to compile fib.ll via assembly with a specific target
architecture (in this case, 32-bit x86):
# Compile the generated LLVM code:
llc -O2 -filetype=asm -march=x86 tmp/fib.ll
# Build an executable that includes the generated code:
gcc -m32 -o tmp/fib -Wl,-no_pie demo/runtime.c tmp/fib.s
You may see a message of the form warning: overriding the module target triple with ... when you run the commands above; we will
need to improve the LLVM code generator to avoid this in a future
release, but you should be able to ignore this message for now.
Finally, you can run the compiled program:
# Run the generated executable:
tmp/fib
The result should look something like the following, with a pair of arbitrarily chosen numbers bracketing two 144s, which have been calculated using two slightly different definitions of the Fibonnacci function:
91
144
144
17
Alternatively, you can use the following command line to run the program directly through the bytecode interpreter:
milc -ilib --standalone -x demo/fib.lc
Replace -x with -m to see the generated MIL code, or with -l to
see the generated LLVM code, etc. (Or just run either milc,
milc.bat or java -jar mil-tools.jar for a summary of additional
command line options.)
mil-tools uses a sequence of compilation "passes" to process
and transform input programs. The specific sequence of passes
that is used for a given task depends on what kind of output
is required: if a user requests LLVM output using the -l
flag, for example, then mil-tools uses extra passes that would
not be included if the user had just requested MIL output using
the -m command line option.
For debugging purposes, however, and to help advanced users to
understand how the compilation process works, it is possible to
override the default and specify a custom sequence of parses
using the -p command line option. We will use this in the
following sequence of examples to illustrate some of the key
passes that are provided by mil-tools. The sample program that
we use in these demonstrations is in a source file called demo/ex.lc
whose contents are as follows:
require "prelude.lc"
data Dup a = Dup a a
entrypoint swap7 :: Dup (Bit 7) -> Dup (Bit 7)
entrypoint swap8 :: Dup (Bit 8) -> Dup (Bit 8)
swap7 = swap
swap8 = swap
swap d = case d of Dup x y -> Dup y x
entrypoint d1
d1 = Dup B001 B110
If you just want to have mil-tools load, check, and compile this
program to MIL, you can use the command line option -p, as in milc -m ex.lc -p. This, however, will produce quite a lot of output, and
we can simplify the code a little by adding the character o to the
end of the -p command line option, producing the following output
(more generally, the -p option takes a sequence of letters, each of
which specifies a particular pass; mil-tools will then attempt to run
exactly that sequence of passes on the MIL code that is loaded in
directly or generated by compiling LC source files):
$ milc -ilib -m ex.lc -po
data Dup (a::*)
= Dup a a
data -> (a::*) (b::*)
= Func ([a] ->> [b])
-----------------------------------------
-- not recursive
export d1 :: Dup (Bit 3)
d1 <-
Dup(B001, B110)
-----------------------------------------
-- not recursive
b73 :: forall (a :: *). [Dup a] >>= [Dup a]
b73[t381] =
assert t381 Dup
t382 <- Dup 0 t381
t383 <- Dup 1 t381
Dup(t383, t382)
-----------------------------------------
-- not recursive
b0 :: forall (a :: tuple). [] >>= a
b0[] =
halt(())
-----------------------------------------
-- not recursive
b71 :: forall (a :: *). [Dup a] >>= [Dup a]
b71[t384] =
case t384 of
Dup -> b73[t384]
_ -> b0[]
-----------------------------------------
-- not recursive
k46 :: forall (a :: *). {} [Dup a] ->> [Dup a]
k46{} t385 = b71[t385]
-----------------------------------------
-- not recursive
s2 :: forall (a :: *). [Dup a] ->> [Dup a]
s2 <-
k46{}
-----------------------------------------
-- not recursive
swap :: forall (arg :: *). Dup arg -> Dup arg
swap <-
Func(s2)
-----------------------------------------
-- not recursive
export swap8 :: Dup (Bit 8) -> Dup (Bit 8)
swap8 <-
return swap
-----------------------------------------
-- not recursive
export swap7 :: Dup (Bit 7) -> Dup (Bit 7)
swap7 <-
return swap
-----------------------------------------
-- Entrypoints: d1 swap8 swap7
$
One of the details to note here is the fact that function values
like swap are wrapped up in uses of the constructor function Func,
which is introduced as the result of a (builtin) data type definition
data d -> r = Func ([d] ->> [r]). You might also notice that the
case construct in block b71 includes a default branch (introduced
with an underscore instead of a constructor name), even though the
Dup type that it is matching on has only one constructor. We can
eliminate these overheads by including the c pass ("constructor
function rewrite") as part of the -p setting:
$ milc -ilib -m ex.lc -pco
data Dup (a::*)
= Dup a a
-----------------------------------------
-- not recursive
export d1 :: Dup (Bit 3)
d1 <-
Dup(B001, B110)
-----------------------------------------
-- not recursive
b70 :: forall (a :: *). [Dup a] >>= [Dup a]
b70[t398] =
t399 <- Dup 0 t398
t400 <- Dup 1 t398
Dup(t400, t399)
-----------------------------------------
-- not recursive
k46 :: forall (a :: *). {} [Dup a] ->> [Dup a]
k46{} t401 = b70[t401]
-----------------------------------------
-- not recursive
swap :: forall (a :: *). [Dup a] ->> [Dup a]
swap <-
k46{}
-----------------------------------------
-- not recursive
export swap8 :: [Dup (Bit 8)] ->> [Dup (Bit 8)]
swap8 <-
return swap
-----------------------------------------
-- not recursive
export swap7 :: [Dup (Bit 7)] ->> [Dup (Bit 7)]
swap7 <-
return swap
-----------------------------------------
-- Entrypoints: d1 swap8 swap7
$
After this transformation, all references to the LC level function
type constructor -> have been replaced by uses of the MIL function
type arrow ->>. More generally, if a source program contains a
definition of a non recursive data type data T a = MkT t, then
the c pass will eliminate all uses of the T a type (and all uses
of the MkT constructor) by replacing them with a corresponding
instance of t.
As a next step, we can eliminate polymorphism and parameterized
data types from a MIL program by running the specializer, represented
by the pass letter s: (We follow that here with a second optimizer
run, just to clean up the resulting code, but that is not technically
required. Alternatively, we could achieve a similar effect to what
is shown here by using -pcso, which only uses one optimizer pass.)
$ milc -ilib -m ex.lc -pcoso
data Dup0
= Dup0 (Bit 7) (Bit 7)
data Dup1
= Dup1 (Bit 8) (Bit 8)
data Dup2
= Dup2 (Bit 3) (Bit 3)
-----------------------------------------
-- not recursive
b70 :: [Dup0] >>= [Dup0]
b70[t410] =
t411 <- Dup0 0 t410
t412 <- Dup0 1 t410
Dup0(t412, t411)
-----------------------------------------
-- not recursive
k46 :: {} [Dup0] ->> [Dup0]
k46{} t413 = b70[t413]
-----------------------------------------
-- not recursive
swap :: [Dup0] ->> [Dup0]
swap <-
k46{}
-----------------------------------------
-- not recursive
export swap7 :: [Dup0] ->> [Dup0]
swap7 <-
return swap
-----------------------------------------
-- not recursive
b701 :: [Dup1] >>= [Dup1]
b701[t414] =
t415 <- Dup1 0 t414
t416 <- Dup1 1 t414
Dup1(t416, t415)
-----------------------------------------
-- not recursive
k461 :: {} [Dup1] ->> [Dup1]
k461{} t417 = b701[t417]
-----------------------------------------
-- not recursive
s1 :: [Dup1] ->> [Dup1]
s1 <-
k461{}
-----------------------------------------
-- not recursive
export swap8 :: [Dup1] ->> [Dup1]
swap8 <-
return s1
-----------------------------------------
-- not recursive
export d1 :: Dup2
d1 <-
Dup2(B001, B110)
-----------------------------------------
-- Entrypoints: swap7 swap8 d1
$
The generated program is longer now because there are essentially
two different copies of the swap function, each operating on a
different version of the Dup type. As intended, however, the
program no longer contains any definitions with polymorphic types.
In order to generate working code for this program, we need to
provide representations for types like Bit 7 and Bit 8 that
show up in this program. In mil-tools, both of these types are
actually represented as values of type Word, treating only the
lower 7 or 8 bits as being significant, in each case. The r
character can be used to specify a "representation transformation"
pass that rewrites input programs to reveal these low level
details:
$ milc -ilib -m ex.lc -pcosoro
data Dup0
= Dup0 Word Word
data Dup1
= Dup1 Word Word
data Dup2
= Dup2 Word Word
-----------------------------------------
-- not recursive
export swap7 :: [Dup0] >>= [Dup0]
swap7[t444] =
t445 <- Dup0 0 t444
t446 <- Dup0 1 t444
Dup0(t446, t445)
-----------------------------------------
-- not recursive
export swap8 :: [Dup1] >>= [Dup1]
swap8[t447] =
t448 <- Dup1 0 t447
t449 <- Dup1 1 t447
Dup1(t449, t448)
-----------------------------------------
-- not recursive
export d1 :: Dup2
d1 <-
Dup2(1, 6)
-----------------------------------------
-- Entrypoints: swap7 swap8 d1
$
For types like Dup0 and Dup1, however, it is possible to
provide more compact representations by replacing each of the
associated data type definitions with corresponding bitdata
definitions. This can be accomplished by using the letter
b to specify a bitdata rewrite pass. Note that b can
only be used immediately after s because it relies on
information that is generated during specialization:
$ milc -ilib -m ex.lc -pcosbo
bitdata Dup2 /6
= Dup2 [ Dup20 :: Bit 3 | Dup21 :: Bit 3 ]
-- predDup2(x :: Bit 6) = true
bitdata Dup1 /16
= Dup1 [ Dup10 :: Bit 8 | Dup11 :: Bit 8 ]
-- predDup1(x :: Bit 16) = true
bitdata Dup0 /14
= Dup0 [ Dup00 :: Bit 7 | Dup01 :: Bit 7 ]
-- predDup0(x :: Bit 14) = true
-----------------------------------------
-- not recursive
b70 :: [Dup0] >>= [Dup0]
b70[t524] =
t525 <- Dup0 0 t524
t526 <- Dup0.Dup0 0 t525
t527 <- Dup0.Dup0 1 t525
t528 <- Dup0.Dup0(t527, t526)
Dup0(t528)
-----------------------------------------
-- not recursive
k46 :: {} [Dup0] ->> [Dup0]
k46{} t529 = b70[t529]
-----------------------------------------
-- not recursive
swap :: [Dup0] ->> [Dup0]
swap <-
k46{}
-----------------------------------------
-- not recursive
export swap7 :: [Dup0] ->> [Dup0]
swap7 <-
return swap
-----------------------------------------
-- not recursive
b701 :: [Dup1] >>= [Dup1]
b701[t530] =
t531 <- Dup1 0 t530
t532 <- Dup1.Dup1 0 t531
t533 <- Dup1.Dup1 1 t531
t534 <- Dup1.Dup1(t533, t532)
Dup1(t534)
-----------------------------------------
-- not recursive
k461 :: {} [Dup1] ->> [Dup1]
k461{} t535 = b701[t535]
-----------------------------------------
-- not recursive
s1 :: [Dup1] ->> [Dup1]
s1 <-
k461{}
-----------------------------------------
-- not recursive
export swap8 :: [Dup1] ->> [Dup1]
swap8 <-
return s1
-----------------------------------------
-- not recursive
s2 :: Dup2.Dup2
s2 <-
Dup2.Dup2(B001, B110)
-----------------------------------------
-- not recursive
export d1 :: Dup2
d1 <-
Dup2(s2)
-----------------------------------------
-- Entrypoints: swap7 swap8 d1
$
If you're wondering why this version of the program looks
a little bit longer, that's because bitdata types use two
levels of constructor functions: for every constructor
function C in a bitdata type T, there is also an associated
layout type called T.C, each of which also has a constructor
function with the same name.
At this point, we can add r back to our list of passes to
run the representation transformation. At this point,
mil-tools determines that complete Dup0 and Dup1 values
can be represented within a single word (only 16 and 14 bits,
respectively, are needed in each case0) and replaces the
algebraic data type selectors and constructos with some
appropriate bit twiddling logic, as illustrated in the
final output shown below:
$ milc -ilib -m ex.lc -pcosboro
-----------------------------------------
-- not recursive
export swap7 :: [Word] >>= [Word]
swap7[t614] =
t615 <- lshr((t614, 7))
t616 <- shl((t614, 7))
t617 <- and((t616, 16256))
or((t615, t617))
-----------------------------------------
-- not recursive
export swap8 :: [Word] >>= [Word]
swap8[t618] =
t619 <- lshr((t618, 8))
t620 <- shl((t618, 8))
t621 <- and((t620, 65280))
or((t619, t621))
-----------------------------------------
-- not recursive
export d1 :: Word
d1 <-
return 14
-----------------------------------------
-- Entrypoints: swap7 swap8 d1
$
Note also that, as part of the r representation transformation,
the definitions of swap8 and swap7 have been changed from
definitions of function closures (using the ->> arrow) to simple
block definitions, which are more easily accessed as standard
functions from other LLVM languages, as can be seen if we switch
from MIL to LLVM output in the previous example: (It is possible
to generate both with a single mil-tools invocation, but we only
generate one at a time here so that the output is a little easier
to follow.)
$ milc -l ex.lc -pcosboro
@d1 = global i32 14
define i32 @swap7(i32 %r0) {
entry:
br label %swap7
swap7:
%r2 = lshr i32 %r0, 7
%r4 = shl i32 %r0, 7
%r3 = and i32 %r4, 16256
%r1 = or i32 %r2, %r3
ret i32 %r1
}
define i32 @swap8(i32 %r0) {
entry:
br label %swap8
swap8:
%r2 = lshr i32 %r0, 8
%r4 = shl i32 %r0, 8
%r3 = and i32 %r4, 65280
%r1 = or i32 %r2, %r3
ret i32 %r1
}
$
The tools described here can be used in various ways to support the development of software libraries, standalone programs, or variations in between. The following examples illustrate some of the possibilities, and the corresponding use of LC/MIL features and milc command line options.
Our first example is a small library of functions that are not
complete programs in themselves, but could potentially be linked
in as useful components of a variety of different applications.
This particular library, whose source code is in demo/funlib.lc
provides two implementations of the Fibonnaci function, and two
implementations of the factorial function, so it will probably
not be useful in many real world settings, but should provide
some familiar examples and should also be sufficient to demonstrate
the main ideas:
$ cat demo/funlib.lc
require "prelude.lc"
-- A standard recursive version of the Fibonnaci function:
entrypoint fib :: Word -> Word
fib n = if eq n 0 then 0
else if eq n 1 then 1
else add (fib (sub n 1)) (fib (sub n 2))
-- An iterative version of the Fibonnaci function:
entrypoint itfib :: Word -> Word
itfib = let loop a b n = if eq n 0 then a else loop b (add a b) (sub n 1)
in loop 0 1
-- A traditional recursive implementation of the factorial function:
entrypoint recfac :: Word -> Word
recfac n = if eq n 0 then 1 else mul n (recfac (sub n 1))
-- An iterative version of the factorial function:
entrypoint itfac :: Word -> Word
itfac = let loop a n = if eq n 0 then a else loop (mul a n) (sub n 1)
in loop 1
$
Perhaps the most important detail here is that each of the four
functions is declared as an entrypoint. (The type signatures that
are provided for each function here could be useful as documentation,
but they can also be inferred automatically, so are not actually
required.) As a result, when this program is compiled using the
following commands, it will produce an LLVM output (in the file
tmp/funlib.ll) that contains an externally visible function
definition for each of the four LC functions.
$ ./milc -ilib demo/funlib.lc -ltmp/funlib.ll
$ clang -c -o tmp/funlib.o tmp/funlib.ll
warning: overriding the module target triple with ...
1 warning generated.
$
[As before, assuming you have a conventional LLVM installation, you should be able ignore the warning message shown here.]
The command sequence shown above follows the use of milc with a
clang command that builds an object file tmp/funlib.o that can
be linked with other programs. Alternatively, we can write a C
program that uses the functions in funlib.lc and then use a
single clang command to compile and link it with the
tmp/funlib.ll source file:
$ cat demo/funlibtest.c
#include <stdio.h>
extern int fib(int);
extern int itfib(int);
extern int recfac(int);
extern int itfac(int);
int main(int argc, char** argv) {
for (int i=0; i<10; i++) {
printf("%d\t%d\t%d\t%d\t%d\n",
i, fib(i), itfib(i), recfac(i), itfac(i));
}
}
$ clang -o tmp/funlibtest demo/funlibtest.c tmp/funlib.ll
$
[Note: On macOS, you may be able (or need) to substitute gcc for
clang in the command lines shown here, depending on details of
your LLVM installation.]
Notice that the C code references the functions we have seen
defined in funlib.lc as regular C functions with conventional
function prototypes. This is possible because, if an entrypoint
in the LC code is a curried function, then it will be exported
from the generated LLVM code as a fully uncurried function (i.e.,
as a function that takes all of its arguments at the same time, as
is the norm in languages like C that do not directly support
higher order functions). The result is an executable that blends
code from the two different source code components into a single
program:
$ tmp/funlibtest
0 0 0 1 1
1 1 1 1 1
2 1 1 2 2
3 2 2 6 6
4 3 3 24 24
5 5 5 120 120
6 8 8 720 720
7 13 13 5040 5040
8 21 21 40320 40320
9 34 34 362880 362880
$
Although there is no need for this in our funlib example,
it is sometimes necessary to perform some kind of initialization
steps before some (or all) of the entrypoints in a library are
used for the first time. The following shows a contrived
example that includes entrypoints called fib12 and
fib15 that are supposed to contain the 12th and 15th
Fibonnaci numbers, respectively:
$ cat demo/needinit.lc
require "prelude.lc"
entrypoint fib :: Word -> Word
fib n = if eq n 0 then 0
else if eq n 1 then 1
else add (fib (sub n 1)) (fib (sub n 2))
entrypoint fib12, fib15
fib12 = fib 12
fib15 = fib 15
$
But now suppose that we link this with a C program that reads the
values of fib12 and fib15 and displays them on the screen. At
what point will those values be calculated? If we compile this
program using a similar milc command line to the previous
example, we get an error informing us that the generated code
requires an initialization function (in this case, we can infer
that it will be used to set the values for fib12 and fib15):
$ ./milc -ilib demo/needinit.lc -ltmp/needinit.ll
ERROR: LLVM program requires initialization function (set using --llvm-main=NAME)
$
The error message also suggests that we can fix this problem by
using an --llvm-main command line option, which is precisely
what we do in the following, choosing to use the name initialize
for this function:
$ ./milc -ilib demo/needinit.lc -ltmp/needinit.ll --llvm-main=initialize
$
Now we can write C programs that use the needinit library. And,
so long as we ensure that the initialize() function is called
before the values of fib12 and fib15 are accessed, then the
everything should work as expected:
$ cat demo/printfibs.c
#include <stdio.h>
extern void initialize();
extern int fib12, fib15;
int main(int argc, char** argv) {
initialize();
printf("fib(12)=%d, fib(15)=%d\n", fib12, fib15);
}
$ clang -o tmp/printfibs demo/printfibs.c tmp/needinit.ll
$ tmp/printfibs
fib(12)=144, fib(15)=610
$
In general, it can sometimes be difficult to determine whether or
not an initialization function is actually required, short of
running a milc command line like the one above to see if an
error is reported. For example, if we replace the fib function
in needinit.lc with the more efficient itfib function, then
the mil-tools optimizer in actually able to compute the values
of fib12 and fib15 at compile-time, and there will not be any
need for an initialization function. For this reason, it is often
a good practice to specify an initialization function name using
the --llvm-main option and to encourage users of the library to
call that function before attempting to accessing any of its other
features. If the library does not actually require special treatment,
then milc will just generate an initialization function with an
empty body. But if something subsequently changes in the program
that does require proper initialization, then users of the library,
at least if they are following your advice, will already have
included the appropriate function call in their code.
In the previous examples, we wrote code in LC that was accessed
and executed from a main function written in C. But what if
we wanted to write our main program directly in LC, as in the
following simple example for displaying the value of fib 12:
$ cat demo/program.lc
require "prelude.lc"
require "io.mil"
itfib :: Word -> Word
itfib = let loop a b n = if eq n 0 then a else loop b (add a b) (sub n 1)
in loop 0 1
-- A program to print the 12th Fibonnaci number:
export main
main = printWord (itfib 12)
$
This program does not specify any explicit entrypoints, but it
does contain a definition for a main routine that is marked as
an export. This allows us to compile the program using the
--standalone flag for milc:
$ ./milc -ilib --standalone demo/program.lc -ltmp/program.ll
$ clang -o tmp/program tmp/program.ll demo/runtime.c
warning: overriding the module target triple with ...
1 warning generated.
$ tmp/program
144
$
Note that, despite what the name --standalone might suggest,
the clang command line shown here does actually mention
the C source file demo/runtime.c, which provides an
implementation for the printWord function in the original
LC code. So, in fact, we are still mixing LC and C code,
but this time we are calling the C code from the LC code,
which is the opposite to what we saw in the earlier examples.
The --standalone flag is really just a shorthand for the
following two command line options:
--llvm-main=main --mil-main=main
As we have already seen, the first of these specifies that the
generated LLVM code should include an initialization function
called main. The second identifies a specific definition, also
with the name main, but this time in the input program, that
will be executed at the end of the initialization function. At
most one main definition can be specified in this way, but it is
possible to use a value with a different name, so long as it is
marked as an export in the source program and identified using
an explicit --mil-main=NAME setting on the command line.)
While --standalone provides sensible defaults for both of
these command line options, the separate --mil-main and
--llvm-main options provide useful additional control
in some more advanced settings.