L2

L2 is an attempt to find the smallest most distilled programming language equivalent to C. The goal is to turn as much of C's preprocessor directives, control structures, statements, literals, and functions requiring compiler assistance (setjmp, longjmp, alloca, assume, ...) into things definable inside L2 (with perhaps a little assembly). The language does not surject to all of C, its most glaring omission being that of a type-system. However, I reckon the result is still pretty interesting.

The approach taken to achieve this has been to make C's features more composable, more multipurpose, and, at least on one occasion, add a new feature so that a whole group of distinct features could be dropped. In particular, the most striking changes are that C's:

irregular syntax is replaced by S-expressions; because simple syntax composes well with a non-trivial preprocessor (and no, I have not merely transplanted Common Lisp's macros into C)
loop constructs are replaced with what I could only describe as a more structured variant of setjmp and longjmp without stack destruction (and no, there is no performance overhead associated with this)

There are 9 language primitives and for each one of them I describe their syntax, what exactly they do in English, the i386 assembly they translate into, and an example usage of them. Following this comes a listing of L2's syntactic sugar. Following this comes a brief description of L2's internal representation and the 7 functions (loosely speaking) that manipulate it. After that comes a description of how a non-primitive L2 expression is compiled. The above descriptions take about 8 pages and are essentially a complete description of L2.

Afterwards, there is a list of reductions that shows how some of C's constructs can be defined in terms of L2. Here, I have also demonstrated closures to hint at how more exotic things like coroutines and generators are possible using L2's continuations. And, finally, this README ends with a description of my L2 system's compilation library, a binary interface for compiling L2 code.

Getting Started	Primitive Expressions	Examples/Reductions
Building L2	Begin	Commenting
The Evaluator	Literal	Dereferencing
Syntactic Sugar	Reference	Numbers
Internal Representation	Storage	Backquoting
Expression	If	Variable Binding
Compilation Library	Function	Switch Expression
	Invoke	Characters
	With	Strings
	Continuation	Conditional Compilation
	Jump	Closures
		Assume

Getting Started

Building L2

./buildl2

This implementation of L2 needs a Linux distribution running on the i386 or x86-64 architecture with the GNU C compiler and musl libc installed to run successfully. To build the system, simply run the buildl2 script at the root of the repository. The build should be fast - there are only about 2200 lines of C code to compile. This will create a directory called bin containing the files l2evaluate, l2compile.so, sexpr.so, and i386.so or x86_64.so. l2evaluate is an evaluator of L2 code: it reads in L2 code, compiles it, then executes it. l2compile.so is the library that contains the compiler (the evaluator uses it to compile L2 code). i386.so or x86_64.so, depdending on which is compiled, is a library of instruction wrappers to provide i386/x86-64 functionality (ADD, SUB, MOV, ...) not exposed by the L2 language. What follows is written assuming that you are on a x86-64 machine.

The Evaluator

./bin/l2evaluate libraries.so ... + inputs.l2 ...

In this initial environment the L2 evaluator begins by loading the shared libraries libraries.so ... into memory. Then the evaluator iterates through each L2 source file after the plus sign. Each of the files read should be of the form expression1 expression2 ... expressionN. For each file, it compiles each expression and emits the corresponding machine code in order. This object code is then linked against the heretofore loaded libraries to produce a shared library. This shared library is then loaded into memory.

If a minus sign is supplied as one of the source files, then a minus-prompt is displayed. You are expected to enter valid L2 source code. Each time you press Enter, a plus-prompt is printed to elicit you to add to the source code so far read. Pressing Ctrl-D on an empty line tells the evaluator to treat all the code entered since the last minus prompt as if it were an L2 source file. If Ctrl-C then Enter is typed in a plus-prompt, heretofore uncompiled code is discarded, and the evaluator returns to minus-prompt state.

Example

file1.l2

(function foo (sexprs)
	(with return (begin
		[putchar [+ (literal 0...01100001) (literal 0...01)]]
		{return [lst [lst [-b-] [lst [-e-] [lst [-g-] [lst [-i-] [lst [-n-] [nil]]]]]] [nil]]})))

[putchar (literal 0...01100001)]

file2.l2

(function bar ()
	[putchar (literal 0...01100011)])
(foo this text does not matter)
[putchar (literal 0...01100100)]

Running ./bin/l2evaluate "./bin/x86_64.so" "./bin/sexpr.so" "/lib/x86_64-linux-musl/libc.so" + file1.l2 file2.l2 should cause the text "abd" to be printed to standard output. The "a" comes from the last expression of file1.l2. It was printed after the compilation of file1.l2, when it was being loaded into the compiler. Why? Because L2 libraries are executed from top to bottom when they are loaded. The "b" comes from within the function in file1.l2. It was executed when the expression (foo this text does not matter) in file2.l2 was being compiled. Why? Because the foo causes the compiler to invoke a function called foo in the environment. The s-expression (this text does not matter) is the argument to the function foo, but the function foo ignores it and returns the s-expression (begin). Hence (begin) replaces (foo this text does not matter) in file2.l2. Now file2.l2 is entirely made up of primitive expressions which are compiled in the way specified below. Finally the text "d" is printed. Why? Because file2.l2's last expression is the only one that is side-effectual, does not get replaced during compilation, and (therefore) gets loaded into memory.

Running ./bin/l2evaluate "./bin/x86_64.so" "./bin/sexpr.so" "/lib/x86_64-linux-musl/libc.so" + file1.l2 - should cause the evaluator to print "a" to standard output for the same reason as above. Now you should be in a minus-prompt. Pasting in the contents of file2.l2, then pressing Enter followed by Ctrl-D should now cause the evaluator to print the text "bd" for the same reasons as above.

Primitive Expressions

Begin

(begin expression1 expression2 ... expressionN)

Evaluates its subexpressions sequentially from left to right. That is, it evaluates expression1, then expression2, and so on, ending with the execution of expressionN. Specifying zero subexpressions is valid. The return value is unspecified.

This expression is implemented by emitting the instructions for expression1, then emitting the instructions for expression2 immediately afterwords and so on, ending with the emission of expressionN.

Say the expression [foo] prints the text "foo" to standard output and the expression [bar] prints the text "bar" to standard output. Then (begin [foo] [bar] [foo] [foo] [foo]) prints the text "foobarfoofoofoo" to standard output.

Literal

(literal b63b62...b0)

The resulting value is the 64 bit number specified in binary inside the brackets. Specifying less than or more than 64 bits is an error. Useful for implementing character and string literals, and numbers in other bases.

This expression is implemented by emitting an instruction to mov an immediate value into a memory location designated by the surrounding expression.

Say the expression [putchar x] prints the character x. Then [putchar (literal 0...01100001)] prints the text "a" to standard output.

Reference

reference0

The resulting value is the address in memory to which this reference refers.

This expression is implemented by the emission of an instruction to lea of some data into a memory location designated by the surrounding expression.

Say the expression [get x] evaluates to the value at the reference x and the expression [set x y] puts the value y into the reference x. Then (begin [set x (literal 0...01100001)] [putchar [get x]]) prints the text "a" to standard output.

Storage

(storage storage0 expression1 expression2 ... expressionN)

If this expression occurs inside a function, then space enough for N contiguous values has already been reserved in its stack frame. If it is occuring outside a function, then static memory instead has been reserved. storage0 is a reference to the beginning of this space. This expression evaluates each of its sub-expressions in an environment containing storage0 and stores the resulting values in contiguous locations of memory beginning at storage0 in the same order as they were specified. The resulting value of this expression is storage0.

N contiguous words must be reserved in the current function's stack-frame plan. The expression is implemented by first emitting the instructions for any of the subexpressions with the location of the resulting value fixed to the corresponding reserved word. The same is done with the remaining expressions repeatedly until the instructions for all the subexpressions have been emitted. And then second emitting an instruction to lea of the beginning of the contiguous words into a memory location designated by the surrounding expression.

The expression [putchar [get (storage _ (literal 0...01100001))]], for example, prints the text "a" to standard output.

If

(if expression0 expression1 expression2)

If expression0 is non-zero, then only expression1 is evaluated and its resulting value becomes that of the whole expression. If expression0 is zero, then only expression2 is evaluated and its resulting value becomes that of the whole expression.

This expression is implemented by first emitting an instruction to or expression0 with itself. Then an instruction to je to expression2's label is emitted. Then the instructions for expression1 are emitted with the location of the resulting value fixed to the same memory address designated for the resulting value of the if expression. Then an instruction is emitted to jmp to the end of all the instructions that are emitted for this if expression. Then the label for expression2 is emitted. Then the instructions for expression2 are emitted with the location of the resulting value fixed to the same memory address designated for the resulting value of the if expression.

The expression [putchar (if (literal 0...0) (literal 0...01100001) (literal 0...01100010))] prints the text "b" to standard output.

Function

(function function0 (reference1 reference2 ... referenceN) expression0)

Makes a function to be invoked with exactly N arguments. When the function is invoked, expression0 is evaluated in an environment where function0 is a reference to the function itself and reference1, reference2, up to referenceN are references to the resulting values of evaluating the corresponding arguments in the invoke expression invoking this function. Once the evaluation is complete, control flow returns to the invoke expression and the invoke expression's resulting value is the resulting value of evaluating expression0. The resulting value of this function expression is a reference to the function.

This expression is implemented by first emitting an instruction to mov the address function0 (a label to be emitted later) into the memory location designated by the surrounding expression. Then an instruction is emitted to jmp to the end of all the instructions that are emitted for this function. Then the label named function0 is emitted. Then instructions to push each callee-saved register onto the stack are emitted. Then an instruction to push the frame-pointer onto the stack is emitted. Then an instruction to move the value of the stack-pointer into the frame-pointer is emitted. Then an instruction to sub from the stack-pointer the amount of words reserved on this function's stack-frame is emitted. After this the instructions for expression0 are emitted with the location of the resulting value fixed to a word within the stack-pointer's drop. After this an instruction is emitted to mov the word from this location into the register eax. And finally, instructions are emitted to leave the current function's stack-frame, pop the callee-save registers, and ret to the address of the caller.

The expression [putchar [(function my- (a b) [- [get b] [get a]]) (literal 0...01) (literal 0...01100011)]] prints the text "b" to standard output.

Invoke

(invoke function0 expression1 expression2 ... expressionN)
[function0 expression1 expression2 ... expressionN]

Both the above expressions are equivalent. Evaluates function0, expression1, expression2, up to expressionN in an unspecified order and then invokes function0, a reference to a function, providing it with the resulting values of evaluating expression1 up to expressionN, in order. The resulting value of this expression is determined by the function being invoked.

N+1 words must be reserved in the current function's stack-frame plan. The expression is implemented by emitting the instructions for any of the subexpressions with the location of the resulting value fixed to the corresponding reserved word. The same is done with the remaining expressions repeatedly until the instructions for all the subexpressions have been emitted. Then an instruction to push the last reserved word onto the stack is emitted, followed by the second last, and so on, ending with an instruction to push the first reserved word onto the stack. A call instruction with the zeroth reserved word as the operand is then emitted. Note that L2 expects registers esp, ebp, ebx, esi, and edi to be preserved across calls. An add instruction that pops N words off the stack is then emitted. Then an instruction is emitted to mov the register eax into a memory location designated by the surrounding expression.

A function with the reference - that returns the value of subtracting its second parameter from its first could be defined as follows:

-:
movl 4(%esp), %eax
subl 8(%esp), %eax
ret

The following invocation of it, (invoke putchar (invoke - (literal 0...01100011) (literal 0...01))), prints the text "b" to standard output.

With

(with continuation0 expression0)

Makes a continuation to the containing expression that is to be jumped to with exactly one argument. Then expression0 is evaluated in an environment where continuation0 is a reference to the aforementioned continuation. The resulting value of this expression is unspecified if the evaluation of expression0 completes. If the continuation continuation0 is jumped to, then this with expression evaluates to the resulting value of the single argument within the responsible jump expression.

5+1 words must be reserved in the current function's stack-frame plan. Call the reference to the first word of the reservation continuation0. This expression is implemented by first emitting instructions to store the program's state at continuation0, that is, instructions are emitted to mov ebp, the address of the instruction that should be executed after continuing (a label to be emitted later), edi, esi, and ebx, in that order, to the first 5 words at continuation0. After this, the instructions for expression0 are emitted. Then the label for the first instruction of the continuation is emitted. And finally, an instruction is emitted to mov the resulting value of the continuation, the 6th word at continuation0, into the memory location designated by the surrounding expression.

Examples

Note that the expression {continuation0 expression0} jumps to the continuation reference given by continuation0 with resulting value of evaluating expression0 as its argument. With the note in mind, the expression (begin [putchar (with ignore (begin {ignore (literal 0...01001110)} [foo] [foo] [foo]))] [bar]) prints the text "nbar" to standard output.

The following assembly function allocate receives the number of bytes it is to allocate as its first argument, allocates that memory, and passes the initial address of this memory as the single argument to the continuation it receives as its second argument.

allocate:
/* All sanctioned by L2 ABI: */
movl 8(%esp), %ecx
movl 16(%ecx), %ebx
movl 12(%ecx), %esi
movl 8(%ecx), %edi
movl 0(%ecx), %ebp
subl 4(%esp), %esp
andl $0xFFFFFFFC, %esp
movl %esp, 20(%ecx)
jmp *4(%ecx)

The following usage of it, (with dest [allocate (literal 0...011) dest]), evaluates to the address of the allocated memory. If allocate had just decreased esp and returned, it would have been invalid because L2 expects functions to preserve esp.

Continuation

(continuation continuation0 (reference1 reference2 ... referenceN) expression0)

Makes a continuation to be jumped to with exactly N arguments. When the continuation is jumped to, expression0 is evaluated in an environment where continuation0 is a reference to the continuation itself and reference1, reference2, up to referenceN are references to the resulting values of evaluating the corresponding arguments in the jump expression jumping to this function. Undefined behavior occurs if the evaluation of expression0 completes - i.e. programmer must direct the control flow out of continuation0 somewhere within expression0. The resulting value of this continuation expression is a reference to the continuation.

5+N words must be reserved in the current function's stack-frame plan. Call the reference to the first word of the reservation continuation0. This expression is implemented by first emitting an instruction to mov the reference continuation0 into the memory location designated by the surrounding expression. Instructions are then emitted to store the program's state at continuation0, that is, instructions are emitted to mov ebp, the address of the instruction that should be executed after continuing (a label to be emitted later), edi, esi, and ebx, in that order, to the first 5 words at continuation0. Then an instruction is emitted to jmp to the end of all the instructions that are emitted for this continuation expression. Then the label for the first instruction of the continuation is emitted. After this the instructions for expression0 are emitted.

The expression {(continuation forever (a b) (begin [putchar [get a]] [putchar [get b]] {forever [- [get a] (literal 0...01)] [- [get b] (literal 0...01)]})) (literal 0...01011010) (literal 0...01111010)} prints the text "ZzYyXxWw"... to standard output.

Jump

(jump continuation0 expression1 expression2 ... expressionN)
{continuation0 expression1 expression2 ... expressionN}

Both the above expressions are equivalent. Evaluates continuation0, expression1, expression2, up to expressionN in an unspecified order and then jumps to continuation0, a reference to a continuation, providing it with a local copies of expression1 up to expressionN in order. The resulting value of this expression is unspecified.

N+1 words must be reserved in the current function's stack-frame plan. The expression is implemented by emitting the instructions for any of the subexpressions with the location of the resulting value fixed to the corresponding reserved word. The same is done with the remaining expressions repeatedly until the instructions for all the subexpressions have been emitted. Then an instruction to mov the first reserved word to 5 words from the beginning of the continuation is emitted, followed by an instruction to mov the second reserved word to an address immediately after that, and so on, ending with an instruction to mov the last reserved word into the last memory address of that area. The program's state, that is, ebp, the address of the instruction that should be executed after continuing, edi, esi, and ebx, in that order, are what is stored at the beginning of a continuation. Instructions to mov these values from the buffer into the appropriate registers and then set the program counter appropriately are, at last, emitted.

The expression (begin (with cutter (jump (continuation cuttee () (begin [bar] [bar] (jump cutter (literal 0...0)) [bar] [bar] [bar])))) [foo]) prints the text "barbarfoo" to standard output.

An Optimization

Looking at the examples above where the continuation reference does not escape, (with reference0 expression0) behaves a lot like the pseudo-assembly expression0 reference0: and (continuation reference0 (...) expression0) behaves a lot like reference0: expression0. To be more precise, when references to a particular continuation only occur as the continuation0 subexpression of a jump statement, we know that the continuation is constrained to the function in which it is declared, and hence there is no need to store or restore ebp, edi, esi, and ebx. Continuations, then, are how efficient iteration is achieved in L2.

Syntactic Sugar

`$a1...aN`

In what follows, it is assumed that $a1...aN is not part of a larger string. If $a1...aN is simply a $, then it remains unchanged. Otherwise at least a character follows the $; in this case $a1...aN turns into ($ a1...aN).

For example, the expression $$hello$bye turns into ($ $hello$bye) which turns into ($ ($ hello$bye))

`&a1...aN`

An analogous transformation to the one for $a1...aN happens.

`,a1...aN`

An analogous transformation to the one for $a1...aN happens.

`a1...aN

An analogous transformation to the one for $a1...aN happens.

Internal Representation

After substituting out the syntactic sugar defined in the invoke, jump, and syntactic sugar sections, we find that all L2 programs are just compositions of the <pre-s-expression>s: <symbol> and (<pre-s-expression> <pre-s-expression> ... <pre-s-expression>). If we now replace every symbol with a list of its characters so that for example foo becomes (f o o), we now find that all L2 programs are now just compositions of the <s-expression>s <character> and (<s-expression> <s-expression> ... <s-expression>). The following functions that manipulate these s-expressions are not part of the L2 language and hence the compiler does not give references to them special treatment during compilation. However, when compiled code is loaded into an L2 compiler, undefined references to these functions are to be dynamically resolved.

`[lst x y]`

x must be a s-expression and y a list.

Makes a list where x is first and y is the rest.

Say the s-expression foo is stored at a and the list (bar) is stored at b. Then [lst [get a] [get b]] is the s-expression (foo bar).

`[lst? x]`

x must be a s-expression.

Evaluates to the complement of zero if x is also list. Otherwise evaluates to zero.

Say the s-expression foo is stored at a. Then [lst? [get a]] evaluates to (literal 1...1).

`[fst x]`

x must be a list.

Evaluates to a s-expression that is the first of x.

Say the list foo is stored at a. Then [fst [get a]] is the s-expression f. This f is not a list but is a character.

`[rst x]`

x must be a list`.

Evaluates to a list that is the rest of x.

Say the list foo is stored at a. Then [rst [get a]] is the s-expression oo.

`[nil]`

Evaluates to the empty list.

Say the s-expression foo is stored at a. Then [lst [get a] [nil]] is the s-expression (foo).

`[-<character>-]`

Evaluates to the character <character>.

The expression [lst [-f-] [lst [-o-] [lst [-o-] [nil]]]] evaluates to the s-expression foo.

`[sexpr= x y]`

x and y must be s-expressions.

Evaluates to the complement of zero if x is the same s-expression as y, otherwise it evaluates to zero.

Say the s-expression (foo (bar bar) foo foo) is stored at both x and y. Then [sexpr= [get x] [get y]] evaluates to (literal 1...1).

`[begin x]`

x must be a list of s-expressions.

Evaluates to an s-expression formed by prepending the s-expression begin to x. The begin function could have the following definition: (function b (sexprs) [lst [lst [-b-] [lst [-e-] [lst [-g-] [lst [-i-] [lst [-n-] [nil]]]]]] [get sexprs]]).