cl-jackc

Introduction

This is a compiler for the Jack language, the high-level language of the Hack platform, an exercise of the book The Elements of Computing Systems, by Nisan and Schocken.

The Jack language is introduced on Chapter 9, and Chapters 10 and 11 are dedicated to building a simple compiler for such language, which is what this project is.

This compiler was completely written in Common Lisp.

This software is beta quality, and is currently on its road to perform full compilations of Jack programs for Chapter 11 compliance.

Qlot support

This project uses qlot for both development and unit testing, so one should be able to use this tool to either run or develop the compiler.

Qlot assumes that you have Roswell installed, preferably running SBCL. To install and configure qlot, simply run:

ros install qlot
cd /path/to/cl-jackc
qlot install

To use qlot along with SLIME on Emacs, see qlot’s repository documentation.

Dependencies

This project depends on a number of Common Lisp systems, which are specified below. All of them can are fetched from Quicklisp’s main repository.

Compilation and usage:

alexandria, 2017-08-30
split-sequence, 2019-08-30

Unit tests:

rove, 2019-08-13
cxml, 2019-08-13

Documentation:

staple, 2019-08-13

Modularization

This project takes a different approach when compared to other projects of the book (cl-hackasm, cl-hackvmtr), with respect to its organization. The book talks about compiler modules, and so the project is organized in modules as well, though not strictly as suggested.

Each one of the project’s packages correspond to a certain module of the compiler. When doing this, instead of having separate binaries, we just make sure that the packages are minimalistic and serve their purpose explicitly. The user shall only interface with a single package, which depends on the other modules and works as a frontend.

Module listing

cl-jackc (interface)
Default interface for compiler. Exports procedures so the user can interact with the compiler.
jackc-utils
Utilities and miscellaneous structures for all other compiler modules, usually related to language extension.
jackc-conditions
Definitions for compiler-related conditions.
jackc-tokenizer
Capable of reading specific tokens and handling a file stream, while also holding the language grammar specification.

Relates to the JackTokenizer module, proposed in the book, except that the generation of the parsing tree was passed to the analyzer.
jackc-analyzer
Takes a single file stream, and matches it with the language grammar, generating a syntax tree. Redirects the output to the parser.

Relates mostly to the JackAnalyzer module, proposed in the book.
jackc-parser
Takes the output of a file analysis, and effectively compiles it to one of the desired outputs (XML, VM or SEXP), outputting it to console.

Relates to the CompilationEngine module, proposed in the book.
jackc-writer
Takes the console output of the parser, and writes it to the desired (.xml, .sexp or .vm) file.
jackc-reader
Takes compilation arguments and configuration, passed by the user. For each given file, generates a file stream and passes it to the analyzer.

Usage

There are three ways to use the compiler:

roswell/jackc.ros, the default compiler script, which can be used from command line with Roswell;
run-jackc.sh, which directly invokes the compiler script for command line, using a Qlot context;
The cl-jackc package, which works as an interface to the compiler.

Please notice that this software is still beta quality and is not ready for usage.

Installing from Roswell

It is possible to install this program directly from Roswell, though it is not recommended while the software is at beta stage.

Simply run:

ros install luksamuk/cl-jackc

After the required operations, the script jackc should be available for use from command line:

jackc /path/to/files/and/dirs [args...]

When using jackc, any separate given path is treated as a single, different project. Each file of a project can be separately compiled. However, if a directory name is passed instead of the path to a Jack code file, jackc will attempt to find all Jack files on the first level of such directory and compile each one of them. Each compiled file will be written in the same directory as the code file.

As for extra arguments, jackc allows --xml and --sexp, which indicate specific syntax analysis steps. When both are informed, --xml takes precedence. More information can be seen on the compiler’s help output.

jackc does not demand any order of arguments, so one can safely interleave arguments and paths when using the script.

Passing no arguments shows a usage and help text.

UPDATE 2019-08-25: At this date, the newest version of alexandria was broken on Quicklisp repositories and could not be properly loaded. This is why the unit testing now uses fixed versions for some systems. I cannot guarantee that these libraries will not break if you install them using plain Roswell; should anything happen, plese refer to the bundled script.

Using the bundled script

Given that Roswell and Qlot are installed, and that Qlot is correctly configured in the project’s directory, the compiler can be invoked from command line using the run-jackc.sh script.

./run-jackc.sh /path/to/files/and/dirs [args...]

More information on script usage can be found in the previous subsection (Installing from Roswell).

Using from REPL

When using cl-jackc from a REPL (SLIME, Roswell, Qlot or any other means), provided that it has access to the cl-jackc system (loadable using Quicklisp), just load and use the default interface:

(ql:quickload :cl-jackc)
(cl-jackc:compile-exec "/path/to/file/or/dir" :analyze case)

COMPILE-EXEC takes a single path, which can point to a single file or directory. The key argument :ANALYZE specifies whether the file should be analyzed; if so, the user may pass one of the :XML or :SEXP keywords.

:ANALYZE can also be ignored. If so, it defaults to NIL, which indicates that a full compilation of the file(s) should be done, and not just syntax analysis.

Notice that, while it is already possible to invoke a full compilation for a path, the compiler will fail unless a syntax analysis case is specified, due to compilation progress (this will change shortly).

Any compilation or syntax analysis files generated are saved with the same name of its Jack file, except for its extension, on the same folder of the source code.

Unit testing

This project uses rove for unit testing. For using it, a system called cl-jackc/test is provided, which includes a number of test suites for parts of the compiler.

The test system is comprised of the following files:

parser-test, which contains tests for the whole analyzer, and compares the exported ASTs;
analyzer-test, which contains tests for checking whether the compilation runs without problems on valid files;
tokenizer-test, which tests for the ability of the tokenizer head to find specific and rule-based tokens on a character stream.

Tests also assume that you have qlot and roswell installed.

Running tests automatically

A run-tests.sh script is included on the repository root, and will automatically run all tests when invoked.

Running tests manually, from Slime

If you are hacking the project’s files, and you have a Slime REPL open (under qlot; for that, see qlot’s documentation), just invoke rove for the testing system:

(ql:quickload :cl-jackc/test)
(rove:run :cl-jackc/test)

Running tests manually, from Bash

Assuming that qlot is installed and configured, navigate to the project’s root directory and open the REPL:

qlot run

When the Lisp REPL is opened, load the test system and use rove to run the unit tests:

(ql:quickload :cl-jackc/test)
(rove:run :cl-jackc/test)

Extra information

For a brief documentation, check the generated documentation file.

Below are links to more information related to this project.

[Portuguese] Integrando testes automáticos para Common Lisp com Docker e Travis CI
[English] A grammar language based on S-expressions (Org / TeX / PDF)

License

This project is distributed under the MIT License.

luksamuk / cl-jackc