Cakelisp

This is a Lisp-like language where I can have my cake and eat it too. I wanted to do this after my LanguageTests experiment revealed just how wacky Common Lisp implementations are in regards to performance. I was inspired by Naughty Dog’s use of GOAL, GOOL, and Racket/Scheme (on their modern titles). I’ve also taken several ideas from Jonathan Blow’s talks on Jai.

The goal is a metaprogrammable, hot-reloadable, non-garbage-collected language ideal for high performance, iteratively-developed programs (especially games).

It is a transpiler which generates C/C++ from a Lisp dialect.

You can see the introduction to Cakelisp and check out the Hacker News announcement thread.

Features

The metaprogramming capabilities of Lisp: True full-power macro support. Macros can use compile-time code execution to conditionally change what is output based on the context of the invocation
The performance of C: No heavyweight runtime, boxing/unboxing overhead, etc.
*”Real” types:* Types are identical to C types, e.g. int is 32 bits with no sign bit or anything like other Lisp implementations do
No garbage collection: I primarily work on games, which make garbage collection pauses unacceptable. I also think garbage collectors add more complexity than manual management
Hot reloading: It should be possible to make modifications to functions and structures at runtime to quickly iterate
Truly seamless C and C++ interoperability: No bindings, no wrappers: C/C++ types and functions are as easy to declare and call as they are in C/C++. In order to support this, I’ve decided to ignore type deduction when possible and instead rely on the C compiler/linker to relay typing errors. Cakelisp will blindly generate what look like C/C++ function calls without knowing if that function actually exists, because the C/C++ compiler will tell us what the answer is
Compile-time code modification: After all macros are expanded, the programmer can specify compile-time functions which can do arbitrary modification of the expanded code. This makes it possible to validate functions, automatically insert profiling instrumentation (similar to this Jai demonstration), and other tasks which would be cumbersome or impossible to do with macros alone
Output human-readable C/C++ source and header files: This makes it possible to use Cakelisp in a subset of your project. It also means Cakelisp will work on any platform C/C++ works on. Generated code closely resembles the source Cakelisp code whenever possible
Build system: Simple projects will automatically be built and linked into an executable. Complex projects can use compile-time code execution to override stages of the build process. The code essentially knows how to build itself!

For more advantages, see doc/NeatUses.org.

Some of these features come naturally from using C as the backend. Eventually it would be cool to not have to generate C (e.g. generate LLVM bytecode instead), but that can a project for another time.

Building Cakelisp itself

Note: Windows support is a work-in-progress.

Run Build.sh in cakelisp/:

cd cakelisp
./Build.sh

This script first builds cakelisp explicitly, then uses the cakelisp_bootstrap executable to build Cakelisp using Cakelisp. Subsequent executions of Build.sh will also build using Cakelisp, which means all the caching features will be in effect (making builds much faster).

You can then run ./bin/cakelisp. Have fun! A simple test:

./bin/cakelisp --execute test/Hello.cake

It shouldn’t be hard to build Cakelisp using your favorite build system. Simply build all the .cpp files in src and link them into an executable. Leave out Main.cpp and you can embed Cakelisp in a static or dynamic library!

Dependencies

Currently, Cakelisp has no dependencies other than:

C++ STL and runtime: These are normally included in your toolset
Child-process creation: On Linux, unistd.h. On Windows, windows.h
Dynamic loading: On Linux, libdl. On Windows, windows.h
File modification times: On Linux, sys/stat.h
C++ compiler toolchain: Cakelisp needs a C++ compiler and linker to support compile-time code execution, which is used for macros and generators

I’m going to try to keep it very lightweight. It should make it straightforward to port Cakelisp to other platforms.

Note that your project does not have to include or link any of these unless you use hot-reloading, which requires dynamic loading. This means projects using Cakelisp are just as portable as any C/C++ project - there’s no runtime to port (except hot-reloading, which is optional).

Building a project using Cakelisp

Cakelisp will automatically figure out how to build simple projects into executables.

For more complex projects, many hooks and variables are provided for overriding the build process. Your code is defined in Cakelisp, and so are all build commands. This gives the code the ability to know how to build itself.

For example, you could have a .cake module which includes a 3rd party graphics library. By importing that module, the module’s compile-time hooks are added to the build process, which can do things like add the 3rd party graphics library’s lib files to the link stage.

The build hooks are all regular Cakelisp code, which means you could do something as advanced as cloning a repository from the internet, launching a subprocess to cmake and make that project, then let Cakelisp finish the build by linking the output libraries.

One huge advantage to defining your build process in a “real” programming language (as opposed to a domain-specific language interpreted by a build system) is that you can attach a debugger and single step through the build process when things go wrong.

C or C++?

Cakelisp itself is written in C++. Macros and generators must generate C++ code to interact with the evaluator.

However, you have more options for your project’s generated code:

Only C: Generate pure C. Error if any generators which require C++ features are invoked
Only C++: Assume all code is compiled with a C++ compiler, even if a Cakelisp module does not use any C++ features
Mixed C/C++, warn on promotion: Try to generate pure C, but if a C++ feature is used, automatically change the file extension to indicate it requires a C++ compiler (.c to .cpp) and print a warning so the build system can be updated

Note: The ability to output only C is not yet implemented.

I may also add declarations which allow you to constrain generation to a single module, if e.g. you want your project to be only C except for when you must interact with external C++ code.

Generators keep track of when they require C++ support and will add that requirement to the generator output as necessary.

Hot-reloading won’t work with features like templates or class member functions. This is partially a constraint imposed by dynamic loading, which has to be able to find the symbol. C++ name mangling makes that much more complicated, and compiler-dependent.

I’m personally fine with this limitation because I would like to move more towards an Only C environment anyway. This might be evident when reading Cakelisp’s source code: I don’t use class, define new templates, or define struct/class member functions, but I do rely on some C++ standard library containers and & references.

Tooling support

Emacs

Open .cake files in lisp-mode:

(add-to-list 'auto-mode-alist '("\\.cake?\\'" . lisp-mode))

Build systems

A build system will work fine with Cakelisp, because Cakelisp outputs C/C++ source/header files. Note that Cakelisp is expected to be run before your regular build system runs, or in a stage where Cakelisp can create and add files to the build. This is because Cakelisp handles its own modules such that adding support to an existing build system would be challenging.

Ideally, you should be able to rely on Cakelisp’s built-in build system. This allows Cakelisp files to know how to build themselves.

Debugging

See doc/Debugging.org. Cakelisp doesn’t really have an interpreter. Cakelisp always generates C/C++ code to do meaningful work. This means the Cakelisp transpiler, macros, generators, and final code output can be debugged using a regular C/C++ debugger like GDB, LLDB, or Visual Studio Debugger.

Mapping files will make it possible to step through code in the Cakelisp language (i.e. not in the generated language). This is similar to how debuggers allow you to step through code in C files, when under the hood it’s actually stepping through machine code. It will require building support into your editor in order to properly jump to the right Cakelisp file and line (among other things).

Why Lisp?

The primary benefit of using a Lisp S-expression-style dialect is its ease of extensibility. The tokenizer is extremely simple, and parsing S-expressions is also simple. This consistent syntax makes it easy to write macros, which generate more S-expressions.

Additionally, S-expressions are good for representing data, which means writing domain-specific languages is easier, because you can have the built-in tokenizer do most of the work.

It’s also a reaction to the high difficulty of parsing C and especially C++, which requires something like libclang to sanely parse.

Similar applications/languages

See doc/VsOtherLanguages.org for projects similar to Cakelisp.

hermanhermitage / cakelisp