CL-CPP-GENERATOR

Introduction

Note: This code is obsolete. I work on https://github.com/plops/cl-cpp-generator2 now.

The purpose of this Lisp package is to bring the power of Common Lisp macros to C or C++ source code.

This project provides the Common Lisp function emit-cpp &key code str. It receives a list of special s-expressions and emits corresponding C++ code into a string.

Please look at test.lisp in order to learn how to use these s-expressions.

The semantics of the s-expressions are very close to C or C++. This is not a transpiler like parenscript that converts s-expressions to javascript or varjo that converts s-expressions to OpenGL shading language. This means it is entirely up to you if you generate C or C++ code. The lisp package knows nothing about the functions you use or define, it can’t check the validity of your type assignments.

There are two advantage compared to parenscript and varjo. The generated code can be easy to read and you can always fall back to dumping a string if you can’t figure out how a certain language feature should be expressed as s-expressions. In particular I gave up very early on writing C++ template types with s-expressions. They are just too much of a mess already.

I feel the s-expressions of cl-cpp-generator give me still a lot of power to generate lot’s of boring, repetetive code.

A tiny bit of ‘intelligence’ is in the function write-source. This function only keeps track of hashes writes code into a C source when the contents have changed.

Recently, I experimented with a code generator for Golang ( https://github.com/plops/cl-golang-generator) that has a bit of cleverness to keep track of variable types.

This is the first in a series of code generators.

https://github.com/plops/cl-cpp-generator I tried this generator with C/C++/ISPC/OpenCL/Cuda. It was my first attempt. The largest project with this is a firmware for a microcontroller. The difficult part is to get placement of semicolons right. I’m not terribly happy with the current solution. People don’t like that every function call has to be explicitly stated as such. Initially I thought it would help me to add function parameter completion in emacs. But I never figured out how to do that and in later code generators I simplified the lisp syntax.
https://github.com/plops/cl-ada-generator (abandoned) I always wanted to have a VHDL code generator and thought that ADA would be a good target to support a similar language. Unfortunately ADA and VHDL have a lot of syntactic constructs which I find difficult to express in Lisp semantics that I can remember. So I stopped working in this direction.
https://github.com/plops/cl-py-generator At a new job I generated LOTS of python code (75k lines) with this generator. The difficult part was to get indents right. It works really well. I should backport some features from cl-golang-generator: Variable type declaration, when, unless, unitary - and unitary /, logior, logand, incf, decf.
https://github.com/plops/cl-js-generator I played around with webrtc and webgl in the browser. I used parenscript before and I really like the way it upholds lisp semantics (every expression returns a value). However, the generated javascript can be difficult to read (and compare to s-expressions). For this project my goal was to have a very minimal mapping from s-expressions to javascript. Turns out converting lisp to javascript is very straightforward.
https://github.com/plops/cl-wolfram-generator (abandoned) At some point I was able to download a wolfram language license. I think this language is very close to Lisp. I tried some code generation but the free license would only work on one computer which is not how I work.
https://github.com/plops/cl-yasm-golang (abandoned for now, FIXME I accidentally called this golang and not generator). I was playing with the idea to program bare to the metal with either LLVM IR or amd64 assembly. Some prototyping indicated that this has extra challenges that can’t be easily addressed in my ‘single-function-code-generator’ approach. These are distributing variables on registers, memory handling. Also I’m not sure how much or if I could immediatly profit from existing tooling if I dump assembly or machine code.
https://github.com/plops/cl-golang-generator I used this to learn a bit of Go. I implemented quite a few examples of the Golang Programming book. In particular I like how straight forward it was to add common lisp type declarations. I’m very happy with how this experiment turned out. Golang is an expressive, simple language. Implementing the code generator was much faster than my C generator (because I don’t have to think about semicolons). Distributing the binaries is much easier than Python code. They are however so large (>20Mb) that I wonder if writing Common Lisp directly wouldn’t be the better approach.
https://github.com/plops/cl-kotlin-generator I just started that. The language looks very similar to python or golang but interacting with the android build environment seems to be rather cumbersome.

I’m not happy with differences that currently exist between the code generators. Currently, a conditional in cl-py-generator code looks like this:

(if (< a 0) (do0 (print “bla”)))

Whereas cl-cpp-generator expects this:

(if (< a 0) (statements (print “bla”)))

My hope is that eventually all these code generators converge to have the same s-expressions with semantics that are as close as possible to Common Lisp.

Occasionally, I add small repositories with self-contained example programs that use cl-cpp-generator. They all contain a gen.lisp file that contains the s-expressions to generates C++ code. Even though the C++ code is automatically generated I keep them in the source/ folder of the repositories.

https://github.com/plops/cl-gen-glfw Interactive programming example: Open a GLFW window and dlopens a library that emits OpenGL calls. This library source can be modified/recompiled/reloaded while the program keeps running.
https://github.com/plops/cl-gen-ispc-mandelbrot Emit C++ code with Intel Threading Building Blocks and Intel ISPC SIMD compiler that computes the mandelbrot set.
https://github.com/plops/cl-gen-opencv Simple example of how to call the C++ image processing library OpenCV. This example shows how to do 2D interpolation. It loads a PNG image and displays a distorted version.
https://github.com/plops/cl-gen-cufft Call Nvidia CUFFT library to perform a multi-dimensional Fourier transform on the GPU. The code draws a spherical shell into a 3D volume.
https://github.com/plops/cl-gen-fft Implementation of a one-dimensional power-of-two Fourier Transform. The algorithm is from Wikipedia. I use Lisp to unroll the C++ code more than it is usually done. I find the function bit_reverse_copy in https://github.com/plops/cl-gen-fft/blob/master/source/main.cpp particularly instructive.
https://github.com/plops/cl-gen-qt-thing A small GUI program that uses a QT canvas. When writing this I learned that writing GUIs with QT takes quite some effort. Too much, in my opinion.
https://github.com/plops/cl-gen-cpp-wasm Spin up a web server in Common Lisp and host a web assembly program that sums up an array of numbers. I deliberately use a very low level approach https://dassur.ma/things/c-to-webassembly/ to learn about web assembly.
https://github.com/plops/cl-gen-cuda-try Code generators for 1d and 2d fast fourier transforms on CPU and CUDA. For CUDA I used Google’s free colab GPU instances. I really like how the CPU implementation turned out. I actually understood the algorithm better by looking at the generated code. I’m not happy with the verboseness of CUDA’s complex variables. The SIMD version is a nightmare.

Installation

Clone the repository into Quicklisps local-projects folder:

cd ~/quicklisp/local-projects
git clone https://github.com/plops/cl-cpp-generator

If you want to add the code to a lisp that is alreay running, you will have to call:

(ql:register-local-projects)

I don’t think this is required if you start a new instance of Common Lisp. In that case quicklisp will find the new folder during its initialization.

I develop with SBCL, so for now that is the only supported implementation.

Place the following code into a .lisp file and execute the file, i.e. in Slime press C-c C-k.

(eval-when (:compile-toplevel :execute :load-toplevel)
  (ql:quickload :cl-cpp-generator))

(in-package :cl-cpp-generator)

(with-open-file (s "emitted_code.cpp"
                   :direction :output
                   :if-exists :supersede
                   :if-does-not-exist :create)
  (emit-cpp
   :str s
   :clear-env t
   :code 
   `(with-compilation-unit
          (include <optixu/optixpp_namespace.h>))))

This will output a line with an include statement into the file emitted_code.cpp. To write more you can add additional code in in the with-compilation-unit expression and re-execute the surrounding with-open-file with C-M-x.

Using macros

This an example to that defines and uses a macro. emit-cpp calls the macroexpand of the host compiler. That is why either defmacro or macrolet can be used to define the macro.

(eval-when (:compile-toplevel :execute :load-toplevel)
  (ql:quickload :cl-cpp-generator))

(in-package :cl-cpp-generator)

(defmacro with-glfw-window ((win &key (w 512) (h 512) (title "glfw")) &body body)
  `(let ((,win :type GLFWwindow*))
     (if (! (funcall glfwInit))
	 (statements (return -1)))
     (setf ,win (funcall glfwCreateWindow ,w ,h (string ,title) NULL NULL))
     (if (! ,win)
	 (statements (funcall glfwTerminate)
		     (return -1)))
     (funcall glfwMakeContextCurrent ,win)
     ,@body
     (funcall glfwTerminate)))


(with-open-file (s "emitted_code.cpp"
		      :direction :output
		      :if-exists :supersede
		      :if-does-not-exist :create)
     (emit-cpp
      :str s
      :clear-env t
      :code 
      `(with-compilation-unit

	   (function (main ((argc :type int)
			    (argv :type char**))
			   int)
	    (decl ((argc :type (void))))
	    (decl ((argv :type (void))))
	    (macroexpand
	     (with-glfw-window (main_window :w 512 :h 512)
	       (for (() (! (funcall glfwWindowShouldClose main_window)) ())

		    (funcall glClear GL_COLOR_BUFFER_BIT)
		    
		       
		    (funcall glfwSwapBuffers main_window)
		    
		    (funcall glfwPollEvents))))
	    (return 0)))))

Support for ISPC (Intel SPMD Program Compiler)

If you add the keyword ispc to `*features*`, you can use the ispc specific control expressions (foreach, foreach_unique, foreach_tiled, foreach_active, cif, bit).

(push :ispc *features*)
(eval-when (:compile-toplevel :execute :load-toplevel)
  (ql:quickload :cl-cpp-generator))

(in-package :cl-cpp-generator)
(with-output-to-string (s)
  (emit-cpp
   :str s
   :clear-env t
   
   :code 
   `(with-compilation-unit
	(dotimes (i (funcall max 2 3))
	  (funcall bla))
      (foreach (i (funcall max  1 0) (funcall min m n))
	       (funcall ata))
      (foreach ((i (funcall max  1 0) (funcall min m n))
		(j 0 n))
	       (funcall ata))
      (foreach-active (i)
		      (+= (aref a index) (bit #b0110)))
      (function (func ((v :type "uniform int")) "extern void"))
      (foreach-unique (val x)
	       (funcall func val)))))

for(int i = 0; (i < max(2,3)); i += 1) {
  bla();
}

foreach(i = max(1,0) ... min(m,n)) {
  ata();
}

foreach(i = max(1,0) ... min(m,n),j = 0 ... n) {
  ata();
}

foreach_active(i) {
  a[index] += 0b110;
}

extern void func(uniform int v);
foreach_uniq(val in x) {
  func(val);
}

Example

(with-open-file (s "o.cpp"
                   :direction :output
                   :if-exists :supersede
                   :if-does-not-exist :create)
  (emit-cpp :str s :code
            '(with-compilation-unit
              (include <complex>)
              (include "org_types.h")
              (with-namespace N
                (class CommandsHandler ()
                 (access-specifier public)
                 (constructord CommandsHandler ((callbacks :type "const DeviceCallbacks")))
                 (functiond HandleRxBlock ((data :type "const uint16_t")) void))
                (function HandleRxBlock ((data :type "const uint16_t")) void
                 (decl ((a :type uint16_t :init 3)
                        (b :type uint16_t)))
                 (+= a data))))))

#include "org_types.h"
#include <complex>
namespace N {
class CommandsHandler {
public:
  CommandsHandler(const DeviceCallbacks callbacks);
  void HandleRxBlock(const uint16_t data);

}

void HandleRxBlock(const uint16_t data){
  uint16_t a = 3;
  uint16_t b;
  ;
  a += data;
}
};

include arg arg either keyword like <stdio.h> or a string

function name params* ret expr1 expr2 … name .. function name parameters .. 0 or more but always a list ret .. return value

constructord name params* functiond name params* ret expr ..

struct union class identifier base-clause identifier .. class name like dfa%%flash base-clause .. (()) or ((public virtual buh%%fcsdf)) or ((public virtual buh%%fcsdf) (private B::C))

with-namespace name &rest cmds

with-compilation-unit &rest cmds

binary operator (+ a b c) a + b + c

setf a b c d a = b; c = d

computed assignemnt a b a += b

logical operator == a == b

compound-statement (a b c) { a; b; c; }

decl ((name :type type :init 0) ( .. ) (.. ) .. )

type name = 0;

let just like lisp, expands into block with decl inside

if cond yes [no]

for (for ((i a :type int) (< i n) (+= i 1))) for(int i=a;i<n;i+=1)

i only allow one variable initialization

statement ensure a semicolon at the end

TAGBODY {tag | statement}*

Define tags for use with GO. The STATEMENTS are evaluated in order, skipping TAGS, and NIL is returned. If a statement contains a GO to a defined TAG within the lexical scope of the form, then control is transferred to the next statement following that tag. A TAG must be an integer or a symbol. A STATEMENT must be a list. Other objects are illegal within the body.

Development with multiple output files

Usually, I develop my code emitter in one common lisp progn that writes all required files upon execution with C-M-x in Slime. Often it is necessary to write into multiple output files. However, writing the same content into a file still changes the modification time and induces a rebuild in make or ninja. The following function write-source hashes the output of each call in `*file-hashes*` and will not emit code of unmodified s-expressions into C source code.

Note: The data in `*file-hashes*` is not persistent. So if you run this code in a fresh Common Lisp instance all C files will be touched and require a rebuild. If you want to prevent that store `*file-hashes*` to disk.

(defparameter *file-hashes* (make-hash-table))

(defun write-source (name extension code)
  (let* ((fn (merge-pathnames (format nil "~a.~a" name extension)
                              (user-homedir-pathname)))
         (code-str (emit-cpp
                    :clear-env t
                    :code code))
         (fn-hash (sxhash fn))
         (code-hash (sxhash code-str)))
    (multiple-value-bind (old-code-hash exists) (gethash fn-hash *file-hashes*)
      (when (or (not exists) (/= code-hash old-code-hash))
        ;; store the sxhash of the c source in the hash table
        ;; *file-hashes* with the key formed by the sxhash of the full
        ;; pathname
        (setf (gethash fn-hash *file-hashes*) code-hash)
        (with-open-file (s fn
                           :direction :output
                           :if-exists :supersede
                           :if-does-not-exist :create)
          (write-sequence code-str s))
        (sb-ext:run-program "/usr/bin/clang-format" (list "-i" (namestring fn)))))))


(let ((header `(with-compilation-unit
                   (include <QGraphicsItemGroup>)
                 (class CustomItemGridGroup ("public QGraphicsItemGroup")
                        (access-specifier public)
                        (function (CustomItemGridGroup ((dx :type int)
                                                        (dy :type int)
                                                        (nx :type int)
                                                        (ny :type int))
                                                       explicit))
                        
                        (access-specifier private)
                        (decl ((m_dx :type "unsigned int")
                               (m_dy :type "unsigned int")
                               (m_nx :type "unsigned int")
                               (m_ny :type "unsigned int"))))))
      (code `(with-compilation-unit
                 (include "CustomItemGridGroup.h")
               (function ("CustomItemGridGroup::CustomItemGridGroup" ((dx :type int)
                                                                      (dy :type int)
                                                                      (nx :type int)
                                                                      (ny :type int))
                                                                     nil
                                                                     :ctor
                                                                     ((m_dx dx)
                                                                      (m_dy dy)
                                                                      (m_nx nx)
                                                                      (m_ny ny)))
                         (with-compilation-unit
                             (raw "// draw grid")
                           (let ((dx :init m_dx)
                                 (dy :init m_dy)
                                 (nx :init m_nx)
                                 (ny :init m_ny))
                             (dotimes (i ny)
                               (let ((x1 :init (* dx i))
                                     (y1 :init (* dy 0))
                                     (x2 :init x1)
                                     (y2 :init (* dy (- ny 1))))
                                 (funcall this->addToGroup (new (funcall QGraphicsLineItem (funcall QLineF x1 y1 x2 y2))))))
                             (dotimes (i nx)
                               (let ((y1 :init (* dy i))
                                     (x1 :init (* dx 0))
                                     (y2 :init y1)
                                     (x2 :init (* dx (- nx 1))))
                                 (funcall this->addToGroup (new (funcall QGraphicsLineItem (funcall QLineF x1 y1 x2 y2))))))))))))
  (write-source "CustomItemGridGroup" "h" header)
  (write-source "CustomItemGridGroup" "cpp" code))

This is how the emitted code in `~/CustomItemGridGroup.cpp` and `~/CustomItemGridGroup.h` looks like:

// cpp 
#include "CustomItemGridGroup.h"
CustomItemGridGroup::CustomItemGridGroup(int dx, int dy, int nx, int ny)
    : m_dx(dx), m_dy(dy), m_nx(nx), m_ny(ny) {
  // draw grid
  {
    auto dx = m_dx;
    auto dy = m_dy;
    auto nx = m_nx;
    auto ny = m_ny;

    for (unsigned int i = 0; (i < ny); i += 1) {
      {
        auto x1 = (dx * i);
        auto y1 = (dy * 0);
        auto x2 = x1;
        auto y2 = (dy * (ny - 1));

        this->addToGroup(new QGraphicsLineItem(QLineF(x1, y1, x2, y2)));
      }
    }

    for (unsigned int i = 0; (i < nx); i += 1) {
      {
        auto y1 = (dy * i);
        auto x1 = (dx * 0);
        auto y2 = y1;
        auto x2 = (dx * (nx - 1));

        this->addToGroup(new QGraphicsLineItem(QLineF(x1, y1, x2, y2)));
      }
    }
  }
}

// header
#include <QGraphicsItemGroup>
class CustomItemGridGroup : public QGraphicsItemGroup {
public:
  explicit CustomItemGridGroup(int dx, int dy, int nx, int ny);

private:
  unsigned int m_dx;
  unsigned int m_dy;
  unsigned int m_nx;
  unsigned int m_ny;
};

Implementation of tests

In order to verify that the code emitted by emit-cpp is valid I implemented unit tests in test.lisp. I also use sb-cover to create an HTML code coverage report.

The function (test <number> <code> <string>) will emit C code as defined by the s-expression in <code> using the emit-cpp function into /dev/shm/1.

The expected output is given to the test function as the third parameter <string> and is written into /dev/shm/2.

Both files are then indented with clang-format so that the test is less independent on the exact white space. Then the files are compared using the diff command.

How to fix a broke test

If the emit-cpp output is not the same as the expected <string>, an assertion error like this will show up:

The assertion
(eq nil
    #1=(with-output-to-string (s)
         (sb-ext:run-program "/usr/bin/diff"
                             '("/dev/shm/1" "/dev/shm/2")
                             :output s)))
failed with #1# = "2,5c2,5
< float f = (3.2e-7);
< double d = (7.2e-31);
< complex float z = ((2.f+0) + (1.f+0i));
< complex double w = ((2.e+0) + (1.e+0i));
---
> float f = (3.2000000000f-7);
> double d = (7.200000000000000000e-31);
> complex float z = ((2.0000000000f+0) + (1.0000000000f+0i));
> complex double w = ((2.000000000000000000e+0) + (1.000000000000000000e+0i));
".

In this case I modified the printing of floating point numbers in emit-cpp, so that the least amount of digits are printed without loosing precision. Of course this broke the previous test. If the code in /dev/shm/1 is correct, just place it into the third argument <string> of test. Don’t forget to quote quotes.

Problem

(if (== current_pattern_number pattern_number) ...

source/libview.cpp:265:41: warning: equality comparison with extraneous parentheses
      [-Wparentheses-equality]
            if ((current_pattern_number == pattern_number)) {
                 ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
source/libview.cpp:265:41: note: remove extraneous parentheses around the comparison to silence this
      warning
            if ((current_pattern_number == pattern_number)) {
                ~                       ^                ~
source/libview.cpp:265:41: note: use '=' to turn this equality comparison into an assignment
            if ((current_pattern_number == pattern_number)) {
                                        ^~
                                        =

Ideas

Destructuring

http://stackoverflow.com/questions/31394507/how-can-i-emulate-destructuring-in-c

struct animal {
    std::string species;
    int weight;
    std::string sound;
};

int main()
{
  auto pluto = animal { "dog", 23, "woof" };

  auto [ species, weight, sound ] = pluto;

  std::cout << "species=" << species << " weight=" << weight << " sound=" << sound << "\n";
}

References

how to write a dsl in lisp: https://www.youtube.com/watch?v=5FlHq_iiDW0
https://github.com/deplinenoise/c-amplify (CL)

http://voodoo-slide.blogspot.de/2010/01/amplifying-c.html

https://github.com/burtonsamograd/sxc (CL) https://news.ycombinator.com/item?id=13199610

the syntax is structured in a way that, though it resembles C, so that it is quickly readable if you know C, seems to be hard to analyze for rudimentary structure in a way that follows semantics.

[he] wrote the syntax as [he] went along going through the examples and problems of K&R. The syntax is more like C as this is more of a tool that is meant to bring C programmers into the Lisp world rather than pulling Lisp programmers into the C world

The expander of a macro is Lisp; its output is SXC.