GeoLIPI: A DSL for Implicit Geometric Modelling

GeoLIPI is a Language for modelling 2D/3D geometric objects. This is to be treated as a meta-language, from which visual programming languages can be derived. Some of the languages/visual programs that can be executed in this framework are:

CSG 3D Variants
GeoCode
SVG 2D

and many more. Check out languages.md for more details. The banner image shows ome of the highlights:

(top-left) Execute and render 3D Geometric expressions in Blender.
(top-right) Different 2D and 3D types of data that can be generated with GeoLIPI.
(mid-left) Batched Execution of expressions - speedy data generation.
(lower) Parameter Optimization for a 3D target shape.
(bottom) Parameter Optimization for 2D shapes.

Important: Research Code - Use at your own risk

I have added some documentation (with the help of ChatGPT) here.

Main usecase

Mainly, GeoLIPI attempts to embed a generic visual language in python, making it easier to use and extend for research. Additionally, it provides the following benefits:

Fast Batched Execution of programs - useful for training neural networks on large batches of program executions. See a demonstration of this in notebooks/compiled_execution.ipynb.
Single "symbolic" object, multiple execution strategies. This helps with "executing" the program in different platforms/systems (such as blender for rendering, and pytorch for optimization). See scripts/blender_render_example.py.
Parameter Optimization of arbitrary visual programs (All operations allow differentiable optimization of its parameters). See notebooks/parameter_optimization.ipynb.
[TBD] Help with searching programs for a desired execution (refer to our recent paper).
Batched PyTorch execution code for all the shader toy 2D and 3D primitives described by Inigo Quilez. See notebooks/torch_sdf2d_example.ipynb and notebooks/torch_sdf3d_example.ipynb.

Installation

Currently, there is no use actually "installing" the package. Just clone the repo and add it to your PYTHONPATH.

git clone git@github.com:BardOfCodes/geolipi.git
# Add path to your PYTHONPATH
export PYTHONPATH=$PYTHONPATH:/path/to/geolipi

Examples

Here is a basic example of using the language to create an SVG image.

import matplotlib.pyplot as plt
# geolipi.symbolic contains the "language" itself - disentangled from its evaluation/execution.
import geolipi.symbolic as gls
from geolipi.torch_compute import recursive_evaluate, Sketcher


# sketcher is used for creating position grids
resolution = 1024
sketcher_2d = Sketcher(device="cuda", resolution=resolution, n_dims=2)
device = "cuda"

# Create a complex SVG shape
# Each star is s Star with a star5 on a reg star
star_expression = gls.ApplyColor2D(
    gls.Star2D((0.7,),(0.25,)), sp.Symbol("burgundy"))
star_reg_expression = gls.ApplyColor2D(
    gls.RegularStar2D((0.5,), (12.,), (5.,)), sp.Symbol("lilac"))
circle_expr = gls.ApplyColor2D(
    gls.Circle2D((0.15,)), sp.Symbol("chartreuse"))
mix_expr = gls.SourceOver(circle_expr, 
    gls.SourceAtop(star_expression, star_reg_expression))


# Convert to pytorch tensors for evaluation
mix_expr.to_tensor()

output = recursive_evaluate(mix_expr.to_tensor(), sketcher_2d)

image = output.reshape(res, res, 4).detach().cpu().numpy()

# Show image.
plt.figure(figsize=(10, 10))
plt.imshow(image)
plt.axis('off')

This results in this output.

Here is an example of optimizing such expressions. Check out other examples in notebooks/.

Remaining TODOs

3D data DiffOpt with primitive soup.

Future TODOs

REVL (Read-Eval-Visualize-Loop) using DearPyGui.
Add DiffVG executor (for correct diff Opt of SVG)
Add SDS optimization example -> Connecting it to natural language directly.
Add differentiable CSG operation and draw operations (probability over types).
Stochastic primitives?

High level issues

Execution time. Something like a BVH is a must for really complex programs.
Initialization time - Sympy Functions are not the simplest to initialize. Furthermore, the lookup_table for tensors is not the most efficient.
What to do about numerical precision? Also SDFs are almost often not exact (after boolean operations or scaling etc.)
Aliasing - If we want beautiful output images, we need to do something about aliasing.
Which symbols should have 2D/3D explicit and which ones not? The code can be made more uniform in general.

Other related works

Many other awesome libraries exist which help with pythonic 3D modelling. I have taken inspiration from many of them. Some of them are:

Acknowledgements

A big shoutout to Inigo Quilez for his awesome work on SDFs and shader toy.
Thanks to Carson Katri for his geometry script work which helped with the blender side of things.
Derek's Blender toolbox (link) was quite helpful for materials.
Patrick-Kidger's sympytorch helped thinking about how to integrate sympy here.
Thanks to Tim Nelson's Logic for Systems course made the DNF/CNF stuff much easier to understand.
Thanks to my PhD Advisor Daniel Ritchie for his support and guidance.
Thanks to my lab mates Arman Maesumi and R. Kenny Jones for their feedback.
Hiroki Sakuma's Torch-Sphere-Tracer helped write my tiny dirty sphere tracer.

Naming

GeoLIPI stands for Geometry-Lipi. Lipi is a Hindi Language term which stands for "script" (So, Geometry-Script basically). Also, the abbreviation gls for geolipi.symbolic (which contains all the sympy expressions etc.) is inspired by OpenGL Shading Language's abbreviation glsl.

BardOfCodes / geolipi