[request] add language support for complex numbers

Question

[request] add language support for complex numbers

franko opened this issue 3 years ago · comments

Please add language support for complex numbers.

Context

I am considering using Lobster for engineering / scientific applications either at work or for my personal projects.

I will probably proceed to implement a "matrix" module to provide matrix support using lapack and an optimized BLAS library. This should be fine and doesn't need special support from the language.

On the other side for scientific applications it is very important to have native support for complex numbers so that they can be used naturally with their algebra operators "+" and "*". It is also important to have the elementary functions like sin, cos, exp etc defined for complex numbers.

Review of current status

I made a quick exploration about introducing complex numbers in Lobster just using some custom definitions and I stumbled into some difficulties.

Here my exploratory implementation:

// Fields could be named re and im but by using x and y we
// stay compatible with xy_f struct which is an advantage.
struct Complex:
    x:float
    y:float

// It would be handy to have a lexer support to enter imaginary number
// in the form 3.5i so that a letter 'i' immediately following a numeric
// part interpret the number as imaginary.

let e = 2.718281828459045235
let pi = 3.141592653589793238

def conj(z):
    return Complex { z.x, -z.y }

def norm(z):
    return sqrt(z.x * z.x + z.y * z.y)

def exp(x:float):
    return pow(e, x)

def exp(z:Complex):
    let n = pow(e, z.x)
    // sincos needs angles in degrees but this is mathematically unfriendly.
    // It doesn't support vector multiplication by a scalar.
    let c = sincos(180 * z.y / pi)
    return Complex { n * c.x, n * c.y }

// Cannot overload a built-in function: use another name.
def sqrt_c(z:Complex):
    let n = sqrt(norm(z))
    let c = sincos(atan2(z) / 2)
    return Complex { n * c.x, n * c.y }

let a = Complex { 3, 4 }

print("value: {a} conjugate: {conj(a)} norm: {norm(a)}")

let b = Complex { 1, 2 }

// Complex multiplication below is wrong because it is done element-wise.
// We would need compiler support to define correctly the complex number
// multiplication.
print("a: {a} b: {b},  a*b: {a*b},  a+b: {a+b}") // Wrong multiplication

// We are missing the exponential function, "exp" so we have defined our own.
print("real number exp(2): {exp(2.0)}")
print("(complex) x: {a} exp(x): {exp(a)}")

// We would need to overload elementary functions like sin, cos, exp, log, sqrt, pow to
// complex numbers. The following result is correct but we cannot call the function sqrt.
print("x: {a} sqrt(x): {sqrt_c(a)}")

Problems

Major problem:

with the definition given above the multiplication is wrong for complex number and there is no way to fix that as far as I know.

Minor problems:

scalar by vector multiplication is not supported but it is a fundamental operation for vector spaces, see: https://en.wikipedia.org/wiki/Vector_space#Definition . That would be important to write someting like w = 2 * z where z is a complex number. That would be useful also for 2D/3D games to scale vectors.

Nice to have:

language support for imaginary number literals in the form 3.0i using an 'i' that follows the numeric part
exp elementary function. I guess it is not needed to implement a game but this is a fundamental function.
pi and e constant definitions

May be, to be discussed:

built-in elementary function that works for either complex and float numbers. For example it is very common to use "exp(z)" or "sqrt(z)" where z is a complex number. Alternatively a "complex" module containing the elementary functions could be fine.

Potentially contentious:

use radians for angles instead of degrees. I understand you may not want to change this but I mention this point because for scientific applications radians make a lot of more sense.

Wouter van Oortmerssen · Answer 1 · Wed Aug 11 2021 00:47:34 GMT+0800 (China Standard Time)

The current support for vector types actually works on ANY structs made out of floats or ints that have size 2..4. For example we have a color type that is treated just like any other vector.

Downside of that is that it is not type specific, i.e. your Complex type above when applied to + is going to do component-wise addition.

To change that, we'd need some type of way to specify the desired semantics for various types, to even be able to implement whatever complex numbers do with two floats for the various operations. That would be a relatively major addition to the language, either involving hard-coding some complex number semantics (to make them as fast as possible) or involving some form of being able to specify operator overloading for specific types.

I am also likely biased, in that in all my years of programming games / game engines I've never had the need for complex numbers. I've used quaternions in character animation, but that is usually an isolated use case buried deep inside C++ rendering code if used at all (current gl_animate_mesh uses only matrices).

If I were to add additional types to the currently supported vectors, my first thought would be matrices. Our current de-facto matrix "type" is simply a vector of float, e.g. gl_set_uniform_matrix accepts vectors of 4/6/9/12/16 floats. That is not great, but its use so far is so minimal, so it works ok.

So I feel direct language support for complex numbers is too niche to me. I'd rather discuss what we could do to the language to make implementing your own complex numbers more usable.

First thing to do would be to allow built-in functions to be overridden with the same number of arguments. I am not sure how hard that is to add to the implementation but should certainly be possible.

There's no need to write norm for complex numbers btw, as the built-in magnitude already does the right thing.

Next would be some form of allowing operator overloading?

Once we get there, custom literals could be a thing maybe, as they'd likely make use of the same mechanisms as operator overloading.

Francesco · Answer 2 · Wed Aug 11 2021 05:53:28 GMT+0800 (China Standard Time)

The current support for vector types actually works on ANY structs made out of floats or ints that have size 2..4. For example we have a color type that is treated just like any other vector.

Downside of that is that it is not type specific, i.e. your Complex type above when applied to + is going to do component-wise addition.

I know about the special behavior for struct with 2..4 all ints or all floats. This makes absolutely sense to me for the addition but for the multiplication I am wondering if it is actually useful. Do you have any example of when the element-wise multiplication is actually useful ?

In term of mathematics the component-wise addition is perfectly right for a vector space but the element-wise multiplication doesn't really make sense. Aactually you can multiply two vectors in Geometric Algebra but this is widly off-topic.

We may consider removing the component-wise behavior for multiplication by default and introduce it only for specific struct objects like complex numbers for example.

This would be the solution like C99 and fortran that natively recognize complex numbers and implement from them the right algebra operations. This is in contrast to C++ that use a templated class and operator and function overloading.

To change that, we'd need some type of way to specify the desired semantics for various types, to even be able to implement whatever complex numbers do with two floats for the various operations. That would be a relatively major addition to the language, either involving hard-coding some complex number semantics (to make them as fast as possible) or involving some form of being able to specify operator overloading for specific types.

See may suggestion above. I don't advocate to add more complexity to the language. I think Lobster is very nice also because it is a simple language. We may just add simply support for complex numbers like C99. We recognize complex number as a fundamental type and add rules for multiplication. We can add also support for imaginary number literals and still keep the language simple without adding new features.

I am also likely biased, in that in all my years of programming games / game engines I've never had the need for complex numbers. I've used quaternions in character animation, but that is usually an isolated use case buried deep inside C++ rendering code if used at all (current gl_animate_mesh uses only matrices).

Right, in games they are not used but in Physics they are absolutely fundamental. If you work doing numerical simulations about maxwell equations for EM fields or any quantum mechanics. This is why in Fortran they have a fundamental status and all the math libraries like BLAS, LAPACK, Eigen support them.

If I were to add additional types to the currently supported vectors, my first thought would be matrices. Our current de-facto matrix "type" is simply a vector of float, e.g. gl_set_uniform_matrix accepts vectors of 4/6/9/12/16 floats. That is not great, but its use so far is so minimal, so it works ok.

Right, matrix would be also useful but that doesn't require special language support as far as I can tell. Only for multiplication because for matrix you don't use element-wise multiplication. Actually some systems use element-wise multiplication but it only make sense for arrays, not real matrices. This is typically for array-based languages like Matlab I think.

So I feel direct language support for complex numbers is too niche to me. I'd rather discuss what we could do to the language to make implementing your own complex numbers more usable.

That's up to you. I don't want to force any decision. I know designing a good programming language requires a lot of careful, well pondered, decisions.

First thing to do would be to allow built-in functions to be overridden with the same number of arguments. I am not sure how hard that is to add to the implementation but should certainly be possible.

That would be definitely nice but I would be okay to use something like complex_sqrt or complex_exp, that's not a deal-breaker for me.

Next would be some form of allowing operator overloading?

Once we get there, custom literals could be a thing maybe, as they'd likely make use of the same mechanisms as operator overloading.

Operator overloading, that's a big deal. I am not very inclined to add this to the language because it adds a significant complexity but to be honest for complex number and matrices this is what we would need.

Just to compare, Nim seems to have full support for complex numbers:

https://nim-lang.org/docs/complex.html

but I don't mean you should do whatever Nim does!

Just for information, Lua also allows operator overloading using the __add and __mul entries in the metatable and LuaJIT as full support for complex numbers including literals.

After all that's not really niche, as long as you venture beyond the domain of games engine this is a need that quickly arise.

Wouter van Oortmerssen · Answer 3 · Wed Aug 11 2021 06:55:43 GMT+0800 (China Standard Time)

Do you have any example of when the element-wise multiplication is actually useful ?

Non-uniform scaling?
Any kind of weighting?
Color correction?
Multi dimensional index stepping?

Not super common but I've had uses for all of these and probably many more.

I think it is more done for consistency than anything else. vector * scalar is more common in code, and therefore vector * vector should do something related, not be an error, or do something which at least at a programming (not geometry) level is a very different operation (like dot or cross, which are easier to distinguish as named functions).

Most importantly, it mimics exactly what GLSL and other shader languages do, which is the other place in Lobster where you'd use similar math. It be confusing if it meant something different.
https://stackoverflow.com/questions/13901119/how-does-vectors-multiply-act-in-shader-language

This would be the solution like C99 and fortran that natively recognize complex numbers and implement from them the right algebra operations. This is in contrast to C++ that use a templated class and operator and function overloading.

I am pretty sure I don't want the language to directly support complex numbers.. the C++ route is the only thing that would make sense to me.

See may suggestion above. I don't advocate to add more complexity to the language. I think Lobster is very nice also because it is a simple language. We may just add simply support for complex numbers like C99. We recognize complex number as a fundamental type and add rules for multiplication. We can add also support for imaginary number literals and still keep the language simple without adding new features.

That to me is more complicated, since complex numbers are just one of many types the language could support natively, so the total support cost would be higher.

Hence my approach to simplicity.. support directly what (at least in games) is 99% of the use cases, then anything beyond that just uses named functions.

Right, in games they are not used but in Physics they are absolutely fundamental. If you work doing numerical simulations about maxwell equations for EM fields or any quantum mechanics. This is why in Fortran they have a fundamental status and all the math libraries like BLAS, LAPACK, Eigen support them.

Right now, Lobster is both a general purpose programming language, but also one that comes with some facilities for its primary intended use case: games.

You're suggesting Lobster to become a language that tailors to games and physics.
I've had friends ask me to make it into a language that supports ML directly (another adjacent field that would require support of new math types).

Right, matrix would be also useful but that doesn't require special language support as far as I can tell.

If you were to be using matrices extensively, you'd probably want special language support, because a) [float] is dynamically allocated, and you'd want it to inline like other struct types, and b) it is not strongly typed, it does not know the dimensions of the matrix and can't error if you're trying to multiply something with the wrong dimensions.

That would be definitely nice but I would be okay to use something like complex_sqrt or complex_exp, that's not a deal-breaker for me.

Or square_root :)
Similarly, you can currently add add and mul methods to your struct.

Just to compare, Nim seems to have full support for complex numbers:

https://nim-lang.org/docs/complex.html

That appears to be a pretty similar direction to C++ also, i.e. not built into the language.

Just for information, Lua also allows operator overloading using the __add and __mul entries in the metatable and LuaJIT as full support for complex numbers including literals.

And thus also not built into the language. Tricks like __add are a bit simpler because Lua is dynamically typed, so it doesn't have to do anything until + on a custom type is actually called. With Lobster's type checker, that is all a bit more work :)

Francesco · Answer 4 · Wed Aug 11 2021 22:25:42 GMT+0800 (China Standard Time)

Okay, I see your point and finally I agree with you. In any case you are much better placed to judge about these matters.

So, let's summarize about some possible modifications:

operator overloading

would be needed for complex numbers for '*' and '/'
potentially useful also for matrix and in agreement with GLSL standard for matrix multiplications

built-in functions overload

to overload elementary functions like exp, sin, cos, sqrt for complex numbers

complex number literals
- to ease the notation
scalar to vector multiplication

To me the most important one is #1.

For the moment I will wait patiently and in the mean time I will proceed with my explorations using home-made complex numbers and a "mul" function to multiply them. I will also implement my own elementary functions to work on float and complex.

I may try to help you implementing this stuff if you give me some indications about where to start but I think it is a complex change at the core of the language and it is better done by yourself.

Wouter van Oortmerssen · Answer 5 · Thu Aug 12 2021 01:21:19 GMT+0800 (China Standard Time)

I think we'd have to start with (2), not only is it much simpler than (1), it could build the foundation for it.

In the simplest case for overloads (1), we do something similar to Lua, but at the typechecker level. If it finds a + node, but the left arg has a type that has __add defined for it, it rewrites it as a function call, and does normal function call type checking from there.

That would not be too hard, if it wasn't for the way function specialization is currently implemented.. hmm..

Wouter van Oortmerssen · Answer 6 · Mon Sep 06 2021 02:48:19 GMT+0800 (China Standard Time)

Been experimenting with simplifying function specialization implementation, which would lay the foundation for more type based operator rewriting.

Francesco · Answer 7 · Mon Sep 06 2021 04:43:55 GMT+0800 (China Standard Time)

Thank you so much! I was wondering if you forgot about this thing but I am glad to see you are on it.

Please let me know if you need some help. I guess this is a complex thing you want to implement by yourself because it require an intimate knowledge of the implementation. On the other side if I can help on some small related (or unrelated) things I would be glad to help. It would make me more familiar with the code base.

Wouter van Oortmerssen · Answer 8 · Mon Sep 06 2021 05:05:51 GMT+0800 (China Standard Time)

Yeah it is somewhat tricky, would not make sense as a way of getting the know the codebase :)

As for helping out, that is always very welcome! I have however no specific list of urgent or easy starter items.. you can either let yourself be driven by what you find yourself important, or have a read thru TODO.txt for inspiration of what possible things to work on are.. it's sorted very roughly in order of priority.

Wouter van Oortmerssen · Answer 9 · Fri Oct 29 2021 01:05:51 GMT+0800 (China Standard Time)

Implementing operator overloading as we speak! And the good news is that after a lot of other refactoring I did to change how functions are specialized, it is now actually really simple.

Current syntax is:

def operator+(a, b):
   ...

Here operator is a keyword (like in C++) that can take any operator token following it, and turn it into a function name.

It will then basically translate x + y into a call to the above function if it can find an operator+ function whose first arg type corresponds to the type of x.

Thus, these functions make most sense to write as methods, but will work as free-standing functions as well.

Will work for pretty much all operators, and for all struct or class types.

One thing I haven't figured out yet is what to do about value arguments for += style operators, as there is no way to "overwrite" the first argument to a function to affect the caller in Lobster. That is fine for class arguments, since you can modify them in-place anyway, but not for struct. So either we say that those operators for now should only be used with class, or I have to invent a way to do "by reference" arguments. That may be a followup to this project if necessary.

Wouter van Oortmerssen · Answer 10 · Thu Nov 04 2021 02:00:26 GMT+0800 (China Standard Time)

First implementation is here: 0f0ae42

As you can tell, that is all relatively simple for such a feature. However, it wouldn't have worked without the 3 preceding commits, in particular dca3935 which makes it possible for the type checker to rewrite parts of the AST as it type checks them, which this feature leans on heavily to be simple. It also simplified some other code in the the compiler so is generally a good direction.

Wouter van Oortmerssen · Answer 11 · Thu Nov 04 2021 02:01:50 GMT+0800 (China Standard Time)

@franko you can now implement your own complex numbers, let me know how that goes ;)
There is already a small example for quaternions included.

Francesco · Answer 12 · Thu Dec 16 2021 22:41:24 GMT+0800 (China Standard Time)

I tested the new implementation with operator overload and it works perfectly. Thank you so much for this great improvement.

Here the modified file I used to test the new implementation

// Fields could be named re and im but by using x and y we
// stay compatible with xy_f struct which is an advantage.
struct Complex:
    x:float
    y:float

    def operator*(o:Complex):
        return Complex { x * o.x - y * o.y, x * o.y + y * o.x }

    def operator/(o:Complex):
        let d = o.x * o.x + o.y * o.y
        return Complex { (x * o.x + y * o.y) / d, (-x * o.y + y * o.x) / d }

let e = 2.718281828459045235
let pi = 3.141592653589793238

def conj(z):
    return Complex { z.x, -z.y }

def norm(z):
    return sqrt(z.x * z.x + z.y * z.y)

def exp(x:float):
    return pow(e, x)

def exp(z:Complex):
    let n = pow(e, z.x)
    // sincos needs angles in degrees but this is mathematically unfriendly.
    // It doesn't support vector multiplication by a scalar.
    let c = sincos(180 * z.y / pi)
    return Complex { n * c.x, n * c.y }

// Cannot overload a built-in function: use another name.
def sqrt_c(z:Complex):
    let n = sqrt(norm(z))
    let c = sincos(atan2(z) / 2)
    return Complex { n * c.x, n * c.y }

let a = Complex { 3, 4 }

print("value: {a} conjugate: {conj(a)} norm: {norm(a)}")

let b = Complex { 1, 2 }

// Multiplication and other operation on Complex are done correctly.
print("a: {a} b: {b},  a*b: {a*b},  a+b: {a+b} a/b: {a/b}")

// Use our defined exp function for complex numbers
print("real number exp(2): {exp(2.0)}")
print("(complex) x: {a} exp(x): {exp(a)}")

// We would need to overload elementary functions like sin, cos, exp, log, sqrt, pow to
// complex numbers. The following result is correct but we cannot call the function sqrt.
print("x: {a} sqrt(x): {sqrt_c(a)}")

We still cannot overload builtin functions but having working operator overload is a great improvement. Please note that float by complex multiplication doesn't work contrary to expectations and it would be convenient to have this feature. I think it should be native to small vectors.

Wouter van Oortmerssen · Answer 13 · Thu Dec 16 2021 23:59:44 GMT+0800 (China Standard Time)

Glad it worked!
Yes, builtin overload with same number of arguments is on the todo list, as well as scalar * vector.