Adding features supporting writing code in a functional style
piotrklibert opened this issue · comments
I started using MoonScript recently with Awesome window manager and with a
couple of toy Web projects based on OpenResty. I find MS generally pleasant to
use, but unfortunately (less so than Lua, but still) it lacks many features that
I came to expect from a transpiled language. I also slightly disagree with how
some features are implemented, but those are really minor details. Anyway, while
MS simplifies writing object-oriented code a lot, it lacks proper support for
other styles, from which the functional style is my focus.
The lacking features I talk about are centered around function definition and
application. For example: syntax for defining currried functions, syntax for
piping/threading values (like |> operator in OCaml/F# and Elixir, -> macro
in Clojure and other Lisps or | in shell), syntax for un-nesting calbacks,
syntax for defining pattern-matching (destructuring) functions (like function
in OCaml, or alternatively allowing destructuring in function heads and adding
syntax for function overloading like in Erlang/Elixir/Prolog/Haskell). There are
many more features I'd like to see in MoonScript, but the most important to me
are obviously the ones I need the most right now, ie. features for writing
callback-based asynchronous code in FP styles.
I said I "came to expect" some features - that's because I have worked for a few
years with a transpiled-to-JS language. The language is/was called LiveScript[1]
and together with its std-lib replacement[2] implements all the mentioned
features, as well as many others. LS is actually too big, with some parts
hastily bolted on without considering the language overall design, but in effect
it's filled to the brim with programming productivity boosters. Said boosters
are of course also dangerous in many ways and the language makes no effort to
fix that. Due to many factors LiveScript is dead already, but the features
(concepts) live on and can be implemented elsewhere.
Ok, now we're getting to the good part: I'm not going to whine about those
missing features and ask people to implement them in MoonScript for me! I'm
going to try implementing them myself - the most vital (to me right now)
features first, one at a time. I'm a fan of incremental improvement ;)
I'm not a professional in the programming languages space by any means, but I've
been toying with PLs, their theory and implementation for many years now. I
implemented a toy TCL interpreter[3] in Nim, got a fix of nested object
comprehensions into LiveScript, and implemented destructuring assignment in
Io[4] (now that I look at this, they're all dead... but I got just enough
knowledge out of them to be dangerous).
I already implemented (after chasing the left-recursion (no memoization in
lpeg?) in the grammar for a bit) the |> pipe operator; a MS fragment like
this:
1 |> f |> g
is now compiled to this Lua code:
g(f(1))
My implementation is bare-bones currently, ie. it it doesn't allow to do this
(for example):
> 1 |> print(2)
1 2
but it'll definitely improve with time. The next feature I want to implement are
"back-calls"[5], which are basically a little syntactic cheat to make working
with nested callbacks more pleasant. Unexpectedly, the feature is more
versatile, and can be used for many kinds of DSLs.
Summing all this up: I'm going to implement many new syntactic constructs by
changing/adding to the MS implementation. I have no idea how long I will keep at
it, which features exactly I'll end up implementing, nor how big the needed
changes/rewrites in MS would be needed - and most of all I don't know when I'll
get stuck - but I'm currently determined to at least try doing this.
I'd like to ask a few questions related to the above:
-
Is there a technical reason for some of the features being impossible to
implement (or simply being a "bad idea" in Lua)? Which ones? -
Is there anyone interested in helping? Or at least discussing the unexpected
problems and the features themselves... I could use a whole lot of that :) -
Is there any hope of ever merging my changes upstream? I'm doing it for my
own use, but if there's a chance, I'll try to make the changes/PR "production
ready", with more comments, an additional round of cleanups, and updated docs.
PS. All of this is written with the assumption that MS is MIT-licensed - please
let me know if there are any legal problems with my ideas!
[1] https://livescript.net/
[2] http://www.preludels.com/
[3] https://github.com/piotrklibert/nimiTCL
[4] https://klibert.pl/posts/adding_destructuring_bind_to_io.html
[5] http://livescript.net/#functions-backcalls
LS is actually too big, with some parts
hastily bolted on without considering the language overall design
I'm not sure "design" is the right word, even... But you also have to remember LS is 3 transpilers in a trenchoat (CoffeeScript, Coco, itself) + its host language, MoonScript doesn't inherit anything like that.
got a fix of nested object comprehensions into LiveScript
Yeah, thanks for 541, this bit of code is hairy.
I completely agree in that MoonScript does not have the best set of features for functional programming. Functional style is also my main focus when programming in MoonScript and I generally do not touch the object-oriented features at all.
I like many of the features you present and miss some of them myself, although I usually use Haskell as a reference for these features because it happens to be the language I am most familiar with (even though I have not written a single program in Haskell).
I will say though, that my best guess is that MoonScript's intentions of the language definitely do not agree with these new features, which probably will not get through, given how things that I deem more necessary in the language haven't even gone through. But a fork is always appreciated.
For your question #1:
Is there a technical reason for some of the features being impossible to implement (or simply being a "bad idea" in Lua)? Which ones?
Some features are actually possible to implement and very much feasible, and it is not like they would disagree with Lua in the sense that some of them are just better syntax for calls. Something that I find missing myself are the function infixing and definition of operators, of course taking Haskell as a reference here. I would love to be able to define ::, $, . and many more as operators. I messed with these ideas in several ways, by applying complex regexes to implement operators such as $, or using MoonScript's transformer features, but essentially always abusing MoonScript's current syntax. Just look at the v7 version of my project typekit, trying to replicate things such as type signatures using functions and strings, or look at the definition of monads. What I want to show with this is that there are cases where extension of the language could be very useful, as you say.
For your question #2:
Is there anyone interested in helping? Or at least discussing the unexpected problems and the features themselves... I could use a whole lot of that :)
I'm interested in this issue and I am mostly free to ping in the Discord server, having written a lot of MoonScript myself, to discuss and work on it whenever.
For your question #3:
Is there any hope of ever merging my changes upstream? I'm doing it for my own use, but if there's a chance, I'll try to make the changes/PR "production ready", with more comments, an additional round of cleanups, and updated docs.
My best guess would be: not a chance. I do not think that Leafo would agree with these changes, let alone maintain them. That is not necessarily a bad thing and I admire Leafo for how he has managed MoonScript, it just simply isn't in the focus of the language. Your best bet is a self-maintained fork. I would have said community-maintained, but us regular and known MoonScript users are like 20 at most.
Other comments:
-
I know that MoonScript is eventually getting a better rewrite of the compiler, but everyone is unsure when. I don't think big changes will be introduced before that.
-
While pattern-matching functions are great, it would break with the current way functions are defined terribly. Either you propose a magically perfect standard that fixes that, or it is not going to look good.
-
Mind compatibility. MoonScript is not a language big enough to survive such a giant syntax change and would essentially phase most known projects out.
-
I am personally not very good at parser and compilers, but I can contribute some of my ideas and always discuss syntax.
(Sorry for the delay, I was busy being in pain after the dentist)
I'm not sure "design" is the right word, even...
...let's agree not to say anymore and to let LS rest in peace :D
MoonScript doesn't inherit anything like that.
Which is great! Compared to hacking on LS, MoonScript feels almost like a clean slate - it's so much easier to implement features without having to update half a dozen others.
That's not to say the current features of MS are entirely orthogonal or that there aren't any landmines in their implementations, but that's a given for any non-trivial language.
... I usually use Haskell as a reference ...
Haha, and here I went out of my way not to mention Haskell too much, assuming it would scare people off :) More precisely, this bit here:
Unexpectedly, the feature is more versatile, and can be used for many kinds of DSLs.
Should read "...used, e.g. for Haskell-like do notation". I think that with coroutines you can jump straight into implementing even the more advanced monads like continuation monad... Well, you'd like first to consider if they're needed before the jump though :)
I will say though, that my best guess is that MoonScript's intentions of the language definitely do not agree with these new features, which probably will not get through, given how things that I deem more necessary in the language haven't even gone through. But a fork is always appreciated.
Which features? Are they unimplemented because of the lack of available workforce, or is said workforce simply opposed to them? [BTW, I still didn't get to digging through the pile of 150 issues :(, so I'd appreciate links to specific feature requests which were turned down.]
It is generally impossible to only learn the parts of a language related to one's desired features. It's possible to miss some dark corners, but to implement the features I want, I need to understand the implementation well enough to be able to implement features in general. The only problem is the finite time for both learning and implementing, which necessitates one to prioritize the features by putting some before/above the others. But, on the other hand, it also means that "if there's spare time there's no problem in implementing features further down the list".
I find missing myself are the function infixing
With a `f` b syntax or something similar? I'd have to think a bit about how to make the parser agree to this, but it doesn't look harder to implement than my a |> f(b).
I would love to be able to define ::, $, . and many more as operators.
That's harder to do; the current parser doesn't really handle the precedence of operators very well (actually, I think it doesn't do it at all - binary operators seem to pass-through to Lua unobstructed where the precedence rules are applied). Unless we're talking about a fixed set of special-cased symbols (which would both match at an operator place in expressions and act as a name in fndef - Elixir does that for example), we will need an operator-precedence-aware parser and possibly a way to declare the precedence and associativity of a new operator (like infixr/infixl). I'm not sure I'd want to go there right off the bat, although the ability to extend the operator set with user code is indeed interesting. (One of the most beautiful uses of that ability I've seen in Prolog, in "definite clause grammars" - https://www.metalevel.at/prolog/dcg - which transform Prolog into a language description DSL with as little as a single new operator definition (the long-arrow: -->))
I messed with these ideas in several ways, by applying complex regexes to implement operators such as $, or using MoonScript's transformer features, but essentially always abusing MoonScript's current syntax.
Please don't! I mean, you should never attempt to parse a non-regular language using regular expressions. Have you seen this? And that's for HTML, which is rather simple as far as languages go! I know that the regexes currently in use are not "regular expressions" at all, with all the extensions, but they're still too weak to describe complex grammars comfortably. Well, I write this without (yet - sorry) reading your code, so just treat it as a piece of general advice (but rooted firmly in CompSci).
[EDIT: and also, MS already uses a (bare-bones, though) implementation of a stronger formalism, ie. PEG. There should (in principle) be no need to revert to regexes, although in practice (esp. where performance matters) they're still used within the PEG grammars. There's a lot of conveniences missing, but they are all (cleanly, mostly) expressible in terms of lpeg.C and lpeg.Cmt. ]
I'm interested in this issue ...
Thank you! I will definitely pester you with features design and esp. testing. My use-case - while vital to me and unlikely to quickly go away (I switch WMs every 5 years on average, and I started with Awesome a month ago :)) - is tiny (not even 1k loc at the moment... but, if it goes anything like with Emacs, I'll have tens of kloc in a matter of months ;)) and it definitely won't be enough to stress-test the new implementation. In terms of compatibility testing, we can compile MoonScript's codebase itself, but the new features are, well, new, so they don't exist in any MS project - and they won't magically appear all over Github just because they get implemented. We'll need to write a lot of MS from scratch to test them.
My best guess would be: not a chance.
That's unfortunate, but, well, kind of expected. I mean, the language we're talking about is a teen already, yet it still remains pretty focused (to say it kindly - you could also fault this as excessive minimalism, but hey, that's a matter of personal taste...) On the other hand, this - if true, I hope @leafo will say something about this at some point - also leaves me free to consider even the most outrageous ideas[1] with a peace of mind.
but us regular and known MoonScript users are like 20 at most.
That's plenty enough, though? I was going to do it for myself actually, i.e. with #expected_user_count == 1, so to have potentially 20 testers is quite a luxury already :)
I know that MoonScript is eventually getting a better rewrite of the compiler, but everyone is unsure when. I don't think big changes will be introduced before that.
Well, that's a bummer. I don't want to go luajit route, where I branch off a version quickly replaced by the newer, yet largely incompatible with my fork. I'd really like @leafo to comment on that. What exactly is/was planned, ie. what objectives would the rewrite be meant to achieve? Which parts of the codebase would be affected? I've seen some ad-hoc hacks here and there, but nothing that would warrant a full rewrite. Well, the AST could use rework, though.
Mind compatibility. [ie. pattern-matching functions]
Sure. I'm talking specifically about language extension, I don't plan (right now at least) to change any existing syntax. It's not needed in the vast majority of cases in general. Consider that there are languages well into their 70s which still compile and run their first implementation (ok, I'm cheating by bringing up Common Lisp here, but you can really see it everywhere: for every Python3 and Perl6 out there we have hundreds of (almost) perfectly backward-compatible language releases). Especially in small languages, if the features are reasonably orthogonal, you can go very far before crashing into the compatibility wall.
Whoa, that's a lot of text, I feel I wrote more just now then on my blog for the last two years, yet I still didn't even get to describing even a single feature I want in detail... I'd love to keep at it for another 2 hours, but unfortunately, there's that thing called life which also requires my attention, so I can't :( I'll probably write a bit more tomorrow, and will use the weekend to clean this up: https://github.com/piotrklibert/moonscript/pull/1/files - the spurious requires and duplicated utils are the remnants of a debugging session, don't let them bother you ;) - and later to set up all the things I ignored for now, such as tests.
[1] If you wonder how outrageous: enough that retargeting MS to bare metal via LLVM seems tame in comparison ;)
I know that the regexes currently in use are not "regular expressions" at all, with all the extensions, but they're still too weak to describe complex grammars comfortably.
Wow, that's nice. Well, it just means I didn't know what the "all the extensions" means in practice, esp. in PERL. TIL.
(EDIT: also an interesting read: https://metacpan.org/pod/PPI)
Wow, that's nice. Well, it just means I didn't know what the "all the extensions" means in practice, esp. in PERL. TIL.
Without derailing the thread further: video on the topic. This is possible with the regex flavors of .NET, Ruby, Python should also be able to do that.
Which features?
In my opinion, as features which are necessary, I would say #3, #15, #122, #156, #375 and more recently #402. That is of course not counting features that would certainly help me such as an operator x <op> y that becomes x = y x, which I felt like I needed countless times.
I honestly have not bothered opening issues for them anymore since I know that they will probably be shortly considered then discarded, like most syntax extensions that are not actually either expected or introduced to Lua. I would love it if MoonScript had something similar to Haskell' where new, experimental features can be discussed and implemented into future versions of the language. This, of course, requires people who are willing to submit changes, people to test them, and people to maintain them.
Are they unimplemented because of the lack of available workforce, or is said workforce simply opposed to them?
The only workforce here really is leafo, I would dare say. There are as many as 151 open issues but barely 11 open PRs, and most actually useful PRs are back from 2016. On top of that, Leafo has, and with reason, been critical to some of these game-changing ideas.
That's harder to do; the current parser doesn't really handle the precedence of operators very well (actually, I think it doesn't do it at all - binary operators seem to pass-through to Lua unobstructed where the precedence rules are applied).
In fact, I would dare say it's worse precedence handling than Lua! Guess what type x == "string" compiles to? That's right, type(x == "string"), which is... understandable if you get what the compiler is doing, but also very confusing. Many times I find myself writing way too many parentheses because the amount of errors I used to get from wrong precedence were countless.
we will need an operator-precedence-aware parser and possibly a way to declare the precedence and associativity of a new operator (like infixr/infixl). I'm not sure I'd want to go there right off the bat, although the ability to extend the operator set with user code is indeed interesting.
In my opinion this is definititely something I would include in a MoonScript rewrite, proper ordered operator precedence.
Please don't! I mean, you should never attempt to parse a non-regular language using regular expressions. Have you seen this?
Haha, I know, I know. I know all the horrors of Regex parsing things you should not use Regex for at all. It's not production-ready code and pretty much for my enjoyment and experimentation with ideas, since I couldn't mess with MoonScript's internals personally. No need to lecture me on that, believe me! I just love pushing Lua/MS to its limits and doing cursed stuff with it.
We'll need to write a lot of MS from scratch to test them.
Good, I write MS daily, count on me. But only kindof. I'm not as availiable as I used to be.
That's plenty enough, though?
You will find that it's not. You will find people using MoonScript, but it is rather individuals than a community. Hell, Lua feels like a giant community compared to MoonScript's which consists of me going off in the #moonscript channel of the Discord server and occasionally helping each other. There won't be enough of an userbase to support great changes, but it is true that if some changes do get passed, they will be used quickly, since it's not the same getting 20 people to change than 1,000. Ask the people who still use python2.
Well, that's a bummer. I don't want to go luajit route, where I branch off a version quickly replaced by the newer, yet largely incompatible with my fork. I'd really like @leafo to comment on that. What exactly is/was planned, ie. what objectives would the rewrite be meant to achieve? Which parts of the codebase would be affected?
I will note that the compiler rewrite notice was like, only superficially mentioned in the Discord? I don't even think I can pinpoint the actual quote but you can always try. But judging by Leafo's behavior towards other smaller issues, I predict that if there is a rewrite it will produce semantically identical compiled code (Lua syntax may change, the generals won't), and that once the language is replicated as it currently is, then new features can be introduced.
A short update from me: I managed to implement pipe operator and backcall syntax. They look like this:
Other than that, I've been thinking about what you @daelvn wrote here and I have to say I got kind of discouraged. First, MoonScript seems dead as a project, or maybe un-dead - a zombie animated by just a few people who feel strongly about it. The original author is not among these people, apparently, and the development efforts lack leadership, which results in them being unfocused and chaotic (why would you ever reimplement the codebase in C++ is beyond me, for example). There are people much more knowledgable than me - lurking on Discord taught me that - but the changes they suggest and prototype are so far removed from the basics (eg. the tickets you mention) that they are of little use to me right now. It's kind of disappointing to see all these smart people arguing about type systems while things like exp1; exp2 are still unimplemented. Also, to be honest, the code of Moonscript could use some love in basically every part of it - it's not the most beautiful thing I've ever read, to say the least.
Due to all that I'm thinking about a more radical departure from MoonScript. My free time is limited and my use case is pretty niche - though I still believe MS is a good fit for Awesome WM - so I'm thinking about simply dropping all the unnecessary parts, backward compatibility be damned. I would rename my fork to something like AwesomeScript and focus on integration with the window manager - this is something I personally want and need; it's obvious that I won't get any help in this, so I need to use my time as efficiently as possible. I will need to cut a lot of corners to get what I want before the heat death of the universe, and I feel that discussing said corners with people who won't contribute any code would be a waste of time.
To summarize: I still intend to work with the codebase, but I'm going to get away from MoonScript-the project and MoonScript-the community. I'm sorry and sad about that, and I of course welcome any contributions to my fork, but I don't have the time to waste on things outside of my immediate needs. I can't let my enthusiasm to diminish any further - there's a huge amount of other languages I could use and I happen to know most of them, but once I start looking at them I'll have to admit I wasted lots of time on MoonScript and Lua already - I don't want that to happen.
First, MoonScript seems dead as a project, or maybe un-dead - a zombie animated by just a few people who feel strongly about it. The original author is not among these people, apparently, and the development efforts lack leadership, which results in them being unfocused and chaotic
Although sad, I think this feels very much true. I would love MoonScript to succeed as a project. I am fairly sure that MoonScript does get used on itch.io, for example, though, so I don't think the author doesn't actually like the language; just that it seems to me that it already fits his needs and therefore doesn't need much more tweaking.
It's kind of disappointing to see all these smart people arguing about type systems while things like
exp1; exp2are still unimplemented.
Oh, yeah, that... Although it wasn't directly related to MoonScript, or something to be implemented in MoonScript, the language does not get much attention, no.
Also, to be honest, the code of Moonscript could use some love in basically every part of it - it's not the most beautiful thing I've ever read, to say the least.
As someone who tried to, unsuccessfully, tweak the code to make syntax changes: big mood. It doesn't feel like code you can just get into.
so I'm thinking about simply dropping all the unnecessary parts, backward compatibility be damned.
As my opinion is with Lua, but not as strongly, sometimes a language just needs many breaking changes to be reborn. MoonScript definitely needs to be reworked completely, no matter what it means for compatibility.
it's obvious that I won't get any help in this, so I need to use my time as efficiently as possible. I will need to cut a lot of corners to get what I want before the heat death of the universe, and I feel that discussing said corners with people who won't contribute any code would be a waste of time.
I'm no expert coder, regardless of my 8 years of script kiddie experience, and I'm not very knowledgeable in languages, but I would definitely try to pitch in for a radical change for MoonScript.
To summarize: I still intend to work with the codebase, but I'm going to get away from MoonScript-the project and MoonScript-the community. I'm sorry and sad about that, and I of course welcome any contributions to my fork, but I don't have the time to waste on things outside of my immediate needs. I can't let my enthusiasm to diminish any further - there's a huge amount of other languages I could use and I happen to know most of them, but once I start looking at them I'll have to admit I wasted lots of time on MoonScript and Lua already - I don't want that to happen.
This is honestly sad to hear, since I was kind of hoping that we would get a driving force for a MoonScript rework, but it seems like that is never happening, since people come to the project and then decide to get away from it, and they are very right to do so! I do not understand why I'm still on this sinking boat but I like to be on it. I just wish we had something better, but if that doesn't come from people who really don't want to drop MoonScript like me, then it just isn't happening.
Looking forward to your fork.