backwards compatibility strategy / library versioning

[Bart Brouns]

The context / "example case" of my question:

As I learn more about compressors, I'm finding things I should have done differently in the libraries.
For example, the smoothing should be done in the lin domain, not in the dB domain.
For the sake of argument, let's assume everyone agrees this sounds better in most cases (it does).
At the same time, for example master_me has it's presets written around the sound of the current implementation.
So even though it would sound better with the new implementation, the presets would have to be rewritten for that.

I could of course come up with a new name for the improved set of compressors and just add them to the pile, but that would get confusing for users and there'd be a lot of code duplication.
I could work around the code duplication by writing clever code, but that would make it harder to read and I think also slower to compile, which matters a lot when working on big algorithms.
And then in two months I find out I made another subjective mistake (spoiler alert: I did...) and I'd have to add yet another set of compressors.

The actual question:

The compressors are just an example.
How do we deal with this in the general case?

The cleanest solution I've come across so far is how Rust does it.
In the code, you (optionally) declare the version of the libraries you want to compile against. (simplifying here)
This allows the libraries to make breaking changes as needed, but downstream authors can upgrade if and when they want to.
Would this be a possible way forward for faust?

Side note:

In Rust, if you don't give a version number, you get the latest. (simplifying again)
This works for Rust, since the users of the libraries know this, so those who don't specify a version are OK with breaking changes.

In our case, it would defeat the whole point, cause none of the current code has any version numbers.
I see two solutions:

1. We make a breaking change to make it work like Rust,
   It'd be like when everybody had to change their code to include library prefixes.
   It was a pain, and some code never got converted, but it was worth it, imho.
2. If you don't specify a version you get a library version frozen in time.

[Stéphane Letz]

• since Rust and Julia have a similar embeddable library model this would help to understand what happen in Julia
• we have some ideas on a more modern way to deal with libraries, see: https://github.com/grame-cncm/faustideas#packaging-system-for-faust-libraries. We can possibly enrich this description there with some of your comments ?

[Bart Brouns]

That looks great!
I don't think my comments would improve it, since you already mentioned Cargo as an inspiration.

[Julius Smith]

Hi Bart, I am very firmly in the camp that preserves backward compatibility forever. Names are cheap. :-) The library version can effectively be part of the name, as you and Rust suggest, but I prefer to be more explicit and either come up with a new name or create a new collection with its own mandatory prefix. In general, I think the Faust Libraries should be considered a massive repo of fodder supporting more specialized and curated downstream collections. The "standard Faust function" collection is one good example. I personally mostly use the "original JOS collection" which essentially has the prefix "old" right now :-). The "compressor overload" situation is a good example of a group that would be hard-pressed to look coherently organized. The oscillators are similarly scattered, and how about all those reverbs and physical models? No one person is going to pull all that together, but someone might define a preferred subset that they would be willing to document and curate. In summary, I would not want to discourage more variations and extensions in the Faust Libraries, but I also can't imagine documenting everything that's already there in a unified way, much less what all is hopefully to come. I think of it as a boiling pot of research ideas and contributions from which polished products can emerge. Cheers, Julius

…

On Thu, May 11, 2023 at 7:03 AM Bart Brouns ***@***.***> wrote: Closed #157 <#157> as completed. — Reply to this email directly, view it on GitHub <#157 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQZKFNIVF4BK5LIX4FMYXTXFTWT3ANCNFSM6AAAAAAX6DMDSI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

-- Julius O. Smith III ***@***.***> Professor Emeritus of Music and by courtesy Electrical Engineering CCRMA, Stanford University http://ccrma.stanford.edu/~jos/

[Bart Brouns]

Hi Julius,

I agree backwards compatibility is very important, that's why I brought this up.
I was just thinking aloud on how to best achieve it.

I can see the advantage of making a new name any time a breaking change is introduced.

On the other hand: sure, names are cheap, but if you can't find what you need, you might as well not have it.

There's a fine and subjective balance there:
I think it makes sense to have lot's of ways to generate a sine wave, since each has it's own advantages and disadvantages.
I don't hink it is a good user experience to have tons of legacy functions cluttering things up.
So far it's not too bad, but eventually there will be more legacy code than current code.

Cheers,
Bart.

[David Braun]

On the other hand, if you can't break things then you end up with too many things. The purpose of a good library versioning system would be to inform people that there are non-backwards-compatible breaks. People should be able to continue to access the older version if they opt into it.

[Julius Smith]

Ok, but how do you then blend functions across a variety of library versions? Maybe something like this is what I want: fi = library("filters.lib"); fi02 = library("02/filters.lib"); os = library("oscillators.lib"); os03 = library("03/oscillators.lib"); process = os.someOsc, os03.someV3Osc : fi.someFilter, fi02.someV2Filter; Etc. Then of course the bloat is that the distribution would include complete copies of older versions of each library (I measure about 1.7M for all the currently installed libraries, so that's not a deal-breaker for me, and readable patch-files could be used in ./01, ./02, etc.). This is easy enough to do individually that I don't mind doing it myself. My opinion now is therefore to simply maintain a very complete <FaustLibraries>/BREAKING-CHANGES.txt file, like the JUCE distribution has. We can then read it carefully before upgrading. However, this is tricky because Faust decides which Faust Libraries version to use in its submodule, so I have to carefully choose when to upgrade both Faust and Faust Libraries together. This is unfortunate because the Faust compiler almost never introduces a breaking change, so I have been very happy upgrading to the latest master-dev routinely without getting bit. I am not worried about API bloat from the creation of new names, because that's already out of control to the point that I think a new set of small, focused, downstream libraries is needed. Those can use new names, or reuse old names incompatibly, but all are protected by newly named namespace environments. I see the current libraries as an anything-goes pile of carefully tested proposals for consideration from many authors who don't have time to fully internalize all relevant pre-existing contributions first. Cheers, Julius

…

On Thu, May 11, 2023 at 3:27 PM David Braun ***@***.***> wrote: On the other hand, if you can't break things then you end up with too many things. The purpose of a good library versioning system would be to inform people that there are non-backwards-compatible breaks. People should be able to continue to access the older version if they opt into it. — Reply to this email directly, view it on GitHub <#157 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQZKFPN4RAKPZ4O2453FTLXFVRVPANCNFSM6AAAAAAX6DMDSI> . You are receiving this because you commented.Message ID: ***@***.***>

-- Julius O. Smith III ***@***.***> Professor Emeritus of Music and by courtesy Electrical Engineering CCRMA, Stanford University http://ccrma.stanford.edu/~jos/

[Bart Brouns]

I am not worried about API bloat from the creation of new names, because that's already out of control to the point that I think a new set of small, focused, downstream libraries is needed.

What you call "a set of focused libraries" is at the core of what I'm talking about.
As far as I'm concerned, they do not need to be small, as long as they are well organized.
Part of that organization should be in terms of functionality and part of it should be in terms of time/version.

As far as I can tell, that is also what the faustideas link is talking about.