model: Improve cross-project support in yaml

Question

model: Improve cross-project support in yaml

oyvindberg opened this issue a year ago · comments

I reimported tapir to check out what it looks like now. A lot is great, but a few things aren't.

This particular build is very big, and different parts are cross-compiled to different targets. Supporting exactly this use case is the most difficult thing to handle in the yaml model, precisely because it's very limited.
The fact that it is limited/limiting is a feature, but ideally bleep can support exactly the feature set which can make these kind of builds a joy to work with.

1) Duplication of cross project settings

The thing which is difficult to express in the current model is differences between the cross projects for the same project.
If you have a big build matrix you currently have to repeat the same lines a bunch of times. The exact incantations can be hidden behind a template, but you still need to extend that template various places.

For instance:

  tapir-core:
    cross:
      js212:
        dependencies:
        - com.softwaremill.magnolia1_2::magnolia:1.1.2
        - org.scala-js::scalajs-dom:2.4.0
        - configuration: provided
          module: org.scala-lang:scala-reflect:${SCALA_VERSION}
      js213:
        dependencies:
        - com.softwaremill.magnolia1_2::magnolia:1.1.2
        - org.scala-js::scalajs-dom:2.4.0
        - configuration: provided
          module: org.scala-lang:scala-reflect:${SCALA_VERSION}
      js3:
        dependencies:
        - com.softwaremill.magnolia1_3::magnolia:1.2.6
        - org.scala-js::scalajs-dom:2.4.0
      jvm212:
        dependencies:
        - com.softwaremill.magnolia1_2::magnolia:1.1.2
        - configuration: provided
          module: org.scala-lang:scala-reflect:${SCALA_VERSION}
      jvm213:
        dependencies:
        - com.softwaremill.magnolia1_2::magnolia:1.1.2
        - configuration: provided
          module: org.scala-lang:scala-reflect:${SCALA_VERSION}
        sources: ./src/${SCOPE}/scalajvm-3-2.13+
      jvm3:
        dependencies: com.softwaremill.magnolia1_3::magnolia:1.2.6
      native212:
        dependencies:
        - com.softwaremill.magnolia1_2::magnolia:1.1.2
        - io.github.cquiroz::scala-java-time:2.4.0-M3
        - configuration: provided
          module: org.scala-lang:scala-reflect:${SCALA_VERSION}
      native213:
        dependencies:
        - com.softwaremill.magnolia1_2::magnolia:1.1.2
        - io.github.cquiroz::scala-java-time:2.4.0-M3
        - configuration: provided
          module: org.scala-lang:scala-reflect:${SCALA_VERSION}
      native3:
        dependencies:
        - com.softwaremill.magnolia1_3::magnolia:1.2.6
        - io.github.cquiroz::scala-java-time:2.4.0-M3
        - forceJvm: true
          module: org.scala-lang::scala3-library:${SCALA_VERSION}
    dependencies:
    - com.softwaremill.sttp.model::core:1.5.4
    - com.softwaremill.sttp.shared::core:1.3.12
    - com.softwaremill.sttp.shared::ws:1.3.12
    extends:
    - template-common-main
    - template-cross-all
    folder: ./core
    sources: ./src/${SCOPE}/boilerplate-gen

Conditionals

My best idea right now is to introduce some kind of conditional with simple predicate language to express this.

  tapir-core:
    conditional:
      scala.epoch == 2:
        dependencies: com.softwaremill.magnolia1_2::magnolia:1.1.2
      scala.epoch == 3:
        dependencies: com.softwaremill.magnolia1_3::magnolia:1.2.6
      platform == js:
        dependencies: org.scala-js::scalajs-dom:2.4.0
      platform == native:
        dependencies: io.github.cquiroz::scala-java-time:2.4.0-M3
    dependencies:
    - com.softwaremill.sttp.model::core:1.5.4
    - com.softwaremill.sttp.shared::core:1.3.12
    - com.softwaremill.sttp.shared::ws:1.3.12
    extends:
    - template-scala-reflect
    - template-common-main
    - template-cross-all
    folder: ./core
    sources: ./src/${SCOPE}/boilerplate-gen

So you may have a project which extends a template with a bunch of cross projects, and you just need to refine the them. A conditional like this can express it much more concisely than before.

It remains to be seen, but I hope we can infer these the same way templates are inferred today in the sbt import.

Conditional templates

This also paves the way for writing more reusable templates, for instance:

  template-macro-paradise:
    conditional:
      scala.binary == 2.12:
        scala:
          compilerPlugins: org.scalamacros:::paradise:2.1.1
      scala.binary == 2.13:
        scala:
          options: -Ymacro-annotations
  template-scala-reflect:
    conditional:
      scala.epoch == 2:
        dependencies:
        - configuration: provided
          module: org.scala-lang:scala-reflect:${SCALA_VERSION}

Since these are now very general they could be bundled as part of bleep:

projects:
  a:
    extends: builtin.template-scala-reflect

Open questions

To which extent could this replace templates?

Definition / refinement of projects in a cross project

I think definition and refinement of cross projects should be separated, because currently you cannot tell what you're looking at.

currently:

projects:
  a: 
    cross:
      foo: 
        ...
    extends: template-cross-all
templates:
  template-cross-all:
    foo:
      ...
    bar:
      ...

When looking at project a you cannot tell if foo was already defined as a cross project.

Refinement can likely be handled by conditionals, but definition should maybe look different.
One idea can be to not use templates for that, and define crossIds at the top-level.

# define crossId's at the top-level, along with what they add to a project
cross:
  foo:
    scala:
      version: 3.2.0
      options: -Dcross=foo
  bar:
    scala:
      version: 3.2.1
      options: -Dcross=bar
projects:
  a: 
    conditional:
      scala.version == 3.2.1:
        options: -Dfoo=true
    cross: [foo, bar]

This example assumes continued usage of arbitrarily named crossIds. These are flexible because they can be used for anything, like cross-compiling libraries across cats-effect 2 and 3 for instance. A suggestion in #251 was to rather consider the structure of a cross project and name/select it after the versions.

Ondra Pelech · Answer 1 · Tue Feb 28 2023 18:55:23 GMT+0800 (China Standard Time)

GitHub Actions is a poplar tool for "building stuff" which also uses YAML for configuration. Maybe taking inspiration from what it does (or at least from the parts that it does well) would be a smart idea! And this proposal seems to be very similar to GHA 👍

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idsteps

Øyvind Raddum Berg · Answer 2 · Wed Mar 01 2023 03:56:03 GMT+0800 (China Standard Time)

Yeah I guess it is @sideeffffect :)

I've come to learn to like that format, though it may be more a quality of the tooling rather than the format itself, not sure! Intellij havs some support for it and the feedback loop is not terrible since the runners start running the actions very soon after push.

Ondra Pelech · Answer 3 · Wed Mar 01 2023 04:36:00 GMT+0800 (China Standard Time)

Intellij havs some support for it

It has support for anything that has a (JSON)Schema, which we have for Bleep, right?

Øyvind Raddum Berg · Answer 4 · Wed Mar 01 2023 06:59:09 GMT+0800 (China Standard Time)

Yes, exactly