"experimental" "variants"
hannesm opened this issue · comments
Dear Madam or Sir,
I notice there are some jobs that are marked as "experimental", some of which are pre-releases of the OCaml compiler; while others are specific operating systems.
How comes that these two are intermingled? And is there a process (and document thereof) how/when they will raise out of the experimental pod? Is the build status indication related to stuff marked as experimental (i.e. overall it'll be green even if all experimental builds fail)?
Thanks for reading and your support for the OCaml community. Is OCaml-CI itself considered to be stable?
I got some replies in ocurrent/opam-repo-ci#230 (comment) and am closing this, looks like there's not much to say here.
I'll copy my longer response here for general documentation.
@hannesm the use of the experimental label is somewhat overloaded as you've noticed. The main functional purpose is to introduce new types of builds without breaking existing project's CI. For example we introduced FreeBSD support for all projects that claim to support it via their opam configuration but in reality some don't properly support it and/or our FreeBSD infrastructure is not considered stable yet. Another other type is for pre-release compilers or opam versions, to help provide visibility of future changes and let library authors prepare for those releases. One further type is linting builds that probably should be looked into as failures but currently aren't.
So we have these types of builds all reported under experimental:
- New operating systems or hardware architectures
- Pre-release OCaml/Opam
- Linting builds
Of those 1 has a clear path to being moved out of experimental, once we are happy with the stability of the underlying infrastructure and have sufficient capacity they will move to non-experimental builds. At that point project builds will fail if they fail on that platform.
2 (Pre-release OCaml/Opam) will probably always stay as experimental but perhaps we can use a different label for them. Pre-release or Preview, I am open to suggestions.
3 is tricky because that is the most opinionated type of build and not all projects work well with lower-bounds checks for example. There is clear scope for improving these checks and we would consider moving them out of experimental once most projects pass these builds.
Is the build status indication related to stuff marked as experimental
experimental builds should not fail CI for a project, if they do that's a bug and it should be reported.
Is OCaml-CI itself considered to be stable?
Yes, we try to provide a stable CI service on https://ocaml.ci.dev. Please report issues against this repo.
We do have limits to our resources to work on fixes and new features so we do need to be selective in what we spend time on.
Yes, we try to provide a stable CI service on https://ocaml.ci.dev/. Please report issues against this repo.
We do have limits to our resources to work on fixes and new features so we do need to be selective in what we spend time on.
Thanks. Just in case that's not yet known, but since several hours (days?) the OCaml CI is not running and ocaml.ci.dev is not reachable.
I tried status.ocaml.org (as hinted in ocaml/infrastructure#31), but that doesn't exist yet it seems.
Have a nice weekend, I hope it's something simple that turned ocaml CI off.
I have restarted the web service and it looks back to normal now.
Thanks a lot @mtelvers. Is there any post-mortem on what caused the service unavailability?