Binary compatibility improvements
daddykotex opened this issue · comments
Series 0.19 release is going to include some changes aimed at improving the experience of introducing changes in the codebase that do not break binary compatibility. You can read more about the topic and how it works in Scala in the docs.
The goal of this issue is to track all the changes required to improve developer experience.
- mima checks for binary incompatibility changes between releases (in place for a while now)
- implement this list of changes for case classes in smithy4s-core
- implement a scalafix rule (as a linter) that fails if case classes are introduced that do not match the convention described in the link above
mima checks for binary incompatibility changes between releases (in place for a while now)
I was thinking of transitive checks (i.e. checking every patch against all the previous patches) but in the case of this repo, it's going to be terribly time-consuming.
Perhaps worth an experiment anyway? Maybe it's not that bad compared to running all the tests, and can be done in parallel.
I'm VERY opposed to transitive checks. I'd rather we invested in scalafix rules to protect against hard-to-evolve patterns.
I'm VERY opposed to transitive checks. I'd rather we invested in scalafix rules to protect against hard-to-evolve patterns.
Any reason in particular?
cost/benefit ratio. It's a lot of computing power to catch what is essentially very edge cases.
Unrelated to the discussion above, but I've had a question regarding the relevant section in the docs
it mentions:
- define a private unapply function in the companion object (note that by doing that the case class loses the ability to be used as an extractor in match expressions)
do you know why the unapply has to be private? if it's there but not private, it won't evolve with changes in the case class parameters and should not affect binary compatibility as you evolve the case class.
I ask because after running the rules to generate some def unapply for our case classes, two things happen:
- most of them are unused (nothing within the repo itself was using unapply of the case class)
- it's unreachable because it's private
my question relates to 2) because if I could make it private it would solve the compilation issue I get after I run the rule.
do you know why the unapply has to be private
It has to be private in order to signal to the compiler that it should NOT generate the traditional case class extractors :
case class Foo(a: Int, b: Int)
object Foo {
private def unapply(foo: Foo): Some[Foo] = Some(foo)
}
def test(foo: Foo) = foo match {
// you get a compile-error here.
case Foo(a, b) => println("hello")
}The traditional extractors look like this :
def unapply(foo: Foo) : Some[(Int, Int)] = Some((foo.x, foo.y))
whenever you make a change to the case class by adding/removing parameters, the signature of this extractor changes in a non binary-compatible way. This is why it's important to prevent its generation.
I ask because after running the rules to generate some def unapply for our case classes, two things happen
To be clear : I don't quite remember how scalafix works, but I'm more interested with a rule that emits a warning/error if the construct doesn't match the expectation than a rule that actually modifies the code.
If it's easy enough to get both things at once, then great.
I was about to respond with a follow up, but maybe I have those questions because I do not quite understand how they're implemented. I assumed: case class Foo(a: Int, b: Int)
would be generated as:
object Foo {
def unapply(foo: Foo): Some[(Int, Int)] = Some((foo.a foo.b))
}and so if you updated the case class by adding a parameter it would update it to:
object Foo {
def unapply(foo: Foo): Some[(Int, Int, Int)] = Some((foo.a foo.b, foo.c))
}which is breaking. But if you had manually defined this unapply yourself on the companion object, then nothing is changing it when you change the case class unless you manually do it.
but if the generated method looks like def unapply(foo: Foo): Some[Foo] = Some(foo) and that the extractors are generated by the compiler, then I understand.
To be clear : I don't quite remember how scalafix works, but I'm more interested with a rule that emits a warning/error if the construct doesn't match the expectation than a rule that actually modifies the code.
I'm trying to do both:
- one rule to do the linting: preventing you from introducing such issues in the future
- one (or multiple) rule(s) to make the changes for you (best effort, run once)
For the sake of the discussion, I'm gonna completely ignore that you're trying to do both, because it creates noise in the discussion. We'll focus exclusively on the linting aspect.
In the context of the discussion, when I say "generated", I'm talking about the methods that the compiler generates for you. When you write : case class Foo(a: Int, b: Int), what happens is the following (non exhaustive):
copymethodstoStringimplementationhashCodeimplementationequalsimplementationdef apply(a: Int, b: Int): Foosmart constructor in the companion objectdef unapply(foo: Foo): ???
From what I gather, the signature of unapply may have changed between Scala 2 and Scala 3. In Scala 3, looks like what gets generated is def unapply(foo: Foo) : Foo when in Scala 2, what gets generated is def unapply(foo: Foo): Option[(Int, Int)]. This implies that the compiler is able to do more in Scala 3 in order to deconstruct case classes.
Now that being said, the mechanism allowing to prevent the compiler from generating extractors is the same in both Scala versions, namely adding private def unapply(foo: Foo): Some[Foo] = foo to the companion should suffice in both versions.
Ok, that aligns with what I had in mind
The only thing I did not know was the difference between scala 2 / 3 which is new to me
In core we have the following list:
case class StaticBinding[A](key: ShapeTag[A], value: A)
case class DynamicBinding(keyId: ShapeId, value: Document)
case class ShapeId(namespace: String, name: String)
case class Member(shapeId: ShapeId, member: String)
case class Total[A](a: A)
case class Partial[A](indexes: IndexedSeq[Int], partialData: IndexedSeq[Any], make: IndexedSeq[Any] => A)
case class DNumber(value: BigDecimal)
case class DString(value: String)
case class DBoolean(value: Boolean)
case class DArray(value: IndexedSeq[Document])
case class DObject(value: Map[String, Document])
case class UnsupportedProtocolError(service: HasId, protocolTag: HasId)
case class ConstraintError(hint: Hint, message: String)
case class PayloadPath(segments: List[PayloadPath.Segment])
case class Label(label: String)
case class Index(index: Int)
case class PayloadError
case class OpenStringEnum[E](unknown: String => E)
case class OpenIntEnum[E](unknown: Int => E)
case class Field[S, A]
case class Alt[U, A]
case class EnumValue[E]
case class TotalMatch[A](schema: Schema[A])
case class SplittingMatch[A](matching: Schema[PartialData[A]], notMatching: Schema[PartialData[A]])
case class NoMatch[A]()
case class PrimitiveSchema[P](shapeId: ShapeId, hints: Hints, tag: Primitive[P])
case class CollectionSchema[C[_], A](shapeId: ShapeId, hints: Hints, tag: CollectionTag[C], member: Schema[A])
case class MapSchema[K, V](shapeId: ShapeId, hints: Hints, key: Schema[K], value: Schema[V])
case class EnumerationSchema[E](shapeId: ShapeId, hints: Hints, tag: EnumTag[E], values: List[EnumValue[E]], total: E => EnumValue[E])
case class StructSchema[S](shapeId: ShapeId, hints: Hints, fields: Vector[Field[S, _]], make: IndexedSeq[Any] => S)
case class UnionSchema[U](shapeId: ShapeId, hints: Hints, alternatives: Vector[Alt[U, _]], ordinal: U => Int)
case class OptionSchema[A](underlying: Schema[A])
case class BijectionSchema[A, B](underlying: Schema[A], bijection: Bijection[A, B])
case class RefinementSchema[A, B](underlying: Schema[A], refinement: Refinement[A, B])
case class LazySchema[A](suspend: Lazy[Schema[A]])
case class StreamingSchema[A](fieldName: String, schema: Schema[A])
case class DecodeError(expectedType: String)
case class StaticSegment(val value: String)
case class ParameterSegment
case class DiscriminatedUnionMember
case class HttpEndpointError(message: String)
case class Metadata
case class OnlyMetadata[A](schema: Schema[A])
case class OnlyBody[A](schema: Schema[A])
case class MetadataAndBody[A](metadataSchema: Schema[PartialData[A]], bodySchema: Schema[PartialData[A]])
case class Empty[A](value: A)
case class HttpPayloadError
case class NotFound(field: String, location: HttpBinding)
case class WrongType
case class ArityError
case class FailedConstraint
case class ImpossibleDecoding
case class UrlFormDecodeError
case class HttpRequest[+A]
case class HttpResponse[+A]
case class FullId(shapeId: ShapeId)
case class NameOnly(name: String)
case class StatusCode(int: Int)
case class OTHER(value: String)
case class UrlForm(values: List[UrlForm.FormData])
case class FormData(path: PayloadPath, maybeValue: Option[String])
case class StaticSegment(value: String)
case class LabelSegment(value: String)
case class GreedySegment(value: String)
case class UnknownErrorResponse
case class HttpUri
case class HeaderBinding(httpName: CaseInsensitive)
case class HeaderPrefixBinding(prefix: String)
case class QueryBinding(httpName: String)
case class PathBinding(httpName: String)
case class MetaDecodeError(f: (String, HttpBinding) => MetadataError)
case class StringValueMetaDecode[A](f: String => A)
case class StringCollectionMetaDecode[A](f: Iterator[String] => A)
case class StringMapMetaDecode[A](f: Iterator[(String, String)] => A)
case class StringListMapMetaDecode[A](f: Iterator[(String, Iterator[String])] => A)
case class StructureMetaDecode[A](f: Metadata => Either[MetadataError, A])
case class RequiredResponseCode[A](f: A => Int)
case class OptionalResponseCode[A](f: A => Option[Int])
case class StringValueMetaEncode[A](f: A => String)
case class StringListMetaEncode[A](f: A => List[String])
case class StringMapMetaEncode[A](f: A => Map[String, String])
case class StringListMapMetaEncode[A](f: A => Map[String, List[String]])
case class StructureMetaEncode[S](f: S => Metadata)
case class Static(value: String)
case class HostLabel(name: String)
Some of them are final, others are not.
Do you want me to address them all or do you find any exception in this list? Should I make em all final?
Some of these are internals, I'd ignore them tbh.
@daddykotex the internal ones can be left alone, provided they are adequately marked as private or package private.
Leave me deal with the ones that extend Document and the ones that extend Schema. You can start tackling the rest
Some of these are internals, I'd ignore them tbh.
yeah definitely leave the internal ones out of it. I have a configuration to exclude some packages and I'm ignoring internals any package with internals in it. but maybe my logic is wrong and I'm still picking them up, I'll double check
checked and I'm grabbing smithy4s.internals.DocumentKeyDecoder, so definitely something wrong there
I'll run scalafixAll beforehand to apply the organizeimports to avoid having those changes mixed up with the rest.
organize import of core: #1386
I'm VERY opposed to transitive checks. I'd rather we invested in scalafix rules to protect against hard-to-evolve patterns.
fun fact, we already have these.
show core3/mimaPreviousArtifacts
[info] Set(com.disneystreaming.smithy4s:smithy4s-core:0.18.0 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.4 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.16 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.5 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.12 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.3 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.2 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.7 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.15 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.9 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.8 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.14 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.13 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.6 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.10 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.11 (e:info.versionScheme=early-semver), com.disneystreaming.smithy4s:smithy4s-core:0.18.1 (e:info.versionScheme=early-semver))