Issue with encoding / decoding optional values when using semiauto derivation in Scala 3

[Anton]

Scala: 3.1.0
Circe: 0.14.3

I found an issue when encoding / decoding optional values, and I'm unsure if this is by design.

Optional values are not encoded as expected when the implicit encoder for that type is not on scope. The example below shows the effect. Is this intended?

import io.circe.Encoder
import io.circe.generic.semiauto.deriveEncoder

object Main extends App {
  case class Location(name: String)
  case class Person(name: String, location: Option[Location])

  {
    // Provides only a Person Encoder
    implicit lazy val personEncoder: Encoder.AsObject[Person] =
      deriveEncoder[Person]

    println(personEncoder(Person("John", Some(Location("New York")))))
    println(personEncoder(Person("Mary", None)))

// Results in a nested object when the location is Some(...):
//    {
//      "name" : "John",
//      "location" : {
//        "Some" : {
//          "value" : {
//            "name" : "New York"
//          }
//        }
//      }
//    }
//    {
//      "name" : "Mary",
//      "location" : {
//        "None$" : null
//      }
//    }
  }

  {
    // Provides both a Person Encoder and a Location Encoder
    implicit lazy val locationEncoder: Encoder.AsObject[Location] =
      deriveEncoder[Location]

    implicit lazy val personEncoder: Encoder.AsObject[Person] =
      deriveEncoder[Person]

    println(personEncoder(Person("John", Some(Location("New York")))))
    println(personEncoder(Person("Mary", None)))
    
// Results are as expected:
//    {
//      "name" : "John",
//      "location" : {
//        "name" : "New York"
//      }
//    }
//    {
//      "name" : "Mary",
//      "location" : null
//    }
  }
}

[Darren Gibson]

Do you know exactly which version of circe this issue was introduced in? 0.14.4 and 0.14.5 if I were to guess.

[Anton]

I just tried these combinations of Scala 3.x with circe 0.14.x, and they all provide the same output as mentioned earlier.

Scala 3.0.2

• circe 0.14.0 => same output as above
• circe 0.14.1 => same output as above

Scala 3.1.3

• circe 0.14.0 => same output as above
• circe 0.14.1 => same output as above
• circe 0.14.2 => same output as above
• circe 0.14.3 => same output as above

Scala 3.2.2

• circe 0.14.0 => same output as above
• circe 0.14.1 => same output as above
• circe 0.14.2 => same output as above
• circe 0.14.3 => same output as above
• circe 0.14.4 => same output as above
• circe 0.14.5 => same output as above

[cacapouh]

Also, Either values are not being encoded as expected.
It seems that this problem occurs with the type representing a sum type.

object I2111 extends App {
  case class LeftLocation(name: String)
  case class RightLocation(name: String)
  case class Person(name: String, location: Either[LeftLocation, RightLocation])

  implicit lazy val personEncoder: Encoder[Person] = Encoder.AsObject.derived[Person]
  println(Person("John", Right(RightLocation("R"))).asJson)

  /*
{
  "name" : "John",
  "location" : {
    "Right" : {
      "value" : {
        "name" : "R"
      }
    }
  }
}
  */
}

And the background of this problem seems to involve the issues described in the following link:
#2126

[Simão Martins]

In Scala 2.13 using the circe-generic dependency we get the following behavior:

import io.circe.Encoder
import io.circe.generic.semiauto
import io.circe.syntax._

case class Location(name: String)
case class Person(name: String, location: Option[Location])
case class Person2(name: String, location: Either[Location, Location])

{ 
  implicit val encoderPerson: Encoder[Person] = semiauto.deriveEncoder
  //  Fails with: could not find Lazy implicit value of type io.circe.generic.encoding.DerivedAsObjectEncoder[Person]
}

{ 
  implicit val encoderLocation: Encoder[Location] = semiauto.deriveEncoder
  implicit val encoderPerson: Encoder[Person] = semiauto.deriveEncoder
  println(Person("John", Some(Location("New York"))).asJson.noSpaces)
  // prints: {"name":"John","location":{"name":"New York"}}
  println(Person("Mary", None).asJson.noSpaces)
  // prints: {"name":"Mary","location":null}
}

// Conclusion: semiauto only derives the class we asked it to derive (its not recursive).
// So it requires the Encoder[Location] to exist.

{ 
  implicit val encoderPerson2: Encoder[Person2] = semiauto.deriveEncoder
  //  Fails with: could not find Lazy implicit value of type io.circe.generic.encoding.DerivedAsObjectEncoder[Person2]
}

{ 
  implicit val encoderLocation: Encoder[Location] = semiauto.deriveEncoder
  implicit val encoderPerson2: Encoder[Person2] = semiauto.deriveEncoder
  // Fails with: could not find Lazy implicit value of type io.circe.generic.encoding.DerivedAsObjectEncoder[Person2]
  // This fails because Circe does not define an implicit for Encoder[Either[L, R]]
}

{ 
  implicit val encoderLocation: Encoder[Location] = semiauto.deriveEncoder
  implicit val encoderEither: Encoder[Either[Location, Location]] = semiauto.deriveEncoder
  implicit val encoderPerson2: Encoder[Person2] = semiauto.deriveEncoder
  println(Person2("John", Right(Location("R"))).asJson.noSpaces)
  // {"name":"John","location":{"Right":{"value":{"name":"R"}}}}
}
{ 
  implicit val encoderLocation: Encoder[Location] = semiauto.deriveEncoder
  implicit val encoderEither: Encoder[Either[Location, Location]] = Encoder.encodeEither("leftKey", "rightKey")
  implicit val encoderPerson2: Encoder[Person2] = semiauto.deriveEncoder
  println(Person2("John", Right(Location("R"))).asJson.noSpaces)
  // {"name":"John","location":{"rightKey":{"name":"R"}}}
}

// As soon as we also defined an Encoder for Either things started working as expected. 

{
  import io.circe.generic.auto._

  println(Person("John", Some(Location("New York"))).asJson.noSpaces)
  // {"name":"John","location":{"name":"New York"}}
  println(Person("Mary", None).asJson.noSpaces)
  // {"name":"Mary","location":null}
  println(Person2("John", Right(Location("R"))).asJson.noSpaces)
  // {"name":"John","location":{"Right":{"value":{"name":"R"}}}}
}

// Conclusion: auto derivation is recursive (so the encoder for either is also derived)

So the current implementation of typeclass derivation in Scala 3 is closer to the auto derivation, which is unfortunate (see #2126 (comment)).