Support encoding/decoding non-Long numbers

[Adam]

Currently when I try to decode an integer to a Int, I get an error. This is inconvenient, and confusing.

Exception in thread "main" com.akuleshov7.ktoml.exceptions.IllegalTypeException: 
Line 2: <Int> type is not allowed by toml specification, use <Long> instead (key = port; value = 8080)

import com.akuleshov7.ktoml.Toml
import kotlinx.serialization.Serializable

fun main() {
  Toml.decodeFromString(ServerConfig.serializer(), """
    host = "localhost"
    port = 8080
  """.trimIndent())
}

@Serializable
data class ServerConfig(
  val host: String,
  val port: Int,
)

Comparison with Kotlinx Serialization

This is confusing because it does not match the behaviour of Kotlinx Serialization JSON. Technically, the JSON spec states that any number can be of any magnitude or precision (so basically KxS should only support BigDecimal or BigInteger), but KxS allows for numbers to be en/decoded with any numeric type - Long, Short, Double, Float - including unsigned numbers.

import kotlinx.serialization.Serializable
import kotlinx.serialization.decodeFromString
import kotlinx.serialization.json.Json

@Serializable
data class ServerConfig(
  val host: String,
  val port: Int, // Int is allowed
)

fun main() {
  val config: ServerConfig = Json.decodeFromString(
    """
    {
      "host": "localhost",
      "port": 123213131321321312313213213132123132131
    }
  """.trimIndent()
  )
}

KxS throws a runtime error when it encounters an invalid number

Exception in thread "main" kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 35: Numeric value overflow at path: $.port
JSON input: {
  "host": "localhost",
  "port": 123213131321321312313213213132123132131
}

Work arounds

Custom Serializer

I have tried to work around this with a custom serializer, but I get the same error.

internal typealias IntAsLong = @Serializable(with = IntAsLongSerializer::class) Int

private object IntAsLongSerializer : KSerializer<Int> {
  override val descriptor: SerialDescriptor =
    PrimitiveSerialDescriptor("IntAsLongSerializer", PrimitiveKind.LONG)

  override fun serialize(encoder: Encoder, value: Int) = encoder.encodeLong(value.toLong())

  override fun deserialize(decoder: Decoder): Int = decoder.decodeLong().toInt()
}

@Serializable
data class ServerConfig(
  val host: String,
  val port: IntAsLong,
)

.toInt()

Manually converting on-the-fly with config.port.toInt() is possible, but inconvenient, especially if the value is used in a lot of places.

Secondary value

I guess it's possible to create a secondary value to expose the actual value...? But this is weird and confusing.

@Serializable
data class ServerConfig(
  @SerialName("port")
  private val portLong: Long = 8080,
  val host: String = "localhost",
) {
  @Transient
  val port: Int = portLong.toInt()
}

Request

I would like ktoml to coerce numbers to the value in the @Serializable class. If an unsuitable value is encountered (for example, it is too large), then ktoml should throw an exception at runtime.

Versions

• Kotlin 1.7.21
• Kotlinx Serialization 1.4.1
• ktoml 0.3.0

[Andrey Kuleshov]

yeah, we thought about that with @NightEule5, TOML standard (as we understand it) prohibits the usage of INT.
But there is absolutely no problem to support it, we just wanted to support standard...

"Arbitrary 64-bit signed integers (from −2^63 to 2^63−1) should be accepted and handled losslessly. If an integer cannot be represented losslessly, an error must be thrown."

Probably my understanding was wrong 🤦

[Andrey Kuleshov]

#153 (comment)

Actually everything will start working if we simply remove this exception. I just wanted to be very straight-forward with spec... 🤣

[Andrey Kuleshov]

Looks like the change should be a little bit more complicated: we will need to support TomlInteger class and use it during parsing with avoiding of overflow: https://stackoverflow.com/questions/74510016/how-to-understand-if-there-was-an-integer-overflow-during-parsing-of-string-with

[Adam]

I didn't think it would be that difficult, so maybe I'm missing something :)

1. The KxS SerialDescriptor requests an Int from the Decoder (implemented by ktoml)
2. The ktoml decoder will attempt to read a number as a Long (as normal) in fun decodeInt()
3. Then the ktoml decoder can try to convert the Long to an Int/Short/Byte - and throw an exception if the Long isn't a valid Int/Short/Byte

This is similar to how Kotlinx Serialization JSON does it.

I've made a demo PR

ktoml/ktoml-core/src/commonMain/kotlin/com/akuleshov7/ktoml/decoders/TomlAbstractDecoder.kt

Lines 39 to 44 in f8cb2e9

 override fun decodeInt(): Int    { 
     val result = decodeLong() 
     if (result !in (Int.MIN_VALUE.toLong()..Int.MAX_VALUE.toLong())) 
         throw ParseException("not an Int", 0) 
     return result.toInt() 
 }