- Author: Cal Stephens
- Implementation: apple/swift#30570
- Status: Waiting for Review (apple/swift-evolution#1133)
- Previous Proposal: Add support for Encoding and Decoding nested JSON keys
Today, encoding and decoding Codable
objects using the compiler's synthesized implementation requires that your object graph has a one-to-one mapping to the object graph of the target payload. This decreases the control that authors have over their Codable
models.
I propose that we add a new CodingKeyPath
type that allows consumers to key into nested objects using dot notation.
Swift-evolution thread: CodingKeyPath
Application authors often have little to no control over the structure of the encoded payloads they receive. It is often desirable to rename or reorganize fields of the payload at the time of decoding.
Here is a theoretical JSON payload representing a Swift Evolution proposal (SE-0274):
{
"id": "SE-0274",
"title": "Concise magic file names",
"metadata": {
"review_start_date": "2020-01-08T00:00:00Z",
"review_end_date": "2020-01-16T00:00:00Z"
}
}
The consumer of this payload may prefer to hoist fields from the metadata
object to the root level:
struct EvolutionProposal: Codable {
var id: String
var title: String
var reviewStartDate: Date
var reviewEndDate: Date
}
Today, this would require writing a fair amount of boilerplate. The consumer would need to either write custom encoding and decoding implementation or proxy to Codable subtypes.
I propose that we add a new CodingKeyPath
type that allows consumers to key into nested objects using dot notation.
struct EvolutionProposal: Codable {
var id: String
var title: String
var reviewStartDate: Date
var reviewEndDate: Date
enum CodingKeyPaths: String, CodingKeyPath {
case id
case title
case reviewStartDate = "metadata.review_start_date"
case reviewEndDate = "metadata.review_end_date"
}
}
NSDictionary.value(forKeyPath:)
supports retrieving nested values using dot notation.
Many existing model parsing frameworks support dot notation for accessing nested keys. Some examples include:
- Mantle, "Model framework for Cocoa and Cocoa Touch"
- Unbox, "The easy to use Swift JSON decoder"
- ObjectMapper, "Simple JSON Object mapping written in Swift"
/// A type that can be used as a key path for encoding and decoding.
public protocol CodingKeyPath {
/// The components of this path. Derived automatically for a `CodingKeyPaths` enum:
///
/// enum CodingKeyPaths: String, CodingKeyPath {
/// /// components = ["rootValue"]
/// case rootValue
///
/// /// components = ["nestedObject", "value"]
/// case nestedValue = "nestedObject.value"
/// }
///
var components: [CodingKey] { get }
}
/// A container for encoding with a `CodingKeyPath` type.
/// - Internally wraps a `KeyedEncodingContainer`.
/// - Recursively follows a `CodingKeyPath` by encoding a `nestedContainer` for each component.
public struct KeyPathEncodingContainer<K: CodingKeyPath> {
public mutating func encode<T>(_ value: T, forKeyPath keyPath: K) throws where T: Encodable
public mutating func encodeIfPresent<T>(_ value: T?, forKeyPath keyPath: K) throws where T: Encodable
}
/// A container for decoding with a `CodingKeyPath` type.
/// - Internally wraps a `KeyedDecodingContainer`.
/// - Recursively follows a `CodingKeyPath` by decoding a`nestedContainer` for each component.
public struct KeyPathDecodingContainer<K> where K: CodingKeyPath {
public func decode<T>(_ type: T.Type, forKeyPath keyPath: K) throws -> T where T: Decodable
public func decodeIfPresent<T>(_ type: T.Type, forKeyPath keyPath: K) throws -> T? where T: Decodable
}
This proposal doesn't add any new requirements on the Encoder
and Decoder
protocols, so all existing implementations (JSONEncoder
, PlistDecoder
, etc.) will receive this behavior automatically.
KeyPathEncodingContainer
andKeyPathDecodingContainer
simply wrap the existingKeyedEncodingContainer
andKeyedEncodingContainer
types, so they don't require any additional support.
public extension Encoder {
func keyPathContainer<KeyPath>(keyedBy type: KeyPath.Type) -> KeyPathEncodingContainer<KeyPath> where KeyPath: CodingKeyPath
}
public extension KeyedEncodingContainer {
mutating func nestedKeyPathContainer<KeyPath>(keyedBy type: KeyPath.Type, forKey key: Key) throws -> KeyPathEncodingContainer<KeyPath> where KeyPath: CodingKeyPath
}
public extension UnkeyedEncodingContainer {
mutating func nestedKeyPathContainer<KeyPath>(keyedBy type: KeyPath.Type) throws -> KeyPathEncodingContainer<KeyPath> where KeyPath: CodingKeyPath
}
public extension Decoder {
func keyPathContainer<KeyPath>(keyedBy type: KeyPath.Type) throws -> KeyPathDecodingContainer<KeyPath> where KeyPath: CodingKeyPath
}
public extension KeyedDecodingContainer {
mutating func nestedKeyPathContainer<KeyPath>(keyedBy type: KeyPath.Type, forKey key: Key) throws -> KeyPathDecodingContainer<KeyPath> where KeyPath: CodingKeyPath
}
public extension UnkeyedDecodingContainer {
mutating func nestedKeyPathContainer<KeyPath>(keyedBy type: KeyPath.Type) throws -> KeyPathDecodingContainer<KeyPath> where KeyPath: CodingKeyPath
}
The compiler with synthesize init(from decoder: Decoder)
and encode(to encoder: Encoder)
implementations for types that provide a CodingKeyPaths
enum.
- It is invalid for a type to provide both a
CodingKeys
enum and aCodingKeyPaths
enum. - If a
Codable
type doesn't provide either aCodingKeys
enum or aCodingKeyPaths
enum, the compiler will synthesize aCodingKeys
enum. - The compiler will never automatically synthesize a
CodingKeyPaths
enum.
struct EvolutionProposal: Codable {
var id: String
var title: String
var reviewStartDate: Date
var reviewEndDate: Date
enum CodingKeyPaths: String, CodingKeyPath {
case id
case title
case reviewStartDate = "metadata.reviewStartDate"
case reviewEndDate = "metadata.reviewEndDate"
}
/// Synthesized by the compiler:
init(from decoder: Decoder) throws {
let container = try decoder.keyPathContainer(keyedBy: CodingKeyPaths.self)
id = try container.decode(String.self, forKeyPath: .id)
title = try container.decode(String.self, forKeyPath: .title)
reviewStartDate = try container.decode(Date.self, forKeyPath: .reviewStartDate)
reviewEndDate = try container.decode(Date.self, forKeyPath: .reviewEndDate)
}
/// Synthesized by the compiler:
func encode(to encoder: Encoder) throws {
var container = encoder.keyPathContainer(keyedBy: CodingKeyPaths.self)
try container.encode(id, forKeyPath: .id)
try container.encode(title, forKeyPath: .title)
try container.encode(reviewStartDate, forKeyPath: .reviewStartDate)
try container.encode(reviewEndDate, forKeyPath: .reviewEndDate)
}
}
This proposal is purely additive, so it has no appreciable effect on source compatibility.
- Code synthesis behavior and/or source validity may change for Codable models that currently have a subtype named
CodingKeyPaths
. - A quick GitHub search for
enum CodingKeyPaths
doesn't yield any relevant results, so this seems like a non-issue.
This proposal is purely additive, so it has no effect on ABI stability.
This proposal is purely additive to the public API of the Standard Library. If this proposal was adopted and implemented, it would not be able to be removed resiliently.
This design could potentially support advanced operations like indexing into arrays (metadata.authors[0].email
, etc). Objective-C Key-Value Coding paths, for example, has a very complex and sophisticated DSL.
- The author believes that there isn't enough need or existing precident to warrant a more complex design.
- Indexing into arrays seems useful on the surface, but would be quite limited in practice.
- For example, you would be able to index into the first element of an array (
[0]
) but not the last element of the array. - Additionally,
UnkeyedEncodingContainer
andUnkeyedDecodingContainer
only support sequential access (no performant support for random access).
- For example, you would be able to index into the first element of an array (
In the pitch thread for this proposal, it was brought up that the name CodingKeyPath
could potentially cause confusion with the existing KeyPath
type. We could potentially choose a different name for this type, like CodingPath
.
We would also need to rename the other types and methods added in this proposal:
encoder.keyPathContainer(keyedBy: CodingKeyPaths.self)
would becomeencoder.pathContainer(keyedBy: CodingPaths.self)
KeyPathEncodingContainer
would becomePathEncodingContainer
Valid JSON keys may contain dots:
{
"id": "SE-0274",
"title": "Concise magic file names",
"metadata.review_start_date": "2020-01-08T00:00:00Z",
"metadata.review_end_date": "2020-01-16T00:00:00Z"
}
It's practically guaranteed that there are existing Codable
models that rely on this behavior. We can't add dot-notation keypath semantics to the existing CodingKeys
type without breaking backwards compatibility for these models.
-
We could make this the default decoding behavior without breaking backwards compatibility by preferring the flat key when an exact match is present.
-
We cannot make this the default encoding behavior without breaking backwards compatibility. Encoding must be a one-to-one mapping (unlike decoding, which can potentially be a many-to-one mapping).
A previous version of this proposal added NestedKeyEncodingStrategy
and NestedKeyDecodingStrategy
configuration flags to Foundation.JSONEncoder
and Foundation.JSONDecoder
:
let decoder = JSONDecoder()
decoder.nestedKeyDecodingStrategy = .useDotNotation
try decoder.decode(EvolutionProposal.self, from: Data(originalJsonPayload.utf8))
Tony Parker (on the Foundation team at Apple) noted two main drawbacks to that approach:
It applies "globally" across the entire archive. That moves part of the behavior of how encode/decode works from the type itself (where the most knowledge about structure lies) into the encoder/decoder.
It does not apply across different kinds of encoders and decoders. If EvolutionProposal specified the keys with the . syntax then it would effectively require JSONEncoder to encode and decode itself, because part of the data structure is now part of the key name instead.
This CodingKeyPaths
approach described in this proposal is:
- Configured on a per-type basis
- Compatible "for free" with all existing
Encoder
andDecoder
implementations.
We could potentially allow authors to opt-in to this behavior by configuring a static flag on their CodingKeys
type:
// In the Standard Library:
public protocol CodingKey {
// A new protocol requirement:
static var options: CodingKeyOptions { get }
}
public struct CodingKeyOptions {
var dotNotationRepresentsNestedPath: Bool
}
// Default configuration to preserve source compatability and existing behavior:
public extension CodingKey {
static var options: CodingKeyOptions {
CodingKeyOptions(dotNotationRepresentsNestedPath: false)
}
}
// EvolutionProposal.swift
struct EvolutionProposal: Codable {
enum CodingKeys: String, CodingKey {
case id
case title
case reviewStartDate = "metadata.review_start_date"
case reviewEndDate = "metadata.review_end_date"
static var options: CodingKeyOptions {
CodingKeyOptions(dotNotationRepresentsNestedPath: true)
}
}
}
This approach seems appealing on the surface:
- We would only need to introduce one new type to the Standard Library (
CodingKeyOptions
) CodingKeyOptions
could be extended in the future to provide other customization points.- For example, we could add a key-transformation option similar to
Foundation.JSONEncoder.KeyEncodingStrategy.convertToSnakeCase
.
- For example, we could add a key-transformation option similar to
The unfortunate downside is that it's not possible to introduce new behavior on the existing CodingKeys
type without breaking backward compatability with existing Encoder
and Decoder
implementations.
- We could update Foundation's encoders and decoders (
JSONEncoder
,PlistEncoder
, etc.) to respect these new options, but existing third-party implementations would also need to be updated. - We shouldn't introduce options that aren't guaranteed to be respected in the concrete
Encoder
orDecoder
implementation being used.
The only way to add new behavior to all existing Encoder
and Decoder
implementations is to introduce a new enhanced version of CodingKey
, along with corresponding enchanced KeyedEncodingContainer
and KeyedDecodingContainer
wrappers:
/// Like a `CodingKey`, but with additional configuration options. ("CodingKey 2.0")
public protocol ConfigurableCodingKey {
var stringValue: String { get }
var intValue: Int? { get }
static var options: CodingKeyOptions { get }
}
public struct CodingKeyOptions {
var dotNotationRepresentsNestedPath: Bool
}
public extension Encoder {
func container<ConfigurableKey: ConfigurableCodingKey>(keyedBy: ConfigurableKey) -> ConfiguredKeyedEncodingContainer<ConfigurableKey>
}
/// This `ConfigurableKeyedEncodingContainer` would wrap existing `KeyedEncodingContainer` implementations,
/// which would allows the Standard Library to apply additional transformations.
/// All existing `Encoder` implementations would get this support "for free".
public struct ConfigurableKeyedEncodingContainer<ConfigurableKey: ConfigurableCodingKey> {
private let underlyingKeyedEncodingContainer: KeyedEncodingContainer<_>
public func encode<T: Encodable>(_ value: T, atKey key: ConfigurableKey) {
// Apply transformations to the key as specified by the `CodingKeyOptions`
// The Standard Library can add arbitrary complex key transformations here
// and it would apply to all existing `Encoder` implementations.
}
}
// along with a corresponding `ConfigurableKeyedDecodingContainer` implementation.
-
The
CodingKeyPath
implementation in this proposal uses this exact approach to add additional behavior on top of the existingKeyedEncodingContainer
andKeyedDecodingContainer
APIs. -
This would be an improvement over the existing
CodingKeys
type, but it has worse ergonomics thanCodingKeys
and the proposedCodingKeyPaths
.-
The author belives there aren't enough additional use cases for a
static
CodingKeyOptions
customization point for it to pull its syntactic weight. -
Static type-level configuration is less useful than per-property configuration, which cannot be done ergonomically using the existing
CodingKeys
design.
-
-
A "key" and a "path" have fundamentally different encoding and decoding semantics. It seems more appropriate to treat a
CodingKeyPath
as a distinct type rather than a flag or option on someCodingKey
type.
Instead of building upon the design of CodingKeys
, we could design an entirely new system using property-wrapper-like annotations.
struct EvolutionProposal: Codable {
// @Key("id") (compiler-synthesized)
var id: String
// @Key("title") (compiler-synthesized)
var title: String
@Path("metadata.review_start_date")
var reviewStartDate: Date
@Path("metadata.review_end_date")
var reviewEndDate: Date
}
The author believes it's more appropriate to extend and built upon the existing CodingKeys
-based system:
CodingKeys
cannot be removed or replaced, since that would be massively source-breaking.- The language should not include two separate / competing
Codable
systems.