Giters

Effect-TS / schema

Modeling the schema of data structures as first-class values

Home Page:https://effect.website

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Make `compose` more flexible

vecerek opened this issue · comments

commented

🚀 Feature request

Current Behavior

import * as S from "@effect/schema/Schema";

const schema = S.record(S.string, S.unknown).pipe(S.compose(S.struct({ a: S.string })))
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                  Type '{ readonly [x: string]: unknown; }' is
                                                  not assignable to type '{ readonly a: string; }'.ts(2345)

Desired Behavior

The above code (and similar) compiles successfully.

Suggested Solution

Change the signature of compose to:

interface Compose {
  <B1, B2 extends B1, C>(bc: Schema<B2, C>): <A>(ab: Schema<A, B1>) => Schema<A, C>
  <A, B1, B2 extends B1, C>(ab: Schema<A, B1>, bc: Schema<B2, C>): Schema<A, C>
}

Who does this impact? Who is this for?

It's for everyone who wants to be able to transform schemas with ease.

Describe alternatives you've considered

Additional context

Your environment

Software Version(s)
@effect/schema
TypeScript
commented

@vecerek yeah it seems handy, I need a little more time to think about all the consequences of this change though (in particular whether it will play well with schemas being invariant)

commented

Thanks for the review @gcanti! Could you explain what you mean by "schemas being invariant"? Is it about the decode-encode roundtrip property or something else?

commented

In Schema<A> (let's consider one type param for simplicity) A must be invariant because we want to derive artifacts like decoders:

(u: unknown) => A // A here is in covariant position

but also artifacts like encoders:

(a: A) => something // A here is in contravariant position

so A can be at the same time in covariant position AND in contravariant position, ergo is an invariant type param.

That's the reason why Schema is defined as:

export interface Schema<To> {
  // ...
  readonly To: (_: To) => To // <= this ensures the invariance of the `To` type paramater
  readonly ast: AST.AST
}
commented

From a practical point of view, if we wanted to define a Decoder it could only be covariant, and in this case Decoder<string> is a subtype of Decoder<string | undefined>. On the other hand, if we wanted to define an Encoder it could only be contravariant, and then Encoder<string | undefined> is a subtype of Encoder<string>. But Schema<string> is not a subtype of Schema<string | undefined> nor Schema<string | undefined> is a subtype of Schema<string> because we could derive both decoders and encoders:

import * as Schema from '@effect/schema/Schema'
import * as AST from '@effect/schema/AST'

interface Decoder<A> {
  readonly To: () => A // only covariant
  readonly ast: AST.AST
}

declare const decoder1: Decoder<string>
export const decoder2: Decoder<string | undefined> = decoder1 // no error, great

interface Encoder<To> {
  readonly To: (to: To) => void // only contravariant
  readonly ast: AST.AST
}

declare const encoder1: Encoder<string | undefined>
export const encoder2: Encoder<string> = encoder1 // no error, nice

declare const string: Schema.Schema<string>
export const stringOrUndefined: Schema.Schema<string | undefined> = string // error! awesome
/*
Type 'Schema<string, string>' is not assignable to type 'Schema<string | undefined, string | undefined>'.
  Types of property 'From' are incompatible.
    Type '(_: string) => string' is not assignable to type '(_: string | undefined) => string | undefined'.
      Types of parameters '_' and '_' are incompatible.
        Type 'string | undefined' is not assignable to type 'string'.
          Type 'undefined' is not assignable to type 'string'.ts(2322)
*/

declare const stringOrUndefined2: Schema.Schema<string | undefined>
const string: Schema.Schema<string> = stringOrUndefined2 // error again! awesome
/*
Type 'Schema<string | undefined, string | undefined>' is not assignable to type 'Schema<string, string>'.
  The types returned by 'From(...)' are incompatible between these types.
    Type 'string | undefined' is not assignable to type 'string'.
      Type 'undefined' is not assignable to type 'string'.ts(2322)
*/
commented

now back to compose:

<A, B1, B2 extends B1, C>(ab: Schema<A, B1>, bc: Schema<B2, C>): Schema<A, C>

the constraint B2 extends B1 is "breaking the rules".

It makes perfectly sense in the world of Encoders (or any other contravariant derivations like Pretty for example):

  • if ab can encode a B1 it can also encode a B2 for sure (e.g. if ab can encode a string | undefined it can also encode a string for sure)
  • if ab can pretty print a B1 it can also pretty print a B2 for sure
  • etc...

but what about covariant derivations like Decoders or Arbitrary or ...?

commented

the constraint B2 extends B1 is "breaking the rules"

@vecerek btw ^ this means that changing the signature of compose without changing its implementation accordingly will lead to bugs, for example the following schema definition in now allowed by the new signature (because string is a subtype of string | null) but it throws a runtime error:

it("(A U B) compose (B -> C)", async () => {
  const schema = S.union(S.null, S.string).pipe(S.compose(S.NumberFromString))
  await Util.expectParseFailure(schema, null, "Expected string, actual null")
})
TypeError: Cannot read properties of null (reading 'trim')
 ❯ Object.decode src/Schema.ts:2205:13
    2203|         return PR.success(-Infinity)
    2204|       }
    2205|       if (s.trim() === "") {
       |             ^
    2206|         return PR.failure(PR.type(schema.ast, s))
    2207|       }
commented

@gcanti thanks for the thorough explanation, it's been super helpful. I need to take some time and reflect on all this 😅

commented

I'm also fine with keeping compose strict but then would need to find another way of making transformations like the ones below more ergonomic for the users of the library:

  • record -> struct
  • array -> tuple
  • string -> literal

Maybe it just means more of such specific transformers instead of one generic such as compose.

commented

record -> struct

@vecerek what's the use case though? Let's say you can do this (your initial feature request):

const schema = S.record(S.string, S.unknown).pipe(S.compose(S.struct({ a: S.string })))

now schema has type:

const schema: S.Schema<{
    readonly [x: string]: unknown;
}, {
    readonly a: string;
}>

which means that you are not gaining anything while parsing but you are losing precision while encoding (with respect to S.struct({ a: S.string }))

commented

@gcanti I have several use cases:

  1. First parsing a jwt token and then refining its type in a second step.
  2. Taking a file, reading it as a string then parse that through JSON.parse and refine that to a further struct.
JWT schema 
import * as ParseResult from "@effect/schema/ParseResult";
import * as Schema from "@effect/schema/Schema";
import {
  TAlgorithm as Algorithm,
  decode as decodeJwt,
  encode as encodeJwt,
} from "jwt-simple";

export interface Args {
  key: string;
  algorithm: Algorithm;
}

const noVerify = false;
const unknownRecord = Schema.record(Schema.string, Schema.unknown);

export const jwt = (
  args: Args
): Schema.Schema<string, Record<string, unknown>> =>
  Schema.transformResult(
    Schema.string,
    unknownRecord,
    (s) => {
      try {
        return ParseResult.success(
          decodeJwt(s, args.key, noVerify, args.algorithm)
        );
      } catch (_) {
        return ParseResult.failure(ParseResult.type(unknownRecord.ast, s));
      }
    },
    (jwt) => ParseResult.success(encodeJwt(jwt, args.key, args.algorithm))
  );
JSONFromString schema 
import * as ParseResult from "@effect/schema/ParseResult";
import * as Schema from "@effect/schema/Schema";
import { pipe } from "effect";

export type Json = boolean | number | string | null | JsonArray | JsonRecord;
export type JsonArray = ReadonlyArray<Json>;
export interface JsonRecord {
  readonly [key: string]: Json;
}

export const Json: Schema.Schema<Json> = Schema.lazy(() =>
  Schema.union(
    Schema.null,
    Schema.string,
    Schema.JsonNumber,
    Schema.boolean,
    Schema.array(Json),
    Schema.record(Schema.string, Json)
  )
);

export const JsonFromString: Schema.Schema<string, Json> =
  Schema.transformResult(
    Schema.string,
    Json,
    (s) => {
      try {
        return ParseResult.success(JSON.parse(s));
      } catch (_) {
        return ParseResult.failure(ParseResult.type(Json.ast, s));
      }
    },
    (json) => pipe(json, JSON.stringify, ParseResult.success)
  );

And then the places I use these:

JWT use case:

import * as Schema from "@effect/schema/Schema";
import { identity } from "effect";
import { jwt } from "../jwt";
import * as error from "./error";

export const UnauthenticatedHeaders = (secret: string) =>
  Schema.struct({
    "X-Authentication-Response-Code": Schema.literal("401"),
    "X-Authentication-Token": Schema.transform(
      // until https://github.com/Effect-TS/schema/pull/362 is accepted
      jwt({ key: secret, algorithm: "HS256" }),
      error.token.Token,
      (a) => a as error.token.Token,
      identity
    ),
  });

JSON file use case:

import * as Schema from "@effect/schema/Schema";
import { identity } from "effect";
import { JsonFromString } from "../json";

export const NonEmptyString = Schema.string.pipe(Schema.nonEmpty());

export const Config = Schema.struct({
  host: NonEmptyString,
  port: Schema.number.pipe(Schema.int(), Schema.positive()),
  schema: NonEmptyString,
});

export interface Config extends Schema.To<typeof Config> {}

export const Endpoints = Schema.struct({
  reader: Config,
  writer: Config,
});

export interface Endpoints extends Schema.To<typeof Endpoints> {}

export const ShardsHosts = Schema.struct({
  nonShardedDB: Endpoints,
  shards: Schema.record(Schema.string, Endpoints),
});

export interface ShardsHosts extends Schema.To<typeof ShardsHosts> {}

export const ShardsHostsFromString = Schema.transform(
  // until https://github.com/Effect-TS/schema/pull/362 is accepted
  JsonFromString,
  ShardsHosts,
  (a) => a as ShardsHosts,
  identity
);
commented

I see, thanks (btw you may want to chime in here https://discord.com/channels/795981131316985866/1136229648058028093/1136409710157901925, we are discussing whether transport serialization is a schema concern or not)

commented

Let's start fresh, given two generic codecs Schema<A, B>, Schema<C, D> we have 3 cases when we consider composition:

  1. B = C: already handled by the current compose
  2. C extends B: encoding is ok, decoding must be "forced"
  3. B extends C: decoding is ok, encoding must be "forced"

Examples:

// (string -> Array<string>) compose (Array<string> -> Array<number>)
const case1 = S.compose(S.split(S.string, ","), S.array(S.NumberFromString)) // OK
// (A U B) compose (B -> C)
const case2 = S.compose(S.union(S.null, S.string), S.NumberFromString) // ERROR
// (A -> B) compose (C U B)
const case3 = S.compose(S.NumberFromString, S.union(S.null, S.number)) // ERROR

We could have 3 compose functions:

// current
export const compose: {
  <B, C>(bc: Schema<B, C>): <A>(ab: Schema<A, B>) => Schema<A, C>
  <A, B, C>(ab: Schema<A, B>, bc: Schema<B, C>): Schema<A, C>
} = dual(
  2,
  <A, B, C>(ab: Schema<A, B>, bc: Schema<B, C>): Schema<A, C> =>
    transform(ab, bc, identity, identity)
)

export const composeForceDecoding: {
  <B, C extends B, D>(cd: Schema<C, D>): <A>(ab: Schema<A, B>) => Schema<A, D>
  <A, B, C extends B, D>(ab: Schema<A, B>, cd: Schema<C, D>): Schema<A, D>
} = dual(
  2,
  <A, B, C extends B, D>(ab: Schema<A, B>, cd: Schema<C, D>): Schema<A, D> =>
    transformResult(ab, cd, P.validateResult(from(cd)), PR.success)
  //                                  ^--- force decoding
)

export const composeForceEncoding: {
  <B extends C, C, D>(cd: Schema<C, D>): <A>(ab: Schema<A, B>) => Schema<A, D>
  <A, B extends C, C, D>(ab: Schema<A, B>, cd: Schema<C, D>): Schema<A, D>
} = dual(
  2,
  <A, B extends C, C, D>(ab: Schema<A, B>, cd: Schema<C, D>): Schema<A, D> =>
    transformResult(ab, cd, PR.success, P.validateResult(to(ab)))
  //                                             ^--- force encoding
)

and then

// (string -> Array<string>) compose (Array<string> -> Array<number>)
const case1 = S.compose(S.split(S.string, ","), S.array(S.NumberFromString)) // OK

// (A U B) compose (B -> C)
const case2 = S.composeForceDecoding(S.union(S.null, S.string), S.NumberFromString) // OK

// (A -> B) compose (C U B)
const case3 = S.composeForceEncoding(S.NumberFromString, S.union(S.null, S.number)) // OK
commented

Closed by #388