Question around JSON schema compatibility
mrmckeb opened this issue · comments
First, great library - thanks!
I see support for JSON schema properties in the documentation, but I wondered if there is a way to restrict a struct so that it is compatible with JSON schema?
Of course there are other libraries that can validate from JSON schema, but it would be nice to be able to use a single validation library for both JSON validation and more complex validation.
+1
Not sure I'm following, what's the use case? Convert a JSON Schema to a Schema? Extract a JSON Schema from a Schema? Could you please expand on this
a way to restrict a struct so that it is compatible with JSON schema
In my case, we read a user-supplied data structure, which may be provided via JSON. Ideally, I'd like to be able to provide a JSON schema (enabling tools like VSCode to highlight issues, etc). But also, I want to validate the input - and I'd prefer not to have both a Schema and a JSON schema.
So I think there are maybe a few parts to this request:
- Could this library have a Schema type that is restricted to create a JSON-compatible Schema?
- Could the library parse to and/or from JSON Schema? This could be a build step for performance reasons.
Use-cases include user config files and data transfer (REST, etc).
Transformation to and from JSON schema is of interest to me.
I think this library is a great sort of mini language to write type definitions with. But I commonly need to share types with other people working in other languages (or even with people who may prefer “regular Typescript types”).
Same goes for other people authoring types. Typically they provide JSON schema if anything, which helps. I’ve found a lot of value in having decoders and arbitraries for types that are at the boundaries of a system, so I always end up converting from JSON schema or sample data, programmatically if I can, otherwise by hand.
Are there any plans on the roadmap for this or anyone in the Effect community who may already be working on it?
Edit:
May be a start or interest here:
https://github.com/sukovanej/effect-http/tree/master
OpenAPI schema format overlaps with JSON schema.
Are you looking for a port of these libraries to effect-ts? If so, I agree, they would be very useful. I don't know if this is within the scope of the core project though.
https://www.npmjs.com/package/ts-to-zod
https://www.npmjs.com/package/json-schema-to-zod
https://www.npmjs.com/package/openapi-to-zod
Are you looking for a port of these libraries to effect-ts?
Yes. And the other way also (effect/schema -> JSON schema, Avro, python, go…)😊
I’ve written some “io-ts to X” in the past (Typescript, JSON schema, Spark/Avro schema…). Unfortunately I committed it all to repos that belong to my former employer with the promise to open source and it still remains closed 😕. I may be able to help with something similar for effect/schema. Just looking to get up to speed with what’s being done/planned already first.
I read through the code and see there is support for JSON schema attributes. Looking to learn more about how these are used currently and future plans for them and the project in general (I seem to remember a roadmap for schema somewhere but I can’t find it).
The only work that I've seen in the x -> @effect/schema direction so far is this lib but I don't know if it's actively being worked on. I've thought about forking them myself(it would probably be better to make them a single cli tool vs several) but I haven't needed something like this badly enough yet to justify the amount of work needed. jaja. Going from @effect/schema -> x is quite a bit easier though sense it just requires interpreting the ast vs parsing+codegen. @effect/schema -> ts & json schema already kind of exist but are currently buried under the test/compilers directory (and @effect/schema -> Spark/Avro schema should be pretty straightforward). I don't know what @gcanti's plans to make them available from the lib are though.
I think they should be external packages, as it happens in the zod ecosystem
That is my intuition as well. Although, I think things like hooks might be a bit awkward to implement on in a third party library though. How would you implement an arbitrary compiler, for example, if the library didn't already attach an [ArbitraryHook]? I realize that arbitrary will probably be part of the core lib, but other compilers will likely need custom hooks. Same with a Json schema compiler, it would likely be difficult if the library didn't already attach the jsonSchema annotation.
The fundamental aspect is to put on refinements enough information to make them recognizable and usable by compilers such as Arbitrary (as it is already doing #256). For the format of this information, it makes sense to use something already known rather than something invented on the spot, so using portions of JSON Schema seems sensible.
Now that we have annotations on the refinements, whether the compiler that uses them is within effect/schema or an external package doesn't matter too much.
@gcanti, I've been giving this more thought over the last week and was thinking to build a simple mapper based around the AST. How stable is that API/implementation?
For example:
const { ast } = S.struct({
hello: S.literal('world'),
});
// {
// "_tag": "TypeLiteral",
// "propertySignatures": [
// {
// "name": "hello",
// "type": {
// "_tag": "Literal",
// "literal": "world",
// "annotations": {}
// },
// "isOptional": false,
// "isReadonly": true,
// "annotations": {}
// }
// ],
// "indexSignatures": [],
// "annotations": {}
// }
const jsonSchema = someFn(ast);
{
// "type": "object",
// "properties": {
// "hello": {
// "type": "string",
// "oneOf": [
// {
// "enum": ["world"]
// }
// ]
// }
// },
// "required": [
// "hello"
// ]
// }If you are looking for a schema -> json compiler there is one in the /test/compilers directory btw. I've also re-exported it here.
@mrmckeb probably there will be some changes (you can get an idea here https://github.com/Effect-TS/schema/blob/poc/split-schema/src/AST.ts), in particular to Refinement and Transform, but they shouldn't affect a mapper to JSON Schema
That's a great find, thanks @jessekelly881 - and for the work on that repo.
@gcanti and @jessekelly881, how would you feel about exposing the JSON compiler from this package directly? I'd be happy to contribute any fixes as we make use of it.
@mrmckeb ok we are considering exposing a JSON Schema compiler from this package. The starting point could be the one in the test folder (though it was kind of an experiment to help the development of the core)
I think exposing it here would probably be a good idea. Especially considering how tightly linked this lib is with json-schema. Exposing it would also encourage PRs that make it more robust. I'd be happy to create a PR to expose it (w/ some slight modifications).
Thanks! I plan to take a look at this at the end of this week.
I've started looking into this. I see some properties are already mapped to jsonSchema, like minimum and maximum for a number type.
I think it would make sense to also support title and description, unless there are concerns around that? We could either:
- Map these to the
jsonSchemaattribute, and then get those values when we build the JSON schema. - Just look for those two properties specifically when building the JSON schema.
I think I prefer the first of those two, but not sure on the implementation yet. I'll dig deeper.
@mrmckeb title and description are currently stored as separated annotations:
import * as S from '@effect/schema/Schema'
const schema = S.number.pipe(S.title('my title'), S.description('my description'))
console.log('%o', schema.ast)
/*
Output:
{
_tag: 'NumberKeyword',
annotations: {
'@effect/schema/TitleAnnotationId': 'my title',
'@effect/schema/DescriptionAnnotationId': 'my description'
}
}
*/Working on exposing the JSON Schema compiler... #493