Data coercion when writing to postgres JSONB column not working

Question

Data coercion when writing to postgres JSONB column not working

alexanderfrankel opened this issue 5 years ago · comments

Describe the bug

I am getting

ROM::SQL::DatabaseError (PG::DatatypeMismatch: ERROR:  column "value" is of type jsonb but expression is of type integer)

when attempting to write to a postgres JSONB column using both Relation#insert and Relation#command(:create).

To Reproduce

Create a table called attributes in your db in postgres with a name::VARCHAR column and a value::JSONB column.
Set up ROM::Relation as such:

class AttributesRelation < ROM::Relation[:sql]
  gateway :default

  schema(:attributes, infer: true)
end

Attempt both of the following:

attributes = ROM.env.relations[:attributes]
attributes.insert(name: "valid name", value: 1)

attributes = ROM.env.relations[:attributes]
attributes.command(:create).call(name:"valid name", value: 1)

Expected behavior

I expected the integer value to automatically be coerced into valid JSON before attempting to write to the database. Effectively:

attributes.insert(name: "valid name", value: 1.to_json)

attributes = ROM.env.relations[:attributes]
attributes.command(:create).call(name: "valid name", value: 1.to_json)

(these both work btw)

Your environment

Affects my production application: NO
Ruby version: 2.6.2
Rails version: 5.2.3
OS: MacOS Mojave Version 10.14.3

Note
Please let me know if coercing data into the JSON format before inserting said data is not an intended feature. I have worked around this by creating a changeset which coerces the data in the value key into valid JSON before attempting insert.

This works:

class NewAttributeChangeset < ROM::Changeset::Create
  map do
    map_value :value, ->(v) { v.to_json }
  end
end

attributes.changeset(NewAttributeChangeset, { name: "valid name", value: 1 }).commit

Any advice you have on best practices around this is much appreciated. Thank you for taking the time to investigate.

Nikita Shilnikov commented 5 years ago

See #333

Peter Solnica · Answer 1 · Sat Apr 20 2019 16:32:22 GMT+0800 (China Standard Time)

This doesn't look like a bug to me. The built-in JSONB type coerces input using Sequel.pg_jsonb which does not convert integers to strings like to_json monkey-patch. Looks like PG can store a plain string in a JSONB column but not an integer. To be honest, this looks like a special case, people typically store hashes or arrays in JSONB columns.

To make it work you can define a custom constructor that would handle to_json coercion:

schema(:attributes, infer: true) do
  attribute :value, Types::PG::JSONB.constructor(&:to_json)
end

Does it make sense?

Nikita Shilnikov · Answer 2 · Sat Apr 20 2019 16:47:07 GMT+0800 (China Standard Time)

@solnic it must be Types::PG::JSONB.prepend(&:to_json) which is in order only available in the recent dry-types releases.

Alexander Frankel · Answer 3 · Tue Apr 23 2019 05:33:53 GMT+0800 (China Standard Time)

@solnic @flash-gordon Thanks for getting back to me so quickly.

I am confused when you say: "Looks like PG can store a plain string in a JSONB column but not an integer." According to the PG docs, the following JSON primitive types can be stored in a JSONB column: string, number, boolean, null. https://www.postgresql.org/docs/9.4/datatype-json.html

Are you suggesting that the problem is Sequel does not support automatically coercing an integer into JSON?

Thanks again for your help and the custom constructor suggestion.

Peter Solnica · Answer 4 · Tue Apr 23 2019 17:16:27 GMT+0800 (China Standard Time)

@alexanderfrankel this is interesting, I just tried plain Sequel:

[8] pry(main)> ds.insert(title: "A test", meta: Sequel.pg_jsonb(1))
Sequel::DatabaseError: PG::DatatypeMismatch: ERROR:  column "meta" is of type jsonb but expression is of type integer
LINE 1: ... INTO "books" ("title", "meta") VALUES ('A test', 1) RETURNI...
                                                             ^
HINT:  You will need to rewrite or cast the expression.
from /Users/solnic/.gem/ruby/2.6.3/gems/sequel-5.19.0/lib/sequel/adapters/postgres.rb:152:in `async_exec'
Caused by PG::DatatypeMismatch: ERROR:  column "meta" is of type jsonb but expression is of type integer
LINE 1: ... INTO "books" ("title", "meta") VALUES ('A test', 1) RETURNI...
                                                             ^
HINT:  You will need to rewrite or cast the expression.

Which gives a pretty clear error message that jsonb was expected but integer was give. So, I'm confused 😄

Alexander Frankel · Answer 5 · Thu Apr 25 2019 03:50:15 GMT+0800 (China Standard Time)

@solnic From the Sequel docs: "The pg_json extension adds support for Sequel to handle PostgreSQL's json and jsonb types. It is slightly more strict than the PostgreSQL json types in that the object returned should be an array or object (PostgreSQL's json type considers plain numbers strings, true, false, and null as valid). Sequel will work with PostgreSQL json values that are not arrays or objects, but support is fairly limited and the values do not roundtrip."

https://sequel.jeremyevans.net/rdoc-plugins/files/lib/sequel/extensions/pg_json_rb.html

This could be why.

Peter Solnica · Answer 6 · Thu Apr 25 2019 15:48:59 GMT+0800 (China Standard Time)

OK so the big question is what do we want to do with it? I actually find it weird that json can be a hash or an integer but it is a standard so if we wanted to respect POLS we should support all valid json values.

Alexander Frankel · Answer 7 · Mon Apr 29 2019 22:50:42 GMT+0800 (China Standard Time)

@solnic Agreed. I have started a discussion with @jeremyevans about our specific use-case and adding support to the Sequel lib to support JSON primitive types in JSONB columns.

He has indicated that there is currently "no object that wraps json/jsonb primitives and therefore no way of saving such an object back to the database."

https://groups.google.com/forum/#!topic/sequel-talk/JDwBDbEPxM8

Peter Solnica · Answer 8 · Tue Apr 30 2019 00:26:51 GMT+0800 (China Standard Time)

@alexanderfrankel OK we'll see what happens on the sequel side then. I'll leave this open until this functionality gets into Sequel and we can verify if it works over here too.

Alexander Frankel · Answer 9 · Tue Apr 30 2019 21:40:37 GMT+0800 (China Standard Time)

@solnic Sounds good to me. I'll keep you updated. Thanks for your help!

Jeremy Evans · Answer 10 · Sat May 11 2019 05:28:24 GMT+0800 (China Standard Time)

I've just pushed a commit to Sequel that allows wrapping JSON primitives: jeremyevans/sequel@035b214

Please test this out in the next two weeks and let me know whether you think there should be any changes before I finalize the API.

Alexander Frankel · Answer 11 · Tue May 14 2019 01:55:14 GMT+0800 (China Standard Time)

@solnic Did you see @jeremyevans 's comment above?

All looks good on my side. Does the api look good on the rom-sql side?

Nikita Shilnikov · Answer 12 · Tue May 14 2019 02:17:04 GMT+0800 (China Standard Time)

@alexanderfrankel I will test the changes tomorrow

Nikita Shilnikov · Answer 13 · Wed May 15 2019 05:34:40 GMT+0800 (China Standard Time)

I did some quick tests, the API seems to be functional, I'll try to come up with a PR tomorrow. Thank you @jeremyevans
As far as I can see, we only need to change this

        JSON = Type('json') do
          (SQL::Types::Array | SQL::Types::Hash)
-            .constructor(Sequel.method(:pg_json))
+            .constructor(Sequel.method(:pg_json_wrap))
            .meta(read: JSONRead)
        end

        JSONB = Type('jsonb') do
          (SQL::Types::Array | SQL::Types::Hash)
-            .constructor(Sequel.method(:pg_jsonb))
+            .constructor(Sequel.method(:pg_jsonb_wrap))
            .meta(read: JSONRead)
        end

wrap_json_primitives is OK to be false for us since we unwrap json(b) values when reading anyway.