denodrivers / postgres

PostgreSQL driver for Deno

Home Page:https://denodrivers.github.io/postgres

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[feature] Support streaming of buffer/bytea parameters

vicary opened this issue · comments

My app manages deep learning models and it's metadata, the models are sized around 50MB - 100MB.

I am using CockroachDB for storage because it has no hard limit on their BYTES / BYTEA / BLOB columns, it is handy when the binary data is infrequently accessed. The binary data is for records and statistical study, they are usually written once and rarely read.

I know it is not the best practice out there, but I don't want to setup object storage just for them.

With the current version, when the buffer goes more than ~30MB V8 will start throwing heap allocation failures, I believe denoland/rusty_v8#427 is not enough to catch up with the sudden memory spike.

Does it make sense if the parameters can be transferred via ReadableStream and WriteableStream?