10kb Item Limit expandable?
ericwilhite opened this issue · comments
The one thing holding me back from going all in on Userbase is the 10kb item limit.
Are there technical limitations behind the 10kb number? If not, any chance of raising it? Or workarounds?
The underlying DynamoDB store has a much larger limit (400kb if a quick google search is correct)
Everything about Userbase is perfect for my current side-project startup minus this limit. I'm needing to store longer form text that quickly goes over the 10kb.
I tried compressing the string using: https://rotemdan.github.io/lzutf8/demo/
It takes my test string from 25kb to around 4kb.
When I pass the compressed data it still throws the 10kb error.
If the 10kb limit is truly a hard ceiling on item size I supposed I could encode the object into a js blob and store as a File.
There doesn't seem to be an API for updating Files though. Or deleting...? Unless the deleteItem call is also used to delete a corresponding File?
Far from ideal. But perhaps a workaround. Would much rather be able to store as a standard item.
If the 10kb limit is truly a hard ceiling on item size I supposed I could encode the object into a js blob and store as a File.
We expect devs who need larger items to use uploadFile() exactly like you did! It's a pattern I've used myself for Hush Docs (a Google Docs alt built on Userbase). In Hush Docs, I also compress items myself before storing, and if >10kb, store the item as a file (source code).
There doesn't seem to be an API for updating Files though. Or deleting...? Unless the deleteItem call is also used to delete a corresponding File?
Calling uploadFile() will overwrite the file stored on an item. Will update the docs to reflect this.
It takes my test string from 25kb to around 4kb... When I pass the compressed data it still throws the 10kb error.
Our calculation is simply 2 bytes per character in the string, after calling JSON.stringify() on the item you pass. What is the length of the string you're passing in here?
Regarding the hard 10kb limit, Userbase items don't map 1-to-1 to DynamoDB items. Note how we also allow transactions where you can insert or update 10 items at a time (and their respective metadata). Which means that when you call putTransaction inserting or updating 10 items that are each 10kb, this ends up storing a 100kb item in DynamoDB. It is true this is significantly less than DynamoDB's 400kb limit. But it gives us more flexibility to expand the API for what can be inserted as a single item in DynamoDB. We settled on a 10kb limit for this reason, and expanded to allow for file storage to account for developers who need larger storage requirements. This likely won't be lifted any time soon.
Understood. I think updating the docs to help navigate this pattern is a great idea. Especially around updating/deleting the file and perhaps even a note on the insert item doc mentioning the recommended pattern for anything >10kb.
Closing issue and opening a docs specific issue.
There doesn't seem to be an API for updating Files though. Or deleting...? Unless the deleteItem call is also used to delete a corresponding File?
Calling uploadFile() will overwrite the file stored on an item. Will update the docs to reflect this.
So, does deleteItem also delete any associated files, @j-berman ? Or do you need to replace the file with an empty 0kb file first to get back that storage space?
Thanks!