Resolve array size limitations (mostly OK now on Node 10+)
jywarren opened this issue · comments
@tech4GT noted:
Hmm, looks like we are beyond the allowed array length while making a canvas this big! We would have to look for a workaround for this. @jywarren Any ideas?
RangeError: Invalid typed array length: 13650935808
@tech4GT opened this asking how to handle larger arrays: scijs/get-pixels#46
Noting that https://github.com/scijs/get-pixels and https://github.com/scijs/save-pixels use https://github.com/scijs/ndarray for arrays.
And the above error references a Uint8Array - so a typed array. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_array_length
There seem to be lots of ndarray implementations:
https://github.com/scijs/ndarray/wiki/ndarray-module-list#core-module
Could one of those help us?
Also noting this lib designed to work around memory limits on untyped arrays:
https://www.npmjs.com/package/lds
@jywarren Just for the sake of completeness, is there some sort of upper limit on the size of the images we may get here?
haha... uhhhh i mean i've seen people try to export 50,000x50,000px images!
Woah! Yeah we'll have to think this one out for sure! Image sequencer can't generate a canvas this big in the first place, and we might have multiple of these! 🤯
haha! I am not sure one of these has successfully exported, to be honest, even in Ruby! But, one thing to think of is potentially rendering in segments - like, breaking it into max 10,000px^2 squares and rendering each square independently. It's complicated. But we can think about this kind of optimization much later.
To refocus back on the issue at hand here, what dimensions are we dealing with in more conventional maps? I would guess we want to be able to export in the range of 5000-10,000px square, max.
Alright! As long as we have only a few of these images and they have a sizeable overlap, I think we can handle it with mininal scaling! I was able to handle the ceres -- 2 map without any scaling of the input!
OK, but that's one of the smaller maps out there... see publiclab/mapknitter-exporter-sinatra#23 for some of the ones we're outputting in Ruby!
Great! Thanks I'll try those out tomorrow, along with the documentation and tests! It's pretty late here haha!
I was wondering if we can discuss the multiprocessor scaling as well right now so that I can code it up tomorrow! 😅
Put it this way - what is the max size arrays can be? MDN says:
The length property of an Array or an ArrayBuffer is represented with an unsigned 32-bit integer, that can only store values which are in the range from 0 to 2^32-1.
So that's: 4294967296 or 4,294,967,296 -- four billion or so. Not sure but imagine RGBA is 4 values, so a billion pixels, and the square root of that is 32,768 -
So is 32768x32768 roughly our canvas max?
Actually it's less for typed arrays, I wasn't able to find the exact value but 2^26 works and 2^27 doesn't. Maybe that is a machine dependent thing but that's what happened on my mac!
Aha! @jywarren this is dependent on the version of node!! I tried this with node 10 and it allows up to 2^30 !!!
I was using node 9 earlier!
Yeah I think 2^30 is a big enough canvas for our purpose!! Great!! 🎉
OK I'll mark this as temporarily resolved...