Proposal: index arrays instead of newline delimited "streams"
fergiemcdowall opened this issue · comments
Personally although I like the promise of streams,I think that they are becoming an increasingly esoteric way of reading large blobs of data, and are creating a barrier to people who just want to get going with Norch.
Whould anybody be unhappy if we changed the /add endpoint to accept arrays instead of streams?
I love the indexing part of the stream API on search-index, but since we're talking about the server-side here, I don't think it's a problem.
But maybe be able to add streams as well, even if through some tools. Today search-index has add and concurrentAdd (for arrays). You're thinking of swapping to the concurrentAdd? The memory issues of indexing went away with streams, so for large data-sets we should maybe keep a way to do this for search-index, while the norch-endpoints could be the user friendlier version?
@eklem I feel like it would be better if Norch's add accepted standard JSON.parse-able objects as default with the option to specify more advanced inputs such as streams as an add-on, or preprocessing stage
Yes, sounds good to have add as the standard choice with easy-to-use arrays as input. Also have some dirty tricks to parse streams in the browser when querying a norch-instance. Could be nice to get rid of those too =)
Upcoming version is de-streamified