Implement `--stream`
MiSawa opened this issue · comments
To really implement this without expand JSON in memory, we need one of followings
- generator feature so that we can yield to pass value into the VM while keeping the intermediate state of the parser,
- spawn a thread that reads input, and send the result into
std::sync::mpsc::sync_channel(0)(or1so the reader thread can read one element ahead) to do a similar with above, or - implement a JSON parser on our own (and perhaps yaml as well).
Actually generator wouldn't work since for example the A of visit_seq<A>(self, mut seq: A) itself may have a lifetime that is shorter than 'de or whatever so can't return a structure including this seq thing.
Also it's hard to do send/recv, since serde_json doesn't support providing DeserializeSeed for iteration mode (Deserializer::from_reader(stdin).into_iter()).
serde_yaml on the other hand is probably fine.
Though serde_yaml doesn't seem to parse until we supply everything till EOF??? So presumably they keep everything in memory??? 😞
Here's a code to see the feasibility. https://gist.github.com/MiSawa/4e4086bd649f4639f390b93ca214f7f0
Here's an implementation. Though it uses GATs in a non-essential part so require a nightly compiler to build. https://github.com/MiSawa/xq-stream-test