Prevent buffering of the output?

[Mikaël Salson]

Hello,

I'm outputting quite huge JSON files under the following format:

{
    "data": [
        {
            "position": 0,
            "infos": [
                {
                    "position": 1,
                    "index": 3.4
                },
                {
                    "position": 15,
                    "index": 76.1
                },
                ....
          }
     ]
}

I can have tens of thousands of entries in one infos array and I can have tens of thousands of objects in data.

I'm writing the JSON on stdout using a PrettyWriter object directly (with writer.Key, writer.StartArray, writer.StartObject, …). I was surprised to see the output generation consume several GBs of RAM. It appeared that rapidjson seems to buffer a first complete object in data before outputting it, because I see a delay between when the program should start outputting and when the program actually outputs something on stdout.
How to prevent such a buffering to occur and to make rapidjson output the data directly?

I'm surprised by such a behaviour because rapidjson doesn't wait for the end of the data array or of the initial object before outputting anything. So why does it wait for the complete object when it is in the data array?

Thanks!

[Milo Yip]

The writer does not buffer internally. It just uses the output stream to write characters.
Please check which output stream type your code is using with.
Or try to post a simple, reproducible example.

[Mikaël Salson]

Hi, thanks for your feedback.
Indeed the problem was not coming from rapidjson.
My apologies for the disturbance.