Sonic Injest Closes Unexpectedly

Question

Sonic Injest Closes Unexpectedly

mhudecheck opened this issue 5 years ago · comments

I'm using the node.js sonic driver to injest a large (100M+) document collection from MongoDB. After ~60 seconds, the connection automatically drops with the following error:

(INFO) - closing channel thread with traceback: Connection reset by peer (os error 104)

I've tried implementing different timeout lengths on my injest stream without any luck. Is there an automatic timeout setting in node-sonic-channel?

Valerian Saliou · Answer 1 · Sat Apr 27 2019 17:26:01 GMT+0800 (China Standard Time)

Hello!

Can you share which Sonic version you’re using, also if it’s running in the Docker image.

As well, which version of node-sonic-channel do you run?

And last, can you import with Sonic in warn log level and share all Sonic console logs of a failed import?

I’ve ran a huge import for Crisp (400M+) objects over ~1 day with the same setup and I’ve not seen this.

mhudecheck · Answer 2 · Sat Apr 27 2019 23:32:10 GMT+0800 (China Standard Time)

I'm running Sonic 1.18 and sonic-channel 1.2.5. Sonic and Mongodb are running on a physical server.
The output I get with debug enabled in the config file is:

(INFO) - took 1ms/1408us/1408102ns to process channel message

(INFO) - closing channel thread with traceback: Connection reset by peer (os error 104)

thread 'sonic-channel-client' panicked at 'closing channel', /home/***/.cargo/registry/src/github.com-1ecc6299db9ec823/sonic-server-1.1.8/src/channel/handle.rs:181:21

(DEBUG) - running a tasker tick...

(DEBUG) - scanning for kv store pool items to janitor

Valerian Saliou · Answer 3 · Sun Apr 28 2019 00:11:57 GMT+0800 (China Standard Time)

This looks like a TCP issue, and not an issue related to Sonic itself. Can you try using Sonic from Git (master branch) and compile it by hand, and run this version instead? We’ve fixed a ton of issues, especially one related to a deadlock, so that may fix yours.

…

-- Valerian Saliou CTO at Crisp

On 27 Apr 2019, at 5:32 PM, mhudecheck ***@***.***> wrote: I'm running Sonic 1.18 and sonic-channel 1.2.5. Sonic and Mongodb are running on a physical server. The output I get with debug enabled in the config file is: (INFO) - took 1ms/1408us/1408102ns to process channel message (INFO) - closing channel thread with traceback: Connection reset by peer (os error 104) thread 'sonic-channel-client' panicked at 'closing channel', /home/***/.cargo/registry/src/github.com-1ecc6299db9ec823/sonic-server-1.1.8/src/channel/handle.rs:181:21 (DEBUG) - running a tasker tick... (DEBUG) - scanning for kv store pool items to janitor — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

mhudecheck · Answer 4 · Fri May 03 2019 16:33:16 GMT+0800 (China Standard Time)

This turned out to be a problem with the MongoDB in-memory engine (specifically, Percona's open source version) and not node-sonic. Switching back to WiredTiger fixed the issue.