Occasional panic: 'socket configured in non-blocking mode'

[Sergey "Shnatsel" Davidoff]

I got the following panic on 87 out 1,000,000 invocations:

thread 'main' panicked at 'socket configured in non-blocking mode', /home/sdavydov/attohttpc/src/streams.rs:56:51
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:77
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1057
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:195
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:215
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:472
  11: std::panicking::begin_panic
  12: attohttpc::streams::BaseStream::connect
  13: attohttpc::request::RequestBuilder<B>::send
  14: attohttpc_test::main
  15: std::rt::lang_start::{{closure}}
  16: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  17: std::panicking::try::do_call
             at src/libstd/panicking.rs:296
  18: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:79
  19: std::panicking::try
             at src/libstd/panicking.rs:272
  20: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  21: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  22: std::rt::lang_start
  23: __libc_start_main
  24: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Code:

use std::env;
use std::time::Duration;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let url = format!("http://{}", env::args().skip(1).next().expect("No URL provided"));

    println!("GET {}", url);

    const time: std::time::Duration = Duration::from_secs(5);
    let res = attohttpc::get(url)
        .connect_timeout(time)
        .read_timeout(time)
        .send();

    if res.is_err() {
        println!("Library error: {:?}", res);
    }

    println!("\nDone.");
    if res.is_err() {
        Err(Box::new(std::io::Error::new(std::io::ErrorKind::Other, "Lib error")))
    } else {
        Ok(())
    }
}

Invoked with

cut -d ',' -f 3 majestic-million-31dec2019.csv | parallel -j 30 /path/to/binary/with/the/above/code

majestic-million.csv obtained here
Tested on Ubuntu 18.04, attohttpc commit aeb2bd2

[Simon Bernier St-Pierre]

Hmm that is pretty weird. I'll look into this.

[Simon Bernier St-Pierre]

The error probably comes from this line. I'm not sure what makes the tls connector return WouldBlock.

The native_tls docs indicate that it returns a MidHandshakeTlsStream, not sure what's up with that.

[Simon Bernier St-Pierre]

Can you add a call to set_nonblocking(false) before the TLS connector performs the handshake and run your test again? Right before this line?

[Sergey "Shnatsel" Davidoff]

Sure, I'll rerun the test with that patch. Can't guarantee a timeframe sadly, but should be within a few days.

[Simon Bernier St-Pierre]

I'll try to run your test since you've provided the csv and all.

[Simon Bernier St-Pierre]

I ran the test program 14,000 times and got 90 non-blocking panics, even with the fix. Will need more investigation.

[Simon Bernier St-Pierre]

This is released in 0.10.1. Thanks for running the library through a test like this. Really helps improve the quality. I'm sure there's a few other bugs in those million websites.

[Sergey "Shnatsel" Davidoff]

Nothing crashing, but probably quite a few errors, yeah. With 50 concurrent requests this completes in under 8 hours, so you can run the test overnight by yourself. Doesn't require a particularly beefy machine or a very fast connection, even a laptop will do.

It's better done via a VPN though because some ISPs frown upon tons and tons of HTTPS connections.

[Simon Bernier St-Pierre]

I noticed that I was losing my network connection while running this test, sometimes my network was getting blocked. I just assumed it was either my VM being underpowered or my distro being unstable, I'm on Debian sid in that VM. It seemed to be fine on the host machine. Who knows.