Invalid log messages `response dropped`

Question

Invalid log messages `response dropped`

ayrat555 opened this issue 3 years ago · comments

Hello. Thanks for the library I'm using it in my telegram bot -https://github.com/ayrat555/el_monitorro/blob/master/src/bot/telegram_client.rs#L114

Most of the logged messages in my synchronization services look like this:

response dropped without fully consuming the response body, which may result in sub-optimal performance

It seems such messages are logged for every request, even though requests are successful.

Stephen M. Coakley · Answer 1 · Thu Aug 05 2021 03:45:15 GMT+0800 (China Standard Time)

Thanks for opening an issue! For some background on what this log message is referencing, see ReadResponseExt::consume:

By default, if a response stream is dropped before it has been completely read from, then that HTTP connection will be terminated. Depending on which version of HTTP is being used, this may require closing the network connection to the server entirely. This can result in sub-optimal performance for making multiple requests, as it prevents Isahc from keeping the connection alive to be reused for subsequent requests.

If you are downloading a file on behalf of a user and have been requested to cancel the operation, then this is probably what you want. But if you are making many small API calls to a known server, then you may want to call consume() before dropping the response, as reading a few megabytes off a socket is usually more efficient in the long run than taking a hit on connection reuse, and opening new connections can be expensive.

Note that in HTTP/2 and newer, it is not necessary to close the network connection in order to interrupt the transfer of a particular response. If you know that you will be using only HTTP/2 or newer, then calling this method is probably unnecessary.

However, in your code it seems that you are reading the whole response body using text(), so it is indeed puzzling why this log might be triggering and worth investigating.

The log message itself is also a bit spartan, so it might be a good idea to update the log message as well to give a little bit more detail.

As a side-note, text() is going to give you sub-optimal performance here. Since JSON is always UTF-8, it is redundant to call text() to parse the response body into a string just to parse it as JSON using serde_json::from_str immediately afterward. Instead, it is better to parse the raw bytes as JSON directly, which can be done by reading the bytes into a Vec<u8>, or better:

let parsed_result = serde_json::from_reader(response.body_mut());

Or even better yet, if you enable the json crate feature you can just do

let parsed_result = response.json();

which is a shortcut for the former.

Ayrat Badykov · Answer 2 · Thu Aug 05 2021 14:29:28 GMT+0800 (China Standard Time)

thanks, I'll do as you suggested later this week.

Stephen M. Coakley · Answer 3 · Fri Aug 06 2021 03:05:26 GMT+0800 (China Standard Time)

I've opened a PR to at least improve this log message to be more useful: #335

Ayrat Badykov · Answer 4 · Sat Aug 07 2021 20:24:50 GMT+0800 (China Standard Time)

after setting let parsed_result = serde_json::from_reader(response.body_mut()) I'm still seeing such error message

Ayrat Badykov · Answer 5 · Sun Aug 08 2021 02:59:06 GMT+0800 (China Standard Time)

I started seeing a new error I haven't seen before when I changed from text :

EOF while parsing a value

I had to revert the new commit

Stephen M. Coakley · Answer 6 · Sun Aug 08 2021 10:16:19 GMT+0800 (China Standard Time)

Hmm, it seems like the server is aborting the response early (or we are aborting the connection early somehow). I haven't seen something like this before. I definitely want to get to the botton of this; hopefully I'll have some time available this week to look deeper!

Using text or not shouldn't have any impact on EOF behavior, that was just an unrelated suggestion on how to improve your code, but the fact that it makes a difference here is very strange.

Here's what could help me diagnose this. Could you turn on debug logs for Isahc and capture those when this happens? Isahc has lots of useful debug logs that can help troubleshoot networking problems like this. (If you're using env_logger that might look like RUST_LOG=isahc=debug). If you could paste the output of some of those logs when this happens that would be very helpful to me!

Ayrat Badykov · Answer 7 · Sun Aug 08 2021 12:20:05 GMT+0800 (China Standard Time)

Using text or not shouldn't have any impact on EOF behavior, that was just an unrelated suggestion on how to improve your code, but the fact that it makes a difference here is very strange.

May it be happening because I'm trying to read the response body twice? Can it be consumed after the first read

Ayrat Badykov · Answer 8 · Sun Aug 08 2021 12:35:46 GMT+0800 (China Standard Time)

You can find logs in this gist - https://gist.github.com/ayrat555/d725da7c2eb911b71f195407f69f8caa

Keep in mind I'm running 10 workers for fetching feeds.

Stephen M. Coakley · Answer 9 · Sun Aug 08 2021 12:46:50 GMT+0800 (China Standard Time)

Thanks! Those logs will be very helpful; I'll keep digging and see if I can figure out the problem.

May it be happening because I'm trying to read the response body twice? Can it be consumed after the first read

Ah, my bad. I did not realize you were trying to read the response body multiple times. No, the body can only be consumed once since it is a "live" socket stream, if you will. In that case you might want to read the body into memory (without trying to parse as UTF-8) and have have Serde parse from that buffer. Something more like:

let mut bytes = Vec::new();
response.copy_to(&mut bytes)?;

serde_json::from_slice(&bytes);

Ayrat Badykov · Answer 10 · Wed Aug 11 2021 02:46:25 GMT+0800 (China Standard Time)

@sagebind you can use this rss feed as an example https://tynan.com/feed/ . isahc always logs response dropped for it

Stephen M. Coakley · Answer 11 · Wed Aug 11 2021 05:35:21 GMT+0800 (China Standard Time)

I'm not able to reproduce the log message with that URL alone; I get no such message with the simple program below (Isahc 1.4.0, macOS):

use isahc::prelude::*;

fn main() -> Result<(), isahc::Error> {
    env_logger::init();

    let mut response = isahc::get("https://tynan.com/feed/")?;

    response.text()?;

    Ok(())
}

Does this emit the log message for you? You're running inside Docker right? Rest assured, I'm still actively looking into this issue. I'm just having difficulty reproducing the problem.

Is the only problem the log message? Or does it fail to parse the response body also when this occurs? If this log message was emitted, I would expect you to not receive the entire response body if you are indeed reading from it.

Ayrat Badykov · Answer 12 · Wed Aug 11 2021 14:40:18 GMT+0800 (China Standard Time)

can you please try sending requests like https://github.com/ayrat555/el_monitorro/blob/master/src/sync/reader.rs#L62
This function is used to fetch feeds

Stephen M. Coakley · Answer 13 · Thu Aug 12 2021 11:47:11 GMT+0800 (China Standard Time)

Thanks, I created a sample program more closely aligning with your usage:

spurious_abort_logs.rs

use std::{error::Error, io, time::Duration};

use isahc::{Request, RequestExt, config::{Configurable, RedirectPolicy}};

fn main() -> Result<(), Box<dyn Error>> {
    env_logger::init();

    read_url("https://tynan.com/feed/")?;

    Ok(())
}

pub fn read_url(url: &str) -> Result<Vec<u8>, String> {
    let client = match Request::get(url)
        .timeout(Duration::from_secs(5))
        .header("User-Agent", "el_monitorro/0.1.0")
        .redirect_policy(RedirectPolicy::Limit(10))
        .body(())
    {
        Ok(cl) => cl,
        Err(er) => {
            let msg = format!("{:?}", er);

            return Err(msg);
        }
    };

    match client.send() {
        Ok(mut response) => {
            let mut writer: Vec<u8> = vec![];

            if let Err(err) = io::copy(response.body_mut(), &mut writer) {
                let msg = format!("{:?}", err);

                return Err(msg);
            }

            Ok(writer)
        }
        Err(error) => {
            let msg = format!("{:?}", error);

            Err(msg)
        }
    }
}

However, running it I don't get the log message in question:

Log output

$ env RUST_LOG=isahc=debug cargo run --example spurious_abort_logs
[2021-08-12T03:44:43Z DEBUG isahc::agent] agent_thread; id=0
[2021-08-12T03:44:43Z DEBUG isahc::agent] agent took 494.4µs to start up 
[2021-08-12T03:44:43Z DEBUG isahc::client] send; method=GET uri=https://tynan.com/feed/
[2021-08-12T03:44:43Z DEBUG isahc::handler] handler
[2021-08-12T03:44:43Z DEBUG isahc::handler] handler; id=0
[2021-08-12T03:44:43Z DEBUG isahc::handler]   Trying 88.198.208.99:443... 
[2021-08-12T03:44:44Z DEBUG isahc::handler] Connected to tynan.com (88.198.208.99) port 443 (#0) 
[2021-08-12T03:44:44Z DEBUG isahc::handler] ALPN, offering h2 
[2021-08-12T03:44:44Z DEBUG isahc::handler] ALPN, offering http/1.1 
[2021-08-12T03:44:44Z DEBUG isahc::handler] successfully set certificate verify locations: 
[2021-08-12T03:44:44Z DEBUG isahc::handler]  CAfile: /usr/lib/ssl/certs/ca-certificates.crt 
[2021-08-12T03:44:44Z DEBUG isahc::handler]  CApath: /usr/lib/ssl/certs 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (OUT), TLS handshake, Client hello (1): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (IN), TLS handshake, Server hello (2): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (IN), TLS handshake, Certificate (11): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (IN), TLS handshake, CERT verify (15): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (IN), TLS handshake, Finished (20): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (OUT), TLS handshake, Finished (20): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 
[2021-08-12T03:44:44Z DEBUG isahc::handler] ALPN, server accepted to use http/1.1 
[2021-08-12T03:44:44Z DEBUG isahc::handler] Server certificate: 
[2021-08-12T03:44:44Z DEBUG isahc::handler]  subject: CN=tynan.com 
[2021-08-12T03:44:44Z DEBUG isahc::handler]  start date: Jun 11 06:00:54 2021 GMT 
[2021-08-12T03:44:44Z DEBUG isahc::handler]  expire date: Sep  9 06:00:53 2021 GMT 
[2021-08-12T03:44:44Z DEBUG isahc::handler]  subjectAltName: host "tynan.com" matched cert's "tynan.com" 
[2021-08-12T03:44:44Z DEBUG isahc::handler]  issuer: C=US; O=Let's Encrypt; CN=R3 
[2021-08-12T03:44:44Z DEBUG isahc::handler]  SSL certificate verify ok. 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 
[2021-08-12T03:44:44Z DEBUG isahc::handler] old SSL session ID is stale, removing 
[2021-08-12T03:44:44Z DEBUG isahc::handler] Mark bundle as not supporting multiuse 
[2021-08-12T03:44:44Z DEBUG isahc::handler] Connection #0 to host tynan.com left intact

Does this sample program emit the log in question in your environment? It may also help to share the output of isahc::version() which includes the versions of dependencies; it could be a version of OpenSSL or curl being used.

Ayrat Badykov · Answer 14 · Mon Aug 16 2021 21:18:28 GMT+0800 (China Standard Time)

I'm sorry for the late response. I'll definitely take a look this week.

btw yes, it seems it's related to os / deps / vps. because locally I can't these messages.
also, it seems they don't cause any issues, they just clatter log messages

Stephen M. Coakley · Answer 15 · Wed Aug 18 2021 12:03:07 GMT+0800 (China Standard Time)

Can you give some more details on the runtime environment? Perhaps we can nail down the underlying cause.

In addition, please try upgrading to version 1.4.1 just released to see if it improves things, which fixes some erroneous emissions of that log message around HTTP/2 usage.

Ayrat Badykov · Answer 16 · Thu Aug 19 2021 02:44:03 GMT+0800 (China Standard Time)

@sagebind the version is

isahc/1.4.1 (features:default,encoding_rs,http2,mime,static_curl,text_decoding) libcurl/7.78.0-DEV OpenSSL/1.0.2n zlib/1.2.11 nghttp2/1.33.90

it started printing longer messages after the update:

Aug 18 20:50:41 vps695005 bash[1336]:     Aborting a response without fully consuming the response body can result in sub-optimal performance. See https://github.com/sagebind/isahc/wiki/Connection-Reuse#closing-connections-early.

Ayrat Badykov · Answer 17 · Sat Aug 21 2021 15:40:58 GMT+0800 (China Standard Time)

I set error log level for isahc

RUST_LOG=info,isahc=error

Stephen M. Coakley · Answer 18 · Mon Aug 23 2021 05:20:07 GMT+0800 (China Standard Time)

it started printing longer messages after the update:

Yes, in addition to trying to avoid spurious printing of the message I also extended it to be more clear as to what the message means.

I set error log level for isahc

RUST_LOG=info,isahc=error

That also works, though I'm still not sure why it is being printed in your scenario. The message represents a real problem, but it at least doesn't seem like your program has the problem in question.