Getting 400 Bad Request on second call to same host

Question

Getting 400 Bad Request on second call to same host

emmettprexus opened this issue 2 years ago · comments

emmettprexus commented 2 years ago

400 Bad Request

First time I request data from my server it works. If I call it again, I get a 400 Bad Request.

The funny thing is it doesn't happen when connecting to the sample URLs.

Steps to Reproduce

Use your own examples but change the URL you are GET'ing from to https://yocal.dk/test.txt

Expected behavior

Same response both times

Actual behavior

First request returns 200, second request returns 400

Log

First request:

[AHTTPS] open( GET , url = https://yocal.dk/test.txt
[AHTTPS] open: connecting to hostname = yocal.dk:443
[AHTTPS] _client->connecting to yocal.dk , 443
[AHTTPS] client.connect OK to yocal.dk , 443
[AHTTPS] _onError handler SSL error = OK

**************************************
Response code: 200
Response text: Test :)
**************************************

Second request

[AHTTPS] open( GET , url = https://yocal.dk/test.txt
[AHTTPS] open: already connected

**************************************
Response code: 400
Response text: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request - Invalid Hostname</h2>
<hr><p>HTTP Error 400. The request hostname is invalid.</p>
</BODY></HTML>

**************************************

Information

Could it be related to: #12 ?

emmettprexus · Answer 1 · Thu Nov 10 2022 01:54:18 GMT+0800 (China Standard Time)

I rolled back to v2.1.2 and everything works as expected!

The logging output is also vastly different with the same log level enabled. Try and compare the two:

[AHTTPS] open( GET , url = https://yocal.dk/test.txt
[AHTTPS] _parseURL(): scheme+host HTTPS:// yocal.dk 
[AHTTPS] _parseURL(): port+path+query 443 /test.txt 
[AHTTPS] open: connecting to hostname = yocal.dk:443
[AHTTPS] _connect()
[AHTTPS] _client->connecting to yocal.dk , 443      
[AHTTPS] client.connect OK to yocal.dk , 443        
[AHTTPS] _connect() true
[AHTTPS] send()
[AHTTPS] _buildRequest()
[AHTTPS] _HTTPmethod = 0
[AHTTPS] GET  /test.txt  HTTP/1.1

[AHTTPS] write _HTTPmethodStringwithSpace : GET
[AHTTPS] write path : /test.txt
[AHTTPS] write query
[AHTTPS] write HTTP/1.1
[AHTTPS] To write hdr
[AHTTPS] hdr->name
[AHTTPS] hdr->value
[AHTTPS] host : yocal.dk:443

[AHTTPS] Write hdr done
[AHTTPS] _buildRequest() done
[AHTTPS] _send(), _request->available = 45
[AHTTPS] !connected yet
[AHTTPS] _onError handler SSL error = 0
[AHTTPS] _onError handler SSL error = OK
[AHTTPS] _onConnect handler
[AHTTPS] _setReadyState : 1
[AHTTPS] _send(), _request->available = 45
[AHTTPS] Done supply
[AHTTPS] *send 74
[AHTTPS] ========= _onData handler =======
[AHTTPS] Vbuf = HTTP/1.1 200 OK
Content-Type: text/plain
Last-Modified: Wed, 09 Nov 2022 16:59:11 GMT
Accept-Ranges: bytes
ETag: "3d3f4c975cf4d81:0"
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Date: Wed, 09 Nov 2022 17:49:25 GMT
Content-Length: 7

Test :)I�U�4␋��=؝�n␓␛t]�:␃С␐�?�0�+␛�X␟��␐8������:�␞���4��d!�u1E[�␂��␎�␝�@�@0␅␆
[AHTTPS] =================================
[AHTTPS] Vbuf len = 254
[AHTTPS] =================================
[AHTTPS] _collectHeaders()
[AHTTPS] _setReadyState : 2
[AHTTPS] _setReadyState : 3
[AHTTPS] *all data received - no disconnect
[AHTTPS] _setReadyState : 4

**************************************
Response code: 200
[AHTTPS] responseText()
[AHTTPS] responseText() = Test :) , size = 7
Response text: Test :)
**************************************
[AHTTPS] open( GET , url = https://yocal.dk/test.txt
[AHTTPS] _parseURL(): scheme+host HTTPS:// yocal.dk
[AHTTPS] _parseURL(): port+path+query 443 /test.txt
[AHTTPS] open: connecting to hostname = yocal.dk:443
[AHTTPS] _connect()
[AHTTPS] _onConnect handler
[AHTTPS] _setReadyState : 1
[AHTTPS] _connect() true
[AHTTPS] send()
[AHTTPS] _buildRequest()
[AHTTPS] _HTTPmethod = 0
[AHTTPS] GET  /test.txt  HTTP/1.1

[AHTTPS] write _HTTPmethodStringwithSpace : GET
[AHTTPS] write path : /test.txt
[AHTTPS] write query
[AHTTPS] write HTTP/1.1
[AHTTPS] To write hdr
[AHTTPS] hdr->name
[AHTTPS] hdr->value
[AHTTPS] host : yocal.dk:443

[AHTTPS] Write hdr done
[AHTTPS] _buildRequest() done
[AHTTPS] _send(), _request->available = 45
[AHTTPS] Done supply
[AHTTPS] *send 74
[AHTTPS] ========= _onData handler =======
[AHTTPS] Vbuf = HTTP/1.1 200 OK
Content-Type: text/plain
Last-Modified: Wed, 09 Nov 2022 16:59:11 GMT
Accept-Ranges: bytes
ETag: "3d3f4c975cf4d81:0"
Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET
Date: Wed, 09 Nov 2022 17:49:27 GMT
Content-Length: 7

Test :)I�U�4␋��=؝�n␓␛t]�:␃С␐�?�0�+␛�X␟��␐8������:�␞���4��d!�u1E[�␂��␎�␝�@�@0␅␆
[AHTTPS] =================================
[AHTTPS] Vbuf len = 254
[AHTTPS] =================================
[AHTTPS] _collectHeaders()
[AHTTPS] _setReadyState : 2
[AHTTPS] _setReadyState : 3
[AHTTPS] *all data received - no disconnect
[AHTTPS] _setReadyState : 4

**************************************
Response code: 200
[AHTTPS] responseText()
[AHTTPS] responseText() = Test :) , size = 7
Response text: Test :)
**************************************

emmettprexus · Answer 2 · Thu Nov 10 2022 01:59:59 GMT+0800 (China Standard Time)

Just tested v2.1.3 and the issue showed up again. So something happened between v2.1.2 and v2.1.3 that gives me the above issue. Guess it's this one as mentioned earlier.

Khoi Hoang · Answer 3 · Thu Nov 10 2022 11:07:24 GMT+0800 (China Standard Time)

The funny thing is it doesn't happen when connecting to the sample URLs.

Sorry I have no issue here with many requests to original example's site, repeatedly, such as

https://worldtimeapi.org/api/timezone/America/Toronto.txt

So you have to check what's wrong / difference to other working https site and fix it.

I really don't have time to deal with this. Sorry.

Good Luck,

Starting AsyncHTTPSRequest_ESP using ESP32_DEV
AsyncTCP_SSL v1.3.1
AsyncHTTPSRequest_Generic v2.2.0
Connecting to WiFi SSID: HueNet1
...
AsyncHTTPSRequest @ IP : 192.168.2.80

**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:02:36.392636-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049356
utc_datetime: 2022-11-10T03:02:36.392636+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:02:45.237136-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049365
utc_datetime: 2022-11-10T03:02:45.237136+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:02:55.235181-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049375
utc_datetime: 2022-11-10T03:02:55.235181+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:03:05.235913-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049385
utc_datetime: 2022-11-10T03:03:05.235913+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:03:15.240199-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049395
utc_datetime: 2022-11-10T03:03:15.240199+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:03:25.244353-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049405
utc_datetime: 2022-11-10T03:03:25.244353+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:03:35.242713-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049415
utc_datetime: 2022-11-10T03:03:35.242713+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:03:45.237077-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049425
utc_datetime: 2022-11-10T03:03:45.237077+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:03:55.235871-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049435
utc_datetime: 2022-11-10T03:03:55.235871+00:00
utc_offset: -05:00
week_number: 45
**************************************
H
**************************************
abbreviation: EST
client_ip: 65.110.220.219
datetime: 2022-11-09T22:04:05.238020-05:00
day_of_week: 3
day_of_year: 313
dst: false
dst_from: 
dst_offset: 0
dst_until: 
raw_offset: -18000
timezone: America/Toronto
unixtime: 1668049445
utc_datetime: 2022-11-10T03:04:05.238020+00:00
utc_offset: -05:00
week_number: 45
**************************************

Khoi Hoang · Answer 4 · Thu Nov 10 2022 13:03:33 GMT+0800 (China Standard Time)

Hi @Yocal

I just rethink and will make a new release to fix this issue, which is similar to Host/Headers not always sent with 1.10.1 #44 of AsyncHTTPRequest_Generic library

emmettprexus · Answer 5 · Thu Nov 10 2022 13:09:31 GMT+0800 (China Standard Time)

Thank you very much! :) I was just about to state my case that when not even https://www.google.com/robots.txt works, then I think the problem lies in the library :P Unfortunately my understanding of this topic in details are not good enough to be able to assist much, otherwise I'd gladly have delved into it to see if I could find the actual issue.

I appreciate that you wanna look into it! :) Thanks.

Khoi Hoang · Answer 6 · Thu Nov 10 2022 13:49:01 GMT+0800 (China Standard Time)

Hi @Yocal

The new AsyncHTTPSRequest_Generic v2.2.1 has just been published. Your contribution is noted in Contributions and Thanks

Please test and report more bugs.

Best Regards,

Releases v2.2.1

Default to reconnect to the same host:port after connected for new HTTP sites. Check Host/Headers not always sent with 1.10.1 #44 and Getting 400 Bad Request on second call to same host #14
Use allman astyle and add utils