Proxy to download http file slow

[YK-Samgo]

Describe the bug
Proxy to download http file slow

To Reproduce
Steps to reproduce the behavior:

1. Run proxy
2. Download one big file on local server with and without proxy. The speed diffs a lot.
3. I can see the proxy process used up one whole core in top

Download without proxy. Everything finished in one second on my server.

sudo curl -v http://127.0.0.1/api/bigfile -o /dev/null
*   Trying 127.0.0.1:80...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 80 (£0)
> GET /api/bigfile HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 28 Feb 2024 09:25:32 GMT
< Server: Apache/2.4.56 (Debian)
< Upgrade: h2,h2c
< Connection: Upgrade
< Last-Modified: Wed, 28 Feb 2024 09:25:24 GMT
< ETag: "100000000-6126db8fc99f6"
< Accept-Ranges: bytes
< Content-Length: 4294967296
<
{ [32513 bytes data]
100 4096M  100 4096M    0     0  3444M      0  0:00:01  0:00:01 --:--:-- 3444M

Download with proxy. I can't wait it to complete.

sudo curl -v http://127.0.0.1/api/bigfile -o /dev/null -x 'http://127.0.0.1:8899'
*   Trying 127.0.0.1:8899...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 8899 (£0)
> GET http://127.0.0.1/api/bigfile HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.74.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 28 Feb 2024 09:25:44 GMT
< Server: Apache/2.4.56 (Debian)
< Upgrade: h2,h2c
< Connection: Upgrade
< Last-Modified: Wed, 28 Feb 2024 09:25:24 GMT
< ETag: "100000000-6126db8fc99f6"
< Accept-Ranges: bytes
< Content-Length: 4294967296
<
{ [32513 bytes data]
  4 4096M    4  170M    0     0   961k      0  1:12:43  0:03:02  1:09:41  467k

[YK-Samgo]

I figured out what cost so much CPU.

In proxy/http/proxy/server.py:279: self.response.parse(raw)

This costs most of the CPU. I don't see anything necessary. Can you just remove it or add a switch to disable it?

Without this line, the speed is acceptable though performance loss

sudo curl -v http://127.0.0.1/api/bigfile -o /dev/null -x 'http://127.0.0.1:8899'
*   Trying 127.0.0.1:8899...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 8899 (£0)
> GET http://127.0.0.1/api/bigfile HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.74.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 28 Feb 2024 09:40:05 GMT
< Server: Apache/2.4.56 (Debian)
< Upgrade: h2,h2c
< Connection: Upgrade
< Last-Modified: Wed, 28 Feb 2024 09:25:24 GMT
< ETag: "100000000-6126db8fc99f6"
< Accept-Ranges: bytes
< Content-Length: 4294967296
<
{ [65281 bytes data]
100 4096M  100 4096M    0     0   581M      0  0:00:07  0:00:07 --:--:--  627M

[Abhinav Singh]

Thank you @YK-Samgo for reporting this. I have been away from proxy.py for too long. I hope to wrap pending PRs and open issues in upcoming release. 🙏