[Feature request] Data streaming

Question

[Feature request] Data streaming

acheong08 opened this issue a year ago · comments

Similar to Python requests: https://stackoverflow.com/questions/60343944/how-does-requests-stream-true-option-streams-data-one-block-at-a-time

Gun · Answer 1 · Sat Feb 25 2023 17:40:45 GMT+0800 (China Standard Time)

Streaming would be great to have because it allows user to make decision about skipping body data or buffer fetching to print result to client, which is great in many situations:

Content is too big, like 2GB and memory limitation like 2GB RAM, then if you fetch everything into memory you'll get OOM error
In proxy environment, you may want to abort the body data if the response header doesn't match some of your criterias

I'll try to look into the code and implement this feature, shouldn't be that hard considering we're fetching data from Go TLS Client which probably allow data streaming.

This line going forward should be what we need to modify: https://github.com/FlorianREGAZ/Python-Tls-Client/blob/master/tls_client/response.py#L40

Gun · Answer 2 · Sat Feb 25 2023 19:43:32 GMT+0800 (China Standard Time)

Actually after researching the API of tls-client by bogdanfinn, I can't find any information about stream being made available, so it looks like to have this stream feature we first need to do something with the core first, which is the Go Client https://github.com/bogdanfinn/tls-client-api

And... tls-client actually sends back JSON data instead of standard HTTP response with headers \r\n\r\n then body, which is ways easier to handle streaming (the same way requests and urllib3 are doing). Something like this, which is pretty pointless to stream data if you want to send buffer to something like web browsers, in this case you need to parse JSON data anyways, then send to web browsers:

{....,"headers":{"Access-Control-Allow-Credentials":["true"],"Access-Control-Allow-Origin":["https://rr6-sn-8qj-i5o6k.gooqlevideo.com"],"Age":["72715"],"Alt-Svc":["h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400"],"Cache-Control":["max-age=31536000"],"Cf-Cache-Status":["HIT"],"Cf-Ray":["79f028e9cf569e28-SIN"],"Content-Encoding":["br"],"Content-Security-Policy":["default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;form-action 'self';frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests"],"Content-Type":["text/html; charset=utf-8"],"Cross-Origin-Embedder-Policy":["require-corp"],"Cross-Origin-Opener-Policy":["same-origin"],"Date":["Sat, 25 Feb 2023 11:39:19 GMT"],"Expect-Ct":["max-age=0"],"Origin-Agent-Cluster":["?1"],"Referrer-Policy":["no-referrer"],"Server":["cloudflare"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"],"Vary":["Accept-Encoding","Origin"],"X-Content-Type-Options":["nosniff"],"X-Dns-Prefetch-Control":["off"],"X-Download-Options":["noopen"],"X-Frame-Options":["SAMEORIGIN"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Powered-By":["centminmod"],"X-Xss-Protection":["0","1; mode=block"]},"cookies":{"__cf_bm":"yxyOVhUi38n1UfQujprACZl5yxXyhYZMYl6boMzKKh0-1677324536-0-AdwLPV4avEeaFZNNLGA7VKzwHM3C2rbm2jL/SPwv1WNoRDaahAKij0hWwlApbK52ErrqT3pGG+pCYWIZlOzkM9o="}}'

FlorianREGAZ · Answer 3 · Sat Mar 18 2023 04:49:56 GMT+0800 (China Standard Time)

Hey, I don't think it'll be possible to add this soon sadly. :/