php-http / react-adapter

PHP version: any

Description

In the client, we are using the ->request() method on the ReactPHP\Http\Browser object. This method will not use the streaming capabilities by default and it seems that all the request content will be stored in memory.

It seems to be a normal behavior on the ReactPHP side but I'm not sure it's a correct behavior here.

How to reproduce

Try to download a big file using a simple HTTP request. I made a sample repository to reproduce the issue, you can find it here : https://github.com/shulard/reactphp-adapter-issue-with-streaming

Possible Solution

In the reactphp/http Readme, there are some details about streaming responses: https://github.com/reactphp/http#streaming-response

I tried to work with those streams but couldn't find a way to retrieve the content. I think that we might change the Promise implementation to handle the stream.

Hello @WyriHaximus !

I would love to have your input here, maybe I misunderstood something regarding ReactPHP behavior…

Hey @shulard, it's not the promise that's the issue, it's the client that calls request and not requestStreaming on the Browser:

react-adapter/src/Client.php

Lines 52 to 57 in e209bef

    
           $this->client->request( 
        
               $request->getMethod(), 
        
               $request->getUri(), 
        
               $request->getHeaders(), 
        
               $request->getBody() 
        
           ),

@dbu You know this better than I do probably. Does php-http have a defined way to deal with streams? If not, the easiest way would be to copy the Client to StreamingClient and make the requestStreaming call instead of request.

Yep you’re right using requestStreaming will solve the current issue. However the current implementation with request can be used by external users without knowing that there is a implementation below that will be limited by the memory allowed to PHP.I thought that it can be interesting to make the default Client using the « requestStreaming » method, we just need to be sure that the promise will be fulfilled only when the request content has arrived locally…WDYT ? If we want to stick with the current implementation we must clarify the limitations in the doc…

@dbu What do php-http users expect? A fully ready to handle body or a stream?

@shulard The thing is that doing it that way could break a lot of existing assumptions. Having a specific streaming client works around that.

@WyriHaximus yes you're right, we can't change the way this library behave. We must have a full ready to handle body.

However, this body can be a local stream which contains the full response. My main concern here is to ensure that the response is completely received before fulfilling the Promise. My first idea was to pipe the streaming readable body to a writable stream and when the response is completely received, use the writable stream to build the final Response.

I know that inside ReactPHP, streaming capabilites means async but is it possible to wrap those streams and expose a Response ?

As a user that's what I expect, I made a request (sync or async), then get my response in a usable way that is not a ReactPHP specific implementation detail. This is only because we are using ReactPHP here as a low level tool and php-http/react-adapter must abstract it.

we are using PSR-7 messages. the MessageInterface says that getBody returns a StreamInterface. https://www.php-fig.org/psr/psr-7/#31-psrhttpmessagemessageinterface

the promise is supposed to return a PSR-7 response, so the body is always a stream.

does that help?

Yep thanks, but it leads to other questions ^^

Today, the react-adapter is exposing a StreamInterface implementation named BufferedBody. That implementation is blocking some StreamInterface feature like detach.

However, this implementation is working fine my main concern is the limitation of an "in memory buffer" in the case of HTTPlug adapters. This buffer is, by design limited to the allowed PHP memory so you can't retrieve bigger data (downloading files, very big API response).

It works fine for most of the API exchanges when it sticks to JSON/TXT responses but not for bigger responses.

This is the same behavior for sync and async calls.

I'll try to work on a solution which I guess can be fine:

Don't rely on BufferedBody but use React stream capabilities ;
Resolve the promise whenever the response was fully retrieved (and not before) ;
Ensure it works in sync and async mode…

sounds good. as long as the implementation respects the StreamInterface, i guess you don't even need the full stream to be downloaded before the promise can be resolved. or is that not the expected behaviour? i guess we then run into problems when an application starts consuming the stream interface, and data transfer breaks at some point before all data is received. then we should be doing a http exception, but that would not be expected anymore?
though for any http implementation, at that point it will already have received the status 200 and all headers. the server has no way of providing information about the error while streaming the response, afaik.

this might be an option on the react adapter to chose if a user prefers to work with actual streaming that can break while processing, or prefers the full download first to have clean exception handling.

Ok to much text to quote. But I'll start working on a package that can create a PSR-7 compatible stream from a react ReadableStreamInterface. This means we most likely can start returning the response earlier and users can decide to do getContents or read + eof calls in a loop to read the body. Note that when using getContents the entire response will still be buffered. And with read + eof the buffer will still fill up when you don't read quickly or big enough.

Got WIP up at https://github.com/WyriHaximus/reactphp-psr-7-stream here is an example on how to use it, be sure to never do an await in that while loop as you might miss data: https://github.com/WyriHaximus/reactphp-psr-7-stream/blob/270bdeaf05a93097b25b8b721fa8dd7241c0fcd2/tests/StreamTest.php#L34-L53

@shulard Let me know how this works for you, will add examples and documentation to the repo later on. It should be a good, at least, starting point.

	$this->client->request(
	$request->getMethod(),
	$request->getUri(),
	$request->getHeaders(),
	$request->getBody()
	),

This adapter doesn't use stream by default since ReactPHP use BufferedBody by default when using ->request(...) method.