When array of coroutines is yielded, exception in one could lead to incorrect state
nick4fake opened this issue · comments
I am currently investigating this problem. My code:
public function warmup(): \Generator
{
$this->logger->info('Warming up active proxy cache');
$active = $this->cache->getItem('proxies.active');
$activeList = $active->isHit() ? $active->get() : [];
$toCheck = array_diff($this->proxyProvider->getProxies(), $activeList);
$this->logger->debug('Total active proxies: ' . count($activeList) . ', to check: ' . count($toCheck));
while (count($toCheck) > 0) {
$batch = array_slice($toCheck, 0, static::BATCH_SIZE);
$toCheck = array_slice($toCheck, static::BATCH_SIZE);
yield array_map(function (Proxy $proxy) {
return function () use ($proxy) {
$client = $this->clientBuilder->getProxifiedClient($proxy, static::CLIENT_TIMEOUT);
try {
yield $client->get('http://yandex.ru', [
'User-Agent' => UserAgent::random(),
])->then(function ($ret) {
echo 1;
});
$proxy->setStatus(Proxy::STATUS_OK);
} catch (\Exception $e) {
$proxy->setStatus(Proxy::STATUS_BAD);
throw $e;
}
yield;
};
}, $batch);
}
return count($activeList);
}If async request fails, I get this:
In StrandTrait.php line 134:
[Error]
Call to a member function throw() on null
It looks like somehow strand queue is broken by the fact that multiple strands fail in one loop iteration, resulting in a more sever problem after StrandWaitAll class cancels them.
Hi, thanks for the submission. Unfortunately, I'm not having any luck reproducing this issue. Are you able to reduce the example code to something that reproduces the issue without depending on the client / proxy parts of your project?
I've attempted to do so with the snippet below. If I understand correctly, multiple promises (as returned by $client->get() are being rejected, however this doesn't seem to cause the issue you are seeing.
Recoil\React\ReactKernel::start(function() {
$batch = ["a", "b", "c"];
yield array_map(
function ($v) {
return function () use ($v) {
yield React\Promise\reject($v);
yield;
};
},
$batch
);
});Could you also please let me know the version of recoil/kernel you have in your project. The version number is in the top of the changelog and can be seen easily by running:
head vendor/recoil/kernel/CHANGELOG.md
Finally, could you attempt to run the code you provided with PHP assertions enabled. The StrandTrait class has many assert() calls that might provide some insight into where this is being caused.
Hi,
Thank you for the quick response.
This is the exception I get after assertions are enabled:
In StrandTrait.php line 120:
[AssertionError (1)]
call-stack must not be empty
Recoil version:
recoil/api 1.0.1 The public Recoil API, for library and application developers.
recoil/kernel 1.0.1 Reusable components for implementing Recoil kernels.
recoil/react 1.0.2 Integrate Recoil with ReactPHP.
recoil/recoil 1.0.1 Asynchronous coroutines for PHP 7.
And react-related stuff:
clue/buzz-react v2.3.0 Simple, async PSR-7 HTTP client for concurrently processing any number of HTTP requests, built on t...
clue/socks-react v0.8.7 Async SOCKS4, SOCKS4a and SOCKS5 proxy client and server implementation, built on top of ReactPHP
react/cache v0.4.2 Async, Promise-based cache interface for ReactPHP
react/dns v0.4.13 Async DNS resolver for ReactPHP
react/event-loop v0.5.2 ReactPHP's core reactor event loop that libraries can use for evented I/O.
react/http v0.8.3 Event-driven, streaming plaintext HTTP and secure HTTPS server for ReactPHP
react/http-client v0.5.9 Event-driven, streaming HTTP client for ReactPHP
react/promise v2.5.1 A lightweight implementation of CommonJS Promises/A for PHP
react/promise-stream v1.1.1 The missing link between Promise-land and Stream-land for ReactPHP
react/promise-timer v1.3.0 A trivial implementation of timeouts for Promises, built on top of ReactPHP.
react/socket v0.8.11 Async, streaming plaintext TCP/IP and secure TLS socket server and client connections for ReactPHP
react/stream v0.7.7 Event-driven readable and writable streams for non-blocking I/O in ReactPHP
I am still working on shrinking the example. I'll post it as soon as reduced version is ready.
It looks like problem is on this line:
https://github.com/clue/reactphp-buzz/blob/master/src/Io/Sender.php#L108
$deferred = new Deferred(function ($_, $reject) use ($requestStream) {
// close request stream if request is canceled
$reject(new \RuntimeException('Request canceled'));
$requestStream->close();
});The problem is somehow related to that "reject" call. Commenting this line out removes the "throw" problem. I am still trying to understand if this is a problem in the @clue's library or in the coroutine handling.
@nick4fake I'm not currently aware of any issues in https://github.com/clue/reactphp-buzz, but if it boils down to an issue there, make sure to send another ping my way and I'm happy to look into this 👍
That Deferred you mentioned from Buzz leads me to believe that there is a mismatch between the cancellation semantics of Recoil's strands and React (or any) promises.
IIRC once the cancellation has triggered, Recoil does not expect the strand to be resumed, which is what the $reject() call will do. I suspect Recoil should be changed to allow the strand to be resumed, after cancellation, but if a given promise's cancellation handler does not reject the promise that leaves a strand in memory that will never be resumed.
I'll need to better my understanding of the promises to work out what the guarantees are here.
reactphp/promise#56 seems to be discussing what I'm talking about. Given that issue remains unresolved, I think the solution for Recoil is probably to keep track of when it cancels a promise then ignore any future call to its resolve/reject handlers for that promise.
This retains Recoil's current semantics, i.e., the strand does not get resumed after a promise is cancelled, and is consistent regardless of whether the promise rejects on cancel, or not.
I've "fixed" this on the no-resume-after-cancel branch of recoil/kernel.
@nick4fake, if you don't mind, could you please try this branch with your project and let me know if you see the behaviour you'd expect. You can override the recoil/kernel version installed by recoil/react by running:
composer require recoil/kernel="dev-no-resume-after-cancel as 1.999.999"@clue, I wouldn't mind your opinion on this change too. Moreso due to your familiarity with promises than anything to do with Buzz. I'm not 100% confident I'm making the right choice here.
@jmalloc I'm not too familiar with this project here, so I'm not sure of how much help my input would be, but here's the promise side FWIW :-)
The cancellation handler is currently optional and allows the creator of the promise to implement any kind of cancellation logic. It is usually assumed and recommended to ensure that the cancellation handler takes care of cleaning up any resource allocations and then rejects the promise. This is currently not enforced as part of the promise API for BC reasons. This implies that promise creators that do not care about cancellation will likely create a promise that will stay pending forever when cancelled. There's ongoing debate if the future Promise API should ensure that cancellation should always result in a rejection.
I hope this helps 👍
@nick4fake did the branch on recoil/kernel solve the problem? I'm running into the same issue.
I've rebased the no-resume-after-cancel branch from master, just in case it was lagging far behind to the point where it wasn't working correctly.
If either of you @bartvanhoutte or @nick4fake can confirm this does indeed fix your problem I am happy to merge it :)
I've run into the same issue, and have some insight:
TLDR:
- simpler code to reproduce issue
no-resume-after-cancelbranch DOES fix the issue. So, I suggest you DO merge it in ASAP!
Here's code to reproduce this issue simply, with no dependencies on reactphp/http (clue/reactphp-buzz):
require __DIR__ . '/vendor/autoload.php';
use Recoil\ReferenceKernel\ReferenceKernel;
function strandThrowingException() {
yield 3;
throw new Exception('expected exception');
}
function strandNotGettingCaught() {
$deferred = new \React\Promise\Deferred(function () { // define with a canceller function
throw new Exception('this exception cannot be caught');
});
return $deferred->promise();
}
ReferenceKernel::start(function() {
try {
yield [
strandThrowingException(),
strandNotGettingCaught()
];
} catch (Exception $e) {
echo 'caught: ' . $e->getMessage() . "\n"; // only catches the first exception
}
});
The issue is that, per the docs, if you yield an array, each array element is executed as a strand. If one of these throws an exception, the others are terminated. If one of those strands is a React Promise that has a canceller function defined (as the above example does), then that canceller function is called. If it, in turn, throws an exception, this exception cannot be caught and gives a PHP fatal error ("Call to a member function throw() on null").
The reactphp/http library has exactly this behaviour (canceller that throws an exception), which is why this behaviour shows up there.
But, good news is that I ran the above code with branch no-resume-after-cancel, and the PHP fatal error went away. The second exception (the uncatchable exception) WAS thrown, but then is silently discarded (uncatchably, from what I can see). Which is not ideal, but not sure what better behaviour there is to handle this situation that produces two exceptions.
Thanks for the confirmation and detailed information! I'll get this merged in.
Tagged in recoil/kernel v1.0.4.