[5.1.0] Deadlock and warnings when Http2 connection breaks
CRC-Mismatch opened this issue · comments
For some context, I've developed a Symfony Console command for a data migration task that requires basically just sending POST requests to a predefined endpoint and logging the results of each request - but since it's a mass-import task, we'll be doing around 1M of these POSTs. The command itself uses AMPHP's Pipelines and Workers to easily speed up the process, and inside the custom Task class, I'm using amphp/http-client to do the actual request/response handling.
Now on to the problem itself: It seems that whenever a connection breakage occurs, the affected amp-process
subprocess "spits" out some PHP warnings, then deadlocks using 100% of a CPU core. The remaining subprocesses are able to continue the batch just fine, until there's only the tasks that were already assigned to the locked process' workers remaining - then the whole command sits running "idle" until manually interrupted (SignalCancellation seems to do the trick), and only then a really long "Fatal Error" trace gets printed out...
The "initial" PHP warnings:
PHP Warning: Undefined array key 2033 in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php on
line 1336
PHP Warning: Attempt to read property "responsePending" on null in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php on line 1347
PHP Warning: Attempt to read property "body" on null in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php on line 1347
PHP Warning: Attempt to read property "trailers" on null in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php on line 1347
PHP Warning: Undefined array key 2033 in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php on line 1336
PHP Warning: Attempt to read property "responsePending" on null in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php on line 1347
PHP Warning: Attempt to read property "body" on null in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php on line 1347
PHP Warning: Attempt to read property "trailers" on null in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php on line 1347
The traces that only get printed after the Cancellation request:
PHP Fatal error: Uncaught Amp\Http\Client\SocketException: The HTTP/2 connection from '<redacted>:59720' to '<redacted>:443' closed unexpectedly in /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php:1432
Stack trace:
#0 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Internal/Http2ConnectionProcessor.php(1134): Amp\Http\Client\Connection\Internal\Http2ConnectionProcessor->shutdown()
#1 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(425): Amp\Http\Client\Connection\Internal\Http2ConnectionProcessor->runReadFiber()
#2 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(616): Revolt\EventLoop\Internal\AbstractDriver->invokeMicrotasks()
#3 [internal function]: Revolt\EventLoop\Internal\AbstractDriver->Revolt\EventLoop\Internal\{closure}()
#4 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/DriverSuspension.php(64): Fiber->resume()
#5 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(425): Revolt\EventLoop\Internal\DriverSuspension::Revolt\EventLoop\Internal\{closure}()
#6 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(616): Revolt\EventLoop\Internal\AbstractDriver->invokeMicrotasks()
#7 [internal function]: Revolt\EventLoop\Internal\AbstractDriver->Revolt\EventLoop\Internal\{closure}()
#8 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(498): Fiber->start()
#9 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(553): Revolt\EventLoop\Internal\AbstractDriver->invokeCallbacks()
#10 [internal function]: Revolt\EventLoop\Internal\AbstractDriver->Revolt\EventLoop\Internal\{closure}()
#11 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(113): Fiber->start()
#12 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop.php(406): Revolt\EventLoop\Internal\AbstractDriver->run()
#13 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/functions.php(83): Revolt\EventLoop::run()
#14 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/process-runner.php(88): Amp\Parallel\Context\Internal\runContext()
#15 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/process-runner.php(89): Amp\Parallel\Context\Internal\{closure}()
#16 {main}
Next Amp\Future\UnhandledFutureError: Unhandled future: Amp\Http\Client\SocketException: "The HTTP/2 connection from '<redacted>:59720' to '<redacted>:443' closed unexpectedly"; Await the Future with Future::await() before the future is destroyed or use Future::ignore() to suppress this exception. Enable assertions and set AMP_DEBUG=true in the process environment to track its origin. in /home/ubuntu/trebbiano/vendor/amphp/amp/src/Internal/FutureState.php:53
Stack trace:
#0 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/Http2Connection.php(105): Amp\Internal\FutureState->__destruct()
#1 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/HttpStream.php(89): Amp\Http\Client\Connection\Http2Connection->request()
#2 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\Connection\HttpStream->Amp\Http\Client\Connection\{closure}()
#3 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/HttpStream.php(89): Amp\Http\Client\processRequest()
#4 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/ConnectionLimitingPool.php(134): Amp\Http\Client\Connection\HttpStream->request()
#5 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/HttpStream.php(89): Amp\Http\Client\Connection\ConnectionLimitingPool->Amp\Http\Client\Connection\{closure}()
#6 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\Connection\HttpStream->Amp\Http\Client\Connection\{closure}()
#7 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/HttpStream.php(89): Amp\Http\Client\processRequest()
#8 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Interceptor/DecompressResponse.php(43): Amp\Http\Client\Connection\HttpStream->request()
#9 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/InterceptedStream.php(56): Amp\Http\Client\Interceptor\DecompressResponse->requestViaNetwork()
#10 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\Connection\InterceptedStream->Amp\Http\Client\Connection\{closure}()
#11 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/InterceptedStream.php(39): Amp\Http\Client\processRequest()
#12 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Interceptor/ModifyRequest.php(40): Amp\Http\Client\Connection\InterceptedStream->request()
#13 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/InterceptedStream.php(56): Amp\Http\Client\Interceptor\ModifyRequest->requestViaNetwork()
#14 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\Connection\InterceptedStream->Amp\Http\Client\Connection\{closure}()
#15 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/InterceptedStream.php(39): Amp\Http\Client\processRequest()
#16 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Interceptor/ModifyRequest.php(40): Amp\Http\Client\Connection\InterceptedStream->request()
#17 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/InterceptedStream.php(56): Amp\Http\Client\Interceptor\ModifyRequest->requestViaNetwork()
#18 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\Connection\InterceptedStream->Amp\Http\Client\Connection\{closure}()
#19 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Connection/InterceptedStream.php(39): Amp\Http\Client\processRequest()
#20 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/PooledHttpClient.php(36): Amp\Http\Client\Connection\InterceptedStream->request()
#21 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\PooledHttpClient->Amp\Http\Client\{closure}()
#22 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/PooledHttpClient.php(29): Amp\Http\Client\processRequest()
#23 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Interceptor/RetryRequests.php(37): Amp\Http\Client\PooledHttpClient->request()
#24 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/InterceptedHttpClient.php(44): Amp\Http\Client\Interceptor\RetryRequests->request()
#25 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\InterceptedHttpClient->Amp\Http\Client\{closure}()
#26 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/InterceptedHttpClient.php(35): Amp\Http\Client\processRequest()
#27 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Interceptor/FollowRedirects.php(131): Amp\Http\Client\InterceptedHttpClient->request()
#28 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/InterceptedHttpClient.php(44): Amp\Http\Client\Interceptor\FollowRedirects->request()
#29 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\InterceptedHttpClient->Amp\Http\Client\{closure}()
#30 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/InterceptedHttpClient.php(35): Amp\Http\Client\processRequest()
#31 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/Interceptor/ModifyRequest.php(52): Amp\Http\Client\InterceptedHttpClient->request()
#32 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/InterceptedHttpClient.php(44): Amp\Http\Client\Interceptor\ModifyRequest->request()
#33 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(20): Amp\Http\Client\InterceptedHttpClient->Amp\Http\Client\{closure}()
#34 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/InterceptedHttpClient.php(35): Amp\Http\Client\processRequest()
#35 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/HttpClient.php(35): Amp\Http\Client\InterceptedHttpClient->request()
#36 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/functions.php(30): Amp\Http\Client\HttpClient->Amp\Http\Client\{closure}()
#37 /home/ubuntu/trebbiano/vendor/amphp/http-client/src/HttpClient.php(32): Amp\Http\Client\processRequest()
#38 /home/ubuntu/trebbiano/src/Parallel/Task/ZedUserMigrationTask.php(106): Amp\Http\Client\HttpClient->request()
#39 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Worker/Internal/task-runner.php(58): App\Parallel\Task\ZedUserMigrationTask->run()
#40 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(425): Amp\Parallel\Worker\Internal\{closure}()
#41 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(616): Revolt\EventLoop\Internal\AbstractDriver->invokeMicrotasks()
#42 [internal function]: Revolt\EventLoop\Internal\AbstractDriver->Revolt\EventLoop\Internal\{closure}()#43 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/DriverSuspension.php(171): Fiber->throw()
#44 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(425): Revolt\EventLoop\Internal\DriverSuspension::Revolt\EventLoop\Internal\{closure}()
#45 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(616): Revolt\EventLoop\Internal\AbstractDriver->invokeMicrotasks()
#46 [internal function]: Revolt\EventLoop\Internal\AbstractDriver->Revolt\EventLoop\Internal\{closure}()
#47 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(497): Fiber->resume()
#48 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(553): Revolt\EventLoop\Internal\AbstractDriver->invokeCallbacks()
#49 [internal function]: Revolt\EventLoop\Internal\AbstractDriver->Revolt\EventLoop\Internal\{closure}()
#50 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(113): Fiber->start()
#51 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop.php(406): Revolt\EventLoop\Internal\AbstractDriver->run()
#52 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/functions.php(83): Revolt\EventLoop::run()
#53 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/process-runner.php(88): Amp\Parallel\Context\Internal\runContext()
#54 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/process-runner.php(89): Amp\Parallel\Context\Internal\{closure}()
#55 {main}
Next Revolt\EventLoop\UncaughtThrowable: Uncaught Amp\Future\UnhandledFutureError thrown in event loop callback Amp\Internal\FutureState::Amp\Internal\{closure} defined in /home/ubuntu/trebbiano/vendor/amphp/amp/src/Internal/FutureState.php:54; use Revolt\EventLoop::setErrorHandler() to gracefully handle such exceptions: Unhandled future: Amp\Http\Client\SocketException: "The HTTP/2 connection from '<redacted>:59720' to '<redacted>:443' closed unexpectedly"; Await the Future with Future::await() before the future is destroyed or use Future::ignore() to suppress this exception. Enable assertions and set AMP_DEBUG=true in the process environment to track its origin. in /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/UncaughtThrowable.php:13
Stack trace:
#0 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(400): Revolt\EventLoop\UncaughtThrowable::throwingCallback()
#1 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/Internal/AbstractDriver.php(116): Revolt\EventLoop\Internal\AbstractDriver::Revolt\EventLoop\Internal\{closure}()
#2 /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop.php(406): Revolt\EventLoop\Internal\AbstractDriver->run()
#3 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/functions.php(83): Revolt\EventLoop::run()
#4 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/process-runner.php(88): Amp\Parallel\Context\Internal\runContext()
#5 /home/ubuntu/trebbiano/vendor/amphp/parallel/src/Context/Internal/process-runner.php(89): Amp\Parallel\Context\Internal\{closure}()
#6 {main}
thrown in /home/ubuntu/trebbiano/vendor/revolt/event-loop/src/EventLoop/UncaughtThrowable.php on line 13
Killed
php -i
summary:
phpinfo()
PHP Version => 8.3.6
System => Linux ip-10-80-1-210 6.5.0-1017-aws #17~22.04.2-Ubuntu SMP Mon Mar 25 20:28:54 UTC 2024 x86_64
Build Date => Apr 11 2024 20:23:38
Build System => Linux
Server API => Command Line Interface
Virtual Directory Support => disabled
Configuration File (php.ini) Path => /etc/php/8.3/cli
Loaded Configuration File => /etc/php/8.3/cli/php.ini
Scan this dir for additional .ini files => /etc/php/8.3/cli/conf.d
Additional .ini files parsed => /etc/php/8.3/cli/conf.d/10-opcache.ini,
/etc/php/8.3/cli/conf.d/10-pdo.ini,
/etc/php/8.3/cli/conf.d/15-xml.ini,
/etc/php/8.3/cli/conf.d/20-bcmath.ini,
/etc/php/8.3/cli/conf.d/20-bz2.ini,
/etc/php/8.3/cli/conf.d/20-calendar.ini,
/etc/php/8.3/cli/conf.d/20-ctype.ini,
/etc/php/8.3/cli/conf.d/20-curl.ini,
/etc/php/8.3/cli/conf.d/20-dom.ini,
/etc/php/8.3/cli/conf.d/20-exif.ini,
/etc/php/8.3/cli/conf.d/20-ffi.ini,
/etc/php/8.3/cli/conf.d/20-fileinfo.ini,
/etc/php/8.3/cli/conf.d/20-ftp.ini,
/etc/php/8.3/cli/conf.d/20-gettext.ini,
/etc/php/8.3/cli/conf.d/20-gmp.ini,
/etc/php/8.3/cli/conf.d/20-iconv.ini,
/etc/php/8.3/cli/conf.d/20-igbinary.ini,
/etc/php/8.3/cli/conf.d/20-intl.ini,
/etc/php/8.3/cli/conf.d/20-mbstring.ini,
/etc/php/8.3/cli/conf.d/20-phar.ini,
/etc/php/8.3/cli/conf.d/20-posix.ini,
/etc/php/8.3/cli/conf.d/20-readline.ini,
/etc/php/8.3/cli/conf.d/20-redis.ini,
/etc/php/8.3/cli/conf.d/20-shmop.ini,
/etc/php/8.3/cli/conf.d/20-simplexml.ini,
/etc/php/8.3/cli/conf.d/20-soap.ini,
/etc/php/8.3/cli/conf.d/20-sockets.ini,
/etc/php/8.3/cli/conf.d/20-sysvmsg.ini,
/etc/php/8.3/cli/conf.d/20-sysvsem.ini,
/etc/php/8.3/cli/conf.d/20-sysvshm.ini,
/etc/php/8.3/cli/conf.d/20-tokenizer.ini,
/etc/php/8.3/cli/conf.d/20-xmlreader.ini,
/etc/php/8.3/cli/conf.d/20-xmlwriter.ini,
/etc/php/8.3/cli/conf.d/20-xsl.ini,
/etc/php/8.3/cli/conf.d/20-zip.ini
PHP API => 20230831
PHP Extension => 20230831
Zend Extension => 420230831
Zend Extension Build => API420230831,NTS
PHP Extension Build => API20230831,NTS
Debug Build => no
Thread Safety => disabled
Zend Signal Handling => enabled
Zend Memory Manager => enabled
Zend Multibyte Support => provided by mbstring
Zend Max Execution Timers => disabled
IPv6 Support => enabled
DTrace Support => disabled
Registered PHP Streams => https, ftps, compress.zlib, php, file, glob, data, http, ftp, compress.bzip2, phar, zip
Registered Stream Socket Transports => tcp, udp, unix, udg, ssl, tls, tlsv1.0, tlsv1.1, tlsv1.2, tlsv1.3
Registered Stream Filters => zlib.*, string.rot13, string.toupper, string.tolower, convert.*, consumed, dechunk, bzip2.*, convert.iconv.*
This program makes use of the Zend Scripting Language Engine:
Zend Engine v4.3.6, Copyright (c) Zend Technologies
with Zend OPcache v8.3.6, Copyright (c), by Zend Technologies
Composer packages versions:
amphp/amp v3.0.1
amphp/byte-stream v2.1.1
amphp/cache v2.0.1
amphp/dns v2.1.2
amphp/hpack v3.2.1
amphp/http v2.1.1
amphp/http-client v5.1.0
amphp/parallel v2.2.9
amphp/parser v1.1.1
amphp/pipeline v1.2.0
amphp/process v2.0.3
amphp/serialization v1.0.0
amphp/socket v2.3.1
amphp/sync v2.2.0
amphp/windows-registry v1.0.1
daverandom/libdns v2.1.0
kelunik/certificate v1.1.3
league/uri 7.4.1
league/uri-components 7.4.1
league/uri-interfaces 7.4.1
monolog/monolog 3.6.0
psr/cache 3.0.0
psr/clock 1.0.0
psr/container 2.0.2
psr/event-dispatcher 1.0.0
psr/http-factory 1.0.2
psr/http-message 2.0
psr/log 3.0.0
revolt/event-loop v1.0.6
symfony/cache v6.4.6
symfony/cache-contracts v3.4.2
symfony/clock v6.4.5
symfony/config v6.4.6
symfony/console v6.4.6
symfony/dependency-injection v6.4.6
symfony/deprecation-contracts v3.4.0
symfony/dotenv v6.4.4
symfony/error-handler v6.4.6
symfony/event-dispatcher v6.4.3
symfony/event-dispatcher-contracts v3.4.2
symfony/expression-language v6.4.3
symfony/filesystem v6.4.6
symfony/finder v6.4.0
symfony/flex v2.4.5
symfony/framework-bundle v6.4.6
symfony/http-foundation v6.4.4
symfony/http-kernel v6.4.6
symfony/messenger v6.4.6
symfony/monolog-bridge v6.4.4
symfony/monolog-bundle v3.10.0
symfony/polyfill-intl-grapheme v1.29.0
symfony/polyfill-intl-normalizer v1.29.0
symfony/polyfill-mbstring v1.29.0
symfony/polyfill-php83 v1.29.0
symfony/property-access v6.4.6
symfony/property-info v6.4.6
symfony/routing v6.4.6
symfony/runtime v6.4.3
symfony/serializer v6.4.6
symfony/service-contracts v3.4.2
symfony/string v6.4.4
symfony/var-dumper v6.4.6
symfony/var-exporter v6.4.6
symfony/yaml v6.4.3
There appears to be a race condition which is releasing an HTTP/2 stream within a callback which later releases the same string. The hang may be then due to the unexpected exception, though I'm not certain.
I pushed a branch, 5.x...issue-361, which a quick fix for checking if the stream was previously released before attempting to release it again. Can you change your dependency on amphp/http-client
to dev-issue-361
and see if the hang still occurs?