Since v22 upgrade, curl requests (downloads in this case) are executing sequentially (blocking) in coroutines
AntonioNav opened this issue · comments
- What did you do? If possible, provide a simple script for reproducing the error.
<?php
declare(strict_types=1);
define ('PATH','/usr/local/deploy/');
require_once PATH.'vendor/autoload.php';
use Swoole\Coroutine;
\Swoole\Runtime::enableCoroutine(true, SWOOLE_HOOK_ALL | SWOOLE_HOOK_CURL | SWOOLE_HOOK_NATIVE_CURL | SWOOLE_HOOK_BLOCKING_FUNCTION | SWOOLE_HOOK_SOCKETS);
class swooleDownloadTest
{
public function run(array $cases, int $timeout): array
{
// $hooks = \Swoole\Runtime::getHookFlags();
//
// $flags['curlFlag'] = $hooks & SWOOLE_HOOK_CURL;
// $flags['nativeCurlFlag'] = $hooks & SWOOLE_HOOK_NATIVE_CURL;
// $flags['blockingFlag'] = $hooks & SWOOLE_HOOK_BLOCKING_FUNCTION;
// $flags['socketFlag'] = $hooks & SWOOLE_HOOK_SOCKETS;
//
// echo 'Flags: '.print_r($flags, true).PHP_EOL;
$tasks = $this->prepareTasks($cases);
$result = $this->runTasks($tasks, $timeout);
return $result;
}
private function prepareTasks(array $cases): array
{
$result = [];
foreach ($cases as $id => $case)
{
$result[] = function () use ($case, $id)
{
$result = $this->downloadFile($case, 'file_'.$id.'.dat');
return $result;
};
}
return $result;
}
private function runTasks(array $tasks, int $timeout): array
{
$result = [];
$tasksIDs = [];
$startTime = microtime(true);
Co::run(function() use ($tasks, &$result, $timeout, &$tasksIDs, $startTime )
{
foreach ($tasks as $id => $task)
{
go(function() use ($id, $task, &$result, &$tasksIDs)
{
$tasksIDs[] = Coroutine::getCid();
echo "Start CID: ".Coroutine::getCid().PHP_EOL;
$result[$id] = $task();
echo "End Task: ".$id.PHP_EOL;
});
}
Swoole\Timer::tick(100, function() use ($tasksIDs, $startTime, $timeout)
{
$totalTime = microtime(true) - $startTime;
$waitCoroutines = true;
foreach($tasksIDs as $taskID)
{
$waitCoroutines = Swoole\Coroutine::exists($taskID);
if ($waitCoroutines) {
break;
}
}
if (($totalTime >= $timeout) || !$waitCoroutines)
{
Swoole\Timer::clearAll();
foreach($tasksIDs as $taskID)
{
Swoole\Coroutine::cancel($taskID);
}
}
});
});
return $result;
}
private function downloadFile($url, $file)
{
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
));
$response = curl_exec($curl);
// $info = curl_getinfo($curl);
//
// echo 'Curl info: '.print_r($info, true).PHP_EOL;
curl_close($curl);
$result = false;
if ($response)
{
// Co::System::writeFile(__DIR__.'/'.$file, $response);
// file_put_contents(__DIR__.'/'.$file, $response);
$result = strlen($response);
}
return [$file => $result];
}
}
$test = new swooleDownloadTest();
$cases = [
'https://proof.ovh.net/files/10Mb.dat',
'https://proof.ovh.net/files/100Mb.dat',
'https://proof.ovh.net/files/1Mb.dat',
'https://proof.ovh.net/files/1Mb.dat',
'https://proof.ovh.net/files/10Mb.dat',
// 'https://proof.ovh.net/files/100Mb.dat',
];
$timeout = 10;
$startTime = microtime(true);
echo 'Start test: '.PHP_EOL;
$result = $test->run($cases, $timeout);
$totalTime = microtime(true) - $startTime;
echo "End test: ".$totalTime.PHP_EOL;
var_dump($result);
- What did you expect to see?
I expect all coroutines start in order, but first finish the smaller downloads and process terminate at the timeout of 5 seg. without wait to download the largest file. With version v4, prior to upgrade to v22 this script worked as expected.
Start test:
Start CID: 2
Start CID: 3
Start CID: 4
Start CID: 5
Start CID: 6
End Task: 2
End Task: 3
End Task: 0
End Task: 4
End test: 5.0000
- What did you see instead?
Start test:
Start CID: 2
End Task: 0
Start CID: 3
End Task: 1
Start CID: 4
End Task: 2
Start CID: 5
End Task: 3
Start CID: 6
End Task: 4
End test: 41.82473897934
- What version of OpenSwoole are you using (show your
php --ri openswoole)?
$ php --ri openswoole
openswoole
Open Swoole => enabled
Author => Open Swoole Group <hello@openswoole.com>
Version => 22.0.0
Built => Dec 22 2022 13:00:45
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.0.2 15 Mar 2022
dtls => enabled
http2 => enabled
pcre => enabled
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
postgresql => enabled
Directive => Local Value => Master Value
openswoole.enable_coroutine => On => On
openswoole.enable_preemptive_scheduler => Off => Off
openswoole.display_errors => On => On
openswoole.unixsock_buffer_size => 8388608 => 8388608
- What is your machine environment used (show your
uname -a&php -v&gcc -v) ?
$ uname -a
Linux ANAVARRO-LPT 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ php -v
PHP 7.4.33 (cli) (built: Jan 13 2023 10:42:56) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Zend OPcache v7.4.33, Copyright (c), by Zend Technologies
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
You can also try the following OpenSwoole support channels:
- Documentation - Documentation for Open Swoole
- Slack - Slack channel of Open Swoole
- Discord - Discord server of Open Swoole
Could you try to install with the following command or based on the installation guide and try it again?
pecl install -D 'enable-sockets="no" enable-openssl="yes" enable-http2="yes" enable-mysqlnd="yes" enable-hook-curl="yes" enable-cares="yes" with-postgres="yes"' openswoole
I installed OpenSwoole in Ubuntu as described in https://openswoole.com/docs/get-started/installation#install-open-swoole-on-ubuntu-2204-lts-jammy-jellyfish) with the packages provided in the ppa.
I'll try to test in another machine without these packages and install with pecl.
The following flag should display if installed correct:
hook-curl => enabled
Open Swoole => enabled
Author => Open Swoole Group <hello@openswoole.com>
Version => 22.0.0
Built => Jan 17 2023 12:36:05
coroutine => enabled with boost asm context
kqueue => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 3.0.7 1 Nov 2022
dtls => enabled
http2 => enabled
hook-curl => enabled
pcre => enabled
zlib => 1.2.11
brotli => E16777225/D16777225
mysqlnd => enabled
postgresql => enabled
I'd tryed with docker oficial image of php7.4 with Openswoole v4.12.1 and php8.1 with Openswoole v22.0.0:
$ docker run -it -v "$PWD":/usr/src/myapp -v "/usr/local/deploy/:/usr/local/deploy" -w /usr/src/myapp openswoole/swoole:4.12.1-php7.4 php swooleDownloadTest.php
Start test: 1673968734.9379
Start CID: 2
Start CID: 3
Start CID: 4
Start CID: 5
Start CID: 6
End Task: 2
End Task: 3
End Task: 4
End Task: 0
End Task: 1
End test: 10.10813999176
$ docker run -it -v "$PWD":/usr/src/myapp -v "/usr/local/deploy/:/usr/local/deploy" -w /usr/src/myapp openswoole/swoole php swooleDownloadTest.php
Start test: 1673969086.8539
Start CID: 2
Start CID: 3
Start CID: 4
Start CID: 5
Start CID: 6
End Task: 2
End Task: 3
End Task: 0
End Task: 4
End Task: 1
End test: 10.110167980194
In both cases seems to work the hook-curl, dowloading first the smaller files but the control of timeout with Coroutine::cancel still doesn't work.
Any ideas about why the official packages don't support 'hook-curl'? Are there any way to activate it or I have to remove the packages and use pecl?
The timeout control also works.
The only problem seems to be with the configuration of the .deb packages in the ppa.
Thanks.