Error building phar with compression enabled

[Johannes Skov Frandsen]

I get the following error:

proc_open(): unable to create pipe Too many open files

when I enable the following option in the box.json file:

"compression": "BZ2",

I'm trying to build a phar app with the Cilex framework as the foundation.
I'm running PHP 5.4.27 on OSX 10.9.2

[Deleted user]

Does this error message happen without the "compression" setting?

[Jánszky Lajos]

http://stackoverflow.com/questions/18744991/symfony2-process-component-unable-to-create-pipe-and-launch-a-new-process

Seems like the gc is still full of bugs (after 10 years). Typical php behavior, don't be afraid... ;-)

[Deleted user]

@localgod Can you use dev-master to see if you still see this issue?

[Johannes Skov Frandsen]

I'm normally using the phar version from http://box-project.org/
When I try to install dev-master with composer I get this message:

Skipped installation of bin bin/box for package kherge/box: file not found in package

So I can not really test it.

[Deleted user]

Try:

composer require kherge/box=2.0.x-dev

[Sebastian Krebs]

It seems, that this is still an issue with the current phar. Anything I can do?

PHP Fatal error:  Uncaught exception 'ErrorException' with message 'proc_open(): unable to create pipe Too many open files' in phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php:974
Stack trace:
#0 [internal function]: KevinGH\Box\Application->KevinGH\Box\{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 974, Array)
#1 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(974): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)
#2 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(784): Symfony\Component\Console\Application->getSttyColumns()
#3 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(745): Symfony\Component\Console\Application->getTerminalDimensions()
#4 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(675): Symfony\Component\Console\Application->getTe in phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php on line 974

[Kevin Herrera]

I'm afraid I don't know what else I can do on my end.

I believe that this is a problem with the phar extension itself, which may not be closing open file handles.

[Jánszky Lajos]

@kherge You should reduce the problem to 1-10 lines of code, write a short example and send a bug report (https://bugs.php.net/) or find an existing one with a workaround. I thought the workaround I linked would solve the problem. :S Maybe different bugs can cause the same error message. Good luck!

[lserni]

I have the same problem. The cause is indeed in the phar extension; the "compressFiles" function compresses each file to a temporary one, and when it has finished, it reads them back together again, and then closes them all. Doing so requires N files to be open at once. (You can increase the limit using ulimit() under Unix/Linux, though).

[Kevin Herrera]

@lserni Thanks for the additional info!

I'll see if I can write a small code example to trigger this issue.

[lserni]

The problem is actually in the "phar.c" file. This is a heavily snipped excerpt. Note that there is one cycle opening as many tmpfiles() as needed, and then another that makes use of them. There seems to be a check that avoids using those handles altogether, though.
An alternative would be to not store entry->cfp, but rather a pair ( offset, length ) with which to seek into a single huge temporary file with fixed cfp. Then all rewind(entry->cfp)'s would be replaced by fseek(hugefp, entry->offset, SEEK_SET), etc.:

newfile = php_stream_fopen_tmpfile();

for (zend_hash_internal_pointer_reset(&phar->manifest);
    zend_hash_has_more_elements(&phar->manifest) == SUCCESS;
    zend_hash_move_forward(&phar->manifest)) {

   ...
   entry->cfp = php_stream_fopen_tmpfile();

}

/* now copy the actual file data to the new phar */
offset = php_stream_tell(newfile);
for (zend_hash_internal_pointer_reset(&phar->manifest);
    zend_hash_has_more_elements(&phar->manifest) == SUCCESS;
    zend_hash_move_forward(&phar->manifest)) {

   if (entry->cfp) {
        file = entry->cfp;
        php_stream_rewind(file);
   }

   ...
   if (entry->cfp) {
        php_stream_close(entry->cfp);
        entry->cfp = NULL;
   }
}

/* finally, close the temp file, rename the original phar,
   move the temp to the old phar, unlink the old phar, and reload it into memory
*/
if (phar->fp && free_fp) {
    php_stream_close(phar->fp);
}

[Kevin Herrera]

Found an open bug on PHP's bug tracker which pretty much describes what we're experiencing.

Here's a snippet that will reproduce the issue:

<?php

ini_set('memory_limit', -1);

$max_handles = (int) file_get_contents('/proc/sys/fs/file-max');

if (!is_dir('src')) {
    echo "Creating source files...\n";

    mkdir('src');

    $total_dirs = ceil($max_handles / 1000);

    for ($current_dir = 0; $current_dir < $total_dirs; $current_dir++) {
        mkdir("src/$current_dir");

        for ($current_file = 0; $current_file < 1000; $current_file++) {
            touch("src/$current_dir/$current_dir-$current_file.php");
        }
    }
} else {
    echo "Source files already exist, skipping.\n";
}

if (file_exists('bug.phar')) {
    echo "Removing previous phar...\n";

    unlink('bug.phar');
}

echo "Building new phar...\n";

$phar = new Phar('bug.phar');
$phar->buildFromIterator(
    new RecursiveIteratorIterator(
        new RecursiveDirectoryIterator(
            'src',
            FilesystemIterator::SKIP_DOTS
        )
    ),
    'src'
);

echo "Compression files in phar...\n";

$phar = new Phar('bug.phar');
$phar->compressFiles(Phar::GZ);

EDIT: I also updated PHP's bug tracker.

[Jánszky Lajos]

PHP devs do not seem to care about PHP extensions since 2010. Lot of bugs are not resolved since then in the soap and phar extensions (but I guess you can find more extensions easily which suffered the same faith). I don't know whether you can do anything about this. You need some c skills, and fork, fix, pull request, but I guess these options are not available by PHP extensions (I may be wrong).

[Kevin Herrera]

Unfortunately I'm not experienced enough in C, and even then it could be a while before we see anything become of the fix.

From what I recall, the PHP extension used to be a PHP native class. I'm thinking that until this issue gets resolved properly, I could re-create the extension in PHP again. It'll be much slower, but it'll be easier and quicker to fix.

[Jánszky Lajos]

I don't know, I am not experienced in this topic.

For me the phar was important because of the digital signature. With that you can write a deploy tool, which copies phar files into a folder via FTP or HTTP. If somebody steals your password they can do nothing, because they cannot sign their own package without the private key.

Maybe others use this technology for different purposes, I think before you write your own phar implementation in PHP you should think about what you want to solve with it. So it should not be a simple copy-paste of the current solution. E.g. some people want to use compressed, password encrypted files, which is not supported by the current extension. Some people use webphar (I guess that was the name), which is only 1.5-2x slower than the native php. By them your slower solution won't be good enough I think. (Note: I don't use PHP for a year or so, I dev now in nodejs.)

[lserni]

I feel quite confident in altering phar.c - I shall do so as soon as my new laptop arrives, the old and faithful one having given up the ghost some days ago - but I think the problem lies rather in compiling (and distributing) the resulting phar.so module, which must match the existing PHP installation.

For me on OpenSuSE for example the easiest option seems to be to download the php5-phar SRPM and use that to rebuild the RPM.

But a quick fix would be to simply increase the open file limit before making the PHAR file:

ulimit -Sn 4096

I've tried it, and (where available) it works.

[Jánszky Lajos]

@lserni If you want to modify the c file, I think the best option to talk with the php guys. They already have the tools necessary to distribute the changes. I cannot find any contact about how to contribute to the project, and the phar extension is built in since php 5.3, but has a pecl registration as well: http://pecl.php.net/package/phar According to the pecl page: Marcus Börger [e-mail] who you should ask about this. (If that mail address still exists. Nobody touched the extension since 2009-07-29. Maybe you should ask too, why these extensions aren't maintained properly.)

[James Halsall]

@lserni thanks for that, works a charm 👍

[Vincent Klaiber]

I'm having the same issue. When I run box build after adding compression to the JSON config I get this error thrown. I'm building the phar with PHP 7.0 and it works fine without the compression.

$ box build
Building...
PHP Fatal error:  Uncaught ErrorException: proc_open(): unable to create pipe Too many open files in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php:978
Stack trace:
#0 [internal function]: KevinGH\Box\Application->KevinGH\Box\{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 978, Array)
#1 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(978): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)
#2 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(788): Symfony\Component\Console\Application->getSttyColumns()
#3 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(749): Symfony\Component\Console\Application->getTerminalDimensions()
#4 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(679): Symfony\Component\Console\Application->getTerminalWidth()
#5 phar in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php on line 978

Fatal error: Uncaught ErrorException: proc_open(): unable to create pipe Too many open files in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php:978
Stack trace:
#0 [internal function]: KevinGH\Box\Application->KevinGH\Box\{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 978, Array)
#1 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(978): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)
#2 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(788): Symfony\Component\Console\Application->getSttyColumns()
#3 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(749): Symfony\Component\Console\Application->getTerminalDimensions()
#4 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(679): Symfony\Component\Console\Application->getTerminalWidth()
#5 phar in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php on line 978

Below is the content of the box.json file.

{
    "chmod": "0755",
    "directories": [
        "src"
    ],
    "files": [
        "LICENSE",
        "./vendor/guzzle/guzzle/src/Guzzle/Http/Resources/cacert.pem"
    ],
    "finder": [
        {
            "name": "*.php",
            "exclude": ["Tests", "tests"],
            "in": "vendor"
        }
    ],
    "git-version": "package_version",
    "main": "bin/climb",
    "output": "climb.phar",
    "compression": "GZ",
    "stub": true
}

[lserni]

Did you try with my "dirty fix", below? (If you did and it did not work,
you may need to use
sudo).

(Also: if you get the same error but some time later, then it means that
even 4096 files are
not enough. You need to increase again the value).

I did write the patch to avoid using temp files while building the phar,
but never got around
to test it to any serious extent.

> But a quick fix would be to simply increase the open file limit before
making the PHAR file:

ulimit -Sn 4096

On Mon, Dec 21, 2015 at 9:06 AM, Vincent Klaiber [e-mail]
wrote:

I'm having the same issue. When I run box build after adding compression
to the JSON config I get this error thrown. I'm building the phar with PHP
7.0 and it works fine without the compression.

$ box build
Building...
PHP Fatal error: Uncaught ErrorException: proc_open(): unable to create pipe Too many open files in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php:978
Stack trace:#0 [internal function]: KevinGH\Box\Application->KevinGH\Box{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 978, Array)#1 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(978): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)#2 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(788): Symfony\Component\Console\Application->getSttyColumns()#3 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(749): Symfony\Component\Console\Application->getTerminalDimensions()#4 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(679): Symfony\Component\Console\Application->getTerminalWidth()#5 phar in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php on line 978

Fatal error: Uncaught ErrorException: proc_open(): unable to create pipe Too many open files in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php:978
Stack trace:#0 [internal function]: KevinGH\Box\Application->KevinGH\Box{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 978, Array)#1 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(978): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)#2 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(788): Symfony\Component\Console\Application->getSttyColumns()#3 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(749): Symfony\Component\Console\Application->getTerminalDimensions()#4 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(679): Symfony\Component\Console\Application->getTerminalWidth()#5 phar in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php on line 978

Below is the content of the box.json file.

{
"chmod": "0755",
"directories": [
"src"
],
"files": [
"LICENSE",
"./vendor/guzzle/guzzle/src/Guzzle/Http/Resources/cacert.pem"
],
"finder": [
{
"name": "*.php",
"exclude": ["Tests", "tests"],
"in": "vendor"
}
],
"git-version": "package_version",
"main": "bin/climb",
"output": "climb.phar",
"compression": "GZ",
"stub": true
}

—
Reply to this email directly or view it on GitHub
#80 (comment).

[Théo FIDRY]

If someone is still having that issue, please give a try to https://github.com/humbug/box. I couldn't reproduce that issue with it but I'm not 100% this is fixed so a reproducer is welcomed :)

[lserni]

I am not certain this fix actually addresses the problem - the temporary file design flaw is within the PHP code, so it will happen whenever you create a compressed PHAR with more than MAX_OPEN_FILES files inside (this is around 1024 or 4096 depending on systems). If you did set the ulimit to a value N and succeeded in creating a PHAR with N files then, unless you assemble the PHAR yourself bypassing the PHP generator, the problem is solved in the PHP version you're using. Cheers, Leonardo

…

On Sun, Apr 8, 2018 at 9:11 PM, Théo FIDRY ***@***.***> wrote: If someone is still having that issue, please give a try to https://github.com/humbug/box. I couldn't reproduce that issue with it but I'm not 100% this is fixed so a reproducer is welcomed :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#80 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGI6ngiBTO04d8b4cvw8q_L4TgP6Qu8fks5tmmDEgaJpZM4B1Fcn> .

[Théo FIDRY]

compressed PHAR with more than MAX_OPEN_FILES files inside

And how do you achieve this? Isn't creating a PHAR with N files and compressing the whole PHAR supposed to be the issue?

[lserni]

On Mon, Apr 9, 2018 at 9:21 AM, Théo FIDRY ***@***.***> wrote: compressed PHAR with more than MAX_OPEN_FILES files inside And how do you achieve this? Isn't creating a PHAR with N files and compressing the whole PHAR supposed to be the issue?

​Yes, it is. I'm sorry, I have probably expressed myself badly. As far as I can see, your commit modifies the list of the files that will be included in the PHAR being built, is it not so? If that is the case, I believe this can't possibly fix, alone, the issue at hand, for it does not change the problem (the open files table in the PHP binary code). Nothing that we can do in a ".php" file can fix the problem (except manually compressing data a file at a time, which allows us to never invoke the problematic function in PHP but is very slow in comparison). Based on a VERY cursory examination of the new PHP code, ​ *I believe​ that the issue has* *indeed been fixed*. I see several references to something called "PHAR_TMP" handles; I strongly suspect that phar code now uses some kind of indirect file mapping, to avoid hogging file handles, not unlike the fix I attempted, and failed, to make back in the day. https://github.com/php/php-src/blob/master/ext/phar/phar.c /* first, iterate over the manifest and close all PHAR_TMP entry fp handles, this prevents unnecessary unfreed stream resources */ Of course, it's also completely possible that I'm disastrously mistaken and am seeing things. It's been a while since I last waded inside PHP innards. Cheers, Leonardo

[Théo FIDRY]

If that just got fixed it's nice then :p I just try to reproduce it by creating a PHAR containing 100K files and compressing it whole but it didn't fail... So I wondered if I fixed that magically at some point or if that got fixed or if I did it wrong