Suggestion: Don't serialize complete scope of Short Closures

[Hartmut Glücker]

Hello,

(if it matters: PHP 7.4.16, opis/closure 3.6.1)

I believe I noticed a minor concept issue during development with Laravel Batches that can easily lead to "OOM" fatal error cases during closure serialization. Assume the following example:

// assume there is sufficient RAM to create and hold $lotsOfJobs
$lotsOfJobs = Collection::times(999999, fn() => new MyJob('some-payload'));

Bus::batch($lotsOfJobs)
     ->finally(fn(Batch $batch) => Log::notice("Jobs finished. {$batch->totalJobs} jobs executed."))
     ->dispatch();

The "finally" callback will then be serialized using opis/closure and cause a fatal:

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/opis/closure/src/SerializableClosure.php on line 621
PHP Stack trace:
[...]
PHP  38. Illuminate\Bus\PendingBatch->dispatch() /var/www/html/app/<redacted>.php:33
PHP  39. Illuminate\Bus\DatabaseBatchRepository->store($batch = class Illuminate\Bus\PendingBatch { protected $container = class Illuminate\Foundation\Application { protected $basePath = '/var/www/html'; protected $hasBeenBootstrapped = TRUE; protected $booted = TRUE; protected $bootingCallbacks = [...]; protected $bootedCallbacks = [...]; protected $terminatingCallbacks = [...]; protected $serviceProviders = [...]; protected $loadedProviders = [...]; protected $deferredServices = [...]; protected $appPath = NULL; protected $databasePath = NULL; protected $langPath = NULL; protected $storagePath = NULL; protected $environmentPath = NULL; protected $environmentFile = '.env'; protected $isRunningInConsole = TRUE; protected $namespace = 'App\\'; protected $absoluteCachePathPrefixes = [...]; protected $resolved = [...]; protected $bindings = [...]; protected $methodBindings = [...]; protected $instances = [...]; protected $aliases = [...]; protected $abstractAliases = [...]; protected $extenders = [...]; protected $tags = [...]; protected $buildStack = [...]; protected $with = [...]; public $contextual = [...]; protected $reboundCallbacks = [...]; protected $globalBeforeResolvingCallbacks = [...]; protected $globalResolvingCallbacks = [...]; protected $globalAfterResolvingCallbacks = [...]; protected $beforeResolvingCallbacks = [...]; protected $resolvingCallbacks = [...]; protected $afterResolvingCallbacks = [...] }; public $name = 'Update Gracenote Stations (all Bearers)'; public $jobs = class Illuminate\Support\Collection { protected $items = [...] }; public $options = ['allowFailures' => TRUE, 'finally' => ...] }) /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/PendingBatch.php:226
PHP  40. Illuminate\Bus\DatabaseBatchRepository->serialize($value = ['finally' => ...]) /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/DatabaseBatchRepository.php:106
PHP  41. serialize($var = [ 'finally' => ...]) /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/DatabaseBatchRepository.php:279
PHP  42. Illuminate\Queue\SerializableClosure->serialize() /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/DatabaseBatchRepository.php:279
PHP  43. Illuminate\Queue\SerializableClosure->mapByReference($data = ['lotsOfJobs' => class Illuminate\Support\Collection { protected $items = [...] }, '¯\_(ツ)_/¯' => TRUE]) /var/www/html/vendor/opis/closure/src/SerializableClosure.php:148
[...]
<final stack trace position is some variable within MyJob>

Note how at stack trace position 43, the local-scope variable $lotsOfJobs is being serialized, despite it never being used within the "finally" callback.

I've worked around the issue by using a traditional closure which removes that variable from the scope, but I figured I should let you know about this. I don't know if it is feasible or possible to scan the closure's code for usage of scoped variables, and remove unused ones from the serialization process - but if so, I would suggest that to happen.

Either way, thank you for providing this package and keep being awesome! Feel free to close this suggestion without further comment if it's not possible or a "wontfix" edge case (fair enough!).

Best regards,
Harti

[Sorin Sarca]

Hi @Harti

I cannot reproduce your issue. If you can, please test this code and post the output of var_dump call.

// assume there is sufficient RAM to create and hold $lotsOfJobs
$lotsOfJobs = Collection::times(999999, fn() => new MyJob('some-payload'));

$f = fn(Batch $batch) => Log::notice("Jobs finished. {$batch->totalJobs} jobs executed.");

var_dump((new \ReflectionFunction($f))->getStaticVariables());

Bus::batch($lotsOfJobs)
     ->finally($f)
     ->dispatch();

[Sorin Sarca]

Anyway, we only serialize the scope variables provided by the PHP (using reflection), and lotsOfJobs is NOT one of them.

[Hartmut Glücker]

Thanks for the swift reply! I shall check this at work next week.

Had to remove work references from the example, maybe I simplified it too much in the process and that's why reproduction failed.
I'm also not entirely sure if it's something due to Laravel where I misinterpreted the stack trace to put the blame on you, I'll double check that as soon as I can and report back.

[Hartmut Glücker]

I'm sorry about the false alert, I'm unable to make a reproduction phpunit test case for this. The var_dump essentially proved that the scope really doesn't bleed.

This rather seems to be a Laravel specific issue with the SerializesModels trait (though I really do wonder where the 'lotsOfJobs' part in the stack trace came from, which led to my original assumption).

Thank you again!