Random "Pusher error: Internal server error." and slow on "larger" arrays, inconsistent speed

[tarreislam]

Reverb Version

beta 4

Laravel Version

11+

PHP Version

8.3

Description

So I decided to randomly test speed and payload size because I believe it's pretty inconsistent. I came up with a test and sometimes reverb would not send small payloads.

The config

max_request_size and max_message_size set to PHP_INT_MAX

The test

    $x = 0;
    $b = 0;
    for ($i = 0; $i < PHP_INT_MAX; $i += 1000) {
        event(new GnaQueueUpdatedEvent(range(0, $i)));
        if ($i % 1000 == 0) {
            $this->info("$i ($x) ~" . round($b / 1024 / 1024, 2) . 'mb');
            $x++;
            $b += $i;
        }
    }

class GnaQueueUpdatedEvent implements ShouldBroadcastNow
{
    use Dispatchable, InteractsWithSockets, SerializesModels;

    public array $gnaQueue;

    public function __construct(array $gnaQueue)
    {
        $this->gnaQueue = array_values($gnaQueue);
    }

    public function broadcastOn(): array
    {
        return [
            new Channel('Engine'),
        ];
    }

    public function broadcastAs(): string
    {
        return 'GnaQueueUpdatedEvent';
    }

}

The result
I get between 1 and 100 iterations before it breaks with "Internal server error"

Sometimes its slow, sometimes its fast. Somethimes it times out. But one thing is consistent, its inconsistent on all my tests.

I wish I could be more concrete. Hint hint

Here is a 32 megabyte gif that show me run a couple of tests. The Reverb server does not have any clients connected besides Laravel.

EDIT
I could not add the gif in the issue: so I uploaded it to a repo

https://github.com/tarreislam/uploaded-files/blob/master/example%20of%20reverb.gif

Steps To Reproduce

See above

[tarreislam]

I could not add the gif in the issue: so I uploaded it to a repo

https://github.com/tarreislam/uploaded-files/blob/master/example%20of%20reverb.gif

[Joe Dixon]

@tarreislam what do you consider a "small payload"?

Reverb isn't really designed for sending MBs of data across the wire.

[tarreislam]

@tarreislam what do you consider a "small payload"?

Reverb isn't really designed for sending MBs of data across the wire.

Yeah sure I send about 1 mb as its highest but thats not the issue, the issue is the random internal server error on 200 kb. Check the gif

[Joe Dixon]

The internal server error seems to come at 2MB?

I'm open to a pull request if you think we can improve the debugging output for this.

[Dries Vints]

This seems unrealistic. Feel free to send in a PR if you like to improve this.

[Jeff]

I also expierience the same issue, it has random error 500

[tarreislam]

This seems unrealistic. Feel free to send in a PR if you like to improve this.

Whats unrealistic, on the 4 or 5th run I would get a 500 on about 1kb but I sigkilled it

Id love to make A PR but "Internal server error" does not help much when debugging and I don't have time to unpack the entire codebase for reverb.

[tarreislam]

Small update

Okay I found where you muted the error and its here:

src/Servers/Reverb/Http/Server.php

$this->close($connection, 500, 'Internal server error. '. $e->getMessage());

When adding this piece of code, my random error is (39 kb payload and 2 mb payload)

Laravel\Reverb\Protocols\Pusher\Http\Controllers\EventsController::validator(): Argument #1 ($payload) must be of type array, null given, called in /var/www/vendor/laravel/reverb/src/Protocols/Pusher/Http/Controllers/EventsController.php on line 30.

So it fails before you can catch it

        $this->verify($request, $connection, $appId);

        $payload = json_decode($this->body, true); // probably null

        $validator = $this->validator($payload); // fail

        if ($validator->fails()) {
            return new JsonResponse($validator->errors(), 422);
        }

[tarreislam]

Ok here again.

I would consider you open this again, because I just lowered the payload to 1 mb and its consistently failing with the above error

when I log the payload $this->body the end of the message is cut of

3,99944,99945,99946,99947,99948,99  // missing 3 digits and end bracket

[tarreislam]

Changed test to this. We are not even talking megabytes anymore...

    $x = 0;

    $this->info("500 iterations of 512");
    while ($x < 512){
        event(new GnaQueueUpdatedEvent(range(0, 512)));
        $x++;
    }
    $this->info("500 iterations of 1024");
    $x = 0;
    while ($x < 512){
        event(new GnaQueueUpdatedEvent(range(0, 1024)));
        $x++;
    }
    return;

The 1024 test always fails at 1004-1006nth entry

952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,100  
[2024-03-28 23:14:39] local.ERROR: Pusher error: Internal server error. Laravel\Reverb\Protocols\Pusher\Http\Controllers\EventsController::validator(): Argument #1 ($payload) must be of type array, null given, called in /var/www/vendor/laravel/reverb/src/Protocols/Pusher/Http/Controllers/EventsController.php on line 31. {"exception":"[object] (Illuminate\\Broadcasting\\BroadcastException(code: 0): Pusher error: Internal server error. Laravel\\Reverb\\Protocols\\Pusher\\Http\\Controllers\\EventsController::validator(): Argument #1 ($payload) must be of type array, null given, called in /var/www/vendor/laravel/reverb/src/Protocols/Pusher/Http/Controllers/EventsController.php on line 31. at /var/www/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:164)
**[stacktrace]**

I am starting to suspect

[tarreislam]

I made a PR

#119

[王登辉]

REVERB_SERVER_HOST=0.0.0.0
REVERB_SERVER_PORT=8080
(A) for reverb:start bind

REVERB_HOST="localhost"
REVERB_PORT=8080
REVERB_SCHEME=http
(B) for laravel app send event to reverb

VITE_REVERB_HOST
(C) for frontend conn to reverb

if (B) is wrong when broadcast event will show red error Syntax error