Video Uploads are extremely slow

[ebitogu]

Prerequisites

• I have written a descriptive issue title
• I have searched existing issues to ensure the issue has not already been raised

Issue

I am not sure what may be the issue but when uploading short length videos of less than 2MB the video upload is blazing fast.

However, the moment I tried to upload a 20MB video, the time it takes before busboy starts to process the video is so so long.

Here is my code snippet:

busboy.on('file', async function (fieldname, file, filename, encoding, mimetype) {

             console.log("Video Reached Busboy"); //This happened very fast

            if (!unProcessedFilePaths.includes(filename)) {

                let fileType = await FileService.getFileType(filename);
                let transferredBytes = 0;

                console.log(`On Data about to be called`); 

                file.on('data', async (data) => {
                    console.log(`On Data is been called`);   //This is called only after about 2mins for a 20MB video. Weird

Please advise.

Thank you.

[Igor Savin]

as a quick thought, you may want to experiment with highWaterMark and fileHwm sizes

[ebitogu]

@kibertoad Thanks for that insight. However, I'm not sure what the optimal sizes maybe.

Do you mind recommending sizes that will make my videos upload faster?

Here is my current busboy configuration:

let busboy = new Busboy({
            headers: req.headers,
            limits: {
                fileSize: 6e+8,
            }
        });

Thank you

[Igor Savin]

Never had to do this kind of tweaking myself. Try increasing them significantly from defaults and measure whether it makes any perf difference for your use-cases.

[Aras Abbasi]

limits.fileSize is limitting the allowed uploadable filesize. That wont help your case.

You have to take care of the highWateMarks of the streams. i think the default value is 16 kb. So imagine chunking 20 mb into 16 kb blocks, results in a big overhead.

Maybe increase it to 512 kb. Would be then 40 chunks for 20 mb.

[ebitogu]

Oh wow. Thanks for that. I made the changes to this and it seems the speed is a bit faster now:

let busboy = new Busboy({
            headers: req.headers,
            highWaterMark: 1024000,
            fileHwm: 1024000,
            limits: {
                fileSize: 6e+8
            }
});

Thanks so much for this insight

[Aras Abbasi]

But is still quite odd, that calling data happens after few minutes?! It should immediatly call data.

---

Just wanted to remark, that 6e8 are about 600 GB. You should obviously reduce the size.

Also maybe 1024000 for 1 MB for a high water mark is a little bit big, but depends on your data. Keep in mind that the high water mark is a protection against high memory usage . A high watermark of 1mb means the stream tries to keep the memory usage at about 1MB. So if you set the memory limit in docker to 1 GB, you can only handle about 1000 uploads per instantiated service. If you use 512 kb your microservice can handle about 2000 uploads in parallel. That in ideal cases, as bigger chunks are also possible and there is also some memory usage by node and fastify too.

And container orchestrators kill containers which go over the limit. So keep these things in mind when you set the limits, highwatermarks etc .

I would say that you should provision more than gracefully your containers. so if you plan to create a youtube clone, you should first benchmark which highwatermark has the best performance and then decide how many uploads an instance should handle. Smaller is better.

E.g. 1mb has good performance. Lets say 100 uploads per instance are desirable. We need about 50-100 mb for node and fastify, 100 uploads with 1 Mb need 100mb, just take 30-50% more. So you could try with 250 MB RAM for your docker container.

Then keep track how many uploads you have in your installation and spin up more instances if you have more and shutdown gracefully if you need less.

[ebitogu]

@Uzlopak Thank you so much for the detailed breakdown.

However for the 6e+8 I think it's just 600MB because I actually crossed checked with Google early on before setting it unless maybe I'm wrong.

Here is what the busboy documentation says about the file size:

fileSize - integer - For multipart forms, the max file size (in bytes) (Default: Infinity).

and here is the conversion result I got from Google

Or am I wrong?

Thank you once again

[Aras Abbasi]

My fault. 600 mb.

[ebitogu]

Alright. Thank you