Slow for very large folders

[adambarnett52]

Hi There

Is there a way to speed this up for very large folder ( aprox 63TB) it take a long time to calculate
I am using it as such

stats.size = await getFolderSize.loose(p);

Thanks
adam

update, i dont need file sizes just folder

[Anton]

Hello Adam

This package is not the fastest because it strikes a balance between usability, maintainability, and speed. There might be some way to shuffle the code around to improve speed without impacting the other concerns, but it is not obvious - you are very welcome to experiment with that and submit a PR if you find anything.

You can take the code from this package and improve it specifically for speed in your use case. One big thing you can do, is implement multi-threaded folder traversal, which would utilize all of your CPU's power, compared to the single core that this package uses. We might even accept a PR to add that functionality here as well.

If you want the absolute best performance, you should look into rewriting the code in Rust and compiling it to WebAssembly. That has the potential to significantly improve speed, but it is hard to say how big the improvement will really be.

Other than that, I can only think of maybe using a faster runtime engine, such as Bun.

[adambarnett52]

Thanks will take a look

[markthree]

Hello Adam

This package is not the fastest because it strikes a balance between usability, maintainability, and speed. There might be some way to shuffle the code around to improve speed without impacting the other concerns, but it is not obvious - you are very welcome to experiment with that and submit a PR if you find anything.

You can take the code from this package and improve it specifically for speed in your use case. One big thing you can do, is implement multi-threaded folder traversal, which would utilize all of your CPU's power, compared to the single core that this package uses. We might even accept a PR to add that functionality here as well.

If you want the absolute best performance, you should look into rewriting the code in Rust and compiling it to WebAssembly. That has the potential to significantly improve speed, but it is hard to say how big the improvement will really be.

Other than that, I can only think of maybe using a faster runtime engine, such as Bun.

I implemented one with go. Currently, cli is faster than the Current package。Later, I will improve a version of WebAssembly 🥰

👉 markthree/go-get-folder-size

[Anton]

@markthree nice work!

[markthree]

Hello Adam
This package is not the fastest because it strikes a balance between usability, maintainability, and speed. There might be some way to shuffle the code around to improve speed without impacting the other concerns, but it is not obvious - you are very welcome to experiment with that and submit a PR if you find anything.
You can take the code from this package and improve it specifically for speed in your use case. One big thing you can do, is implement multi-threaded folder traversal, which would utilize all of your CPU's power, compared to the single core that this package uses. We might even accept a PR to add that functionality here as well.
If you want the absolute best performance, you should look into rewriting the code in Rust and compiling it to WebAssembly. That has the potential to significantly improve speed, but it is hard to say how big the improvement will really be.
Other than that, I can only think of maybe using a faster runtime engine, such as Bun.

I implemented one with go. Currently, cli is faster than the Current package。Later, I will improve a version of WebAssembly 🥰

👉 markthree/go-get-folder-size

I tested that wasm implemented with go cannot call the io api 😂
After investigation, it is found that esbuild is also a subprocess call 😂
Go's wasm seems to be for the browser at the beginning 😵
It seems that for go users, it can only be compiled into binary executable files 🥵

[markthree]

Hello Adam
This package is not the fastest because it strikes a balance between usability, maintainability, and speed. There might be some way to shuffle the code around to improve speed without impacting the other concerns, but it is not obvious - you are very welcome to experiment with that and submit a PR if you find anything.
You can take the code from this package and improve it specifically for speed in your use case. One big thing you can do, is implement multi-threaded folder traversal, which would utilize all of your CPU's power, compared to the single core that this package uses. We might even accept a PR to add that functionality here as well.
If you want the absolute best performance, you should look into rewriting the code in Rust and compiling it to WebAssembly. That has the potential to significantly improve speed, but it is hard to say how big the improvement will really be.
Other than that, I can only think of maybe using a faster runtime engine, such as Bun.

I implemented one with go. Currently, cli is faster than the Current package。Later, I will improve a version of WebAssembly 🥰
👉 markthree/go-get-folder-size

I tested that wasm implemented with go cannot call the io api 😂 After investigation, it is found that esbuild is also a subprocess call 😂 Go's wasm seems to be for the browser at the beginning 😵 It seems that for go users, it can only be compiled into binary executable files 🥵

All right, my node version is too high 😂

The wasm of go can be run 🫣