BUG: Documentation for configuring Aleph workers is lacking
stchris opened this issue · comments
Describe the bug
It is not immediately obvious that aleph and ingest-file workers are using "number of cpus" threads, which can cause scaling issues as seen in numerous reports, the last of which is #3383 .
To Reproduce
Steps to reproduce the behavior:
- Follow the installation / deployment docs
- Massively scale up ingest-file, workers etc.
- Run out of postgres connections
Expected behavior
Have better control over the number of workers running. Make it visible that we are using threads by default. Explain what the servicelayer workers default to and how to disable that.
Aleph version
3.15.1
Screenshots
Additional context
Related #3064
Would be good to set the limit to 0, and disable threads entirely. Also update the documentation to talk more about how we (don't) use this feature and what we recommend other people do.