Lazy connection pooling
nulian opened this issue · comments
Would be nice to have a new version of the lazy connection pooling.
Only after upgrading I found out that the feature was removed and now different pool's are no longer possible it makes it impossible for us to upgrade to ecto 3 or just the mysql driver because of package codependency.
We have a lot of different databases because of different customers each needed his own connection. We only by default assigned 1 pool_size and used overflow to scale to the amount of jobs/request we could handle at the same time.
But now with the change it means we no longer can do that and increasing pool_size for all the apps isn't a possibility because that causes a high amount of db connections.
Hi @nulian. We currently have no plans to add overflow back but DBConnection does allow custom pools via the message passing API. For example, we still have the Pool and OwnershipPool. Unfortunately, those messages are not very well documented but if the goal is a custom pool, you could explore it to implement your own with overflow.
Also keep in mind that overflow in itself may reduce the latency considerably because if one connection is taken, even for a very quick query, another one will be spawned thanks to the overflow, while waiting for the previous one would likely be faster.
Another point to consider is that Ecto now supports dynamic repositories, so this feature could be implemented at a higher level too: https://hexdocs.pm/ecto/Ecto.Repo.html#c:put_dynamic_repo/1. I.e. you can keep and manage your own pool of Ecto repositories.
The reason we use multiple connections is because it can be fairly large queries. And the max was the same as the amount of requests the customers can do at the same time. Found this problem out by the queue_target and queue_interval error we got in the logs.
Will try to find out the best way to handle our pooling with this.
Closing this. IF you have more questions, please let us know!