Reducing Memory Usage
kiler129 opened this issue · comments
This is more a question than an issue. We are running some workloads requiring bath processing of records. To speed up the process multiple workers are used.
The problem is each new worker requires 2GB of memory to load libpostal data. This seems wasteful, since the data AFAIK doesn't change. Effectively this cause memory to have multiple copies of libpostal model, while also significantly delaying startup of each of the workers.
Is there any way to share the memory containing data model between workers, or is making a simple REST api our only option?
same here!
We finally went with https://github.com/johnlonganecker/libpostal-rest-docker and after seeeeveral millions of records it proven solid and fast :)