how to create HNSW indexes in parallel?
neicun1024 opened this issue · comments
How to create HNSW indexes in parallel or insert data in parallel after creating index?
I saw a project called hnswlib, which supports parallel insertion of data, but it is not used for databases.
And I noticed that hnswlib's code is a bit similar to yours.
So do you have any ideas?
Many thanks!
Sorry, do you mean https://github.com/nmslib/hnswlib project?
I do not see any support of parallel insertion in it.
Parallel index construction is supported by DiskANN (using Vamana algorithm)
We also working on extension using Vamana index but this work is in progress.
Due to its graph structure it is hard to parallize construction of HNSW index.
The simplest solution is partitioning: have N indexes instead of one. In this case each index can be constructed individually in parallel with each other. But search should be performed in all indexes (also can be done in parallel) with subsequent merging of results. Unfortunately support of parallel query execution in Postgres is not so good. Parallel workers (processes) are launched at for each query execution. It makes it reasonable to use parallel plan only for expensive OLAP queries. But time of HNSW search is about 1msec. In this case spawning parallel workers can only slowdown query execution.
By the way, we have implemented PQ optimisation for HNSW and it uses OpenMP library to speedup index construction process. But it is also in development stage.