Understanding HNSW and IVFFLAT index creation and storage

[taimur1991]

I am running a benchmark against some cloud databases for postgres which support pgvector extension and I am wondering where is the index created; either in RAM or disk, where is index stored and what happens if the memory available is less than index size? Although I have read that pgvector supports creation of index that is greater than memory available. The pgvector extension version is 0.6.0. Two cases come in my mind when the available memory is less than index size:-

1. Index is created on maintenance_work_mem and stored in shared memory (shared_buffers) while the surplus index is stored on the disk
2. Index is created on maintenance_work_mem and stored on disk alongwith the database files and then imported into the shared memory (shared_buffers). The index is partially loaded into the memory since the available memory is less than index size and relies on disk for the remaining portion of index.

My hunch is that the second case makes more sense since there's a possibility to lose index if database restarts. Any help in clearing this idea would be greatly appreciated.

[Jonathan S. Katz]

First, I'd recommend running with v0.6.2 (or v0.7.0) as v0.6.2 has changes that impact index build performance that will be noticeable in what you're testing.

The index build occurs in the portion of maintenance_work_mem that's allocated -- at the point of the index build shared_buffers is used for pages already persisted the disk (i.e. heap pages).

If the data used for building the index exceeds maintenance_work_mem, the "excess" data is written to temporary storage, which is wherever you've defined where your PostgreSQL temporary storage goes.

Once the index is built, pgvector allocates the pages for flushing to disk and handles that portion in bulk. This is when the new index pages make their way into the shared buffer pool.

My hunch is that the second case makes more sense since there's a possibility to lose index if database restarts.

I'm not following this -- assuming this is the "builk build" scenario, if your database restarts while your index is still building, you have to start the index build again as the transaction never finished.

[taimur1991]

Thanks @jkatz, that clears the concept. I'd really appreciate your comment on the size of index with more recent version of pgvector as I have gone through your study here with pgvector 0.5.0. Your study shows that index size depends on the configuration of hnsw and type of dataset used. Similar analysis was concluded by this study as well although this study does not provide the pgvector version. Could it be possible to estimate the size of index from the table or from any relation? For instance, this answer shows the estimation of index memory with the help of a relation proposed by FAISS. Can similar relation be made for index size as well?

[Andrew Kane]

Hi @taimur1991, someone could come up with a formula for estimating the index size based on the # of rows, # of dimensions, data type, index type, and index options (see #545 for the layout / storage of HNSW index tuples). You'd need to take into account the Postgres block size / page layout as well.