Giters

microsoft / DiskANN

Graph-structured Indices for Scalable, Fast, Fresh and Filtered Approximate Nearest Neighbor Search

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG] deadlock when loading disk index when num_threads is zero

bddap opened this issue · comments

commented

Expected Behavior

When constructing a return StaticDiskIndex from a previously generated index.

diskannpy.StaticDiskIndex(
    index_directory=...,
    num_threads=0,
    num_nodes_to_cache=...,
    cache_mechanism=...,
    distance_metric=...,
    vector_dtype=...,
    dimensions=...,
)

I would expect the program to terminate eventually. From the docs, I would assume the constructor to use the number of logical processors available.

Actual Behavior

The program stops, never returning. The final thing to be printed is:

Setting up thread-specific contexts for nthreads: 0

Example Code

import diskannpy
import numpy as np
import tempfile

num_threads = 0

with tempfile.TemporaryDirectory() as tmpdir:
    diskannpy.build_disk_index(
        np.array([[0.0] * 32] * 1000, dtype=np.float32),
        "cosine",
        tmpdir,
        75,
        60,
        1,
        1,
        num_threads,
    )
    diskannpy.StaticDiskIndex(
        tmpdir,
        num_threads,
        0,
    )

Dataset Description

Please tell us about the shape and datatype of your data, (e.g. 128 dimensions, 12.3 billion points, floats)

  • Dimensions: 32
  • Number of Points: 1000
  • Data type: f32

Logs

logs 
Compressing 32-dimensional data into 32 bytes per vector.
Opened: /tmp/tmp0e5dbxiu/ann_vectors.bin, size: 128008, cache_size: 128008
Training data with 1000 samples loaded.
Processing chunk 0 with dimensions [0, 1)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 1 with dimensions [1, 2)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 2 with dimensions [2, 3)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 3 with dimensions [3, 4)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 4 with dimensions [4, 5)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 5 with dimensions [5, 6)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 6 with dimensions [6, 7)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 7 with dimensions [7, 8)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 8 with dimensions [8, 9)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 9 with dimensions [9, 10)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 10 with dimensions [10, 11)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 11 with dimensions [11, 12)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 12 with dimensions [12, 13)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 13 with dimensions [13, 14)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 14 with dimensions [14, 15)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 15 with dimensions [15, 16)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 16 with dimensions [16, 17)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 17 with dimensions [17, 18)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 18 with dimensions [18, 19)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 19 with dimensions [19, 20)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 20 with dimensions [20, 21)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 21 with dimensions [21, 22)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 22 with dimensions [22, 23)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 23 with dimensions [23, 24)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 24 with dimensions [24, 25)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 25 with dimensions [25, 26)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 26 with dimensions [26, 27)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 27 with dimensions [27, 28)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 28 with dimensions [28, 29)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 29 with dimensions [29, 30)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 30 with dimensions [30, 31)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Processing chunk 31 with dimensions [31, 32)
Residuals unchanged: 3.40282e+38 becomes 0. Early termination.
Writing bin: /tmp/tmp0e5dbxiu/ann_pq_pivots.bin
bin: #pts = 256, #dims = 32, size = 32776B
Finished writing bin.
Writing bin: /tmp/tmp0e5dbxiu/ann_pq_pivots.bin
bin: #pts = 32, #dims = 1, size = 136B
Finished writing bin.
Writing bin: /tmp/tmp0e5dbxiu/ann_pq_pivots.bin
bin: #pts = 33, #dims = 1, size = 140B
Finished writing bin.
Writing bin: /tmp/tmp0e5dbxiu/ann_pq_pivots.bin
bin: #pts = 4, #dims = 1, size = 40B
Finished writing bin.
Saved pq pivot data to /tmp/tmp0e5dbxiu/ann_pq_pivots.bin of size 37148B.
Opened: /tmp/tmp0e5dbxiu/ann_vectors.bin, size: 128008, cache_size: 128008
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin...
Metadata: #pts = 4, #dims = 1...
done.
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin...
Metadata: #pts = 256, #dims = 32...
done.
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin...
Metadata: #pts = 32, #dims = 1...
done.
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin...
Metadata: #pts = 33, #dims = 1...
done.
Loaded PQ pivot information
Processing points  [0, 1000)...done.
Time for generating quantized data: 0.300130 seconds
Full index fits in RAM budget, should consume at most 0.000499934GiBs, so building in one shot
L2: Using AVX2 distance computation DistanceL2Float
Using only first 1000 from file..
Starting index build with 1000 points...
0% of index build completed.Starting final cleanup..done. Link time: 0.02997s
Index built with degree: max:60  avg:60  min:60  count(deg<2):0
Not saving tags as they are not enabled.
Time taken for save: 0.000863s.
Time for building merged vamana index: 0.032453 seconds
Opened: /tmp/tmp0e5dbxiu/ann_vectors.bin, size: 128008, cache_size: 128008
Vamana index file size=244024
Opened: /tmp/tmp0e5dbxiu/ann_disk.index, cache_size: 67108864
medoid: 0B
max_node_len: 372B
nnodes_per_sector: 11B
# sectors: 91
Sector #0written
Finished writing 376832B
Writing bin: /tmp/tmp0e5dbxiu/ann_disk.index
bin: #pts = 9, #dims = 1, size = 80B
Finished writing bin.
Output disk index file written to /tmp/tmp0e5dbxiu/ann_disk.index
Finished writing 376832B
Time for generating disk layout: 0.022513 seconds
Opened: /tmp/tmp0e5dbxiu/ann_vectors.bin, size: 128008, cache_size: 128008
Loading base /tmp/tmp0e5dbxiu/ann_vectors.bin. #points: 1000. #dim: 32.
Wrote 94 points to sample file: /tmp/tmp0e5dbxiu/ann_sample_data.bin
Indexing time: 0.35653
Cosine metric chosen for (normalized) float data.Changing distance to L2 to boost accuracy.
L2: Using AVX2 distance computation DistanceL2Float
L2: Using AVX2 distance computation DistanceL2Float
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_compressed.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_compressed.bin...
Metadata: #pts = 1000, #dims = 32...
done.
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin...
Metadata: #pts = 4, #dims = 1...
done.
Offsets: 4096 36872 37008 37148
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin...
Metadata: #pts = 256, #dims = 32...
done.
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin...
Metadata: #pts = 32, #dims = 1...
done.
Reading bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin ...
Opening bin file /tmp/tmp0e5dbxiu/ann_pq_pivots.bin...
Metadata: #pts = 33, #dims = 1...
done.
Loaded PQ Pivots: #ctrs: 256, #dims: 32, #chunks: 32
Loaded PQ centroids and in-memory compressed vectors. #points: 1000 #dim: 32 #aligned_dim: 32 #chunks: 32
Disk-Index File Meta-data: # nodes per sector: 11, max node len (bytes): 372, max node degree: 60
Opened file : /tmp/tmp0e5dbxiu/ann_disk.index
Setting up thread-specific contexts for nthreads: 0

Your Environment

  • Ubuntu 22.04.2 LTS
  • diskannpy 0.6.0
commented

Excellent find: I'll get a fix out for this super shortly. The fault is in python/src/static_disk_index.cpp where I don't do const uint32_t _num_threads = num_threads != 0 ? num_threads : omp_get_num_threads();. I should have this fixed today along with some unit tests to make sure it doesn't sneak back in.