Giters

facebookresearch / faiss

A library for efficient similarity search and clustering of dense vectors.

Home Page:https://faiss.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

To use sparse matrix directly with Faiss or some other way around #need help

Anujcol opened this issue · comments

commented

Summary

<I am encoding a column in a dataframe using Tfidf which creates a sparse matrix and which is then converted to an array to be used with faiss. I am getting memory error while coverting from matrix to array and I am not able to use matrix directly with Faiss.

Please help me how to use sparse matrix directly with Faiss or some other way around-->

Platform

OS: <Windows 10>

Faiss version: <1.7.3>

Installed from: <!anaconda pip install faiss-cpu>

Faiss compilation options: <!not sure-->

Running on:

  • CPU

Interface:

  • Python

Reproduction instructions

vectorizer = TfidfVectorizer()
encoded_data = vectorizer.fit_transform(df['description'])

encoded_data_dims = encoded_data.shape[1]
index = faiss.IndexIDMap(faiss.IndexFlatIP(encoded_data_dims))

encoded_data = encoded_data.toarray().astype('float32')
index.add_with_ids(encoded_data, indices)

commented

Kindly help or just point me in the right direction, anyone?