Giters

run-llama / llama_index

LlamaIndex is a data framework for your LLM applications

Home Page:https://docs.llamaindex.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Question]: why PostgresKVStore table class does not match my postgres vector table schema?

jodiewuxi opened this issue · comments

commented

Question Validation

  • I have searched both the documentation and discord for an answer.

Question

Question
I am using PostgresDocumentStore and PostgresKVStore to get Next and Prev node . But I got the follow error

psycopg2.errors.UndefinedColumn) column data_small_db.key does not exist
[SQL: SELECT public.data_small_db.id, public.data_small_db.key, public.data_small_db.namespace, public.data_small_db.value
FROM public.data_small_db
WHERE public.data_small_db.key = %(key_1)s AND public.data_small_db.namespace = %(namespace_1)s]

I have a vector database small_db that created by llama-index and default schema is

  Column   |       Type        | Collation | Nullable |                  Default
-----------+-------------------+-----------+----------+-------------------------------------------
 id        | bigint            |           | not null | nextval('data_small_db_id_seq'::regclass)
 text      | character varying |           | not null |
 metadata_ | json              |           |          |
 node_id   | character varying |           |          |
 embedding | vector(1536)

Column definition in PostgresKVStore

class AbstractData(base):  # type: ignore
        __abstract__ = True  # this line is necessary
        id = Column(Integer, primary_key=True, autoincrement=True)
        key = Column(VARCHAR, nullable=False)
        namespace = Column(VARCHAR, nullable=False)
        value = Column(metadata_dtype)

I found the issue is that column in PostgresKVStore not match my vector database.
Am I missing something, or is there a bug here?

commented

To align the PostgresKVStore class with your small_db vector database schema, you'll need to modify the column definitions in the AbstractData class within the get_data_model function. This involves adjusting the column names and types to match those in your small_db schema. Here's a step-by-step guide:

  1. Modify Column Definitions: Adjust the Column definitions in AbstractData to reflect your small_db schema. For example, if your schema includes columns for text, metadata_, node_id, and embedding, you should ensure these are represented in the class.

  2. Example Modification:

from sqlalchemy import Column, Integer, String
from sqlalchemy.dialects.postgresql import JSON, VECTOR  # Assuming VECTOR is a custom type

def get_data_model(base: Type, index_name: str, schema_name: str, use_jsonb: bool = False) -> Any:
    tablename = f"data_{index_name}"
    class_name = f"Data{index_name}"

    class AbstractData(base):
        __abstract__ = True
        id = Column(Integer, primary_key=True, autoincrement=True)
        text = Column(String, nullable=False)
        metadata_ = Column(JSON)
        node_id = Column(String)
        embedding = Column(VECTOR(1536))

    return type(
        class_name,
        (AbstractData,),
        {
            "__tablename__": tablename,
            "__table_args__": (
                Index(f"{tablename}_idx_text", "text"),
                {"schema": schema_name},
            ),
        },
    )

  1. Adjust for Custom Types or Constraints: If your schema uses custom types (like VECTOR), ensure these are correctly defined or imported. You might need to define custom types if they're not directly supported by SQLAlchemy.

  2. Integration with Existing Code: Make sure the modified get_data_model function is correctly integrated with your PostgresKVStore class, particularly in methods that interact with the database, such as put and get.

This approach requires you to closely align the class definitions with your database schema, ensuring that the column names and types in the AbstractData class match those in your small_db. For more detailed guidance, consider referring to the source code of the PostgresKVStore class in the LlamaIndex repository.

Details

Sources


About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

commented

The kvstore and vector store should be created as two different tables, with different table names and different schemas

It seems like you are querying the vector store thinking it has the same fields as the kv store?