Add support for Pinecone namespaces

[Ed Burnette]

Pinecone takes a namespace (PINECONE_NAME_SPACE) that can either be empty (default) or a string. I didn't see a way in llamaindex.ts to pass that in when inserting or querying, and setting the environment variable had no effect, so it's currently always using the default namespace.

Documentation: https://docs.pinecone.io/docs/namespaces

[Alexandre Beauvois]

so a workaround could be:

    const pcvs = new PineconeVectorStore({ indexName: 'my-index' });

    const ctx = serviceContextFromDefaults();
    const index = await VectorStoreIndex.fromVectorStore(pcvs, ctx);

    // list all namespaces (the default one is '')
    await index.describeIndexStats();
    // Response: {
    //   namespaces: {
    //     '': { recordCount: 10 }
    //     foo: { recordCount: 2000 },
    //     bar: { recordCount: 2000 }
    //   },
    //   dimension: 1536,
    //   indexFullness: 0,
    //   totalRecordCount: 4010
    // }

    // Add a namespace arbitray named "ns-1" to our index
    index.namespace('ns-1');

    console.log(index.namespace('ns-1').target); // { index: 'my-index', namespace: 'ns-1', indexHostUrl: undefined }

    // Now perform index operations in the targeted index and namespace
    await index.fetch(['3']);

    await index
      .namespace('ns-1')
      .query({ topK: 3, vector: [0.22, 0.66] });
    // {
    //   matches: [
    //     {
    //       id: '556',
    //       score: 1.00000012,
    //       values: [],
    //       sparseValues: undefined,
    //       metadata: undefined
    //     },
    //     {
    //       id: '137',
    //       score: 1.00000012,
    //       values: [],
    //       sparseValues: undefined,
    //       metadata: undefined
    //     },
    //     {
    //       id: '129',
    //       score: 1.00000012,
    //       values: [],
    //       sparseValues: undefined,
    //       metadata: undefined
    //     }
    //   ],
    //   namespace: 'ns-1',
    //   usage: {
    //     readUnits: 5
    //   }
    // }

Should be necessary to improve the queryEngine params to accept the concept of namespaces? @marcusschiesser

[Dudu Lasry]

I also need this solution :) I'm currently using Pinecone's index object directly:

const embedModel = new OpenAIEmbedding({
  apiKey: process.env.OPENAI_API_KEY as string,
  model: "text-embedding-3-large",
  dimensions: 768,
});
const vector = await embedModel.getTextEmbedding("Hello world");

const pc = new Pinecone({ apiKey: process.env.PINECONE_API_KEY as string });
const response = await pc
  .index("my-index")
  .namespace("my-namespace")
  .query({ topK: 5, vector, includeMetadata: true });

[Marcus Schiesser]

I think we can easily add a namespace parameter to the PineconeVectorStore, then you can create an index per namespace, like this:

const ctx = serviceContextFromDefaults();
const pcvs1 = new PineconeVectorStore({ indexName: 'my-index', namespace: 'ns-1' });
const index1 = await VectorStoreIndex.fromVectorStore(pcvs1, ctx);
const pcvs2 = new PineconeVectorStore({ indexName: 'my-index', namespace: 'ns-2' });
const index2 = await VectorStoreIndex.fromVectorStore(pcvs2, ctx);

[Marcus Schiesser]

This feature is implemented in the new 0.1.19 release. Please give it a try!