Prerequisites
- Openai account with payment option.
- Check in this section, if your openai account is appropiate: issues-with-openai.
- As rough estimate: 1000pdf pages cost about 1€.
- Pinecone account.
Foreword
Every change I did in the gpt4-pdf-chatbot-langchain project, I have included in the "gpt4-pdf-chatbot-langchain" folder. It should give you a quick overview on what changes need to be done. It needs your api-keys, namespace, .. though. It is not ment to be copied to your gpt4-pdf-chatbot-langchain project.
Join the discord if you have questions
Project setup
Download the project
Download the project: https://github.com/mayooear/gpt4-pdf-chatbot-langchain
Install nodejs and npm
Install Node.js and npm. Official website: https://nodejs.org/en/download
Verfiy that everything is installed with the following terminal commands.
- Your version may differ, it is only important that node has version 18 or higher.
Install all required packages
Open a terminal at the top level of the project and run "npm install" to install all necessary dependencies.
Create a ".env"
Create a file named ".env" inside your project.
Openai api key
Go to https://platform.openai.com/overview and got "View API keys".
Click "+ Create new secrete key" and copy the key in your ".env" file
Pinecone api key
Go to https://app.pinecone.io/organizations/ and create an api key there as well. Add them to the ".env" file.
Pinecone index
Create a new pinecone index with the Dimension 1536
.
It should look similar to this:
Add pinecone index to the ".env" file
Add the information from your index correct to the .env file. At this point it should look like this:
(Optional) Set namespace in config/pinecone.ts
Give the namespace for your vectors a name in "config/pinecone.ts".
All vectors that are ingested will be available by this namespace in your pinecone index. The name you defined in "config/pinecone.ts" will be used to store your ingested data in this namespace, but also lookup your data on a question to the chatbot.
- It is not possible to search data in multiple namespaces simultaneously.
Docs folder
Create a folder "docs" where you have your pdf files.
Ingest your pdf
Run "npm run ingest" in a terminal. This will ingest your pdf into the pinecone database. Remove the pdf document from the "docs" folder, otherwise it will be ingested a second time if you run "npm run ingest" again.
The terminal must be in the project folder at the top level. I have installed "gpt4-pdf-chatbot-langchain" in "Documents".
Start appliation
Start the application with "npm run dev" in the terminal. Open a browser and go to the address http://localhost:3000/. You should see the project.
Alternative ingest with python
To trouble shoot it might be useful to take a look at my python reimplementation at ingest. It gives more control over each step of ingest.
https://github.com/ucl98/pinecone_ingest_python_implementation
Issues with openai
Paid account
Check if you have a paid account. If you see the following message, you need to set up a paid account.
Api keys
Another issue can be that your api key is not working. Create a new one and test it again.
Contribution
- ucl
- angelina-magidova-synder
- chaudhary_181