Dagchain

Quick Install

pip install dagchain

🤔 What is this?

This is a proof of concept repo for vectorstore loader orchestration Ⓒ. It combines LangChain with Dagster.

✅ Run an example

We have created an example to show how to get started with this.

Clone this repo
Go into repository for examples cd examples
Install dependencies: pip install -r requirements.txt
Set OPENAI_API_KEY environment variable export OPENAI_API_KEY={YOUR_API_KEY}
Run make ingest-local and navigate to http://127.0.0.1:3000/ to view the dag jobs you have created.See alternative storage options below.
Run make run-local to interact with a langchain agent using the local vectorstore.

To edit ingest and run files you should only need to update the examples/{storage_type}/ingest.py and examples/{storage_type}/query.py files. Right now for local storage I have set up two dags that leverage common loading logic. To add a new loader run simply create new document loader and set it up in the examples/main.py file using dagchain. i. You can use any Langchain Document Loaders to load your own data.

Storage

Weaviate

To run with Weaviate Vector Database you will need to:

Sign up for an account here: Weaviate
Create a cluster with no auth, start the cluster, and get the url of the running instance (should look like: https://{your-endpoint-name}.weaviate.network/),and set environment variable export WEAVIATE_URL={YOUR_URL}
Run make ingest-weaviate and navigate to http://127.0.0.1:3000/ to view the dag jobs you have created.
Run make run-weaviate to interact with a langchain agent using the weaviate vectorstore.

Pinecone

To run with Pinecone Vector Database you will need to:

Sign up for an account here: Pinecone
Navigate to API Key page and set two environment variables export PINECONE_API_KEY={YOUR_API_KEY} and export PINECONE_ENVIRONMENT={YOUR_ENVIRONMENT}
Run make ingest-pinecone and navigate to http://127.0.0.1:3000/ to view the dag jobs you have created.
Run make run-pinecone to interact with a langchain agent using the pinecone vectorstore.

Future

This can be used to replace the manual run of ingest.sh in ChatLangChain to provide scheduled updates of vectorstores as well as handling and managing inputs from various sources.

Sample View

Contributing Guide

To build package:

Clean up any artifacts from before: rm -rf dist
Build package: python -m build (will create dist folder)
Upload package: twine upload dist/*

zanderchase / dagchain