daviddwlee84 / DuckDB_Chat

Access file as a database with interactive SQL query experience. Built on Streamlit and DuckDB.

DuckDB Chat

Access file as a database with interactive SQL query experience.

• Online Demo

Getting Started

pip install -r requirements.txt

# Optional
cp example.env .env
# Fill-in your keys
# ...

Streamlit

streamlit run app.py

https://raw.githubusercontent.com/daviddwlee84/DuckDB_Chat/main/demo/demo.csv

Chainlit

chainlit run --watch chainlit_app.py
chainlit run --watch chainlit_app_pandasai.py

Todo

• Add demo of DuckDB with Jupyter Notebook
• Able to use customized table alias
  • Implement with more elegant way => register table name
• Support multiple file loading (same extension and schema)
• Able to use DuckDB extensions - Extensions - DuckDB
• Add row limit for potential large file
  • <= 0 as no limit
  • Add df.head() at initial state
  • Add LIMIT at the end of SQL query
  • Use more elegant way...?
• Support no header CSV (or see what will happened)
• Support DuckDB, SQLite, ... (Connecting to a database — Python documentation)
• Able to plot (maybe indicate by some prefix like Jupyter magics)
  • Plot button (for each DataFrame or at the end) => this will make code very mess
  • tvst/st-execbox
  • Running Jupyter cells inside streamlit? - 🎈 Using Streamlit - Streamlit
  • Chart visualization — pandas 2.1.4 documentation
• Retrieve file from URL (DuckDB — Python documentation)
• Maybe user can ignore "FROM" clause, which by default indicate latest generated table original file
  • Think of option between latest generated table or the original file
• Option to print statistics (e.g. running time, rows, columns, ...)
• Add clear page button (clear the chat history but keep the file pointer) => Delete file or upload new one to clear history
• Add SQL hint and external resources on page
  • Add cheat sheet image
• Add option of auto SELECT * FROM table when the file loaded => Add session initial options
  • Also print some memory information
• Deploy this repository to Streamlit
• Support more than SQL command (i.e. DuckDB statement like CREATE TABLE, DESCRIBE, SHOW TABLES, etc.)
• Support DuckDB extensions
  • httpfs Extension - DuckDB
• Support web file path / API e.g. https://api.github.com/search/repositories?q=jupyter&sort=stars&order=desc
  • Analyzing Github Data with JupySQL + DuckDB — Python documentation
• Remove the file size limit of the file uploader
• Add natural language query support
  • Can refer to this neural-maze/talking_with_hn: The full experience of chatting with your favourite news website.
  • Inference LLM using API Key
  • Streaming user experience
  • Making agent that can generate DuckDB SQL query
  • Making agent that can summary table
• See how to change the main page name (i.e. app) without re-deploy Streamlit app => Currently, not possible
• Manage OpenAI keys in global settings
  • Maybe move duplicate code block into one place => refactor
  • Somehow failed to load from .env when disabling "key" in st.text_input
• Solve DuckDB read_parquet() require filename (instead of file object) issue => Currently using pd.read_parquet + duckdb.from_df
  • Make Python read_parquet() (and read_json()) support file object as input like read_csv() does · duckdb/duckdb · Discussion #9857
  • Let file_uploader return path, instead of reading the file in · Issue #904 · streamlit/streamlit
  • Get path from file_uploader() - 🎈 Using Streamlit - Streamlit
    • tempfile — Generate temporary files and directories — Python 3.12.0 documentation
• Support more file type (e.g. Excel, ...)
• Pipe (|) in SQL => multiple SQL description, and apply one after the other.
• Support PRQL
  • ywelsch/duckdb-prql: PRQL as a DuckDB extension
  • PRQL PRQL/prql: PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
• More Demo
  • NL to SQL
  • DBQA
    • Able to construct SQLDatabase from uploaded file
    • LangChain
    • LlamaIndex
      • NLSQLTableQueryEngine
      • SQLTableRetrieverQueryEngine + ObjectIndex (retriever)
      • Add Azure embed model
      • Option for metadata
      • Cost analysis
        • Cost Analysis - LlamaIndex 🦙 0.9.22
        • Token Counting Handler - LlamaIndex 🦙 0.9.22
        • Usage Pattern - LlamaIndex 🦙 0.9.22
      • Streaming
    • PandasAI: gventuri/pandas-ai: PandasAI is the Python library that integrates Gen AI into pandas, making data analysis conversational
      • Show plot image properly
      • Count Token: Large language models (LLMs) - PandasAI
      • Multiple files: SmartDatalake
      • Agent: Clarification questions

Chainlit

• Try Chainlit
  • Overview - Chainlit
  • Chainlit/chainlit: Build Python LLM apps in minutes ⚡️
• Format user message as SQL code
• Support more file types (e.g. parquet)
• New table as attachment
• Manually set default table name

Resources

DuckDB

• Python API - DuckDB
• Python DB API - DuckDB
• SQL Introduction - DuckDB
• DuckDB Documentation

LangChain x DuckDB

• DuckDB | 🦜️🔗 Langchain
• Unlocking ChatGPT's Potential: LangChain Tutorial with DuckDB - YouTube

SQLite

• In-Memory Databases

Streamlit

• Build a basic LLM chat app - Streamlit Docs
• Chat elements - Streamlit Docs

Deploy

• Deploy your app - Streamlit Docs
• App dependencies - Streamlit Docs
• Configuration - Streamlit Docs
• Secrets management - Streamlit Docs
  • Copy .env settings to the Streamlit App Settings > Secrets

LangChain x Streamlit

• Streamlit | 🦜️🔗 Langchain
• Streamlit • Generative AI
  • streamlit/llm-examples: Streamlit LLM app examples for getting started

LangChain

• SQL | 🦜️🔗 Langchain
  • Case 1: Text-to-SQL query
  • Case 2: Text-to-SQL query and execution
  • Case 3: SQL agents
• SQL Database | 🦜️🔗 Langchain
  • langchain.utilities.sql_database.SQLDatabase — 🦜🔗 LangChain 0.0.339rc1
  • feat: parquet file support for SQL agent · Issue #2002 · langchain-ai/langchain (this guy use parquet with duckdb => convert to SQLite)
• CSV | 🦜️🔗 Langchain
• sugarforever/LangChain-SQL-Chain
• LLMs and SQL

Database / SQL

• What is the process of databases parsing SQL? - Quora

Chainlit

• AskFileMessage - Chainlit
• Text to SQL - Chainlit