🚀 AdvancedRAG

A state-of-the-art implementation of Retrieval Augmented Generation with advanced techniques

Features • Getting Started • Documentation • Contributing

🎯 Overview

A sophisticated implementation of advanced Retrieval Augmented Generation (RAG) techniques, featuring multi-strategy retrieval, automated evaluation, and modular architecture.

✨ Key Features

🔍 Multi-Strategy Retrieval Pipeline

AutoMerging Retrieval with hierarchical node parsing
Sentence Window Retrieval for granular context
Cross-encoder reranking for enhanced relevance
Multi-hop reasoning capabilities

📊 Advanced Evaluation Framework

Integrated TruLens evaluation
Confidence scoring and analysis
Automated groundedness assessment
Performance metrics dashboard

📁 File Structure

AdvancedRAG/
├── AutoMergingRetrieval/
│   ├── utils.py              # Core utilities
│   └── AutoMergingRetrieval.py
├── AdvancedRAGPipeline/
│   ├── src/
│   │   ├── utils.py         # Pipeline utilities
│   │   └── pipeline.py      # RAG orchestration
│   └── data/                # Evaluation sets
└── data/                    # Shared resources

🛠️ Techniques and Methodologies

AutoMerging Retrieval

Utilizes hierarchical node parsing to merge document nodes across varying levels of granularity, resulting in more contextualized retrieval.

Sentence Window Retrieval

Extracts text in overlapping windows to capture granular context, enhancing retrieval precision.

TruLens Evaluation

Integrates feedback mechanisms that measure answer relevance and groundedness, ensuring high-quality responses.

🔧 Implementation Details

AutoMergingRetrieval

Implements dynamic node size adjustment
Uses similarity-based merging strategies
Supports customizable merging thresholds

Advanced RAG Pipeline

Integrates multiple retrieval strategies
Features automated evaluation loops
Provides detailed performance metrics

🚦 Setup & Usage

Prerequisites

Python 3.8+
OpenAI API key
HuggingFace API key

Installation

Clone the repository:

git clone https://github.com/YanCotta/AdvancedRAG.git
cd AdvancedRAG

Install dependencies:
```
pip install -r requirements.txt
```

Configure environment: Create .env in project root:

OPENAI_API_KEY=your_openai_api_key
HUGGINGFACE_API_KEY=your_huggingface_api_key

Run the pipelines:

# For basic and auto-merging retrieval
python src/run_retrieval.py

# For full RAG pipeline with evaluations
python AdvancedRAGPipeline/src/run_pipeline.py

📝 Contributing & License

We welcome contributions! See our Contributing Guidelines for details.

Licensed under the MIT License.

Built with ❤️ by the AdvancedRAG Team

Report Bug • Request Feature

YanCotta / AdvancedRAG