semantic-chunking

There are 2 repositories under semantic-chunking topic.

isaacus-dev / semchunk
A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.
chunking nlp python splitting text text-chunking text-splitting semantic-chunking isaacus
Language:Python 418
mirth / chonky
Fully neural approach for text chunking
ai chunking llms ml rag semantic-chunking text-splitter
Language:Python 392
Unsiloed-AI / Unsiloed-Parser
hacktoberfest ai-agents chunking document-processing llm ocr openai pdf-parsing python rag semantic-chunking yolo
Language:Python 157
jparkerweb / semantic-chunking
🍱 semantic-chunking ⇢ semantically create chunks from large document for passing to LLM workflows
chunking embeddings llm semantic-chunking text-chunking text-splitter text-splitting vector
Language:JavaScript 116
jparkerweb / llm-distillery
🍶 llm-distillery ⇢ use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.
ai-text-reduction large-language-model llm openai-api semantic-chunking text-compression text-distillation text-processing text-summarization token-management tokenization
Language:JavaScript 11
prajwal10001 / semantic-chunker-langchain
Token-aware, LangChain-compatible semantic chunker with PDF, markdown, and layout support
ai langchain markdown nlp pdf python rag semantic-chunking
Language:Python 10
ThanhHung2112 / Semantic_chunking
Semantic Chunking is a Python library for segmenting text into meaningful chunks using embeddings from Sentence Transformers.
chunking nlp rag semantic-chunking text-split text vector vector-database
Language:Python 7
bazilicum / axonode-chunker
Advanced semantic text chunking with custom structural markers, whole-text coherence preservation, and flexible token management. Features async processing, LangChain integration, and dynamic drift detection. Ideal for RAG systems, augmented text processing, and domain-specific document analysis.
lang langchain rag semantic-chunking test-split text-spl
Language:Python 5
ProfEngel / OpenTuneWeaver
All in One-Solution for converting documents to finetune LLMs
ai all-in-one benchmarking educational-ai finetuning gradio llm lora pdf-processing personal-ai qa-generation qlora semantic-chunking opentuneweaver dataset-generation gguf model-deployment quantization vllm
Language:Python 2
dcirne / rag_fundamentals
Retrieval-Augmented Generation (RAG) Fundamentals and Semantic Chunking
artificial-intelligence machine-learning rag semantic-chunking
Language:Jupyter Notebook
gokhaneraslan / llm-qa-dataset-pipeline
🤖 Automated Q&A Dataset Generation Pipeline powered by LLMs. Multi-stage pipeline that searches, filters, extracts and transforms web content into high-quality question-answer datasets for LLM training. Supports multiple LLM providers (Groq, Mistral, Ollama) and search engines.
automated-pipeline content-extraction crawl4ai dataset-generation document-processing groq llama-index llm machine-learning mistral natural-language-processing nlp ollama qa-dataset question-answering semantic-chunking trafilatura web-scraping ai-training-data
Language:Python
Jayandhan03 / HR-Asst-rag
HR Policy Assistant (RAG-based Chatbot) A conversational AI assistant for employees to query company HR policies. Built with LangChain and Qdrant, it semantically ingests HR documents, retrieves relevant policy information, reranks results with BM25/MMR, and delivers precise LLM-generated responses.Cloud-based vector storage ensure quick responses.
dense-retrieval huggingface-spaces hybrid-retrieval langchain qdrant-vector-database rag-chatbot semantic-chunking streamlit-webapp
Language:Python
jsonusuman351 / Langchain_Text_Splitter
An exploration of advanced text splitting strategies in LangChain for RAG, from basic character splitting to state-of-the-art semantic chunking.
langcahin llm openai python rag semantic-chunking text-splitting
Language:Python
pipewrk / llm-core
Lightweight, composable TypeScript library for semantic chunking, workflow pipelining, and LLM orchestration.
bun chunking cosine-similarity data-processing embeddings llm nlp ollama openai pipeline semantic-chunking typescript
Language:TypeScript
smart-models / Progressive-Summarizer-RAPTOR
Cutting-edge semantic text processing system that uses hierarchical clustering and advanced language models to automatically organize and summarize large volumes of text.
gpu-acceleration hierarchical-clustering llm ollama-integration rag raptor rest-api semantic-chunking progressive-summarization docker
Language:Python
url4irl / vectors-gateway
A Sidecar service for applications that need vector database functionality to augment their LLMs. This service provides embeddings and retrieval capabilities by abstracting embeddings generation (LiteLLM) and vector storage and search (Qdrant).
embeddings litellm qdrant sidecar vectors rag semantic-chunking
Language:TypeScript
Utsav-J / chunking_strategies
``retrieval is all you need`` All in 1 repo for different levels of chunking along with their main logic and reusable code. No API keys used. Highly portable and pluggable
chunking langchain rag retrieval semantic-chunking
Language:Python

semantic-chunking

isaacus-dev / semchunk

mirth / chonky

Unsiloed-AI / Unsiloed-Parser

jparkerweb / semantic-chunking

jparkerweb / llm-distillery

prajwal10001 / semantic-chunker-langchain

ThanhHung2112 / Semantic_chunking

bazilicum / axonode-chunker

ProfEngel / OpenTuneWeaver

dcirne / rag_fundamentals

gokhaneraslan / llm-qa-dataset-pipeline

Jayandhan03 / HR-Asst-rag

jsonusuman351 / Langchain_Text_Splitter

pipewrk / llm-core

smart-models / Progressive-Summarizer-RAPTOR

url4irl / vectors-gateway

Utsav-J / chunking_strategies