paulpierre / markdown-crawler

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

Home Page:https://pypi.org/project/markdown-crawler/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

paulpierre/markdown-crawler Stargazers