thepi.pe

Extract markdown and visuals from PDFs URLs, slides, videos, and more, ready for multimodal LLMs. ⚡

thepi.pe is an API that can scrape multimodal data via thepipe.scrape or extract structured data via thepipe.extract from a wide range of sources. It is built to interface with vision-language models such as GPT-4o, and works out-of-the-box with any LLM or vector database. It can be used right away with a hosted cloud, or it can be run locally.

Features 🌟

Extract markdown, tables, and images from any document or webpage
Extract complex structured data from any document or webpage
Works out-of-the-box with all LLMs and RAG frameworks
AI-native filetype detection, layout analysis, and structured data extraction
Multimodal scraping for video, audio, and image sources

Get started in 5 minutes 🚀

thepi.pe can read a wide range of filetypes and web sources, so it requires a few dependencies. It also requires vision-language model inference for AI extraction features. For these reasons, we host an API that works out-of-the-box at thepi.pe.

For more detailed setup instructions, view the docs.

pip install thepipe-api

Hosted API (Python)

from thepipe.scraper import scrape_file
from thepipe.core import chunks_to_messages
from openai import OpenAI

# scrape markdown, tables, visuals
chunks = scrape_file(filepath="paper.pdf")

# call LLM with clean, comprehensive data
client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=chunks_to_messages(chunks),
)

Local Installation (Python)

For a local installation, you can use the following command:

pip install thepipe-api[local]

You must have a local LLM server setup and running for AI extraction features. You can use LiteLLM to run your local LLM server. Then, set the OPENROUTER_BASE_URL environment variable to your LLM server's endpoint URL and set OPENROUTER_API_KEY to the API key for your LLM of choice. the DEFAULT_VLM environment variable can be set to the model name of your LLM, such as openai/gpt-4o-mini if you are using OpenRouter or gpt-4o-mini if using the OpenAI API.

For full functionality with media-rich sources, you will need to install the following dependencies:

apt-get update && apt-get install -y git ffmpeg tesseract-ocr
python -m playwright install --with-deps chromium

When using thepi.pe, be sure to append local=True to your function calls:

chunks = scrape_url(url="https://example.com", local=True)

You can also use thepi.pe from the command line:

thepipe path/to/folder --include_regex .*\.tsx --local

Supported File Types 📚

Source	Input types	Multimodal	Notes
Webpage	URLs starting with `http`, `https`, `ftp`	✔️	Scrapes markdown, images, and tables from web pages. `ai_extraction` available for AI content extraction from the webpage's screenshot
PDF	`.pdf`	✔️	Extracts page markdown and page images. `ai_extraction` available for AI layout analysis
Word Document	`.docx`	✔️	Extracts text, tables, and images
PowerPoint	`.pptx`	✔️	Extracts text and images from slides
Video	`.mp4`, `.mov`, `.wmv`	✔️	Uses Whisper for transcription and extracts frames
Audio	`.mp3`, `.wav`	✔️	Uses Whisper for transcription
Jupyter Notebook	`.ipynb`	✔️	Extracts markdown, code, outputs, and images
Spreadsheet	`.csv`, `.xls`, `.xlsx`	❌	Converts each row to JSON format, including row index for each
Plaintext	`.txt`, `.md`, `.rtf`, etc	❌	Simple text extraction
Image	`.jpg`, `.jpeg`, `.png`	✔️	Uses pytesseract for OCR in text-only mode
ZIP File	`.zip`	✔️	Extracts and processes contained files
Directory	any `path/to/folder`	✔️	Recursively processes all files in directory
YouTube Video (known issues)	YouTube video URLs starting with `https://youtube.com` or `https://www.youtube.com`.	✔️	Uses pytube for video download and Whisper for transcription. For consistent extraction, you may need to modify your `pytube` installation to send a valid user agent header (see this issue).
Tweet	URLs starting with `https://twitter.com` or `https://x.com`	✔️	Uses unofficial API, may break unexpectedly
GitHub Repository	GitHub repo URLs starting with `https://github.com` or `https://www.github.com`	✔️	Requires GITHUB_TOKEN environment variable

How it works 🛠️

thepi.pe uses computer vision models and heuristics to extract clean content from the source and process it for downstream use with language models, or vision transformers. The output from thepi.pe is a list of chunks containing all content within the source document. These chunks can easily be converted to a prompt format that is compatible with any LLM or multimodal model with thepipe.core.chunks_to_messages, which gives the following format:

[
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "..."
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "data:image/jpeg;base64,..."
        }
      }
    ]
  }
]

You can feed these messages directly into the model, or alternatively you can use chunker.chunk_by_document, chunker.chunk_by_page, chunker.chunk_by_section, chunker.chunk_semantic to chunk these messages for a vector database such as ChromaDB or a RAG framework. A chunk can be converted to LlamaIndex Document/ImageDocument with .to_llamaindex.

⚠️ It is important to be mindful of your model's token limit. GPT-4o does not work with too many images in the prompt (see discussion here). To remedy this issue, either use an LLM with a larger context window, extract larger documents with text_only=True, or embed the chunks into vector database.

emcf / thepipe