Menghuan1918 / pdfdeal

A python wrapper for the Doc2X API and comes with native PDF processing (to improve PDF recall in RAG). | Doc2X API的python封装，同时附带本地的PDF处理(提升PDF在RAG中的召回率)。

pdfdeal

Better RAG Effect!

🗺️ ENGLISH | 简体中文

Easily handle PDFs, extract readable text, recognize image text with OCR and clean up formatting to make it more suitable for building knowledge bases.

Introduction

What's NEW

Added documentation tutorial on how to integrate with graphrag

Doc2X Support

Doc2X is a new universal document OCR tool that can convert images or PDF files into Markdown/LaTeX text with formulas and text formatting. It performs better than similar tools in most scenarios. pdfdeal provides abstract packaged classes to use Doc2X for requests.

Processing PDFs

Use various OCR or PDF recognition tools to identify images and add them to the original text. You can set the output format to use PDF, which will ensure that the recognized text retains the same page numbers as the original in the new PDF. It also offers various practical file processing tools.

After processing PDFs, you can achieve better recognition rates when used with knowledge base applications such as graphrag, Dify, and FastGPT.

It is recommended to use Doc2X for the best results.

Cases

For example, if graphrag does not support recognizing PDFs, you can use doc2x to convert it into txt documents for use.

Or for knowledge base applications, you can also use pdfdeal to enhance documents. Below are the effects of original PDF/OCR enhancement/Doc2X processing in Dify:

Documentation

You can view new features under development here!

For details, please refer to the documentation

Or check out the documentation repository pdfdeal-docs.

Quick Start

For details, please refer to the documentation

Installation

Install from PyPI:

pip install --upgrade pdfdeal

Using pytesseract as an OCR engine

When using "pytesseract", make sure that tesseract is installed first:

pip install 'pdfdeal[pytesseract]'

from pdfdeal import deal_pdf, get_files

files, rename = get_files("tests/pdf", "pdf", "md")
output_path, failed, flag = deal_pdf(
    pdf_file=files,
    output_format="md",
    ocr="pytesseract",
    language=["eng"],
    output_path="Output",
    output_names=rename,
)
for f in output_path:
    print(f"Save processed file to {f}")

Using Doc2X as PDF deal tool

from pdfdeal import Doc2X
from pdfdeal import get_files

client = Doc2X()
file_list, rename = get_files(path="tests/pdf", mode="pdf", out="pdf")
success, failed, flag = client.pdfdeal(
    pdf_file=file_list,
    output_path="./Output/test/multiple/pdfdeal",
    output_names=rename,
)
print(success)
print(failed)
print(flag)