PDF to RAG Converter

Convert PDFs to text and split the text into RAG-ready JSON chunks with metadata, BM25 sparse vectors, and (optionally) Azure OpenAI embeddings.

A real run on the sample PDF in examples/: PDF to text, text to 6 chunks with BM25 sparse vectors, then one of the resulting chunks (text, metadata, top BM25 terms). Dense embeddings were not run because they need Azure OpenAI credentials.

Try it in 1 minute

No API keys needed. This reproduces the demo above with BM25 sparse vectors only (tested with Python 3.12).

git clone https://github.com/AceAtDev/RAG-chunker.git
cd RAG-chunker/pdf2text2chunks
python -m venv venv && source venv/bin/activate
pip install PyPDF2 openai spacy tqdm rank_bm25
python -m spacy download en_core_web_sm

python main.py convert --input ../examples --output demo_output/text
python main.py chunk --input demo_output/text --output demo_output/chunks \
  --bm25-only --chunk-size 150 --chunk-overlap 30

The chunks are written to demo_output/chunks/us_constitution_article1_chunks.json. The sample PDF is the Preamble and Article 1, Sections 1 to 3 of the U.S. Constitution (public domain).

To regenerate the GIF from a fresh run: pip install pillow, then python ../demos/record_terminal_demo.py.

Features

• PDF to text: Extracts the text layer of PDFs with PyPDF2.
• Chunking: Sentence-aware chunks (spaCy) with configurable size and overlap, measured in words.
• BM25 sparse vectors: One sparse vector per chunk. Indices are hashed term ids (the same term gets the same index in every file), values are BM25 term weights computed over the chunks of the same file. No API key needed.
• Dense embeddings (optional): Azure OpenAI embeddings (text-embedding-3-small by default). Requires Azure OpenAI credentials.
• Translation (optional): Arabic to English with Azure Translator.
• Metadata: File, chunk, and content metadata (word counts, entities and key phrases from spaCy) on every chunk.
• Batch processing: Processes every file in a directory with progress bars.

OCR for scanned PDFs is not included yet: --method ocr exits with an error because src/ocr_converter.py is not in the repository.

Installation

git clone https://github.com/AceAtDev/RAG-chunker.git
cd RAG-chunker/pdf2text2chunks
pip install -r requirements.txt
python -m spacy download en_core_web_sm

requirements.txt also lists the optional Azure, Google Cloud, OCR, and development packages (including easyocr, which pulls in PyTorch). If you only need PDF to chunks, the five packages in "Try it in 1 minute" are enough.

Usage

All commands are run from the pdf2text2chunks/ folder.

1. Configuration (only needed for embeddings or translation)

Dense embeddings need an Azure OpenAI key and endpoint. Set them as environment variables:

export AZURE_OPENAI_API_KEY="your-azure-openai-api-key"
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"

or create a config file. The tool reads config.json from the current directory:

python -c "from src.config import Config; Config().create_sample_config()"
cp config.sample.json config.json   # then edit config.json with your credentials

The sample config looks like this:

{
  "azure": {
    "api_key": "your-azure-openai-api-key",
    "endpoint": "https://your-resource.openai.azure.com/",
    "api_version": "2024-05-01-preview",
    "embedding_model": "text-embedding-3-small",
    "translator_location": "eastus"
  },
  "google": {
    "credentials_path": "path/to/your/google-credentials.json",
    "project_id": "your-google-project-id",
    "bucket_name": "your-bucket-name",
    "location": "us-central1"
  },
  "vectors": {
    "enable_embeddings": true,
    "embedding_retries": 3,
    "embedding_delay": 0.06,
    "enable_bm25": true,
    "bm25_max_dim": 1000,
    "bm25_k1": 1.2,
    "bm25_b": 0.75
  },
  "processing": {
    "chunk_size": 512,
    "chunk_overlap": 50,
    "max_workers": 4,
    "default_language": "en",
    "supported_languages": ["en", "ar"]
  }
}

Embeddings are enabled by default. Without Azure credentials, pass --bm25-only.

2. Convert PDFs to text

python main.py convert --input pdfs/ --output text/

3. Create chunks

# BM25 sparse vectors only (no API key needed)
python main.py chunk --input text/ --output chunks/ --bm25-only

# Embeddings + BM25 (default, needs Azure OpenAI credentials)
python main.py chunk --input text/ --output chunks/ --azure-key YOUR_API_KEY

# Embeddings only (needs Azure OpenAI credentials)
python main.py chunk --input text/ --output chunks/ --azure-key YOUR_API_KEY --embeddings-only

# Custom chunk settings (sizes are in words)
python main.py chunk --input text/ --output chunks/ --bm25-only \
  --chunk-size 1024 \
  --chunk-overlap 100

4. Translate text files (Arabic to English)

Needs an Azure Translator key and pip install azure-ai-translation-text. The --translator google option is not wired up in the CLI yet.

python main.py translate --input text_ar/ --output text_en/ --translator azure --key YOUR_TRANSLATOR_KEY

5. Full pipeline

Runs convert and chunk in one step (chunk size 512, overlap 50) and writes to output/text/ and output/chunks/.

# No API key needed
python main.py pipeline --input pdfs/ --bm25-only

# With embeddings (needs Azure OpenAI credentials)
python main.py pipeline --input pdfs/ --azure-key YOUR_API_KEY

--translate adds a translation step and passes the --azure-key value to Azure Translator, so it only works if that key is valid for Azure Translator. Running translate as a separate step is clearer.

Python API

PDF to text

from src.pdf_converter import PDFConverter

converter = PDFConverter(input_dir="pdfs", output_dir="text")
results = converter.process_directory()

print(f"Converted {results['successful']} files")

Chunks

from src.text_chunker import TextChunker

chunker = TextChunker(
    input_dir="text",
    output_dir="chunks",
    chunk_size=512,
    chunk_overlap=50,
    enable_embeddings=False,   # set True and pass azure_api_key / azure_endpoint for embeddings
    enable_bm25=True,
    bm25_max_dim=1000
)

results = chunker.process_directory()
print(f"Created {results['total_chunks']} chunks")

Translation

from src.translator import Translator

translator = Translator(
    service="azure",
    api_key="your-translator-key",
    source_lang="ar",
    target_lang="en",
    input_dir="text_ar",
    output_dir="text_en"
)

results = translator.process_directory()

Output format

Each text file produces one <name>_chunks.json file containing a list of chunks. This is chunk 2 of the demo run, shortened:

{
  "id": "us_constitution_article1_chunk_0001",
  "content": "No Person shall be a Representative who shall not have attained to the Age of twenty five Years, ...",
  "metadata": {
    "file": {
      "filename": "us_constitution_article1.txt",
      "file_path": "demo_output/text/us_constitution_article1.txt",
      "title": "us constitution article1",
      "source_type": "text_file"
    },
    "chunk": {
      "chunk_id": "us_constitution_article1_chunk_0001",
      "chunk_index": 1,
      "total_chunks": 6,
      "chunk_position": "2/6",
      "word_count": 140,
      "character_count": 805,
      "content_hash": "5f2c9b8f92eba7af8f441b2e97230f49"
    },
    "content": {
      "word_count": 742,
      "sentence_count": 22,
      "entities": ["the house of representatives", "the united states", "..."],
      "key_phrases": ["this constitution", "the several states", "..."]
    },
    "categories": ["general"],
    "processing": {
      "chunk_size": 150,
      "chunk_overlap": 30,
      "embeddings_enabled": false,
      "bm25_enabled": true,
      "bm25_max_dim": 1000
    }
  },
  "bm25_vector": {
    "indices": [1625267812, 2091544603, 3524121191, "..."],
    "values": [2.134, 1.821, 1.821, "..."]
  }
}

• metadata.content describes the whole source file (entities and key phrases come from its first 1000 characters), not the single chunk.
• bm25_vector indices are unsigned 32-bit hashes of the terms (MD5-based, see TextChunker.bm25_term_index). IDF statistics are computed per file.
• embedding (a list of floats) is added only when embeddings are enabled and Azure OpenAI credentials are set.

Customization

Custom chunking

create_chunks takes the cleaned text and returns a list of chunk strings. Metadata, BM25 vectors, and embeddings are added afterwards by process_text_file.

from typing import List
from src.text_chunker import TextChunker

class FixedWindowChunker(TextChunker):
    def create_chunks(self, text: str) -> List[str]:
        words = text.split()
        return [" ".join(words[i:i + 200]) for i in range(0, len(words), 200)]

chunker = FixedWindowChunker(input_dir="text", output_dir="chunks", enable_embeddings=False)
chunker.process_directory()

Metadata

def custom_metadata_extractor(text: str) -> dict:
    # Extract domain-specific metadata
    return {"word_count": len(text.split())}

chunker.extract_metadata_from_text = custom_metadata_extractor

Troubleshooting

spaCy model not found

python -m spacy download en_core_web_sm

"Azure OpenAI API key is required when embeddings are enabled" Embeddings are on by default. Pass --bm25-only, or set AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT.

Azure API errors

• Check your API key and endpoint
• Check rate limits
• Make sure the embedding model deployment name matches embedding_model

Memory issues with large PDFs

• Reduce --max-workers
• Process files individually

No text extracted from a PDF The PDF probably has no text layer (a scanned document). OCR is not included in this repository yet.

Debug mode

--verbose goes before the command:

python main.py --verbose convert --input pdfs/ --output text/

System check

from src.utils import print_system_summary
print_system_summary()   # needs psutil

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/my-change)
3. Commit your changes
4. Push the branch and open a pull request

git clone https://github.com/AceAtDev/RAG-chunker.git
cd RAG-chunker/pdf2text2chunks
pip install -r requirements.txt

black .
mypy src/

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

Acknowledgments

• Built with PyPDF2, OpenAI, spaCy, rank_bm25, and other open-source libraries
• Thanks to the Arabic NLP community for guidance on multilingual processing

Support

• Issues: GitHub Issues
• Email: aceatdeveloping@gmail.com