Visual RAG

1. pip install colpali-engine (ColPali — retrieve documents by visual appearance, no OCR needed).
2. Download the model: ColPali/ColQwen2 will auto-download on first use (~2-5 GB).
3. Index your PDF documents as page images:

from colpali_engine import ColPali
import fitz  # PyMuPDF
model = ColPali.from_pretrained("vidore/colpali-v1.2")
pdf = fitz.open("report.pdf")
page_embeddings = []
for page in pdf:
    pix = page.get_pixmap(dpi=200)
    img = pix.tobytes("png")
    emb = model.embed_images([img])  # multi-vector embedding per page
    page_embeddings.append(emb)

1. Query: query_emb = model.embed_queries(["Q3 revenue chart"]) → similarity search over page embeddings → returns the most visually-similar pages.
2. First retrieval in <1 second per query on GPU, ~5 seconds on CPU for a 1,000-page corpus.
3. The key advantage: ColPali retrieves based on visual layout, not OCR text. It finds the page with the revenue chart even if "revenue" isn't in the OCR text (embedded in the chart image).

Visual RAG is surprisingly hardware-friendly. ColPali (Vision Transformer based) indexes 5-10 pages/second on CPU, ~50-100 pages/second on a used GTX 1060 6 GB ($60). A 10,000-page corpus indexes in ~15-30 minutes on a $100 GPU. Retrieval is sub-second on any hardware. Pair with Ryzen 5 5600 + 32 GB DDR4 + 1TB NVMe (for storing page images). Total: ~$320-370. For the full pipeline (retrieve pages → VLM answers), add 12 GB VRAM for a 7B VLM. Total with VLM: ~$400-500. Visual RAG is one of the most practical $300-500 local AI setups.

Used RTX 3090 24 GB (~$700-900, see /hardware/rtx-3090). Runs the full Visual RAG pipeline end-to-end: ColQwen2 indexing at 200+ pages/second, retrieval in <100ms, Qwen2-VL 72B for answer generation. Handles 1M+ page corpora with sub-second query latency. For enterprise Visual RAG (legal document archives, scientific paper libraries): combine with a vector DB (Qdrant) that supports multi-vector ColBERT-style indexing. Total: ~$1,800-2,200. Visual RAG eliminates the brittle OCR step entirely — a paradigm shift for document retrieval.

The mistake: Running OCR on every page, embedding the OCR text, and calling it "visual RAG" — then being confused why the system can't find a page with an embedded chart. Why it fails: OCR extracts text, not visual structure. A chart labeled "Q3 Revenue" with bars showing $10M, $12M, $15M might OCR as "Q3 Revenue $10M $12M $15M" with no indication these are values on a chart. The embedding of that OCR text is similar to any page mentioning "Q3 Revenue" — it can't distinguish a chart from a text mention. The fix: Use true Visual RAG (ColPali/ColQwen) that embeds the page image directly. The multi-vector embedding captures visual patterns — a chart page looks visually different from a text page, even if the OCR text is similar. ColPali finds the chart because it "sees" bars and axes, not because it reads "revenue."