Qwen 2.5-VL 72B

Qwen 2.5 VL 72B is Alibaba's 72B vision-language model — 72B dense text backbone + Qwen-VL vision encoder. Run at Q4_K_M via llama.cpp with llava-server for vision, or Ollama if the VL tag is available. Q4_K_M file size ~41 GB (text) + ~4-6 GB (vision). Minimum VRAM: 48 GB — RTX A6000 at Q3_K_M with vision, or text-only Q4_K_M. Recommended: A100 80GB at AWQ-INT4 for full vision serving. Throughput: ~10-18 tok/s on A6000 at Q4_K_M text-only; vision encoding adds ~2-4s per image. Qwen-VL's vision encoder is well-optimized — lower VRAM overhead than InternVL's InternViT. Strong OCR, document understanding, and visual reasoning. 128K context advertised; practical for vision at Q4 on 80 GB is 4-8K. Qwen 2.5 VL uses the same text backbone as Qwen 2.5 72B — broad ecosystem support. For production: vLLM with multimodal pipeline and tensor-parallel if needed.

Minimum: RTX A6000 48GB at Q3_K_M + vision (tight). Recommended: A100 80GB at AWQ-INT4. VRAM math: 72B text at Q4_K_M ≈ 41 GB. Qwen-VL encoder: 3-5 GB. KV cache at 8K: ~10 GB. Total with vision: ~54-56 GB. A6000 48GB: must use Q3_K_M (31 GB) + vision at 4K context. A100 80GB: comfortable for Q4 + vision + 8K. Dual RTX 4090 48 GB: Q4_K_M text-only, or Q3_K_M + vision. Mac Studio M4 Ultra 128GB: Q4_K_M + vision, 3-6 tok/s. Cloud: A100 at $5-10/hr. Qwen-VL's encoder is smaller than InternViT — better VRAM efficiency for vision. AWQ-INT4 drops text weights to ~36 GB, enabling 16K+ context on 80 GB.

1. Vision tag availability in Ollama. Qwen 2.5 VL may not have an official Ollama tag. Community tags may exist but aren't verified. Test with raw llama.cpp if Ollama fails. 2. Image preprocessing mismatch. Qwen-VL expects specific image preprocessing (resolution, normalization). Feeding raw images without preprocessing degrades vision quality. Use the model's image processor from HF transformers. 3. KV cache with vision. Vision tokens are prepended to the text prompt — each image adds 256-1024 tokens to context. Multiple images inflate context and KV cache proportionally. Budget for image tokens. 4. Qwen 2.5 vs Qwen 3 VL. Qwen 2.5 VL and Qwen 3 VL use different vision encoders. Don't mix model files or chat templates between versions.

llama.cpp with llava-server for local vision. vLLM for production multimodal serving. Qwen-VL is supported in vLLM's multimodal pipeline. Ollama for quick-start if the VL tag is available. MLX-LM on Apple Silicon if Qwen-VL is supported.

Mistake: Using a standard text-only GGUF and expecting vision to work. Fix: Vision requires a multimodal GGUF with the Qwen-VL encoder included. Download from bartowski or convert from hf. Mistake: Ignoring image token count in context budget. Fix: Each image in Qwen-VL consumes 256-1024 tokens. Subtract image tokens from your available context window. Mistake: Using Llama 3.2 Vision chat template for Qwen-VL. Fix: Different architectures, different templates. Use Qwen's chat template from tokenizer_config.json. Mistake: Sending large images without preprocessing. Fix: Qwen-VL expects images within a specific resolution range. Use Qwen's image processor or resize manually before inference.