Qwen 2.5 Math 72B

Qwen 2.5 Math 72B is Alibaba's math-specialized 72B dense model. Run at Q4_K_M via Ollama (ollama pull qwen2.5-math:72b) or llama.cpp with -ngl 999 -fa -c 8192. Q4_K_M file size ~41 GB on disk. Minimum VRAM: 48 GB — RTX A6000 (48GB) at Q4_K_M for 4K context. RTX 4090 24GB: Q3_K_M with KV offload. Recommended: A100 80GB at AWQ-INT4 for serving. Throughput: ~12-20 tok/s on A6000 at Q4_K_M; ~25-40 tok/s on A100. Qwen 2.5 architecture — well-supported in llama.cpp, vLLM, and Ollama. Math specialization means this model was trained with math-specific data and reasoning formats. Use for: theorem proving, mathematical reasoning, step-by-step problem solving, competition math. Not ideal for: general chat, creative writing, coding (use Qwen 2.5 Coder instead). The math tuning may make the model verbose on non-math tasks. Context: 32K advertised; practical at Q4 on 48 GB is 8-16K.

Minimum: RTX 3090 24GB at Q3_K_M with KV offload (4K context). Recommended: RTX A6000 48GB at Q4_K_M (8K context). Optimal: A100 80GB at AWQ-INT4. VRAM math: 72B dense, Q4_K_M ≈ 41 GB. KV cache at 8K: ~10 GB. Total: ~51 GB at 8K. Single A6000 48GB: borderline at 8K — trim to 4K or offload KV. RTX 4090 24GB: Q3_K_M ≈ 31 GB + KV offload. RTX 5090 32GB: Q4_K_M ≈ 41 GB — must offload KV to RAM. Mac Studio M4 Max 64GB: Q4_K_M at 5-10 tok/s. Dual RTX 4090 48 GB: Q4_K_M at 8K context — viable. Cloud: A100 at $5-10/hr.

1. Math formatting expectations. Qwen 2.5 Math expects problems formatted with specific delimiters (e.g., "Problem:" / "Solution:"). Free-form prompts may trigger verbose, off-target outputs. 2. General-task quality degradation. The math specialization narrows the model's general knowledge. Non-math factual queries may hallucinate more than base Qwen 2.5 72B. 3. Quantization sensitivity on math. Mathematical reasoning degrades more sharply at Q3 than general language tasks. The precise numerical weights matter for arithmetic fidelity. Use Q4_K_M minimum for math. 4. Chain-of-thought explosion. The model is trained to produce detailed CoT — generation length can be 3-5× longer than non-math models for the same problem. Budget extra tokens in your pipeline.

llama.cpp for local math work — precise control over temperature (set to 0 for deterministic math). Ollama for quick testing. vLLM for serving. Qwen 2.5 is well-supported. For math specifically, set temperature=0 and top_p=0.95 — avoid high temperature (0.7+) which introduces arithmetic noise.

Mistake: Using high temperature for math. Fix: Temperature >0.3 introduces randomness that corrupts arithmetic. Use temp=0 for deterministic math outputs. Mistake: Expecting Qwen 2.5 Math to be good at coding. Fix: Math tuning ≠ code tuning. Use Qwen 3 Coder 32B for code. Mistake: Truncating CoT output mid-generation. Fix: Qwen 2.5 Math's CoT is verbose — configure max_tokens high enough (4K+) to avoid cutting off the solution. Mistake: Using Q3 quantization for math. Fix: Math is precision-sensitive. Q4_K_M minimum; Q8 if VRAM permits. Q3 loses ~5-10% on math benchmarks vs Q4.