Hallucination

Hallucination is when an LLM generates plausible-sounding but factually incorrect information — citing papers that don't exist, inventing API methods, fabricating quotes. It's not a bug; it's a fundamental property of next-token prediction trained on noisy text.

Hallucination correlates inversely with model size and training data quality. Smaller models hallucinate more. Reasoning-tuned models (DeepSeek R1, QwQ, Phi-4 Reasoning) hallucinate slightly less because they fact-check intermediate steps in their visible chain-of-thought.

Mitigations: RAG (ground in real documents), tool use (let the model call calculators/APIs/databases), prompt engineering (ask for citations, acknowledge uncertainty), and avoiding tasks where the model will obviously make things up (asking a 3B model for niche historical facts).