Qwen 3 8B

Positioning

Qwen 3 8B introduces a hybrid "thinking" / "non-thinking" toggle into the 7B class. In non-thinking mode it's a tier with Qwen 2.5 7B; in thinking mode it produces visible chain-of-thought and lifts hard-task performance closer to 14B-class models at the cost of latency.

Strengths

• Hybrid reasoning toggle — /think and /no_think per turn lets you pay for reasoning only when needed.
• Improved tool-use over Qwen 2.5 — function-call format more standardized.
• Strong multilingual carryover from the 2.5 generation.

Limitations

• Thinking-mode output is verbose — tokens-per-answer roughly doubles, eating speed.
• Some prompt-injection vectors specific to the /think toggle that haven't been fully audited.
• License remains Qwen-flavored with usage caps.

Real-world performance on RTX 4090

• Q4_K_M (5.0 GB): 95–115 tok/s decode (non-thinking); 90–110 tok/s thinking but 2× output
• Q5_K_M (5.9 GB): 85–100 tok/s
• Q8_0 (8.4 GB): 65–82 tok/s

Should you run this locally?

Yes, for users who want the best 8B-class capability and are willing to use thinking mode selectively for hard prompts. No, for users who don't need reasoning — Qwen 2.5 7B is simpler and similar speed.

How it compares

• vs Qwen 2.5 7B → Qwen 3 8B with thinking mode wins on reasoning; without thinking, near-equal. Pick Qwen 3 if reasoning matters.
• vs Llama 3.1 8B → Qwen 3 8B wins on raw capability; Llama wins on instruction polish + ecosystem maturity.
• vs QwQ 32B → QwQ is the dedicated reasoning specialist at 32B; Qwen 3 8B's thinking mode is a poor man's QwQ at lighter VRAM.
• vs Phi-4 14B → Phi-4 has cleaner reasoning at higher VRAM; Qwen 3 8B fits in less memory.

Run this yourself

ollama pull qwen3:8b
ollama run qwen3:8b
# Toggle reasoning per turn:
#   /think    — enable chain-of-thought
#   /no_think — disable

Settings: Q4_K_M GGUF, 8192 ctx, llama.cpp/CUDA, RTX 4090