DeepSeek MoE 16B Base

DeepSeek MoE 16B Base is DeepSeek's small Mixture-of-Experts base model — 16B total parameters with ~2.8B active per token. Ultra-efficient MoE architecture: 16B total for broad knowledge, 2.8B active for fast generation. This is a base model — not instruction-tuned, not chat-ready. Generates completions, not responses. Run at Q4_K_M via llama.cpp with -ngl 999 -fa -c 8192. Q4_K_M file size ~9 GB on disk. Minimum VRAM: 6 GB — RTX 2060 (6GB) at Q4_K_M with expert offload. RTX 3060 12GB: Q4_K_M with all experts in VRAM. Recommended: any GPU with 8+ GB at Q4_K_M. Throughput: ~80-120+ tok/s on RTX 4090 at Q4_K_M — extremely fast due to 2.8B active. DeepSeek MoE architecture — verify llama.cpp support for DeepSeek MoE specifically. Designed as a research base model: fine-tune for specific tasks, use for few-shot completion, or as a fast embedding/labeling model. Strong for its size on: text completion, classification, simple extraction. Not for: direct chat (no instruction tuning), complex reasoning (2.8B active limits), creative generation. Context: 4K baseline (DeepSeek MoE); short context is fine for base model use cases. For instruction-tuned small MoE: Granite 3 MoE 3B-Active. For larger DeepSeek base: DeepSeek V3 Base.

Minimum: 4 GB RAM CPU-only at Q4_K_M (~4-8 tok/s). Recommended: any GPU with 6+ GB at Q4_K_M. VRAM math: 16B total, ~2.8B active. Q4_K_M ≈ 9 GB for full weights. Expert offload: ~2 GB active experts in VRAM. KV cache at 4K: ~1 GB. Total with all experts in VRAM: ~10 GB — fits 12 GB GPUs easily. RTX 2060 6GB: Q4 with expert offload at 4K. RTX 3060 12GB: all experts on-GPU. RTX 4090 24GB: overkill — 120+ tok/s. CPU-only on modern laptop: 5-12 tok/s. Raspberry Pi 5 8GB: Q4 at 3-6 tok/s. This is one of the most deployable models — fits anywhere. The 2.8B active makes it ideal for high-throughput, low-latency applications where quality requirements are modest.

1. Base model, not chat. No instruction tuning means raw completions. For chat, use DeepSeek-Chat or an instruct-tuned variant. Few-shot prompting can approximate chat but quality varies. 2. 2.8B active ceiling. The active parameter count limits reasoning depth. Complex tasks that need multi-step reasoning will fail. This is a lightweight model — know its limits. 3. DeepSeek MoE architecture. Not standard Mixtral MoE — verify llama.cpp supports DeepSeek's specific MoE implementation. Shared experts + routed experts differ from Mixtral/Dbrx. 4. Fine-tuning complexity. Fine-tuning a MoE model is more complex than a dense model — expert routing adds training instability. Use established MoE fine-tuning recipes (QLoRA on routed experts, etc.).

llama.cpp for local use — CPU and GPU backends. Ultra-lightweight makes it ideal for CPU-only deployment. vLLM for serving (verify DeepSeek MoE support). Avoid Ollama for base model — no chat template, Ollama is designed for instruct/chat. For fine-tuning: Axolotl or Unsloth with MoE-aware config.

Mistake: Chatting with DeepSeek MoE Base and wondering why responses are garbled continuations. Fix: Base models complete text — they don't follow instructions. Use few-shot completion format or fine-tune. Mistake: Expecting 16B dense quality from a 16B MoE. Fix: Quality is driven by active parameters (~2.8B), not total parameters. The model has broad knowledge from 16B training but limited reasoning depth. Mistake: Using standard Mixtral GGUF conversion scripts. Fix: DeepSeek MoE differs from Mixtral/Dbrx MoE. Use DeepSeek-specific conversion scripts. Mistake: Fine-tuning with standard LoRA on all layers. Fix: MoE fine-tuning requires careful handling of expert routing layers. Use MoE-aware QLoRA or only fine-tune specific expert subsets.