> State of Local AI, 2026.

Three sentences, sharable verbatim:

Local AI is no longer fringe. A used RTX 3090 runs Llama 3.3 70B at usable speed. Apple Silicon ships as a serious inference platform out of the box. The remaining gap to frontier-cloud models is real — but for chat, code, and RAG, local is the default, not the alternative.

The interesting question isn't “is local viable yet” — that's settled. It's “under what specific workloads does cloud still win, and how fast is that gap closing.” The honest 2026 answer: long-context reasoning over 200K, frontier multimodal, anything needing the latest weights week-of-release.

The 2026 story is consolidation rather than revolution. Three platforms now matter for local inference, and each owns a clear slice of the buyer decision:

The mobile/NPU class — Snapdragon X, Lunar Lake, current Apple M-series laptops — turned out narrower than vendors promised. These ship with NPUs marketed for 45+ TOPS that nobody uses for LLM inference because the software toolchain isn't there. The actual LLM inference path on these machines runs on the GPU/CPU, not the NPU. NPU marketing is ahead of NPU utility by about 18 months.

What we got right last year: the 3090 staying relevant through a full new generation. What we got wrong: we underestimated how fast Apple Silicon would close the gap on dense models. Top-tier Mac Studios at $5,000 now compete credibly with $8,000 dual-GPU CUDA builds for everything except multi-user inference.

The open-weight frontier moved roughly one model-class behind the closed frontier this year (70B-dense open vs 400B+ closed; 6–12 month release lag; ~15–25% gap on common reasoning benchmarks) — a faster gap-close than 2025. Three shifts matter:

The 14B-class sweet spot got crowded — in a good way

Three families now compete head-to-head for the most-asked workload (chat / RAG / light coding on 12-16GB consumer hardware): Phi-4 14B (with reasoning + multimodal variants), Gemma 3 12B (Google's small-model line), and the perennial Qwen 2.5 / 3 14B class. Each has a different tradeoff — Phi-4 leads on reasoning + multimodal, Gemma 3 on Google ecosystem compatibility, Qwen on raw breadth and Apache 2.0 licensing. For most buyers the differences sit inside the noise floor.

MoE is the new dense

Qwen 3 235B-A22B, DeepSeek V3, and Llama 4 Scout / Maverick all use sparse mixture-of-experts. The implication for buyers: VRAM gates total params, but speed gates active params. A 235B/22B MoE on 96GB of pooled VRAM runs faster than a 70B dense on the same hardware, while delivering broader capability. The mental model for “how big a model can I run” needs both numbers now.

Coding-specific models earned their keep

DeepSeek-Coder-V2, Qwen 2.5 Coder, and the StarCoder lineage now produce code that rivals frontier APIs for tasks in their training distribution. The pattern that actually shipped in 2026: 14-32B coder models pair with Aider / Cursor / Continue for the editor loop, with a larger generalist model held in reserve for design-level questions.

Vision finally became deployable

Llama-3.2-Vision (11B and 90B variants) and Pixtral cleared the “actually useful for daily work” bar. 2025 vision models could caption photos; 2026 vision models can extract structured data from a screenshot, parse a UI mockup, transcribe a whiteboard. The 12GB-VRAM floor for an 11B vision model is what made this accessible.

Four runtimes now matter, each owning a clear seat:

• Ollama — the default front door. Wins on developer ergonomics. The vast majority of “I want to run a local model” first-time installs go here. The cost is leaving 20-30% of throughput on the table vs ExLlamaV2 or vLLM.
• llama.cpp — the engine underneath. GGUF format won the quant wars. Custom kernels, every architecture supported, runs on anything from a Pi to a 4090.
• vLLM — multi-user inference, paged attention, continuous batching. The right answer when more than one human is hitting the rig at once. Overkill for a single-operator setup.
• MLX / mlx-lm — the Apple-native path. Higher throughput than llama.cpp Metal, fewer features. Best when you're sure you're staying on Apple silicon.

What didn't happen: no fifth runtime emerged. The consolidation is real and probably permanent. Bet accordingly when picking a stack.

2026 was the year “agentic AI” stopped being a demo and started being deployable. Three patterns earned their place on the catalog, and one specific model family made local agents viable at all.

Hermes 3 / 4 as a popular local-agent default

Nous Research's Hermes 3 (and now 4) Llama fine-tunes have emerged as a popular default for tool-use on local hardware — observed across community recipes and the most- cited recommendations on r/LocalLLaMA + Ollama threads, though we don't hold an audited usage count. The 8B Hermes 3 runs comfortably on 12GB cards and handles structured function-calling reliably enough for an autonomous loop — see model page. The 70B variant covers the larger agent context windows where the 8B starts losing thread. Both are now first-class targets in the catalog alongside Llama, Qwen, and Phi.

Coding-agent stack consolidation

The ecosystem coalesced around three coding-agent shapes:

• Terminal-driven — Aider + a 32B coder. The lowest-friction path. Git-aware, diff-first, runs against any OpenAI-compatible local endpoint.
• VS Code-integrated — Cline and Roo Code. Autonomous loops inside the editor. Multi-mode personas and project-level rules make them version-controllable.
• Browser-native — OpenHands and Bolt.diy. Full app generation with sandboxed execution. The “show your friend what local AI can do” demos.

For the full side-by-side — runtime model support, autonomy levels, repo-awareness, and 2026 release cadence across all of the above plus Continue, Cursor, Cody, and the long tail — see the coding-agents map (2026).

The local-first agent wave (messaging-native UX)

The breakout 2026 story isn't a multi-agent framework — it's an entire new category of personal-AI agents that run locally and reach users through messaging platforms (WhatsApp, Telegram, Slack, Discord, iMessage, Signal) instead of a dedicated browser tab. The breakout reference is OpenClaw (~347k★ as of May 2026, now foundation-stewarded), but the category already has imitators (NVIDIA's NemoClaw, Letta experimenting with messaging bridges, several smaller forks). Backend-agnostic — every option pairs with Ollama + Hermes 3. Honest deep-dive at /guides/openclaw-personal-ai-agent-2026.

Multi-agent frameworks settled on three winners

The orchestration layer split into three credible frameworks: AutoGen for free-form multi-agent conversation, CrewAI for role-based crews, LangGraph for deterministic graph flows. All three route through OpenAI-compatible endpoints, so any local runtime (Ollama, vLLM, llama.cpp server) is a valid backend. The smaller smolagents from HuggingFace earned a niche as the “CodeAgent writes Python instead of JSON tool calls” alternative — faster but riskier without sandboxing.

Memory frameworks went from research to standard

Letta (formerly MemGPT) and Mem0 moved from interesting papers into actually-shipped dependencies. Multi-turn agentic workflows that need to remember across days now have a sane default. The memory-frameworks map tracks the active set.

The 2025 question was “can agents work at all on local models?” The 2026 answer is yes — with Hermes 3 / 4, a 32B coder, and any of the three big frameworks, autonomous loops feel less like a hype demo and more like an actual tool.

• Used GPU markets stabilized. The 2023-era panic-buying premium is gone. Used 3090s at $700, used A6000s at $2,800. The market priced in that used GPUs work fine for inference.
• Quantization research delivered. IQ3_M and Q4_K_M are now the universal defaults; AWQ and GPTQ-Marlin saw real production use. Q2 is finally viable for the largest models on small hardware.
• Coding agents that work. Aider + a 32B coder is now a real alternative to Copilot for the bulk of solo developer work.
• Local RAG as a default workflow. AnythingLLM, Open WebUI's document layer, and LangChain-on-Ollama removed the friction. “Drop a folder of PDFs, ask questions” is now a 5-minute install.

• The NPU story. Two years in, no mainstream local-LLM workflow uses the marketed NPU compute on Snapdragon X / Lunar Lake. These chips run LLMs fine — on their GPU/CPU. NPU marketing promises haven't matched software-stack reality yet.
• Dedicated AI accelerators below H100. Tenstorrent, Groq cards, others — promising hardware that lacks the software ecosystem to be a credible local-inference target for individuals. They'll stay datacenter or stay niche.
• Voice cloning at scale. Whisper for STT is solid. TTS that sounds genuinely human, fast, on local hardware, with arbitrary voice — still not there. The closest models (XTTS-v2, Kokoro, Bark) all trade quality, speed, or flexibility.
• Local long-context. Frontier APIs ship 200K-1M context. Local stops being viable around 32K on most consumer hardware. The KV cache math is brutal at scale and no consumer setup has the VRAM headroom for the high-context workloads that the frontier enables.

Bets we'd actually take if we had a year-long horizon:

The bet we're not making: that any new hardware vendor cracks the inference-accelerator market against NVIDIA + Apple. The CUDA + MLX moat is wider than it looks, and the second-tier vendors that exist still treat consumer inference as an afterthought.

Live counts from the RunLocalAI catalog as of 2026-05-14 — the same database that drives every page on this site: