Jamba 1.5 Large

Jamba 1.5 Large is AI21's SSM-hybrid model (94B total, ~52B active via MoE). The SSM backbone means KV cache overhead is lower than pure attention — practical for long-context workloads. Run at Q4_K_M via llama.cpp with -ngl 999 -fa -c 32768. Q4_K_M file size ~45-55 GB on disk. Minimum VRAM: 48 GB — RTX A6000 works at 32K context. RTX 4090 24GB needs Q3_K_M (35 GB) with partial offload or dual-card setup. Recommended: single RTX A6000 48GB at Q4_K_M, or dual RTX 3090 (24GB each) at Q3_K_M row-split. SSM layers reduce per-token KV cache growth, so 32K context is realistic on 48 GB. Throughput: ~15-25 tok/s on RTX A6000 at 32K bulk decode. The SSM architecture limits token-by-token generation speed compared to pure attention on GPU-heavy setups, but the reduced KV cache footprint enables much longer contexts.

Minimum: RTX 3090 24GB at Q3_K_M (4K context). Recommended: RTX A6000 48GB at Q4_K_M (32K context). VRAM math: ~94B total, active subset ~52B. SSM layers (Mamba) use fixed state size regardless of context — this is the key advantage. Q4_K_M ~0.7 bytes/param for active subset ≈ 36 GB for weights. KV cache for attention layers: ~8-15 GB at 32K context (less than pure-attention models because only ~30-50% of layers are attention). Total: ~44-51 GB at 32K on Q4_K_M — tight fit on 48 GB. Mac Studio M4 Max 64GB can run Q4_K_M at 16K context. RTX 6000 Ada 48GB gives headroom. Cloud: single A10 24GB at Q3_K_M works for testing.

1. SSM kernel on older GPUs. Mamba kernels require CUDA 11.8+ and SM 7.5+ (Turing or newer). GTX 10-series (Pascal) won't run Jamba. Verify CUDA compute capability before deploying. 2. Ollama SSM support is immature. Jamba's hybrid architecture may not be fully wired in Ollama's default llama.cpp backend. Test with raw llama.cpp first. 3. Per-token latency on SSM layers. SSM decode is sequential — generation speed at small batch sizes is slower than attention on high-end GPUs. Jamba trades throughput for context efficiency. 4. Training data cutoff. Jamba 1.5's knowledge stops at its training date. RAG or web grounding needed for current information.

llama.cpp with -ngl 999 is the primary option — it has the most mature Jamba/SSM support. vLLM may have experimental Jamba support but verify before committing. Avoid Ollama unless you confirm Jamba 1.5 Large is in their supported model list. Avoid MLX-LM — SSM kernel on Apple Silicon is not optimized.

Mistake: Assuming Jamba runs like a standard Llama. Fix: Jamba is SSM-hybrid — different architecture, different bottlenecks (sequential SSM decode vs parallel attention). Benchmark your workload specifically. Mistake: Expecting 100+ tok/s decode on RTX 4090. Fix: SSM layers bottleneck per-token generation. Jamba Large at Q4_K_M on RTX 4090-class hardware gets ~15-25 tok/s — not 80+. Long-context efficiency is the tradeoff. Mistake: Setting 256K context and expecting it to work. Fix: While SSM enables longer contexts than pure attention, 256K on 48 GB is unrealistic. Start at 32K, benchmark, and scale up. Mistake: Using automatic mixed precision (AMP) without testing. Fix: SSM precision sensitivity differs from attention. Q4_K_M is well-tested; custom quants may produce numerical instability.