AMD Radeon RX 6800 XT

This card is for the operator who wants a 16 GB VRAM workhorse for local LLMs without paying Nvidia's premium. The RX 6800 XT runs 7B Q4 models at 55-75 tok/s and 13B Q4 at ~30-40 tok/s, comfortably handling 30B Q4 models in the ~15-20 tok/s range. The 16 GB VRAM fits 30B Q4 with room for context, and even 70B Q2 or Q3 quantizations can run, though slowly (5-8 tok/s).

What breaks: ROCm is officially supported but still has rough edges compared to CUDA. Some inference frameworks (e.g., llama.cpp with ROCm backend) may need specific versions or patches. Flash attention and other optimizations are less mature on AMD. The card also lacks tensor cores, so compute-heavy operations like training or large batch inference are slower than Nvidia equivalents.

When to pass: If the workload requires CUDA-only features (e.g., vLLM, TensorRT-LLM) or if the operator wants plug-and-play compatibility with the widest range of tools. Also pass if training or fine-tuning is a primary use case, as Nvidia's ecosystem is significantly more mature.

Price/value note: At ~$450 used, this card offers the best VRAM-per-dollar for local inference, beating similarly priced Nvidia options (RTX 3080 10 GB, RTX 4060 Ti 16 GB) on both VRAM and bandwidth.