ROCm

What ROCm actually is

ROCm is AMD's open-source GPU compute stack — the equivalent layer to NVIDIA's CUDA. It includes the HIP programming model (a C++ runtime that ports cleanly between AMD and NVIDIA at the source level), HIPify tooling that auto-translates CUDA source to HIP, and accelerated math libraries (rocBLAS, rocFFT, MIOpen) that AMD GPUs need to compete with cuDNN-class performance on AI workloads.

Crucially, ROCm is what makes AMD GPUs exist in the local-AI conversation at all. Without it, every other tool on this site that supports AMD — llama.cpp, vLLM, SGLang, PyTorch — would silently fall through to CPU inference. ROCm is the runtime layer; the tools above are clients of it.

Where it fits in the stack

ROCm is a driver / runtime layer, not an inference engine. The stack:

• Hardware: AMD GPU — RDNA3 (RX 7900 series, MI300, MI250) is the realistic 2026 floor for serious AI work
• Driver / runtime: AMD's amdgpu kernel driver + ROCm userspace
• Compute libraries: rocBLAS, MIOpen, hipBLASLt, rocSPARSE
• Inference engines: llama.cpp, PyTorch (with ROCm wheels), vLLM, SGLang, ExLlamaV2 (limited), bitsandbytes (limited)
• Frontends: Ollama, LM Studio, Open WebUI

Picking AMD for local AI in 2026 means picking the ROCm path. There's no second option, and the size of the gap to CUDA is the size of the gap to whatever ROCm ships next.

Best use cases

• High-VRAM-per-dollar inference workstations. A used RX 7900 XTX gives 24 GB VRAM at roughly half the price of a 24 GB NVIDIA equivalent. For solo / homelab inference, that delta is real.
• Datacenter MI300X / MI250X clusters. Where AMD has invested most heavily; ROCm 6+ is competitive with CUDA on H100-class workloads at the kernel level for the workloads AMD has tuned.
• PyTorch researchers with AMD hardware. ROCm PyTorch wheels are a one-line install; most research code runs unchanged.

OS support

For Windows AMD users, the practical truth in May 2026 is: ROCm on Windows works for llama.cpp and Ollama for most inference cases, but breaks under more advanced workloads (multi-GPU tensor-parallel, some PyTorch ops, FlashAttention variants). If you can dual-boot Linux, do it.

Hardware support

ROCm officially supports a narrower hardware list than CUDA. The 2026 working list:

• Datacenter: MI300X, MI300A, MI250X, MI250, MI210
• Consumer (RDNA3): RX 7900 XTX, RX 7900 XT, RX 7900 GRE, RX 7800 XT, RX 7700 XT
• Consumer (RDNA2): RX 6900 XT, RX 6800 XT (community-supported; gfx1030 target)
• Consumer (older): Vega 64, Vega VII (legacy support)

If you are buying new AMD for local AI in 2026, RDNA3 is the floor; older cards work but the Linux toolchain quality drops sharply below RDNA2.

Model / inference engine compatibility

This is where the rubber meets the road, and the picture has improved a lot but still has gaps:

• llama.cpp — full GGUF support via HIPBLAS; the most reliable AMD inference path in 2026
• PyTorch (ROCm wheels) — installs in one line; most research code runs unchanged
• vLLM (ROCm) — supported on MI300X / MI250X; consumer RDNA3 support exists but is rougher
• SGLang — partial AMD support; lags vLLM
• ExLlamaV2 — limited AMD support; the EXL2 quant kernels are CUDA-tuned
• bitsandbytes — partial; AMD support has been the long pole for a year+
• TensorRT-LLM — NVIDIA-only

Quant formats: GGUF works everywhere; AWQ / GPTQ work on PyTorch + ROCm but slower than NVIDIA equivalents; FP8 / EXL2 are NVIDIA-territory.

Setup path

On Ubuntu 24.04, the canonical ROCm 6.x install:

wget https://repo.radeon.com/amdgpu-install/latest/ubuntu/noble/amdgpu-install_*.deb
sudo apt install ./amdgpu-install_*.deb
sudo amdgpu-install --usecase=rocm
sudo usermod -aG render,video $USER
# log out and back in
rocminfo                    # should list your GPU

Then for PyTorch:

pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm6.0

For llama.cpp:

cmake -B build -DGGML_HIPBLAS=ON -DAMDGPU_TARGETS=gfx1100   # 7900 XTX
cmake --build build -j

What breaks first

1. Wrong gfx target compiled in. ROCm binaries are GPU-arch-specific. Building for gfx1100 (7900 XTX) and trying to load on gfx1030 (6900 XT) silently fails. See /errors/rocm-device-not-found.
2. Driver / ROCm version mismatch. amdgpu kernel driver and ROCm userspace must agree. Distro upgrades break this.
3. Out-of-tree kernel. Custom kernels (Zen4-tuned, TKG, etc.) need the amdgpu-dkms package re-built; usually fails silently.
4. vLLM / SGLang version drift. ROCm support tracks behind CUDA support by 1-2 minor versions; pin everything.
5. Multi-GPU tensor-parallel. Works on MI300X clusters; consumer RDNA3 multi-GPU TP is still flaky in 2026.

Alternatives by intent

Best pairings

• RX 7900 XTX + ROCm + llama.cpp = the cheapest 24 GB VRAM local inference path in 2026
• MI300X cluster + ROCm + vLLM = the AMD datacenter answer to H100 + TensorRT-LLM
• Ubuntu 24.04 LTS — the reference OS; everything else is harder

Who should avoid ROCm

• Time-constrained operators. ROCm has caught up enormously but still requires more debugging time than CUDA. If you are paid by the hour to ship local AI, NVIDIA is cheaper.
• Anyone whose stack depends on bleeding-edge NVIDIA-tuned kernels (FP8 transformer engine, latest FlashAttention variants, EXL2).
• macOS users. Use MLX-LM instead.

Related

• Hardware: RX 7900 XTX
• Tools: llama.cpp, vLLM, Ollama
• System guides: /setup, /compatibility
• Errors: /errors/rocm-device-not-found, /errors/wsl2-gpu-not-detected