MLC LLM

MLC LLM (Machine Learning Compilation for Large Language Models) is a framework that compiles LLMs into deployable binaries for a wide range of hardware, including consumer GPUs, Apple Silicon, mobile devices, and web browsers. It uses Apache TVM to optimize model execution through operator fusion, memory planning, and quantization. Operators encounter MLC LLM when they need to run models on non-NVIDIA hardware (e.g., AMD GPUs, Apple M-series) or on edge devices where standard runtimes like llama.cpp lack support. The framework produces platform-specific executables that can run models efficiently without requiring a full Python stack at inference time.

MLC LLM is built on top of Apache TVM, an open-source machine learning compiler. It takes a model in a standard format (e.g., Hugging Face Transformers) and compiles it into a shared library or executable tailored to the target hardware. The compilation process includes automatic operator scheduling, memory optimization, and optional quantization (e.g., INT4, INT8). Unlike llama.cpp, which is CPU-first with GPU offload, MLC LLM is designed to exploit GPU acceleration across vendors via Vulkan, Metal, CUDA, and OpenCL backends. This makes it a strong choice for running LLMs on AMD GPUs (via ROCm or Vulkan) or on Apple Silicon (via Metal). MLC LLM also supports WebGPU for browser-based inference. The framework includes a chat CLI, a REST server, and Python/C++ APIs. Its main trade-off is longer compilation time compared to just loading a GGUF file, but the resulting binary can be more performant on non-CUDA hardware.

On an AMD RX 7900 XTX (24 GB VRAM), running Llama 3.1 8B via llama.cpp with Vulkan offload might achieve ~30 tok/s. Using MLC LLM with the Vulkan backend, the same model can reach ~45 tok/s due to better operator fusion and memory scheduling. The trade-off: MLC LLM requires compiling the model first, which takes ~10-15 minutes, whereas llama.cpp loads a GGUF file in seconds.

To run a model with MLC LLM, an operator first installs the package (pip install mlc-llm) and then compiles the model: mlc_llm compile --model Llama-3.1-8B --target vulkan -o lib.so. This produces a shared library. Then they run the chat CLI: mlc_llm chat lib.so. For a REST server, they use mlc_llm serve lib.so --port 8080. The compilation step is unique to MLC LLM; other runtimes skip it by loading pre-quantized weights directly.