Ollama

Ollama is a runtime and CLI tool for running large language models locally on consumer hardware. It wraps llama.cpp and other backends to handle model downloading, quantization, and inference with a simple command-line interface. Operators use Ollama to pull models from a curated library, load them into VRAM, and serve them via a REST API or interactive chat. It abstracts away manual model file management and backend configuration, making local LLM deployment accessible on Windows, macOS, and Linux.

Ollama simplifies the process of running LLMs by providing a single command to download and run models. Under the hood, it uses llama.cpp for CPU/GPU inference with support for various quantization levels (Q4, Q8, etc.). It manages model files in a local cache (~/.ollama/models) and automatically selects the best backend based on available hardware (CUDA, Metal, Vulkan). Ollama also offers a REST API compatible with OpenAI's API format, enabling integration with tools like Open WebUI or custom scripts. Its model library includes popular models like Llama, Mistral, and Gemma, but operators can also import custom GGUF files. The runtime handles context management, batching, and offloading to system RAM when VRAM is insufficient.

On an RTX 3090 with 24 GB VRAM, running ollama run llama3.1:8b loads the Q4 quantized model (~5 GB) and achieves ~40 tokens/sec. The same command on a 16 GB card might offload some layers to system RAM, dropping speed to ~10 tok/s. Operators can adjust context length with --num-ctx 4096 to fit larger sequences within VRAM.

An operator starts by pulling a model: ollama pull llama3.1:8b. This downloads the GGUF file into ~/.ollama/models/blobs. Then they run ollama run llama3.1:8b to start an interactive session. For API access, they run ollama serve and send requests to http://localhost:11434/api/generate. To use a custom model, they create a Modelfile with FROM and PARAMETER directives, then run ollama create mymodel -f Modelfile.