2026-03-28 · William Kennedy
How to Set Up Ollama for Local AI Inference
Ollama makes running local LLMs dead simple. Here's how to install it, download models, and integrate it with your applications.
What Is Ollama?
Ollama is a tool that makes running large language models locally as easy as running a Docker container. It handles model downloading, GPU optimization, and provides a simple API for inference.
Installation
On Linux: curl -fsSL https://ollama.com/install.sh | sh
On macOS: Download from ollama.com
On Windows: Use WSL2 or the native installer
Running Your First Model
ollama run llama3.1:8b
That's it. Ollama downloads the model and starts an interactive session. For API access, it runs on localhost:11434 by default.
GPU Requirements
- 7B models: 8GB VRAM minimum
- 13B models: 16GB VRAM recommended
- 70B models: 48GB+ VRAM or quantized versions
We help businesses size their hardware correctly and optimize Ollama for production workloads.