The most common Ollama failure mode is pulling a model that doesn't fit your VRAM. This guide gives you exact model recommendations by GPU tier, explains quantization, and shows the fine-tuning pathway for when you outgrow off-the-shelf models.
Quick Answer
8 GB VRAM or unified RAM: run ollama run llama3.1:8b — the sweet spot of quality, speed, and compatibility. On Apple Silicon, treat unified memory as VRAM. On NVIDIA, use nvidia-smi to check free VRAM before pulling.
All recommendations use Q4_K_M quantization (Ollama's default) unless noted.
| VRAM / RAM | Recommended Models | Ollama Command |
|---|---|---|
| 4–6 GB | llama3.2:3b, phi3:mini, gemma2:2b | ollama run llama3.2:3b |
| 8 GB ★ | llama3.1:8b, mistral:7b, qwen2:7b | ollama run llama3.1:8b |
| 16 GB | llama3.1:70b (Q4), deepseek-coder:33b | ollama run llama3.1:70b |
| 24 GB+ | llama3.1:70b (Q8), any 13b at F16 | ollama run llama3.1:70b:q8_0 |
★ 8 GB is the recommended entry point — good model quality without hardware compromise.
Ollama uses Apple's Metal GPU framework on M-series chips. Unified memory is shared between CPU and GPU, so the full amount counts as effective VRAM.
M1 8 GB3B models (7B is on the edge — very slow)M1/M2 16 GB7–8B models comfortably; 13B is feasible at Q4M2/M3 Pro 32 GB13B at Q8, or 70B at Q4 (slow but works)M2/M3 Max 96 GB70B models at Q8, multiple models loaded simultaneouslyQuantization compresses model weights to smaller numerical precision, letting bigger models fit in less VRAM with a small quality trade-off.
Q4_K_MDefault — Best Balance
~60% memory reduction vs F16. Minimal quality loss for most tasks. What 'ollama pull' fetches by default.
Q8_0Higher Quality
~30% memory reduction vs F16. Near-lossless quality. Needs roughly 2× the VRAM of Q4. Good for coding tasks.
F16Full Precision
No compression. Full model quality. Only feasible on 24 GB+ VRAM workstations. Use for benchmarking.
NVIDIA: run "nvidia-smi" and look at "Free" memory under VRAM. AMD: use "rocm-smi". Apple Silicon: open Activity Monitor → Memory tab and note Unified Memory. Use the free (unoccupied) figure, not total.
Use the VRAM table above to pick a model, then run "ollama pull [model]". Example: "ollama pull llama3.1:8b". Downloads happen once; subsequent runs use the cached weights.
Start the model with "ollama run llama3.1:8b". In a second terminal, run "ollama ps" to see which model is loaded and how much VRAM it occupies. If inference is extremely slow, the model is CPU-offloading — consider a smaller quantization level.
Ollama exposes an OpenAI-compatible REST API at http://localhost:11434. You can point Open WebUI, Jan, or any OpenAI-SDK-compatible app at this endpoint to use your local model via a chat interface.
If off-the-shelf models don't fit your domain, fine-tune with Unsloth (LoRA, uses ~40% less VRAM than standard fine-tuning), export as GGUF via llama.cpp, then import with 'ollama create my-model -f Modelfile'.
HookFlow's live heat scores for the local AI ecosystem.
Run large language models locally with a simple CLI and REST API. Supports Llama, Mistral, Gemma, and dozens of other models out of the box.
View on HookFlow54
heat
Desktop app for running local LLMs with a ChatGPT-like interface. Supports GGUF models, model management, and a local API server compatible with OpenAI.
View on HookFlow18
heat
Jan is a free, open-source AI chat application that runs entirely on your local machine. Unlike cloud-based tools like ChatGPT, Jan sends no data to external servers — all inference happens on-device using local language models including LLaMA, Mistral, and other GGUF-format models. It supports Mac, Windows, and Linux and works fully offline. Jan is the go-to choice for developers and privacy-conscious users who want a self-hosted AI assistant with complete data sovereignty.
View on HookFlow46
heat
Fine-tune LLMs 2x faster with 70% less memory. Optimized kernels for Llama 3, Mistral, and Gemma on consumer GPUs. Runs free on Google Colab.
View on HookFlow52
heat
Offline-capable, self-hosted web interface for Ollama and OpenAI-compatible APIs. ChatGPT-like UI that runs entirely on your own machine.
View on HookFlow30
heat
deepseek-coder:6.7b (fits 8 GB VRAM) or deepseek-coder:33b (needs 16 GB). For general coding with strong reasoning, qwen2.5-coder:7b is a strong alternative. Pull with 'ollama pull deepseek-coder:6.7b'.
Run 'ollama list' to see all downloaded models with their sizes and quantization levels. 'ollama ps' shows what's currently loaded in VRAM.
Yes, if you have enough VRAM. Ollama will load each model into VRAM on first call and keep it resident. Models are unloaded after 5 minutes of inactivity by default (configurable with OLLAMA_KEEP_ALIVE env var).
'ollama pull' downloads the model weights without starting inference. 'ollama run' downloads if needed and immediately opens an interactive chat session. Use 'ollama pull' to pre-cache models without starting a session.
HookFlow ranks local AI tools daily by heat score — see what the community is building with right now.
View Ollama Heat Score →Learn how to run AI models locally on your own hardware — completely offline, private, and free. Step-by-step guide covering the best local AI tools including Jan and Open WebUI.
Ollama, Axolotl, Unsloth, Haystack, and Burn are co-moving — a rare multi-tool signal. This guide shows how to wire them into a complete run-fine-tune-serve pipeline you own.
Distilled reasoning models pack frontier-level chain-of-thought into smaller, faster, cheaper packages. Here are the top performers tracked by HookFlow heat scores.