bryanashley.dev

Running Claude Code at Home with Local Models

I've recently been experimenting with running Claude Code locally using Ollama and Docker. It's been a fascinating journey trying to get a private, high-performance coding agent running on my own hardware without relying on external APIs. This setup allows for a completely private development environment with zero token costs.

🚀 "Local Claude" Quick Start Guide (WSL2)

⚠️ WARNING This needs a lot of VRAM (and probably system memory) to even load the initial context for claude code. See the "System Wall" problem in the Troubleshooting section.

This Quick Start guide assumes you have WSL2 and Docker Desktop (or Docker engine) already running. This setup will give you a private, $0/token coding agent powered by your own hardware.

1. Spin up the Ollama Container

Run this in your WSL2 terminal to start the backend. We are skipping the --restart flag per your request.

docker run -d \
  --gpus=all \
  --name ollama \
  -v ollama_data:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  ollama/ollama

2. Pull the Powerhouse Models

Download the two best models for agentic coding in 2026.

docker exec -it ollama ollama pull qwen3-coder
docker exec -it ollama ollama pull glm-4.7-flash

3. Install Claude Code CLI

Install the official Anthropic tool. While it's built for their API, our environment variables will "hijack" it.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

(Note: If you prefer npm, use npm install -g @anthropic-ai/claude-code, but the curl script is the new 2026 standard for auto-updates).

4. Configure Your Shell

Add these lines to your ~/.bashrc or ~/.zshrc to automate the "trick" and create a shortcut.

# Redirect Claude to your local Docker container
export ANTHROPIC_BASE_URL="http://localhost:11434"
export ANTHROPIC_AUTH_TOKEN="ollama"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1

# Create a convenient alias
alias lclaude='claude --model qwen3-coder'

Apply the changes: source ~/.bashrc (or ~/.zshrc).

🛠 Everyday Usage

Action Command
Start Backend docker start ollama
Start Coding lclaude (in any project folder)
Swap Model claude --model glm-4.7-flash
Check VRAM docker exec -it ollama ollama ps
Stop Backend docker stop ollama

⚠️ Troubleshooting: "Error response from daemon: OCI runtime create failed: !!! could not select device driver "" with capabilities: [gpu]"

This error means your Docker engine is trying to use your GPU, but it doesn't have the "bridge" (the NVIDIA Container Toolkit) to talk to the hardware.

Since you are in WSL2, you specifically need to install this inside your Linux distro, even if you have NVIDIA drivers on Windows.

The Fix (Step-by-Step)

Run these commands in your WSL2 terminal to install the toolkit and configure Docker to use it:

  1. Add the Package Repositories
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
  1. Install the Toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
  1. Configure Docker This command tells Docker how to use the NVIDIA driver as a "runtime":
sudo nvidia-ctk runtime configure --runtime=docker
  1. Restart Docker Since you are in WSL, you might need to restart the service manually:
sudo systemctl restart docker

🧱 Troubleshooting: The "System Prompt" Wall

Claude Code injects a massive 16.5k token system prompt before you even say "Hello." This prompt contains the instructions on how to use the tools.

If your local model is running with the default Ollama context (usually 2k or 4k), the tool-calling instructions are literally being "chopped off" before the model can read them.

The Fix: You must manually increase the context window to at least 32k (64k is better).

How to force 32k Context:

docker stop ollama && docker rm ollama && \
docker run -d \
  --gpus=all \
  --name ollama \
  -v ollama_data:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  -e OLLAMA_CONTEXT_LENGTH=32768 \
  ollama/ollama