Docker + Ollama: Production-Ready Local AI API in 15 Minutes

Running Ollama in Docker gives you a reproducible, containerized local AI API that can be deployed on any machine with a GPU. This setup is ideal for home servers, dev environments, and anywhere you need Ollama to start automatically and survive reboots.

Basic Docker Setup

# CPU only
docker run -d   -v ollama:/root/.ollama   -p 11434:11434   --name ollama   --restart always   ollama/ollama

# With NVIDIA GPU
docker run -d --gpus all   -v ollama:/root/.ollama   -p 11434:11434   --name ollama   --restart always   ollama/ollama

Docker Compose Setup

# docker-compose.yml
version: '3.8'
services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: always
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: always
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open_webui_data:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama_data:
  open_webui_data:

docker compose up -d
# Pull your first model
docker exec -it ollama ollama pull qwen3.6:14b

AMD GPU Support

docker run -d --device /dev/kfd --device /dev/dri   -v ollama:/root/.ollama   -p 11434:11434   --name ollama   ollama/ollama:rocm

Expose to Your Home Network

# Add to docker-compose environment
environment:
  - OLLAMA_HOST=0.0.0.0  # Listen on all interfaces

# Access from other machines on your network:
# http://192.168.1.100:11434/v1

For hardware recommendations for running Ollama as a home server, see LLMHardware.io's home server build guide.