Prompt Engineering for Local LLMs: Get Better Answers Locally

Local LLMs respond differently to prompts than cloud APIs because they have different training, RLHF tuning, and instruction formats. Getting consistently good results requires understanding how your specific model was trained and what prompting strategies work best locally.

Instruction Format Matters

Each model family uses a specific chat template. Sending prompts in the wrong format gets worse results:

import ollama

# BAD: raw prompt (model sees it as continuation, not instruction)
resp = ollama.generate(model='qwen3.6:14b', prompt='Fix this Python bug: ...')

# GOOD: chat format (model understands this as instruction)
resp = ollama.chat(model='qwen3.6:14b', messages=[
    {'role': 'system', 'content': 'You are an expert Python developer.'},
    {'role': 'user', 'content': 'Fix this Python bug: ...'}
])

System Prompt Patterns That Work

SYSTEM_PROMPTS = {
    'code': "You are an expert software engineer. Write clean, efficient code. Include error handling. Do not add unnecessary comments.",
    'analysis': "You are a data analyst. Be precise and quantitative. Show your reasoning step by step. Use tables when comparing options.",
    'writing': "You are a technical writer. Write clearly and concisely. Avoid jargon. Use active voice. Target a senior developer audience.",
}

Chain-of-Thought for Hard Problems

response = ollama.chat(
    model='deepseek-r1:32b',  # reasoning model
    messages=[{
        'role': 'user',
        'content': '''Let's think step by step.
        
Problem: I have 24GB VRAM. I want to run a 32B model with 32K context.
Which quantization should I use?

Show your calculation.'''
    }],
    options={'temperature': 0.6}
)

Temperature Guide

Temperature	Output Style	Best For
0.0	Deterministic	Code, factual Q&A
0.6-0.7	Balanced	Most use cases ★
0.9-1.0	Creative	Brainstorming, creative writing
>1.0	Random/chaotic	Rarely useful

For hardware that provides the fastest response times for interactive prompting, see LLMHardware.io's latency benchmark page.