RAG with Local LLMs: Private Document Chat Without the Cloud

Retrieval Augmented Generation (RAG) lets you chat with your own documents using a local LLM. No data leaves your machine. This guide builds a working RAG pipeline from scratch using Ollama, ChromaDB, and Python in under 50 lines of code.

How RAG Works

RAG has three stages:

1. Index: Split documents into chunks, embed each chunk as a vector, store in a vector database
2. Retrieve: Embed the user's question, find the most similar document chunks
3. Generate: Pass retrieved chunks + question to the LLM as context, get an answer

Setup

pip install chromadb langchain langchain-community langchain-ollama

# Pull embedding model (runs locally)
ollama pull nomic-embed-text

# Pull LLM
ollama pull qwen3.6:14b

Build the RAG Pipeline

from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain.chains import RetrievalQA

# 1. Load and chunk documents
loader = TextLoader("./my_docs/hardware_specs.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

# 2. Embed and store (all local, no API calls)
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./rag_db")

# 3. Build QA chain
llm = ChatOllama(model="qwen3.6:14b", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    return_source_documents=True
)

# 4. Query
result = qa_chain.invoke({"query": "What GPU has the most VRAM?"})
print(result["result"])
print("Sources:", [d.metadata for d in result["source_documents"]])

Supported Document Types

• PDF files: PyPDFLoader
• Web pages: WebBaseLoader
• Word documents: Docx2txtLoader
• YouTube transcripts: YoutubeLoader
• GitHub repos: GithubFileLoader

Hardware for RAG

Embedding generation is fast — even a CPU can embed thousands of documents in minutes. The bottleneck is the LLM for answer generation. A 14B model on an RTX 4080 delivers 55 tok/s — more than enough for interactive RAG. For hardware comparisons at different price points, check LLMHardware.io's RAG hardware guide.