Ask Questions About Your Own Documents: Build a RAG System

What you will build

Yes: this is a complete small Retrieval-Augmented Generation (RAG) system. You will ingest plain-text documents, split them into passages, create local embeddings, retrieve relevant passages for a question, and give those passages to a local chat model with numbered source references.

The reference implementation uses an inspectable JSON index and Python's standard library. A dedicated vector database is not necessary for a tiny single-user collection. The result is a learning build, not a multi-user document service.

Before you start

Use Python 3.11+, a loopback-only Ollama service, and enough RAM to run an embedding model and small chat model, potentially one at a time. Allow 1-2 hours plus downloads. No hosted model key is needed.

ollama pull nomic-embed-text
ollama pull llama3.2:3b
python3 --version

Create a new directory named rag-lab and a docs subdirectory. Put two small UTF-8 .txt files in it. Use synthetic material, not private contracts, for the first run. For example, write that the fictional workshop opens at 09:30 on Saturdays and closes at 15:00.

1. Get the reference implementation

Download rag.py into rag-lab, inspect it, and run its offline checks:

python3 rag.py --self-test

The checks exercise chunk boundaries and similarity scoring; they do not prove model accuracy or a complete live deployment. The script uses a fixed loopback endpoint and does not follow HTTP redirects. It reads only local plain-text files selected by the build command.

2. Build the index

From rag-lab:

python3 rag.py build docs --index index.json

Inspect index.json. It contains the embedding model name, document hashes, source paths, chunk text, and vectors. The index contains your document content: protect it like the originals.

Checkpoint: each sample document has passages, embeddings have a consistent nonzero dimension, and the build reports a bounded number of chunks. The chunker advances by a positive step and stops at the end rather than creating hundreds of repeated tail fragments.

This example indexes characters, not tokens. Its conservative chunk size is appropriate for short learning documents; use token-aware boundaries and structural parsing when moving to large or complex documents.

3. Ask a grounded question

python3 rag.py ask "When does the workshop open on Saturdays?" --index index.json

The program prints retrieved sources before its answer. Compare the source passages with the generated claim. The answer should cite a source number such as [1]; the printed source record maps that number to the original file and chunk.

These are references to local files, not fabricated web URLs. The script does not guarantee that a generated citation supports a claim. You are responsible for checking that relationship.

4. Test missing and changed information

Ask a question the documents cannot answer, such as the fictional workshop's phone number. The model should say it cannot find the information. The prompt requests this behavior, but prompts do not eliminate hallucination.

Edit the workshop opening time and ask again before rebuilding. The program should reject a stale source hash rather than silently answer from the old index. Rebuild, then repeat the question. Also test a deleted document.

Record five answerable questions and five unanswerable questions. Check retrieved evidence, answer accuracy, and citation support separately. A correct-looking answer from the model's training knowledge is not proof that retrieval worked.

5. Understand the limits before expanding

This build has no authentication, shared tenancy, OCR, PDF parsing, or automatic watching. Do not expose it as a public API. Source paths and file permissions are your access boundary.

If the embedding model changes, rebuild the index; matching vector dimensions do not make different embedding models compatible. Keep sources and provenance authoritative, and treat document instructions as untrusted content rather than commands to the assistant.

Move to a vector database when collection size or concurrent use justifies it. Add permission filtering before retrieval, not after generating an answer. Introduce reranking only after measuring retrieval failures on a reference set.

When it breaks

Cleanup and next steps

No system service is installed by this script. Keep or remove the disposable rag-lab files after reviewing their contents. Unload models with ollama stop MODEL_NAME if desired; do not delete shared model data.

Continue with MCP tools or use the wiki's evaluation checklist before trusting a larger collection.

References

• Ollama embedding API
• Wiki: RAG with Local Embeddings
• Wiki: Evaluating RAG Answers and Citations