RAG Evaluation: Chunks, Embeddings and Top-k

Problem

RAG systems are usually built on defaults: a chunk size from a tutorial, a popular embedding model and a top-k of 5. Those choices decide what the model sees, but teams rarely test them. This lab replaces guesswork with a repeatable comparison that reports its own uncertainty.

Approach

• Five open-access arXiv papers and 20 hand-written questions, including 4 unanswerable ones and a near miss.
• The prompt, overlap, search method and LLM stay fixed, so only the three factors change.
• Hit rate is matched on verbatim evidence snippets, which is stricter than matching a page.
• A blind LLM judge scores answers against a written rubric, and the judge itself was audited.
• 95% bootstrap intervals and paired comparisons show which gaps can be trusted.

Results

• Only the top-k effect on hit rate clears its interval. Other results point in a consistent direction but are evidence, not proof.
• The system fails by refusing, not by inventing: all 48 unanswerable cases were correctly declined, and none of the 143 real answers was entirely wrong.
• Retrieval decides the outcome. Answers were fully correct 93% of the time after a hit and 11% after a miss.
• The judge agreed with deterministic fact matching on 94.4% of answers, and all 6 disagreements were read by hand.

Limitations

• With 16 answerable questions, one question moves a score by about 6 points.
• The corpus is 5 machine learning papers, so results may differ for contracts or manuals.
• One LLM and one run; the judge is the same model as the generator.
• Rerankers, hybrid search and semantic chunking were not tested.

Built with

• Python
• sentence-transformers
• NumPy
• pypdf
• Gemini
• Anthropic
• OpenAI