Research Paper RAG with Page-Level Citations
A retrieval-augmented generation assistant that answers questions from a folder of engineering research PDFs. Every answer cites its document and page, and if the sources do not contain the answer, it refuses.
View the code on GitHubFull results
- answer accuracy on 12 answerable questions
- 83.3%
- of cited pages were actually retrieved
- 92.3%
- unanswerable questions correctly refused
- 3/3
Problem
Engineers need precise answers from papers: which hyperparameters, what error rate, what architectural trick. A general chatbot answers from memory, can invent numbers, and cannot show where a claim came from. This tool answers only from your PDFs, so every claim can be checked against a page.
Approach
- Text is extracted page by page, and chunks never cross a page boundary, so each page reference is exact.
- Chunks are embedded locally with sentence-transformers and stored in Chroma, so indexing makes no API calls.
- The model must cite every claim or return a fixed refusal sentence.
- A similarity gate refuses weak matches before any LLM call, which also saves cost.
- Citations are parsed from each answer and checked against the passages actually retrieved.
Results
- The evaluation uses 15 hand-written questions over 5 open-access arXiv papers, scored by deterministic string and page matching with no LLM judge.
- Overall score was 86.7%. Retrieval alone found the right page for 75.0% of questions; the LLM added synthesis and proper refusals.
- The first real run scored 75%. Two of the failures were bugs in my own citation parser and scorer, not in the model.
Limitations
- Fifteen questions is a smoke test, not a benchmark.
- Text extraction flattens equations and tables, so content in algorithm boxes retrieves poorly.
- Scanned PDFs without a text layer cannot be indexed.
- Follow-up questions do not use chat history for retrieval.
Built with
- Python
- sentence-transformers
- ChromaDB
- pypdf
- Streamlit
- Gemini
- Anthropic
- OpenAI