The problem
Every RAG system has the same painful iteration loop: change the chunking strategy, re-embed everything, run eval queries, compare results manually. There's no fast way to know whether BM25 or dense retrieval is better for your corpus, or what chunk size maximizes recall without hurting precision. Most teams eyeball it or pick defaults and never revisit them.
RAGBench is a no-code platform for this experimentation loop — upload a corpus, configure a pipeline, run it, see the numbers, compare against a previous run. Built it after spending too much time on manual eval loops at Yonata.
What you can configure
Each pipeline run is a combination of:
- Chunking strategy — fixed-size, recursive, or semantic splitting, with configurable chunk size and overlap.
- Retrieval mode — BM25 (keyword), dense (vector), or hybrid (BM25 + dense + RRF fusion).
- LLM config — model, temperature, system prompt.
- Embedding model — swap the embedding model independently of the retrieval strategy.
Runs are stored in PostgreSQL with their full config. The comparison dashboard diffs any two runs side by side — scores, retrieved chunks, generated answers.
Why hybrid retrieval won
Before building this, I assumed dense retrieval would dominate — semantics beat keywords. It doesn't. BM25 is better for exact product names, error codes, and domain jargon that the embedding model has never seen. Dense is better for paraphrase and concept search. Neither alone is robust.
Reciprocal Rank Fusion merges the two result lists by rank rather than score — which avoids the problem of incomparable similarity scales between BM25 and cosine distance. On mixed-query test sets (half exact-match, half semantic), hybrid RRF consistently outperformed either alone by 8–15% on context precision.
Evaluation: the RAG Triad
Integrated DeepEval's RAG Triad — three metrics that together catch most RAG failure modes:
- Faithfulness — does the answer contain only claims supported by the retrieved chunks? Catches hallucination.
- Answer relevancy — does the answer actually address the query? Catches off-topic responses.
- Context recall — were the right chunks retrieved? Catches retrieval failures.
These run automatically after each pipeline execution. The context recall metric requires a ground-truth answer set — RAGBench generates a synthetic one using the LLM itself for corpora without labeled data.
Dev experience
The entire stack is Dockerized: Qdrant, PostgreSQL, the FastAPI backend, and the Next.js frontend all start with make up. No manual setup, no environment drift. The async FastAPI backend handles embedding and retrieval concurrently — large corpus ingestion doesn't block the UI.