Same chunks, different order: context assembly for long-document RAG
Sole author of both harnesses ·Forward Data Lab, UIUC (Prof. Kevin Chang) · –
Relevance order: answered wrong.
misleading, 1st, gold, 42nd
Document order: answered right.
gold, 22nd · misleading, 23rd
One QuALITY question I annotated by hand: the same retrieved passages, one bar each, in two orders. Only in document order did the gold passage sit beside the misleading one, and only then did the model answer right.
Putting the same retrieved passages in document order, not relevance order, raised accuracy on QuALITY and cut it on ContractNLI, by about 4–5 points each. The best order depends on the task.
What I did
Built two controlled long-document RAG harnesses as sole author: a SCROLLS context-assembly benchmark and a Hamlet QA instrument.
Evaluated 10+ published RAG methods in those harnesses, official code where it exists and faithful ports otherwise, under the same retrieval, reader and token budget.
Found that at high budgets the context already covered nearly whole documents on most tasks, then isolated passage order with the chunks held fixed.
Showed that document order raises QuALITY accuracy by 5.13 points and cuts ContractNLI by 3.95 on the full validation set.
Designed, implemented and unit-tested an evidence-role context assembler: it finds up to five evidence roles a question needs, builds source-faithful units, covers every role greedily under the token budget and orders the result anchor-first, then by document.
Figures
one order per arm
The controlled study: every arm reads the same chunks, under the same budget and reader; only their order changes.Hover over a box, or tab to it, for how that step works.Tap a box for how that step works.Long documents. QuALITY, ContractNLI, Qasper, QMSum and NarrativeQA, with the official SCROLLS loading and scoring.Retrieved chunks. The retriever picks one set of chunks under the token budget; a selected-set signature proves that every arm reads exactly the same chunks.Relevance order. The usual RAG prompt: the most relevant chunk first.Document order. The same chunks, in the order they appear in the document.Anchor first. The best match first, then the rest in document order: 39.56 against 36.72 for relevance order, on the 5-task average at 10K tokens.Same reader. One Qwen2.5 reader and the official SCROLLS scorer for every arm, at the same budget, so only the order differs.Accuracy moves. On 7,804 examples per arm, document order raised QuALITY by 5.13 points and cut ContractNLI by 3.95, at 5K tokens.
Mean document coverage of the assembled context, by task and context budget
Context budget (tokens)
Task
500
1.5K
5K
10K
Qasper
11%
35%
91%
100%
ContractNLI
14%
46%
94%
100%
QuALITY
7%
22%
75%
96%
QMSum
5%
15%
51%
89%
NarrativeQA
1%
5%
16%
32%
How much of each document is already in the context at each token budget. At 10K tokens it is about 100% of Qasper and ContractNLI and 96% of QuALITY, so wins there were mostly about order.
Built with
Python ·vLLM ·FAISS and BM25 ·rerankers ·Qwen2.5 and Qwen3
Credits
Builds on published RAG methods compared on the SCROLLS benchmark: DOS-RAG, RAPTOR, ReadAgent, SetR, CRAG, MacRAG and RECOMP, by their original authors. With Prof. Kevin Chang.