Exploring PageIndex in RAG: What We’re Learning About Reasoning-Based Retrieval

By Nkosinathi Sangweni

Why explore PageIndex?

Most traditional RAG systems ask a narrow question: Which chunks are closest to the query?

PageIndex asks a different question: Where would a knowledgeable reader look inside this document?

That difference matters.

A vector retriever may find passages that sound similar to the query, but similar does not always mean useful. PageIndex’s own technical blog calls out this problem directly: vector-based RAG assumes that the most semantically similar text is also the most relevant text, but that assumption can fail when the user’s query expresses intent rather than exact content.

For example, a user might ask: What was the total deferred asset value?

A traditional retriever may look for chunks containing “deferred,” “asset,” and “value.” But the actual answer might live in a table, appendix, or referenced section that does not closely match the wording of the query. A human would not search only by phrase similarity. A human would scan the table of contents, inspect likely sections, follow references, and check whether the evidence is sufficient.
PageIndex tries to make retrieval behave more like that.

---

The core idea:
​Retrieval as document navigation

The most useful way to understand PageIndex is not as “RAG without vectors,” but as RAG with document navigation.
In PageIndex, a document is organised into a hierarchy:

The LLM can inspect this structure and decide where to look next. PageIndex’s blog describes this as an iterative process: read the table of contents, select a likely section, extract information, check whether it is sufficient, and continue if needed. 
That makes retrieval feel less like a nearest-neighbor lookup and more like a guided search through a document.
This is the part worth exploring deeply: 
​PageIndex changes the retrieval primitive from “find matching chunks” to “reason over structure.”

---

What makes this promising?

1. It preserves document structure

Many documents are not just text. They are organised knowledge objects.
A financial report has statements, notes, tables, appendices, and page references. A legal contract has clauses, schedules, definitions, and exceptions. A technical manual has chapters, diagrams, warnings, and troubleshooting sections.
Chunk-based RAG can flatten that structure. PageIndex keeps the hierarchy visible.
That is promising because users often do not only need an answer. They need to know where the answer came from.

2. It may reduce the “similar but wrong” retrieval problem

One of the biggest practical issues in RAG is retrieving content that appears related but does not actually answer the question.
PageIndex’s blog argues that semantic similarity is not the same as true relevance, especially in domain-specific documents where many passages may sound similar but differ in meaning or importance.
That framing is useful. It suggests that better retrieval may require more than better embeddings. It may require a retrieval strategy that understands the shape of the document.

3. It can follow references more naturally

​Long documents often say things like:

Traditional vector retrieval may miss the referenced section if the referenced content does not share the same wording as the original query. PageIndex’s blog highlights this as a weakness of conventional vector RAG and argues that a ToC-based or hierarchical retrieval system can follow those references more like a human reader.
This is one of the strongest reasons to explore PageIndex for document-heavy workflows.

4. It makes retrieval more explainable
A vector result can be hard to explain. The system says, in effect: This chunk was mathematically close to the query.
A PageIndex-style system can say: I inspected the financial statements section, then followed a reference to Appendix G, then retrieved the relevant table.​
That retrieval path is easier to audit.
The PageIndex GitHub page specifically lists better explainability and traceability as core features, including page and section references.

---

What we are learning?

Learning 1: PageIndex is less about replacing vectors & more about the retrieval question

The easy headline is “vectorless RAG.” But the more useful insight is that PageIndex reframes retrieval.
Traditional RAG asks: What text is similar?
PageIndex asks: Where should we look?
That distinction is subtle but important. It moves retrieval closer to reasoning, planning, and evidence gathering.

Learning 2: Structure matters more than we often admit

A lot of RAG work focuses on embeddings, chunk size, reranking, and prompt design. Those are important. But PageIndex reminds us that document structure itself is a powerful signal.
For structured documents, the table of contents, section hierarchy, appendix layout, page numbers, table captions, and references are not metadata decorations. They are part of the meaning.
Ignoring that structure can make retrieval weaker than it needs to be.

Learning 3: Chunking is not neutral

Chunking sounds like a preprocessing detail, but it can change what the model is able to understand.
PageIndex’s blog criticises hard chunking because fixed-size chunks can split sentences, paragraphs, sections, or logical units of meaning. 
That is an important learning: chunking is not just a technical step. It is an information design choice.
When chunks break the natural structure of a document, retrieval can become noisy, incomplete, or misleading.

Learning 4: Reasoning-based retrieval has trade-offs

PageIndex is promising, but it is not magic.
The PageIndex documentation notes that LLM-based tree search can be slower because it requires LLM reasoning. It also notes that selecting nodes based only on summaries can lose details from the original content.
That means PageIndex-style retrieval may be strongest when accuracy, traceability, and structured navigation matter more than raw speed.
For low-latency FAQ search, traditional vector RAG may still be simpler and cheaper. For complex document analysis, reasoning-based retrieval may be worth the extra cost.

Learning 5: Hybrid retrieval may be the practical middle ground

The PageIndex docs describe both LLM Tree Search and Hybrid Tree Search. Hybrid Tree Search combines LLM reasoning with vector database techniques. 
This feels like an important practical direction.
Rather than framing the decision as: vectors vs. no vectors
A better framing may be: when should vectors help, and when should reasoning take over?

A strong production architecture might use vector or keyword search to identify candidate documents quickly, then use PageIndex-style reasoning to navigate within the most relevant documents.

---

Where PageIndex seems especially useful

PageIndex looks most interesting for documents where:
​
- answers are buried inside long reports
- page-level citations matter
- tables and appendices are important
- internal references need to be followed
- users ask multi-step questions
- the document has a strong section hierarchy

Examples include:
- financial filings
- legal contracts
- policy documents
- insurance documents
- academic papers
- technical manuals
- compliance reports

In these cases, retrieval is not just about finding text. It is about finding evidence.

---

Open questions we would want to test

Because this is exploratory, the right next step is not to declare PageIndex “better.” The right next step is to test it against specific retrieval problems.
​
The questions worth testing are:

1. Does PageIndex retrieve the correct evidence more often than vector RAG?
2. Does it improve citation accuracy?
3. How much latency does tree search add?
4. How much does the quality of the document tree affect results?
5. Does it work equally well on messy PDFs, scanned PDFs, and poorly structured documents?
6. When does hybrid retrieval outperform pure tree search?
7. How does it perform on multi-document questions?

​These are the questions that would determine whether PageIndex is a better fit for a specific RAG system.

---

A possible exploratory architecture

Example of an exploratory PageIndex-based RAG workflow

​

---

An example of architecture for a hybrid system. This pattern feels especially practical for large corporates.

---

Final reflection

PageIndex is interesting because it challenges one of the default assumptions in RAG: that better retrieval mainly means better similarity search.
It suggests a different path.
For long, structured documents, retrieval may need to become more like reading. The system should understand the document map, inspect likely sections, follow references, and verify whether the evidence is enough.

That is the real learning from exploring PageIndex:
​
The future of RAG may not be only about retrieving the closest text. It may be about teaching systems where to look, why to look there, and how to know when they have found enough evidence.

PageIndex is one approach pointing in that direction. Whether it becomes the right choice for a given system depends on the documents, the latency budget, the need for citations, and the complexity of the questions. But as an idea, it is worth exploring because it moves RAG from passive similarity matching toward active document reasoning.