I Tested 6 Leading Models on a 35,000-Word Document. The Cheapest Model Won.

TL;DR

The most expensive reasoning models gave the worst results. The cheapest fast model excelled. Hierarchical processing rescued failing models but added overhead to working ones. And disabling reasoning improved long-document accuracy more than paying for premium models.

How This Started

I had been asking frontier models to analyse long documents and getting confident answers that were subtly wrong. Not hallucinations - systematic mischaracterisations. The models would claim the document didn't address objections that were explicitly addressed, or misrepresent positions based on intro/conclusion while missing middle sections. They sounded authoritative while being factually wrong.

The insidious part: You can only catch these errors if you know the source document intimately. If you're using LLMs to analyse documents you haven't read yourself - the most valuable use case - you have no way to know they're wrong.

Plenty of research literature confirms this: LLM attention degrades on long inputs (e.g. https://arxiv.org/abs/2505.10570). Quality is reliable around 2k tokens, acceptable to 8k, increasingly unreliable beyond - despite models advertising 200k-2M token windows.

Why This Matters

Customer service AI handling insurance claims needs to cite the correct policy clause. If the model skims and pattern-matches to "typical claims," it might miss exception clauses or cite wrong procedures.

Legal discovery requires finding composite facts across hundreds of pages. Missing exculpatory evidence on deposition page 147 because the model skimmed the middle section could cost someone their freedom.

Medical systems synthesising patient histories from lengthy records can't afford to miss contraindications or hallucinate drug interactions.

RAG systems likely inherit the same risks. Synthesis across 10 retrieved chunks totaling 15k tokens puts you back in the degradation zone.

The failure mode is confident wrongness. Models sound authoritative. Errors are subtle. You need domain expertise to detect them - which defeats the purpose of using the LLM.

Solution: Hierarchical Processing

To try and solve this I built a system with strict token budgets:

• Split documents into 1,800-token chunks

• Summarise each chunk (~700 tokens)

• Compress upward recursively: group summaries (5→1) until total < 8k tokens

• LLM selects relevant spans and generates citation IDs (linked to original source and used for selective decompression)

• Final synthesis from compressed top level

• Every LLM call stays in the 2-8k "safe zone"

e.g. For a 150 page document: 49 summaries → 8 parents → final synthesis sees ≈5.6k tokens, not ≈45k.

Result: Hierarchical processing was more accurate – it avoided the skimming errors that direct reading produced – but less comprehensive, as each compression step lost cross-document synthesis and nuance.

However: When I tested simpler (and in some cases older) non-reasoning models – Gemini 2.5 Flash Lite and Claude 3.5 Sonnet – on the full document in a single prompt, they worked perfectly. They could faithfully read and synthesise 45k tokens directly: no skimming, no hallucination. The hierarchical processing had been solving a problem these models barely showed...

As far as I can tell, this contrast isn’t yet covered in the research, which tends to assume all models struggle equally with long-context fidelity. I had expected newer, larger-context models to perform better, not worse!

How This Was Built

The entire platform - variable-depth chunking engine, citation tracking infrastructure, 4-way comparison harness, AWS deployment automation - was collaboratively built with an LLM on a rainy Sunday afternoon. Infrastructure that would have taken weeks to hand-code was generated and deployed in hours. The code handles complex state across hierarchical levels, tracks citations through recursive trees, manages parallel processing, and serves a polished web interface.

The productivity multiplier is real and astonishing. And it lets us follow lines of curiosity like this that wouldn’t have been worth the time even a year ago.

The Test

Document: 35,000-word book ≈45,000 tokens (this one!)

Task: "Write a comprehensive review for publication as a long read. Go into depth about strengths, weaknesses, and positioning compared to other books."

Models tested:

• Gemini 2.5 Flash Lite (reasoning disabled)

• Gemini 2.5 Pro

• OpenAI GPT-5

• OpenAI GPT-4o

• Anthropic Claude 3.5 Sonnet

• Anthropic Claude Sonnet 4.5

Test design - 4 paths:

1. Model A: Hierarchical compression → synthesis

2. Model A: Direct (full document, single prompt)

3. Model B: No prompt, synthesis guide from path 1 + full document

4. Model B: Direct (full document, single prompt)

This let me compare the hierarchical approach directly and see how a second, more advanced model performed when given the hierarchical synthesis as a guide. It gave a clear view of the effect of using hierarchy as a pre-processing step versus simple prompting. The tests were informal and manual rather than structured evals - roughly fifty runs in total.

Results

Winner: Gemini 2.5 Flash Lite

• Followed all instructions (compared to specific books e.g. The Shallows, Atlas of AI, The Alignment Problem)

• 1,500+ words of substantive analysis

• 7 seconds, costs fractions of a cent

• Largely indistinguishable from careful human reading

GPT-4o: Template-Driven Failure

• Generic 'strengths and weaknesses' structure

• Invented irrelevant criticisms ('Limited Focus on Ethical Implications')

• Ignored explicit instructions

• ~500 words of shallow observations

• Significantly more expensive ~30 cents per run

Claude Sonnet 4.5: Confident Mischaracterisation

• Claimed book didn't address objections it explicitly covered

• Mischaracterised book's position into a stereotyped bucket

• Reasoned about expected content rather than actual content

• Specific, detailed, confident - and factually wrong

GPT-5: The API Quality Gap

• Via ChatGPT interface: excellent prose, natural flow, proper depth, some document skimming detectable

• Via OpenRouter API: compressed, note-like style, still reasonably accurate but less readable

• Same model, different output quality - likely hidden system prompts, different parameters, or post-processing

• Implication: Don't assume UI behaviour matches your API integration and be ready to do tuning

Gemini 2.5 Pro: Elegant But Non-Compliant

• Best prose style ("pincer movement of competing hysterias")

• Missed key instructions, light on weaknesses, overpraising tone

Claude 3.5 Sonnet: Solid Baseline

• Reliable and accurate without the failure modes of newer reasoning variants

Conclusion: Reasoning interferes with long-document reading

Two models excelled: Flash Lite (reasoning disabled) and Claude 3.5 Sonnet (no reasoning capability).

Four reasoning models struggled: GPT-4o produced generic templates. Claude Sonnet 4.5 confidently mischaracterised content. GPT-5 and Gemini 2.5 Pro were mixed - accurate but with style or instruction-following issues.

Working theory: For long-document synthesis, reasoning may interfere with faithful representation. Reasoning models excel at math and coding, but for accurately reading and analysing long texts, simpler pattern-matching appears more reliable.

What Hierarchical Processing Revealed

For failing models (GPT-4o):

• Synthesis guide rescued it from terrible direct performance

• Still inferior to Flash Lite reading the full document directly

For working models (Flash Lite):

• Hierarchical: Accurate with citations, less comprehensive, 40-80 seconds

• Direct: Accurate AND comprehensive, 7 seconds

• Both were faithful to source material; direct was better overall (faster, richer synthesis)

The citation advantage

Hierarchical processing does create one thing worth having: verifiable citations. When the output cites [L1_P3], you can walk back through the tree to the exact byte offset in the source document. For compliance-heavy applications, this auditability matters.

The LLM picks which spans matter; the code tracks them through compression. Building span-selection logic by hand would take weeks of brittle rule-coding. It's a nice example of how LLMs can be leveraged as components in regular software - not replacements for entire systems, but new building blocks that unlock things that weren't practical before.

Practical Takeaways

1. Test on documents you know intimately - The only way to catch subtle mischaracterisations

2. Cheaper often beats expensive - Flash Lite crushed GPT-4o on this task

3. Disable reasoning for long-document tasks - If the model allows it

4. Hierarchical processing when models fail - But test if baseline works first

5. Citations matter - Build systems that track claims to source text

6. Don't trust advertised context windows - Capacity ≠ reliable attention

Coda: Why Reasoning Makes Models Wrong with Greater Confidence

After writing this, I wanted to understand why reasoning models behave this way. The answer is structural.

More reasoning steps mean more opportunities to lock into an interpretation. Once the model commits - say, by skimming the introduction and conclusion - each subsequent reasoning step treats that as ground truth. Mistakes compound rather than correct. There’s no 'did I actually read that right?' loop.

The result is a model that’s wrong with extraordinary conviction. The smug and at times passive aggressive and doubling-down tone people sometimes notice isn’t personality - it’s architecture. The system rewards coherence, not correctness. More capability just means more elaborate confabulation.

This pattern extends beyond document reading. Autonomous agents inherit the same mechanism: misidentify context early, then each action reinforces the mistake. Without external verification at each step, more autonomy means more confident wrongness.

Reasoning works when tasks have external validation - maths, code with tests, formal logic. It fails when interpretation is the task itself, because there’s no mechanism to check the premise.

For production use, reasoning models are useful inside bounded, verifiable loops. Outside those constraints, 'smarter' doesn’t mean more accurate - it means better at being wrong.