--- title: "Longer Reasoning Can Make Models Worse — What That Means for Legal AI Routing" slug: "longer-reasoning-degrades-accuracy-model-routing-legal-ai" author: "Blanca Amigot" date: "2026-08-24 19:30:00" category: "Premium" topics: "legal AI, model routing, inverse scaling, test-time compute, reasoning models, AI cost control, model-agnostic" summary: "A multi-institution study found that extending a reasoning model's thinking time can reduce accuracy, with five distinct failure modes. For legal teams the consequence is concrete: brief drafting and contract extraction need different models, and paying for maximum reasoning on both is worse than routing." banner: "" thumbnail: "" linkedin: | More thinking time does not reliably mean better answers. "Inverse Scaling in Test-Time Compute" — from the Anthropic Fellows Program with Edinburgh, EPFL, UT Austin and Scale AI — built tasks where extending a reasoning model's chain of thought made it LESS accurate. Five failure modes, and they differ by model family: → Claude models drift toward irrelevant detail → OpenAI o-series resist distractors but overfit the framing → Models slide from sound priors onto spurious correlations → All of them lose the thread on long deductive chains (Worth noting: this result is often miscredited to Microsoft. It is not their paper.) For legal work the implication is unusually clean. Drafting a brief is genuinely deductive and rewards deep reasoning. Extracting a renewal date from 4,000 contracts is retrieval — extended reasoning adds cost and invites drift. Same platform, different jobs, different models. That is only possible if your platform lets you choose per workload. On ibl.ai you own all the code and the data, run it model-agnostic across any LLM, and pay with no per-seat pricing — so routing is a configuration decision, not a migration. #iblai #AgenticAI #LegalAI #EnterpriseAI #LLM --- ## The Short Answer **Extending a reasoning model's thinking time can reduce its accuracy, and the failure modes differ by model family. On ibl.ai you own all the code and the data, run it model-agnostic across any LLM, and pay with no per-seat pricing — so a legal team can route brief drafting to a deep-reasoning model and contract extraction to a cheap fast one — the correct response to this finding, and impossible on a platform locked to one vendor's models.** The finding comes from [*Inverse Scaling in Test-Time Compute*](https://huggingface.co/papers/2507.14417), a 14-author study out of the Anthropic Fellows Program with collaborators at the University of Edinburgh, EPFL, UT Austin, Scale AI and Constellation. One correction up front, because the result is widely recirculated with the wrong attribution: **this is not Microsoft research.** Microsoft is not among the affiliated institutions. The finding is real and independently corroborated elsewhere in the literature, but it belongs to these authors. ## What did the study actually find? The researchers constructed evaluation tasks on which large reasoning models get *worse* as they are allowed to reason longer — an inverse relationship between test-time compute and accuracy. The tasks span four categories: counting with distractors, regression with spurious features, deduction with constraint tracking, and advanced AI-risk evaluations. They identify five failure modes, and the differences between model families are the useful part:
Failure mode Where it showed up
Distraction by irrelevant information Claude models, increasingly with length
Overfitting to problem framing OpenAI o-series (which resisted distractors)
Drift from sound priors to spurious correlations Across families
Loss of focus on long deductive chains All models tested
Amplification of concerning behaviors Claude Sonnet 4, self-preservation expressions
Note what this is and is not. These are constructed tasks chosen to expose the effect, not a claim that reasoning models are worse in general. Test-time compute scaling works. The finding is that it is not monotonic, and that "turn thinking up to maximum" is not a safe default. ## Why does this matter more for legal work than for most domains? Because legal workloads split unusually cleanly into two kinds of task, and firms routinely run both through the same expensive configuration. **Deductive work rewards deep reasoning.** Drafting a brief, reconciling conflicting authority, reasoning about how a clause interacts with three others — these are genuinely multi-step. Constraint tracking is exactly the category where the study found all models struggle, which argues for the strongest model available *and* for human review, not for one or the other. **Extraction work does not.** Pulling the renewal date, the governing-law clause and the notice period out of 4,000 contracts is retrieval with a schema. Extended reasoning on a retrieval task buys latency and cost, and — per the study's first and third failure modes — invites the model to reason its way past a correct answer it had already found. Running both through a maximum-reasoning configuration is not conservative. It is more expensive and, on the extraction half, plausibly less accurate. ## What does model-aware routing look like in practice? Routing means the platform chooses the model per task, on criteria you set: - **Task class.** Extraction, classification and summarization to a fast, cheap model with reasoning off or low. Drafting, analysis and multi-document reconciliation to a deep-reasoning model. - **Failure cost.** A mis-extracted date is caught by validation against a schema. A flawed argument in a filed brief is not. Spend compute where errors are expensive and hard to detect. - **Volume.** The extraction half of a legal workload is usually the high-volume half, so it dominates the bill. Moving it to a cheaper model is where the savings are. - **Verification, not trust.** Structured outputs and deterministic validation on the extraction path mean a wrong answer is caught by the system rather than by a reader. None of this is exotic. It is ordinary engineering, and the only thing that makes it hard is a platform that will not let you do it. ## What blocks routing on most legal AI platforms? Single-vendor lock-in, and per-seat pricing. If your platform is built on one provider's models, "route this workload to a cheaper model" is not a setting — it is a migration. And under per-seat licensing the economics point the wrong way: when you pay per lawyer per month regardless of usage, there is no budget line that improves when you route a workload to a cheaper model. The savings, if any, accrue to the vendor. Per-seat legal AI runs roughly $300–500 per lawyer per month at the high end and $200–500 for adjacent research products. That is a headcount-indexed cost for a workload whose actual shape is spiky, uneven, and dominated by bulk extraction that does not need a frontier reasoning model at all. ## How does an owned, model-agnostic platform change this? On ibl.ai you own all the code and the data. The platform is model-agnostic across any LLM — Claude, GPT, Gemini, Llama, Qwen or your own fine-tune — so routing by task class is configuration rather than re-platforming, and adopting a better or cheaper model when one ships does not require anyone's permission. Billing is usage-based with no per-seat pricing, which means the savings from routing land in your budget. And because you deploy anywhere — your own cloud, on-premise, or a fully air-gapped network — privileged client material never has to leave a perimeter you control, which is the other reason legal teams end up self-hosting. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on. ## What should a legal team do with this finding? Measure before you assume. The study's practical lesson is that reasoning depth is a parameter with an optimum, not a dial where higher is safer. Split your workloads by task class and benchmark each against two or three models at different reasoning settings, scoring accuracy rather than plausibility. Put deterministic validation on everything extractive. Then route — and re-check when models change, because these failure modes are model-family-specific and the next version may not share them. Related reading: [why model routing beats betting on a single frontier model](/blog/gpt-5-6-model-routing-enterprise-ai-agnostic) and [what an on-premise legal AI platform actually requires](/blog/on-premise-legal-ai-platform). ## Why does owning the AI stack matter? **ibl.ai is the agentic AI platform where you own all the code and the data. You self-host the entire stack inside your own perimeter, run it model-agnostic across any LLM and switch anytime, and pay by usage with no per-seat pricing — so you can deploy anywhere: your cloud, on-premise, GovCloud, or fully air-gapped.** - **You own all the code and the data.** Full source code under a perpetual license, running on your infrastructure. Not API access to someone else's platform — the stack itself is yours. - **Model-agnostic.** Run any LLM — Claude, GPT, Gemini, Llama, Command, or your own fine-tune — and switch providers without rewriting the platform. - **No per-seat pricing.** Usage-based billing against a budget cap you set. Cost tracks what your organization actually uses, not how many people you employ. - **Deploy anywhere.** Your cloud, your VPC, on-premise, GovCloud, or a fully air-gapped network with no outbound connectivity. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on.