The Short Answer
Model architecture is close to a solved commodity: a 2021 Google study of Transformer modifications found most do not meaningfully improve performance. The money in financial AI is at inference: caching, routing, batching. A cache hit on Claude Opus 5 costs $0.50 per million tokens against $5.00 uncached, a 90% cut. With ibl.ai you own all the code and the data, so that serving policy is yours to tune.
A firm that cannot set its own cache TTL or routing threshold does not have an inference strategy. It has a subscription.
Does model architecture actually matter less than financial firms think?
Yes, and the version of this claim currently circulating is worth correcting before it gets repeated further.
It travels as a quotation from an unnamed "Stanford LLM lecturer" to the effect that everyone obsesses over architecture and it is the least important part. The circulating phrasing is unattributed, so it is not quoted here.
The argument does closely track a real lecture: Yann Dubois's Stanford CS229 guest lecture Building Large Language Models, which ranks architecture below data, evaluation and systems in what determines a model's quality.
Whether that lecture is the origin of the circulating sentence is a separate question, and not one worth asserting. The published evidence is stronger than either version of the anecdote.
In Do Transformer Modifications Transfer Across Implementations and Applications?, a Google team led by Sharan Narang evaluated a large set of proposed architectural variants and reported that most modifications do not meaningfully improve performance.
The paper was submitted in February 2021 and revised that September. Its second observation is narrower than it is usually reported, and the distinction is worth keeping.
What the authors observed is that the variants that did help were "either developed in the same codebase that we used or are relatively minor changes."
What they drew from that is offered as a hypothesis, not a result: they "conjecture that performance improvements may strongly depend on implementation details."
For a bank or an asset manager the conclusion is blunter still, because architecture is not a decision you make. You select a vendor's model from a list, and every remaining choice is about how that model is served.
Model architecture is settled before you arrive. Serving architecture is settled by you, every day, and it is where the cost and the latency actually live.
How much does prompt caching actually save on an inference bill?
Enough that it is usually the first thing worth fixing. All three major vendors publish the same discount shape on cache reads — one tenth of the uncached input price — and differ on what a write costs.
| Model | Input $/MTok | Cached input $/MTok | Output $/MTok | Cache discount |
|---|---|---|---|---|
| Claude Opus 5 | $5.00 | $0.50 | $25.00 | 90% |
| Claude Haiku 4.5 | $1.00 | $0.10 | $5.00 | 90% |
| GPT-6 Astra | $10.00 | $1.00 | $50.00 | 90% |
| Gemini 3.8 Flash † | $0.75 | $0.075 | $3.75 | 90% |
† The Gemini 3.8 Flash rates above are published only through 31 December 2026 on Google's Gemini API pricing page, so treat that row as dated rather than standing.
Caching is not free on the write, and this is where the three vendors stop agreeing. Anthropic publishes a 1.25x multiplier on a five-minute cache write and 2x on a one-hour write, which means a five-minute cache pays for itself after one read and a one-hour cache after two.
OpenAI charges no write premium at all. Google charges nothing to write either, but bills cached content for storage by the hour, which prices a cache by how long you hold it rather than by how much you put in it.
Financial workloads have close to the ideal shape for this. A research desk prompt is mostly a stable prefix, the coverage policy, the risk taxonomy, the instrument master, the compliance instructions, followed by a short variable question.
Take an eight-hour desk running 2,000 requests against a shared 40,000-token prefix on Opus 5, with a one-hour cache refreshed each hour. Uncached, that prefix alone is 80 million input tokens, or $400.00 a day.
Cached, it is eight writes at $10.00 per million tokens plus 1,992 reads at $0.50, which comes to $43.04 a day. Over 250 trading days the difference is roughly $89,000 a year, on one prompt prefix, with no change to the model or the output.
Why does routing cheap requests to small models beat standardizing on one frontier model?
Because the price spread inside a single vendor's family is already 5x, and most desk traffic does not need the expensive end of it.
Anthropic lists Opus 5 at $5.00 input and $25.00 output per million tokens, and Haiku 4.5 at $1.00 and $5.00. Same vendor, same API, five times the cost on both sides of the request.
A large share of what a financial firm actually sends is not hard: extract a ticker, normalize a counterparty name, classify an inbound email, summarize one section of a filing. Sending those to a frontier model is paying five times over for headroom the task never uses.
The research supports doing this systematically rather than by hand.
RouteLLM, from a Berkeley and LMSYS team led by Isaac Ong in June 2024, trained routers to choose between a stronger and a weaker model per request and reported cost reductions of over 2x in certain cases while maintaining response quality.
Its less-quoted result matters more for a regulated buyer. The routers transferred: they kept working when the underlying models were swapped at test time, which is the difference between a durable cost policy and one you rebuild at every model release.
Batch processing is the other unglamorous lever. Anthropic, OpenAI and Google all publish a 50% discount on both input and output for asynchronous work, which covers overnight document classification, backfills and validation reruns.
This is also where per-seat SaaS reveals itself as the wrong shape rather than a different option. A per-seat subscription prices headcount, so a 90% cut in token cost lands nowhere on the invoice. You cannot optimize a bill that is not measuring what you optimized.
What does speculative decoding change, and who actually gets to use it?
It buys latency without changing the answer, which is a rarer property than it sounds and the reason it belongs in a regulated stack.
Fast Inference from Transformers via Speculative Decoding, by Yaniv Leviathan, Matan Kalman and Yossi Matias at Google, reported 2x to 3x acceleration on T5-XXL with identical outputs compared to the standard implementation.
Identical is the load-bearing word. A latency optimization that alters outputs is a change to the model and has to be documented and revalidated as one. A latency optimization that provably does not is an infrastructure change.
The same distinction runs through continuous batching, KV cache reuse and eviction policy. These are properties of the serving stack, not of the model, and they are available to you only if you run the serving stack.
On a hosted API you get the caching window, batch behavior and routing the vendor chose to expose, at the granularity the vendor chose.
Running it yourself, you set the cache TTL, the batch window, the draft model and the routing threshold, and you change them when your traffic changes.
That is the same argument for owning the platform that applies to AI-ready architecture in financial services, arriving one layer lower in the stack.
Does inference routing create model risk under SR 26-2?
Probably yes for the router itself, and no for the model it routes to — a split that most coverage of the guidance misses, because it stops reading the relevant footnote halfway.
SR 26-2, Revised Guidance on Model Risk Management, was issued jointly by the Federal Reserve, the OCC and the FDIC on April 17, 2026, superseding SR 11-7 from April 2011 and SR 21-8 from April 2021, and is described as most relevant to banking organizations with over $30 billion in total assets.
Footnote 3 is the passage that settles this, and it has to be read to the end. It opens: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance."
It continues: "Nonetheless, a banking organization's risk management and governance practices should guide the determination of appropriate governance and controls for any tools, processes, or systems not covered in this document."
Then its last sentence reverses the answer most readers take from the first two: "However, the principles described in this guidance apply to traditional statistical and quantitative models and non-generative, non-agentic AI models."
A RouteLLM-style router is a trained classifier. It generates nothing and it takes no actions, which places it squarely inside that final sentence rather than inside the generative-AI carve-out.
So the split is narrow and worth getting right. The served LLM sits outside the guidance; the router that decides which LLM serves a request probably sits inside it, as a non-generative AI model the principles expressly cover.
Caching and batching are not models at all, so they fall to the footnote's middle sentence: a governance question the institution answers for itself, which is a heavier obligation than a covered-model designation, not a lighter one.
What that demands in practice is a record. Which model served which request, under which routing policy, against which cached prefix, and when that policy last changed.
A router that silently downgrades a request from a frontier model to a small one has changed the system's behavior with no evidence trail.
That is a governance defect regardless of whether a supervisor currently calls it a model, and it is only fixable inside a serving layer you operate and can log.
How does ibl.ai give a financial firm control of the serving layer?
With ibl.ai you own all the code and the data.
The platform runs on the firm's own infrastructure with full source code access, so caching policy, routing thresholds, batch windows and audit logging are configuration you set rather than vendor behavior you observe.
It is model-agnostic across any LLM, so a routing policy survives a model swap instead of being rebuilt around it.
Billing is usage-based with no per-seat pricing, which is what makes inference optimization show up on the invoice at all, and you can deploy anywhere from your own cloud to on-premise, GovCloud or a fully air-gapped network.
For a model risk function, the useful property is inspectability. The routing decision, the cached prefix and the model that served each request are yours to log and produce, because the code that makes those decisions is running inside your perimeter.
1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University.
ibl.ai is family-owned and operated from New York, NY.
Related reading: AI-ready architecture for financial services — the same ownership argument one layer up, at the platform rather than the serving tier.
Sources: the architecture finding and the implementation-details conjecture from Narang et al., Do Transformer Modifications Transfer Across Implementations and Applications?; the ranking of architecture below data, evaluation and systems from Yann Dubois's Stanford CS229 guest lecture, Building Large Language Models; Claude per-token, cache-multiplier and batch prices from Anthropic's pricing documentation; GPT-6 Astra prices and the batch discount from OpenAI's API pricing page; Gemini prices, context-caching and batch rates from Google's Gemini API pricing; routing results from RouteLLM; the 2x-3x identical-output speedup from Leviathan et al.; the April 17, 2026 issuance, $30 billion applicability and the full text of footnote 3, including its extension to non-generative, non-agentic AI models, from SR 26-2.