--- title: "Constrained Decoding Models Are Not a New Architecture" slug: "constrained-decoding-models-not-a-new-architecture" author: "Mikel Amigot" date: "2026-10-16 12:00:00" category: "Premium" topics: "constrained decoding, structured output, information extraction, named entity recognition, GLiNER, ScaleDown, small language models, open weights, self-hosted AI, model-agnostic AI, JSON schema, AI cost control" summary: "ScaleDown launched a family of task-specific models it calls Constrained Decoding Models in mid-October 2026. Constrained decoding is a well-documented decoding-time technique, not a new architecture — and on ScaleDown's own benchmark page, its paid extraction model leads GLiNER2.5, a free Apache-2.0 model anyone can download, by 0.2 percentage points." banner: "" thumbnail: "" linkedin: | "Constrained Decoding Models just dropped — a new architecture purpose-built for structured tasks." That line is circulating this week. Three things in it are worth checking, because the underlying shift is real and the framing around it is not. What actually shipped: ScaleDown, a startup, launched a family of small task-specific models — Classify, Extract, Summarize — on 13 October 2026, alongside a LangChain middleware integration. $0.05 per 1M input tokens on the managed API, output free, up to 1M tokens of input per call. There is also a self-hosted tier of licensed container images, documented in a newsletter rather than the docs. → "A new architecture." Constrained decoding is a decoding-time technique with peer-reviewed literature behind it. A September 2026 arXiv study benchmarked it across 5 models and 14 structured-output tasks using two existing open-source libraries, Outlines and XGrammar, and raised schema validity from 78.6–92.9% to 100% on models nobody retrained. → "Skips autoregressive token gen entirely." Not for all of them. ScaleDown's own docs say Classify softmax-normalises over your label set and Extract returns spans with character offsets — both non-generative. But Summarize is documented as abstractive: "a shorter, fluent rewrite in the model's own words." That is generation. → "80% of enterprise AI is extraction, not generation." We could not find a survey behind that number, so we are not repeating it. And the part that matters most, from ScaleDown's own benchmark page, in their words: "GLiNER is the one comparison where ScaleDown is not the faster or cheaper option. GLiNER runs locally, so inference is free, and it is faster, while ScaleDown is more accurate." Fastino released GLiNER2.5 on 24 August 2026 under Apache 2.0 — 0.3B parameters, CPU-capable. Their numbers: ScaleDown +15.5 points on classification, and **+0.2 points on extraction.** So on the task the teaser calls the dominant one, the paid model leads a free model you can download by two tenths of a point — on the vendor's own unreproduced figure, which is worth exactly what that caveat implies. The architectural insight is correct: most production AI is a constrained decision, not an essay, and a 340M non-generative encoder is the right instrument. ScaleDown will even run it inside your network — as a licensed container image, per deployment. The question is whether you license the artifact or hold the weights. With ibl.ai you own all the code and the data, self-hosted inside your own perimeter, model-agnostic across any LLM, usage-based with no per-seat pricing, deployable anywhere from your own cloud to a fully air-gapped network. #iblai #OpenWeights #EnterpriseAI #AgenticAI #InformationExtraction #AIInfrastructure --- ## The Short Answer **Constrained decoding is a decoding-time technique, not a new architecture, and ScaleDown's October 2026 launch is a product line of small task-specific models, sold hosted or licensed as containers. The non-generative, Apache-2.0 version shipped seven weeks earlier as GLiNER2.5. For structured extraction the real question is whether you license the artifact or own it — on ibl.ai you own all the code and the data.** A teaser circulating this week reads: *"Constrained Decoding Models just dropped — a new architecture purpose-built for structured tasks. 80% of enterprise AI is extraction, not generation. CDMs skip autoregressive token gen entirely."* The thing underneath it is real and worth acting on. Three of the four claims in that teaser do not survive contact with the primary sources, including the vendor's own documentation. This page checks each one, then does the comparison nobody ran: what the hosted product costs against the open-weight model that ScaleDown itself uses as a benchmark comparator. ## What are Constrained Decoding Models, and what actually shipped in October 2026? Constrained Decoding Models, or CDMs, is a product name. **ScaleDown**, a startup, introduced the term in a launch announcement on X by co-founder and CEO Neal Patel, describing it as "a new AI model architecture." What shipped is a family of three small, single-purpose models — **Classify, Extract** and **Summarize** — alongside an existing fourth ScaleDown model, **Compress**. This page examines the three CDM-branded models and leaves Compress aside. [LangChain published an integration post on 13 October 2026](https://www.langchain.com/blog/building-long-running-agents-with-scaledown-and-deep-agents), which is the firmest public date anchor for the launch, and shipped `langchain-scaledown`, an open-source middleware package. LangChain's own [integration documentation](https://docs.langchain.com/oss/python/integrations/middleware/scaledown) describes the models as "trained for one narrow task each, so they handle the high-volume steps around an agent." The default shape is a metered endpoint. ScaleDown's [quickstart](https://docs.scaledown.ai/quickstart) documents a hosted base URL at `api.scaledown.xyz` authenticated with an `x-api-key` header, and no on-premise, VPC or downloadable-weights option appears anywhere in the 28-page documentation index. Pricing on the managed API, as listed on [LiteLLM's model directory](https://models.litellm.ai/models/decisions) on 16 October 2026, is **$0.05 per 1M input tokens and $0 per 1M output tokens**. LangChain's post puts the input limit at up to 1M tokens per call. The models are also listed [on OpenRouter](https://openrouter.ai/provider/scaledown). **There is also a self-hosted tier, and it is not in the docs — so correct the record if you have read elsewhere that ScaleDown is API-only.** In a [newsletter post from June 2026](https://tinyml.substack.com/p/self-hosted-deployment-of-scaledown), ScaleDown co-founder and CTO Soham Chatterjee describes a deployment that "can run entirely within your infrastructure. Same models, same API, same performance." The details matter more than the headline, and they are where the ownership question actually lives. You get "the full ScaleDown model suite packaged as container images" plus deployment manifests; "licensing is per-deployment rather than per-token"; and the company explicitly supports air-gapped environments with "no license server that requires an outbound connection." So the deployment axis is genuinely covered. The licence axis is not. A licensed container image that happens to contain weights is not the same artifact as weights published under Apache 2.0, and that distinction is the one this post keeps coming back to. That is a coherent product and a reasonable thing to buy. It is still not a new architecture, and the distinction changes what you should do about it. ## Is constrained decoding a new architecture, or an existing decoding-time technique? Constrained decoding is an existing decoding-time technique, and it has peer-reviewed literature behind it. It restricts the set of tokens a model may emit at each step so the output has to conform to a formal grammar or a JSON schema. The decisive evidence is that it works on models nobody retrained. A September 2026 arXiv paper, [*Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap*](https://arxiv.org/abs/2609.23742) by Akash Chavan, benchmarked **five models from three families across 14 structured-output tasks** under three decoding conditions. The two non-native conditions were **Outlines** and **XGrammar** — existing open-source libraries you bolt onto a model you already run. Schema validity went from **78.6–92.9% to 100%** across every model tested. So the capability the teaser attributes to a new architecture is available today as a library over open-weight models in the 0.6B–4B range. No new vendor relationship is required to get it. What *is* genuinely new is a product category: a vendor selling the narrow tasks as separate metered endpoints instead of selling you a general model and letting you constrain it yourself. That is a packaging decision, and packaging decisions are where ownership gets decided. We made an adjacent argument about where the real leverage sits in [The Alpha Is in Inference Engineering, Not Architecture](/blog/inference-engineering-not-architecture-financial-firms): the serving layer you control moves more cost than the model family you pick. ## Do Constrained Decoding Models really skip autoregressive token generation? Of the three CDM-branded models, two do and one does not — and ScaleDown's own documentation says so, which is why this claim is the one to drop rather than repeat. **Classify** is non-generative. ScaleDown's [classify overview](https://docs.scaledown.ai/api-reference/classify-overview) states that it returns a probability for each label you define and that "scores are softmax-normalised across all labels so they sum to 1.0." That is a scoring head over a fixed label set, not token generation. **Extract** is extractive. The [extract overview](https://docs.scaledown.ai/api-reference/extract-overview) describes it as running "Named Entity Recognition (NER) over arbitrary text," where the model "finds matching spans in the text, returning each one with a confidence score, character offsets, and surrounding context." Character offsets are the tell. A model returning offsets into your input is pointing at text, not writing text. **Summarize** generates. The [summarize overview](https://docs.scaledown.ai/api-reference/summarize-overview) is explicit that it is abstractive: it "condenses text into a shorter, fluent rewrite in the model's own words," producing output "without being constrained to the original phrasing" — in contrast to extractive summarization, "which lifts sentences directly from the source." A fluent rewrite in the model's own words is generated text. So "CDMs skip autoregressive token generation entirely" is false for one of the three models by the vendor's own description. This matters beyond pedantry. The non-generative models carry a hard guarantee — the output is a label from your list or a span from your input — and the generative one does not. Those two things need different review controls in production, and a blanket architectural claim erases the difference. ## Does constrained decoding make extraction accurate, or only well-formed? Only well-formed. This is the most commonly missed consequence of constrained decoding, and the Chavan paper states it in one line: **"CD's reach ends exactly where schema conformance ends."** The paper separates two axes — structural correctness, meaning schema validity, from semantic correctness, meaning whether the content is right. Constrained decoding took structural failures to **zero across all five models tested**. Content accuracy did not follow. The gap is scale-dependent and has shape to it. Type coercion failures — a number arriving as a string — are, in the paper's words, "fully CD-rescuable." Instruction-semantic failures, such as multi-step function calling, "remain CD-resistant." For anyone building a production extraction pipeline, the operational reading is blunt: guaranteed-valid JSON is not guaranteed-correct JSON, and a schema validator will tell you nothing about whether the model pulled the right due date out of the contract. That is why confidence scores and character offsets are the features worth paying attention to in this product category. An extraction you can trace back to a span in the source document is auditable. A generated field is not. We have made this argument before in a regulated setting — [Most Healthcare AI Pilots Never Reach Production](/blog/healthcare-ai-pocs-fail-architecture-not-model) identified structured outputs plus deterministic fallbacks plus domain evaluation as the architecture that separates pilots from production systems. ## Is 80% of enterprise AI really extraction rather than generation? We could not find a primary source for that number, so this page does not use it. There is no survey, benchmark or vendor filing we could locate that puts enterprise AI workloads at 80% extraction, and a number without a source is not evidence. The directionally similar claims we did find are weaker than the teaser's phrasing and come from secondary commentary rather than published research, so they are not worth laundering into a statistic either. What can be stated without a survey is the pricing behaviour, which is observable. Vendors are now shipping metered endpoints specifically for classification, extraction and summarization, and pricing them far below general-purpose model APIs. LangChain's integration post cites ScaleDown's own benchmarks at **"200x faster and 467x cheaper than frontier models on these narrow tasks"**, with Summarize at 41× cheaper and 2.7× faster and Compress at 138× cheaper and 1.8× faster. ScaleDown's own [benchmark overview](https://docs.scaledown.ai/benchmarks/overview) reports accuracy advantages against frontier lineups of **+8.72% versus OpenAI, +8.0% versus Anthropic, +9.0% versus Gemini, +7.93% versus Grok, +19.5% versus DeepSeek and +16.71% versus Qwen**, with latency advantages from 2.2× to 13.9×. Those are vendor-run numbers, and they are better documented than most. The [methodology page](https://docs.scaledown.ai/benchmarks/methodology) names real public datasets — **CLINC OOS** for classification, **CUAD** for extraction, **QMSum** for summarization and **FinanceBench** for compression. Sample sizes and metric definitions are published too, in the company's newsletter rather than its docs. The [extraction write-up](https://tinyml.substack.com/p/benchmarking-scaledowns-extraction) cites CUAD's "510 real commercial contracts, expert-annotated by lawyers from The Atticus Project," defines exact-match and token-overlap F1, and reports 0.4961 against 0.4196 exact match versus its GPT-5.4 Mini baseline. The harness is on GitHub as [scalebench](https://github.com/scaledown-team/scalebench). That is more transparency than the category norm, which makes the one arithmetic problem worth naming precisely rather than hand-waving at. **ScaleDown's head-to-head page reports +15.5 pp on classification and +0.2 pp on extraction, then states the average as "+10.5%."** The mean of 15.5 and 0.2 is 7.85. We do not know how 10.5 was derived, and the page does not say. Either way, the headline average is the number to discount and the two per-task numbers are the ones to use — which is what the next section does. ## How does a hosted Constrained Decoding Model compare to the open-weight equivalent? This is the comparison the launch coverage skipped, and the numbers come from ScaleDown's own benchmark page rather than from us. ScaleDown's [head-to-head page](https://docs.scaledown.ai/benchmarks/providers) names its open-source comparator directly: **GLiNER2.5-multi-v1**, which the page correctly describes as "a local open-source model for classification and extraction." Credit where it is due, because the vendor says the hard part out loud: "GLiNER is the one comparison where ScaleDown is not the faster or cheaper option. GLiNER runs locally, so inference is free, and it is faster, while ScaleDown is more accurate." GLiNER2.5 was released by **Fastino on 24 August 2026 under Apache 2.0** — seven weeks before the CDM launch — in three sizes: gliner2.5-small-v1 at 74M parameters, gliner2.5-base-v1 at 0.2B and gliner2.5-multi-v1 at 0.3B, per [Fastino's release post](https://fastino.ai/blog/gliner2-5-span-free-information-extraction). And the architecture claim lands here, not on the hosted product. In Fastino's words, the architecture "replaces span enumeration with boundary prediction, scoring entity start and end positions directly." The lineage is documented in [GLiNER2](https://arxiv.org/abs/2507.18546) and the original [GLiNER paper](https://aclanthology.org/2024.naacl-long.300.pdf), with code at [github.com/fastino-ai/GLiNER2](https://github.com/fastino-ai/GLiNER2). Here is what ScaleDown reports against it:
Task ScaleDown accuracy lead GLiNER2.5 latency GLiNER2.5 token cost
Classification +15.5 pp 22 ms $0
Extraction +0.2 pp 480 ms $0
Average across both +10.5% — $0
Read it honestly in both directions. On **classification**, a 15.5-point accuracy lead is a real product advantage, and $0.05 per million tokens is a fair price for it. On **extraction** — the task the teaser calls the dominant enterprise workload — the paid model leads a free, downloadable, Apache-2.0 model by **two tenths of a percentage point**, and that is the vendor's own figure for its own product. One caveat we have to state, because it cuts against the conclusion we like: ScaleDown publishes no GLiNER configuration for that run — no checkpoint hash, no schema, no prompt, no split, no sample size — and the scalebench repository does not appear to contain it. So +0.2 pp is an unreproduced number about a competitor's model, published by the competitor. Treat it as the vendor's own claim, not as a measurement. Then go and run `gliner2.5-multi-v1` on your own documents, which is the only number that decides anything. On that figure as published, the open-weight model is not a compromise on extraction. It is a rounding error away on accuracy, faster by the vendor's own account, costs nothing per token, and runs inside your own perimeter where the documents never leave. ## Which model actually skips autoregressive generation, and can you download it? The model that most precisely matches the teaser's description is open-weight, and it shipped under three weeks before the launch that was described as new. **GLiNER2.5-Decide**, released by Fastino on **24 September 2026 under Apache 2.0** at **340M parameters**, is described in [Fastino's announcement](https://fastino.ai/blog/gliner-2-5-decide-open-weight-decision-model) as "a specialized, non-generative encoder" that uses "a constrained decoder" to search only the answers "allowed by the declared rules." It returns "valid answers, probabilities, confidence scores, and constraint-feasibility metadata" against user-defined typed questions, and reports **167.3 ms** latency on a 48-vCPU Intel Xeon Platinum 8581C at 64 tokens. Fastino also reports **60.1% average accuracy**, but on its own wording that is across "an unseen, internally generated classification suite containing 5,100 test examples across 17 datasets" — so it earns exactly the discount we just applied to ScaleDown's numbers, and for the same reason. That is, literally, a constrained-decoding model purpose-built for structured decisions that skips autoregressive generation entirely — with weights you can download, inspect, fine-tune and run on a CPU you already own. It is also not an isolated release. We surveyed the whole cohort in [Six Decision Models in Three Weeks](/blog/decision-layer-models-nine-days-own-the-router), covering six decision models that shipped between 15 September and 6 October 2026 and comparing their licences. Read that one for the category. Read this one for what the 13 October launch adds to it. So the architectural thesis in the teaser is correct and the attribution is wrong. The industry *is* moving from "ask a frontier model to write JSON" toward small non-generative encoders that score a constrained output space. That movement is happening in the open, under permissive licences, at parameter counts that fit on a laptop. The 340M model and the commercial product are the same idea on different ownership terms. One is a file published under Apache 2.0; the other is a licensed container image, whether you run it on ScaleDown's hardware or your own. This is the same pattern we examined in [A 9B Fine-Tune Beat Opus 4.5. Who Holds the Weights?](/blog/coinbase-9b-fraud-fine-tune-who-owns-the-weights) — a task-specific small model outperforming a frontier model on one company's benchmark, where the durable question was not parameter count but who held the artifact afterwards. ## What should you do this quarter about structured extraction? Five concrete steps, in order, none of which require buying anything first. **Inventory what your pipelines actually ask for.** Separate the calls that need a label from a fixed list, the calls that need spans from a document, and the calls that genuinely need prose. The first two categories do not need a frontier model, and they are usually the majority of the call volume. **Measure on your own documents.** CLINC OOS, CUAD, QMSum and FinanceBench are legitimate public datasets, but none of them is your contracts, your tickets or your claims. A headline average from either vendor tells you nothing about your corpus. **Test the open-weight baseline before the hosted one.** GLiNER2.5 is Apache-2.0, CPU-capable and installable in an afternoon. If it clears your accuracy bar on extraction, every token you would have metered is a token you do not pay for, and your documents never leave your network. **Put constrained decoding on whatever you already run.** Outlines and XGrammar raised schema validity to 100% on five different open-weight models in the Chavan study. That is a library change, not a procurement cycle. **Then evaluate semantics separately from structure.** Valid JSON is not correct JSON. Build the content-accuracy evaluation on your own labelled data, because that is the axis constrained decoding cannot help with. ## Can you run constrained, structured extraction on infrastructure you own? **ibl.ai is the agentic AI platform where you own all the code and the data. You self-host the entire stack inside your own perimeter, run it model-agnostic across any LLM and switch anytime, and pay by usage with no per-seat pricing — so you can deploy anywhere: your cloud, on-premise, GovCloud, or fully air-gapped.** Note what the distinction is and is not. Running a vendor's container inside your own network is a real capability, and several vendors in this category now offer it. But a per-deployment licence over an image you did not build is a tenancy, not a title. Source code under a perpetual licence is the different thing. Concretely, for a structured-extraction workload: the small non-generative models run beside the large generative ones inside the same perimeter, so a classification call never crosses a network boundary. [Agentic OS](/product/agentic-os) is the flagship, and every other product is a module built on it. Model-agnostic matters more here than it does for chat. An extraction pipeline tuned on one model's output is the easiest thing in an AI stack to get locked into, because the schema, the prompts and the post-processing all encode assumptions about one provider. For deployments where the documents cannot leave — contracts, claims, PII, classified material — [air-gapped AI](/service/air-gapped-ai) runs the whole stack with no outbound connectivity, which is the only configuration in which "we don't send your documents anywhere" is a verifiable statement rather than a policy. And because extraction pipelines process documents by the million, [no per-seat pricing](/pricing) is the structurally correct shape: cost tracks tokens actually processed, not headcount. A per-seat contract on a workload with no seats is a pricing mismatch, not a discount negotiation. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on. ## Frequently asked questions about Constrained Decoding Models **Is "Constrained Decoding Model" a standard technical term?** No. It is ScaleDown's product name, introduced in October 2026. The underlying technique, constrained decoding, is standard and predates the product by years. **Can ScaleDown run on my own infrastructure?** Yes. The company documents a self-hosted tier of container images with per-deployment licensing, including air-gapped configurations, in a June 2026 newsletter post. It is not in the product documentation, which is why several write-ups have called it API-only. **Can I get constrained decoding without a new vendor?** Yes. Outlines and XGrammar are open-source libraries that apply grammar or schema constraints to models you already run, and the Chavan study measured both taking schema validity to 100%. **Does constrained decoding slow inference down?** It adds work at each decoding step. That cost is model- and grammar-dependent, and we have not measured it on our own stack, so we are not quoting a figure for it here. **Is the open-weight option always better?** No. ScaleDown's own benchmark shows a 15.5-point classification lead over GLiNER2.5, which is a real advantage if it holds on your data. The near-tie is on extraction. Measure both on your corpus. **What is the one claim from the original teaser worth keeping?** That bigger models are not the answer for structured tasks. That part is right, and the open-weight 340M non-generative encoders are the clearest evidence for it. ## Why does owning the AI stack matter? **ibl.ai is the agentic AI platform where you own all the code and the data. You self-host the entire stack inside your own perimeter, run it model-agnostic across any LLM and switch anytime, and pay by usage with no per-seat pricing — so you can deploy anywhere: your cloud, on-premise, GovCloud, or fully air-gapped.** - **You own all the code and the data.** Full source code under a perpetual license, running on your infrastructure. Not API access to someone else's platform — the stack itself is yours. - **Model-agnostic.** Run any LLM — Claude, GPT, Gemini, Llama, Command, or your own fine-tune — and switch providers without rewriting the platform. - **No per-seat pricing.** Usage-based billing against a budget cap you set. Cost tracks what your organization actually uses, not how many people you employ. - **Deploy anywhere.** Your cloud, your VPC, on-premise, GovCloud, or a fully air-gapped network with no outbound connectivity. 1.6M+ users across 400+ organizations run the platform this way, including NVIDIA, MIT, and Syracuse University. ibl.ai is family-owned and operated from New York, NY — a U.S.-headquartered, domestically-owned long-term partner, not a vendor that sells licenses and moves on.