Chapter 7Hardware You Can Put Your Hand On

The previous two chapters assumed that an institution can run open-weight models on hardware it controls. This chapter is about that hardware: what to buy for which scale, what the honest limits are, and how to read a specification sheet without being misled. I will name products because a CIO has to buy specific things, and I will give the numbers that determine fit rather than the ones that sell.

Why memory, not FLOPS, is the number

Marketing for AI hardware leads with floating-point operations per second. For running a language model to serve people, the number that matters first is memory, and the number that matters second is memory bandwidth.

A model has to fit in memory to run at all. A model with 70 billion parameters needs roughly 140 gigabytes at 16-bit precision, 70 at 8-bit, and about 35 to 40 at the 4-bit quantizations that have become standard for serving, with a modest quality cost. A 400-billion-parameter model needs five to six times that. If the machine does not have the memory, no amount of compute helps; the model does not load.

Once the model fits, the speed at which it produces each token is bounded, for a single conversation, by how fast the machine can read the whole model from memory once per token. Divide memory bandwidth by model size in bytes and you have a ceiling on tokens per second for one user. A machine with 800 gigabytes per second of bandwidth serving a 40-gigabyte model cannot exceed about 20 tokens per second for one conversation, regardless of how many FLOPS it advertises. Serving many users at once amortizes this, which is why the throughput figures for data-centre systems look so different from the figures for a desktop.

With those two numbers in hand, the hardware market sorts itself into three tiers.

The department tier: Apple silicon

Apple's desktop machines have become a serious, slightly accidental, option for local inference because they put a large pool of fast unified memory next to a capable GPU in a quiet box that plugs into a wall socket.

A Mac mini with an M4 Pro processor can be configured with 64 gigabytes of unified memory and offers memory bandwidth in the high two hundreds of gigabytes per second. That is enough to run quantized models in the 30-billion-parameter class comfortably, for a handful of simultaneous users, at conversational speed. It costs about what a well-equipped laptop costs and fits on a shelf.

A Mac Studio with an M3 Ultra can be configured with up to 512 gigabytes of unified memory and roughly 800 gigabytes per second of bandwidth. That memory ceiling is the remarkable part: it will load quantized models in the several-hundred-billion-parameter class that otherwise require a multi-GPU server. Single-conversation speed on the largest models is modest, a handful of tokens per second, but for a department that wants a capable private model for a few dozen people, with no data leaving the building, it is a credible answer at a price that does not require a capital committee.

Honest limits: Apple's platform is not where the inference-serving ecosystem is most mature, concurrency is limited compared with data-centre GPUs, and there is no path to scale a single machine beyond its box. It is a department tier, and a good one.

The team tier: DGX Spark and DGX Station

NVIDIA's desk-side systems bring the data-centre software stack, CUDA and everything built on it, to a form factor a team can own.

DGX Spark is a compact system built around the GB10 Grace Blackwell chip with 128 gigabytes of unified memory. NVIDIA positions it for models up to about 200 billion parameters at reduced precision on one unit, and two units can be linked to roughly double that. It draws ordinary wall power. For a team that wants to develop, fine-tune and serve mid-sized models on the same software it will later deploy in a data centre, it is the natural first purchase.

DGX Station is the larger desk-side system, built around the GB300 Grace Blackwell Ultra, with roughly 784 gigabytes of coherent memory across the CPU and GPU. It is a workstation in the sense that it sits in an office, and a server in every other sense. It will load the largest open-weight models at useful precision and serve a team with real concurrency.

The team tier is where I would point most institutions beginning a sovereign deployment that needs local inference for restricted data. It is enough compute to matter, it runs the same software as the production systems, and it can be bought, installed and returned without a procurement cycle measured in quarters.

The institution tier: DGX in the data centre

Above the desk sit the rack-scale systems: multi-GPU DGX servers and the larger configurations built from them, in the institution's own data centre or a colocation facility it controls. This is where an institution serves thousands of concurrent users from open-weight models, fine-tunes on its own data at scale, and runs the restricted-classification workloads for a whole enterprise.

The economics at this tier are a straightforward capital-versus-operating comparison. A rack of GPUs is a large purchase with a multi-year life and a predictable power bill. Against it stands the commercial token price for the same volume of work, which is falling, and the per-seat price, which is not. For institutions with the volume, the data classification requirements, or both, the rack pays for itself; for institutions without them, the team tier plus commercial APIs is the better spend. The routing layer from Chapter 6 is what lets an institution start at one tier and move to another without changing anything above the hardware.

Routers across machines

Institutions rarely have one machine. They have a Mac Studio in one department, a Spark in another, a GPU workstation in a lab, and increasingly a desk-side system or two. Software now exists to treat them as one pool.

NVIDIA's Personal AI Router, released as open source in September 2026, is a representative example. It is a router rather than a cluster: it sends each independent request to one eligible machine and never splits a model across machines or pools their memory. Nodes can be RTX-class GPUs, DGX Spark, or Apple M4-class systems. Its value is not raw speed; a reference three-device group was measured in minutes for a heavy workload, not seconds. Its value is utilization: idle machines across an institution become an inference pool for the volume workloads that do not need the data centre, and the routing policy decides what goes there.

What to budget and what not to believe

Three rules for reading a hardware proposal.

Ask for memory capacity and bandwidth first, then concurrency at a stated model size and precision, then price. A proposal that leads with FLOPS and a benchmark headline is selling something other than serving capacity.

Do not believe single-user token rates as a measure of what a machine will do for a team. Ask for throughput at ten, fifty and two hundred concurrent conversations at the model you intend to run, and ask how it was measured. If the vendor cannot answer, run the test yourself before buying; the models and serving software are free.

And budget for the software and the people, not just the box. A machine serving models needs an inference server, monitoring, a model catalogue, access control and someone who updates it. A sovereign platform supplies most of that; the institution supplies the person. Appendix D sets out sizing tables for the three tiers against users, model class and latency, with the arithmetic shown so it can be redone when the next generation of hardware arrives, which it will.