---
title: "Hardware Sizing Tables"
slug: "appendix-d-hardware-sizing-tables"
kind: "appendix"
order: 26
label: "Appendix D"
letter: "D"
summary: "Sizing guidance for local inference at the three tiers from Chapter 7. The arithmetic is shown so that it can be redone when the next generation of hardware arrives. The figures for memory and band…"
words: 714
---

Sizing guidance for local inference at the three tiers from Chapter 7. The arithmetic is shown so that it can be redone when the next generation of hardware arrives. The figures for memory and bandwidth are the manufacturers' published specifications at the time of writing; the throughput figures are estimates derived from them by the method below, not measurements, and should be validated on the institution's own workload before purchase.

## The arithmetic

Two quantities determine whether a machine can serve a model and how fast.

**Fit.** A model must fit in memory with room for the working context. Parameters times bytes per parameter, plus a margin of roughly 20 percent for the context and runtime.

|Precision|Bytes per parameter|70B model|200B model|400B model|
|---|---|---|---|---|
|16-bit|2|140 GB|400 GB|800 GB|
|8-bit|1|70 GB|200 GB|400 GB|
|4-bit|0.5|35 GB|100 GB|200 GB|

**Speed.** For one conversation, each generated token requires reading the whole model from memory once, so tokens per second for a single user is bounded by memory bandwidth divided by model size in bytes. Serving many users at once batches that read, so aggregate throughput rises with concurrency until compute becomes the limit.

Single-user ceiling (tokens/second) ≈ bandwidth (GB/s) ÷ model size (GB).

Conversational use feels fluent above roughly 15 tokens per second; batch and background work tolerates far less.

## The department tier: Apple silicon

|Machine|Unified memory (max)|Bandwidth (approx.)|Largest practical model (4-bit)|Single-user ceiling on a 35 GB model|Suited to|
|---|---|---|---|---|---|
|Mac mini, M4 Pro|64 GB|~270 GB/s|~70B class, tight|~7 tok/s|A small team; classification and summarization at volume; a private assistant for a handful of people|
|Mac Studio, M4 Max|128 GB|~550 GB/s|~200B class|~15 tok/s|A department; a capable private model for a few dozen people|
|Mac Studio, M3 Ultra|512 GB|~800 GB/s|Several-hundred-billion class|~22 tok/s|A department needing the largest open-weight models locally, at modest concurrency|

Concurrency on this tier is limited; plan for a handful of simultaneous conversations at fluent speed, more for background work.

## The team tier: NVIDIA desk-side systems

|Machine|Memory|Notes from the manufacturer|Suited to|
|---|---|---|---|
|DGX Spark (GB10)|128 GB unified|Positioned for models up to ~200B parameters at reduced precision; two units can be linked for roughly double|A team developing, fine-tuning and serving mid-sized models on the data-centre software stack; restricted workloads for a group|
|DGX Station (GB300)|~784 GB coherent|Desk-side form factor with data-centre class GPU memory|The largest open-weight models at useful precision for a team, with real concurrency; a restricted-data inference node for a division|

Both run the same CUDA software stack as the rack systems, so a model served here moves to the data centre without change.

## The institution tier: data-centre GPUs

Rack-scale DGX systems and equivalent multi-GPU servers. Sizing at this tier is a function of three inputs the institution supplies: peak concurrent conversations, the model class to serve, and the latency target. The method:

1. Choose the model and precision; compute its size from the fit table.
2. Choose the GPU memory per node; the model must fit on one node or be sharded across a node's GPUs with the serving software's support.
3. Measure, with the free serving software and the real model, aggregate tokens per second at the target concurrency on one node.
4. Divide peak demand by that figure for the node count; add one for failover.

Do not accept a vendor's single-user token rate as a proxy for step 3.

## Routing across machines

Where an institution has machines at several tiers, a local inference router (NVIDIA's Personal AI Router, open-sourced in September 2026, is one example) treats them as one pool for independent requests: each request goes to one eligible node; models are never split across nodes. Eligible nodes include RTX-class GPUs from the 20 series onward, DGX Spark and Apple M4-class systems. Use it to put idle machines to work on volume and background tasks, not to replace a data-centre node for interactive load.

## Budget lines beyond the box

- Inference serving software (free, open source) and the person who runs it.
- Monitoring and a model catalogue, supplied by the platform.
- Power and cooling, especially at the institution tier.
- A refresh cycle: assume the hardware is superseded in two to three years and the model it was bought for is superseded in one. Both are fine, because the platform's router makes the hardware a setting too.