Choosing hardware for local LLM inference

The companion article “Capacity and throughput bounds for local LLM inference” builds a performance model: capacity decides whether a checkpoint fits, memory traffic usually limits batch-one generation, and compute and kernel quality usually matter more for long-prompt prefill and larger batches. This article turns that model into a decision a reader can reproduce: a fixed workload, a fixed quality reference, and two metrics — intelligence per dollar and intelligence per kilowatt — computed from documented inputs rather than vendor token-rate claims.

Every price and availability claim below is dated. GPU and DRAM prices moved sharply through 2026; a hardware decision is only reproducible if the reader knows which price was quoted on which day.

Ground: the objects and the fixed inputs

What “intelligence” means in the metric

The word is deliberately defined, not used as a general label. A task instance is one execution of a fixed prompt against a fixed grader under a fixed sampling configuration. A task suite is a fixed set of those instances. Quality for this article means the model’s measured score on that suite, normalized against a chosen reference model.

The metric is therefore task-specific. It does not rank “intelligence” in the abstract; it ranks how much usable task performance a complete system delivers per dollar spent and per kilowatt drawn while meeting latency limits. Every term is defined in the Derive section before it appears in a formula.

The fixed task suite and latency constraints

The worked example intends a fixed suite of 256 instances of an arithmetic word-problem task, each with a 512-token prompt and a 512-token reference-length generation, run at batch/concurrency 1. Reproducibility caveat: the exact task dataset and revision, the 256-instance selection, the prompt/chat template, the expected-answer extraction and grader, and the sampling configuration are not specified. Until that definition is fixed, no run is reproducible, so R (measured completed-task rate) and any suite-measured Q are unreproducible pending this task definition: only the metric formulas and the reasoning that does not depend on a measured run stand. The operator-chosen latency constraints are a 95th-percentile time to first token (p95 TTFT) of at most 60 s and a 95th-percentile inter-token latency (p95 ITL) of at most 150 ms. These are constraints, not measurements; they come from the operator’s own interactive tolerance and must be stated so the reader sees what the decision is optimizing against.

The documented-quality reference is published AIME (American Invitational Mathematics Examination) accuracy, a fixed, externally graded problem set. The reference model is the dense baseline below; its own published score is normalized to Q = 1.0. Details and citations are in Derive and Apply.

The three fixed model artifacts

Each model below is an example of a different constraint shape: a dense baseline, a mid-size Mixture-of-Experts (MoE) model with low-bit weights, and a very large sparse model that no consumer machine can hold. They are not quality tiers; their task performance, licenses, and prompt formats differ. What is fixed is the exact revision and artifact, so the capacity arithmetic is verifiable.

Dense baseline — Qwen3-32B. Qwen3-32B is a dense 32.8B-parameter causal language model. Its pinned configuration records 64 layers, 8 key/value heads, head dimension 128, and 32,768 tokens of native context. The selected artifact is Qwen3-32B-Q4_K_M.gguf at revision 938a7432affaec9157f883a87164e2646ae17555, exactly 19,762,149,024 bytes with LFS SHA-256 efd971561896866f0e910cce52761ca77b1b138090c7f15fe284676d57d1f689. This is the same artifact pinned in the first article, so the two rewrites share one consistent dense baseline throughput constraint.

Candidate — gpt-oss-120b (MXFP4). gpt-oss-120b is OpenAI’s open-weight, 117B-total-parameter decoder MoE model with 5.1B active parameters per token. Its weights shipped pre-quantized in MXFP4; the model card states this stores MoE weights at a nominal 4.25 bits/parameter and that the larger 120B model runs within 80 GB. In the Hugging Face repository (revision b5c939de8f754692c1647ca79fbf85e8c1e70f8a), the MXFP4 distribution is the 14 model-<n>-of-14.safetensors shards, which total 65.25 GB by derivation from the pinned repository tree; the full-precision original/ shards are separate and are not the artifact that runs. This 65.25 GB is the capacity input, not a claim about the executed kernel.

Capacity-boundary example — DeepSeek-V4.1-Flash. DeepSeek-V4.1-Flash is a 552B-backbone plus 196B-Engram MoE model with a Causal Encoder-Decoder architecture, 8B active parameters in prefill and 16B in decode, and a 890-byte persistent cache per token (from the DeepSeek technical report). Its small active count does not make the checkpoint small; total parameters exceed 700B, so it cannot fit any system in this article’s candidate set. It is included to show how the decision procedure rejects a model on capacity before any dollar or watt arithmetic, and its metrics are marked unavailable for that reason.

The three candidate systems and their dated prices

The comparison holds the workload constant and varies the machine plus the artifact it runs. All numbers below are facts with dated prices. Prices and availability are sourced to first-party product pages or the manufacturer’s own pricing announcement where available; where only reporting exists, the source is explicitly a secondary dated report/tracker with its retrieval date stated. All entries are specifications, not measured inference rates.

System	Memory (unified unless noted)	Published bandwidth	Relevant compute path	Dated price (USD, all dates 2026)
Apple M5 Max Mac Studio (40-core GPU, 128 GB)	128 GB unified	614 GB/s	Neural Accelerators; Metal TensorOps for BF16 and INT4/INT8	≈ $4,799 (base $2,499 + $300 GPU + $2,000 memory); ships 2026-09-22
NVIDIA DGX Spark / GB10	128 GB coherent unified	273 GB/s	Blackwell FP4/FP8 Tensor Cores	$4,699 MSRP since 2026-02-23
NVIDIA RTX 5090 (GPU only; needs a host)	32 GB GDDR7	1,792 GB/s	Blackwell FP4/FP8 Tensor Cores	MSRP $1,999 at launch; street ≈ $5,199–6,900 in Sep 2026

Sources: first-party — Apple M5 Mac Studio announcement, Apple M5 Mac Studio specifications, NVIDIA DGX Spark, NVIDIA developer-forum price-change announcement (2026-02-23), and RTX 5090. Secondary dated reporting — Apple M5 Mac Studio price/availability reporting (MacRumors; retrieved 2026-09-17, covering the announced config/price through mid-September availability) and September 2026 RTX 5090 price trackers (VideoCardPrices; retrieved 2026-09-17, covering the September street-price range).

The RTX 5090 row is deliberately not used in the worked decision: a decision must include the complete required system in the price, and the 5090 requires a separate host, power supply, and cooling, so its $1,999 MSRP is not a complete-system cost. It appears only as a reference point for what a Blackwell FP4 path looks like, as requested by the issue brief.

Vendor bandwidth caveat. Every bandwidth figure above is a manufacturer specification, not sustained application bandwidth. A ceiling derived from it is optimistic and falsifiable, not a benchmark. The first article required this caveat for every vendor peak; this article observes it for every machine it computes a ceiling on.

Derive: the metrics, the rate model, and the ceilings

Define the four inputs

Define the four quantities in their natural units, then combine them.

• Q — normalized quality score on the fixed task suite. Q is dimensionless. It equals the model’s measured score s on the fixed suite (here proxied by published AIME accuracy) divided by the reference model’s score s₀ on the same suite: Q = s / s₀. The reference is the baseline, so the baseline has Q = 1.0 by construction. The scoring rule and the reference must be documented with the numbers; a different grader or reference changes Q.

• R — completed task instances per hour at a fixed concurrency while meeting latency limits. R has units instances/hour. It is derived from measured end-to-end latency — time to first token and inter-token latency — not from a vendor token-rate claim. For concurrency 1, if a task instance takes mean wall time T seconds (TTFT plus G generated tokens at mean ITL), then R = 3600 / T instances/hour, with T restricted to runs whose p95 TTFT and p95 ITL both meet the stated constraints. At concurrency B > 1 the formula changes because aggregate throughput replaces a single stream; this article pins B = 1.

• C — dated purchase price of the complete required system. C has units USD. It must include the host, memory, accelerators, and required interconnect — a GPU’s MSRP alone is not C. Prices are dated at the point of quote.

• W — measured steady-state wall power in kilowatts. W has units kW, measured at the wall with a power meter under the same run and workload, after clocks settle, with the idle/included-power convention stated. It is a measurement, not a rated TDP.

Combine them into the decision metrics

The decision metrics are

task intelligence per dollar  = Q × R / C      [normalized-qualities × instances per hour / USD]
task intelligence per kilowatt = Q × R / W      [normalized-qualities × instances per hour / kW]

Both are derivations from the four definitions: Q × R is usable task throughput weighted by quality (1/hour), divided by the dated price or measured power. Dividing by C gives useful work per dollar spent; dividing by W gives useful work per kilowatt drawn. Neither metric is meaningful until each of its four inputs is documented with units and a date or measurement record. A vendor token-rate claim must not be substituted for the measured R behind these quotients.

Derive the throughput ceiling from sourced bandwidth (allowed upper bound)

R itself is a measured quantity. When no measured run exists, the article may substitute a derived upper-bound ceiling built from a sourced bandwidth, never a fixed efficiency factor. From the first article’s decode bound, decode rate ≤ BW / Q_decode tokens/second, where Q_decode is bytes read from the limiting memory tier per generated token, at batch one. With a predominantly resident checkpoint that reads its full weight payload W_bytes each token and a cache that adds K_bytes per token, the ceiling is

r_ceiling = BW / (W_bytes + K_bytes)      [tokens/second]

The ceiling’s task-throughput form, for G generated tokens and negligible prefill, is

R_ceiling = 3600 × r_ceiling / G          [instances/hour]

This is labelled an upper ceiling, not a prediction. Real kernels do not sustain full advertised bandwidth; the ceiling only says that no honest measurement may exceed it. Because prefill is unmeasured here (no sustained compute throughput is published for the selected shapes on either machine), TTFT is marked unavailable and the ceiling covers decode only, which is the bandwidth-bound regime the first article identified for batch-one generation.

The decision procedure

The procedure is deterministic once the inputs are filled:

1. Capacity screen. Check W + K + A ≤ M (checkpoint bytes plus allocated KV cache plus live allocation ≤ machine memory), as derived in the first article. A model that fails the screen is rejected before any dollar or watt arithmetic; its metrics are unavailable.
2. Quality screen. Confirm the model’s documented Q on the fixed suite meets the operator’s minimum quality threshold.
3. Rate screen. Run the fixed workload on the machine and record TTFT, ITL, and aggregate throughput; confirm p95 TTFT and p95 ITL meet the constraints. Only then is R defined.
4. Measure power. Measure steady-state wall power W under the same run.
5. Combine. Compute Q × R / C and Q × R / W. The higher value on each axis is the better choice under that axis.

The procedure cannot rank machines until step 3 and 4 produce measurements. What a vendor table alone can do is steps 1 and 2: reject what cannot fit and record what quality each model documents.

Apply: reproduce the comparison for one workload

This section completes the procedure as far as the evidence allows, by performing a single constrained baseline-versus-candidate decision and marking exactly which inputs remain unmeasured.

Worked baseline: Qwen3-32B Q4_K_M on an M5 Max Mac Studio

Fixed inputs.

• Quality. Qwen3-32B’s technical report publishes an AIME’25 score of 72.9 in thinking mode (a fact about the published checkpoint, first-party; the flagship Qwen3-235B-A22B scores 81.5 — the two must not be conflated). Per the metric’s own definition, Q must be the selected Q4_K_M artifact’s measured suite score s, not the source checkpoint’s published score. No such measured run is recorded, so the artifact’s absolute Q on the fixed suite is unavailable, not estimated. As the reference model, Q = 1.0 by construction (the reference is normalized to 1.0), independently of the unmeasured suite score.
• Capacity. Artifact W = 19,762,149,024 bytes = 18.405 GiB (file, SHA efd971…f689). At a bounded 8,192-token context with a 2-byte K/V cache, K = 262,144 bytes/token × 8,192 = 2.147 GB (2 GiB), by the KV formula from the first article. W + K ≈ 21.9 GB (fits the 128 GB M5 Max with ~106 GB of memory left for A). Derivation: capacity screen passes by wide margin.
• Price. Configurator price ≈ $4,799 (base $2,499 + $300 for the 40-core GPU + $2,000 for 128 GB), dated 2026-09-17, sourced to Apple’s Mac Studio announcement and configuration reporting. This is C.
• Rate R and power W. Unavailable. No measured llama.cpp Metal runtime TTFT/ITL/throughput or steady-state wall power exists in hand for this exact artifact, machine, and workload. No fixed efficiency factor is substituted.

Derived ceiling (allowed). From Apple’s 614 GB/s specification, and including the article’s stated 2,147,483,648-byte (2 GiB) KV-cache traffic per generated token read, the single-stream decode ceiling is

r_ceiling = 614×10^9 bytes/s / (19,762,149,024 + 2,147,483,648) bytes/token ≈ 28.0 tokens/s
R_ceiling = 3600 × 28.0 / 512 ≈ 197 instances/hour (decode-only, prefill excluded)

This is labelled an optimistic ceiling derived from a sourced bandwidth, not a measurement.

Metric state. Q × R / C and Q × R / W: not computable until R and W are measured. The computable subterm is Q / C = 1.0 / 4,799 ≈ 2.08 × 10⁻⁴.

Worked candidate: gpt-oss-120b (MXFP4) on a DGX Spark

Fixed inputs.

• Quality. The gpt-oss model card publishes 95.8 on AIME 2024 without tools at high reasoning effort (fact, first-party, reasoning-effort-dependent). Compared only as published checkpoint results against the baseline’s AIME’25 72.9, the raw ratio is 95.8 / 72.9 ≈ 1.31 (recomputed from the corrected baseline of 72.9, not 81.5). This ratio is not a usable Q: the two AIME years are incomparable, the baseline’s 72.9 is the source checkpoint’s published score rather than the selected GGUF artifact’s measured suite score, and no common-suite run is recorded. The candidate’s Q on the fixed suite is therefore unavailable, not estimated.
• Capacity. MXFP4 artifact W = 65.25 GB (derived from the pinned repository tree and shards). Attention, embedding, LM head, and router tensors are excluded from MXFP4 quantization per the repository config (modules_to_not_convert), so resident payload exceeds 65.25 GB by the size of those BF16 tensors. Derivation: the payload fits 128 GB with a wide margin, consistent with OpenAI’s documented “within 80 GB” requirement. Exact resident bytes need a measurement. The KV cache for this MoE, at 8,192 tokens, is a small addition relative to the 65 GB payload.
• Price. DGX Spark MSRP = $4,699 since 2026-02-23 (as-source NVIDIA forum announcement) — this is C for a complete system (host + 128 GB unified memory + Blackwell, no separate GPU).
• Rate R and power W. Unavailable. No measured runtime (vLLM or llama.cpp) TTFT/ITL/throughput or steady-state wall power for MXFP4 gpt-oss-120b on a GB10 exists in hand.

Derived ceiling (allowed). None reported. A per-token decode ceiling is not derived here: the 65.25 GB MXFP4 distribution is the resident payload, not the per-token decode traffic. For this MoE only 5.1B parameters are active per token (4 experts/token), so dividing NVIDIA’s 273 GB/s by the full 65.25 GB payload would misstate per-token traffic. A defensible ceiling would need the active experts’ byte footprint (active parameters at their storage width plus attention/embedding/router tensors read each token) and the KV traffic, none of which is documented in hand. The candidate’s rate ceiling is therefore unavailable rather than substituted with the full-resident-payload figure.

Metric state. Q × R / C and Q × R / W: not computable until R and W are measured. The computable subterm Q / C for the candidate is unavailable because the candidate’s Q on the fixed suite is unmeasured (see Quality above).

The rejected model: DeepSeek-V4.1-Flash

Capacity screen fails. With >700B total parameters and no shipping local artifact sized for consumer memory, no system in this article’s candidate set holds it. Its metrics are unavailable — it is rejected at step 1 of the procedure, before any dollar or watt arithmetic, which is exactly what the capacity screen is for. Its small 890-byte/token persistent cache does not rescue its total capacity, so it is not a candidate here.

Resulting choice under the explicit constraints

The operator’s constraints are p95 TTFT ≤ 60 s, p95 ITL ≤ 150 ms, and Q ≥ 1.0 (no quality regression against the baseline on the fixed suite).

The price screen is complete: both dated C values are recorded. The quality screen is unavailable, not complete: the baseline’s Q = 1.0 is the reference normalization, but no measured artifact score exists on the fixed suite, and the candidate’s only documented quality evidence (AIME’24 95.8 vs AIME’25 72.9) consists of incomparable published results from different suites, so neither model’s Q ≥ 1.0 can be confirmed until a common-suite artifact run exists. The capacity screen rejects DeepSeek-V4.1-Flash outright.

The rate and power screens are unfilled: neither R nor W is measured for either machine candidate on this workload. Consequently the full metrics Q × R / C and Q × R / W cannot be calculated yet, and no honest ranking follows them. The decision procedure is therefore not a finished purchase verdict; it is a reproducible method whose remaining terms require the two measurements below. What can be stated now:

• On quality per dollar (Q / C), only the baseline is computable (1.0 / 4,799 ≈ 2.08 × 10⁻⁴); the candidate’s Q / C is unavailable because its Q on the fixed suite has no measured run. No quality-per-dollar comparison between the two is possible yet.
• Only the baseline has a derived decode ceiling — ≈ 197 instances/hour including KV-cache traffic — while the candidate’s rate ceiling is unavailable (no reliable per-token active-tensor figure is documented). A ceiling below the required rate rejects a configuration; the baseline ceiling does not contradict the workload’s feasibility, but it bounds neither TTFT nor the candidate.

The exact measurements that would finish the decision:

1. On the M5 Max Mac Studio: run the fixed 256-instance suite under llama.cpp Metal on the pinned Q4_K_M artifact (same runtime, context, batch, and cache flags as the first article), record p95 TTFT, p95 ITL, and aggregate tasks/hour, and measure steady-state wall power at the wall with a meter during the run. That yields R and W.
2. On the DGX Spark: run the same suite with gpt-oss-120b MXFP4 under a supported runtime (llama.cpp GGUF or vLLM), record the same three latency figures and wall power, keeping the tokenizer differences explicit rather than comparing raw tokens/s.

Once R and W exist for both, the two metrics compute directly and the procedure returns a ranking.

Uncertainty and limits

• Quality. Q uses published AIME scores from different suites (AIME’25 vs AIME’24) and effort settings (Qwen thinking mode vs gpt-oss high reasoning); the operator must re-run a single fixed suite on both models to make s comparable. Sample confidence intervals apply.
• Latency/throughput. Not measured; the ceilings are vendor-bandwidth derivations and are optimistic. The harness record must include tokenizer/chat template, sampling, context/output lengths, batch, warmup, runtime/driver/kernel revisions, quantization and cache dtype, power mode, and thermal state, with TTFT, ITL, and aggregate throughput reported separately.
• Power. W requires a wall-meter reading with idle/included-power convention stated; TDP is not W. The values are unavailable here.
• Price. C is dated to 2026-09-17 and moves; the RTX 5090 row shows how volatile street prices became through 2026. The DGX Spark price reflects an announced 2026-02-23 increase.
• Capacity. A (live allocation) is estimated, not measured; the analysis cites the arithmetic and the needed measurement.

A fixed, re-runnable comparison always records the artifact hash, runtime commit, context, batch, cache types, and power mode. With those, a reader who obtains one of these machines and runs the two measurements above completes this decision procedure and obtains a defensible, dated answer instead of a vendor ranking.