Local inference hardware, models, and techniques in September 2026
A dated map of what current hardware can hold, which low-precision paths it exposes, and which techniques change the result.
The first article develops a performance model: capacity decides whether a model fits; memory traffic usually limits batch-one generation; compute and kernel quality usually matter more for long-prompt prefill and larger batches.
This article applies that model to hardware and models available or announced in September 2026. It is dated on purpose. Product availability and software support change faster than the underlying arithmetic.
The tables below contain vendor specifications, not predicted token rates. A token-rate claim is useful only when it names the model revision, artifact, runtime commit, context, batch, cache type, and power mode. Most product pages do not.
Terms used in the tables
| Term | Meaning |
|---|---|
| GPU | Graphics processing unit. The matrix units inside modern GPUs may be branded as NVIDIA Tensor Cores, Apple Neural Accelerators/TensorOps, AMD matrix cores, or Intel XMX engines. The names do not imply identical instruction support. |
| VRAM | Memory attached to a discrete GPU. HBM means High Bandwidth Memory; GDDR means Graphics Double Data Rate memory. |
| Unified memory | One physical memory pool addressable by CPU and GPU. Apple M-series, AMD Ryzen AI Max, and NVIDIA GB10 expose different forms of this design. |
| LPDDR | Low-Power Double Data Rate memory, commonly used for unified-memory systems. Published bandwidth can be derived from transfer rate and bus width. |
| FP / BF / INT | Floating point, brain floating point, and integer. The number that follows is nominal bits per value: FP8, BF16, or INT4. Scales and metadata can raise effective storage. |
| FLOPS / TOPS | Floating-point or generic operations per second. Vendor TOPS may use integer math, structured sparsity, or a different counting convention, so it is not automatically comparable to dense FLOPS. |
| GQA / MQA | Grouped-query and multi-query attention. They share key/value heads to reduce the KV cache. |
| MLA / CED | Multi-head Latent Attention and Causal Encoder-Decoder. Both change what is cached, but they are different model architectures rather than runtime flags. |
| MoE | Mixture of experts: many expert parameter sets exist, but a router selects a subset for each token. |
| MTP | Multi-token prediction: auxiliary predictions that can support self-speculative decoding. |
| PCIe / RDMA | PCI Express connects devices to a host. Remote Direct Memory Access lets machines exchange memory with less CPU involvement; neither link is equivalent to local device memory. |
| CUDA / ROCm / Metal | NVIDIA’s, AMD’s, and Apple’s GPU software stacks. Hardware support matters only when the selected runtime and kernel expose it. |
| TTFT / ITL | Time to first token includes prompt processing. Inter-token latency measures delay between generated tokens. Aggregate tokens/s at concurrency is a different metric. |
Hardware: capacity first
The most useful first comparison is memory capacity and bandwidth. Peak low-precision compute comes after that because a model that does not fit on the fast memory path cannot use the peak continuously.
| System | Memory | Published bandwidth | Relevant compute path | Status on September 11, 2026 |
|---|---|---|---|---|
| MacBook Pro with M5 Max | up to 128 GB unified | 460 or 614 GB/s, depending on GPU configuration | Neural Accelerators in each GPU core; Metal TensorOps for BF16 and INT4/INT8 | Shipping product |
| Mac Studio with M5 Max | up to 128 GB unified | up to 614 GB/s | Same M5 TensorOps path; up to 40 GPU cores | Announced; general availability September 22 |
| Mac Studio with M5 Ultra | up to 512 GB unified | 1.2 TB/s | Same M5 TensorOps path; up to 80 GPU cores | Announced; general availability September 22; 512 GB due in late October |
| NVIDIA DGX Spark / GB10 | 128 GB coherent unified | 273 GB/s | Blackwell FP4/FP8 Tensor Cores; NVIDIA advertises up to 1 PFLOP FP4 with sparsity | Shipping product |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition | 96 GB GDDR7 ECC | 1,792 GB/s | Blackwell FP4/FP8 Tensor Cores, including NVFP4 software paths | Shipping product |
| NVIDIA RTX 5090 | 32 GB GDDR7 | 1,792 GB/s | Blackwell FP4/FP8 Tensor Cores | Shipping product |
| AMD Ryzen AI Max+ 395 | up to 128 GB unified | 256 GB/s theoretical from LPDDR5X-8000 on a 256-bit bus | RDNA 3.5 GPU; do not infer FP4 support from the memory specification | Shipping systems exist |
| AMD Ryzen AI Max+ PRO 495 | up to 192 GB unified | 273 GB/s theoretical from LPDDR5X-8533 on a 256-bit bus | 40-core RDNA 3.5 GPU; no first-party FP4 claim | Product announced; system availability depends on the vendor |
| AMD Radeon AI PRO R9700 | 32 GB GDDR6 | 640 GB/s | AMD advertises FP8, FP16, and INT8 support | Shipping product |
| Intel Arc Pro B60 | 24 GB GDDR6 | up to 456 GB/s | 160 XMX engines; the runtime still needs a compatible quantized kernel | Shipping product |
Sources: Apple M5 Mac Studio, Apple M5 MacBook Pro specifications, Apple M5 TensorOps, DGX Spark, RTX PRO 6000, RTX 5090, Ryzen AI Max+ 395, Ryzen AI Max+ PRO 495, Radeon AI PRO R9700, and Arc Pro B60.
A few qualifications matter more than the table’s rankings.
Unified memory is a capacity advantage
Apple M-series and AMD Ryzen AI Max systems let the GPU address a large system-memory pool without copying resident weights over PCIe for every layer. DGX Spark also presents a coherent CPU/GPU memory pool. This makes models larger than ordinary consumer VRAM practical.
Unified does not mean unlimited. The operating system and applications share the pool. The GPU still has the listed bandwidth, and CPU, GPU, and display traffic can contend for it. Firmware may also limit how much memory a GPU workload can reserve.
Discrete VRAM is a bandwidth advantage
The RTX 5090 and Radeon AI PRO R9700 hold only 32 GB, but their local memory is much faster than the LPDDR pools in the table. If the entire model and KV cache fit, they have a strong batch-one decode path.
Once dense layers spill to host memory, placement matters. Some runtimes execute offloaded layers on the CPU and move activations across PCIe. Others stream weights to the GPU. These have different traffic costs, but both introduce a slower tier. Saying that a model “runs” with CPU offload says nothing about whether its latency is useful.
The RTX PRO 6000 combines 96 GB with 1,792 GB/s. Its capacity, not a different Blackwell instruction set, is the important difference from a 5090 for local inference.
Low-precision peaks are not comparable by default
NVIDIA’s DGX Spark headline uses FP4 with structured sparsity. Apple publishes relative application and matrix speedups rather than directly comparable FP4 FLOPS. AMD and Intel use their own operand and peak conventions.
Before dividing FLOPS by bandwidth, check:
- dense or sparse peak;
- operand format and accumulator format;
- multiply-add counted as one operation or two;
- boost clock and power envelope;
- matrix shape required to reach the peak;
- whether the runtime invokes that instruction at all.
Without those details, a cross-vendor balance-point table creates precision without comparability.
Three models that expose different constraints
One model cannot represent the local-inference landscape. A dense model, an MoE model with low-bit weights, and a very large sparse model stress different parts of the system.
| Model | Published architecture facts | What it tests |
|---|---|---|
| Qwen3-32B | 32.8B dense parameters; 64 layers; 8 KV heads; native 32,768-token context | Ordinary dense weight traffic and a conventional GQA KV cache |
| gpt-oss-120b | 117B total, 5.1B active parameters per token; MoE weights distributed in MXFP4; OpenAI documents operation on one 80 GB GPU | Total capacity versus active expert traffic, plus the difference between stored MXFP4 and the kernel that executes it |
| DeepSeek-V4.1-Flash | 552B backbone plus a 196B Engram table; 8B active parameters in prefill and 16B in decode; 890-byte persistent KV cache per token; one-million-token context | A model with small active compute but enormous total capacity and architecture-specific cache compression |
These are examples, not quality tiers. Their task performance, licenses, prompt formats, and software support differ. The table is about inference shape.
Qwen3-32B: the dense baseline
Qwen3-32B has an ideal BF16 weight payload of 65.6 GB. A raw four-bit payload is 16.4 GB before scales, metadata, alignment, and tensors kept at higher precision.
Its ordinary two-byte KV layout costs 256 KiB per cached token:
2 × 64 layers × 8 KV heads × 128 dimensions × 2 bytes = 256 KiB/tokenAt 32,768 tokens, that is 8 GiB for one sequence. A 24 GB card may fit a particular four-bit artifact only with a shorter or quantized cache and little workspace headroom. A 32 GB card has more room, but the exact artifact and runtime still decide the answer. A 96 or 128 GB pool can hold higher-precision weights or more cache, but bandwidth and kernels decide speed.
This is a good baseline because nearly every byte in the dense layers is relevant to every token.
gpt-oss-120b: active and total parameters separate
OpenAI reports 117B total parameters and 5.1B active per token for gpt-oss-120b. The MoE weights are stored with MXFP4. OpenAI’s own implementations illustrate why the storage format is not the execution path: the readable PyTorch implementation can upcast weights, while the optimized Triton implementation provides an MXFP4-aware MoE kernel.
Capacity must accommodate the distributed artifact and runtime state. Per-token expert work touches only selected experts plus the dense and shared parts of the model. At higher batch or concurrency, different tokens select different experts, increasing the union of weights read.
OpenAI’s documented single-80-GB-GPU requirement is a useful capacity reference. It does not imply that every 80 GB device, framework, or kernel produces the same speed.
DeepSeek-V4.1-Flash: active compute does not erase capacity
DeepSeek-V4.1-Flash was released on September 10, 2026. Its model card reports a 552B-parameter backbone, a 196B conditional-memory table, and only 8B/16B active parameters per token for prefill/decode.
The small active count reduces compute and active expert traffic. It does not make the checkpoint small. Even a hypothetical 4.5-bit representation of 748B parameters would have a raw payload of about 421 GB before higher-precision tensors, scales, metadata, and workspaces. That arithmetic does not establish that such an artifact exists or that a given runtime can execute it efficiently.
The model’s 890-byte persistent KV cache is genuinely small: one million tokens occupy about 0.89 GB. That result comes from its Causal Encoder-Decoder, sparse attention, replay strategy, and FP4 main-KV design. It cannot be transferred to a conventional GQA model by changing one cache flag.
The official minimal inference documentation uses multi-process tensor parallelism and describes itself as a readable reference. It does not establish laptop SSD streaming as the supported production path. This model is useful as an upper boundary: active parameters predict only part of the machine requirement.
Techniques: what each one changes
The useful question for a technique is not whether it makes inference “faster.” It is which term in the model it changes.
| Technique | What it changes | What it costs or requires |
|---|---|---|
| Weight quantization | Reduces resident weight bytes and weight traffic | Quantization error, scales/metadata, and a compatible kernel |
| KV-cache quantization | Reduces cache capacity and attention traffic | Scales, conversion, and model/task-dependent quality impact |
| GQA / MQA | Stores fewer K/V heads than ordinary multi-head attention | Must be part of the model architecture or introduced through retraining/uptraining |
| MLA / compressed latent cache | Changes what attention stores and reconstructs | Architecture-specific kernels and reconstruction compute |
| Sparse or sliding-window attention | Reduces the number of past positions attended by some layers | It changes context semantics; discarded positions are not available to those layers |
| MoE | Reduces parameters computed and read per token relative to total capacity | Routing, load balance, expert placement, and all-to-all traffic at scale |
| Speculative decoding / MTP | Amortizes target-model passes over multiple accepted tokens | Draft work, verification compute, acceptance loss, and runtime support |
| Prefix caching | Reuses precomputed KV state for an identical prefix | Cache capacity, lookup, invalidation, and exact-prefix constraints |
| PagedAttention | Reduces KV allocation waste and supports larger dynamic batches | Page/block management; it does not shrink the represented cache by itself |
| CPU offload | Moves capacity pressure from VRAM to system RAM | CPU memory bandwidth and activation/weight traffic across PCIe |
| SSD streaming | Moves some capacity below RAM | SSD latency, IOPS, bandwidth, prefetch accuracy, and access locality |
| Tensor parallelism | Splits each layer’s compute and weights across devices | Per-layer collectives and interconnect latency/bandwidth |
| Pipeline parallelism | Places different layers on different devices | Pipeline bubbles; one autoregressive stream remains sequential across stages |
Quantization needs both a representation and a kernel
A four-bit file is useful immediately for capacity. Its speed depends on execution.
On Blackwell, a runtime may use a native FP4 path for a compatible layout such as NVFP4. On other GPUs, a fused mixed-precision kernel can read packed weights, apply scales in registers, and feed wider values to matrix hardware. Marlin demonstrates that this can retain weight-traffic speedups through moderate batch sizes on its tested NVIDIA GPUs.
On M5, Apple documents BF16 and INT4/INT8 Metal TensorOps using the Neural Accelerators. That is not an NVFP4 path. It does show why “quantization only saves memory on a Mac” is now too broad.
The checkpoint name is insufficient. Record the quantizer, effective bits per weight, group or block size, scale type, tensors left at higher precision, runtime version, and selected kernel.
Speculative decoding spends compute to save weight reads
A draft mechanism proposes more than one token. The target model verifies several candidates in parallel and accepts a prefix. Correct speculative sampling can preserve the target distribution, but the speedup is not free or fixed.
It improves latency when accepted tokens save expensive target-model passes and the machine has enough spare compute for verification. It loses value when acceptance is poor, the draft is expensive, verification becomes compute-bound, or concurrency already reuses the weights efficiently. Built-in MTP heads remove the need for a separate draft model but not the acceptance and verification terms.
A hardware balance point can explain the direction of the effect. It cannot produce a trustworthy “2× on this GPU” number without a measurement.
Offload has more than one traffic pattern
“CPU offload” may mean that CPU kernels execute whole layers whose weights stay in RAM, exchanging activations with the GPU. It may instead mean streaming weight blocks over PCIe for GPU execution. Those need different equations.
“SSD offload” also covers several cases. Reading a model once into RAM affects startup time. Loading inactive MoE experts on demand affects per-token latency and depends on expert locality. Persisting reusable KV blocks affects cache hits across requests. Sequential SSD bandwidth does not predict random expert lookup or small-page service.
An offload benchmark should report which bytes cross which boundary. Without that, it is impossible to transfer the result to another model.
Matching models to memory pools
The following table is a capacity screen, not a speed prediction.
| Available fast memory | Plausible starting point | Main constraint to check next |
|---|---|---|
| 24 GB | Qwen3-32B-class dense model at roughly four bits, with controlled context | Actual artifact size, KV-cache precision, and workspace headroom |
| 32 GB | Same dense class with more context, or smaller models at higher precision | Kernel support and memory bandwidth |
| 64 GB | 32B dense at higher precision or larger low-bit dense/MoE artifacts | Exact fit; 32.8B BF16 is already 65.6 GB before overhead |
| 80–96 GB | gpt-oss-120b’s documented MXFP4 single-GPU class; 70B dense low-bit artifacts | Optimized low-bit/MoE kernel and context capacity |
| 128–192 GB unified | Larger MoE artifacts and long contexts without PCIe weight movement | Lower LPDDR bandwidth, GPU compute, and runtime maturity |
| 512 GB unified | Very large quantized checkpoints may fit by raw arithmetic | Artifact availability, higher-precision exceptions, kernel support, and announced-hardware availability |
The phrase “plausible starting point” is deliberate. Use the downloaded artifact size, not the row, before spending money.
What to measure
A useful comparison holds the software and workload constant. For every result, record:
hardware SKU and memory configuration
OS, driver, and power mode
model repository and exact revision
artifact filenames and total bytes
quantizer, effective bits/weight, and cache dtype
runtime and commit
prompt tokens and attended context
output tokens
batch and concurrency
speculative or prefix-cache configurationThen report at least:
- load time;
- resident memory by tier;
- time to first token;
- inter-token latency or batch-one tokens/s;
- aggregate throughput at stated concurrency;
- wall power and whether the run sustained clocks;
- paired quality results if quantization changed.
This is enough to distinguish capacity, bandwidth, compute, and software problems. It also makes a result reproducible when a newer kernel changes the answer.
What I would test next
The hardware table suggests a small experiment matrix rather than one winning machine:
- Dense baseline: Qwen3-32B with the same four-bit artifact class and context across a 32 GB discrete GPU, a 96 GB Blackwell GPU, and a 128 GB unified-memory system.
- MoE and low-bit kernel: gpt-oss-120b on DGX Spark, an 80–96 GB discrete GPU, and a 128 GB unified-memory system. Record which implementation actually uses MXFP4 and which upcasts it.
- Long context: Qwen3-32B at increasing context with two-byte and quantized KV caches. Measure when attention and KV traffic become visible.
- Speculation: one model/runtime pair with and without its supported MTP or draft path, at batch one and at moderate concurrency.
- Offload: one dense and one MoE artifact that exceed VRAM. Separate CPU-computed layers from GPU weight streaming and record PCIe traffic.
The result would be a useful buying guide because the measurements would connect the specifications to real workloads. Until those measurements exist, the honest output is a map of constraints, not a ranked list.