Capacity and throughput bounds for local LLM inference

Running a large language model (LLM) on local hardware raises three separate questions: whether the selected checkpoint fits, how quickly a prompt can be processed, and how quickly new tokens can be generated. This article builds bounds for those questions. A bound can reject an impossible configuration or an impossible performance claim. It is not a benchmark.

Ground: the objects in an inference system

An autoregressive Transformer model maps a sequence of tokens to a probability distribution for the next token. A token is an integer produced by that model’s tokenizer, not necessarily a word. In ordinary causal self-attention, each position may attend to permitted earlier positions; the underlying scaled dot-product operation is defined in the Transformer paper.

The model architecture defines that computation: its layers, tensor shapes, attention arrangement, and vocabulary. A checkpoint is a particular set of learned tensors and configuration metadata for that architecture at a particular revision. The same model can have several checkpoint artifacts that differ in tensor representation without changing the architecture.

A number format maps a fixed bit pattern to a numeric value. A quantizer chooses approximations in a target representation, often together with scales or zero points. A tensor encoding lays those values and scale data out in bytes. A container packages typed tensors and metadata into files. These are different decisions: a file named “four-bit” need not contain only four-bit tensors, and compact storage does not establish which arithmetic an accelerator executes.

A runtime loads the checkpoint, allocates working state, and schedules operations. A kernel implements one such operation for particular tensor layouts and hardware. The kernel may read packed weights, unpack and scale them in registers or local memory, and then use a wider arithmetic instruction. Marlin’s FP16×INT4 kernel is a published example of that fused path. Hardware-native support is narrower: it means that the documented instruction itself accepts the low-precision operand representation.

The names commonly placed in one “format” list therefore occupy different layers. GGUF is a typed model container. MLX is an array framework, while MLX-LM is a model loading and generation toolkit. Triton is a language and compiler for parallel kernels. None of those names, by itself, specifies the stored bits, the executed instruction, or achieved throughput.

Numeric representations and block scaling

Floating-point (FP) formats divide their bits among sign, exponent, and fraction fields. IEEE 754 binary32 uses one sign bit, eight exponent bits, and 23 stored fraction bits; IEEE binary16, commonly called FP16, uses one, five, and ten. Bfloat16 (BF16) instead uses one sign bit, eight exponent bits, and seven fraction bits. BF16 preserves binary32’s exponent width while giving up fraction precision; FP16 allocates more of its 16 bits to the fraction and fewer to the exponent.

The Open Compute Project (OCP) eight-bit floating-point specification defines FP8 E4M3 as one sign, four exponent, and three mantissa bits, and E5M2 as one sign, five exponent, and two mantissa bits. E4M3 allocates one more bit to precision; E5M2 allocates it to range. The format name alone still does not say which tensors use it or how they are scaled.

At four bits, block scaling becomes part of the block-scaled representation. The OCP microscaling specification distinguishes scalar FP4 E2M1 values from the block-scaled MXFP4 vector format, which packs 32 values per block under one shared eight-bit E8M0 scale. For the MXFP4 vector format the local payload is therefore (32 × 4 + 8) / 32 = 4.25 bits/value before container and alignment overhead. NVIDIA’s NVFP4 description uses groups of 16 E2M1 values with an eight-bit block scale plus a tensor-level scale for its one-dimensional (1D) case; its local payload there is (16 × 4 + 8) / 16 = 4.5 bits/value before the tensor-level scale and other layout overhead. NVFP4 also supports 16×16 two-dimensional (2D) blocks, and uses 2D scaling for weights by default.

Those two calculations are derivations from the specifications, not claims about a GGUF file. In llama.cpp, Q4_K_M is a quantization preset that can assign different encodings to different tensors; the quantizer documentation explicitly exposes mixed-tensor controls. Q4_K_M, MXFP4, and NVFP4 must not be treated as synonyms. The quality change from any quantization is an empirical property of the exact model, quantizer, calibration data, and evaluation; bit count alone cannot establish it.

Workload phases and memory tiers

Prefill processes the input tokens and creates attention state for them. Its linear layers can reuse a weight tile across many token rows. Decode then generates one token at a time; at batch size one, a linear-layer weight used for one token has much less reuse. These statements describe data reuse, not fixed bottleneck labels. The operation is compute-bound or bandwidth-bound only after its work and data movement are compared with the same machine’s relevant ceilings.

The retained attention state is the key/value (KV) cache. Grouped-query attention (GQA) shares one key head and one value head across a group of query heads, which reduces the number of KV heads and the cache data loaded during decoding, as defined in the GQA paper. A runtime may allocate that logical cache in blocks; PagedAttention changes physical allocation and fragmentation, not the logical number of retained key and value elements.

Memory is also not one interchangeable capacity. The relevant memory tier is where bytes reside when an operation needs them: accelerator-attached memory, unified system memory, host random-access memory (RAM) across an interconnect, or solid-state-drive (SSD) storage. Capacity answers whether the state can be resident. Bandwidth and latency answer how quickly a required byte can arrive. If some weights are fetched across a slower tier for every token, that traffic belongs in the per-token bound even if faster memory has unused bandwidth.

Derive: capacity, traffic, and time bounds

Capacity starts with the artifact

Let W be the exact checkpoint artifact size in bytes, K the allocated KV cache in bytes, and A every other live allocation: runtime state, temporary workspaces, allocator slack, and the operating system’s share. A configuration fits a memory pool of capacity M only if

W + K + A ≤ M                                      [bytes]

This is a capacity derivation. It does not assume that a label such as “32B Q4” predicts the artifact size. If a file represents a model advertised with P_total parameters, its complete-file effective density is

file bits per advertised parameter = 8W / P_total [bits/parameter]

That quotient includes metadata, scale data, alignment, and any tensors stored at other widths. It describes the artifact as a capacity input; it does not identify its tensor encoding.

KV cache size follows the model configuration

For a cache that retains an ordinary key and value vector for each token and layer, define L as layer count, H_KV as key/value head count, d as values per head, e as bytes per stored value, N as retained tokens per sequence, and B_batch as simultaneous sequences. The logical cache size is

KV bytes/token/sequence = 2 × L × H_KV × d × e          [bytes/token]
K = 2 × L × H_KV × d × e × N × B_batch                 [bytes]

The leading two represents one key and one value. This formula is a derivation under the stated cache layout. Multi-query attention, latent caches, KV quantization, recurrent state, and bounded attention windows require different inputs or formulas.

With ordinary full causal attention, a new query at retained length N uses the preceding keys and values, so logical KV traffic and attention work grow with context. The separate attention-method issue owns exactly the taxonomy of techniques that change the quadratic token-pair term, reduce only KV storage or traffic, or move work into lookup or retrieval. This article assumes the ordinary full-attention path only to size its cache and traffic.

Roofline separates prefill from decode

For one operation or inference step, let F be useful floating-point operations (FLOPs), Q bytes transferred from the limiting memory tier, C sustained compute throughput in FLOP/s, and BW sustained bandwidth in bytes/s. The Roofline model gives the bound

time ≥ max(F / C, Q / BW)                               [seconds]
throughput ≤ min(C / F, BW / Q)                          [steps/second]
arithmetic intensity I = F / Q                           [FLOP/byte]
machine balance R = C / BW                               [FLOP/byte]

I < R identifies the bandwidth side of the bound; I > R identifies the compute side. C and BW must describe the same machine, data path, tensor shapes, precision, and operating condition. A vendor peak is a theoretical input, not a sustained measurement.

This article counts one multiply and one add as two operations. If P_linear weights participate in dense linear operations for one token, their first-order work is approximately

F_linear/token ≈ 2 × P_linear                            [FLOP/token]

For batch-one decode, if those weights occupy W_linear bytes and are read once, their arithmetic intensity is approximately 2P_linear / W_linear. If their effective storage width is b bits per weight with no other traffic, that becomes 16 / b FLOP/byte. The familiar values—about 1 for 16-bit, 2 for 8-bit, and 4 for ideal four-bit storage—are therefore derived weight-only intensities, not end-to-end measurements.

During prefill of S token rows, one read of a reusable weight tile can support roughly S times the linear work. The weight-only intensity can rise toward 2SP_linear / W_linear. Activation movement, cache writes, attention, short matrices, and kernel behavior change the realized value. Therefore “prefill is compute-bound” and “decode is bandwidth-bound” are hypotheses to test with Roofline inputs, not laws.

For full-attention, batch-one decode, a useful traffic estimate is

Q_decode ≈ W_read + K_read                               [bytes/token]
decode rate ≤ BW / Q_decode                              [tokens/second]

The estimate is optimistic when it omits other tensors and transfers. At long context it is also incomplete if attention compute, rather than KV traffic, sets the tighter side of the Roofline bound.

Multiple memory tiers produce multiple bounds

If an operation must move Q_i bytes through tier i with sustained bandwidth BW_i, a non-overlapped path has

time ≥ Σ(Q_i / BW_i)                                    [seconds]

Perfect overlap can reduce the transfer part no further than max(Q_i / BW_i). Actual scheduling lies between those cases. This is why adding RAM, accelerator memory, and SSD capacity says nothing by itself about token rate: the placement determines which Q_i / BW_i terms apply.

Apply: Qwen3-32B Q4_K_M on a 48 GiB M4 Pro

The worksheet below applies the model to one immutable public artifact and one measured machine. Decimal gigabytes (GB) use 10^9 bytes; binary gibibytes (GiB) use 2^30 bytes.

Evidence and selected configuration

Facts. The base checkpoint is Qwen/Qwen3-32B revision 9216db5781bf21249d130ec9da846c4624c16137. Its model card reports 32.8 billion total parameters and 32,768 tokens of native context. Its pinned configuration records 64 layers, 8 KV heads, head dimension 128, no sliding window, and caching enabled.

The selected file is Qwen3-32B-Q4_K_M.gguf at repository revision 938a7432affaec9157f883a87164e2646ae17555. The pinned repository API reports W = 19,762,149,024 bytes and SHA-256 efd971561896866f0e910cce52761ca77b1b138090c7f15fe284676d57d1f689.

The runtime is llama.cpp v0.4.1 at commit 391fac16460f15233a7740550d858ac96df3419d. Its pinned architecture map names Qwen3, the Qwen GGUF card supplies a llama.cpp invocation, and llama.cpp’s build documentation says Metal is enabled by default on macOS. Those are compatibility facts; they do not establish optimized sustained throughput for this exact tensor mix.

The machine is a 14-inch 2024 MacBook Pro with an M4 Pro and 48 GiB of installed unified memory. Local measurement on 2026-09-15: system_profiler identified Mac16,8, M4 Pro, and 48 GB; sysctl -n hw.memsize returned 51,539,607,552 bytes, exactly 48 GiB. Apple’s technical specification lists the 48 GB M4 Pro configuration and advertises 273 GB/s memory bandwidth.

The selected cache setting is one sequence, 32,768 retained tokens, and two-byte FP16 keys and values. llama.cpp exposes those choices as --cache-type-k and --cache-type-v. A reproducible launch pins the artifact and runtime revisions, uses -ctk f16 -ctv f16, sets -c 32768, and offloads all possible layers to Metal with -ngl 99; -fa on selects the runtime’s flash-attention path. The model, file hash, runtime commit, context, batch, cache types, and flags are therefore explicit rather than inherited from defaults.

Capacity result

Derivation. The complete-file density is

8 × 19,762,149,024 bytes / 32.8×10^9 parameters
= 4.820 bits per advertised parameter

The file itself occupies 19,762,149,024 / 2^30 = 18.405 GiB. This is why the actual artifact, not 32.8B × 4 bits, is the capacity input.

The cache occupies

2 × 64 layers × 8 KV heads × 128 values/head × 2 bytes/value
= 262,144 bytes/token = 256 KiB/token

262,144 bytes/token × 32,768 tokens × 1 sequence
= 8,589,934,592 bytes = 8 GiB

Weights plus the fully allocated logical cache require

19,762,149,024 + 8,589,934,592
= 28,352,083,616 bytes
= 26.405 GiB

The measured 48 GiB pool then leaves

51,539,607,552 - 28,352,083,616
= 23,187,523,936 bytes
= 21.595 GiB

for the operating system, runtime, workspaces, and allocator slack. Inference: this capacity model accepts the Q4_K_M artifact at a 32,768-token, batch-one FP16 cache on this machine. The conclusion is conditional on all omitted allocations fitting inside the remaining 21.595 GiB; it is not a process-memory measurement.

Decode and prefill bounds

Derivation from a vendor specification. If every generated token reads the entire 19.762 GB file payload from the memory tier and the workload somehow sustains Apple’s advertised 273 GB/s, the short-context weight-only ceiling is

273×10^9 bytes/s / 19,762,149,024 bytes/token
= 13.81 tokens/s

At 32,768 retained tokens, adding one logical read of the 8 GiB KV cache gives

273×10^9 bytes/s / 28,352,083,616 bytes/token
= 9.63 tokens/s

These correspond to optimistic transfer times of 72.39 ms/token and 103.85 ms/token. Inference: any observed batch-one rate above these values would falsify at least one traffic assumption—for example, not all file bytes were read per token, fewer than 32,768 tokens were retained, cache bytes were reused above the counted tier, or the advertised bandwidth did not describe the modeled path. Rates below the ceilings do not identify a cause.

No corresponding prefill ceiling is reported. Apple publishes the bandwidth used above but the cited machine specification does not publish a sustained FP16 or quantized llama.cpp compute rate for the selected shapes. Substituting an unrelated peak would violate the Roofline requirement that compute and bandwidth inputs describe the same execution path. The prefill result here is therefore limited to the derived direction: weight reuse raises arithmetic intensity with prompt rows, while the actual limiting side remains unmeasured.

Limitations of the worked result

The file API does not expose a tensor-by-tensor encoding inventory, so file bytes are a conservative proxy for bytes read per token. The 273 GB/s input is an Apple peak specification, not sustained application bandwidth. The capacity calculation excludes measured process overhead. The decode calculation omits activation traffic, output projection details, sampling, kernel launches, synchronization, thermal state, and attention arithmetic. It assumes ordinary full attention and one complete logical cache read at maximum retained context.

A measurement capable of replacing the ceilings must record the same artifact hash and runtime commit, whether all layers stayed on Metal, prompt and generated lengths, batch and concurrency, FP16 cache types, flash-attention setting, peak resident memory, prompt-processing rate, and generation rate. Without that record, a tokens-per-second number would not test this worksheet.

Resulting decision

Use the pinned Q4_K_M artifact for this 48 GiB M4 Pro configuration at batch one and at most 32,768 retained tokens: the explicit capacity calculation leaves 21.595 GiB outside weights and KV cache. Treat 13.81 tokens/s at short context and 9.63 tokens/s at full retained context only as optimistic vendor-bandwidth ceilings, not expected or observed performance. The worksheet supports a fit decision and a falsifiable decode bound; it does not support a prefill-rate claim, a quantization-quality claim, or a claim that Metal executes the GGUF encoding as native four-bit arithmetic.