What limits local LLM inference

Suppose you want to run a model on hardware you own. Three questions come first:

Does the model fit?
How long will it take to process the prompt?
How quickly will it generate tokens afterward?

Those are different questions. A single number from a GPU specification cannot answer all three.

This article develops a small performance model for answering them. It is not a substitute for a benchmark. It is a way to reject impossible configurations, understand benchmark results, and know which specification matters before buying hardware.

Terms and symbols

This article uses a few abbreviations repeatedly:

Term	Meaning
LLM	Large language model. The examples here are autoregressive Transformer models based on the architecture introduced in Attention Is All You Need.
Token	One model input or output unit. A token is not necessarily a word; the tokenizer defines the mapping.
`P`	Parameter count participating in the operation being estimated. For a dense model this is close to the model’s linear-layer parameter count; for a mixture-of-experts model it may be the active subset.
`b`	Stored bits per weight before converting to bytes. A nominal four-bit scheme has `b = 4`, but block scales and higher-precision tensors raise its effective bits per weight.
`B`	Batch size: the number of token rows processed together.
`S`	Prompt length in tokens during prefill.
`T`	Prior tokens actually attended during a decode step. This may be smaller than the allocated context for sliding-window or sparse attention.
FLOP	One floating-point operation. This article follows the usual hardware convention of counting a fused multiply-add as two FLOPs.
KV cache	Cached attention key and value vectors from prior tokens. PagedAttention is one method for managing this cache without large contiguous allocations.
VRAM	Memory attached to a discrete GPU. HBM and GDDR are high-bandwidth memory families used by accelerators and graphics cards; LPDDR is commonly used for unified system memory.
GB / GiB	Decimal gigabytes use `10^9` bytes. Binary gibibytes use `2^30` bytes. The distinction is visible when caches reach tens of gigabytes.

Later sections introduce grouped-query attention (GQA), mixture-of-experts (MoE), and PCI Express (PCIe), with the term expanded on first use.

Start with the two phases

A decoder-only language model does two visibly different kinds of work.

Prefill processes the prompt. For a prompt of many tokens, the large linear layers run as matrix-matrix multiplications. The same weights contribute to many prompt tokens, so the computation can reuse weight data and attain high arithmetic intensity. Long-prompt prefill is therefore often limited by compute throughput.

Decode generates tokens autoregressively. At batch size one, each step processes one new token. The large linear layers look more like matrix-vector multiplications. They do little work for every byte of weights read, so token generation is usually limited by memory bandwidth.

This is a useful default, not a law. Short prompts, small matrices, poor kernels, long-context attention, large batches, and speculative decoding can move the bottleneck. The roofline model gives the more precise test:

arithmetic intensity = useful FLOPs / bytes moved
machine balance       = peak FLOPs / peak memory bandwidth

If a kernel’s arithmetic intensity is below the machine balance, memory bandwidth is the upper bound. If it is above the machine balance, compute is the upper bound. Real performance can be lower than either bound.

The `2P` approximation

For a dense model with P parameters, one forward pass requires roughly:

FLOPs per token ≈ 2P

The reason is simple: a weight used in a linear layer participates in a multiply and an add. Counting a fused multiply-add as two floating-point operations gives about two operations per weight. Embeddings, normalization, activation functions, routing, and attention add work, so 2P is an approximation rather than an accounting identity.

For batch-one decode, consider only weight traffic for a moment. If weights occupy b bits each and a fused kernel reads each stored weight once, the weight-side arithmetic intensity is approximately:

Iweight ≈ 2 FLOPs / (b / 8 bytes) = 16 / b FLOPs per byte

That gives about 1 FLOP/byte for FP16 or BF16 weights, 2 for 8-bit weights, and 4 for 4-bit weights. Lower-bit weights increase arithmetic intensity because the same conceptual multiply requires fewer bytes from memory. This is one reason quantization can accelerate decode.

Batching increases reuse. With a decode batch of B, the same weight tile can serve roughly B input vectors, so weight-side arithmetic intensity initially grows with B. Eventually compute, activation traffic, KV-cache traffic, or kernel details take over. Single-user latency and multi-user throughput are therefore different workloads.

First question: does it fit?

The minimum payload for P parameters stored at b bits each is:

weight bytes = P × b / 8

For a 32.8-billion-parameter model, the ideal payload is 65.6 GB at 16 bits, 32.8 GB at 8 bits, and 16.4 GB at 4 bits. A real quantized file is usually larger than the 4-bit minimum because it also contains block scales, metadata, alignment, and some tensors kept at higher precision.

The file size is a better capacity input than the model’s marketing name. It still is not the whole memory requirement. Add:

the KV cache;
temporary workspaces used by kernels;
model metadata and runtime allocations;
allocator fragmentation;
any draft model, adapters, or vision encoder loaded alongside the model.

There is no universal “add 1.5 GB” or “add 5 percent” correction. Runtime, model, context length, batch size, and backend all matter. Leave headroom and measure the process after loading the exact model you intend to use.

The KV cache

Standard autoregressive attention stores a key and a value for every cached token at every layer. For one sequence, a common layout has this size per token:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per element

The leading 2 accounts for keys and values.

Take Qwen3-32B as a worked example. Its published configuration has 64 layers, 8 KV heads, a head dimension of 128, and BF16 weights. If its KV cache uses two-byte elements:

2 × 64 × 8 × 128 × 2 = 262,144 bytes = 256 KiB per token

That is 8 GiB for 32,768 cached tokens and 32 GiB for 131,072 tokens. Context includes prompt and generated tokens, and batch size multiplies the total.

Which techniques change the quadratic attention term?

For a sequence of n tokens, ordinary full self-attention forms an n × n score matrix: every query is compared with every key. The attention term is therefore quadratic in sequence length. With a KV cache, one decode step at position T compares one new query with T cached keys, so that step is linear in T; generating a long sequence still accumulates quadratic work.

Several techniques are often grouped together even though they change different costs:

Technique	What it changes	Sequence-length effect	Paper and model example
KV cache	Reuses K and V from prior tokens instead of recomputing them	Full-attention decode remains `O(T)` per new token	Standard in autoregressive Transformer inference; PagedAttention improves cache allocation
MQA / GQA	Shares fewer KV heads across more query heads, reducing cache size and KV traffic	Does not remove the quadratic full-attention term	MQA, GQA, and Qwen3-32B
FlashAttention	Tiles exact attention to reduce reads and writes to high-bandwidth memory	Arithmetic remains `O(n²)`	FlashAttention and FlashAttention-2
Sliding-window or local attention	Restricts each token to a window of `w` nearby tokens	Reduces attention to `O(nw)`	Longformer and Mistral 7B
Fixed block-sparse or local-plus-global patterns	Computes only selected blocks or gives a small set of tokens global reach	Depends on the pattern; commonly near-linear for fixed window/global counts	BigBird
Learned top-k sparse attention	Uses an indexer or router to select a small set of keys for each query	Selected attention can be `O(nk)` for fixed `k`, plus index-selection cost	Native Sparse Attention and DeepSeek-V4.1-Flash
MLA or another compressed KV representation	Stores lower-dimensional latent state instead of ordinary K and V heads	Reduces cache capacity and traffic; full attention can still be quadratic	DeepSeek-V2
Linear/recurrent replacements	Replaces softmax attention with a recurrent or kernelized state update	Can make sequence processing linear in `n`, but changes the operator	RetNet and the attention-free Mamba state-space model

The useful distinction is between reducing constants and changing the sequence-length term. GQA and MLA primarily reduce KV storage and traffic. FlashAttention reduces data movement. Sliding-window, sparse, and recurrent approaches are the techniques that actually avoid computing every token-to-token pair.

This taxonomy deserves a separate article because each method changes context semantics and model quality differently. The rest of this article uses the ordinary GQA cache formula as a capacity example, not as a claim that GQA solves quadratic attention.

This calculation applies to the stated layout. Multi-head attention, grouped-query attention, multi-query attention, latent attention, sliding windows, sparse attention, and KV quantization change it. Paged allocation changes how efficiently capacity is used, but it does not change the bytes represented by an already allocated cache. Read the model configuration and the runtime’s cache settings rather than assuming every model behaves like the example.

During standard full-attention decode, the attention kernels read the prior keys and values. As context grows, KV traffic and attention computation grow even though the model weights stay fixed. The simple “bandwidth divided by weight size” estimate becomes optimistic at long context.

A bandwidth ceiling for decode

If a dense model is fully resident in one memory pool and batch-one decode is bandwidth-bound, a useful upper bound is:

decode tokens/s ≤ memory bandwidth / bytes read per generated token

A first estimate for the denominator is the resident weight payload plus the KV data read by full attention. It is a ceiling because it assumes perfect sequential use of peak bandwidth and ignores every other cost.

For example, suppose an actual quantized model occupies 19 GB and the machine sustains 500 GB/s on the inference workload. Ignoring KV traffic gives a ceiling of about 26 tokens/s:

500 GB/s / 19 GB ≈ 26 tokens/s

If full attention also reads an 8 GiB KV cache, the combined traffic ceiling is closer to 18 tokens/s. At longer context, attention computation may become a separate limit. Neither number predicts observed speed exactly; both explain why an observed result cannot exceed the available data path indefinitely.

Do not multiply every vendor bandwidth number by a fixed “60 percent efficiency” factor. Achieved bandwidth varies with tensor shapes, quantization layout, kernel implementation, cache behavior, and runtime. Use the formula as a bound, then replace assumptions with a benchmark of the exact model, quantization, context, batch, and runtime.

Number formats are only one layer

The names are easier to understand if we separate ordinary floating-point formats from scaled low-precision formats.

Format	Value bits	Basic layout	Typical role
FP32	32	1 sign, 8 exponent, 23 fraction	higher-precision accumulation, optimizer state, reference computation
FP16	16	1 sign, 5 exponent, 10 fraction	weights, activations, and compute where its range is sufficient
BF16	16	1 sign, 8 exponent, 7 fraction	weights, activations, and training compute with FP32-like exponent range
FP8 E4M3	8	1 sign, 4 exponent, 3 fraction	lower-precision weights and activations, usually with scaling
FP8 E5M2	8	1 sign, 5 exponent, 2 fraction	lower-precision values where more exponent range is needed
FP4 E2M1	4	1 sign, 2 exponent, 1 fraction	element format inside a block-scaled representation
INT8 / INT4	8 / 4	signed or unsigned integer	quantized values interpreted with scales and sometimes zero points

FP16 and BF16 use the same number of bits for different purposes. FP16 has more fraction precision; BF16 keeps FP32’s eight exponent bits and therefore much more range. Neither is simply “more precise” without saying whether range or spacing matters.

The OCP FP8 specification defines E4M3 and E5M2 variants. The split expresses the same tradeoff: E4M3 spends one more bit on the fraction, while E5M2 spends it on exponent range.

Four bits are too coarse for most model tensors without additional structure. Modern FP4 schemes divide a tensor into small blocks and store a scale for each block. The scale moves a block’s values into the narrow range that E2M1 can represent.

Two names that look similar use different scale formats:

MXFP4, standardized by the OCP Microscaling Formats specification, uses 32 E2M1 values with one E8M0 scale. The payload is 4 bits per value plus 8 scale bits per 32 values: 4.25 bits per value before container and alignment overhead.
NVFP4, documented by NVIDIA Transformer Engine, uses E2M1 values, an E4M3 scale for each block of 16 values, and a higher-level FP32 scale. The block payload is 4.5 bits per value before the per-tensor scale and other overhead.

That extra scale precision and finer block size are part of NVFP4’s accuracy story. They are also why “4-bit” does not mean exactly half the bytes of an 8-bit tensor.

A model described as FP8 or FP4 is rarely FP8 or FP4 everywhere. Weights, activations, KV cache, scales, outputs, and accumulators can use different formats. Training often keeps optimizer state or accumulation at higher precision. Inference checkpoints often retain sensitive tensors at higher precision. Ask which tensor uses the format, how it is scaled, and what precision the matrix operation accumulates into.

Quantization quality is empirical. It depends on the model, tensor, task, calibration data, block size, and quantizer. Rules such as “4-bit is always the sweet spot” are useful folklore, not portable guarantees.

“Native support” has several meanings

A machine can “support” a format at four different levels:

It can store the bytes.
A runtime can read the file and understand its metadata.
A kernel can unpack and scale the values while performing a matrix operation.
A matrix instruction can consume the low-precision operands directly.

Only the fourth is hardware-native arithmetic. The third can still be fast: a fused kernel reads compact weights, applies scales in registers, and feeds higher-precision values to matrix hardware without materializing a full converted copy in memory.

This distinction corrects a common but misleading claim: a weight-only 4-bit model is not necessarily “computed at FP16 speed with extra overhead.” A good mixed-precision kernel can save memory traffic and overlap unpacking with computation. Marlin, for example, implements FP16×INT4 matrix multiplication and reports near-ideal weight-traffic speedups through moderate batch sizes on the NVIDIA GPUs it evaluates. Results do not automatically transfer to another quantization layout, GPU, or runtime.

There are real native low-precision paths. NVIDIA Hopper introduced FP8 Transformer Engine support; Blackwell adds FP4-capable Tensor Cores and NVIDIA’s NVFP4 path. NVIDIA’s Transformer Engine documentation describes the supported recipes and hardware requirements. AMD’s CDNA matrix-core documentation documents its own FP8 variants and the FP4/FP6 additions in CDNA4.

Apple’s M5 TensorOps documentation describes Neural Accelerators in each GPU core and Metal operations for BF16 plus 4-bit and 8-bit integer tensors. That is not FP4 floating point or NVFP4. It does mean that the blanket claim “quantization on a Mac can never accelerate compute” is false for M5. For any Apple generation, the useful question is still which MLX or Metal kernel the runtime selects.

A silicon feature is not enough by itself. The runtime must select a kernel that accepts the checkpoint’s layout for the relevant shapes. Otherwise it may convert the weights, choose a slower fallback, or fail to load the model. Peak FP4, FP8, and FP16 numbers on vendor tables are also not directly comparable unless they use the same rules for sparsity, multiply-add counting, operand formats, and accumulation.

File formats, quantizers, runtimes, and kernels

Several names commonly presented as competing “model formats” belong to different layers.

Name	What it is
GGUF	A container for model tensors and metadata used by GGML-based runtimes. A GGUF can contain F32, F16, BF16, or one of many quantized tensor encodings.
Safetensors	A simple container consisting of a metadata header and tensor byte buffer. It records each tensor’s dtype, shape, and offsets.
GPTQ / AWQ	Methods and associated layouts for choosing quantized weights, usually for weight-only inference.
MLX	Apple’s array framework and compute stack, not a checkpoint container. MLX-LM model directories normally store tensors in Safetensors files with MLX metadata.
llama.cpp	An inference runtime whose native model container is GGUF.
vLLM	An inference and serving runtime. Its PagedAttention design manages KV-cache blocks efficiently; it is not a weight format.
Triton	A language and compiler for writing GPU kernels. It is neither a checkpoint format nor a model runtime.

The GGUF specification describes a single-file, extensible, memory-mappable container with model metadata and typed tensors. Seeing Q4_K_M in a filename describes tensor encoding choices inside that container; GGUF alone does not mean four-bit.

The Safetensors specification defines an eight-byte header length, a JSON tensor index, and a contiguous byte buffer. It is intended to load tensors without executable deserialization. It does not prescribe how an inference kernel should compute with those tensors.

MLX illustrates why the distinction matters. MLX-LM is a runtime and model toolkit for Apple silicon. Its conversion code writes model.safetensors files and records format: mlx in metadata. Quantization information lives with the model configuration and tensors; “MLX format” is convenient shorthand for a model prepared for that stack, not a separate peer to GGUF at the file-container layer.

Triton operates farther down the stack. A Triton program describes blocked GPU work; the compiler handles details such as coalescing, shared memory, vectorization, and tensor-core-aware instruction selection. A runtime can use a Triton kernel to load packed weights, apply scales, and execute a matrix multiplication. Triton does not decide what the checkpoint means.

The complete path is:

checkpoint container
    → tensor encoding and quantization metadata
    → runtime chooses a compatible kernel
    → kernel moves packed bytes and applies scales
    → hardware executes instructions and accumulates results

A mismatch anywhere in that path can erase a theoretical hardware advantage.

Memory has levels

“Does it fit in memory?” is incomplete unless it names the memory.

A discrete GPU has high-bandwidth VRAM connected to system RAM over PCIe. If a dense model spills layers to RAM, those layer weights must cross the slower link during inference. A simplified non-overlapped lower bound is:

time per token ≥ bytes read from VRAM / VRAM bandwidth
               + bytes transferred over PCIe / PCIe bandwidth

Real runtimes can overlap some transfer and computation, but they cannot remove the slower link. A model that technically loads with CPU offload can therefore generate much more slowly than one that fits entirely in VRAM.

Unified-memory systems expose one physical pool to CPU and GPU. They avoid a separate PCIe copy for resident data and can make much larger models practical. Capacity is shared with the operating system and applications, and the GPU still has a finite memory bandwidth. “Unified” removes one boundary; it does not create unlimited bandwidth.

SSD is another level. If the model fits in RAM or VRAM, SSD speed mostly affects loading. If inference streams weights from SSD on every token, storage bandwidth and access pattern become part of the steady-state limit. Sparse mixture-of-experts models can sometimes offload inactive experts more gracefully than dense models because each token selects only part of the expert weights, but routing, prefetch accuracy, batching, and cache locality determine whether that works. SSD streaming is a capacity technique with workload-specific performance, not free memory.

For a mixture-of-experts model, distinguish two counts:

total parameters determine checkpoint size and the capacity needed if all experts are resident;
active parameters approximate the expert computation and weight traffic for one token.

The active count is not a complete speed prediction. Attention and shared layers still run, and a batch routed across many experts can touch far more of the checkpoint than one token does.

A practical worksheet

For a model and machine, work through these steps in order.

Choose the exact artifact. Record the model revision, file size, quantization scheme, and runtime. A parameter count and “4-bit” label are not enough.
Calculate KV size. Read layer count, KV heads, head dimension, cache dtype, context, and batch from the model and runtime configuration.
Check the memory level. Decide what resides in VRAM, unified memory, system RAM, or SSD. Do not add their capacities as if their bandwidth were equal.
Estimate batch-one decode. Divide available memory bandwidth by the bytes that must be read per step. Treat the result as a ceiling.
Estimate prefill separately. Use compute throughput for the precision and kernel that will actually run. Do not substitute a sparse FP4 marketing peak for dense BF16 performance.
Check software support. Confirm that the runtime has an optimized kernel for the model architecture, tensor layout, quantization, and hardware.
Benchmark the intended workload. Hold model revision, quantization, runtime, prompt length, generated length, batch, cache dtype, and sampling settings constant. Report time to first token and inter-token latency separately.

This order prevents a common mistake: comparing FLOPS before checking whether the model fits, then comparing memory bandwidth before checking whether the runtime can use the advertised low-precision path.

What the simple model leaves out

The formulas here deliberately omit several effects:

tokenization and sampling overhead;
embeddings and output projection, which can matter with a large vocabulary;
attention computation at long context;
kernel-launch and synchronization overhead;
tensor and pipeline parallel communication;
prefix caching and paged KV allocation;
speculative decoding, which changes target-model passes per accepted token;
thermal and power limits;
runtime-specific graph capture, fusion, and scheduling.

These are reasons to benchmark, not reasons to skip the model. A simple bound tells you which measurements need explanation.

The short version

Ask whether the exact model artifact fits before looking at peak compute.

For long-prompt prefill and large batches, compute throughput and kernel quality usually matter most. For batch-one generation, start with bytes moved per token and memory bandwidth. Add KV-cache cost as context grows.

Keep number format, quantization method, file container, runtime, kernel, and hardware instruction separate. FP8, NVFP4, GGUF, MLX, and Triton do not name interchangeable things.

Then measure the workload you actually care about. The formulas provide the ceiling; the distance from that ceiling tells you what to investigate.