Local Inference Hardware: Reading the Cycle Without the Hype

Every few months a launch event implies that local inference just became the obvious choice, and every few months the people actually serving models from a desk quietly swap a part or re-rent a cloud instance. Unified-memory desktops, consumer GPUs and on-die NPUs are all being reworked at the same time, so the category feels volatile. It is not volatility you should trade on. It is constraints: capacity, bandwidth, software support and power.

Three shapes, three different bottlenecks

A unified-memory machine shares one physical pool between CPU and accelerator. That buys capacity — models that would never fit inside a discrete card’s VRAM — at the cost of bandwidth, because LPDDR5X sits in the low hundreds of GB/s while GDDR6X cards in a similar price class push toward 1 TB/s, and datacenter HBM parts sit far above both. Big models fit; token generation at batch size one tends to lag a discrete card holding the same weights.

A consumer GPU is the opposite trade. High bandwidth, mature CUDA tooling, real thermals and a real power bill, and a hard VRAM ceiling that decides which quantizations and context lengths are even on the table. Two cards do not add up the way people assume unless the model is sharded and the interconnect cooperates.

An NPU is a third thing again: low power, already present in laptops and mini-PCs, and almost entirely dependent on the vendor’s runtime and compiler. When it works, it works quietly. When it does not, you are reading release notes for a kernel you cannot patch.

That last row is not a cop-out. It is a control.

The four numbers that actually decide the purchase

Capacity decides what fits. Weights are the easy part; the KV cache grows with context length and concurrency, and that is the number people forget when they size a box. If you have felt this squeeze before, the failure modes are the same ones covered in Agent memory in production.

def kv_cache_gb(layers, kv_heads, head_dim, tokens, batch=1, dtype_bytes=2):
    """KV cache footprint for one batch. Arithmetic, not a benchmark."""
    per_token = 2 * layers * kv_heads * head_dim * dtype_bytes  # K and V
    return per_token * tokens * batch / (1024 ** 3)

# 32 layers, grouped-query attention with 8 KV heads of dim 128,
# 32k context, 4 concurrent requests, fp16 cache:
print(kv_cache_gb(32, 8, 128, 32768, batch=4))

Run the same function with the multi-head numbers some models actually use and the answer moves by an order of magnitude. That is the point: size the machine with your model’s config, not a rounded-up guess.

Bandwidth decides how fast a single stream generates tokens, and it is the spec most loudly marketed and least useful on its own. Once you batch requests, compute and scheduling start to dominate, and the unified-memory box that looked slow at batch one can look reasonable at batch eight.

Software support decides whether you spend the weekend on kernels or on your product. Check three things before buying anything: does the runtime you use have a maintained backend for this accelerator, does it support the quantization format you actually load, and is that backend owned by the vendor or by one person with a day job. The third question is the one that ages badly, and restart discipline becomes its own problem when you are serving — the mechanics are close to what Swapping models on a single-GPU box without downtime drama describes.

Power and thermals decide whether the machine can live where you work. A 600 W card is a space heater with a fan curve. Idle draw matters if the box runs around the clock.

Inventory what you already have before you shop:

lscpu | grep -E 'Model name|Core|MHz'
free -h
nvidia-smi --query-gpu=name,memory.total,memory.used,power.draw,power.limit --format=csv
nvidia-smi topo -m                     # interconnect layout, matters for multi-GPU
ls /dev/dri /dev/accel* 2>/dev/null    # accelerator nodes exposed by the kernel

What is genuinely unknown

Several things that should affect a purchase decision are not settled, and no launch post will settle them:

• Whether NPU compilers will support the quant formats you already use, or only the ones shown in the vendor demo.
• How long unified-memory platforms keep receiving driver and runtime updates after the next generation ships.
• What memory prices and availability do over the next two years, which sets the resale floor.
• Which serving stack becomes the default for these boxes: a vendor runtime, vLLM, or llama.cpp — each with a different support surface.
• Whether your 2027 workload resembles your 2026 workload. Long-context agents and local embedding pipelines stress hardware differently than chat does.

Write the unknowns down instead of arguing about them. A file beats an opinion:

# hardware_risk_register.yaml
decision: buy-or-rent-2026
unknowns:
  - id: npu-toolchain
    question: Does the vendor compiler ship kernels for my quant format?
    resolve_by: Run my model, not the demo, inside the return window.
    fallback: Keep the discrete GPU in the box.
  - id: driver-window
    question: How long does this platform get driver updates?
    resolve_by: Read the vendor lifecycle document, not the launch post.
    fallback: Budget a replacement at 3 years, not 5.
  - id: sizing
    question: Does my peak concurrency fit, or only my average?
    resolve_by: Sample real traffic for two weeks before ordering.
    fallback: Rent the peak, own the baseline.

The rule for when renting still wins

Renting wins whenever your configuration is not yet stable. Concretely, if any of these is true this quarter, rent:

• Your model mix changes more often than once a quarter, so the memory and bandwidth targets move under you.
• Peak demand is a large multiple of typical demand, and the peak lasts hours rather than days.
• You need a specific accelerator to answer a question (“does this quant fit?”, “how does this runtime behave?”) rather than to run a known workload.
• Nobody on the team owns driver updates, thermals and disk failure. An unattended box is a liability, and the ops time is the expensive part, not the electricity. If this is the sticking point, the habits in Triage habits for automation that fails while you sleep are the relevant ones, not the GPU specs.

The rule worth writing on the wall: rent until one fixed configuration survives two months of your real traffic without you changing it. When a single shape — capacity, bandwidth, runtime — holds steady that long, buying converts a monthly line item into a depreciating asset plus a small ops job, and that trade starts to pay. Before that, you are buying a moving target.

What to do next

1. Measure your actual workload for one week: longest context you really send, peak concurrent requests, and what a cold start costs you. Most hardware regret comes from sizing against the average.
2. Run the inventory commands above on the machine you have. If your answer to “what is my bandwidth ceiling” is a model name and a hope, look the number up before you shop.
3. Pick the one unknown that would change your decision and resolve it inside a return window, with your own model and your own prompt shapes rather than a vendor benchmark.
4. Put a buy trigger in writing — the staying-still rule above — with a date attached, so the decision does not drift into next year by default.
5. If you already own a GB10-class box and the plan is to serve with vLLM, the DGX Spark LLM Deployment Kit ($49, one-time) is a deployment kit for running vLLM models on an NVIDIA GB10 (DGX Spark): systemd units, memory planning, health checks, a watchdog and a troubleshooting playbook. It removes the unglamorous part of the job so your time goes to the workload instead.

Get DGX Spark LLM Deployment Kit

DGX Spark LLM Deployment Kit — $49, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory