Plan Model Load Time So the Server Comes Up Predictably

Model load time is a budget line, not a mystery. It is disk read, memory placement, and a one-time warm-up, and each part fails in a different way. Once you can name which part is growing, “it takes a while to start” becomes a number you either accept or change.

What moving weights into unified memory actually costs

Weights are bytes: parameters times bytes per parameter. At fp16 that is 2 bytes per parameter, at fp8 one byte, at 4-bit roughly half a byte. A 32B model is about 64 GB of weights at fp16 and about 32 GB at fp8, before any cache. On a GB10, CPU and GPU share one 128 GB pool of unified memory, so weights, KV cache and runtime overhead come out of the same number. A configuration that fits with a short context can fail with a long one, and it fails slowly, after a long load.

The load runs in three phases:

1. Read. The checkpoint comes off disk. With safetensors and mmap, vLLM’s default, this is lazy: the file is mapped and pages fault in as they are touched. “Started” comes fast, “serving” comes much later.
2. Place. Weights are copied into the memory the engine will use. Peak memory is highest here because source and destination briefly coexist.
3. Initialize. KV cache blocks are allocated for --max-model-len and --gpu-memory-utilization, CUDA graphs are captured, kernels compile, the tokenizer loads. None of that depends on your prompt.

Phase 3 is the one usually blamed on “the model is big”. It scales with context length and batch shape, not parameter count. Raising --max-model-len on an unchanged model makes startup longer and steady-state memory higher.

Measure the load path before you tune it

Four timestamps per start matter: process start, “model loaded” in the log, /health returning 200, and the first completed request. The gap between the last two is what users feel, and it is invisible if you only watch service status.

#!/usr/bin/env bash
# /usr/local/bin/measure-load.sh — one line per restart
set -euo pipefail
LOG=/var/log/vllm/load.log; API=http://127.0.0.1:8000
t0=$(date +%s.%N)
systemctl restart vllm.service
until curl -fsS -o /dev/null "$API/health"; do sleep 0.5; done
t1=$(date +%s.%N)
curl -fsS "$API/v1/completions" -H 'Content-Type: application/json' \
  -d '{"model":"local","prompt":"warm","max_tokens":16}' >/dev/null
t2=$(date +%s.%N)
awk -v a="$t0" -v b="$t1" -v c="$t2" -v d="$(date -Is)" \
  'BEGIN{printf "%s ready=%.1fs cold=%.1fs\n", d, b-a, c-b}' >> "$LOG"

Run it after every deploy for a week. A bimodal ready time means page cache, not the model. A flat line that steps up after a config change points at context length or memory utilisation. Keep the startup log line where vLLM reports KV cache size and maximum concurrency: it should move when you change --max-model-len and stay put when you do not.

Warm both caches so the first request is not the slowest

There are two caches and they need separate warm-ups.

The file cache belongs to the operating system. Read the weight files once before the engine starts:

[Service]
Type=simple
TimeoutStartSec=0
ExecStartPre=/usr/local/bin/preload.sh /models/qwen-safetensors
ExecStart=/opt/vllm/bin/vllm serve /models/qwen-safetensors --port 8000
Restart=on-failure
RestartSec=5

preload.sh can be a loop of cat "$f" > /dev/null over the *.safetensors files, or vmtouch -t on the directory, which also reports what stayed resident. ExecStartPre blocks startup and counts against TimeoutStartSec, whose default of 90 seconds is not sized for a model load, so raise it.

The runtime cache belongs to the engine: CUDA graphs for the batch shapes it will see, and the prefix cache if enabled. /health returning 200 only means the HTTP layer is up. After it passes, send one request shaped like your worst realistic call, full system prompt and maximum max_tokens, and discard the answer. That request pays the graph capture and the prefill; later requests with the same system prompt hit the prefix cache.

Two flags are worth knowing. --enforce-eager skips graph capture and shortens startup at the cost of per-token throughput. --load-format dummy skips weight loading entirely and is for testing the server path, never for serving.

Keep heavy jobs off the boot path

The worst slow start is not slow at all, it is two things starting at once. A backup, an index rebuild, a dataset download or a second model server competes with the loader for disk bandwidth and, under unified memory, for pages. Weights get paged back out, the watchdog deadline passes, systemd restarts the unit, and the cycle repeats with the job still running. This is the same failure mode covered in adding a health check and watchdog to a local LLM server, with timing added.

Order units explicitly and gate on readiness rather than elapsed time:

# vllm-ready.service
[Unit]
After=vllm.service
Requires=vllm.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/warmup.sh

# nightly-index.service
[Unit]
After=vllm-ready.service
Requires=vllm-ready.service

warmup.sh is the same warm request used in the measurement script; keep one script called from both places so the gate and the measurement never disagree. Set the watchdog deadline above your worst measured load, never against a warm restart, and use Conflicts= for any unit that cannot share memory with the engine. The boot ordering notes in one week of watching a home DGX LLM server are the same lesson from the other direction.

When slow is fine, and when the config is wrong

Slow is fine when it is proportional to weight bytes, stable across restarts, written down, and followed by a warm first request. Anything else is information.

The last row is the honest one. If the number is large but stable and budgeted, schedule around it: longer TimeoutStartSec, a readiness gate, no cron at boot. If it is large and unstable, the config is wrong, not the model. Before you blame the model, re-read upgrading one layer of the stack: most load-time surprises arrive with a flag change, not a checkpoint change.

What to do next

1. Add the measurement script to your deploy path and collect a week of ready and cold numbers before changing any flag.
2. Set TimeoutStartSec and the watchdog deadline from your worst measured cold load, not from a warm restart.
3. Pre-read the weight files in ExecStartPre, send one warm request after /health, then re-measure.
4. Move every heavy timer behind a vllm-ready.service gate and check dmesg for the OOM killer after the next restart.
5. Write the load budget into the runbook: expected range, what makes it grow, what to check when it does.

If you would rather not assemble the unit ordering, readiness gate and watchdog yourself, the DGX Spark LLM Deployment Kit is a $79 one-time download for running vLLM on an NVIDIA GB10: systemd units, memory planning, health checks, a watchdog and a troubleshooting playbook. It is available at https://slashmaster6.gumroad.com/l/bppdqp.

Get DGX Spark LLM Deployment Kit

DGX Spark LLM Deployment Kit — $79, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory