Swapping Models on a Single-GPU Box Without Downtime Drama

Swapping the model behind a single-GPU inference server is not a rolling deploy. There is one device, one set of weights in VRAM, and no second node to take over while the first one restarts. The failure mode is predictable: you kill the old process, the new one fails to allocate, and now nothing is serving while you debug in a panic. The fix is a fixed sequence you follow every time, including a way back.

Why concurrent model swaps fail on one GPU

A 24 GB or 48 GB or 128 GB card can hold one model comfortably and two models badly. When you start a new vLLM server while the old one is still running, one of three things happens: the new process fails with a CUDA out-of-memory error, both processes allocate smaller KV caches and both serve badly, or the driver lets them run and you get throughput collapse from contention. None of these is a swap. They are a degraded state you then have to unwind while users are waiting.

The other trap is process-level thinking. Killing the server process does not mean the memory is gone. vLLM workers, NCCL communicators, and the CUDA context can take seconds to tear down. If you start the replacement in the same second, you race the teardown and lose. So the procedure has an explicit wait step, and that wait is checked, not assumed.

Finally, ports. If both servers bind the same port, the new one exits immediately with an address-in-use error and you may not notice until a health check fails later. Sequencing the stop before the start is not optional.

The swap procedure, step by step

Treat this as a runbook. Write it down, put it in the repo, and do not improvise under pressure.

1. Stop accepting work. If anything is in front of the GPU box, take it out of rotation first. For most single-box setups that means flipping a health endpoint or a reverse-proxy upstream so new requests are not routed here, while in-flight requests finish.

# systemd-managed services: stop the intake side first
sudo systemctl stop vllm-gateway.service

# confirm the service is actually down before touching the GPU
systemctl is-active vllm-gateway.service || echo "gateway down"

If you do not have a gateway, the equivalent is a small proxy like nginx or Caddy in front of the model server. Point the upstream at a 503 or remove it, reload, and wait for active connections to drain. A 30-second drain window is usually enough for chat-style traffic; long-running generation needs longer.

2. Stop the old model server. Now, and only now, stop the process that holds the weights.

sudo systemctl stop vllm.service

# Wait for the process to disappear, not just for the command to return
for i in $(seq 1 30); do
  if ! pgrep -f "vllm.entrypoints.api_server" > /dev/null; then
    echo "server process gone after ${i}s"
    break
  fi
  sleep 1
done

3. Confirm VRAM is actually released. This is the step people skip, and it is the one that causes confusing OOM errors on the next start. Check both the driver view and the process view.

nvidia-smi --query-gpu=memory.used,memory.total --format=csv
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

What you want to see is memory.used back down near idle (a driver and display stack may hold a small amount) and an empty or near-empty compute-apps list. If a PID is still holding several gigabytes, do not start the new server. Investigate that PID first — a stuck worker process will block the new allocation no matter how many times you retry.

4. Start the new server with an explicit command. Keep the launch command in a file or a systemd unit, not in your shell history. When something goes wrong at 2 a.m., you want the exact flags that worked yesterday, not a reconstruction.

# /etc/systemd/system/vllm.service (excerpt)
[Service]
ExecStart=/usr/local/bin/vllm serve /models/Qwen3-32B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --served-model-name qwen3-32b
Restart=on-failure
RestartSec=5

Two flags deserve attention. --gpu-memory-utilization sets the fraction of VRAM vLLM will target for weights plus KV cache; on a box with a display or another small consumer, leaving it at 0.90 instead of 0.95 buys headroom. --max-model-len directly affects KV cache size, so lowering it is your first lever when a model that used to fit no longer fits after a config change.

5. Health-check with a real request, not just a port check. A listening socket means the HTTP layer is up. It does not mean the model loaded. Send an actual completion and assert on the response body.

until curl -sf http://127.0.0.1:8000/health > /dev/null; do sleep 2; done

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-32b",
    "messages": [{"role": "user", "content": "reply with the word ok"}],
    "max_tokens": 8
  }' | python3 -c "import json,sys; d=json.load(sys.stdin); assert d['choices'][0]['message']['content'].strip(), 'empty completion'; print('model serving')"

The assertion matters. A server that returns HTTP 200 with an empty or truncated completion is not healthy, and it will pass a naive uptime check. Only after this passes do you put the gateway back and resume traffic.

Keep the previous launch as a rollback

Before you start the new model, write down the old launch parameters somewhere you can paste them. Not “the previous model” — the full command, the model path, the revision or commit if you pin one, and the serving name your clients expect.

# /opt/rollback/vllm-previous.sh
exec /usr/local/bin/vllm serve /models/Llama-3.3-70B-Instruct-AWQ \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --served-model-name llama-3.3-70b

Rollback is then a two-command drill: stop the new unit, and start the previous script as the unit’s ExecStart (or run it directly and fix the unit after). Practice it once on a quiet afternoon so it is muscle memory rather than a first attempt during an incident. If you pin model weights by revision, rollback also protects you from a silent upstream update to the same model name.

If clients address the model by the --served-model-name value, keep that name stable across swaps where the API surface is compatible. Changing the name forces every client to redeploy, which turns a five-minute swap into a multi-service release.

What to do next

1. Write the swap runbook into your repo as a Markdown file with the exact commands above, ordered stop-then-start, including the VRAM verification step.
2. Move your launch command out of shell history and into a systemd unit or a versioned script, so both start and rollback use the same flags.
3. Replace your port-based health check with one real completion request that asserts on the response body, and wire it into whatever notifies you when the box is unhealthy.
4. Record the VRAM baseline of your current model at idle and under load, so the next swap has a number to compare against instead of a guess.
5. Do one dry run of the rollback path while nothing is broken. It is cheap now and expensive later.

If you would rather not assemble the systemd units, memory math, watchdog and health checks yourself, the DGX Spark LLM Deployment Kit ($49, one-time) bundles those pieces for running vLLM models on an NVIDIA GB10, including a troubleshooting playbook for the allocation and startup failures this procedure is designed to catch. If you are still deciding where automation fits around your serving stack at all, /blog/ultimate-ai-automation-guide-2026/ is the broader map, and /blog/triage-habits-for-automation-that-fails-while-you-sleep/ covers what to do when the box fails at 3 a.m.

Get DGX Spark LLM Deployment Kit

DGX Spark LLM Deployment Kit — $49, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory