Standing Routines That Keep a Home LLM Server Dependable

A home LLM server almost never fails in one dramatic moment. It drifts: a probe nobody reads, a cold start that got slower after a config edit, a restart only one person knows how to run at 11pm. The five routines below are the standing habits that keep the box boring, and each one is small enough to survive a busy week. If you are still deciding how the machine fits into the rest of your tooling, the layering advice in how to wire an AI dev stack so each layer has one job applies here too — the probe, the restart path and the fallback are separate layers with separate jobs.

A health probe that runs on a timer

There are three different questions people lump into “is it up”: is the process alive (a TCP connect answers that), is the HTTP server ready (/health returning 200 on vLLM), and is the model I expect actually loaded (/v1/models lists it). Most “the server is down” reports are the first two. Most “the server is broken” reports are the third, usually after someone edited a model path or a --max-model-len value.

Keep the probe stateless, quiet on success, and non-zero on failure. Let systemd do the scheduling.

#!/usr/bin/env bash
# /usr/local/bin/llm-probe.sh
set -uo pipefail
ENDPOINT=${ENDPOINT:-http://127.0.0.1:8000}
MODEL=${MODEL:-llama-3.1-8b-instruct}
LOG=${LOG:-/var/log/llm-probe.log}

state=$(systemctl show -p ActiveState --value vllm.service)
if [ "$state" = "activating" ] || [ "$state" = "deactivating" ]; then
  printf '%s state=%s skipping\n' "$(date -Is)" "$state" >> "$LOG"; exit 0
fi

code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 8 "$ENDPOINT/health" || echo 000)
models=$(curl -s --max-time 8 "$ENDPOINT/v1/models" | grep -c "\"$MODEL\"" || true)
vram=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -n1)
printf '%s state=%s http=%s model_hits=%s vram_mib=%s\n' \
  "$(date -Is)" "$state" "$code" "$models" "$vram" >> "$LOG"
[ "$code" = "200" ] && [ "$models" -ge 1 ]

Two units run it. A failed run can trigger an alert unit, and the timer does the cadence.

# /etc/systemd/system/llm-probe.service
[Unit]
Description=Probe the local LLM endpoint
[Service]
Type=oneshot
User=llm
ExecStart=/usr/local/bin/llm-probe.sh
OnFailure=llm-alert.service

# /etc/systemd/system/llm-probe.timer
[Unit]
Description=Run the LLM probe every five minutes
[Timer]
OnBootSec=3min
OnUnitActiveSec=5min
AccuracySec=30s
Persistent=true
[Install]
WantedBy=timers.target

One honest caveat: vLLM does not call sd_notify, so adding WatchdogSec= to the vLLM unit does not give you a real liveness watchdog — systemd cannot distinguish a hung process from a busy one. The external timer is the watchdog. Also decide up front how noisy failure is: OnFailure= fires on every failed run, so during a genuine outage you get a message every five minutes. Either alert only on the second consecutive failure (a tiny counter file inside the script) or let the alert receiver deduplicate.

A log review with a fixed appointment

Fifteen minutes, same slot every week, no dashboard — a person reading text.

# last week's warnings and errors, newest last
journalctl -u vllm.service --since '7 days ago' -p warning --no-pager | tail -n 40
# counts for the messages worth tracking over time
journalctl -u vllm.service --since '7 days ago' --no-pager \
  | grep -Eic 'out of memory|timeout|cuda error'

Write one line into /srv/llm/ops/weekly-notes.md: date, anything odd, anything changed. The point is not curation, it is that weeks three through ten establish what normal looks like on your hardware, which is the only baseline that matters when something shifts.

What to watch for: repeated out-of-memory messages (memory planning too tight, or a context length that is more generous than your workload), queue and preemption messages (that is demand, not a bug), client disconnects during long generations (a timeout mismatch between the proxy and the server), and a rising count of start and stop events you did not trigger (something else is restarting the service). Give the probe log a logrotate rule at the same time; an unattended append-only file is a slow disk leak. If your home setup is still small enough that a full automation framework feels heavy, the smaller-guide framing in the AI automation guide is a reasonable calibration point.

A restart path written before you need it

Write /srv/llm/ops/RUNBOOK.md with exact commands, the order to run them, the expected duration, the success criterion, and the first command to run for each known failure mode. Order matters: stop or redirect the proxy first, restart vLLM, wait for /health to return 200, then bring the proxy back. Starting the proxy first just aims requests at a dead port.

sudo systemctl stop llm-proxy.service
sudo systemctl restart vllm.service
# in the runbook, put a timeout around this loop; do not let it hang forever
until curl -sf -o /dev/null http://127.0.0.1:8000/health; do sleep 5; done
sudo systemctl start llm-proxy.service

Known failure modes worth writing down while the box is healthy: VRAM still held by an orphaned worker (nvidia-smi, then kill the PID, then start the unit again), port 8000 already in use (ss -ltnp | grep 8000), and weights that are missing or unreadable by the unit’s User= (check permissions on the model directory, not just the file). One thing to note inside the runbook: the first load after a boot reads weights from disk, while a restart minutes later often finds them in the page cache. Both are valid numbers, but they are not comparable.

A load-time baseline you compare against

Define the measurement precisely so it means something: wall clock from issuing the start to the first 200 on /health. Record it on every restart and compare today’s number against the median of the last ten runs. A simple rule that works: investigate if the load is more than about 25% above the median twice in a row, not once.

#!/usr/bin/env python3
# compare today's cold load time against the stored baseline
import json, pathlib, statistics, sys

path = pathlib.Path("/var/lib/llm/load-times.json")
times = json.loads(path.read_text()) if path.exists() else []
latest = float(sys.argv[1])
baseline = statistics.median(times[-10:]) if times else None
times.append(latest)
path.write_text(json.dumps(times[-50:]))

if baseline is None:
    print(f"first run recorded: {latest:.1f}s")
    raise SystemExit(0)
delta = (latest - baseline) / baseline
print(f"load {latest:.1f}s vs median {baseline:.1f}s ({delta:+.0%})")
raise SystemExit(1 if delta > 0.25 else 0)

A written fallback for when the server is down

One page, four lines: how long you are willing to spend fixing it, what you switch to, how you switch, and who gets told. Keep the switch as configuration rather than code — clients read LLM_BASE_URL and LLM_MODEL from the environment, and the fallback values sit in the same file with a comment. When the box is down, changing one variable beats editing a client. If you share the server with anyone, name the notification channel in the page, because “I will mention it” is not a plan.

Two rules keep this honest. First, keep the fallback model name current — a fallback pointing at a retired model is worse than no fallback, because the failure is silent. Second, exercise the switch once a quarter: point a client at the fallback endpoint, run one request, switch back. A fallback you have never exercised is a rumor. This is the same reasoning behind upgrading one layer of a stack without breaking the rest: the fallback works only if it is a separate, tested layer.

What to do next

1. Install the probe script and timer this week, then check after a day that the log has roughly one line per interval and that a good line shows http=200 model_hits=1.
2. Book a 15-minute weekly log review. For the first session, only count how often each warning appears and write the numbers down — that is your definition of normal.
3. Write the restart section of RUNBOOK.md, then perform one restart by hand and paste the real elapsed time into the document.
4. Add load-time recording to the restart step, storing results in /var/lib/llm/load-times.json, and take a first baseline before you change any configuration.
5. Write the fallback page and test the environment-variable switch once with a single request.

If you would rather start from working scaffolding than write the units yourself, the DGX Spark LLM Deployment Kit ($79 USD, one-time) is a deployment kit for running vLLM models on an NVIDIA GB10 (DGX Spark) that includes systemd units, memory planning, health checks, a watchdog and a troubleshooting playbook; it is a download from https://slashmaster6.gumroad.com/l/bppdqp. The routines above still apply — the kit just removes the blank-page step.

Get DGX Spark LLM Deployment Kit

DGX Spark LLM Deployment Kit — $79, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory