Keeping a Prompt Library Alive When Models Change

A prompt that worked in March can behave differently in June without a single character of it changing. The model moved; the text did not. That is survivable if prompts are versioned artifacts with a test or two, and it is a slow disaster if they live in a browser tab, a notes app, or six people’s heads.

Prompt rot has two causes, and only one of them is the provider

When a provider ships an update, several things can shift at once: tokenization, default verbosity, how strictly a JSON instruction is honoured, how the model weighs a system prompt against a user turn, refusal thresholds, tool-call formatting. None of that is announced as a breaking change, and improvements are not uniform. A model can get better at reasoning and worse at following your formatting rule in the same release.

The second cause is your own drift. The context you paste into a prompt changes, the input distribution changes, the code around the call changes. If you have read what context engineering actually covers, you already know the prompt string is the smallest part of the input.

The real failure is diagnostic: when output quality drops, you cannot tell whether it was the model swap, a prompt edit, or a shift in traffic, because none of those three is recorded anywhere. Fix that first. Everything else here is easier afterwards.

Separate intent from phrasing

An intent file answers questions that stay true across model generations:

• What job is this? One sentence, imperative.
• What are the inputs, and which are required?
• What is the output contract, meaning schema, required fields, forbidden content?
• What counts as a failure?

Phrasing is the actual text you send, and it is disposable. It is the part you rewrite when a model changes. Keep the two in separate files so a migration produces a diff in one file instead of a rewrite of the spec.

# prompts/summarize_ticket/prompt.yaml
id: summarize_ticket
version: 4
intent: >
  Turn a raw support thread into a structured handoff summary for the
  on-call engineer.  
inputs:
  - name: thread
    required: true
contract:
  output: json
  schema: ./schema.json
  must_include: [customer_impact, steps_tried, current_state]
  must_not: [speculation_about_root_cause, promised_timelines]
model:
  provider: openai
  snapshot: gpt-4o-2024-08-06
  temperature: 0
phrasing: ./v4.txt
tests: ./cases/

Two rules keep the split honest. First, if a prompt change requires editing contract, that is a product decision and should be reviewed as one, not slipped in as “just wording”. Second, when a candidate model fails your tests, you rewrite v4.txt and never the intent.

Version prompts next to the code that calls them

A prompt is a behaviour change to a shipped feature. It belongs in the same repository, the same pull request and the same review path as the code that sends it. Prompt platforms that non-engineers edit at runtime are fine for marketing copy; for anything with a schema, they add a sync step and a second source of truth.

Pin snapshot IDs instead of “latest”. Dated snapshot identifiers exist so an evaluation is reproducible; an alias means the model under your tests is not the model in production. Upgrade deliberately, one prompt at a time, and log enough to reconstruct what ran:

# app/llm/summarize.py
import hashlib, pathlib, yaml

PROMPT_DIR = pathlib.Path(__file__).parents[2] / "prompts" / "summarize_ticket"

def load(version: int = 4):
    meta = yaml.safe_load((PROMPT_DIR / "prompt.yaml").read_text())
    text = (PROMPT_DIR / f"v{version}.txt").read_text()
    return {
        "prompt": text,
        "prompt_id": f"{meta['id']}@v{version}",
        "prompt_sha": hashlib.sha256(text.encode()).hexdigest()[:12],
        "model": meta["model"]["snapshot"],
    }

Write prompt_id, prompt_sha and the model snapshot into every response record. When someone reports bad output three weeks later, that row is the difference between a five-minute answer and an afternoon of guessing.

Keep a small regression set, retire everything else

Coverage matters less than specificity. For each prompt that actually drives revenue or support load, keep 8-20 cases encoding what you would notice if it broke: a typical input, an edge case your users hit, one adversarial input, one that must produce empty output. Store them as JSONL next to the prompt and assert on invariants rather than exact strings.

Structural checks survive model changes: valid JSON, required keys present, no forbidden phrases, length within bounds, correct language. Exact-match golden outputs do not. A tokenization change alone can break them while quality is unchanged. If you need a model to judge the output, treat that judge prompt as another prompt with its own version and tests, because it rots too. When automation misbehaves in ways you did not plan for, the same instincts from handling failures that happen while you sleep apply: keep the failure list short and specific.

# baseline first, then the candidate
pytest tests/prompts -k summarize_ticket

MODEL_SNAPSHOT=gpt-4o-2024-11-20 pytest tests/prompts -k summarize_ticket

The failing case IDs are your rewrite work list. That is the point: a model upgrade becomes a bounded task with a known end, instead of a week of eyeballing outputs.

Then delete. A prompt with no caller in the repository and no test is not an asset; it is a thing you will re-evaluate during the next migration for no reason. Git history is your archive. If you need an old prompt back, git log --diff-filter=D -- prompts/ finds it. The instinct that makes teams keep one-off documents around is the same one that turns a prompt folder into a museum, as covered in why templates beat one-off documents. Retire aggressively. You can always recover a file; you cannot recover the review time you spend on dead prompts.

What to do next

1. Inventory every place a prompt string is constructed in your codebase. A crude start is grep -rn "You are a" --include=*.py. List each one with its caller, and delete anything with no caller this week.
2. Pick the three prompts that matter most. Write prompt.yaml with intent, inputs, contract and a pinned snapshot, plus 10 test cases as JSONL.
3. Move them into the repository next to the calling code, and add prompt_id, prompt_sha and the model snapshot to your response logging.
4. Run the suite against the pinned snapshot for a baseline, then against one candidate model. File the failures as rewrite tasks against the phrasing file only.
5. Rewrite or delete the remaining prompts as you touch them. No exceptions, no keeping one “just in case”.

If you would rather start from working examples than a blank file, AI Prompt Library ($29, one-time) is a curated set of ready-to-use prompts organised by job, covering writing, coding, research and ops, delivered as copy-paste templates. Treat them as raw material. The intent file, the contract and the regression cases still have to be yours, because those are the parts that make a prompt survive when the model under it changes.

Get AI Prompt Library

AI Prompt Library — $29, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory