Grading Evidence, Not Effort: A Standard for Course Assessments

Most course assessments fail quietly. The requirement asks for understanding, the rubric rewards presence, and nobody can point to the file that proves anything happened. The fix is a standing rule you apply twice: once before an assignment ships, and once when you review it a year later.

The rule: one requirement, one named artifact

Every requirement in an assessment must map to a named piece of evidence the learner produces, and the rubric scores that evidence — not the effort, not the enthusiasm, not the hours logged.

“Named” is doing the work here. A name means a path, a field, a commit hash, a file you can open without the learner in the room. If the evidence only makes sense with the learner narrating it, it is a note about the learner, not evidence of the work.

What the right column shares: durable, produced by the learner, inspectable by a third party. Three properties, no exceptions. A second reviewer should be able to grade the assignment without talking to the first.

What “evidence” excludes

This list is most of the work, because the things that feel like evidence are usually the things that cannot be checked.

• Effort proxies. Hours logged, attempt counts, “actively participated”. These describe the learner’s calendar, and they reward people with more free time.
• Self-report. Reflections that restate the lesson, confidence ratings, “I feel comfortable with X”. Useful as feedback, useless as a grade, because they cannot be wrong in a way anyone can inspect.
• Presence signals. Forum replies, reactions, meeting attendance. They correlate with engagement and nothing else.
• Bare screenshots. A screenshot of a chat window is evidence that a screen once displayed something. The prompt plus the raw output text is evidence of work. Ship the text.
• Instructor artifacts. If you wrote the config and the learner approved it, the config is yours.
• Undifferentiated dumps. “My whole repo” is not named evidence, it is a haystack. Name the two files.

One exclusion that gets missed: evidence produced under unstated conditions. If the assignment does not say whether AI assistance is allowed and how it must be disclosed, the artifact is uninterpretable and so is the grade. Put that in the assignment, not in a comment thread.

For a related version of this drift, see how prompt libraries rot when nobody names the artifacts: The Maintenance Rhythm That Keeps a Prompt Library Trustworthy.

Designing a new assignment: map first, rubric second

Write the evidence map before you write the prompt prose. Prompt first means you end up with a paragraph of verbs — analyse, reflect, demonstrate — and the rubric inherits them.

Order of work:

1. State the requirement as an observable capability (“can revise a prompt without losing prior behaviour”, not “understands versioning”).
2. Name the artifact that would prove it.
3. Write the decision rule that separates pass from fail.
4. Only now write the learner-facing description, generated from the map.

Keep the map in a file beside the assignment so the two cannot drift:

# assignment: prompt-versioning
requirements:
  - id: R1
    claim: "Revises a prompt without losing prior behaviour"
    evidence:
      - path: prompts/rewrite.v1.md
      - path: prompts/rewrite.v2.md
        must_contain: ["## changelog", "model:"]
      - path: diff/v1-to-v2.patch
  - id: R2
    claim: "Identifies which single change shifted behaviour"
    evidence:
      - path: notes/ablation.md
rubric:
  R1:
    pass: "v2 exists, changelog names the model and the change, patch applies cleanly"
    fail: "changelog missing, or patch does not apply"
  R2:
    pass: "ablation.md names one variable and the observed difference"
    fail: "more than one variable changed, or no observed difference recorded"

Two or three artifacts per requirement is the practical ceiling. Beyond that the grader starts sampling, and sampling is how a rubric becomes a vibe.

When the requirement is a judgement call — picking a tool for a job — the evidence is the written criteria, not the choice. Workflow Builder vs a Plain Script is the kind of comparison a learner should be able to reproduce against their own constraints, and decisions/*.md is where that shows up.

Reviewing an old assignment: walk it backwards

Do not start from requirements. Start from rubric rows and ask, for each one, which evidence field it scores. A row with no citation is either an unmeasured requirement or an effort proxy. Both get deleted or repaired.

import pathlib, sys, yaml

spec = yaml.safe_load(pathlib.Path("assignment.yaml").read_text())
declared = {e["path"] for r in spec["requirements"] for e in r["evidence"]}

problems = []
for req_id, bands in spec["rubric"].items():
    for band, row in bands.items():
        src = row.get("evidence")
        if not src:
            problems.append(f"{req_id}/{band}: no evidence cited")
        elif src not in declared:
            problems.append(f"{req_id}/{band}: cites undeclared path {src}")

if problems:
    print("\n".join(problems))
    sys.exit(1)
print("ok")

Run it in CI beside the course build. It is boring, it takes a minute, and it catches the failure mode that matters: a rubric that quietly drifted into adjectives while the assignment stayed still.

Then read the rubric language out loud. Rows that describe the learner (“shows good understanding”, “actively iterated”) produce two graders who disagree, and both will be right.

If you want the background on why a context-budget row is worth grading at all, What Is Context Engineering? covers the underlying skill.

What to do next

• Take one existing assignment and build the requirement-to-evidence table. Any requirement you cannot name an artifact for gets deleted, not rewritten into softer language.
• Save the map as assignment.yaml and run the checker above against your current rubric. Expect two or three rows with no citation on the first pass.
• Rewrite five rubric rows that describe the learner so they describe a file instead. If a row survives the rewrite unchanged, it was already fine.
• Add one line to every assignment stating whether AI assistance is allowed and how it must be disclosed.

If you want to see the standard applied to a language course rather than a developer course, the Traditional-Chinese AI Course is $69 USD one-time: a four-unit, seventeen-lesson course that takes a beginner from zero to applied AI workflows. It is a download from https://slashmaster6.gumroad.com/l/vzalgb.

Get Traditional-Chinese AI Course

Traditional-Chinese AI Course — $69, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory