Harbor Evals Part 3: Writing Tasks with Pass and Fail Fixtures

You have a scaffold. Now the task has to mean something. This post is about writing Harbor tasks:

• Reference pass and fail outputs.
• Choose deterministic pytest versus LLM-as-judge.
• Keep instruction.md neutral so you do not leak the answer.
• Name the output file so your verifier can find it.

The living samples are in CodeSloth Cursor Samples (opens in a new tab) under harbor-evals/simple/ (opens in a new tab), next to Team Brain intro (opens in a new tab). Snippets below are from that repo - when the samples change, prefer the GitHub files over an outdated fence in this post.

Back: Part 2 - Install Harbor and Scaffold Tasks (opens in a new tab). Ahead: Part 4 - Task Shapes and DRY (opens in a new tab). Series map: Harbor Evals for Agentic Skills (opens in a new tab).

Start from a contract, not a vibe

Good skill evals pin a contract you already asked the agent to honor. From Building Scalable Skills for Agentic Workflows (opens in a new tab), Team Brain announces classification with:

Intent identified: <intent-id>

That line is a strong eval target. It is short, stable, and human-visible. The verifier can assert the prefix + intent id. The agent-facing instruction should not spoon-feed that string.

Our two simple tasks assert different ids (see test_reply.py for intent (opens in a new tab) and for calculator (opens in a new tab)):

Intent identified: reference-answer
Intent identified: structured-calculator

If the agent skips the announcement, routes the wrong intent, or buries the line in a paragraph of preamble, the eval should fail - not “sort of okay if you squint.”

Reference pass and fail outputs

Keep fixtures next to the task so humans (and future-you) know what pass and fail look like without re-running models. In team-brain-intent/ (opens in a new tab) that looks like:

# Task folder with pass/fail fixtures beside the Harbor surfaces:
team-brain-intent/
├── instruction.md
├── task.toml
├── environment/
├── solution/
├── tests/
└── fixtures/
    ├── pass_reply.txt
    └── fail_reply.txt

Pass fixture (fixtures/pass_reply.txt (opens in a new tab)) - a reply that should satisfy the verifier:

Intent identified: reference-answer
 
Harbor is a framework for evaluating and optimizing AI agents in container environments. It grew out of lessons from Terminal-Bench and focuses on modular tasks, environments, and agents.
 
- Tasks ship with instruction.md, a Dockerfile sandbox, and a verifier that writes a reward.
- Use it to assert skill behavior (for example an intent announcement prefix) reproducibly.

Fail fixture (fixtures/fail_reply.txt (opens in a new tab)) - a near-miss that must not pass (here: useful prose, missing intent line):

Harbor evaluates agents in Docker. Here is a long answer without the required intent line.
 
This fixture must fail the verifier: missing `Intent identified: reference-answer`.

Other useful fail shapes when you add more fixtures:

• Wrong intent id (structured-calculator when you asked a docs question)
• Missing prefix (I think this is reference-answer)
• Correct intent buried after a long preamble with no clean first-line match
• Empty file / wrong path

The calculator task mirrors the same idea: pass_reply.txt (opens in a new tab) has the right intent + result: 60; fail_reply.txt (opens in a new tab) keeps the calculation body but announces reference-answer instead.

Wire fixtures into local pytest if you can - run the same assertions against fixture files without paying for an agent trial. Then the Harbor trial becomes “agent produced something; verifier grades it,” while fixture tests protect the grader itself.

Deterministic Python / pytest vs LLM-as-judge

Harbor’s verifier story is flexible. The task tutorial (opens in a new tab) shows tests/test.sh installing deps, running pytest, and writing /logs/verifier/reward.txt (1 or 0). For multi-criterion rubrics and LLM judging, Harbor points you at Reward Kit-style tooling.

For Team Brain evals, prefer deterministic checks. Asserting Intent identified: reference-answer does not need a judge model. Save LLM-as-judge for later tasks where “was this on-call summary useful?” is the real question - and even then, consider hybrid scoring (hard checks first, soft rubric second).

Here is the real verifier from team-brain-intent/tests/test_reply.py (opens in a new tab):

"""Deterministic checks for Team Brain intent announcement.
 
Harbor runs these via tests/test.sh (pytest). A pass writes reward 1; a fail writes 0.
"""
 
from pathlib import Path  # pathlib.Path = filesystem path helper in the standard library
 
# Absolute path inside the Harbor sandbox where the agent must write its reply.
REPLY = Path("/app/output/reply.txt")
# Exact first-line string the skill contract requires (kept out of instruction.md).
EXPECTED_INTENT = "Intent identified: reference-answer"
 
 
def test_reply_file_exists():
    # Fail early if the agent never created the output file.
    assert REPLY.is_file(), f"Missing agent output: {REPLY}"
 
 
def test_first_line_is_intent_identified_reference_answer():
    # Read the whole reply as UTF-8 text.
    text = REPLY.read_text(encoding="utf-8")
    assert text.strip(), "reply.txt is empty"
    # First line only - that is where Team Brain announces the intent id.
    first_line = text.splitlines()[0]
    assert first_line == EXPECTED_INTENT, (
        f"Expected first line {EXPECTED_INTENT!r}, got {first_line!r}"
    )
 
 
def test_reply_has_body_after_intent():
    # Split into lines, then keep non-empty lines after the intent announcement.
    lines = REPLY.read_text(encoding="utf-8").splitlines()
    body = [ln for ln in lines[1:] if ln.strip()]
    assert body, "Expected at least one non-empty line after the intent announcement"

Harbor does not call pytest directly. tests/test.sh (opens in a new tab) runs it and writes the reward file under /logs/verifier/ (see the task structure docs (opens in a new tab)):

#!/bin/bash
# Harbor verifier: write 0|1 to /logs/verifier/reward.txt
# Treat unset variables as errors (safer bash)
set -u
 
# Ensure the verifier log directory exists
mkdir -p /logs/verifier
 
# Harbor copies tests/ into /tests at runtime - run pytest from there
cd /tests
# Run the pytest file; -v = verbose, --tb=short = shorter failure traces
if python3 -m pytest test_reply.py -v --tb=short; then
  # pytest exited 0: write reward 1 (pass)
  echo "1" > /logs/verifier/reward.txt
  echo "Success: intent eval passed"
  exit 0
else
  # pytest failed: write reward 0 (fail)
  echo "0" > /logs/verifier/reward.txt
  echo "Failure: intent eval failed"
  exit 1
fi

The calculator twin adds asserts for result: 60 and the structured markers - see team-brain-calculator/tests/test_reply.py (opens in a new tab).

Instruction must name the expected output file

This is the footgun that wastes the most hours: your pytest looks for /app/output/reply.txt, but instruction.md never told the agent to write there. The model prints a long answer to the terminal transcript… and the verifier reads an empty path.

Spell the artifact contract in the instruction - path and “write what you would show the user” - without leaking the grading contract. From team-brain-intent/instruction.md (opens in a new tab):

# Team Brain intent eval
 
Use the Team Brain skill with the user message below.
 
Write everything you would show the user to `/app/output/reply.txt` (create `/app/output/` if it does not exist). Do not only print to stdout.
 
## User message
 
> What is Harbor used for when evaluating AI agents?

team-brain-calculator/instruction.md (opens in a new tab) is the same shape with a different user message (Calculate (12 + 8) * 3).

Be explicit about:

• Absolute path (or a path relative to a documented WORKDIR)
• Create directories if missing
• User-facing reply (the full thing you would present, not a private scratchpad)

What the Oracle actually does

A Harbor trial has two phases:

1. Something produces the artifact (usually /app/output/reply.txt).
2. The verifier grades that artifact (tests/test.sh → pytest → reward 1 or 0).

When you run a real agent, step 1 is the model. When you run the Oracle (-a oracle or harbor tasks test … --solution), step 1 is your solution/solve.sh (opens in a new tab) instead. No model is called. The script’s only job is to put a known-good file in the same path an agent would write to - in our sample, that means copying the pass fixture:

# Copy the known-good pass fixture into the path the verifier will read
cp fixtures/pass_reply.txt /app/output/reply.txt

Then Harbor runs the same verifier it would after a real agent. Oracle is not a second grading system. It is preparation for the verification step, so you can prove the task and rubric work before you spend API tokens.

If Oracle fails, fix solve.sh, paths, or pytest - not the skill. If Oracle passes and a real agent fails, then you have an agent / skill problem worth debugging.

Full script from the samples:

#!/bin/bash
# Oracle solution: install the known-good fixture as the agent output.
# Exit on error (-e), treat unset vars as errors (-u), fail pipelines on first error (-o pipefail)
set -euo pipefail
 
# Create the output directory the instruction asks the agent to use
mkdir -p /app/output
# Harbor copies solution/ to /oracle/ (or runs from task dir). Prefer /tests sibling fixtures via task layout.
# When run via `harbor tasks test --solution`, this script executes in the sandbox.
# Resolve this script's directory so relative fixture paths work no matter where we start
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
# Prefer fixtures next to solution/ (task-dir layout)
if [ -f "${SCRIPT_DIR}/../fixtures/pass_reply.txt" ]; then
  cp "${SCRIPT_DIR}/../fixtures/pass_reply.txt" /app/output/reply.txt
# Fallback when Harbor mounts solution under /oracle/
elif [ -f /oracle/../fixtures/pass_reply.txt ]; then
  cp /oracle/../fixtures/pass_reply.txt /app/output/reply.txt
else
  # Inline fallback so oracle still works if fixtures are not mounted
  # Write a heredoc into reply.txt (EOF quoted so $VARS are not expanded)
  cat > /app/output/reply.txt << 'EOF'
Intent identified: reference-answer
 
Harbor is a framework for evaluating and optimizing AI agents in container environments. It grew out of lessons from Terminal-Bench and focuses on modular tasks, environments, and agents.
 
- Tasks ship with instruction.md, a Dockerfile sandbox, and a verifier that writes a reward.
- Use it to assert skill behavior (for example an intent announcement prefix) reproducibly.
EOF
fi

Avoid leaking the answer into the instruction

A skewed instruction tells the agent the secret the verifier will check:

# Skewed - do not do this (leaks the verifier assert into the agent prompt)
 
1. Classify intent the way Team Brain would.
2. The first line MUST be exactly: `Intent identified: reference-answer`

That mostly tests “can the model copy a string from the prompt?” It does not test whether the skill routed and announced correctly for a real user message.

Keep expected intent ids, envelope templates, and “success criteria” restatements in:

• tests/ (pytest)
• fixtures/ (pass / fail examples for humans and grader tests)
• solution/ (Oracle)

…not in the text the trial agent reads as its job description.

When you first write evals, prefer neutral instructions: use the skill, here is the user message, write the user-facing output to this file. Let the skill and the model do the routing work your verifier cares about.

When steering the agent is still fair game

Default to neutral instructions. You can still tell the agent to do almost anything when you invoke a skill: run only the beginning of a workflow, jump into the middle, or exercise just the end if those are the behaviours you care about measuring.

That is similar to unit tests that mock an HTTP response. In production the service makes a real request; in the test you fake the payload so you can assert what your code does with known data. In a Harbor task you might:

• Seed a mid-workflow state (“assume intent X was already chosen; continue from step 3”)
• Ask for only one phase (“classify and stop; do not write the long answer”)
• Pin a tool outcome (“treat the calculator result as 60 and format the envelope”)

Use that flexibility when it is pragmatic - isolating a flaky middle step, cutting cost, or grading one contract without paying for the whole skill path. Just label the skew honestly for yourself: you are no longer measuring full end-to-end routing from a blank user prompt. For early evals, stay neutral until you know which slice needs a mock-style instruction.

Team Brain eval as the running example

Why start with these simple tasks?

• Fast feedback while you learn Harbor
• Cheap models can pass if the skill is wired correctly
• Failures localize to routing / announcement, not deep domain quality

Mounting or copying the Team Brain skill into the container is an environment concern - Part 4 covers mounts, shared base images, and keeping sibling tasks DRY. For this post, assume the task environment can see the skill files the agent needs, and focus on instruction + verifier clarity.

A short authoring checklist

Before you call a task “done”:

1. Instruction states the goal, user message, and exact output path - without leaking the assert string when you want a full routing eval.
2. Pass fixture satisfies the verifier; fail fixtures do not.
3. Oracle (solution/solve.sh) prepares the same artifact path a successful agent would; then the verifier grades it.
4. Verifier is deterministic unless you have a documented reason otherwise.
5. Reward file is written by tests/test.sh (reward.txt or reward.json under /logs/verifier/).
6. You can explain in one sentence what a failed trial means for the skill author.

Running matrices and browsing trials in harbor view is Part 5. Here, “done” means the task is gradable.

Sloth Summary

Task writing without the fluff:

• Pin skill contracts (Intent identified: …) in the verifier, not by pasting them into instruction.md.
• Keep pass and fail reference outputs so graders and humans share the same examples.
• Prefer deterministic pytest for these evals; reserve LLM-as-judge for soft rubrics.
• Put the expected output file path in instruction.md - if the agent was never told to write it, the verifier cannot save you.
• Oracle prepares a known-good file in that path; the verifier grades it - use that before paying for a model.
• Default to neutral instructions; steer or “mock” mid-skill behaviour only when that isolation is the point.
• Follow the living files under harbor-evals/simple/ (opens in a new tab) - the repo is the source of truth when samples move on.

Next we zoom out to task shapes - how to share images, mounts, and generators without copy-pasting the same fragments into every task.

May your fail fixtures fail for the reason you intended. 🦥