You ship a Cursor skill. It works in your chat. A teammate tries it tomorrow and… the agent routes the wrong intent, skips the announcement line, or invents a workflow that does not exist. You nudge the markdown, try again, feel better, and hope nothing regresses next week.
That hope is not a quality bar. What are evals in this world? They are repeatable checks that ask: given this instruction and this environment, did the agent do the thing we care about - and can we prove it without staring at transcripts every time?
This is Part 1 of Harbor evals for agentic skills. See the series overview (opens in a new tab) for the full map. We stay conceptual here: why evals matter for skills, how Eval-Driven Development (EDD) sits next to Test-Driven Development (TDD), and what Harbor (opens in a new tab) is good for. Install and scaffolding wait for Part 2 (opens in a new tab). Writing tasks, task shapes, runs, and a full EDD loop follow after that.
If you have not read Building Scalable Skills for Agentic Workflows (opens in a new tab) yet, skim it first - Team Brain’s Intent identified: contract is the example we will build evals around later.
The painful manual skill loop
Skill authoring without evals usually looks like this:
- Edit
SKILL.md/ a workflow / a reference file. - Open a fresh chat (or pray the old one is “clean enough”).
- Prompt the agent with a few happy-path questions.
- Squint at the reply and check the contract pieces by hand: intent announcement, loaded file, structured envelope.
- Fix prose. Repeat.
That loop is slow, noisy, and biased toward the prompts you remember to try. It also does not scale when you care about multiple models, multiple agents, or regression suites that grow with every skill change. You are the verifier - and checks you only run by hand will not catch regressions overnight the way continuous integration (CI) would.
Manual checks still have a place (exploratory demos, user-experience feel). They just should not be the only verification you rely on.
What an eval actually is
An eval is a small experiment with three pieces:
| Piece | Role |
|---|---|
| Instruction | What you ask the agent to do |
| Environment | Where it works (tools, files, operating system, network constraints) |
| Verifier | How you score success after the attempt |
One attempt is a trial. A batch of trials across tasks / agents / models is a job. Pass/fail (or a numeric reward) is the point - not a vibes-based “looks fine to me.”
For skills, good evals pin contracts rather than essay quality:
- Did the reply start with
Intent identified: reference-answer? - Did the calculator path emit the required structured block?
- Did the agent write a known output file with an expected line?
Those are boring on purpose. Boring checks are cheap to run and hard to argue with.
EDD vs TDD (conceptually)
You already know TDD: write a failing test, change the code until the test passes, then refactor while the suite keeps catching regressions.
Eval-Driven Development (EDD) is the same rhythm aimed at agent behavior:
- Write an eval that fails against the current skill (or against “no skill”).
- Change the skill / prompts / workflow so the agent passes.
- Keep the eval. Add more. Refuse to “fix by vibes” alone.
Differences worth naming:
- Unit tests usually exercise deterministic code in-process. Evals often exercise a model-backed agent in a container, so they are slower and cost API tokens.
- Flakiness is a first-class concern. Prefer checks that do not need a second model to grade the first.
- Oracles matter. Harbor’s Oracle agent can run your known-good
solution/script to prove the task is solvable before you blame the model.
Part 6 of this series turns EDD into a concrete Cursor skill loop (interview → write Harbor evals → improve Team Brain until the evals pass, with a hard iteration cap). Here we only need the mindset: the eval is the product requirement, not a retrospective screenshot.
What Harbor is
Harbor (opens in a new tab) is a framework for evaluating and optimizing agents and models in container environments. It grew out of lessons from shipping Terminal-Bench (opens in a new tab): people wanted custom evals, prompt experiments, reinforcement learning (RL) / supervised fine-tuning (SFT) traces, and continuous integration (CI)-style agent testing - and defining containerized tasks at scale was painful without shared plumbing.
Harbor gives you modular pieces for:
- Tasks - instruction + environment + verifier (directory format)
- Agents - including popular command-line interface (CLI) agents already wired in
- Environments - usually Docker images; local Docker or cloud sandboxes
- Jobs / trials - run matrices and collect rewards
- Datasets / registry - share and reuse benchmarks
The open-source project lives at github.com/laude-institute/harbor (opens in a new tab). Official docs start at harborframework.com/docs (opens in a new tab).
Why containers matter
Each trial runs in a container shaped by your task’s environment/ (typically a Dockerfile, sometimes Compose or a pre-built image). That isolation is how you keep “agent installed random packages” from contaminating your laptop - and how you replay the same setup tomorrow.
API keys are on you (and so is the bill)
When you run real agents against real models, you supply the application programming interface (API) keys and you pay for the tokens. Harbor orchestrates the trials; it does not magically subsidize Claude / GPT / Cursor usage. Budget like you would for any other model-heavy CI job: start small, fail fast, then widen the matrix.
Model matrices and cost
A Harbor job can fan out across agents and models. That is useful for skill authors: “does our intent announcement survive Haiku and a stronger model?” Matrices also multiply cost. Prefer a cheap model that still passes your deterministic checks for day-to-day passing runs, and save the expensive models for spot checks or release gates.
Prefer deterministic verifiers over LLM-as-judge
Harbor supports rich verifiers (including large language model (LLM)-as-a-judge style scoring via ecosystems like Reward Kit). Use that when the task truly needs rubric judgment - prose quality, open-ended user experience, soft constraints.
For skill contracts, reach for pytest / string / file assertions first:
- Exact prefix:
Intent identified: structured-calculator - File exists at a path the instruction required
- JavaScript Object Notation (JSON) / text fixture matches a golden output
Deterministic checks are faster, cheaper, and do not grade one LLM with another LLM’s mood. Save the judge model for cases where you cannot write a crisp assert.
Coming in this series
| Part | Focus |
|---|---|
| Part 2 (opens in a new tab) | Install Harbor, harbor tasks init, default folder layout |
| Part 3 (opens in a new tab) | Pass/fail fixtures, deterministic vs LLM-judge, expected output files |
| Part 4 (opens in a new tab) | Task granularity, shared images, mounts, generators |
| Part 5 | Running jobs, API keys, multi-model matrices, harbor view |
| Part 6 | EDD in practice against Team Brain |
Companion samples will live under harbor-evals/ (opens in a new tab) in CodeSloth Cursor Samples (opens in a new tab), next to the Team Brain intro (opens in a new tab) skill from the prior post.
Sloth Summary
Before we touch the CLI, keep these points straight:
- Manual skill loops rely on human memory; evals turn skill contracts into repeatable trials with a reward.
- EDD follows the same rhythm as TDD for agents: fail the eval, change the skill, keep the suite so regressions stay visible.
- Harbor packages containerized tasks, agents, and jobs so you can run those trials locally or in cloud sandboxes.
- You bring API keys and pay for model usage - size matrices carefully and prefer cheap passing models for routine runs.
- Prefer deterministic verifiers (pytest, file checks) over LLM-as-judge whenever the contract is assertable.
Next up: install the CLI and scaffold a task directory without inventing the folder tree from scratch.
May your regressions announce themselves before your teammates do. 🦥