Harbor Task Shapes and DRY (Harbor Evals Part 4)

Parts 2 and 3 gave you a single task that can pass or fail on purpose. Start there. The pain shows up when you have ten of them and every Dockerfile re-clones the same tree, every task.toml drifts a timeout, and nobody remembers which copy of the skill mount is “the real one.”

This post is about Harbor task shapes and DRY:

• How granular to make each eval.
• When a shared base image beats per-task fat images.
• How sibling tasks and volume mounts keep skill trees out of image layers.
• When to leave harbor run for your own orchestration.
• How a tiny generate/copy script keeps task.toml, Dockerfiles, and Compose files from drifting out of sync.

Samples live under harbor-evals/advanced/ (opens in a new tab) in CodeSloth Cursor Samples (opens in a new tab).

Back: Part 3 - Writing Tasks (opens in a new tab). Ahead: Part 5 - Running and Results (opens in a new tab). Series start: Part 1 - What Are Evals? (opens in a new tab).

Two granularities (and why both exist)

Think in two shapes:

The harbor-evals/simple/ (opens in a new tab) suite is the simple shape. Each task owns a slim environment/Dockerfile, an instruction that forces /app/output/reply.txt, and pytest that asserts Intent identified: ….

The advanced suite keeps the same success contract (reply file + intent prefix). What changes is how the environment is built and shared:

• Shared base image - one Docker image holds the common toolchain and repo tree.
• Sibling task images - each task Dockerfile starts FROM that base and only adds what that task needs.
• Compose mounts - the compose file shows how the skill tree is volume-mounted into the container.
• Shared-snippet generator - a small script keeps repeated Dockerfile or instruction fragments in sync.

Keep both shapes in the suite. Small tasks stay cheap for continuous integration (CI); complex flows catch multi-step regressions.

Shared base image: shallow clone once

When sibling tasks need the same repo tree, cloning it in every task Dockerfile is slow. Build a base image once, tag it locally, and keep sibling Dockerfiles small.

From the samples:

# Enter the shared base-image directory in the samples repo
cd harbor-evals/advanced/base-image
# Build a Docker image and tag it codesloth-harbor-base:local (`.` = this directory as build context)
docker build -t codesloth-harbor-base:local .

The base Dockerfile is deliberately honest about the pattern. It installs git, keeps a placeholder tree by default (so offline builds stay cheap), and leaves a commented shallow-clone block pinned to a commit SHA (not a branch name).

The clone is done as small Git steps instead of git clone --branch main. That is still a normal checkout - you init an empty repo, add the remote, fetch one commit, then check it out. The point is to pin a SHA with --depth 1. A branch name is a weak Docker cache key: remote main can move while a cached layer still holds an old tree.

From harbor-evals/advanced/base-image/Dockerfile (opens in a new tab) (uncomment the ARG / RUN lines to use):

# Build-time remote URL. Override with --build-arg REPO_URL=...
# Not set as a runtime ENV in the finished image.
ARG REPO_URL=https://github.com/example/your-samples.git
 
# Full commit hash to fetch. Same SHA -> Docker can reuse this layer.
# Bump SHA (or pass --build-arg REPO_SHA=...) when you want a new tree.
ARG REPO_SHA=0123456789abcdef0123456789abcdef01234567
 
# One RUN = one image layer. Steps (do not put # comments between \ continuations):
#   mkdir -p          create /opt/base/repo
#   git … init        empty repo at that path (-C = run as if cwd were that path)
#   remote add origin attach REPO_URL (no download yet)
#   fetch --depth 1   download only that commit (shallow), not full history
#   checkout FETCH_HEAD  make the working tree match the commit just fetched
RUN mkdir -p /opt/base/repo \
    && git -C /opt/base/repo init \
    && git -C /opt/base/repo remote add origin "${REPO_URL}" \
    && git -C /opt/base/repo fetch --depth 1 origin "${REPO_SHA}" \
    && git -C /opt/base/repo checkout FETCH_HEAD

What each RUN step does:

• mkdir -p /opt/base/repo - create the directory that will hold the checkout.
• git -C … init - create an empty Git repo there (-C runs the command as if that path were the working directory).
• remote add origin - attach REPO_URL; nothing is downloaded yet.
• fetch --depth 1 origin "${REPO_SHA}" - download only that commit (shallow), not full history and not whatever main is today.
• checkout FETCH_HEAD - make the working tree match the commit just fetched.

Pinning the SHA is what makes Docker layer cache useful on purpose:

• Same REPO_SHA → Docker can reuse the clone layer.
• Bump REPO_SHA (or pass a new --build-arg REPO_SHA=…) → that layer rebuilds and fetches the new commit.

Notes from the field:

• Prefer --depth 1 on the fetch for speed; always pin a full commit SHA in the ARG.
• Do not bake API keys into the base image.
• Sibling tasks still own their own WORKDIR, output dirs, and any extra layers - share common setup in the base, not one-off task details.
• Host checkout + COPY is another fine option when you want cache invalidation to follow files on disk instead of a build-arg.

Sibling task images

After the base exists, sibling Dockerfiles start with a generated header:

# Start from the shared local base image (built in the previous section)
FROM codesloth-harbor-base:local

Then a short task-specific block (link the placeholder repo, ensure /app/output, and stop). In the samples, sibling-task-a and sibling-task-b share that base and diverge only on instruction + verifier intent:

• A expects Intent identified: reference-answer and mentions of task.toml / instruction.md / environment/ / tests/.
• B expects Intent identified: structured-calculator and result: 56 for 7 * 8.

That is the DRY win: environment plumbing is shared; the contract under test stays per-task. Harbor still builds each task’s environment/ as usual - you just made those builds cheap.

Volume mounts for skills

Baking a skill into every image layer makes rebuilds slow when the skill changes. Skills change often; image layers do not.

Harbor’s Docker environment can use Compose when you need mounts or sidecars. The advanced sample’s docker-compose.yaml is a commented template around a main service:

volumes:
  # Example: mount a local skill checkout read-only into the sandbox
  # path-on-host:path-in-container:ro  (:ro = read-only)
  # - ../../team-brain-intro/.cursor/skills/team-brain-intro:/opt/skills/team-brain-intro:ro
  # Example: mount task fixtures from the host for faster iteration
  # - ./sibling-task-a/fixtures:/app/fixtures:ro

Use mounts when:

• The skill tree is large or private.
• You iterate on skill markdown between trials and refuse to rebuild the image each time.
• Fixtures or corpora live on the host and should stay out of the Dockerfile.

Mount the skill, not the answer sheet. If you bind-mount a whole samples repo (or a task folder that also holds fixtures/, tests/, and expected intent strings), the agent can read the grading material and pass without earning it. Prefer one of these:

• Mount only the skill directory (for example .cursor/skills/team-brain-intro), not the Harbor task tree beside it.
• Bake or COPY only that skill subtree into the image.
• Keep pass/fail fixtures and verifier asserts off paths the agent can list or open.

Host-side fixture mounts for your iteration are fine when the agent never sees them - for example verifier-only paths, or files copied in after the agent step. Same rule as neutral instruction.md in Part 3 (opens in a new tab): do not leak the grading contract into the agent’s filesystem.

For agent-native skill injection at run time (Harbor uploads skill dirs into the agent’s skills location), use harbor run --skill in Part 5 (opens in a new tab). Mounts stay useful when you need a custom path, a whole skills tree for skill-calls-skill layouts, or non-skill files the instruction refers to.

Pass secrets through the host environment at run time. Do not commit key values into Compose, Dockerfiles, or task.toml. The Harbor MCP / Compose tutorial (opens in a new tab) is the official companion for multi-container shapes; cloud sandboxes often require a single Dockerfile, so keep Compose for local Docker when that is your constraint.

When to leave harbor run for orchestration

harbor run owns one trial (or a batch Harbor already knows how to schedule): sandbox → agent → verifier → reward. Treat that one trial as the unit Harbor already owns.

Move outside Harbor when you need portfolio concerns:

Harbor owns each task’s sandbox contract. You own portfolio orchestration - sweeping tasks and models, building shared images, mounts, and post-processing. Part 5 (opens in a new tab)’s run script is that orchestration - desired state first, then harbor run.

Deduplicate with generate / copy scripts

Once you have two siblings, shared timeouts and FROM lines drift. A small script beats tribal memory.

advanced/scripts/generate_common.sh copies:

• scripts/templates/task.toml.common → each sibling’s task.toml (shared verifier/agent/environment defaults)
• scripts/templates/Dockerfile.from-base.snippet → each sibling’s environment/Dockerfile header

It preserves anything under a # --- task-specific … marker so metadata and layers stay local. Run it after you change shared defaults:

# Copy shared task.toml / Dockerfile headers into each sibling task
./scripts/generate_common.sh

Then rebuild the base tag before you expect Harbor to build siblings cleanly.

You can grow this into full task generation (instruction + tests from a YAML matrix). Start with the boring copy-stamp. Boring is how suites stay honest.

Harbor task shapes and DRY in one map

# Where simple vs advanced Harbor samples live in the repo:
harbor-evals/
  simple/      # Parts 2-3 (+ Part 5 runs) - cheap simple evals
  advanced/    # Part 4 - base image, siblings, mounts, generator

If you are still writing your first simple eval, stay in simple/ until the verifier is boring and passes under the Oracle (or a fixture walk). Only then copy the advanced patterns. Premature DRY is how you invent a framework instead of an eval suite.

Official task structure: harborframework.com/docs/tasks (opens in a new tab). Series companions: Part 2 scaffold (opens in a new tab), Part 5 runs (opens in a new tab), Part 6 EDD (opens in a new tab).

Sloth Summary

• Keep simple evals for cheap contracts; reserve complex flows for shared trees, mounts, and heavier fixtures.
• Build a SHA-pinned shallow-clone (or host COPY) base image once; let sibling Dockerfiles FROM it.
• Mount skill checkouts and large fixtures when they change faster than images should - mount only the skill tree, never pass/fail fixtures the agent can read.
• At run time, prefer harbor run --skill when you only need Harbor to inject skill dirs (see Part 5).
• Leave harbor run for the per-task sandbox; put matrices, aggregation, and cross-repo prep in your scripts.
• Deduplicate task.toml / Dockerfile headers with a generate/copy script that preserves task-specific markers.
• Follow harbor-evals/advanced/ (opens in a new tab) as the pattern kit.

Next we actually run jobs, mind the API bill, and open harbor view.

May your shared base stay shared and your sibling contracts stay distinct. 🦥