Install Harbor and Scaffold Tasks with harbor tasks init

Part 1 (opens in a new tab) covered why we want evals. In this post we install Harbor, look at the CLI, and scaffold a task so you are not inventing directory names from memory.

If you are joining mid-series, start with Harbor Evals for Agentic Skills Part 1: What Are Evals? (opens in a new tab) or the series overview (opens in a new tab). Writing the actual instruction and verifier content is Part 3 - Writing Tasks (opens in a new tab). Official docs live at harborframework.com/docs (opens in a new tab); the code is at github.com/laude-institute/harbor (opens in a new tab).

Install Harbor

Harbor publishes a CLI you can install with uv (opens in a new tab). uv is Astral’s fast Python package and project manager - think of it as a modern replacement for common pip / venv / tool-install workflows, with a dedicated uv tool command for putting CLIs on your PATH.

# Install Harbor as a standalone CLI on your PATH (via uv's tool installer)
uv tool install harbor

That matches the Getting Started (opens in a new tab) guide. Prefer uv over a one-off pip install into a random environment - tool installs stay on your PATH and are easier to upgrade.

Nightly / pre-release builds exist if you need bleeding-edge main, but most skill-eval work is fine on the stable tool install. Upgrade later with uv tool upgrade harbor when you want it.

Confirm the binary:

# Print Harbor's top-level commands and usage
harbor --help

You should see top-level commands such as run, view, task / tasks, dataset, job, init, and friends. You do not need every subcommand on day one - tasks init, run, and (later) view carry most of this series.

Scaffold a task with `harbor tasks init`

Harbor’s task format is a directory. The fastest way to get the expected files is:

# Create a new task folder with the standard Harbor layout
# "codesloth/team-brain-intent" is the task name (dataset/task path style)
harbor tasks init "codesloth/team-brain-intent"

Useful flags when you grow past the defaults:

Flag	Why you might use it
`--tasks-dir` / `-p`	Drop the task under a dataset folder instead of `.`
`--metadata-template`	Pre-fill `task.toml` from a shared TOML template
`--no-pytest`	Skip the pytest template if you are not using it
`--no-solution`	Skip `solution/` when you are not wiring an Oracle yet
`--steps N`	Scaffold a multi-step task

The task tutorial (opens in a new tab) walks the same scaffold with a sample name like ssh-key-pair.

Default folder structure

A fresh task looks like this:

# Default Harbor task layout after `harbor tasks init` (comments on each role):
team-brain-intent/
├── instruction.md         # What the agent must do
├── task.toml              # Config + metadata
├── environment/
│   └── Dockerfile         # Container definition (common default)
├── solution/
│   └── solve.sh           # Known-good Oracle solution
└── tests/
    ├── test.sh            # Verifier entrypoint (writes reward)
    └── test_outputs.py    # Pytest checks (when pytest template is kept)

That layout is the Harbor task format in miniature:

instruction.md - agent-facing instructions (markdown, not buried in YAML)
task.toml - timeouts, metadata, environment knobs, optional network policy
environment/ - how the container is built or selected
solution/ - optional Oracle script so you can prove the task is solvable
tests/ - verifier scripts that must emit a reward under /logs/verifier/

We fill these files in Part 3 - Writing Tasks (opens in a new tab). For now, treat the scaffold as a checklist of surfaces you must eventually own.

Your agent can scaffold without `init`

You do not have to run harbor tasks init yourself. Harbor ships a create-task skill for coding agents:

# Download Harbor's create-task skill into your agent's skills folder
# npx runs the skills CLI without a global install
# --skill create-task picks that skill from the harbor-framework/harbor package
npx skills add harbor-framework/harbor --skill create-task

Point your agent at the task structure docs (opens in a new tab) and ask it to create a task directory in the same shape. That is handy when the task is being generated as part of an EDD loop (Part 6) - the agent writes the folder tree as part of the workflow instead of waiting for you to type init.

Two caveats:

Agent-scaffolded tasks still need to match Harbor’s expectations (tests/test.sh writing a reward, environment files Harbor can start, and so on).
init remains the fastest way for you to see the canonical template when you are learning.

Environment expectations (Dockerfile and Compose)

Harbor keeps environment definitions under environment/. For local Docker (--env docker), any of these can work:

environment/Dockerfile
environment/docker-compose.yaml
[environment].docker_image in task.toml (pre-built image; Dockerfile optional)

High-level expectations for skill evals:

Keep the image small and purposeful. Install only what the agent needs to perform the task and what the verifier needs to grade it (or use a separate verifier environment later).
Prefer a Dockerfile for portability. Many cloud sandbox providers support Dockerfile-defined environments and not Compose.
Compose is fine locally when you need sidecar services - just know you may need a different shape to scale out.
WORKDIR matters. Tutorial tasks often use /app; tests and solution trees are copied to /tests and /solution at runtime, and verifier logs go under /logs/verifier/.
OS is configurable. [environment].os defaults to "linux"; Windows containers are a separate path if you need them.

You can also omit both Dockerfile and Compose and drop other files into environment/ - Harbor uploads them into the container workdir on start. That is a niche escape hatch, not the default for this series.

Where the CodeSloth samples live

The companion repo is CodeSloth Cursor Samples (opens in a new tab). For this series, start under:

harbor-evals/simple/ (opens in a new tab)

That folder is where Parts 2-3 put simple Team Brain eval tasks (intent announcement, calculator path, and similar). Advanced shapes (shared base images, sibling tasks, generators) move to harbor-evals/advanced/ in Part 4.

Quick post-install checklist

After install + init, confirm you are unblocked:

harbor --help prints without errors.
harbor tasks init … created the five surfaces listed above.
You know where instruction.md and tests/test.sh live.
You have a samples path (harbor-evals/simple/) ready for Part 3 content.

Running agents and inspecting harbor view waits for Part 5 - no need to spend API credits just to look at an empty Dockerfile.

Sloth Summary

What you should have after this post:

Install with uv tool install harbor, then learn the map via harbor --help.
Scaffold with harbor tasks init to get instruction.md, task.toml, environment/, solution/, and tests/.
Agents can create the same tree via Harbor’s create-task skill - useful once EDD starts generating evals for you.
Docker environments usually mean a Dockerfile under environment/; Compose and pre-built images are supported with tradeoffs.
Follow along in harbor-evals/simple/ (opens in a new tab).

Next: Part 3 - Writing Tasks (opens in a new tab) - instructions and verifiers that actually pass and fail on purpose.

Happy scaffolding - may your task.toml stay boring and correct. 🦥