v0 PREVIEW

HarnessKit docs

Everything the v0 engine does today. It is the same engine in the browser runner and in the local CLI. Anything not on this page is not built yet.

SuiteTraceImportTemplatesCheck typesScoringBatchBaselineExport & shareCLI

Suite

A suite is a JSON object with a name, an optional version and a checks array. Every check has a type plus the fields for that type. These fields are optional on every check:

{
  "name": "support-agent ship-readiness",
  "version": "0.1.0",
  "checks": [
    { "id": "answer-present", "type": "final_answer_present", "critical": true, "weight": 2 },
    { "id": "no-refund", "type": "tool_not_called", "tool": "issue_refund", "critical": true }
  ]
}

Trace

One recorded agent run. You record it yourself today; SDK auto-capture is not built.

{
  "trace_id": "run-0001",
  "metadata": { "total_tokens": 4210, "latency_ms": 3800, "cost_usd": 0.021 },
  "steps": [
    { "type": "llm",  "name": "plan", "output": "..." },
    { "type": "tool", "name": "order_lookup", "args": { "order_id": "A-7731" }, "result": "...", "error": null }
  ],
  "final_output": "..."
}

A tool step counts as errored when it has a truthy error or status: "error". A budget metric that is missing from metadata makes its budget check UNKNOWN (see scoring).

Importing runs from other formats

Paste or drop a run recorded in one of these formats. It is converted to a HarnessKit trace in your browser (or by the CLI), and the page says which format it detected.

Metrics a format does not record (often cost, sometimes tokens or latency) stay missing, so the matching budget checks are UNKNOWN rather than a silent pass. A JSON array of runs in any mix of formats is a batch.

Starter templates

Suggest checks from trace drafts a suite from the loaded trace (or batch). It includes tools every trace called, their shared order, error-free tools, JSON shape when the output is JSON, and step, tool-call, token, latency and cost budgets at 1.5× the worst observed value, for metrics the trace actually records. Every suggested check passes on the source trace; tighten it from there.

Pick one from the Template menu on the runner. Each one comes with a sample trace that grades SHIP, and each one blocks an empty run (both are tested). Copy one and edit it for your agent.

Check types (19)

Content

final_answer_present

The final output is non-empty.

{ "type": "final_answer_present" }

contains / not_contains

A substring must (not) appear in the target.

{ "type": "not_contains", "target": "final_output", "value": "grounding_check" }

contains_any / contains_all

At least one, or every one, of values appears.

{ "type": "contains_all", "values": ["A-7731", "$84.20"] }

regex / not_regex

A JavaScript regular expression, with optional flags, must (not) match. An invalid pattern is an ERROR.

{ "type": "not_regex", "target": "all_outputs", "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b" }

Structured output

json_valid

The target parses as JSON. One surrounding ```json fence is allowed.

{ "type": "json_valid" }

json_has_keys

The target is JSON and every dotted path in keys exists as an own property. A null value counts as present.

{ "type": "json_has_keys", "keys": ["order.id", "status"] }

Behavior

tool_called / tool_not_called

A tool step named tool did (not) happen.

{ "type": "tool_called", "tool": "kb_search" }

tool_sequence

The tools in sequence were called in this order. Other calls in between are allowed.

{ "type": "tool_sequence", "sequence": ["kb_search", "order_lookup"] }

tool_arg_equals

Some call of tool had args[arg] equal to value (compared as strings).

{ "type": "tool_arg_equals", "tool": "order_lookup", "arg": "order_id", "value": "A-7731" }

no_tool_errors

No tool step reported an error.

{ "type": "no_tool_errors" }

Budgets

max_steps

The number of steps is at most value.

{ "type": "max_steps", "value": 8 }

max_tool_calls

The number of tool calls, or calls to one tool, is at most value.

{ "type": "max_tool_calls", "tool": "web_search", "value": 3 }

max_tokens / max_latency_ms / max_cost_usd

Compared with metadata.total_tokens, metadata.latency_ms or metadata.cost_usd. A missing metric gives UNKNOWN, not a pass.

{ "type": "max_cost_usd", "value": 0.05 }

Scoring and grades

Batch runs

Paste a JSON array or JSONL of traces, or drop a .jsonl file. You get per-trace grades, the mean score, the worst trace and, for each check, how many traces failed it. The batch grade is the worst trace grade, so one BLOCK blocks the batch.

Baseline diff

Load an exported Report JSON, or click Set as baseline after a run. The next run lists checks that regressed (pass rate dropped, critical ones first), checks that were fixed, checks that became unknown, added and removed checks, and the score delta. A grade drop counts as a regression too.

Export and share

Local CLI gate

The same engine in Node 18+, with no dependencies. It runs on your machine or in your own CI job. Hosted CI gating is not built.

node cli.js suite.json trace.json|traces.jsonl \
  [--fail-on block|review] [--baseline base.json [--fail-on-regression]] \
  [--junit junit.xml] [--json report.json] [--quiet]

Exit codes: 0 = gate passed, 1 = gate failed, 2 = usage or input error. Lint messages go to stderr.