Everything the v0 engine does today. It is the same engine in the browser runner and in the local CLI. Anything not on this page is not built yet.
SuiteTraceImportTemplatesCheck typesScoringBatchBaselineExport & shareCLI
A suite is a JSON object with a name, an optional version and a checks array. Every check has a type plus the fields for that type. These fields are optional on every check:
id - a stable name, used in reports, JUnit and baseline diffs. Keep it unique; the lint flags duplicates.description - human text shown in the results table.weight - a non-negative number, default 1.critical - true means a failure blocks the run whatever the score.target (content checks) - final_output (default), all_outputs (every step output and result plus the final output), or step:<name>.{
"name": "support-agent ship-readiness",
"version": "0.1.0",
"checks": [
{ "id": "answer-present", "type": "final_answer_present", "critical": true, "weight": 2 },
{ "id": "no-refund", "type": "tool_not_called", "tool": "issue_refund", "critical": true }
]
}
One recorded agent run. You record it yourself today; SDK auto-capture is not built.
{
"trace_id": "run-0001",
"metadata": { "total_tokens": 4210, "latency_ms": 3800, "cost_usd": 0.021 },
"steps": [
{ "type": "llm", "name": "plan", "output": "..." },
{ "type": "tool", "name": "order_lookup", "args": { "order_id": "A-7731" }, "result": "...", "error": null }
],
"final_output": "..."
}
A tool step counts as errored when it has a truthy error or status: "error". A budget metric that is missing from metadata makes its budget check UNKNOWN (see scoring).
Paste or drop a run recorded in one of these formats. It is converted to a HarnessKit trace in your browser (or by the CLI), and the page says which format it detected.
messages array whose content blocks include tool_use / tool_result. Results are paired by id, is_error becomes a tool error, usage tokens are summed, and thinking blocks are dropped.tool_calls and role: "tool" results. Arguments are parsed from their JSON string, and a top-level usage.total_tokens is read.resourceSpans) or a span array with gen_ai.* attributes. Tool spans become tool steps, error status becomes a tool error, and latency is taken from the span bounds.Metrics a format does not record (often cost, sometimes tokens or latency) stay missing, so the matching budget checks are UNKNOWN rather than a silent pass. A JSON array of runs in any mix of formats is a batch.
Suggest checks from trace drafts a suite from the loaded trace (or batch). It includes tools every trace called, their shared order, error-free tools, JSON shape when the output is JSON, and step, tool-call, token, latency and cost budgets at 1.5× the worst observed value, for metrics the trace actually records. Every suggested check passes on the source trace; tighten it from there.
Pick one from the Template menu on the runner. Each one comes with a sample trace that grades SHIP, and each one blocks an empty run (both are tested). Copy one and edit it for your agent.
[doc-3], no open-web fallback, a retrieval-call budget.final_answer_presentThe final output is non-empty.
{ "type": "final_answer_present" }
contains / not_containsA substring must (not) appear in the target.
{ "type": "not_contains", "target": "final_output", "value": "grounding_check" }
contains_any / contains_allAt least one, or every one, of values appears.
{ "type": "contains_all", "values": ["A-7731", "$84.20"] }
regex / not_regexA JavaScript regular expression, with optional flags, must (not) match. An invalid pattern is an ERROR.
{ "type": "not_regex", "target": "all_outputs", "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b" }
json_validThe target parses as JSON. One surrounding ```json fence is allowed.
{ "type": "json_valid" }
json_has_keysThe target is JSON and every dotted path in keys exists as an own property. A null value counts as present.
{ "type": "json_has_keys", "keys": ["order.id", "status"] }
tool_called / tool_not_calledA tool step named tool did (not) happen.
{ "type": "tool_called", "tool": "kb_search" }
tool_sequenceThe tools in sequence were called in this order. Other calls in between are allowed.
{ "type": "tool_sequence", "sequence": ["kb_search", "order_lookup"] }
tool_arg_equalsSome call of tool had args[arg] equal to value (compared as strings).
{ "type": "tool_arg_equals", "tool": "order_lookup", "arg": "order_id", "value": "A-7731" }
no_tool_errorsNo tool step reported an error.
{ "type": "no_tool_errors" }
max_stepsThe number of steps is at most value.
{ "type": "max_steps", "value": 8 }
max_tool_callsThe number of tool calls, or calls to one tool, is at most value.
{ "type": "max_tool_calls", "tool": "web_search", "value": 3 }
max_tokens / max_latency_ms / max_cost_usdCompared with metadata.total_tokens, metadata.latency_ms or metadata.cost_usd. A missing metric gives UNKNOWN, not a pass.
{ "type": "max_cost_usd", "value": 0.05 }
Paste a JSON array or JSONL of traces, or drop a .jsonl file. You get per-trace grades, the mean score, the worst trace and, for each check, how many traces failed it. The batch grade is the worst trace grade, so one BLOCK blocks the batch.
Load an exported Report JSON, or click Set as baseline after a run. The next run lists checks that regressed (pass rate dropped, critical ones first), checks that were fixed, checks that became unknown, added and removed checks, and the score delta. A grade drop counts as a regression too.
harnesskit.report/v1) is the whole result. It also works as a baseline.<failure>, ERROR becomes <error>, UNKNOWN becomes <skipped>, and grade and score go in suite properties. A batch has one testsuite per trace.#, which browsers never send to a server. Anyone with the link can read them. The result is recomputed on open, never read from the link.The same engine in Node 18+, with no dependencies. It runs on your machine or in your own CI job. Hosted CI gating is not built.
node cli.js suite.json trace.json|traces.jsonl \ [--fail-on block|review] [--baseline base.json [--fail-on-regression]] \ [--junit junit.xml] [--json report.json] [--quiet]
Exit codes: 0 = gate passed, 1 = gate failed, 2 = usage or input error. Lint messages go to stderr.