v0 PREVIEW

HarnessKit - the eval harness for AI agents

Define an eval suite. Run it against a recorded agent trace. Get a ship-readiness score you can argue with. Every team shipping agents rebuilds this from scratch; HarnessKit is the kit so you don't.

Try the runner now

This v0 runs entirely in your browser against recorded traces. Nothing leaves the page. A sample suite and trace are loaded - hit Run evals. Pick another starter template (support agent, RAG answerer, coding agent, structured extractor), or paste, load or drop your own.

Eval suite (JSON)

or drop a .json file on the box. It is read locally and never uploaded.

Agent trace (JSON, or an array / JSONL of traces for a batch run)

or drop a .json file on the box. It is read locally and never uploaded.

Also accepts Anthropic Messages transcripts, OpenAI Chat Completions messages and OpenTelemetry GenAI spans (OTLP JSON). They are converted to a HarnessKit trace in the box.

Baseline: none - load an exported Report JSON, or use "Set as baseline" after a run.

A share link contains your full suite and trace, so anyone who has it can read them. The data sits after the #, which browsers never send to a server, so it is not logged or sent to analytics.

-
Export:JUnit can be uploaded to most CI test-report viewers.
CheckStatusDetail

Check types in v0

Content checks

  • final_answer_present - the agent actually answered
  • contains / not_contains - string must (not) appear in an output
  • regex / not_regex - pattern guards, e.g. no SSN-shaped strings in any output
  • contains_any / contains_all - one of, or every one of, a list of values
  • json_valid / json_has_keys - structured output parses (one ```json fence allowed) and has the required keys (dotted paths like order.id)

Behavior checks

  • tool_called / tool_not_called - required and forbidden tools
  • tool_sequence - calls happened in the required order
  • tool_arg_equals - a call used the exact argument, e.g. the right order ID
  • no_tool_errors - no tool step has an error field or status: "error"

Budget checks

  • max_steps, max_tool_calls (optionally per tool), max_tokens, max_latency_ms, max_cost_usd

Scoring

  • Every check carries a weight; the score is weighted pass rate.
  • Checks marked critical fail the run outright (grade BLOCK) no matter the score.
  • Grades: SHIP (≥90%), REVIEW (70-89%), BLOCK (<70% or a critical failure).
  • A budget check whose metric is missing from the trace is UNKNOWN, not a pass: it is left out of the score and caps the grade at REVIEW.
  • A malformed check (missing field, invalid regex) is an ERROR and counts as a failure. Every run lints the suite first and lists problems by check id.

What v0 is - and what it is not

Working today

  • The eval engine on this page: suites, 19 check types, weighted scoring, SHIP / REVIEW / BLOCK grades.
  • Imports runs recorded as Anthropic Messages transcripts, OpenAI Chat Completions messages or OpenTelemetry GenAI spans, converting them in your browser.
  • Runs on any recorded trace that follows the schema below. Paste yours and run it. Paste or drop an array or a JSONL file of traces to run a batch: you get per-trace grades, the mean score, the worst trace and failure counts per check. The batch grade is the worst trace grade.
  • Export each run as a JSON report or as JUnit XML for a CI test-report viewer. The file is generated in your browser.
  • Compare against a baseline report: checks that regressed (critical ones first), checks that were fixed, checks that became unknown, and the score delta.

Not built yet (no fake claims)

  • Hosted runner - executing suites against live agents and models on our infra. This is the paid tier we are building toward; it does not exist yet.
  • Hosted CI gating - a GitHub app that fails a pull request when an eval regresses. Planned. Today you can run the same engine yourself in your own CI job with the local CLI (node cli.js suite.json trace.json, see the repo README).
  • SDKs and auto-capture - today you record traces yourself. The open-source HarnessKit runtime (runner, SDKs, connectors) is scaffolding in a separate repo and is not wired to this page.
  • Accounts, teams, dashboards - not in v0.

Trace schema

A trace is one JSON object: metadata (total_tokens, latency_ms, cost_usd), a steps array of {type: "llm"|"tool", name, output|result, args}, and final_output. The loaded sample is a complete example.

Where it fits

HarnessKit scores a recorded run against your expectations. It pairs with replay tools rather than replacing them: ConeReplay replays a trace through a causal cone to predict what a model swap breaks; HarnessKit then grades the replayed runs against your suite. Replay with ConeReplay, score with HarnessKit.

The category (LangSmith, Braintrust, Langfuse) is forming now and the comparison guides read like a menu. HarnessKit's bet: the harness itself should be a kit you install in an afternoon - open formats, no seat pricing, runner as the paid tier.