Hive Commons · Agent benchmarks

Outcome, cost and quality of coding agents, models and sandbox runtimes, measured on real pull requests by Hive Commons and Project Bluefin hives.

loading data/index.json…

Leaderboard by model

By agent harness

By sandbox runtime

Contributing hives

Hives are identified by a stable anonymous hash (shown to each owner in their dashboard under Settings → Benchmarks). Repository names appear only for public repositories.

Methodology

How these numbers are computed
  • Unit. One record = one unit of agent work on one pull request, exported from a hive's signed per-PR evidence bundle: agent-authored (the agent opened the PR) or agent review (the swarm reviewed someone's PR; the model shown is the majority reviewer model). Schema: schema/result.schema.json.
  • Backfill. Records with exporter: backfill-from-cluster were converted from existing evidence by scripts/convert-evidence.py; sandbox_runtime: none because those hives ran agents in-process. task.kind is unknown unless the agent name says otherwise.
  • Median time to merge is read from a 12-bucket log-scale histogram, so it is a bucket midpoint, not an exact value.
  • Merge rate = merged ÷ records with pr_opened. CI green first try = ci_green_first_try true ÷ records where CI was captured (nulls excluded). Cost / merged PR = Σ cost.usd_est over merged PRs in the group ÷ merged PRs with a cost estimate; cost.basis = unknown records are excluded from the numerator and the denominator. The small "attr." percentage says how many of the group's records have a PR-attributed cost rather than a fleet-median imputation.
  • Intervals. Every rate shows n and a 95% Wilson score interval. Groups with n < 10 are shown as not reported, the same rule hivecommons/hive uses for its monthly outcome benchmark.
  • Cost basis — every dollar figure is an estimate, not a bill. Each record carries cost.usd_est and cost.basis:
    • reported — the harness or gateway returned a billed figure (OpenRouter/LiteLLM). None yet.
    • tokens_x_pricing — hive's token collector counts input/output/cache tokens per agent kick and prices them with the same list-price table its dashboard Cost panel uses (schema/pricing.json, mirrored from hive/src/pkg/tokens/pricing.go). Spend is attributed to a PR through hive's launch attribution (/api/launch-cost): the kick that named the issue(s) the PR closed, or the PR itself for review-swarm kicks; a kick naming several refs is split evenly. Subscription plans (Claude Max, Copilot premium requests) and self-hosted inference do not bill at list price, so this is an order-of-magnitude proxy.
    • fleet_model_median — the PR has no attributed kick (older than the attribution store, kicked without a ref, or usage outside the 6-hour ownership horizon). It gets the median tokens_x_pricing cost of attributed PRs for the same model and role across the whole fleet (≥5 samples), else the fleet's median token mix for the role priced at that model's list price. Imputed at index-build time; the raw record stays unknown.
    • unknown — nothing available; excluded from all cost aggregates. Tiles say "n=… with cost data".
    Why not duration × $/h? wall_clock_s is PR open→merge time, not agent runtime, so it would mostly measure review latency.
  • Synthetic records (synthetic: true) are excluded unless you tick the box. They exist so the page renders before real exports land.
  • Privacy. Every hive publishes anonymized records by default (opt-out: benchmarks.publish: false). Repository and org names appear only for public repositories (repo_consent); private repos stay anonymous. No logins, prompts, finding text or code are published. See GOVERNANCE.md.
  • Confounders. Models are not assigned to tasks at random: harder tasks go to bigger models, review agents never open PRs, and hives differ in ACMM policy. Filter by task kind and ACMM level before comparing; treat cross-model differences as observational.