Original report (sample): How to read an agent benchmark

Sample original report on how this publication will read agent leaderboards: task match, contamination, cost, and human review — with pointers to public evals, not invented scores.

Byline:: AI Agents Info Research
Published:: October 25, 2026
Last updated:: October 31, 2026
Paper:: How to read an agent benchmark (sample editorial report) — AI Agents Info Research (2026)

Abstract / summary

Sample original report: leaderboards for language-model agents are easy to misread. This note lists checks this desk will use — task match, contamination risk, hidden cost, human baselines, and harness identity — and points at public eval projects without quoting unpublished or invented numbers.

Key findings

A benchmark is a frozen task suite plus a scoring rule. It is not the job you run in production.
The harness (tools, retries, browsing, extra models) is part of the system under test. Naming only the base model is incomplete.
Cost, latency, and human escalation rates belong next to accuracy. They are often missing from screenshots of leaderboards.

Methodology

This is a methods note, not an experiment. It does not run models. It proposes a reading checklist and cites public evaluation projects so future tracker pages have a documented standard.

Limitations

Sample editorial content. No new empirical results.
Public benchmarks change; always read the current dataset card.
We do not rank vendors here and we do not invent scores.

Sample original report. Not a new scientific result. Use it as the skeleton for future desk notes. Do not cite it as evidence of model quality.

The checklist

When a chart claims an agent is “state of the art,” this desk will ask:

Task match. Is the suite about tickets, browsers, shopping, or exams? SWE-bench is not GAIA; neither is your internal workflow.
System identity. Which harness, tools, retries, and time budget? A model name is not a system.
Contamination and leakage. Was the task in training data or in previous traces? If nobody says, assume uncertainty.
Human baseline. Compared with whom, with what time limit?
Cost and failure mode. Dollars, tokens, wall time, and how it fails (refuses, loops, silently wrong).
Date. Leaderboards move. We will timestamp every number we eventually publish on the benchmark tracker.

Public starting points (not scores)

SWE-bench — software engineering tasks.
GAIA — assistant-style questions that often need tools.
HELM — broader language-model evaluation, not agent-only.

We are not copying tables of numbers into this sample. When the tracker leaves stub status, each cell will name a source and a retrieval date.

How this feeds the rest of the site

ReAct breakdown for a historical prompting pattern.
How agent harnesses work for the runtime around the eval.
Glossary: benchmark, agent.