It passed the test. Would you ship it?

Picture a small bug fix that earns a perfect score. The patch compiles, the tests pass, and the assistant reports success. Then a customer follows a slightly different path through the application and encounters the same problem.

This imagined failure is familiar because software quality has always depended on what gets checked. AI adds another complication: the system producing the work may also help decide which evidence counts.

An evaluation is a structured way to judge performance. It can be as simple as a collection of tasks with clear success conditions. The challenge is making sure those conditions still represent the work the business needs done.

A score is a set of choices

A coding assistant might be judged on whether a patch works, how long it takes, how much it costs, and how often a person has to intervene. Those measures can disagree. A solution may be correct but require extensive review. Another may arrive quickly while introducing maintenance work elsewhere.

Combining those outcomes into one number can make comparison easier while hiding the tradeoffs. A manager needs enough detail to understand the result: what was completed, what was checked, what remained uncertain, and what assistance was required.

The test environment matters too. An assistant working in an empty project faces a different job from one entering a mature application with incomplete documentation and competing requirements. Both can be useful experiments. They answer different questions.

Keep some surprises in reserve

If researchers repeatedly tune a system against the same tasks, the result may reflect familiarity with those tasks. A separate set of unseen work helps test whether the improvement travels beyond the examples used during development.

That work should include ordinary complications: a change spanning several files, a review comment that requires a new approach, or a failure in an external tool. The aim is to represent the conditions the factory would actually encounter.

Repeated runs also matter. When a system can produce different answers to the same request, a single success says little about how reliably it will behave. A useful report shows variation and explains the conditions under which the work was attempted.

Confidence in a release depends on the quality of the questions asked before it.

Inspect the evidence

A practical evaluation proposal would preserve the original task, the exact code version, the resulting patch, the checks performed, the cost, and the human interventions. Reviewers could then examine the work independently of the assistant’s account.

A failed task should remain visible. So should a task that passed only after a person corrected the plan. These details are part of the result, particularly when the research question concerns autonomy.

We are interested in evaluations that help people decide what responsibility a factory can reasonably take on. That requires a closer look than a leaderboard can provide.