Agent evaluation needs an artifact, an environment, and a failure budget

An AI agent should earn access to a production workflow through evidence about its output, execution conditions, and cost of failure. A convincing final answer cannot establish that it followed the right instructions, respected constraints, or produced a usable artifact. Four October publications support a more useful evaluation design than a model leaderboard alone.

Docker announced Harbor integration with Sandboxes on October 14, centering isolated execution and network controls, with a stable release planned for October 23. Expo's October 8 account of evaluating skills separates finding guidance from using it. NVIDIA's October 14 Siemens example emphasizes verification of engineering artifacts. The same day's OpenAI account of Third Moment Research describes trajectory analysis and an outer harness deciding whether unfinished work needs retrying or human review. These are the publishers' accounts, rather than tests conducted by this studio.

An identical bad output can hide different failures

An incorrect report might mean the agent never found its instructions, misapplied instructions it did find, or faithfully followed a procedure against stale inputs. A single final score merges these causes. Changing the model under those conditions becomes an expensive guessing exercise.

The useful connection between the four accounts is a sequence of evidence. Freeze inputs and execution conditions, observe the selected path, then independently verify the artifact. Instruction discovery becomes measurable. Isolation reduces accidental help from a previous run or an unrestricted network. Artifact checks establish whether the work actually satisfies the intended business requirement.

The evaluator also needs a different trust boundary from the worker. If the worker can rewrite its checker or reference data, a passing result ceases to be independent evidence. Equally, disabling every network connection can create an unrealistic assessment for a job that depends on a current external API. Match restrictions to the intended deployment. Recorded service responses suit reproducible regression checks; a separate live integration exercise can assess real dependencies.

This distinction affects procurement. A platform demonstrating a high score on its own tasks has shown something narrower than suitability for your process. Ask for the input distribution, allowed tools, stopping rules, and artifact criteria before comparing percentages. A demanding checker that rejects a plausible but incomplete deliverable can be more valuable than a friendly score.

Retry policy belongs inside the evaluation

A success threshold has little meaning without an attempt budget. Success on attempt ten establishes that a solution exists, while leaving its economic viability unanswered. Record tool charges, waiting time, retries, and human review alongside inference consumption. Waiting deserves particular attention because an agent can repeatedly ask for status while an external computation continues without needing its involvement.

Our engineering inference from these sources is to remove useless state transitions before shopping for a cheaper model. Represent a long operation as a job with an identifier and a completion signal. Distinguish a temporary dependency failure from invalid input or a terminal rejection before retrying. Otherwise, an evaluation rewards persistence that the business later pays for.

A fixed deadline matters as well. Some tasks can wait overnight; an interactive support process cannot. Report successful outcomes within the agreed window, and retain the unfinished runs in the denominator. Excluding them would quietly change the question being answered.

Hypothetical example: extracting order lines

Consider a proposed document workflow, not a delivered client result. An agent extracts purchase-order lines from PDFs into a defined JSON structure. The evaluation includes a clear digital document, a scan, a table crossing a page boundary, and a document whose total contradicts its lines. Checks cover required fields, types, arithmetic, and the page supporting each extracted item. Ambiguity has an explicit manual-review outcome.

The first run fixes the documents and tool versions. A second changes only instruction availability. Another simulates OCR failure and a resumed task. Creating the same order twice fails the business test even when both JSON files look correct. Escalating a contradictory total may be a better result than confidently inventing a reconciliation.

Maintain a separate set of rare, consequential cases. Easy-document averages say little about embedded malicious instructions or accidentally mixing records from two organizations. Those cases need explicit acceptance criteria and enough runs to inspect the failure mechanism.

Before choosing a platform, decide who owns reference answers, how input versions are retained, and what stops an unproductive run. Our guide to AI in data processing connects this boundary to extraction checks. The document parsing case supplies a practical task context. You can discuss your inputs and acceptance criteria before deciding how much autonomy the process should receive.