Measure the system, not just the model

When people evaluate agents, they usually evaluate models. They swap one model for another, run a benchmark and compare scores. That's useful, but in a real agent system the model is one component among many. The harness that runs it, the monitors that watch it, the proxies that route its calls and the numbers you use to judge its cost all shape the result, and they're almost never tested.

Your harness has opinions

Agent harnesses are usually built around one model family, and they carry assumptions about it. How big the context window is. What default reasoning settings to send. Which tools exist. Swap in a different model and the harness doesn't complain. It just keeps applying assumptions that no longer hold. In one of my research systems, a model name the harness didn't recognize was silently treated as having a particular window size, which changes when it summarizes and how much it can hold.

Sometimes the assumption runs backward from what you'd expect. When I routed a non-Claude model through a translation layer, a setting requesting "high" reasoning effort cut that model's reasoning by close to a factor of ten compared with sending nothing at all, and the two distributions didn't even overlap.

So when you compare models, keep the surrounding settings as close to identical as you can, even when changing them would save money, or you're comparing harness configurations instead of models. And fail closed. If the route to a model is broken, the request should fail loudly instead of falling back to a different model. Otherwise your evaluation is measuring a mix of two models and reporting it as one.

Evaluate the watchers

Monitoring systems for agents need their own evaluations, and they almost never get them. One of the first monitors I built ran a model over everything it saw and produced a stream of false alarms that people learned to ignore, which is the worst outcome a monitor can have.

The fix was a labeled test set for the monitor itself. It contains snapshots of normal runs, which must produce no alert at all, and snapshots of known failures, which must produce the right alert. A detector ships only if it passes both. False alarms on normal runs aren't tuned down over time. They fail the test. That turns "is the monitor too noisy?" from a feeling into a check you run before anything changes.

Unknown is not zero

Cost is the measurement teams most want and most often get wrong, and it usually goes wrong without any error. At one point every single call in a sample of more than a thousand from one of my research runs reported its cost as unknown. The proxy in the middle wasn't passing a figure through, so the dashboards showed less spend than there really was.

The rule that came out of that is simple. Never record an unknown cost as zero. A zero reads as "this was free," which is the opposite of the truth. Keep unknowns visible, and report how many calls actually had a cost alongside the total, so a partial number can't pass as a complete one.

Record two figures for each call, what the provider billed and what you'd estimate from its published prices. When they agree, good. When they drift apart, that gap is a signal worth chasing. And don't half-translate a pricing scheme you only partly understand. If a provider charges more for long prompts and you can only model part of that, decline the whole price card rather than falling back to the base rate. A partial translation under-reports on the expensive calls.

Where the money actually goes

Agent cost doesn't behave like most software cost, and the differences are worth measuring rather than guessing.

Every turn re-sends the whole conversation. Cached tokens are much cheaper, but they're not free, and one oversized tool result keeps getting billed on every turn after it. That's why I cap large outputs and write the overflow to a file the agent can read if it really needs it. Caching itself is fragile, too. A routing layer in one of my research setups was silently discarding the provider's roughly tenfold discount on cached input, and nothing reported it as a failure. Waiting costs money as well. An agent that checks for news and finds none still pays for a full turn, which is one more reason to keep waiting in plain code, outside the model.

The goal is making the same work cost less, and that takes spend broken down by role and by phase, so "the agents are expensive" turns into "this reviewer is expensive in this phase, and here's why."