AI Agents Need Coffee Filters Too

Recently I bought this stainless steel coffee dripper in a convenience store, in Korea to replace my old one.

The box of a stainless steel coffee dripper, printed in Korean. Three numbered steps show the dripper sitting on a mug while hot water is poured in. Below them, a caution block set apart in red. — Three steps, then a caution block. The line in the brown bar reads: depending on how finely the beans are ground, grounds may escape through the filter.

The instructions are trivial.

Put it on a cup.
Add coffee.
Pour hot water.

But there is a warning.

Depending on the grind size, coffee grounds may pass through the filter.

That’s the entire purpose of the filter.

Not to make better coffee. Just to prevent the wrong things from reaching your cup.

Everyone is focused on making agents more capable.

Better models.
More tools.
Longer context windows.
More autonomy running in infinite loop.

Very few people are asking a different question:

How do you know when the wrong things are getting through?

AI agents rarely make catastrophic mistakes. They make small ones. The kind that compile, deploy, pass validation, satisfy the tests, and still leave the system a little worse than before. A response that’s slightly outdated. A configuration that’s technically correct but inconsistent. A workflow that costs a few more cents every time it runs. Individually, they’re almost invisible. At scale, they’re everything.

These are the coffee grounds slipping through the filter.

You don’t notice them after the first request. You notice them after the ten-thousandth.

This is why I believe observability will become one of the defining pieces of AI infrastructure over the coming years. Traces tell you that an agent executed. Token counts tell you how much it cost. Neither tells you whether the output was actually good or if the right chain of tool calling was executed.

What we need are filters. Evaluation pipelines that continuously sample outputs. Guardrails that detect policy violations before they reach users. Business KPIs that measure whether agents improve the metric they were introduced to optimize. Quality scorecards that reveal slow regressions long before customers do.

In other words, not more telemetry about the agent itself, but more observability of its decisions.

The equivalent of checking whether anything made it into the cup that shouldn’t have.

I put telemetry on the list of engineering fundamentals worth fixing before scaling up your agents. This is the next layer. Instrumentation tells you what happened. Evaluation tells you whether it should have happened.

And like everything else, it belongs to the system. AI agents fall to the level of the system they run in. Either the filter is part of that system, or it isn’t there at all.

The dripper has only one purpose. It stands between what should be kept and what should be discarded. It doesn’t concern itself with the water, nor with the beans, nor with the ritual of brewing. It simply decides what reaches the cup.

AI observability deserves the same humility.

An execution trace is only the echo of an action. A token count is only the measure of its cost. Neither says anything about what crossed the boundary between the machine and the outside world.

What matters is not that the agent acted. What will matter more and more is what escaped.

If you have suggestions or thoughts, keep the conversation going on my Substack.