Why an agent can’t observe itself
When an engineer hears “I instrument my agents,” they picture application observability: add a logging library, emit a span at the top of each request, ship the traces, read a dashboard. That model is decades mature and it is correct for software you wrote, because software you wrote cooperates. You own the call site, so you put the probe there. A coding agent gives you none of that, for one uncomfortable reason that names this whole guide: you can’t ask an agent to log itself.
An agent is a vendor’s binary running a vendor’s model behind a vendor’s API, exposing only the surfaces the vendor chose. It will not emit your span or carry your trace ID. When it spawns a subagent, that child is born and dies outside your instrumentation, leaving only a transcript written after the fact and a line on someone’s bill. The thing you most want to measure is the thing least willing to be measured. And there is a sharper edge to this than mere indifference: an agent does not know your logger is special. To the agent it is one more service in a list, one more file it might tidy up. I learned that the on-the-nose way when an agent doing unrelated housekeeping inside a session reached out and cleanly disabled the very service watching it. You cannot ask an agent to log itself, and you cannot trust it not to turn the logger off.
So the entire discipline starts from an inversion of the normal assumption. In application observability you instrument a cooperative subject from the inside. In agent observability you instrument an uncooperative — sometimes actively hostile — subject from the outside, deriving everything you need from facts you can observe without its help, and designing every part of the system to survive the subject trying to bury it. Get that framing wrong and you build a dashboard that lies to you the day it matters most: a dead capture service looks exactly like a healthy one with nothing to report.
Five surfaces, no view
The reason this stops being optional is that the surfaces multiply fast. Count the AI surfaces in one of my ordinary days. Claude Code’s main thread. Its subagents — more of them in an hour than I type prompts. A Codex session and a Gemini session for second opinions on sensitive changes. A local model on a LAN box draining cheap work while a frontier model grades it. That is five producers, three vendors plus my own hardware, on a single machine.
Every vendor shows me my usage — inside its own walls. None can see across the fence to the next, and none knows the thing I actually care about, which is not “how many tokens did Claude burn” but “what did the work cost, end to end, across whichever models touched it.” That cross-vendor, per-developer view is structurally not for sale. No vendor has an incentive to build the dashboard that puts its own bill next to a competitor’s. So you build it or you fly blind, and for a long time I flew blind and told myself the billing pages were enough. They were not. A CFO knows the AWS bill to the dollar; almost nobody can tell you the AI bill the same way, let alone which team, which person, or which project drove it. The gap between those two states of knowledge is the entire reason this track of my work exists.
The capture layer: one schema, every vendor, never in the way
The capture layer is the foundation, and one constraint swallows all the others while designing it. The capture code sits on the agent’s hook path — it fires on every single tool call. If it is slow, my whole day is slow. If it throws, the agent inherits the error. The first time telemetry is the reason a session hangs, I rip it out, and so would you. A tool that degrades the thing it observes does not get a second chance.
So the design splits in two. On the hot path is a tiny native binary whose only job is to read the hook payload, hand it off, and exit — fast, swallowing every failure it can meet, because the contract with the agent is absolute: it never sees an error that came from my telemetry. Behind it sits an ordinary background service that does the slow work — walking transcripts, rolling up tokens, writing to the database — safely off the critical path. Fail-open means more than failing quietly: when the background service is down, the hot path spools its data to disk and the service drains it on the way back up. The agent never waits, nothing is lost, and an outage becomes a delay rather than a hole in the record.
The other half of capture is normalization. Every vendor’s traffic has to land in one schema — the fields that different models name differently lined up, the data pseudonymized so it is groupable without being personally legible. That single schema is what turns five disconnected billing consoles into one queryable instrument. Once every model call across every vendor lands in one analytics database, you can finally ask questions nobody could answer before. This is the capture layer I run today, the thing the AI Telemetry & Governance track on my own site calls “Wood Fired Telemetry”: it intercepts every model call across every vendor into one analytics database, and it is the precondition for everything that follows. The full build — the fire-and-forget hot path, the fail-open spooling, the self-healing watchdog — is written up in you can’t ask an agent to log itself, which is the companion essay to this guide and goes a layer deeper than I can here.
Attribution without cooperation
Capture is the easy half. The hard half is the question the title makes concrete: when data lands, who produced it? The agent did not say. My telemetry is handed no stable per-developer identity, and none at all for the anonymous subagents the main thread spawns and reaps inside a single prompt. To attribute a token to a context, I have to derive the identity from the outside, with no cooperation from the thing I am identifying.
The trick is to compute identity, not assign it — a hash of the stable facts I can already observe. The main thread’s identity is a hash of its working directory and session id; a subagent’s is a hash of its type, its agent id, and the session it lives in. No registry, no lookup, no I/O — a pure function from observable facts to a stable id. Two different processes that see the same context compute the same identity independently, with nothing shared to coordinate through. The agent cannot alter it, cannot spoof it, and pays nothing for it. That same discipline carries the rest of the system. When the start and stop events for a subagent arrive out of order — which they do, because they come from separate processes the OS schedules as it likes — the stop handler does not depend on a prior registration at all. It reconstructs what it needs from the subagent’s own transcript and flags the gap when even that is missing. You instrument an uncooperative subject the way you design a protocol for a lossy link: assume the message you needed is the one that got dropped, and recover without it.
Per-commit and per-release cost attribution
Tokens are a vanity metric, and everyone who has stared at a usage graph knows it. The question a business actually has is not how many tokens an agent burned but what it got for them. This is where observability becomes governance: the number that lets leadership reason about AI spend is not a token count, it is cost per shipped unit of work.
I can produce that number because every git commit rides the same pipeline as a telemetry event — branch, insertions, deletions and all — and every closed task in my orchestration layer is keyed to the same session that produced it. So I can divide spend by output: dollars per commit, dollars per verified task, for any window, any person, any project, and by extension per release once you sum the commits a release contains. That per-commit and per-release attribution is exactly what the telemetry track promises — AI token spend attributed per commit and per release, drillable from a top-line number all the way down to a single prompt.
Two corrections were needed to make that number honest, and they are worth stating because anyone building this will hit both. The first was time. The capture path drains its backlog in roughly ten-second bursts, so a ninety-second run with nine subagents first showed up as ten seconds of work; I had to derive true duration from the event-side timestamps written in real time rather than from when the rows happened to land. The second was generation versus context: well over ninety-nine percent of all tokens are cache reads — the agent re-reading context it already paid for — and only the output tokens are the actual work. Measure “tokens” without splitting those and you are measuring noise. In one real month, of ten billion tokens captured, all but about seventy-four million were cached reads billed at a fraction of list; the bill was driven almost entirely by that thin slice of full-price input and output. Once both corrections were in, every session could be priced against what it shipped, which is the only token number a manager should ever be shown.
Vendor-neutral by design
The neutrality is not a feature I bolted on for fairness; it is the only design that answers the actual question. If my record of what AI costs depended on any one provider’s reporting, it would have a hole exactly where the cross-vendor comparison lives — which is the comparison that matters. Anthropic is the cost driver in my own usage, OpenAI a rounding error, Gemini near zero, the local model carrying its tokens for free; the only way to see that shape at all is to capture all of them into one schema that belongs to me, not to any of them.
This shows up again one layer up, in the orchestration tooling. The shipped task-and-grader skills I run are vendor-neutral on purpose — the commit that introduced the parallel executor explicitly removed cross-references to other agent-focused tooling from the skill text, so the primitives work whether the agent on the other end is Claude, Codex, Cursor, or Gemini. The same instinct runs through the whole stack: the record of what your agents did and cost must not be hostage to the vendors being recorded. When I want two frontier models from two different vendors to grade each other’s work, they do it through my task system without ever meeting inside the same CLI session, and the telemetry sees both sides in the same units. Vendor-neutral capture is what makes that legible; a per-provider dashboard never could.
Governance, disclosure, and EU-AI-Act exposure
For a solo operator the first payoff of all this is purely operational — I can see my own spend and price my own work. But the reason I treat it as a governance discipline rather than a personal convenience is that the questions it answers are about to become questions organizations are required to answer. Finance sees a token bill, engineering sees activity, and nobody can draw a straight line between the two. When leadership asks what the AI is worth, the honest answer today is usually a shrug. The moment platform-disclosure obligations or EU-AI-Act exposure land on the horizon, “we don’t really track that” stops being acceptable.
What the capture-and-attribution layer produces is the artifact those regimes ask for: a per-developer, per-project, per-provider ledger that a finance or compliance team could actually read. You start at the org, drill to a department, then a user, then a single session, then the exact prompt that ran — and because the schema is pseudonymized, the pseudonymity holds the whole way down, so an org gets accountability without surveillance. That distinction is the governance design, not an afterthought: the goal is a record complete enough to satisfy a disclosure requirement and structured so that producing it does not turn into monitoring individual people’s keystrokes. The same instrument that tells me what a commit cost is the instrument that lets an organization say, with evidence rather than a shrug, what its AI is doing and what it is spending — which is exactly the position disclosure and the EU AI Act are going to require.
Observing multi-agent fleets
A single agent on a single task is the unit you instrument. Governing a fleet of them is a different problem, and it is the one most teams hit second. The moment you have more than one agent draining the same backlog, two failure modes appear that no per-call telemetry catches: two agents grab the same task, or an agent quietly does work it was never supposed to be able to do. Both are governance problems, and both want a structural answer rather than a watchful human.
My answer is a shared task layer with two coordination primitives that double as governance controls. The first is atomic claiming: when a task is unclaimed, any number of agents can race to claim it, exactly one wins, and the rest get a conflict and move to other work. This is verified end-to-end — twenty concurrent agents race the same task, one success, nineteen conflicts, zero errors — and stale claims auto-release so a crashed agent does not lock the work forever. Atomic claiming is what lets a fleet run without a human babysitting the queue, and the claim history is itself an observability surface: every claim and every conflict lands in the database as searchable record. I unpack the full design in introducing wood-fired-tasks, the orchestration layer I open-sourced for exactly this.
The second primitive is the one that matters most for governance: read-only grader agents. The agent that wrote the code never grades the code. A separate, read-only agent does — and “read-only” is enforced at the capability level, not by instruction. The verifier’s tool list has no edit, no write, no mutating database call; its shell access is allowlisted to read-only commands and explicitly denies anything that could mutate state. It physically cannot alter the code it is grading or the task it is grading against, and it emits a structured verdict with cited evidence per acceptance criterion — never “looks good,” always a file, command, or commit. A second grader catches composition bugs the per-task verifier cannot see, because that one only ever sees a single task’s diff, never the union of two workers’ edits to the same file. This generator/critic separation is the governance mechanism that makes a fleet trustworthy: a hard rule written in prose is a request that competes with everything in the model’s training, while a grader that cannot edit cannot be talked out of its job. The governance of an agent fleet is, in the end, the same move as instrumenting one agent — you do not ask the subject to police itself, you build a structure outside it that it cannot subvert.
Asking the instrument why — and acting on the answer
Observability earns its keep when it changes a decision. The piece of this that changed how I work is a telemetry query interface — the Telemetry Query MCP on my telemetry track — that lets me put questions to the analytics database in plain language instead of hand-writing SQL against it. The example that earned its keep: I had a hunch that draining a task graph in parallel might deliver a batch as well as running a long phase-gated workflow I had relied on for months. Rather than argue it from taste, I asked the data — seventy thousand API calls across three hundred sessions — and it answered cleanly. Per token generated the two approaches cost about the same. But a scoped graph project landed five-to-twelve times cheaper in absolute dollars and roughly an order of magnitude faster: a complete twenty-seven-task project verified in 1.7 hours for $118, against a milestone spread over five days for $1,480.
The cause was right there in the same data. The long-lived workflow re-reads its entire accumulated context every turn — around 780 million cache-read tokens for a single milestone — while the graph fans the work into short-lived parallel subagents with small fresh contexts, about eighteen times less. I did not switch my batch-delivery workflow on a feeling. I switched it because the instrument I built to watch myself told me, in dollars and hours, which approach was wasting both. That is the loop the whole discipline is for: measure, see the pattern, change the practice, measure again. Observability that never moves a decision is just a more expensive billing page.
What to instrument first
If you are starting from a stack of billing consoles, the order that worked for me is this. First, build the capture layer fail-open before anything else — a hot path that can never slow or crash the agent, spooling to disk when the backend is down. If capture is not invisible to the agent, you will turn it off within a week, and a telemetry system you have switched off is worse than none because it lulls you. Second, solve attribution by derivation, not cooperation — compute identity from observable facts so the anonymous subagents get attributed too, because in my days the subagents outnumber the prompts and an unattributed majority is a blind majority. Third, make the absence of data visible: add the heartbeat and the watchdog so a dead capture service stops looking identical to a quiet healthy one, and so the instrument can resurrect itself when the thing it observes buries it. Only once those three hold should you build the dashboards and the cost-per-commit rollups, because a beautiful dashboard over an unreliable capture path is a confident lie.
And when you turn the instrument on your fleet rather than a single agent, lead with the structural controls — atomic claiming so agents do not collide, read-only graders so the thing checking the work cannot also do the work. Those are the governance primitives that make a fleet legible and safe at the same time, and they are the same idea as the capture layer one level up: never ask the subject to report on itself, and never give the watcher a reason to trust rather than verify.
I document this practice as I build it. The two essays that anchor this guide go deeper than it can: you can’t ask an agent to log itself, the full build of the capture, attribution, and self-healing telemetry layer, and introducing wood-fired-tasks, the coordination layer with the atomic claiming and read-only graders that govern a fleet. The live commit and source-line evidence behind all of it is on /practice, where it can be weighed instead of asserted.