Hermes Roadmap: Structured Observability for Agent Runs

You’ve just spent an hour waiting for an AI assistant to finish a task, and when it finally replies, the answer is wrong. Worse, you have no idea what took so long, where the money went, or why it failed.

What this direction is: a flight recorder for your AI assistant

Imagine you’re a busy manager who sends a personal assistant on a complicated errand: book a flight, call a client, research a competitor, draft a memo, and email the team. The assistant comes back and says, “Done!” But you have questions: Which airline did you use? How long was that client call? Did the memo get sent before or after the email? If the assistant was slow or blew the budget, you’d have no way to audit the work.

Hermes is building exactly that audit trail. The official roadmap item, “structured observability,” means giving every AI run a mandatory, automatic work log. This isn’t a handwritten diary the assistant fills in after the fact. It’s a system that records every single step in real time: what action was taken, which tool was used, how many tokens were spent, how long it took, and what the result was. Think of it like the black box on an airplane—invisible during normal flight, but the only source of truth when something goes wrong.

What the official plan includes: from “what to record” to “how to use it”

The plan breaks down into four practical pieces, each with a clear job.

First, every key action gets a timestamped “span.” You can think of a span as one line in a structured diary entry. A model call gets a span. A tool call gets a span. A plugin hook, a memory lookup, even the final delivery of a message to a chat platform—each gets its own span. These aren’t scattered notes. They’re nested like Russian dolls. One overall run is the big doll. Inside it, you have medium dolls for “called the model,” “used the web search tool,” and “sent the result.” Inside those, you have smaller dolls for each retry or sub-step. If anything goes wrong, you can peel back the layers and find the exact millisecond where things broke.

Second, every span gets a “finance and performance sticker.” For each model call, the system automatically tags how many tokens were used, what that cost in dollars, and how long the call took. These tags are then grouped automatically by session, by tool, by provider, and by platform. This is like your assistant’s travel expenses being automatically coded to the correct project number. At the end of the month, you can see at a glance which client ate the most budget or which airline was consistently late. The roadmap specifically calls out “slow tool and slow provider diagnostics.” That means the system will flag any step that takes unusually long—no manual comparison of historical data needed.

Third, the log records decision paths, not just actions. Suppose the primary AI model provider times out. Hermes automatically switches to a backup provider. That fallback decision gets a full entry: why the switch happened, how long the system waited before giving up, and how the backup provider performed. This is like your assistant having to rebook a canceled flight. The log shows the original delay, the rebooking process, and whether the new flight arrived on time. The plan also includes recording prompt cache hits. When a prompt is cached, the system doesn’t have to pay to recompute it. Knowing whether a cache hit occurred is crucial for explaining why a run was cheap or expensive.

Fourth, the plan includes “exportable run bundles.” This is a single packaged file containing all the logs, inputs, outputs, and configuration settings from one run. You can use it for your own debugging, or send it to support for deeper analysis. But two red lines are built into the design. First, observability is opt-in by default when it leaves your machine. Nothing is recorded or transmitted without your explicit consent. Second, the debug bundles are scrubbed of secrets by default. Even if you send a bundle to someone else, they won’t see your API keys or other sensitive information. It’s like a work diary where private conversations are automatically redacted—only you can see the full text.

The acceptance criteria for this whole feature is simple and practical: an operator must be able to answer three questions about any run—why was it slow, why was it expensive, and why did it fail? If the log can answer those three questions, the feature passes.

What it means for users: from guessing to knowing

Before this, when an AI run was slow, you waited. When it was expensive, you stared at the bill in confusion. When it failed, you just retried and hoped. Structured observability gives you the power of post-mortem analysis for the first time.

The next time your AI “suddenly gets dumber,” you can open the log and see exactly what happened. Maybe the memory retrieval didn’t find the key document. Maybe a tool call timed out. Maybe the model provider had a temporary outage. You’ll know, not guess.

For developers, this cuts debugging time from hours to minutes. For regular users, it means you can finally ask, “Why did this cost so much?”—and get a precise answer like, “You called the same tool three times in step three, and each time it re-processed the same long document.” No more vague “model cost fluctuations.”

The privacy protections are baked into the design philosophy. Logs stay on your machine by default. Telemetry only goes out if you explicitly opt in. Debug bundles are pre-sanitized. Your assistant’s work log is detailed, but anything sensitive is automatically masked.

In short, Hermes is turning “AI explainability” from a buzzword into a working tool. It doesn’t make the AI smarter. It makes every mistake auditable. For anyone who relies on AI to get things done, this is the closest thing to a practical warranty—a way to hold the system accountable, one log entry at a time.