Fallback Providers — AI Never Down

Fallback Providers — AI Never Down

Ever been in the middle of a big task when your AI provider suddenly goes down? It’s like your favorite coffee shop closing right when you need that caffeine fix. With Hermes Agent, that’s no longer a problem. Let’s look at how fallback providers keep your agent running, no matter what.

The Fallback Chain at a Glance

Hermes walks these layers in order, and each one has its own config key:

Order	Layer	What it covers	Where it is configured
1	Main model	Every user message, tool call and streamed response	model in ~/.hermes/config.yaml
2	Auxiliary task chain	A single side-job (vision, compression, title generation, …)	auxiliary.<task>.fallback_chain
3	Main fallback chain	The whole session, when the main model fails or hits capacity	fallback_providers (managed with hermes fallback); legacy key fallback_model
4	Built-in discovery chain	Last resort for auxiliary tasks	Automatic — nothing to set

Two rules worth remembering: each fallback_providers entry needs both provider and model, and an entry missing either field is ignored. Fallback is also turn-scoped — the primary model is restored at the start of every new user message, and within one turn fallback activates at most once.

Two Kinds of Models, One Goal

Hermes uses two types of model slots:

• Main model — this is the brain. Every user message, every tool call, every streamed response goes through it.
• Auxiliary models — these are the helpers. They handle side-jobs like context compression, image analysis, web-page summarization, approval scoring, MCP tool routing, session-title generation, and skill search.

Each auxiliary task has its own slot and can be overridden independently. That means you can use a cheap, fast model for summarizing web pages while keeping your main model for complex reasoning.

Setting Up Your Main Model

Open the dashboard and click Models in the sidebar. You’ll see two sections:

1. Model Settings — where you assign models to slots
2. Usage analytics — ranked cards showing token counts, costs, and capability badges for every model that ran a session

To set your main model, click Change on the Main model row. A picker dialog opens with two columns:

• Left — authenticated providers (only ones you’ve set up with API keys or OAuth)
• Right — the curated model list for the selected provider

Type in the filter box to narrow by provider name, slug, or model ID. Pick a model, hit Switch, and Hermes writes it to ~/.hermes/config.yaml.

What Happens When You Switch Mid-Session?

Here’s a crucial detail: when you switch models inside an active session, Hermes checks whether your next message will need context compression against the new model’s window. If the session is already near that threshold, you’ll get a warning. The switch still applies immediately, but compression runs on your first message after the switch.

Important warning: Mid-session switches reset the prompt cache. Prompt caches are keyed to the model serving the request, so any model change means the next message re-reads the entire conversation at full input-token price instead of the cached (~75–90% discounted) rate. On a long session, this one-time re-read can cost more than the per-token difference between models. Switch when you need to, but prefer doing it early in a conversation.

The Magic of Fallback Providers

Now, the star of the show: fallback providers. When your main model fails or hits capacity, Hermes automatically tries the next provider in your fallback chain. This happens seamlessly — you don’t even notice.

The fallback chain works like this:

1. Try the main model
2. If unavailable, try any task-specific auxiliary.<task>.fallback_chain
3. Then try the main fallback_providers / fallback_model chain
4. Finally, use Hermes’ built-in auxiliary discovery chain

Configure the chain with the interactive hermes fallback manager (subcommands add, list/ls, remove/rm, and clear), or edit the top-level fallback_providers list in ~/.hermes/config.yaml directly — each entry needs both provider and model, and entries missing either field are ignored. There are no environment variables for the primary fallback chain; this is intentional, so a stale shell export can’t override a deliberate choice.

Fallback is turn-scoped: each new user message starts with the primary model restored, and within a single turn fallback activates at most once — if the fallback also fails, normal error handling takes over. When a rate-limit response names its reset time, the primary is benched until exactly then (a provider that says nothing gets the exponential 60 s → 4 h backoff). Optionally, skip the switch when the primary reopens soon:

fallback:
  min_switch_reset_seconds: 120   # 0 (default) = always switch

Key	Default	Effect
fallback.min_switch_reset_seconds	0 (off)	A rate-limited primary whose declared reset is sooner than this many seconds is not swapped for a fallback; the retry backoff waits out the window instead.

Fallback also re-resolves the reasoning effort for the fallback model — its own agent.reasoning_overrides entry if set, otherwise the global agent.reasoning_effort. The same happens when the CLI falls back at startup because the primary’s auth fails before the first request, so the fallback model gets its own configured effort rather than the primary’s. An explicit hermes chat --reasoning <level> is kept across that startup switch, since it’s your intent for the run.

One more trigger worth knowing: an HTTP-200 response whose only assistant text is a router’s Connect timeout, please try again later. with zero completion tokens counts as an invalid response, so it’s retried instead of shown as the answer. A streamed refusal (the model declining with an explanation) is a terminal content_filter result, not an empty response, so it’s surfaced rather than retried.

Using Your Local Model as a Fallback

You can point a fallback entry at the local model server Hermes manages (Desktop Settings → Providers → Local models). Use the provider name llamacpp — the aliases llama.cpp and llama-cpp work too — and you don’t need to set a base_url, because Hermes already knows where its own server lives.

The important safety detail: if that local server isn’t running, the entry is simply skipped. Hermes never sends your local model name to another provider, so a stopped local server can’t leak the name into a cloud request. The same rule applies when you force the managed server with "llamacpp": if it’s off, the local model name is never handed to a different provider. You can also use "llamacpp" as an auxiliary provider to force the managed local llama.cpp server; when it is off, the local model name is never sent to another provider.

Setting Up Auxiliary Models

Click Show auxiliary to reveal the 11 task slots. Every auxiliary task defaults to auto — meaning Hermes tries your main model for that job too. Override a specific task when you want a cheaper or faster model for a side-job.

Fastest Path: Nous Portal

If you want 300+ models under one subscription, check out Nous Portal. On a fresh install, run:

hermes setup --portal

This logs you in and sets Nous as your provider in one command. Inspect what’s wired up with:

hermes portal info

Portal subscribers also get 10% off token-billed providers.

Quick Setup Tips

On a brand-new install, your config has model: "" (an empty string meaning “not configured yet”). The first time you run hermes setup or hermes model, that key upgrades to a mapping with provider, default, base_url, and api_mode sub-keys. If you ever see an empty string in config.yaml, run hermes model (or click Change in the dashboard) and Hermes will write the dict form for you.

Data Training Tiers

Some models like muse-spark-1.2-contributor are discounted because the vendor may train on your prompts and completions. Interactive model selection always shows a confirmation prompt. Non-interactive startup paths (like Kanban workers and cron agents) fail closed because they can’t ask that question.

If training on the unattended workload’s data is acceptable, record a persistent acknowledgement:

hermes config set security.allow_data_training_tiers_noninteractive true

Revoke it with:

hermes config unset security.allow_data_training_tiers_noninteractive

Never Down, Always On

With fallback providers, your agent keeps working even when one provider has a bad day. The setup is straightforward, the switching is automatic, and you get peace of mind knowing your AI is never down.

---

Keep reading: Hermes credential pools — Where fallback providers swap models, Hermes credential pools rotate several keys for the same provider.