Resilience for multi-agent AI systems

Your agent swarm
will survive this.

Ballast adds circuit breakers, backpressure, and cost-aware fallbacks to any AI agent pipeline — and chaos testing to prove they work. One Python decorator. No orchestrator rewrite.

$ pip install git+https://github.com/s3ak6i-dev/Ballast Star on GitHub
Live simulation — 40 agents · 1 flaky API
0served live
0served by fallback
0hard failures
CLOSEDbreaker state
The problem

One slow API can take down your whole swarm.

Multi-agent systems ship with almost none of the hardening microservices learned over 15 years. The failure modes are old; the ecosystem is new.

Cascading failure

Retries pile up until everything dies

One degraded tool makes every agent retry. Retries exhaust connections, budgets, and patience — then the whole pipeline fails, not just the sick dependency.

No backpressure

Fan-outs hammer your dependencies

A task spawns 50 agents; 50 agents hit one API at once. Nothing queues, nothing sheds, nothing throttles — until the API does it for you, with 429s and bans.

Uncontrolled cost

Swarms burn budgets in minutes

Every retry is a token bill. Without cost-aware routing, an incident isn't just downtime — it's the most expensive hour of your month.

How it works

Wrap. Detect. Degrade.

01Wrap

Decorate any call — a tool, an LLM API, a database query. That's the entire integration. Works with LangGraph, CrewAI, AutoGen, or a plain agent loop.

from ballast import guarded

@guarded(dependency="openai_api", timeout=10)
def call_llm(prompt: str) -> str:
    return client.chat.completions.create(...)

02Detect

Every dependency gets a circuit breaker: a rolling window of failures and latency versus a learned healthy baseline. When an API degrades, the breaker trips in milliseconds — and stops your swarm from hammering a dying service.

# the event stream, live:
breaker_trip      openai_api  failure_rate=0.55, cooldown=10s
breaker_half_open openai_api  probing recovery…
breaker_close     openai_api  probes_succeeded

03Degrade

While the breaker is open, calls don't fail — they walk a fallback chain: a cached answer, a cheaper model (when a rules-based classifier says it's safe), or a static response. Your pipeline keeps moving, and it costs less during the incident, not more.

@guarded(
    dependency="openai_api",
    cache_ttl_s=300,          # rung 1: recent identical answer
    cheaper=call_small_model, # rung 2: downgrade simple prompts
    fallback="degraded",      # rung 3: static last resort
    cost_fn=lambda r: r.cost, # meter spend vs. your budget
)
def call_llm(prompt: str) -> str: ...
What's in the box

Distributed-systems hardening, agent-shaped.

⚡ Per-dependency circuit breakers

Trip on failure rate or p95 latency vs. a learned baseline. Half-open probing, exponential-backoff cooldowns, manual overrides.

🚦 Global backpressure

A concurrency ceiling with strict-FIFO queueing. Excess load queues, then sheds — your system degrades instead of collapsing.

🪜 Fallback chain

Cache → cheaper model → static value → explicit error. Configured per dependency, served automatically when the breaker opens.

🧮 Cost-aware routing

Rolling-hour budgets with soft and hard ceilings. Past the soft ceiling, simple requests route to a cheaper model automatically.

💥 Chaos injection

Time-boxed failure, latency, and corruption rules against your real code path. Your resilience demos double as CI tests.

📊 Live dashboard

Breaker states, traffic and cost charts, a coalescing event feed, and a one-click built-in incident demo — over WebSocket, on localhost.

Chaos testing

Don't claim resilience. Prove it.

Everyone says their pipeline survives an outage. Ballast lets you kill a dependency on purpose — in dev, time-boxed, against the real code path — and watch the system detect, reroute, and recover. This is the actual output:

$ python examples/demo.py
[ 3.03s] >> CHAOS INJECTED mock_api {failure, 85%, 4s}
[ 3.06s] !! BREAKER TRIP mock_api {failure_rate: 0.55} ← detected in 0.02s
[ 3.06s] -> FALLBACK ACTIVE mock_api (serving cached responses)
[ 5.06s] ?? HALF-OPEN mock_api probe failed → cooldown doubled
[ 9.06s] << CHAOS CLEARED mock_api {expired}
[ 9.11s] OK BREAKER CLOSE mock_api {probes_succeeded}
 
calls: 10,466 | live: 4,553 | fallback: 5,901 | errors: 12
The dashboard

Watch your swarm survive, live.

Breaker cards with cooldown countdowns, traffic and cost-vs-budget charts, and a coalescing event feed — streamed over WebSocket. This is a real recorded incident from the built-in demo (chaos at 0:08, recovery at 0:17):

docker compose uplocalhost:8080, or python -m ballast.dashboard. The demo needs no API keys.

Get started

Your config, in 60 seconds.

Answer five questions; copy the result into your project. That's the whole integration.

1 · What are you protecting?
2 · Which framework?
3 · How big is the swarm?
4 · Hourly cost budget?
5 · Fallbacks when things break?

      
Questions

The honest FAQ.

How is this different from an LLM gateway (LiteLLM, Portkey, …)?

Gateways proxy your LLM traffic and can retry or reroute API requests. Ballast lives in-process with your orchestrator, so it can do what a proxy can't: throttle agent fan-out with backpressure, guard non-LLM dependencies (databases, tools), and chaos-test the whole pipeline. They compose — many teams will want both.

What's the overhead on the happy path?

A lock-protected counter check (backpressure), a breaker state read, and a window append per call — microseconds, entirely in-process, no network hop. With chaos disabled the injector is a plain pass-through.

Does it support async?

Yes. @guarded detects async def functions automatically — backpressure waits without blocking the event loop, fallbacks can be async callables, and timeout becomes a real cancelling timeout (unlike the sync flavor, where a thread can't be safely killed). aguard() is the async context-manager form.

Is it production-ready?

It's a well-tested v1 (96 deterministic tests, CI on Linux and Windows across Python 3.11–3.13) that's honest about its limits: single-process in-memory state and a localhost dashboard with no auth. Read the security section in the README before deploying anything.

Give your swarm some ballast.

Clone it, run the built-in incident demo, and watch a breaker trip and recover in your browser — no API keys required.

$ pip install "ballast-agents[dashboard] @ git+https://github.com/s3ak6i-dev/Ballast" && python -m ballast.dashboard