Circuit Breakers Can Break What They Protect

After every big exploit, the same fix gets proposed: put a circuit breaker on it. Rate-limit the outflows, cap the drains, freeze funds before they leave. It sounds obviously correct, and lately it’s being mandated across the board.

The problem is that a naive circuit breaker can easily hurt more than it helps. A rate limiter tight enough to stop an attacker draining the protocol is also tight enough to stop a liquidator closing an underwater position during a volatile move, or a user who needs to exit right now. Set it loose enough to never interfere with legitimate activity, and a real drain slips under it. And getting it wrong in the tight direction isn’t harmless: a cap that blocks liquidations during volatility creates the very bad debt it was supposed to prevent.

Having designed and built one of these systems end to end, I’ve come to the conclusion that a circuit breaker only works as one layer of a larger system, what I’d call an auto-remediation framework. These are my notes on how I think the whole thing should be approached.

What it’s for

This system doesn’t prevent exploits; prevention belongs to the contract logic and everything that hardens it (audits, tests, formal verification). It’s for when prevention fails. The cause can be anything (a missed bug, a manipulated oracle, a misbehaving dependency, a leaked key), but the damage almost always looks the same: funds leaving the protocol faster than they should. Humans can’t respond fast enough. By the time the team is awake and the multisig is assembled, the funds are gone. If you want to limit the damage, the response has to be automated.

It also helps to be precise about what you’re protecting: the capital the protocol holds on behalf of users. A user losing money on their own position is not a protocol loss, and as we’ll see, keeping that distinction sharp is what allows the protections to be tight without getting in everyone’s way.

The stack

I think of it as four layers plus a settlement mechanism. Each layer exists because the previous one has a blind spot, and none of them work well alone. The interesting part is how they fit together.

• L1: the smart contract’s own checks. Per-transaction invariants, enforced synchronously.
• L2: on-chain rate limiters. Cap how much can flow out per window, whatever the cause.
• The settlement layer. Lets users exit even while the cap is binding.
• L3: off-chain checks with an automated pause. Continuously verify the protocol’s global state; pause on breach.
• L4: ecosystem-level coordination. Because in DeFi, no protocol is an island.

The stack. L3 watches from off-chain and reaches down with an automated pause; settlement is the escape valve that keeps L2 livable.

L1: the contract checks itself

The smart contract already does a lot of checking. That’s what it’s for. Amounts on sensitive paths aren’t caller-chosen (a liquidator doesn’t pick how much to seize; the model computes it), and every state transition asserts its invariants. The contracts also carry hard limits on state itself: a ceiling on how much a lending facility can have outstanding, a cap on pool utilization. These are static bounds on levels; they limit how large a quantity is allowed to get at any point in time. For properties that are hard to pin down with tests, like share accounting and rounding direction, formal verification helps a lot; I wrote about that in the Sui Prover post.

The layers above exist to cover what the contract structurally can’t see (aggregate state, anything spanning transactions) and to double-check the contract itself, since the checks were written by the same fallible process that wrote the bugs.

L2: limit the losses until L3 can react

If something goes wrong anyway, L2 limits how much can be lost before the off-chain system reacts. Where L1 bounds levels, L2 is rate limiting: it bounds movement, how much can flow out per window of time. It also lives at the smart-contract level, because that’s where the funds move, and it catches the movement itself regardless of what caused it.

The concept is simple; the application is anything but. Where in the code do you meter: only the user↔protocol boundary, or internal flows too? What counts toward the cap? How do you set the limits? This is app-specific, and a few of the nuances we ran into:

• Cap the protocol’s losses, not user withdrawals. Users pulling out their own funds isn’t a loss event. Metering it just means firing on every busy day.
• Track gross and net outflow separately. A net-only cap can be gamed: deposit first to buy outflow headroom, then drain. The gross cap is the backstop a wash deposit can’t move.
• Use fixed cap values. If the cap scales off live TVL, an attacker who can inflate TVL mid-attack loosens the cap exactly when it should bind. Set a fixed number, revisit it periodically.
• Be careful with defensive paths. Liquidations, repayments, deleveraging: these protect the protocol, and a limiter that blocks them under stress starts creating bad debt instead of preventing it. Metering them can still make sense, but the limits have to be sized for exactly those stress conditions, and exempting them is reasonable where L1’s model checks already cover the path.

There’s no universal setting here. Each protocol needs to be looked at individually, and small protocols have it hardest: even a generous percentage cap can sit below a single busy day’s legitimate flow, so at small scale some false fires are unavoidable.

The settlement layer: don’t trap users

A rate limiter immediately raises a question: what happens to a withdrawal that exceeds the window’s headroom? Reverting it is a bad answer. An attacker will happily retry every window, while honest users are stuck with their funds exposed to market and protocol risk, which is exactly the situation they were trying to leave.

Users must be able to remove their exposure at any time, even when the cash can’t leave yet. Closing a position always works in full: debt repaid, market exposure unwound, liquidation risk gone. Only the cash is paced. The over-cap portion goes into escrow as a claim ticket and pays out as the window drains. If it later turns out an exploit was in progress, whatever is still in escrow can be frozen and clawed back. What already left is gone, but the system fails in the right direction. It also changes what a false alarm costs: a delayed payout rather than a failed exit, which is what makes running tighter caps tolerable in the first place.

This isn’t free. The frontend has to present the flow in a way users understand (“your position is closed, your cash arrives shortly” is a new concept, and the UI carries it). It also puts a burden on integrators: other contracts composing with the protocol need to become familiar with the ticket flow instead of assuming withdrawals settle atomically.

L3: recreate the state off-chain and check everything

L3 recreates the protocol’s entire state off-chain, as a consistent snapshot at each point in time, and continuously runs checks against it. This is more feasible than it sounds. On-chain state is small compared to what off-chain systems handle daily, because smart contracts are so constrained. It fits comfortably in memory.

On Sui, checkpoints arrive roughly every 250ms, and that sets the rhythm: consume the checkpoint stream, update the state, run the full check suite, repeat. We found that a comprehensive suite for Kai runs well within that budget. Everything is in memory, so checking every checkpoint is practical. When a check fails, the response is automated: a pause transaction, scoped to the affected part of the protocol, submitted by a key whose only capability is pausing. Pausing is reversible, which is what makes automating it acceptable; irreversible actions like clawbacks stay with humans, who now get to deliberate behind an already-paused protocol instead of during an active drain.

The infrastructure is the easy part of L3; streaming state and running checks is ordinary engineering. The crux is knowing what to check. That takes deep, protocol-specific knowledge: which relations are invariant for this design, what the state should look like, which quantities matter. It can’t be generalized into a one-size-fits-all product; you really have to sit down with each protocol individually, and that analysis is where most of the value is.

That said, the checks we ended up with fall into a few recognizable types:

• Accounting identities. What the protocol has issued versus what backs it: shares against assets, debt records against pool books. These hold exactly, by construction, so any deviation is proof of a breach rather than a statistical anomaly.
• Solvency checks. Value positions through oracle prices and watch for bad debt. These are noisier (prices move), so they use thresholds, and velocity matters as much as level: bad debt jumping in minutes looks like an exploit, the same amount drifting over days looks like market conditions.
• Unexpected changes. Parameters or privileged state changing with no corresponding governance action, objects appearing where none should.
• Oracle sanity. The feeds the protocol prices against, checked against independent sources.

Testing is its own puzzle: the whole job of these checks is recognizing states that should never exist, so no honest environment can produce a true positive. We test by attacking ourselves. Backdoor functions get injected into throwaway builds of the real contracts on a local network (mint shares with no backing, understate a recorded debt), and then the real pipeline has to notice, and the pause it produces has to block the malicious action on-chain. Before any of it is trusted live, it runs in shadow mode: every decision recorded, nothing submitted.

This is what ties the stack together. With L3 checking every checkpoint, L2’s only job is to hold the line long enough for L3 to react. That’s seconds in theory; in practice you’d give it more room, but still, windows can be short and caps can be tight, which means escrowed cash clears quickly and the limiter is barely noticeable in normal operation. Without L3, the rate limiter would have to bound damage on its own, which forces long windows and painful UX. With it, each layer carries only the load it’s built for.

How it plays out

To make the cooperation concrete, say prevention fails: a bug in a vault’s share-issuance path lets an attacker mint shares backed by nothing, and they start redeeming them for real assets.

Every transaction is individually well-formed, so the contract sees nothing wrong. The first redemptions clear within the rate limiter’s headroom; that money is gone. Then the cap binds: further redemptions still succeed, but the cash routes to escrow instead of leaving. Within a few checkpoints, the off-chain checks notice that shares no longer reconcile against assets (an exact identity, so there’s no judgment call), and the automated pause lands, scoped to the affected vault. Everything freezes, including tickets that matured in the meantime.

By the time anyone is paged, the incident is already contained. The team investigates behind the pause with all the time in the world, voids the exploit-linked tickets, fixes the bug, unpauses. The total loss is what slipped out before the pause: roughly one window of cap headroom, a number chosen in advance. Not the whole vault. And the honest users caught in the same window had their positions closed throughout; only their cash waited.

The same attack, on a clock. Most of the vault never leaves; what the caps deferred is recovered.

None of this makes exploits impossible. The system bounds damage rather than preventing it: whatever exits before the pause is gone, and an attack slow and quiet enough to look like organic flow stays under the caps. It only loses if it also breaks an invariant the checks watch.

L4: no protocol is an island

Everything so far is per-protocol, and a purely siloed defense has a problem: DeFi protocols are tied together. That’s the whole point of DeFi. One protocol’s rate limiter can become another protocol’s outage. An AMM that rate-limits in isolation can block liquidations that a lending protocol depends on, at exactly the moment they matter. Defense done separately by everyone can end up degrading the system as a whole.

The way through, I think, is recognizing that what ultimately needs protecting is the user↔protocol boundary. Between protocols that trust each other, funds can flow with higher limits, preserving composability, as long as flows halt properly where they cross out to users. That becomes practical if protocols share the same on-chain rate-limiting framework, so the trust relationships and limits are expressed in one consistent system rather than N incompatible ones.

This is where DeFi’s nature works in its favor. Composability is what blockchains ultimately enable; it’s the edge DeFi has over traditional finance, and worth defending rather than giving up. It’s also why L4 is possible at all: in legacy finance, coordination like this means committees and closed books, while here every flow lives on a shared ledger and the defense itself can be shared code.

But this only makes sense in a mature ecosystem. There’s a pyramid of prerequisites, and each level rests on the ones below:

1. A team that builds it right, in good faith
2. Audits, tests, formal verification
3. A team that runs it well: understanding the protocol deeply enough to react correctly during incidents and stress
4. Per-protocol auto-remediation (L1–L3)
5. Open-source contracts
6. And only then: ecosystem-level coordination

The maturity pyramid: each level rests on the ones below.

Getting there needs coordinated effort and probably a dedicated security team driving it. We’re not there yet as an ecosystem, but I hope we get there.

Putting it together

Contract-level invariants, flow caps that hold the line for a few seconds, a settlement path that never traps users, off-chain checks against full state snapshots with an automated pause, and eventually ecosystem-level coordination. Built like that, auto-remediation adds real security without degrading UX or normal operation. That, I think, is the bar a circuit breaker has to meet. Anything less and it risks breaking the thing it’s protecting.

We’re continuing to build this out. If you’re thinking about the same problems, especially the ecosystem layer, I’d like to compare notes.