Circuit Breaker Middleware

Introduction

Note: This middleware is only available with Kipchak Enterprise.

When a service your API depends on stops answering, every request that calls it waits for the timeout. Workers fill up with requests that are only waiting, the API slows down for everyone, and the struggling service gets a steady stream of retries just when it needs fewer. A circuit breaker notices the failures, stops calling the service for a while, and answers at once instead, then lets one call through to see whether the service is back.

• For every HTTP call, without code changes: name the hosts a breaker guards, and every request to them made through the HTTP driver goes through it, including the requests the OPA, Vault, ClickHouse, OAuth 2 Token and other HTTP-based drivers make.
• Fleet-wide: with Valkey, every worker on every host shares each breaker. The fleet opens it after a handful of failures between them, and one worker, anywhere, makes the trial call.
• Fails as callers already expect: a refused call is a connection error, so the OPA driver reports OPA as unreachable, the Vault driver Vault as unavailable, and your own code's existing handling applies.
• Backs off: each failed trial keeps the breaker open twice as long, up to a limit.
• Routes too: FailFast answers a route with 503 and Retry-After while a service it needs is down.
• Anything else: database calls, SDKs and queues go through a breaker with ->call().
• Never the cause of an outage: if Valkey is unreachable, each worker keeps its own breakers until it returns.

Installation

To install this middleware, you need access to the Enterprise Composer repository at https://php.pkgs.1x.ax/.

composer require kipchak/middleware-circuit-breaker

It needs kipchak/driver-http 2.15 or later, which Composer installs with it. For shared breakers, add the Valkey driver to drivers/drivers.php.

Copy the sample config to config/kipchak.circuit-breaker.php and initialise the middleware in middlewares/middlewares.php:

use Kipchak\Middleware\CircuitBreaker\CircuitBreaker;

Error::initialise($app);
CircuitBreaker::initialise($app);

Configuration

<?php

return [
    'enabled' => true,
    'store' => 'valkey',            // or 'local': each worker keeps its own breakers
    'valkey_pool' => 'cache',
    'key_prefix' => 'kipchak:circuit:',
    'http' => true,                 // guard requests made through the HTTP driver

    'defaults' => [
        'window' => 10,
        'min_requests' => 10,
        'min_failures' => 5,
        'failure_ratio' => 0.5,
        'consecutive_failures' => 10,
        'failure_statuses' => [500, 502, 503, 504],
        'slow_ms' => null,
        'open_for' => 30,
        'max_open_for' => 300,
        'probe_timeout' => 10,
    ],

    'breakers' => [
        'opa' => ['hosts' => ['opa:8181'], 'open_for' => 10],
        'payments' => ['hosts' => ['api.stripe.com'], 'failure_statuses' => [500, 502, 503, 504, 429]],
        'fx-feed' => ['min_requests' => 5, 'min_failures' => 3],
    ],
];

The config is checked when the worker starts. Valkey is not contacted until the first call, so an API starts even while Valkey is down.

Breakers

Each breaker takes the defaults and overrides what it needs:

A breaker opens on whichever comes first: consecutive_failures failures in a row, or a window with at least min_requests calls, min_failures failures and a share of failures of at least failure_ratio. A run of failures matters on its own because a busy service's window is full of earlier successes: in testing, an API serving 3,400 requests a second through OPA never reached a 50% failure ratio during a 12-second OPA outage, while three failures in a row opened the breaker within a second.

The window slides: calls are counted in buckets one window long, and the previous bucket counts in proportion to how much of it is still inside the window, so there is no cliff at a bucket boundary.

States

A breaker that opened and never saw a trial call, because nothing called the service, closes on its own after three times max_open_for.

What callers see

A request refused by an open breaker throws HttpCircuitOpenException, a Guzzle ConnectException:

• Laravel's client, and the drivers built on it, turn it into a ConnectionException: the OPA driver throws OPAConnectionException, the Vault driver VaultConnectionException, and so on. The Policy middleware answers OPA-backed routes with 503 under its on_error: closed.
• PSR-18 callers see a NetworkExceptionInterface.

The message names the breaker and when to try again, and retryAfter() gives the seconds:

use Illuminate\Http\Client\ConnectionException;
use Kipchak\Middleware\CircuitBreaker\Exception\CircuitOpen;

try {
    $rates = Http::get()->get('https://fx.example.com/latest')->json();
} catch (ConnectionException $exception) {
    if ($exception->getPrevious() instanceof CircuitOpen) {
        $rates = $this->cachedRates();   // a fallback while the breaker is open
    }
}

Routes

FailFast answers a route with 503 and Retry-After while a breaker it depends on is open, without running the route:

use Kipchak\Middleware\CircuitBreaker\Handlers\FailFast;

$app->post('/v1/payments', [Payments::class, 'create'])->add(new FailFast('payments'));
$group->add(new FailFast('opa', 'ledger'));

A half-open breaker lets the request through, so it can be the trial call.

Other calls

use Kipchak\Middleware\CircuitBreaker\CircuitBreaker;

$rates = CircuitBreaker::get('fx-feed')->call(fn () => $feed->latest());

Any exception counts as a failure and is rethrown; pass a second argument to say which do not:

$order = CircuitBreaker::get('ledger')->call(
    fn () => $ledger->find($id),
    fn (\Throwable $e): bool => !$e instanceof NotFound,
);

An open breaker throws CircuitOpenException. For a status endpoint, CircuitBreaker::states() gives each breaker's state and the milliseconds until it may change.

Valkey

Each breaker is a few keys under key_prefix, all in one cluster slot through a {name} hash tag, so standalone, Sentinel and cluster pools all work (not RedisArray, which cannot run them atomically). Every decision is one Lua script: an ordinary call costs two round trips, one before and one after, and an open breaker none, since each worker remembers until when it is open.

If Valkey cannot be reached, each worker keeps its own breakers in memory and tries Valkey again every store_retry_after seconds (default 5), logging the switch each way. A phpredis client broken by the outage is replaced when Valkey returns.

Breakers are shared by every API that uses the same Valkey and key_prefix, so two APIs calling the same service can share its breaker, or keep their own with different prefixes.

Logs

Each names the breaker; refused calls are not logged one by one.

Performance

In testing on the Kipchak Enterprise runtime in worker mode, two hosts sharing one Valkey served about 3,400 requests a second between them through the Policy middleware and OPA. With OPA paused, the fleet opened the breaker within a second of the first timeouts, and the hosts went from 4 requests a second, each waiting two seconds, to answering every request in about 11 milliseconds. One trial call went to OPA every three seconds, then six, and the first after OPA returned closed the breaker for both hosts. With Valkey stopped mid-traffic instead, every request still succeeded, and each worker logged the switch to its own breakers and back once.

Git Repository

The source code for this middleware is available on 1x.ax at https://1x.ax/mamluk/enterprise/kipchak/middlewares/circuit-breaker.