When a dependency is failing, the worst thing your service can do is wait for it.

That sentence is the whole idea behind a circuit breaker, and behind shurutech/circuit-breaker, a small Go package we built after watching one slow database take down a fleet of services that had nothing wrong with them. This post is about the failure it prevents, the state machine that prevents it, and the design choices that make the package fit services running as many replicas.

Take Me Along

A failing component has a message for everyone that calls it: take me along.

Picture the boring, healthy version of a system. A client talks to Service A. Service A fans out to B and C. Everything eventually lands on one database. Now the database gets slow. Not down—slow. Slow is worse than down, because down fails immediately and slow makes you wait.

Service A sends a query and waits. The query times out. A retries, and waits again. Meanwhile two hundred more requests have arrived, and each one is holding a goroutine, a connection, and a slice of memory while it waits for a database that is not going to answer. A stops answering too. The load balancer, which only knows that A's health check still returns 200, keeps sending it traffic. B and C call A. Now they're waiting.

A client behind a load balancer fanning out to three services, all marked as waiting, all with timed-out requests pointing at a single failed database One slow database, and every service in the fleet is parked on a request that will never return.

Nothing else in this picture broke. Everything else was polite. It waited its turn, and waiting is what took it down.

The fix is old and it comes from your house. A short in one appliance should trip one switch, not melt the wiring in the walls. A circuit breaker in software does the same thing: after enough failures, it stops sending current to the thing that's failing, and it tells callers immediately instead of making them wait to find out.

Three States, Six Knobs

Every circuit breaker is the same little state machine. The package makes the states explicit and the transitions configurable.

State diagram: CLOSED moves to OPEN on MaxFailures within one minute; OPEN moves to HALF OPEN when the OpenToHalfOpenWait timer expires; HALF OPEN returns to CLOSED on HalfOpenMaxSuccess or back to OPEN on HalfOpenMaxFailures The whole design in one picture. Every arrow is a knob you can turn.

  • Closed is the normal state. Requests go through. Failures are counted, and if enough of them cluster together the breaker trips.
  • Open is the tripped state. Nothing is sent. Every call returns instantly with either a fallback or a 503 Circuit is open. After a cooling-off period, the breaker moves on.
  • Half open is the probe. A handful of real requests are let through. Enough successes close the circuit; enough failures open it again.

A failure here means the request couldn't be made, timed out, or came back with a 5xx. A 404 is the service working correctly, so it doesn't count against it.

KnobWhat it controlsDefault
TimeoutIntervalHow long a single request may take10s
MaxFailuresFailures within a minute before the circuit opens5
OpenToHalfOpenWaitHow long to stay open before probing30s
HalfOpenMaxSuccessSuccessful probes needed to close5
HalfOpenMaxFailuresFailed probes needed to reopen3
RetryIntervalsThe pauses between retries of one request1s, 2s, 3s, 5s, 8s

One detail I'd defend: failures only count if they arrive within a minute of each other. A dependency that throws one error an hour is not down. A breaker that trips on a slow drip of errors is a breaker somebody eventually switches off, and then you have nothing.

What It Offers

Four properties, and what each one means in the code.

Graceful recovery. The half-open state means a dependency has to earn its way back. Nobody flips the circuit closed on a hunch; a few real requests succeed, and then the traffic follows.

Configurable retry. Retries are a slice of durations, not a count. []time.Duration{500ms, 1s, 2s} is three attempts with that backoff. Want exponential? Write the numbers. Want none? Pass one interval.

Fallback. Register a function that receives the original *http.Request and returns a response. Serve from cache, return a sensible default, call a different service. While the circuit is open, callers get the fallback instantly and never touch the network.

Plug and play. You hand it an http.Request and get back a typed response that is one of three things: success, fallback, or error. Your handler switches on the type and moves on.

Flowchart of DoRequest: the state is read from Redis; if OPEN, the request goes to the fallback if one is set, otherwise returns 503; if CLOSED or HALF OPEN, the request is sent, a sub-500 status records success, a failure records a failure and sleeps the next retry interval before trying again, and exhausted retries fall back or return an error What one call to DoRequest does. The left branch, the open one, is the reason the package exists.

One Key in Redis

This is the design choice I care most about, and it's the one that makes the package fit the systems it was built for.

Most circuit breaker libraries keep their state in process memory. That's fine for one process. It's not fine for a service running as twelve replicas on Kubernetes, because now you have twelve breakers, and each one has to discover the outage on its own. Twelve replicas, each burning MaxFailures worth of timeouts before it trips, is a lot of goroutines waiting on a database you already know is down.

So the state lives in Redis, under a key named after the breaker. One replica trips the circuit; every replica reads OPEN on its next request. The outage is discovered once.

The cost is a Redis round trip per request. That's a real trade, and it's why the Redis client is passed in as a small interface you control rather than something the package spins up on your behalf.

Using It

go
1import circuitbreaker "github.com/shurutech/circuit-breaker/v1"
2
3cb := circuitbreaker.NewCircuitBreaker(circuitbreaker.Config{
4 TimeoutInterval: 5 * time.Second,
5 MaxFailures: 3,
6 OpenToHalfOpenWait: time.Minute,
7 HalfOpenMaxSuccess: 2,
8 HalfOpenMaxFailures: 1,
9 RetryIntervals: []time.Duration{500 * time.Millisecond, time.Second, 2 * time.Second},
10}, "inventory-service", redisClient)
11
12cb.SetFallbackFunc(func(req *http.Request) *circuitbreaker.CircuitBreakerResponse {
13 return &circuitbreaker.CircuitBreakerResponse{
14 HttpStatus: http.StatusOK,
15 ResponseType: circuitbreaker.Fallback,
16 Data: map[string]any{"items": cachedItems},
17 }
18})
19
20req, _ := http.NewRequest(http.MethodGet, "http://inventory/items", nil)
21resp := cb.DoRequest(req)
22
23switch resp.ResponseType {
24case circuitbreaker.Success:
25 // resp.Data holds the decoded JSON body
26case circuitbreaker.Fallback:
27 // served without touching the network
28case circuitbreaker.Error:
29 // resp.Error.Code and resp.Error.Message explain why
30}

The full example lives at examples/main.go. The name you give the breaker is the Redis key, so two services that share a dependency can share a circuit by sharing a name, or keep separate ones by not.

Rough Edges

A post about a package I wrote shouldn't read like an advertisement for it, so here is what I'd tell a colleague before they imported it.

  • Retries and the trip counter share a ledger. Every failed attempt inside one DoRequest counts toward MaxFailures. With the defaults, five retry intervals and a threshold of five, a single request against a dead service is enough to open the circuit. That's aggressive on purpose, but set the two numbers together, not separately.
  • The cool-down timer is local. The replica that trips the circuit is the one holding the timer that half-opens it. If that pod is gone before the timer fires, the key stays open until a fresh replica starts up and notices. A TTL on the Redis key would be the more honest design, and it's where I'd take the next version.
  • It's HTTP-and-JSON shaped. The package wraps http.Request and decodes JSON bodies. That's the scope it was built for, not a claim that it's the right scope for everything.

A failing dependency will take you along if you let it. Don't let it: count the failures, trip the switch, tell your callers the truth right away, and let the dependency prove it's back before you trust it again. The package is a few hundred lines that do exactly that.

Resources