Circuit Breaker

The Circuit Breaker pattern is a design approach used to prevent a system from repeatedly trying to execute an operation that is likely to fail. It wraps a protected function call (like a call to a remote service or another component) in a circuit breaker object, which monitors failures.

In the context of actor systems, this pattern can be applied to an actor that needs to send messages to another actor or external service which might be unreliable. Instead of the calling actor directly sending messages and potentially blocking or consuming resources on repeated failures, a "circuit breaker" logic is incorporated, often within the sending actor itself or a dedicated intermediary actor.

The circuit breaker typically operates in three states:

  1. Closed

    The default state. Messages are sent to the target actor/service. The breaker monitors for failures (e.g., timeouts, error responses). If the failure rate or count exceeds a threshold within a certain time window, the breaker trips and transitions to the Open state.

  2. Open

    The breaker immediately fails messages without attempting to send them to the target. It might return an error or a fallback result. This prevents overwhelming the failing target and saves resources. After a configured timeout period, the breaker transitions to the Half-Open state.

  3. Half-Open

    A limited number of trial messages are allowed through to the target. If these trial messages succeed, the breaker assumes the target has recovered and transitions back to the Closed state. If any of these trial messages fail, the breaker assumes the target is still unhealthy and immediately transitions back to the Open state, typically resetting or increasing the timeout period.

Implementing a circuit breaker involves adding state management within an actor's message processing logic. An actor might hold variables for the current state (Closed, Open, Half-Open), a failure counter, a success counter (for Half-Open state), and a timestamp for state transitions. When processing an outgoing message, the actor checks its circuit breaker state before attempting delivery. Incoming error or success messages from the target update the state. Timers can be managed through scheduled messages within the actor system itself.