Circuit Breaker Pattern 101

This blog is part of the series where we discuss 101 concepts from Ground Zero for an audience that has limited starting knowledge. This article comes in the Intermediate-Level Series since it involves understanding the of Priority Queues — used mainly for handling asynchronous messages of different priorities based on tenants/business use-cases.

Some of the earlier blogs in the 101 Series are as follows:

Priority Queues 101
API 101
Async Communication 101
Databases 101
Database Design 101
Deployment Strategies 101
Ambassador Pattern 101

What is a Circuit Breaker?

Circuit Breaker is a design pattern used in distributed systems to prevent non-transient failures at one tier of the application from bottling up resources at other tiers of the application hierarchy. Non-transient failure events such as maintenance, slow network connections, timeouts etc which persistently can lead to remote calls terminating without a desired response are ideal cases for implementing circuit breakers. Asymmetric to retry pattern used in in case of transient failures, where a client makes successive calls (with/without the exponential back-off), this pattern is essential to preventing Denial of Service scenarios in case of a multi-tenant platform architecture. Unblocking resources by opening a circuit for long terms failures, helps to optimise on the system resources such as memory, threads, database connections, thus preventing any further failures in downstream systems.

End 2 End Flow

Circuit breakers are usually implemented as application proxy which monitors the remote calls failures and then allows the subsequent operations/actions. The proxy is usually configured to run as a simple state machine as depicted below —

Closed State

• In a Closed State, all the remote calls to downstream services are allowed as expected. The proxy counts the occurrences of failures/exception and once reaches certain threshold, the state changes to an Open State.

Open State

• In an Open State, all remote calls fail intermittently with a pre-defined exception and all calls to downstream are blocked until the set timeout expires. Post timeout expiry, the circuit goes to a Half Open state.

Half Open State

• In a Half Open State, a percentage of remote calls to downstream micro-services are allowed by the circuit. In case of further failures, the circuit moves back to an Open state and if the successful threshold is hit, the circuit moves back to healthy Closed state.

Use Cases

• Multi-Tenant — Large Scale systems where the downstream platform service such as Identity, Authentication etc is shared across multiple Business domains.
• Single Tenant — Micro-services within the same tenant which have high visibility in terms of stricter performance and availability and hence circuits help maintain the desired end to end SLA.

Advantages

• The primary objective of this pattern is to isolate Single Point of Failures cascading into large, distributed failures affecting the overall availability of entire Business Domain of multiple micro-services.
• Optimise system resources (threads, memory, CPU etc) by reducing unnecessary retires and exhaustion at the time of transient failures.
• Shielding the affected downstream system and helping reduce the overall Mean Time to Recover ( MTTR) by avoiding flow of traffic in failure state.

Summary

We discussed basics of the Circuit Breaker Design Pattern which is popular for distributed systems with high availability and millisecond performance SLA’s. Netflix through its open sourced library Hystrix had solved some the major heavy-lifting integration needs for the clients to implement on their end. Later on Hystrix was deprecated and replaced by Resilience4j a Spring Library.

We will talk about the working of the library with a code example in a future Advanced blog, to solve for an actual use-case.

For feedback, please drop a message to amit[dot]894[at]gmail[dot]com or reach out to any of the links at https://about.me/amit_raj.