{"id":65278,"date":"2022-06-23T09:00:54","date_gmt":"2022-06-23T09:00:54","guid":{"rendered":"https:\/\/www.cryptocabaret.com\/?p=65278"},"modified":"2022-06-23T09:00:54","modified_gmt":"2022-06-23T09:00:54","slug":"how-to-use-a-circuit-breaker-pattern-for-site-reliability-engineering","status":"publish","type":"post","link":"https:\/\/www.cryptocabaret.com\/?p=65278","title":{"rendered":"How to use a circuit breaker pattern for site reliability engineering"},"content":{"rendered":"

How to use a circuit breaker pattern for site reliability engineering<\/span>
\nRobert Kimani<\/a><\/span>
\nThu, 06\/23\/2022 – 03:00<\/span><\/p>\n

\n
Register<\/a> or Login<\/a> to like<\/div>\n
Register<\/a> or Login<\/a> to like<\/div>\n<\/div>\n
\n

Distributed systems have unique requirements for the site reliability engineer (SRE). One of the best ways to maintain site reliability is to enforce a certain set of best practices. These act as guidelines for configuring infrastructure and policies.<\/p>\n

What is a circuit breaker pattern?<\/strong><\/h2>\n

A circuit breaker pattern saves your service from halting or crashing when another service is not responding.<\/p>\n

\n
\"Image<\/div>\n
Image by: <\/span><\/p>\n

(Robert Kimani, CC BY-SA 40)<\/p>\n<\/div>\n<\/article>\n

For instance, in the example image, the business logic microservices talk to the User Auth service, Account service, and Gateway service.<\/p>\n

As you can see, the “engine” of this business application is the business logic microservice. There’s a pitfall, though. Suppose the account service breaks. Maybe it runs out of memory, or maybe a back end database goes down. When this happens, the calling microservice starts to have back pressure until it eventually breaks. This results in poor performance of your web services, which causes other components to break. A circuit breaker pattern must be implemented to rescue your distributed systems from this issue.<\/p>\n

Implementing a circuit breaker pattern<\/strong><\/h2>\n

When a remote service fails, the ideal scenario is for it to fail fast. Instead of bringing down the entire application, run your application with reduced functionality when it loses contact with something it relies upon. For example, keep your application online, but sacrifice the availability of its account service. Making a design like a circuit breaker helps avoids cascading failure.<\/p>\n

Here’s the recipe for a circuit breaker design:<\/p>\n

    \n
  1. \n

    Track the number of failures encountered while calling a remote service.<\/p>\n<\/li>\n

  2. \n

    Fail (open the circuit) when a pre-defined count or timeout is reached.<\/p>\n<\/li>\n

  3. \n

    Wait for a pre-defined time again and retry connecting to the remote service.<\/p>\n<\/li>\n

  4. \n

    Upon successful connection, close the circuit (meaning you can re-establish connectivity to the remote service.) If the service keeps failing, however, restart the counter.<\/p>\n<\/li>\n<\/ol>\n

    Instead of hanging on to a poorly performing remote service, you fail fast so you can move on with other operations of your application.<\/p>\n<\/p>\n

    \n
    More DevOps resources<\/div>\n
    \n
    What is DevOps?<\/a><\/div>\n
    The ultimate DevOps hiring guide<\/a><\/div>\n
    DevOps monitoring tools guide<\/a><\/div>\n
    A guide to implementing DevSecOps<\/a><\/div>\n
    Download the DevOps glossary<\/a><\/div>\n
    eBook: Ansible for DevOps<\/a><\/div>\n
    Latest DevOps articles<\/a><\/div>\n<\/p><\/div>\n<\/p><\/div>\n

    Open source circuit breaker pattern<\/strong><\/h2>\n

    There are very specific components provided by open source to enable circuit breaker logic in your infrastructure. First, use a proxy layer, such as Istio<\/a>. Istio is a technology independent solution and uses a\u00a0 “Blackbox” approach (meaning that it sits outside your application).<\/p>\n

    Alternately, you can use Hystrix, an open source library from Netflix that’s widely and successfully used by many application teams. Hystrix gets built into your code. It’s generally accepted that Hystrix can provide more functionality and features than Istio, but it must be “hard coded” into your application. Whether that matters to your application depends on how you manage your code base and what features you actually require.<\/p>\n

    Implementing effective load balancing<\/strong><\/h2>\n

    Load balancing<\/a> is the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing as efficient as possible. Load balancing can optimize response time, and avoid unevenly overloading some computer nodes while other computer nodes are left idle.<\/p>\n

    \n
    \"Image<\/div>\n
    Image by: <\/span><\/p>\n

    (Robert Kimani, CC BY-SA 40)<\/p>\n<\/div>\n<\/article>\n

    Users on the left side talk to a virtual IP (vIP) on a load balancer. The load balancer distributes tasks across a pool of systems. Users are not aware of the back-end system, and only interact with the system through the load balancer.<\/p>\n

    Load balancing helps to ensure high availability (HA) and good performance. High availability means that failures within the pool of servers can be tolerated, such that your services remain available a high percentage of the time. By adding more servers to your pool, effective performance is increased by horizontally spreading the load (this is called horizontal scaling<\/em>.)<\/p>\n

    \u00a03 ways to do load balancing<\/strong><\/h2>\n
      \n
    1. DNS Load Balancing:<\/strong> This is the most straight forward and is free. You do not have to purchase any specialized hardware because it’s the basic functionality of DNS protocol, you’d simply return multiple A-records An A-record maps a hostname to an IP address. When the DNS server returns multiple IP addresses for a single hostname, the client will choose one IP address at random, in other words the load balancing is automatically done for you from the client.<\/li>\n
    2. \n

      Dedicated Hardware-Based Load Balancer: <\/strong>This is what is typically used within a data center, it’s feature-rich and highly performant.<\/p>\n<\/li>\n

    3. \n

      When you have a Software Load Balancer<\/strong>, you do not need to have a dedicated hardware for this type of load balancing, you can simply install the load balancer software on commodity hardware and use it for your load balancing use cases.<\/p>\n<\/li>\n<\/ol>\n

      Nginx and HAProxy are mostly used as software load balancers and can be very useful in development environments where you can quickly spin up their functionality.<\/p>\n

      DNS load balancing<\/strong><\/h2>\n

      A DNS server may return multiple IP addresses when a client looks up a hostname. This is because in a large-scale deployment, a network target isn’t just running on one server or even just on one cluster. Generally, the client software chooses an IP address at random from the addresses returned. It’s not easy to control which IP a client chooses.<\/p>\n

      Unless you have custom software built to determine the health of a server, pure DNS load balancing doesn’t know whether a server is up or down, so a client could be sent to a server that is down.<\/p>\n

      Clients also cache DNS entries by design, and it’s not always easy to clear them. Many vendors offer the ability to choose which datacenter a client gets routed to based on geographic location. Ultimately, an important way to influence how clients reach your services is through load balancing.<\/p>\n

      Dedicated load balancers<\/strong><\/h2>\n

      Load balancing can happen at Layer3 (Network), Layer4 (Transport), or Layer7 (Application) of the OSI (Open Systems Interconnection) model.<\/p>\n