Why Your PHP app NEEDS a Circuit Breaker

April 8, 2009

More and more websites these days are relying on external resources for data and functionality. Each one of these services introduces a stability hole in your application. You're now relying on the uptime of an external resource. Yahoo, Google, etc.. ALL have downtime at some point. No one has a 100% SLA. This means that more than ever you need to code defensively to prevent external resources from tarnishing the user experience of your site. This is where the Circuit Breaker Stability Pattern comes into play (From the book "Release It").

The Circuit Breaker Pattern is a simple concept. If too many connections fail while accessing a resource the circuit is "opened" or stopped at that point. After a certain period of time we let one connection go through, if that one works the circuit is re-closed and operations resume as normal. Let's break this down further.

Your site connects to Google to get a list of widgets. The Google service goes down and now your customer is stuck in a timeout waiting for the connection to fail. You have no idea this is happening. You think everything is fine, until people get tired of waiting, your connections pool up and you run out of resources on your machines while 10,000 people are waiting 30 seconds to timeout. This makes your site look horrible to the user and they find somewhere else to query for these widgets. The goal is that if something is constantly failing you stop calling it and notify administrators and users.

The Circuit Breaker pattern defines a max number of failures (threshold) that can occur before the circuit is tripped. So let's say we want to stop connecting to Google after 5 people's connections have failed in a row. Now we can immediately return responses back to our users letting them know the service is down or they're query has been stored for processing at a later time. An informed user is a happy user, even if something does work they like to know about it immediately. After 10 minutes or so we want to open up ONE connection to test the waters to see if Google is still down so we open the circuit "halfway". If that connections fails we reopen the circuit and await the next timeout. Once a connection is successful again we re-close the circuit and operations can resume as normal. When a circuit does open we want to alert our operations group or the site admin so they know there is a problem with this service.

This allows you to automatically shut off and re-instate services if failures occur. It's a very powerful concept in increasing the stability of your system.

I've created a very simple, sample Circuit Breaker class in PHP that can be used for this purpose. It currently assumes a mysql database to store this information but you can replace it with a memcached or other in-memory caching option for increased performance. You can find all the code and samples here: http://code.google.com/p/plushcode/source/browse/#svn/trunk/stability_patterns/circuit_breaker

The schema is listed out below for the table

CREATE TABLE IF NOT EXISTS `circuit_breaker` (
`id` int(5) NOT NULL auto_increment,
`app_id` varchar(255) NOT NULL,
`failure_count` int(5) NOT NULL,
`threshold` int(5) NOT NULL,
`state` enum('open','closed','half') NOT NULL,
`timeout` int(5) NOT NULL COMMENT 'timeout in seconds',
`last_open_time` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;


Let's start going through the fields....
app_id - Each external resource you have (including databases!) should get it's own application id that is unique. Something such as "google_widgets" works fine.

failure_count - Each time a failure occurs this is incremented by 1. Once a successful call has been made it is reset to 0

threshold - The number of times a service can fail before we open the circuit and stop all processing

state - The current state of the service. It's either closed (normal), open(failure occurred) or half(we're trying one more time)

timeout - The number of seconds to wait before attempting a connection again

last_open_time - A timestamp when the circuit was last opened which would indicate a failure


One of the nice aspects to having this in the database is that you can easily monitor all of your external services from one site. All you have to do is query the table and you can see exactly what the health of your system is. Here is a quick and dirty sample report http://www.litfuel.net/tutorials/cb_report.htm

To use this class you simply call it as seen below:
  1.  
  2.  
  3. require_once('CircuitBreaker.php');
  4.  
  5.  
  6. $cb = new CircuitBreaker('myapp');
  7. $cb->logging = true;
  8.  
  9. if($cb->isClosed()) {
  10. if(!$res = @file_get_contents("http://litfuel.net/tutorials/testserver.php")) {
  11. $cb->fail();
  12. } else {
  13. $cb->success();
  14. }
  15. var_dump($res);
  16. } else {
  17. // notify the user, send them to a different service or anything!
  18. echo 'CIRCUIT TRIPPED! NO CALLS MADE';
  19. return false;
  20. }
  21.  


What we're doing is including the CircuitBreaker class, then we're making sure the circuit is closed which means everything is A-OK. We make a web services call over to my test server page. If that connection fails then we call $cb->fail which then tests to see if our failure count is over our threshold and if not, increases it by one. If we are over the threshold it will open the circuit and all future calls will return false when isClosed() is called. If the connection is successful then we call $cb->success() which resets the failures back to 0.

To test this out:
1.create a test database
2.import the schema into the new database
3.enter in one record called "myapp" as the app_id, put in 5 as the threshold and 10 as the timeout value
4.paste the Test.php code into your own file
5.When you run the code you should see a var dump of some xml data which indicates success
6.Now change the url in the file_get_contents line to a url that does not exist
7.Start to refresh your page and you start to see the connections fail and then the circuit trips, after 10 seconds it opens up halfway, does one test then reopens for another 10 seconds



Comments

RSS feed for comments on this post.

  1. Louis Vuitton Ha says:
    January 29, 2010 @ 21:17 — Reply

    Comment pending moderation

  2. computer science degree says:
    April 10, 2010 @ 10:32 — Reply

    Comment pending moderation

  3. World Cup Bets says:
    May 4, 2010 @ 12:05 — Reply

    Comment pending moderation

  4. LEED Certification India says:
    May 5, 2010 @ 02:01 — Reply

    Comment pending moderation

  5. usa casinos says:
    May 29, 2010 @ 05:34 — Reply

    Comment pending moderation

  6. online forex trading says:
    May 5, 2010 @ 04:01 — Reply

    Comment pending moderation

  7. wow gold says:
    May 17, 2010 @ 09:59 — Reply

    Comment pending moderation

  8. forex trade says:
    May 19, 2010 @ 04:08 — Reply

    Comment pending moderation

  9. low cost health insurance says:
    May 19, 2010 @ 04:09 — Reply

    Comment pending moderation

  10. fire damage says:
    May 19, 2010 @ 10:54 — Reply

    Comment pending moderation

  11. Gambling Info says:
    May 19, 2010 @ 20:48 — Reply

    Comment pending moderation

  12. seo keyword says:
    May 31, 2010 @ 02:27 — Reply

    Comment pending moderation

  13. cheap replica watches says:
    June 1, 2010 @ 19:15 — Reply

    Comment pending moderation

  14. broward county inmate search says:
    June 1, 2010 @ 20:07 — Reply

    Comment pending moderation

  15. free arrest records says:
    June 2, 2010 @ 19:38 — Reply

    Comment pending moderation

  16. electric towel rails says:
    June 2, 2010 @ 20:04 — Reply

    Comment pending moderation

  17. nc doc inmate search says:
    June 2, 2010 @ 20:59 — Reply

    Comment pending moderation

  18. www.towelradiators.net says:
    June 2, 2010 @ 21:00 — Reply

    Comment pending moderation

  19. Online Slots says:
    June 5, 2010 @ 08:23 — Reply

    Comment pending moderation

  20. Louis VUITTON OutleT says:
    June 9, 2010 @ 21:19 — Reply

    Comment pending moderation

  21. focal speakers says:
    June 10, 2010 @ 13:05 — Reply

    Comment pending moderation

  22. Hot drinks vending London says:
    June 14, 2010 @ 09:53 — Reply

    Comment pending moderation

  23. glass doors says:
    June 14, 2010 @ 10:10 — Reply

    Comment pending moderation

  24. dollhouse miniatures says:
    June 15, 2010 @ 11:39 — Reply

    Comment pending moderation

  25. West Des Moines Chiropractor says:
    June 15, 2010 @ 12:18 — Reply

    Comment pending moderation

  26. Louis Vuitton handbags says:
    June 16, 2010 @ 00:35 — Reply

    Comment pending moderation

  27. christian louboutin shoes says:
    June 16, 2010 @ 08:43 — Reply

    Comment pending moderation

  28. muscle building protein foods says:
    June 17, 2010 @ 01:27 — Reply

    Comment pending moderation

  29. therapy greenville sc says:
    June 18, 2010 @ 10:31 — Reply

    Comment pending moderation

  30. health insurance quotes says:
    June 20, 2010 @ 09:49 — Reply

    Comment pending moderation

  31. quantum swing trader review says:
    June 27, 2010 @ 01:37 — Reply

    Comment pending moderation

Leave a Comment

Line and paragraph breaks automatic, HTML allowed: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <code> <em> <i> <strike> <strong>

Comments disabled due to spammers being losers that lead sad lives.