Memcached failover with PHP 5

Memcached failover with PHP 5

Implementing a custom failover for memcached, in PHP environment

After having a (lot of) trouble trying to configure Memcached failover under PHP, I thought it would be useful to write down my experience and solutions.

Versions

First, I must specify that I'm running Debian stable, so at the time of writing:
- Memcached: 1.4.13
- libmemcached: 1.0.8
- PHP: 5.4.4
- Memcached extension for PHP: 2.0.1

First test

I'll use a default «$servers» variable, which should represent all your servers, for example:
$servers = array(
array('host1', 11211),
array('host2', 11211));

Important note: you must always use the same order of memcached servers, and you don't have to remove «dead servers»: else your distribution of keys (hash) will change from one script to another, and you'll never perfom with memcached as you should. So set the very same array of servers, in every script you'll use.

After reading several posts, I tried the following setup:
$memcached = new Memcached();
$memcached->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
$memcached->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$memcached->addServers($servers);

Note that «addServers()» must be called after all options are set, else the options won't apply to those servers.

With this setup, a call to a $memcached->set() won't work on the DEAD server, and always return an error. You should be aware that OPT_LIBKETAMA_COMPATIBLE changes the OPT_DISTRIBUTION value, hence it was useless.

Enabling failover

After many other posts, I come up with the following options:
$memcached->setOption(Memcached::OPT_SERVER_FAILURE_LIMIT, 5);
$memcached->setOption(Memcached::OPT_REMOVE_FAILED_SERVERS, 5);

The first one is in fact deprecated since 0.48 version of libmemcached (be careful: there's currently two instance of the variable on the page, the second one shows the deprecation).

The second one is the one you need. But I've found it strange, as it was not working when used as an integer:
- «0»: was returning an error
- «1»: made the failover active after 2 errors
- «2» or more: failover wasn't working anymore

At last, I've found everything on the libmemcached documentation page: it's a boolean value, so either true or false. When you enable it, as a default, the failover starts its job after 2 failures.

I think that the «2 failures» can be configured within libmemcached, but has no setOption() for now.

Final setup

Here's what I've set up on my servers:
$memcached = new Memcached();
$memcached->setOption(Memcached::OPT_CONNECT_TIMEOUT, 10);
$memcached->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
$memcached->setOption(Memcached::OPT_SERVER_FAILURE_LIMIT, 2);
$memcached->setOption(Memcached::OPT_REMOVE_FAILED_SERVERS, true);
$memcached->setOption(Memcached::OPT_RETRY_TIMEOUT, 1);
$memcached->addServers($servers);

Memcached::OPT_DISTRIBUTION: set it to consistent hashing. If one memcached node is dead, its keys (and only its keys) will be evenly distributed to other nodes. This is where the magic is done. This is really different from removing one server in your ->addServers() call.
Memcached::OPT_SERVER_FAILURE_LIMIT: number of connection issues before a server is marked as DEAD, and removed from the list of servers (default: 5).
Memcached::OPT_REMOVE_FAILED_SERVERS: set it to «true», to enable the removal of dead servers.
Memcached::OPT_RETRY_TIMEOUT: after a node is declared DEAD, libmemcached will try it again after that many seconds. Here I've set it to 1 second, but I'm working on PHP scripts that run for less than 100ms most of the time. That would only be useful for cron/daemonize scripts.
Memcached::OPT_CONNECT_TIMEOUT: the timeout after which a server is considered DEAD. As my servers are on the same LAN, ping is ~0.5ms, so 10ms is large enough to consider the server is DEAD. Note that you have to wait twice that time before a node is marked DEAD, so if it's 1000ms, your script will lock for 2 seconds before ignoring the DEAD server. That may affect your response times a lot, and that's why I've set it very low.

Photo by Bengt Nyman.

 
Published by 119 on December 16, 2013 at 01:17 p.m..

Comments

Abdul Jabbar WebBestow 11 year(s) 1 month(s) ago at 13:32
Abdul Jabbar WebBestow Website Very good and really helpful article. Just wanted to know that do we have to hash our keys or memcached client library will take care of it? Please attach an example of the code in php on how to store key=>value in memcached so it will clear us out that what we really need to do either we need to hash our keys in some way or what. Please, looking forward for your response.
Yvan 11 year(s) 1 month(s) ago at 14:19
119 Yvan Hello Abdul,

memcached client will hash your keys automatically. Say you have 3 servers, A, B and C and 3 keys «K1» to «K9». For example, the client hash algorithm will store as follow : K1/K2/K3 stored on A, K4/K5/K6 stored on B, K7/K8/K9 stored on C. If your server B goes down, its keys (K4/K5/K6) will be stored evenly on the 2 remaining servers (A and C). For example, K4 will go to A, and K5/K6 will go to server C.

That's just an example, not the real algorithm. You can find out which key goes on which server with the function $memcached->getServerByKey('K4') . Then make one server go down, and see what the getServerByKey() sends you after this failure.
Abdul Jabbar WebBestow 11 year(s) 1 month(s) ago at 09:37
Abdul Jabbar WebBestow Website Thanks alot Yvan for prompt response. Now We are totally clear about Memcache. Thanks alot. I will must forward others to your blog.
William Eaton 9 year(s) 2 month(s) ago at 14:49
William Eaton Fantastic instructions, I too was baffled by the options and why it didn't failover by default. Ive implemented what you suggested with a few tweaks (easy to understand thanks to your explanation) and it worked just as easily as that
Name(Mandatory field)
Email(Mandatory field)
Email(Mandatory field)
Website or blog
Characters left: 1000