Swift/MemcachedConnPool

Memcached Conn Pool - Behaviors and Issues

In the Swift code base prior to Chuck Thier's commit, https://review.openstack.org/45134, connections to memcached servers could grow unbounded in code using the swift/common/memcached.py module. Chuck's commit introduced the concept of a connection pool, with a configured maximum number of connections.

This allows administrators to calculate how many concurrent connections their memcached server configurations can handle, and divide that up amongst the nodes and worker processes (typically proxy server) available.

For example, if one were administrating 4 memcached servers, each able to handle 64 concurrent connections, with 4 proxy servers nodes, each with 4 proxy server workers, then the max_memcached_connections parameter would be set to 4, allowing each worker process on a node to have 4 connections to a memcached server (4 connections * 4 workers * 4 nodes = 64).

It is left to the administrator to configure the proper max_memcached_connections parameter value that works well in their environment.

Problems

There is an existing problem with Chuck's patch above, as documented by https://code.launchpad.net/bugs/1235027, where connections are created beyond the configured maximum number of connections, and there is no bound on their creation. The reason these extra connections are created stems from an assumption in the code that any Exception or Timeout exception encountered while trying to "get" a connection to use must mean a connection to a memcached server is bad, and a new one must be created. It turns out that the code would not only time how long it took for a connect() system call to return a connection, but also timed how long the calling greenlet waited to retrieve a connection from the pool. Since it unconditionally created a place holder for a new connection to be created later when the retrieval of a connection took too long (even though all connections to a memcache server had been created), these new connections would grow unbounded.

The fix, as proposed by https://review.openstack.org/49739, removes the leak by simply not timing out on the connection retrieval, connection creation still times out.

However, there is a concern that if connection retrieval has no timeout, then request service times can drop precipitously if connection retrieval takes too long.

Scenarios of Concern

There seem to be three scenarios which we need to consider for the reason that connection retrieval times out:

Case #1: The memcached server is slow, where operations are timing out

In this scenario, if a given operation times out, the connection to the memcache server will be closed, and a place holder created to allow the next connection attempt to create a new connection. If errors for a given server reach a configured threshold, then that server is not considered for a configured time period, and future operations will skip this server.

However, all the greenlets waiting to connect to this server will still attempt to create connections when they see the place holder, and go on to execute their operation, moving to the next server when their operation subsequently fails due a timeout.

With a timeout on connection retrieval

Having a separate timeout for just the retrieval operation will cause that server to be skipped before the error limit is reached, currently (without further code changes). This can cause the cache to be in-effective, resulting in more backend acct/cont/obj operations.

Without a timeout on connection retrieval

By removing the Timeout entirely, as proposed in 50031, once the error limit has been reached (10 errors with-in 1 min), the memcached server will no longer be considered for the error-limit time period (1 min).

Case #2: The memcached server is slow, where operations are NOT timing out

In the case where a memcached server is slow, but not slow enough to timeout memcache operations, the service times of requests which involve that memcached server will be slowed as well.

With a timeout on connection retrieval

When there is a timeout placed on the connection retrieval, only when there are sufficiently more requests requiring connections will the slow memcached server be detected

Without a timeout on connection retrieval

When there is NO timeout placed on connection retrieval, there will be no possibility of detecting a slow server, and so requests will continue to be slow.

Case #3: The memcached server is NOT slow, but there are many more concurrent requests queued waiting for a connection than there are available connections

In the case where there are many requests queued waiting for a connection to a memcached server, it is likely then, given a fairly uniform distribution over the memcache key space, that all memcache servers are experiencing the same problem. If the key space is not uniform, then there is not much that can be done to address this problem.