KeystonePerformance

Keystone Performance

This is to track the performance work related to Keystone.

Work Items

Identify CPU, Disk, Memory, Database bottlenecks

Methodology

1. Install Havana Stable [1] on a bare metal.
2. Create one instance of m1.medium flavor and other of type m1.large, so that we can have different CPU and memory config
3. Install Keystone Manually on both of the above created instances
4. Using python multiprocessing module create users in parallel using keystoneclient.v2_0 module on each one of them.
   1. key.users.create(<user>, "test", "test@test.com") where key = client.Client( .... )
5. Collect the CPU, Disk, Memory and Database related stats while user creation is in progress.

Compare PKI vs UUID, SQL vs LDAP

Methodology

In this story we will be measuring keystone performance for those sequencial tests:

• User creation
• User token generation
• User token validation

Test setup

Devstack on ubuntu server 13.10, with most services disabled (beside keystone, mysql and ldap):

for service in g-api g-reg n-api n-crt n-obj n-cpu n-net n-cond \
                     cinder c-sch c-api c-vol n-sch n-novnc n-xvnc n-cauth \
                     horizon rabbit tempest; do
    echo disable_service $service >> localrc
done

Added at the end of localrc:

ADMIN_PASSWORD=nomoresecrete
SERVICE_TOKEN=ADMIN
OFFLINE=True

KEYSTONE_TOKEN_FORMAT=UUID
KEYSTONE_IDENTITY_BACKEND=ldap

enable_service ldap
KEYSTONE_CLEAR_LDAP=yes
LDAP_PASSWORD=ldappass

Test scripts

Used scripts can be found there: https://github.com/TristanCacqueray/keystone-perfs/tree/master/src Basically:

• User creation (perf-create-users.sh):
• Generate user's tokens list (perf-gen-tokens.sh):
• Token validation (perf-validate-tokens.sh):

Performance data acquisition

And the main script to automate the benchmark is bench.sh. This will write real time (userland + kernelland) in /tmp/perf-test_name-TOKEN_FORMAT-BACKEND:

Performances results

Note that this is a sequencial benchmarks (no parallel requests). After running the data acquisition several time, here is the average observed time in seconds for 100 items (users or tokens):

	LDAP	SQL
UUID	user creation  : 29.36 token generation: 27.74 token validation: 2.29	user creation  : 31.35 token generation: 29.73 token validation: 2.32
PKI	user creation  : 29.39 token generation: 30.40 token validation: 2.84	user creation  : 31.34 token generation: 32.27 token validation: 2.86

First remarks:

• UUID token generation is faster than PKI (obviously).
• User creation does not depends on token type, and is faster on LDAP.
• Overall, LDAP performs better, even for token validation.

The concurrency value against PKI V3

Description

In this story, we will measure the max number of PKI_token_authenticate or PKI_token_revocation_list_get requests that Keystone V3 can handle per second. And then we will try to improve the value.

The test environment

Hardware info of host:8 cores, 24G memory OS:suse11.2 Keystone log level:WARN Keystone DB:postgres sql Token format:PKI Api version:V3

Methodology

1. When server is ready, the client process will start to send sequencial requests constantly.
2. Change the number of client process to make the number of reponse reached the maximum.
3. Run the client processes 20 seconds
4. Calculate the concurrency value per second.

A single-process Pthon-only keystone server

Deployment

There are two hosts. The one used to run client processes, the other used to run keystone server and sql server.

Performances results

token_authenticate	revocation_list_get
10/s	58/s

A multi-processes Keystone server running inside an Apache container server

Deployment

There are two hosts. The one used to run client processes, the other used to run keystone server running inside Apache and sql server.

Performances results

token_authenticate	revocation_list_get
57/s	330/s

Two multi-processes Keystone servers running inside an Apache container server

Deployment

There are there hosts:client processes, keystone server running inside Apache and sql server, keystone server running inside Apache and haproxy.

Performances results

token_authenticate	revocation_list_get
105/s	560/s

There multi-processes Keystone servers running inside an Apache container server

Deployment

There are four hosts:client processes, keystone server running inside Apache, keystone server running inside Apache and sql server, keystone server running inside Apache and haproxy.

Performances results

token_authenticate	revocation_list_get
140/s	800/s

Four multi-processes Keystone servers running inside an Apache container server

Deployment

There are five hosts:client processes, keystone server running inside Apache, keystone server running inside Apache, keystone server running inside Apache and sql server, keystone server running inside Apache and haproxy.

Performances results

token_authenticate	revocation_list_get
140/s	1000/s

Next step

Through the data above we can find that the concurrency value per second does not increase when the number of hosts increased from three to four. We guess that the postgres sql is bottleneck. So, we will try to improve sql next step.