KeystonePerformance
Contents
- 1 Keystone Performance
- 1.1 Work Items
- 1.1.1 Identify CPU, Disk, Memory, Database bottlenecks
- 1.1.2 Effect of caching - memcached
- 1.1.3 Effect of expired tokens
- 1.1.4 Overhead of syncing revocation list
- 1.1.5 Improvement with multi-core keystone service
- 1.1.6 Compare PKI vs UUID, SQL vs LDAP
- 1.1.7 The concurrency value against PKI V3
- 1.1.7.1 Description
- 1.1.7.2 The test environment
- 1.1.7.3 Methodology
- 1.1.7.4 A single-process Pthon-only keystone server
- 1.1.7.5 A multi-processes Keystone server running inside an Apache container server
- 1.1.7.6 Two multi-processes Keystone servers running inside an Apache container server
- 1.1.7.7 There multi-processes Keystone servers running inside an Apache container server
- 1.1.7.8 Four multi-processes Keystone servers running inside an Apache container server
- 1.1.7.9 Next step
- 1.1 Work Items
Keystone Performance
This is to track the performance work related to Keystone.
Work Items
Identify CPU, Disk, Memory, Database bottlenecks
Test #1, Create users in parallel and look for CPU, disk or memory bottleneck.
Methodology
- Install Havana Stable [1] on a bare metal.
- Create one instance of m1.medium flavor and other of type m1.large, so that we can have different CPU and memory config
- Install Keystone Manually on both of the above created instances
- Using python multiprocessing module create users in parallel using keystoneclient.v2_0 module on each one of them.
- key.users.create(<user>, "test", "test@test.com") where key = client.Client( .... )
- Collect the CPU, Disk, Memory and Database related stats while user creation is in progress.
Effect of caching - memcached
Effect of expired tokens
Overhead of syncing revocation list
Improvement with multi-core keystone service
Compare PKI vs UUID, SQL vs LDAP
Methodology
In this story we will be measuring keystone performance for those sequencial tests:
- User creation
- User token generation
- User token validation
Test setup
Devstack on ubuntu server 13.10, with most services disabled (beside keystone, mysql and ldap):
for service in g-api g-reg n-api n-crt n-obj n-cpu n-net n-cond \ cinder c-sch c-api c-vol n-sch n-novnc n-xvnc n-cauth \ horizon rabbit tempest; do echo disable_service $service >> localrc done
Added at the end of localrc:
ADMIN_PASSWORD=nomoresecrete SERVICE_TOKEN=ADMIN OFFLINE=True KEYSTONE_TOKEN_FORMAT=UUID KEYSTONE_IDENTITY_BACKEND=ldap enable_service ldap KEYSTONE_CLEAR_LDAP=yes LDAP_PASSWORD=ldappass
Test scripts
Used scripts can be found there: https://github.com/TristanCacqueray/keystone-perfs/tree/master/src Basically:
- User creation (perf-create-users.sh):
- Generate user's tokens list (perf-gen-tokens.sh):
- Token validation (perf-validate-tokens.sh):
Performance data acquisition
And the main script to automate the benchmark is bench.sh. This will write real time (userland + kernelland) in /tmp/perf-test_name-TOKEN_FORMAT-BACKEND:
Performances results
Note that this is a sequencial benchmarks (no parallel requests). After running the data acquisition several time, here is the average observed time in seconds for 100 items (users or tokens):
LDAP | SQL | |
UUID | user creation : 29.36 token generation: 27.74 token validation: 2.29 |
user creation : 31.35 token generation: 29.73 token validation: 2.32 |
PKI | user creation : 29.39 token generation: 30.40 token validation: 2.84 |
user creation : 31.34 token generation: 32.27 token validation: 2.86 |
First remarks:
- UUID token generation is faster than PKI (obviously).
- User creation does not depends on token type, and is faster on LDAP.
- Overall, LDAP performs better, even for token validation.
The concurrency value against PKI V3
Description
In this story, we will measure the max number of PKI_token_authenticate or PKI_token_revocation_list_get requests that Keystone V3 can handle per second. And then we will try to improve the value.
The test environment
Hardware info of host:8 cores, 24G memory OS:suse11.2 Keystone log level:WARN Keystone DB:postgres sql Token format:PKI Api version:V3
Methodology
- When server is ready, the client process will start to send sequencial requests constantly.
- Change the number of client process to make the number of reponse reached the maximum.
- Run the client processes 20 seconds
- Calculate the concurrency value per second.
A single-process Pthon-only keystone server
Deployment
There are two hosts. The one used to run client processes, the other used to run keystone server and sql server.
Performances results
token_authenticate | revocation_list_get |
10/s | 58/s |
A multi-processes Keystone server running inside an Apache container server
Deployment
There are two hosts. The one used to run client processes, the other used to run keystone server running inside Apache and sql server.
Performances results
token_authenticate | revocation_list_get |
57/s | 330/s |
Two multi-processes Keystone servers running inside an Apache container server
Deployment
There are there hosts:client processes, keystone server running inside Apache and sql server, keystone server running inside Apache and haproxy.
Performances results
token_authenticate | revocation_list_get |
105/s | 560/s |
There multi-processes Keystone servers running inside an Apache container server
Deployment
There are four hosts:client processes, keystone server running inside Apache, keystone server running inside Apache and sql server, keystone server running inside Apache and haproxy.
Performances results
token_authenticate | revocation_list_get |
140/s | 800/s |
Four multi-processes Keystone servers running inside an Apache container server
Deployment
There are five hosts:client processes, keystone server running inside Apache, keystone server running inside Apache, keystone server running inside Apache and sql server, keystone server running inside Apache and haproxy.
Performances results
token_authenticate | revocation_list_get |
140/s | 1000/s |
Next step
Through the data above we can find that the concurrency value per second does not increase when the number of hosts increased from three to four. We guess that the postgres sql is bottleneck. So, we will try to improve sql next step.