Revision as of 17:11, 16 December 2013

Keystone Performance

This is to track the performance work related to Keystone.

Work Items

Identify CPU, Disk, Memory, Database bottlenecks

Test #1, Create users in parallel and look for CPU, disk or memory bottleneck.

Methodology

1. Install RDO Havana Stable [1] on a bare metal.
2. Create one instance of m1.medium flavor and other of type m1.large, so that we can have different CPU and memory config
3. Install Keystone Manually (from RDO release) on both of the above created instances
4. Using python multiprocessing module create users in parallel using keystoneclient.v2_0 module on each one of them.
   1. key.users.create(<user>, "test", "test@test.com") where key = client.Client( .... )
5. Collect the CPU, Disk, Memory and Database related stats while user creation is in progress.

Effect of caching - memcached

Effect of expired tokens

Overhead of syncing revocation list

Improvement with multi-core keystone service

Compare PKI vs UUID, SQL vs LDAP

Methodology

In this story we will be measuring keystone performance for those sequencial tests:

• User creation
• User token generation
• User token validation

Test setup

Devstack on ubuntu server 13.10, with most services disabled (beside keystone, mysql and ldap):

for service in g-api g-reg n-api n-crt n-obj n-cpu n-net n-cond \
                     cinder c-sch c-api c-vol n-sch n-novnc n-xvnc n-cauth \
                     horizon rabbit tempest; do
    echo disable_service $service >> localrc
done

Added at the end of localrc:

ADMIN_PASSWORD=nomoresecrete
SERVICE_TOKEN=ADMIN
OFFLINE=True

KEYSTONE_TOKEN_FORMAT=UUID
KEYSTONE_IDENTITY_BACKEND=ldap

enable_service ldap
KEYSTONE_CLEAR_LDAP=yes
LDAP_PASSWORD=ldappass

Test scripts

User creation (perf-create-users.sh):

#!/bin/sh

export OS_SERVICE_TOKEN=ADMIN
export OS_SERVICE_ENDPOINT=http://localhost:35357/v2.0
for i in $(seq 100); do
    keystone user-create --name perf_testuser_$(printf "%04d" $i) \
                         --tenant demo --pass demopass 2>&1 > /dev/null
    if [ "$?" != 0 ]; then
        echo "User creation failed"
        break
    fi
done

Generate user's tokens list (perf-gen-tokens.sh):

#!/bin/sh

get_id () {
    echo `"$@" | awk '/ id / { print $4 }'`
}

unset OS_SERVICE_TOKEN OS_SERVICE_ENDPOINT
export OS_AUTH_URL=http://localhost:5000/v2.0
export OS_TENANT_NAME=demo
export OS_PASSWORD=demopass
echo -n > /tmp/tokens_list.txt
for i in $(seq 100); do
    export OS_USERNAME=perf_testuser_$(printf "%04d" $i)·
    get_id keystone token-get >> /tmp/tokens_list.txt
    if [ "$?" != 0 ]; then
        echo "Token get failed"
        break
    fi
done

Token validation (perf-validate-tokens.sh):

#!/bin/sh

get_id () {
    echo `"$@" | awk '/ id / { print $4 }'`
}
unset OS_SERVICE_TOKEN OS_SERVICE_ENDPOINT
export OS_AUTH_URL=http://localhost:5000/v2.0
export OS_TENANT_NAME=demo
export OS_PASSWORD=nomoresecrete
export OS_USERNAME=admin
ADMIN_TOKEN=$(get_id keystone token-get)
while read USER_TOKEN; do
    curl -H "X-Auth-Token:${ADMIN_TOKEN}" http://0.0.0.0:5000/v2.0/tokens/${USER_TOKEN} \
        2>&1 | grep "issued_at" > /dev/null
    if [ "$?" != 0 ]; then
        echo "Token validation failed"
        break
    fi
done < /tmp/tokens_list.txt

Performance data acquisition

This script will write real time (userland + kernelland) in /tmp/perf-test_name-TOKEN_FORMAT-BACKEND:

#!/bin/sh

cd ~/devstack
for TOKEN_FORMAT in UUID PKI; do
    for BACKEND in ldap sql; do
        echo "== ${TOKEN_FORMAT} - ${BACKEND} =="
        echo "[+] Unstack // stack..."
        ./unstack.sh

        sed -e "s/^KEYSTONE_TOKEN_FORMAT=.*/KEYSTONE_TOKEN_FORMAT=${TOKEN_FORMAT}/"     \
            -e "s/^KEYSTONE_IDENTITY_BACKEND=.*/KEYSTONE_IDENTITY_BACKEND=${BACKEND}/"  \
            -i localrc
        if [ "${BACKEND}" = "ldap" ]; then
            sed -e "s/^disable_service ldap$/enable_service ldap/" -i localrc
        else
            sed -e "s/^enable_service ldap$/disable_service ldap/" -i localrc
        fi
        ./stack.sh 2>&1 > /dev/null
        if [ "$?" != 0 ]; then
            echo "Stack.sh failed"
            exit 1
        fi

        echo "[+] Create users..."
        PERF_OUTPUT="/tmp/perf-create-users_${TOKEN_FORMAT}_${BACKEND}"
        /usr/bin/time -o ${PERF_OUTPUT} -f "%e" -a ~/bin/perf-create-users.sh
        echo -n "-> "; tail -n 1 ${PERF_OUTPUT}
        sleep 1

        echo "[+] Generate tokens..."
        PERF_OUTPUT="/tmp/perf-gen-tokens_${TOKEN_FORMAT}_${BACKEND}"
        /usr/bin/time -o ${PERF_OUTPUT} -f "%e" -a ~/bin/perf-gen-tokens.sh
        echo -n "-> "; tail -n 1 ${PERF_OUTPUT}
        sleep 1

        echo "[+] Validate tokens..."
        PERF_OUTPUT="/tmp/perf-validate-tokens_${TOKEN_FORMAT}_${BACKEND}"
        /usr/bin/time -o ${PERF_OUTPUT} -f "%e" -a ~/bin/perf-validate-tokens.sh
        echo -n "-> "; tail -n 1 ${PERF_OUTPUT}
        sleep 1
    done
done

Performances results

Note that this is a sequencial benchmarks (no parallel requests). After running the data acquisition several time, here is the average observed time in seconds for 100 items (users or tokens):

user creation   : 29.36
token generation: 27.74
token validation:  2.29

user creation   : 31.35
token generation: 29.73
token validation:  2.32 

user creation   : 29.39
token generation: 30.40
token validation: 2.84

user creation   : 31.34
token generation: 32.27
token validation:  2.86

First remarks:

• UUID token generation is faster than PKI (obviously).
• User creation does not depends on token type, and is faster on LDAP.
• Overall, LDAP performs better, even for token validation.