Jump to: navigation, search

Difference between revisions of "Manila/docs/HOWTO use tempest with manila"

< Manila‎ | docs
(Utilisation of resources)
 
(17 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 +
'''WARNING: Information on this page may be obsolete due to changes in tempest and manila_tempest_tests. Refer to the developer documentation here instead: https://docs.openstack.org/manila/ocata/devref/tempest_tests.html'''
 +
 +
 
=Foreword=
 
=Foreword=
This page describes how can be used Tempest project for testing purposes of Manila project.
+
This page describes how Tempest project for Manila project testing can be used.
  
It is assumed that person who reads this is acquainted with, at least, common information about Tempest:
+
It is assumed that user who reads this is acquainted with, at least, common information about Tempest:
 
  http://docs.openstack.org/developer/tempest/
 
  http://docs.openstack.org/developer/tempest/
 
  https://github.com/openstack/tempest
 
  https://github.com/openstack/tempest
  
Also, it is assumed, that test environment has next projects installedː
+
Since Liberty release Manila is using tempest plugins which are documented here:
  cinder
+
 
  glance
+
http://docs.openstack.org/developer/tempest/plugin.html
  keystone
+
 
  manila
+
Also, it is assumed, that the following projects has been installed in test environmentː
  neutron
+
  - cinder
  nova
+
  - glance
 +
  - keystone
 +
  - manila
 +
  - neutron
 +
  - nova
  
=Where does Tempest tests for Manila live?=
+
=Where do Tempest tests for Manila live?=
 
Manila stores its tests for Tempest in its repo here:
 
Manila stores its tests for Tempest in its repo here:
  https://github.com/stackforge/manila/tree/master/contrib/tempest
+
  https://github.com/openstack/manila/tree/master/manila_tempest_tests
 +
 
 +
Since Manila is using the Tempest plugin mechanism all test are automatically registered by running setup.py.
 +
In case devstack is used it is fully sufficient to just enable the Manila devstack plugin.
 +
 
 +
You can list all Manila tests by browsing to the Tempest source directory and execute the following command:
 +
 
 +
  testr list-tests | grep manila_tempest_tests
  
It is pluggable-like files without requirements to change core Tempest files.
+
To see if the plugin is registered correclty you can run:
But the way of its pluggability is work-around for Tempest, which hasn't pluggable functionality for exceptions, config and clients modules.
 
It is compatible with master branch of Tempest.
 
  
Just copy these files to Tempest dirː
+
  echo 'import pkg_resources; print list(pkg_resources.iter_entry_points("tempest.test_plugins"))' | python
$ TEMPEST_DIR=/path/to/tempest
 
$ MANILA_DIR=/path/to/manila
 
$ cp ${MANILA_DIR}/contrib/tempest/tempest/* ${TEMPEST_DIR}/tempest
 
  
 
=Configuring Tempest before running tests=
 
=Configuring Tempest before running tests=
Any test run with Tempest requires update of Tempest's config file located in
+
Any test running with Tempest requires update of Tempest's config file located in
 
  etc/tempest.conf
 
  etc/tempest.conf
If it does not exist, just copy template "tempest.conf.sample"
+
 
 +
If the tempest plugin is installed you can use the following command to generate all config options:
 +
oslo-config-generator --config-file tools/config/config-generator.tempest.conf
 +
 
 +
Just copy template "tempest.conf.sample" and configure the needed options:
 
  cp tempest.conf.sample tempest.conf
 
  cp tempest.conf.sample tempest.conf
  
Manila's config extension is defined in
+
Subsections below describe sections and options that can be updated with Tempest's config file.
https://github.com/stackforge/manila/blob/master/contrib/tempest/tempest/config_share.py
 
 
 
Subsections below describes sections and options that can be updated with Tempest's config file.
 
  
 
==Cluster related options in general==
 
==Cluster related options in general==
Below described required options for Manila tests
+
Required options for Manila tests are described below.
  
 
===Section "cli"===
 
===Section "cli"===
Line 50: Line 60:
  
 
====Option "uri"====
 
====Option "uri"====
Using localhost it looks like nextː
+
If using localhost it should look likeː
 
  http://127.0.0.1:5000/v2.0/
 
  http://127.0.0.1:5000/v2.0/
 
Used for all tests, if v2 auth enabled.
 
Used for all tests, if v2 auth enabled.
  
 
====Option "uri_v3"====
 
====Option "uri_v3"====
Using localhost it looks like nextː
+
If using localhost it should look likeː
 
  http://127.0.0.1:5000/v3/
 
  http://127.0.0.1:5000/v3/
  
 
====Option "username"====
 
====Option "username"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  demo
 
  demo
 
====Option "password"====
 
====Option "password"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  secrete
 
  secrete
 
Password can be redefined, just put here password from 'demo' user.
 
Password can be redefined, just put here password from 'demo' user.
  
 
====Option "tenant_name"====
 
====Option "tenant_name"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  demo
 
  demo
  
 
====Option "alt_username"====
 
====Option "alt_username"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  alt_demo
 
  alt_demo
  
 
====Option "alt_password"====
 
====Option "alt_password"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  secrete
 
  secrete
 
Password can be redefined, just put here password from 'alt_demo' user.
 
Password can be redefined, just put here password from 'alt_demo' user.
  
 
====Option "alt_tenant_name"====
 
====Option "alt_tenant_name"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  alt_demo
 
  alt_demo
  
 
====Option "admin_username"====
 
====Option "admin_username"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  admin
 
  admin
  
 
====Option "admin_password"====
 
====Option "admin_password"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  secrete
 
  secrete
 
Password can be redefined, just put here password from 'admin' user.
 
Password can be redefined, just put here password from 'admin' user.
  
 
====Option "admin_tenant_name"====
 
====Option "admin_tenant_name"====
Using devstack installation it should have next valueː
+
If using devstack installation it should have the following valueː
 
  admin
 
  admin
  
 
===Section "network"===
 
===Section "network"===
 
====Option "public_network_id"====
 
====Option "public_network_id"====
Used for creation of isolated tenants. Its value can be got by
+
This option is used for creation of isolated tenants. Its value can be retrieved from:
 
  $ neutron net-list
 
  $ neutron net-list
  
 
====Option "public_router_id"====
 
====Option "public_router_id"====
Used for creation of isolated tenants. Its value can be got by
+
This option is used for creation of isolated tenants. Its value can be retrieved from:
 
  $ neutron router-list
 
  $ neutron router-list
  
Line 108: Line 118:
  
 
===Section "service_available"===
 
===Section "service_available"===
This section is defined withinin vanila Tempest. But "manila" option is defined within Manila's extension.
+
This section is defined within vanila Tempest. But "manila" option is defined within Manila's extension.
 
====Option "manila"====
 
====Option "manila"====
 
  typeː boolean
 
  typeː boolean
Line 130: Line 140:
 
  typeː boolean
 
  typeː boolean
 
  default valueː True
 
  default valueː True
  descriptionː This option used to determine backend driver type, multitenant driver uses share-networks, but single-tenant doesn't.
+
  descriptionː This option is used to determine backend driver type, multitenant driver uses share-networks, but single-tenant doesn't.
  
 
  Current singletenant drivers: LVM, GlusterFS, NetApp 7-mode
 
  Current singletenant drivers: LVM, GlusterFS, NetApp 7-mode
Line 139: Line 149:
 
  typeː list
 
  typeː list
 
  default valueː nfs,cifs
 
  default valueː nfs,cifs
  descriptionː First value of list is protocol by default, items of list show enabled protocols at all.
+
  descriptionː The first value of the list is protocol by default, items of the list show enabled protocols.
  
 
  Expected values are:
 
  Expected values are:
Line 150: Line 160:
 
  typeː list
 
  typeː list
 
  default valueː nfs,cifs
 
  default valueː nfs,cifs
  descriptionː Whether allow run of ip rules tests or not, and which share protocols should be used.
+
  descriptionː Whether allow running of ip rules tests or not, and which share protocols should be used.
  
 
  Expected values are:
 
  Expected values are:
Line 161: Line 171:
 
  typeː list
 
  typeː list
 
  default valueː (empty)
 
  default valueː (empty)
  descriptionː Whether allow run of sid (username/usergroup) rules tests or not, and which share protocols should be used.
+
  descriptionː Whether allow running of sid (username/usergroup) rules tests or not, and which share protocols should be used.
  
 
  Expected values are:
 
  Expected values are:
Line 169: Line 179:
 
  cifs,nfs - sid rules will be tested with both share protocols
 
  cifs,nfs - sid rules will be tested with both share protocols
  
====Option "username_for_sid_rules"====
+
====Option "username_for_user_rules"====
 
  typeː str
 
  typeː str
 
  default valueː Administrator
 
  default valueː Administrator
  descriptionː What name should be used with sid rule tests? Specify it if 'sid rule' tests are enabled.
+
  descriptionː What name should be used with user rule tests? Specify it if 'user rule' tests are enabled.
  
 
====Option "storage_protocol"====
 
====Option "storage_protocol"====
Line 182: Line 192:
 
  typeː str
 
  typeː str
 
  default valueː "" (empty)
 
  default valueː "" (empty)
  descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for alternative member creds. At the moment there are no tests, that use share_network with creds for alternative user.
+
  descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for member creds.
  
 
====Option "alt_share_network_id"====
 
====Option "alt_share_network_id"====
 
  typeː str
 
  typeː str
 
  default valueː "" (empty)
 
  default valueː "" (empty)
  descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for admin creds.
+
  descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for alternative member creds. At the moment there are no tests, that use share_network with creds for alternative user.
  
 
====Option "admin_share_network_id"====
 
====Option "admin_share_network_id"====
 
  typeː str
 
  typeː str
 
  default valueː  
 
  default valueː  
  descriptionː  
+
  descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for admin creds.
  
 
====Option "multi_backend"====
 
====Option "multi_backend"====
Line 232: Line 242:
  
 
==If something went wrong with test launch==
 
==If something went wrong with test launch==
On step where tests are gathered and parsed can be raised Exception and as a result we would see something like nextː
+
Exception may be raised on step where tests are gathered and parsed and as a result we would see something similar to the followingː
 
  running testr
 
  running testr
 
  Non-zero exit code (2) from test listing. stdout='\xb3)\x01@d@Ytempest.api.share.admin.test_admin_actions.AdminActionsTest.test_force_delete_share[gate]<Z\x90"\xb3)\x01@g@\\
 
  Non-zero exit code (2) from test listing. stdout='\xb3)\x01@d@Ytempest.api.share.admin.test_admin_actions.AdminActionsTest.test_force_delete_share[gate]<Z\x90"\xb3)\x01@g@\\
Line 257: Line 267:
 
There could be different causes for that, in most cases - dependency issues, when some dependency is not installed and can not be imported. Or python syntax error, if something were changed in code, etc...
 
There could be different causes for that, in most cases - dependency issues, when some dependency is not installed and can not be imported. Or python syntax error, if something were changed in code, etc...
  
As we can see, it does not show us real problem and we need dig root cause up. It can be done by a lot of cases, for example, try nextː
+
As we can see, it does not show us real problem and we need dig root cause up. It can be done by a lot of means, for example, try nextː
 
  nosetests -sv path/to/tests
 
  nosetests -sv path/to/tests
 
or
 
or
 
  .tox/venv/bin/python -m testtools.run discover -t ./ path/to/tests
 
  .tox/venv/bin/python -m testtools.run discover -t ./ path/to/tests
 +
or
 +
testr run path/to/tests
  
 
=How to write own tests=
 
=How to write own tests=
Line 291: Line 303:
  
 
==Utilization of resources==
 
==Utilization of resources==
Utilisation stuff defined in next moduleː
+
Utilization stuff is defined in the following moduleː
 
  tempest/api/share/base.py
 
  tempest/api/share/base.py
  
 
Basic actionsː
 
Basic actionsː
 
  1) All setUpClass methods should be decorated with "@test.safe_setup" decorator.
 
  1) All setUpClass methods should be decorated with "@test.safe_setup" decorator.
  Reasonː if we get Exception on some step in setUpClass, tearDownClass won't start, where created resources are deleted. So, this decorator makes 'tearDownClass' run in any case.
+
  Reasonː if we get Exception on some step in setUpClass, tearDownClass won't start. So, this decorator runs 'tearDownClass' in any case.
  
 
  2) Create resources that should be deleted with methods from base class.
 
  2) Create resources that should be deleted with methods from base class.

Latest revision as of 09:50, 24 January 2018

WARNING: Information on this page may be obsolete due to changes in tempest and manila_tempest_tests. Refer to the developer documentation here instead: https://docs.openstack.org/manila/ocata/devref/tempest_tests.html


Contents

Foreword

This page describes how Tempest project for Manila project testing can be used.

It is assumed that user who reads this is acquainted with, at least, common information about Tempest:

http://docs.openstack.org/developer/tempest/
https://github.com/openstack/tempest

Since Liberty release Manila is using tempest plugins which are documented here:

http://docs.openstack.org/developer/tempest/plugin.html

Also, it is assumed, that the following projects has been installed in test environmentː

- cinder
- glance
- keystone
- manila
- neutron
- nova

Where do Tempest tests for Manila live?

Manila stores its tests for Tempest in its repo here:

https://github.com/openstack/manila/tree/master/manila_tempest_tests

Since Manila is using the Tempest plugin mechanism all test are automatically registered by running setup.py. In case devstack is used it is fully sufficient to just enable the Manila devstack plugin.

You can list all Manila tests by browsing to the Tempest source directory and execute the following command:

 testr list-tests | grep manila_tempest_tests

To see if the plugin is registered correclty you can run:

 echo 'import pkg_resources; print list(pkg_resources.iter_entry_points("tempest.test_plugins"))' | python

Configuring Tempest before running tests

Any test running with Tempest requires update of Tempest's config file located in

etc/tempest.conf

If the tempest plugin is installed you can use the following command to generate all config options:

oslo-config-generator --config-file tools/config/config-generator.tempest.conf

Just copy template "tempest.conf.sample" and configure the needed options:

cp tempest.conf.sample tempest.conf

Subsections below describe sections and options that can be updated with Tempest's config file.

Cluster related options in general

Required options for Manila tests are described below.

Section "cli"

Option "enabled"

Set it to

True

value, to be able to run Manila cli tests.

Section "identity"

Option "uri"

If using localhost it should look likeː

http://127.0.0.1:5000/v2.0/

Used for all tests, if v2 auth enabled.

Option "uri_v3"

If using localhost it should look likeː

http://127.0.0.1:5000/v3/

Option "username"

If using devstack installation it should have the following valueː

demo

Option "password"

If using devstack installation it should have the following valueː

secrete

Password can be redefined, just put here password from 'demo' user.

Option "tenant_name"

If using devstack installation it should have the following valueː

demo

Option "alt_username"

If using devstack installation it should have the following valueː

alt_demo

Option "alt_password"

If using devstack installation it should have the following valueː

secrete

Password can be redefined, just put here password from 'alt_demo' user.

Option "alt_tenant_name"

If using devstack installation it should have the following valueː

alt_demo

Option "admin_username"

If using devstack installation it should have the following valueː

admin

Option "admin_password"

If using devstack installation it should have the following valueː

secrete

Password can be redefined, just put here password from 'admin' user.

Option "admin_tenant_name"

If using devstack installation it should have the following valueː

admin

Section "network"

Option "public_network_id"

This option is used for creation of isolated tenants. Its value can be retrieved from:

$ neutron net-list

Option "public_router_id"

This option is used for creation of isolated tenants. Its value can be retrieved from:

$ neutron router-list

Manila related options

Section "service_available"

This section is defined within vanila Tempest. But "manila" option is defined within Manila's extension.

Option "manila"

typeː boolean
default valueː True
descriptionː Whether or not manila is expected to be available

Section "share"

This section is defined with Manila's extension.

Option "catalog_type"

typeː str
default valueː share
descriptionː Catalog type of the Share service. Should be changed only if it is changed in Manila itself.

Option "endpoint_type"

typeː str
default valueː publicURL
descriptionː The endpoint type to use for the share service. Expected values are 'public', 'admin', 'internal', 'publicURL', 'adminURL' and 'internalURL'.

Option "multitenancy_enabled"

typeː boolean
default valueː True
descriptionː This option is used to determine backend driver type, multitenant driver uses share-networks, but single-tenant doesn't.
Current singletenant drivers: LVM, GlusterFS, NetApp 7-mode
Current multitenant drivers: Generic, NetApp Cluster-mode (cmode)
Default driver in manila is Generic driver, so "multitenancy_enabled" option is enabled by default.

Option "enable_protocols"

typeː list
default valueː nfs,cifs
descriptionː The first value of the list is protocol by default, items of the list show enabled protocols.
Expected values are:
nfs - will be tested only NFS type of shares and common actions
cifs - will be tested only CIFS type of shares and common actions
nfs,cifs - will be tested both share types and common actions, for common actions will be used nfs protocol
cifs,nfs - will be tested both share types and common actions, for common actions will be used cifs protocol

Option "enable_ip_rules_for_protocols"

typeː list
default valueː nfs,cifs
descriptionː Whether allow running of ip rules tests or not, and which share protocols should be used.
Expected values are:
nfs - sid rules will be tested only for nfs protocol
cifs - sid rules will be tested only for cifs protocol
nfs,cifs - sid rules will be tested with both share protocols
cifs,nfs - sid rules will be tested with both share protocols

Option "enable_sid_rules_for_protocols"

typeː list
default valueː (empty)
descriptionː Whether allow running of sid (username/usergroup) rules tests or not, and which share protocols should be used.
Expected values are:
nfs - sid rules will be tested only for nfs protocol
cifs - sid rules will be tested only for cifs protocol
nfs,cifs - sid rules will be tested with both share protocols
cifs,nfs - sid rules will be tested with both share protocols

Option "username_for_user_rules"

typeː str
default valueː Administrator
descriptionː What name should be used with user rule tests? Specify it if 'user rule' tests are enabled.

Option "storage_protocol"

typeː str
default valueː "" (empty)
descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for member creds.

Option "share_network_id"

typeː str
default valueː "" (empty)
descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for member creds.

Option "alt_share_network_id"

typeː str
default valueː "" (empty)
descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for alternative member creds. At the moment there are no tests, that use share_network with creds for alternative user.

Option "admin_share_network_id"

typeː str
default valueː 
descriptionː If you want Tempest to use some specific share-network instead of autocreated, specify it. This value is used for admin creds.

Option "multi_backend"

typeː boolean
default valueː False
descriptionː Does your environment has more than 1 backend or not? If yes, than you can set it to True to enable multibackend tests in tempest.

Option "backend_names"

typeː list
default valueː (empty)
descriptionː names of backends in Manila, that Tempest should use. Tempest will use first two values from list and will use it only if option "multi_backend" is Enabled.

Option "build_interval"

typeː int
default valueː 3
descriptionː time to wait between resend of requests.

Option "build_timeout"

typeː int
default valueː 500
descriptionː time to wait until raise of Exception with timeout.

How to run tests

Run only cli tests

tox -evenv bash tools/pretty_tox.sh \"tempest.cli.*manila*\"

Run only api tests

tox -evenv bash tools/pretty_tox.sh \"tempest.api.share*\"

Run only api and cli tests

tox -evenv bash tools/pretty_tox.sh \"tempest.api.share* tempest.cli.*manila*\"

Run only api and cli tests with specified amount of threads

export TEMPEST_CONCURRENCY=4
tox -evenv bash tools/pretty_tox.sh \"tempest.api.share* tempest.cli.*manila* -- --concurrency=$TEMPEST_CONCURRENCY\"

Above action will try to run Tempest in 4 threads, if your host does have at least 4 cores. By default it uses amount of threads equal to amount of available cores.

Manila tests are OK using concurrent threads.

If something went wrong with test launch

Exception may be raised on step where tests are gathered and parsed and as a result we would see something similar to the followingː

running testr
Non-zero exit code (2) from test listing. stdout='\xb3)\x01@d@Ytempest.api.share.admin.test_admin_actions.AdminActionsTest.test_force_delete_share[gate]<Z\x90"\xb3)\x01@g@\\
...
tempest.scenario.test_volume_boot_pattern6B\x1e\xcc', stderr=None
error: testr failed (3)
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./tempest/test_discover} --list 

======
Totals
======
Run: 0 in 0.0 sec.
 - Passed: 0
 - Skipped: 0
 - Failed: 0
ERROR: InvocationError: '/bin/bash tools/pretty_tox.sh tempest/api/share/test_extensions.py'
__________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
ERROR:   venv: commands failed

There could be different causes for that, in most cases - dependency issues, when some dependency is not installed and can not be imported. Or python syntax error, if something were changed in code, etc...

As we can see, it does not show us real problem and we need dig root cause up. It can be done by a lot of means, for example, try nextː

nosetests -sv path/to/tests

or

.tox/venv/bin/python -m testtools.run discover -t ./ path/to/tests

or

testr run path/to/tests

How to write own tests

Where to put new tests

All tests, that cover admin API are stored in

tempest/api/share/admin/*

All tests, that cover non-admin API are stored in

tempest/api/share/*

Add method as API caller

All such methods are stored in

tempest/services/share/json/shares_client.py

XML is considered as deprecated and tests for it were removed, so, only json "client" is expected.

Positive/Negative

Positive and negative tests are separated into different modules, module with "negative" tests should have "_negative" suffix. Example of module with positive testsː

tempest/api/share/test_shares.py

Example of module with negative testsː

tempest/api/share/test_shares_negative.py

Resolving concurrency issue

If some test is sensible to changes that can be done by other tests, this test should use isolated tenant, example of it can be found in quota tests, located in

tempest/api/share/admin/test_quotas.py

Reusing resources

If several tests have same dependencies, they should use same resources if possible, it can be created in "setUpClass" method. Example can be found hereː

tempest/api/share/test_shares_actions.py

Utilization of resources

Utilization stuff is defined in the following moduleː

tempest/api/share/base.py

Basic actionsː

1) All setUpClass methods should be decorated with "@test.safe_setup" decorator.
Reasonː if we get Exception on some step in setUpClass, tearDownClass won't start. So, this decorator runs 'tearDownClass' in any case.
2) Create resources that should be deleted with methods from base class.
correctː self.create_share() - will add resources to deletion queue
incorrectː self.shares_client.create_share() - won't add resource to deletion queue

Common infoː

All resources in deletion queue will be deleted in LIFO sequence - "last in first out" or "last created first deleted".
There are two deletion queuesː
- runs after each test
- runs after each test-suite (class with tests)
Be attentive setting up it.