This page serves as a proposal to unify the command-line and configuration file options processing for OpenStack projects.
Launchpad Entry: openstack-common:bexar-options-processing
Created: 2010-08-31
Contributors: JayPipes
Current status
Command-line options processing
Currently, Nova uses python-gflags for processing command-line options. This style includes the following way of specifying program options:
In a module, you do this:
1 import flags
2 FLAGS = flags.FLAGS
3
4 DEFINE_string('connection_type', 'libvirt', 'libvirt, xenapi, or fake')
5
The above would define an option "--connection_type" that can appear on the command line (or in a special "flag file", see below)
While a coder defines their module-level flags in the module they are working on (as opposed to a single global file), there is no support for either:
- enforcement of module-level option names
- auto-prefixing of module name to option name
for example, having an option in /nova/datastore.py called "driver" automatically be labeled "--datastore-driver" or "--nova-datastore-driver"
In Swift, there is a mixture of usage of either the older getopt module or the newer optparse module (which has now been deprecated in favour of the argparse module in 2.7).
Configuration file processing
In Nova, there is no support for configuration file processing other than the use of gflag's --flag-file=FILE argument, which indicates a file that can contain a non-standard newline-delimited list of flag options, like so:
--connection_type=fake --s3_port=3334
As mentioned, this is non-standard, and does not conform to any RFC regarding configuration files.
In Swift, some things use the ConfigParser module for allowing configuration options in files. There doesn't seem to be any support, however, for configuration option groups in common configuration files for specific modules (see proposal below).
Importantly, Swift makes heavy use of paste.deploy configuration files, which are identical to normal INI-like files, and use a special category:name designation for its option groups in configuration files.
Proposal for a unified options processing module
I propose consolidating all of the CLI and configuration file options processing into a single common module: openstack.common.config
This module would expose an API for adding system and module-level program options in a simple, straightforward manner, with automatic support for program options entered into standard configuration files.
Adding program options
openstack.common.config.add_module_option
Program options may be added using openstack.common.config.add_module_option.
Example: /nova/datastore.py
Current options stuff (abbreviated):
1 from nova import flags
2
3 FLAGS = flags.FLAGS
4 flags.DEFINE_string('redis_host', '127.0.0.1',
5 'Host that redis is running on.')
6 flags.DEFINE_integer('redis_port', 6379,
7 'Port that redis is running on.')
8
Would become:
1 from openstack.common import config
2
3 config.add_module_option(__name__, '--redis-host', '127.0.0.1',
4 'Host that redis is running on')
5 config.add_module_option(__name__, '--redis-port', 6379,
6 'Port that redis is running on', type="int")
7
Since the underlying processing engine would be optparse, you can use any of the keyword arguments to openstack.common.config.add_module_option that you would use with optparse.OptionParser.add_option, as illustrated below:
1 from openstack.common import config
2
3 _engines = ['redis', 'sqlite', 'memory']
4
5 config.add_module_option(__name__, '-e'. '--engine', default='redis',
6 choices=_engines, metavar="DRIVER",
7 help="Driver to use for data storage "
8 " (default: %default)")
9
The difference between the gflags and proposed addition of program options is that options registered with openstack.common.config.add_module_option will automatically have the module name prefixing the option name. This allows us to adhere to a common naming convention of --module-name-option-name and also to specify module-specific options in configuration files as option groups (see below).
What this means is the following: if an option "--engine" is defined using openstack.common.config.add_module_option in a module /nova/datastore.py, then the resulting option name will be --nova-datastore-engine.
Possible variation
It's just as easy to have the resulting option name be --datastore-engine and not have the "nova" prefix...
openstack.common.config.ModuleConfig
Some may prefer the following method of adding a module-specific configuration option:
1 from openstack.common import config
2
3 mc = config.ModuleConfig(__name__)
4
5 mc.add_option('-e'. '--engine', default='redis',
6 choices=_engines, metavar="DRIVER",
7 help="Driver to use for data storage "
8 " (default: %default)")
9
Retrieving program options
Program options are retrieved using the standard optparse.OptionParser methods after calling openstack.common.config.parse_options. Optionally, an openstack.common.config.ModuleConfig wrapper can be used. The two styles are shown below.
Assume that the --nova-datastore-engine option from above has been registered using openstack.common.config.add_module_option
1 from openstack.common import config
2
3 config.parse_options()
4
5 # Print the value of the --nova-datastore-engine option
6 print config.options.nova_datastore_engine
7
8 # Same thing, only using the ModuleConfig convenience wrapper
9 mc = config.ModuleConfig('nova.datastore')
10 print mc.engine
11
Configuration files
I am proposing that the openstack.common.config.add_module_option (and its ModuleConfig convenience wrapper would automatically register the program option as something that may be placed in any openstack configuration file, in an option section named after the module. This means that no longer will openstack projects need to import ConfigParser nor handle the parsing and merging those config values with those of the parsed optparse.OptionParser.Options values.
An example speaks volumes.
Assume the following code is in /nova/datastore/__init__.py:
1 from openstack.common import config
2
3 import nova.datastore.sqlite
4
5 config.add_module_option(__name__, '--driver', 'sqlite', 'Which datastore driver')
6
Assume the following code is in /nova/datastore/sqlite/__init__.py:
1 from openstack.common import config
2
3 config.add_module_option(__name__, '--db', 'nova.db', 'Name of SQLite DB')
4
Now assume that the following is in /etc/openstack/core.cnf:
[nova.datastore] driver=sqlite [nova.datastore.sqlite] db=testing.db
And assume the following code is in /test_options.py:
1 from openstack.common import config
2
3 from nova import datastore
4
5 config.parse_options()
6 print config.options.nova_datastore_sqlite_db
7
If the test_options.py were invoked like so:
./test_options.py --nova-datastore-sqlite-db=fred.db
The following would be printed:
fred.db
The configuration file value of "testing.db" would replace the default of "nova.db" all other times.