CommonOptionsProcessing

This page serves as a proposal to unify the command-line and configuration file options processing for OpenStack projects.


 * Launchpad Entry: openstack-common:bexar-options-processing
 * Created: 2010-08-31
 * Contributors: JayPipes

= Current status =

Command-line options processing
Currently, Nova uses `python-gflags` for processing command-line options. This style includes the following way of specifying program options:

In a module, you do this:

import flags FLAGS = flags.FLAGS

DEFINE_string('connection_type', 'libvirt', 'libvirt, xenapi, or fake')

The above would define an option "--connection_type" that can appear on the command line (or in a special "flag file", see below)

While a coder defines their module-level flags in the module they are working on (as opposed to a single global file), there is no support for either:
 * enforcement of module-level option names
 * auto-prefixing of module name to option name
 * for example, having an option in `/nova/datastore.py` called "driver" automatically be labeled "--datastore-driver" or "--nova-datastore-driver"

In Swift, there is a mixture of usage of either the older `getopt` module or the newer `optparse` module (which has now been deprecated in favour of the `argparse` module in 2.7).

Configuration file processing
In Nova, there is no support for configuration file processing other than the use of gflag's `--flag-file=FILE` argument, which indicates a file that can contain a non-standard newline-delimited list of flag options, like so:

--connection_type=fake --s3_port=3334

As mentioned, this is non-standard, and does not conform to any RFC regarding configuration files.

In Swift, some things use the `ConfigParser` module for allowing configuration options in files. There doesn't seem to be any support, however, for configuration option groups in common configuration files for specific modules (see proposal below).

Importantly, Swift makes heavy use of `paste.deploy` configuration files, which are identical to normal INI-like files, and use a special `category:name` designation for its option groups in configuration files.

= Proposal for a unified options processing module =

I propose consolidating all of the CLI and configuration file options processing into a single common module:

This module would expose an API for adding system and module-level program options in a simple, straightforward manner, with automatic support for program options entered into standard configuration files.

openstack.common.config.add_module_option
Program options may be added using.

Example: `/nova/datastore.py`

Current options stuff (abbreviated):

from nova import flags

FLAGS = flags.FLAGS flags.DEFINE_string('redis_host', '127.0.0.1',                   'Host that redis is running on.') flags.DEFINE_integer('redis_port', 6379,                   'Port that redis is running on.')

Would become:

from openstack.common import config

config.add_module_option(__name__, '--redis-host', '127.0.0.1',                        'Host that redis is running on') config.add_module_option(__name__, '--redis-port', 6379,                        'Port that redis is running on', type="int")

Since the underlying processing engine would be `optparse`, you can use any of the keyword arguments to  that you would use with , as illustrated below:

#!highlight python from openstack.common import config

_engines = ['redis', 'sqlite', 'memory']

config.add_module_option(__name__, '-e'. '--engine', default='redis',                        choices=_engines, metavar="DRIVER",                         help="Driver to use for data storage "                         " (default: %default)")

The difference between the `gflags` and proposed addition of program options is that options registered with  will automatically have the module name prefixing the option name. This allows us to adhere to a common naming convention of --module-name-option-name and also to specify module-specific options in configuration files as option groups (see below).

What this means is the following: if an option "--engine" is defined using  in a module `/nova/datastore.py`, then the resulting option name will be.

Possible variation
It's just as easy to have the resulting option name be  and not have the "nova" prefix...

openstack.common.config.ModuleConfig
Some may prefer the following method of adding a module-specific configuration option:

from openstack.common import config

mc = config.ModuleConfig(__name__)

mc.add_option('-e'. '--engine', default='redis',             choices=_engines, metavar="DRIVER",              help="Driver to use for data storage "              " (default: %default)")

Retrieving program options
Program options are retrieved using the standard `optparse.OptionParser` methods after calling. Optionally, an  wrapper can be used. The two styles are shown below.

Assume that the  option from above has been registered using

from openstack.common import config

config.parse_options

print config.options.nova_datastore_engine
 * 1) Print the value of the --nova-datastore-engine option

mc = config.ModuleConfig('nova.datastore') print mc.engine
 * 1) Same thing, only using the ModuleConfig convenience wrapper

Configuration files
I am proposing that the  (and its   convenience wrapper would automatically register the program option as something that may be placed in any openstack configuration file, in an option section named after the module.  This means that no longer will openstack projects need to import ConfigParser nor handle the parsing and merging those config values with those of the parsed   values.

An example speaks volumes.

Assume the following code is in `/nova/datastore/ init .py`:

from openstack.common import config

import nova.datastore.sqlite

config.add_module_option(__name__, '--driver', 'sqlite', 'Which datastore driver')

Assume the following code is in `/nova/datastore/sqlite/ init .py`:

from openstack.common import config

config.add_module_option(__name__, '--db', 'nova.db', 'Name of SQLite DB')

Now assume that the following is in `/etc/openstack/core.cnf`:

[nova.datastore] driver=sqlite

[nova.datastore.sqlite] db=testing.db

And assume the following code is in `/test_options.py`:

#!highlight python from openstack.common import config

from nova import datastore

config.parse_options print config.options.nova_datastore_sqlite_db

If the test_options.py were invoked like so:

./test_options.py --nova-datastore-sqlite-db=fred.db

The following would be printed:

fred.db

The configuration file value of "testing.db" would replace the default of "nova.db" all other times.

= References =


 * Python optparse module