Jump to: navigation, search

Difference between revisions of "CommonOptionsProcessing"

(import cleanup)
(import cleanup)
Line 76: Line 76:
  
  
<pre><nowiki>#!highlight python
+
<pre><nowiki>
 
from openstack.common import config
 
from openstack.common import config
  
Line 115: Line 115:
  
  
<pre><nowiki>#!highlight python
+
<pre><nowiki>
 
from openstack.common import config
 
from openstack.common import config
  
Line 135: Line 135:
  
  
<pre><nowiki>#!highlight python
+
<pre><nowiki>
 
from openstack.common import config
 
from openstack.common import config
  
Line 158: Line 158:
  
  
<pre><nowiki>#!highlight python
+
<pre><nowiki>
 
from openstack.common import config
 
from openstack.common import config
  
Line 170: Line 170:
  
  
<pre><nowiki>#!highlight python
+
<pre><nowiki>
 
from openstack.common import config
 
from openstack.common import config
  

Revision as of 21:43, 16 February 2013

This page serves as a proposal to unify the command-line and configuration file options processing for OpenStack projects.

Current status

Command-line options processing

Currently, Nova uses `python-gflags` for processing command-line options. This style includes the following way of specifying program options:

In a module, you do this:


import flags
FLAGS = flags.FLAGS

DEFINE_string('connection_type', 'libvirt', 'libvirt, xenapi, or fake')


The above would define an option "--connection_type" that can appear on the command line (or in a special "flag file", see below)

While a coder defines their module-level flags in the module they are working on (as opposed to a single global file), there is no support for either:

  • enforcement of module-level option names
  • auto-prefixing of module name to option name
    • for example, having an option in `/nova/datastore.py` called "driver" automatically be labeled "--datastore-driver" or "--nova-datastore-driver"

In Swift, there is a mixture of usage of either the older `getopt` module or the newer `optparse` module (which has now been deprecated in favour of the `argparse` module in 2.7).

Configuration file processing

In Nova, there is no support for configuration file processing other than the use of gflag's `--flag-file=FILE` argument, which indicates a file that can contain a non-standard newline-delimited list of flag options, like so:


--connection_type=fake
--s3_port=3334

As mentioned, this is non-standard, and does not conform to any RFC regarding configuration files.

In Swift, some things use the `ConfigParser` module for allowing configuration options in files. There doesn't seem to be any support, however, for configuration option groups in common configuration files for specific modules (see proposal below).

Importantly, Swift makes heavy use of `paste.deploy` configuration files, which are identical to normal INI-like files, and use a special `category:name` designation for its option groups in configuration files.

Proposal for a unified options processing module

I propose consolidating all of the CLI and configuration file options processing into a single common module: openstack.common.config

This module would expose an API for adding system and module-level program options in a simple, straightforward manner, with automatic support for program options entered into standard configuration files.

Adding program options

openstack.common.config.add_module_option

Program options may be added using openstack.common.config.add_module_option.

Example: `/nova/datastore.py`

Current options stuff (abbreviated):


from nova import flags

FLAGS = flags.FLAGS
flags.DEFINE_string('redis_host', '127.0.0.1',
                    'Host that redis is running on.')
flags.DEFINE_integer('redis_port', 6379,
                    'Port that redis is running on.')


Would become:


from openstack.common import config

config.add_module_option(__name__, '--redis-host', '127.0.0.1',
                         'Host that redis is running on')
config.add_module_option(__name__, '--redis-port', 6379,
                         'Port that redis is running on', type="int")


Since the underlying processing engine would be `optparse`, you can use any of the keyword arguments to openstack.common.config.add_module_option that you would use with optparse.[[OptionParser]].add_option, as illustrated below:


#!highlight python
from openstack.common import config

_engines = ['redis', 'sqlite', 'memory']

config.add_module_option(__name__, '-e'. '--engine', default='redis',
                         choices=_engines, metavar="DRIVER",
                         help="Driver to use for data storage "
                         " (default: %default)")


The difference between the `gflags` and proposed addition of program options is that options registered with openstack.common.config.add_module_option will automatically have the module name prefixing the option name. This allows us to adhere to a common naming convention of --module-name-option-name and also to specify module-specific options in configuration files as option groups (see below).

What this means is the following: if an option "--engine" is defined using openstack.common.config.add_module_option in a module `/nova/datastore.py`, then the resulting option name will be --nova-datastore-engine.

Possible variation

It's just as easy to have the resulting option name be --datastore-engine and not have the "nova" prefix...

openstack.common.config.ModuleConfig

Some may prefer the following method of adding a module-specific configuration option:


from openstack.common import config

mc = config.ModuleConfig(__name__)

mc.add_option('-e'. '--engine', default='redis',
              choices=_engines, metavar="DRIVER",
              help="Driver to use for data storage "
              " (default: %default)")


Retrieving program options

Program options are retrieved using the standard `optparse.OptionParser` methods after calling openstack.common.config.parse_options. Optionally, an openstack.common.config.[[ModuleConfig]] wrapper can be used. The two styles are shown below.

Assume that the --nova-datastore-engine option from above has been registered using openstack.common.config.add_module_option


from openstack.common import config

config.parse_options()

# Print the value of the --nova-datastore-engine option
print config.options.nova_datastore_engine

# Same thing, only using the ModuleConfig convenience wrapper
mc = config.ModuleConfig('nova.datastore')
print mc.engine


Configuration files

I am proposing that the openstack.common.config.add_module_option (and its [[ModuleConfig]] convenience wrapper would automatically register the program option as something that may be placed in any openstack configuration file, in an option section named after the module. This means that no longer will openstack projects need to import ConfigParser nor handle the parsing and merging those config values with those of the parsed optparse.[[OptionParser]].Options values.

An example speaks volumes.

Assume the following code is in `/nova/datastore/init.py`:


from openstack.common import config

import nova.datastore.sqlite

config.add_module_option(__name__, '--driver', 'sqlite', 'Which datastore driver')


Assume the following code is in `/nova/datastore/sqlite/init.py`:


from openstack.common import config

config.add_module_option(__name__, '--db', 'nova.db', 'Name of SQLite DB')


Now assume that the following is in `/etc/openstack/core.cnf`:


[nova.datastore]
driver=sqlite

[nova.datastore.sqlite]
db=testing.db


And assume the following code is in `/test_options.py`:


#!highlight python
from openstack.common import config

from nova import datastore

config.parse_options()
print config.options.nova_datastore_sqlite_db


If the test_options.py were invoked like so:


./test_options.py --nova-datastore-sqlite-db=fred.db


The following would be printed:


fred.db


The configuration file value of "testing.db" would replace the default of "nova.db" all other times.

References