Jump to: navigation, search

Difference between revisions of "Python3"

(Other OpenStack Applications and Projects)
(Other OpenStack Applications and Projects)
Line 682: Line 682:
| [https://pypi.python.org/pypi/designate designate] || style="background-color: orange;" | work in progress || style="background-color: red;" | No || Pradeep Kumar Singh is working on the Python 3 port, see [https://review.openstack.org/#/q/owner:%22Pradeep+Kumar+Singh%22+project:openstack/designate,n,z Pradeep's patches] ([https://review.openstack.org/#/q/owner:%22Pradeep+Kumar+Singh%22+status:open+project:openstack/designate,n,z only open patches]).
| [https://pypi.python.org/pypi/congress congress] || style="background-color: orange;" | work-in-progress || style="background-color: orange;" | non-voting ||
| [https://pypi.python.org/pypi/congress congress] || style="background-color: red;" | No || style="background-color: red;" | No ||
| [https://pypi.python.org/pypi/cue cue] || style="background-color: orange;" | work-in-progress || style="background-color: orange;" | non-voting ||
| [https://pypi.python.org/pypi/cue cue] || style="background-color: red;" | No || style="background-color: red;" | No ||
| [https://pypi.python.org/pypi/designate designate] || style="background-color: orange;" | work in progress || style="background-color: red;" | No || Pradeep Kumar Singh is working on the Python 3 port, see [https://review.openstack.org/#/q/owner:%22Pradeep+Kumar+Singh%22+project:openstack/designate,n,z Pradeep's patches] ([https://review.openstack.org/#/q/owner:%22Pradeep+Kumar+Singh%22+status:open+project:openstack/designate,n,z only open patches]).

Revision as of 13:38, 22 September 2015

This page tracks the progress of Python 3 effort porting for OpenStack.

Python 3

Why should OpenStack move to Python 3 right now?

Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.

Python 2: Python 2.6 support dropped, Python 2.7 only

OpenStack Liberty targets Python 2.7 and 3.4.

Python 2.6 support is being dropped in OpenStack since OpenStack Juno for servers. Python 2.6 support is currently kept in Oslo libraries and clients. See juno-cross-project-future-of-python etherpad.

Python 3.3 support is being dropped since OpenStack Liberty.

Python 2.6 is slowly being dropped in the whole OpenStack project, ex: Remove p26 job from DIB.

Port Python 2 code to Python 3

OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. As we are targeting Python 3.4 and up, there is no need to avoid u'unicode' syntax. Do not use six.u('unicode').

sixer tool

The sixer tool helps to replace most basic patterns to add Python 3 compatibility and it respects OpenStack coding style.

Common patterns

  • Replace dict.iteritems() with dict.items()
  • Replace dict.itervalues() with dict.values()
  • Replace "for key in dict.iterkeys()" with "for key in dict"
  • Replace iterator.next() with next(iterator)
  • Replace basestring with six.string_types
  • Replace unicode with six.text_type
  • Replace (str, unicode) with six.string_types
  • Replace (int, long) with six.integer_types
  • Replace text.encode('base64') with base64.b64encode(text) which requires to encode (Unicode) text to UTF-8 on Python 3
if six.PY3:
   text = text.encode('utf-8')
text_b64 = base64.b64encode(text)
  • Replace raw.decode('base64') with base64.b64decode(raw). The output may be decoded from UTF-8 to get back text (Unicode) on Python 3.
text = base64.b64decode(raw)
if six.PY3:
   text = text.decode('utf-8')
  • Replace raw.decode('hex') with binascii.unhexlify(raw)
  • Replace bytes.encode('hex') with binascii.hexlify(bytes)

Note: Replacing dict.iteritems() with six.iteritems(dict) was preferred in the past, but there was a discussion suggesting to avoid six for this.


To replaced contextlib.nested is to use contextlib.ExitStack. It's available on Python 2 using contextlib2.ExitStack. For contextlib.nested, nova/test.py defines:

if six.PY3:
    def nested(*contexts):
        with contextlib.ExitStack() as stack:
            yield [stack.enter_context(c) for c in contexts]
    nested = contextlib.nested

Another option to replaced contextlib.nested is to use the @mock.patch decorator. Example with nested function:

def test_thing(self):
   def do_test(...):

More options:

bytes.decode and unicode.encode

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write data.decode() or text.encode() without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: data.decode('utf-8') or text.encode('utf-8'). The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).


Olso Incubator has a function safe_decode() which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_decode(data, 'utf-8'): decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:

  • safe_decode(data)

By default, the decoder is strict. You can specify a different error handler using the optional errors parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.


Olso Incubator has a function safe_encode() which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.

  • safe_encode(text) encodes text to the output encoding
  • safe_encode(bytes) may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (incomding parameter) is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (encoding parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_encode(data, incoming='utf-8'): encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:

  • safe_encode(data)


  • safe_encode(b'\xe9', incoming='latin-1') returns b'\xc3\xa9'.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional errors parameter. Example: safe_encode(b'[\xff]', incoming='ascii', errors='ignore') returns b'[]'.

logging module and format exceptions

The exception_to_unicode(exc) function of oslo_utils.encodeutils is the recommanded way to format an exception to Unicode. This function works on Python 2 and Python 3 and it should avoid mojibake is most cases.

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs b'hello' instead of 'hello'.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

  • str(exc): native string, so use bytes on Python 2
  • six.text_type(exc): always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: unicode(Exception("nonascii:\xe9")).
  • six.u(str(exc)): unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: unicode(str(Exception("\xff")))
  • LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using str(exc). Example: LOG.debug(str(exc)).


The HTTP protocol is based on bytes:

  • HTTP body contains bytes. For example, use io.BytesIO for a stream storing an HTTP body.
  • HTTPConnection.getresponse().read() returns bytes (in Python 3, str which is bytes in Python 2)
  • On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
  • It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and HTTP header should use what character encoding?

References to port Python 2 code to Python 3

Common pitfalls

What is a string ?

You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:

   >>> type('foo')
   <type 'str'>
   >>> type(u'foo')
   <type 'unicode'>
   >>> type(b'foo')
   <type 'str'>
   >>> isinstance('foo', six.text_type)
   >>> isinstance(u'foo', six.text_type)
   >>> bytes is str
   >>> b'foo'[0]

Python 3:

   >>> type('foo')
   <class 'str'>
   >>> type(u'foo')
   <class 'str'>
   >>> type(b'foo')
   <class 'bytes'>
   >>> isinstance('foo', six.text_type)
   >>> isinstance(b'foo', six.text_type)
   >>> bytes is str
   >>> b'foo'[0]

tox/testr error: db type could not be determined

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/" and then run "tox -e py34" before running "tox -e py27". You only have to do this once. The problem only occurs with "tox -e py34" when .testrepository/ was created by Python 2.

Python 3 Status of OpenStack projects

Common Libraries (Oslo Projects)

For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160

Project Python 3 compatibility Comment
cliff Yes
oslo.concurrency Yes
oslo-incubator Yes py34 gate is voting
oslo.config Yes
oslo.context Yes
oslo.db Yes

PyMySQL driver is now used by default for MySQL. setup.cfg contains the Python 3 classifier.

oslo.i18n Yes
oslo.log Yes
oslo.messaging Yes, except ...

oslo.messaging 1.11.0 fully works on Python 3... except of Qpid and AMQP 1.0 transports which are Python 2 only.


Note: We care only about AMQP1.0 driver. The legacy Qpid driver is deprecated, buggy and unmaintened.

The py34 check job is not more voting because of ubuntu still uses Python 3.4.0 which has a severe bug.

oslo.middleware Yes
oslo.rootwrap Yes
oslo.serialization Yes
oslosphinx  ? The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible?
oslotest Yes
oslo.versionedobjects Yes
oslo.vmware Yes Supports Python 3 since oslo.vmware 0.13.0
oslo.utils Yes
pylockfile Yes
stevedore Yes
taskflow Yes

Development tools

Project Python 3 compatibility Comment
cookiecutter yes
hacking yes py33 gate is not voting
pbr yes
stackforge/python-jenkins yes py33 gate is voting
openstack-infra/jenkins-job-builder partial https://review.openstack.org/172238

OpenStack clients

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
keystonemiddleware Yes Voting On PyPI python-memcached keystonemiddleware 1.6.1 supports Python 3.
python-barbicanclient Yes Voting On PyPI
python-ceilometerclient Yes Voting On PyPI
python-cinderclient Yes Voting On PyPI
python-fuelclient Yes voting Not yet

Patches: topic:py3.

python-glanceclient Yes Voting On PyPI
python-heatclient Yes Voting On PyPI
python-ironicclient Yes Voting On PyPI
python-keystoneclient Yes Voting On PyPI
python-manilaclient Yes Voting On PyPI
python-marconiclient Yes Voting On PyPI
python-novaclient Yes Voting On PyPII
python-neutronclient Yes Voting On PyPI
python-openstackclient Yes Voting On PyPI As of 0.9
python-saharaclient Yes Voting On PyPI
python-swiftclient Yes Voting On PyPI
python-tuskarclient Yes Voting On PyPI
python-troveclient Yes Voting On PyPI
python-designateclient work in progress Voting No

OpenStack Libraries

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Comment
django_openstack_auth Yes voting Yes

django_openstack_auth 2.0 is fully compatible with Python 3.

glance_store Yes voting Yes glance_store 0.7.0 added the Python 3 support.
os-brick Yes voting Yes os-brick 0.3.2 added Python 3 support.
sqlalchemy-migrate Yes voting Yes
taskflow Yes voting Yes


There are two ways to express dependencies that should differ between Python 2.x and 3.x. We currently support using multiple requirements files, and will shortly be supporting and recommending the use of environment markers. Recent releases of all the tools in our ecosystem are needed for environment markers.


This approach generates a different requirements list depending on what Python version is running when the egg info is created (which goes into wheels and sdists). As a consequence this can't work with tox, publishing sdists to PyPI, or wheels. It is however useful for anything installing from git. To use it, create two requirements files:

  • requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
  • requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
  • (same for test-requirements.txt)

You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:

deps = -r{toxinidir}/requirements-py3.txt

Environment markers

Environment markers provide a mini-language to express when a dependency is relevant. We hope the next release of pbr will support them. Recent pip, setuptools and wheel versions do support them, but until pbr exports the requirements metadata with them intact and structured correctly for setuptools, we can't use them anywhere.

When placed in a requirements.txt file, the markers are interpreted directly by pip in `pip install -r requirements.txt`, and indirectly via setuptools when installing an sdist or wheel (including `pip install .`. For instance:

futures; python_version < '3.2'

See also:

Oslo Messaging issue:

  • futures is not needed on Python 3, since concurrent.futures is now part of the Pyhon stdlib since Python 3.2
  • futures was added to requirements-py3.txt because tox creates the source tarball using Python 2 which reads requirements.txt
  • The correct fix would be to use environment markers on futures

Dependencies status

Porting status for global-requirement.txt.

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
boto Yes N/A Yes See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32).
django-compressor Yes N/A Yes Requirements upgraded: https://review.openstack.org/94357
tablib Yes but... N/A Yes setup.py installs tablib.packages.markup which is not compatible with Python 2. setup.py displays a warning on compiling the bytecode of markup.py and many others in the tablib/packages folder. Overall, the package cannot be built for Python 3 in distributions (zigo has issues to package it on Debian).

tablib is needed by cliff-tablib, itself needed by python-neutronclient, then neutronclient is itself needed by many other libs.

dnspython Yes N/A Yes Must use the Python 3 version, see https://github.com/rthalley/dnspython/issues/60
ecdsa Yes N/A Yes Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
eventlet Yes Yes Yes eventlet 0.17.3 now fully support Python 3
jsonrpclib No N/A No The project seems dead :(
libvirt-python Yes N/A Yes
MySQL-Python No, but mysqlclient or PyMySQL are compatible N/A No


MySQL-Python: 2 pending pull requests for Python 3.

INADA Naoki, the maintainer of PyMySQL, wrote mysqlclient: fork of MySQL-Python using libmysqlclient.so which is Python 3 compatible. He recommends to use mysqlclient instead of MySQL-python even on Python 2. mysqlclient fixes some bugs, build issues and it support Python 3. For example: support MariaDB's libmysqlclient.so, support microsecond in TIME column.

PyMySQL is compatible with Python 3: see PyMySQL evaluation discussion to eventually replace mysql-python with PyMySQL in OpenStack.

netifaces Yes N/A Yes Patch sent by Victor Stinner (in private): netifaces_python3.patch, Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 .
nose-exclude Yes N/A No Python 3 is supported since nose-exclude 0.4.
nosehtmloutput No No No
  • nose-exclude (tests only)
  • openstack.nose-plugin
nosexcover No N/A On PyPI Python 3 support since 1.0.9
openstack.nose-plugin No No No
pam No for pam, Yes for simplepam NA No The fork simplepam works on Python 2 and 3
paramiko Yes N/A On PyPI Requirements upgraded: https://review.openstack.org/#/c/81132/
paste Yes N/A Yes Compatible with Python 3 since Paste 2.0
pycadf Yes Yes Yes
python-ldap No No No The project seems dead. See ldap3 which is Python 3 compatible (license: LGPL, one of the optional deps is BSD-4-Clause).
python-memcached Yes N/A Yes

python-memcached 1.56 is compatible with Python 3, but there are still some bugs: More Python 3 fixes (not part of python-memcached 1.57). These bugs impact running keystonemiddleware tests when the memcached server is not running. It can also impact a setup with multiple memcached servers.

qpid-python No No No Required if using oslo.messaging's qpid backend. That is the old pure python amqp 0-10 client library. That only speaks 0-10 and works with qpidd broker. It is used by the impl_qpid.py driver in oslo.messaging. That client is legacy - it isn't being ported to use AMQP 1.0.
pyngus Yes N/A No

pyngus 2.0 supports Python 3.

pyngus is required if using oslo.messaging's AMQP 1.0 driver. pyngus is a wrapper for proton and proton has not been ported to python 3 yet: see Proton issue #490.

Proton is a C library with various bindings in other languages. The language bindings are auto generated via the SWIG tool, plus a the python wrapper bits. All the python stuff is found under proton-c/bindings/python in the git tree. The first bit of work is to refactor the SWIG file cproton.i to work for both Python 2 and 3.

rtslib-fb No No No
Routes Yes Yes Yes Routes 2.2 works Python 2 and 3 using a single code base.
sphinxcontrib-docbookrestapi Yes Yes In the Git repo, not on PyPI
sphinxcontrib-httpdomain Yes N/A No
sphinxcontrib-pecanwsme No No No
thrift No No No
websockify Yes N/A Yes python3 works with the 0.7.0 version released on Jul 23, 2015

OpenStack applications (tc-approved)

OpenStack applications approved by the OpenStack Technical Committee.

Project Python 3 compatibility py34 gate Comments
ceilometer Yes voting
ironic Yes voting

Python 3.4 unit tests are now being run for openstack/ironic. The unit tests are a voting job. Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4: Run tests in py34 environment

sahara Yes voting
cinder work in progress voting

Update: We have passed feature freeze, should hold this for Mitaka.

cinder-python3 blueprint (written by Victor Stinner) has been accepted for Liberty.

Patches: bp/cinder-python3 topic.

glance work in progress voting

Victor Stinner is working on porting Glance to Python3.

Patches: Patches for glance (topic: py3).

heat work in progress voting

Python34 Support spec (by Sirushti Murugesan) accepted for Liberty.

Patches: bp/heat-python34-support topic.

horizon work in progress voting

Update: David Lyle blocked Python 3 patches until the next M cycle to focus on stabilize

George Peristerakis wrote Make Horizon python3.4 compatible in May 2015, the blueprint was approved.

Patches: bp/porting-python3 topic.

Some tests are still blocked a Python 3 bug in django_openstack_auth 1.4.0. The fix is already merged, but a release is still needed.

babel 2.0: openstack_dashboard.dashboards.settings.user.tests.UserSettingsTest fails with a TypeError on babel.core.get_global(): it's a bug in babel, see Locale.parse('zh_CN') raises a TypeError on Babel 2.0 with Python 3.3 and 3.4 and the fix Avoid incompatible *.dat files between Python 2 and 3. See the bug Need a new release to fix Python 3 compatibility.

keystone work in progress voting

The spec Add spec for python-3 compatibility (by Morgan Fainberg) was accepted for Liberty. See also the blueprint python3 started by Dolph Mathews on 2014-07-22; David Stanek is working on it.

Patches: bp/python3 topic.

pysaml2 3.0 now supports Python 3.

Aslo blocked by ldappool and python-ldap. pyldap is a fork of python-ldap which adds Python 3 support. python-ldap may be replaced with ldap3, but it requires more work.

neutron work in progress voting

Porting to Python 3 spec (by Cyril Roelandt) accepted for Liberty, port in progress. Patches:

nova work in progress voting

Update:"Given all the current patches have merged, and the agreement to timebox this effort, lets call this the end of things for liberty. --johnthetubaguy 5th August 2015".

Adding Python 3.4 support to Nova spec (by Victor Stinner) accepted for Liberty. Port in progress.


See also Py3.4 Compatibility in Liberty nova priorities tracking.

swift work in progress non voting

September 2015: Development now focused on fixing last erasure code (EC) bugs, Python 3 is out of the scope of this development cycle (september).

swift now depends on six. (Jaivish Kothari is also working on this)

Fixing "tox -e py34" requires:



trove No No

Victor Stinner proposed the trove-python3 blueprint for Mitaka.

Patches: search for the bp/trove-python3 topic.

Other OpenStack Applications and Projects

List of all OpenStack projects: projects.yaml.

Project Python 3 compatibility py34 gate Comments
aodh Yes voting
gnocchi Yes voting
manila Yes but... voting

Valeriy Ponomaryov wrote the py3-compatibility blueprint.

SSL tests are skipped because of the bug requests to SSL wrapped sockets hang while reading using py3.

rally Yes voting A big thank to Andrey Kurilin for a lot of work in this direction
searchlight Yes voting
zaqar Yes voting
mistral work in progress non voting
solum work-in-progress non-voting
barbican work in progress non-voting Pradeep Kumar Singh proposed the blueprint barbican-py3 for Liberty.

Patches: search for the bp/barbican-py3 topic.

congress work-in-progress non-voting
cue work-in-progress non-voting
designate work in progress No Pradeep Kumar Singh is working on the Python 3 port, see Pradeep's patches (only open patches).
magnetodb No No
magnum No No
murano No No

Unsorted projects:

  • App Catalog
  • Chef OpenStack (Ruby)
  • Kolla: ~2100 lines of Python, but not "tox -e py27" target
  • OpenStack Ansible (Python): Ansible playbooks and roles for deployment
  • Puppet OpenStack (Ruby)
  • TripleO

Functional tests

Doug Hellmann wrote a cross-project specification to run integration tests on Python 3: Enabling Python 3 for Application Integration Tests (spec approved for Liberty).

Patch for DevStack: Enable optional Python 3 support.

Status of Python 3 in Linux distributions

Reports at OpenStack Summits


Pycon Montreal 2014: Sprint Port OpenStack to Python 3

Enovance organized a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday) 2014. See the page Python3/SprintPycon2014.