Jump to: navigation, search

Python3

Revision as of 14:31, 3 March 2014 by CyrilRoelandteNovance (talk | contribs) (Dependencies)

This page tracks the progress of Python 3 effort porting for OpenStack.

Python 3

Why should OpenStack move to Python 3 right now?

Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.

Port Python 2 code to Python 3

OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions.

References to port Python 2 code to Python 3

Common pitfalls

What is a string ?

You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:

   >>> type('foo')
   <type 'str'>
   >>> type(u'foo')
   <type 'unicode'>
   >>> type(b'foo')
   <type 'str'>
   >>> isinstance('foo', six.text_type)
   False
   >>> isinstance(u'foo', six.text_type)
   True
   >>> bytes is str
   True
   >>> b'foo'[0]
   'f'

Python 3:

   >>> type('foo')
   <class 'str'>
   >>> type(u'foo')
   <class 'str'>
   >>> type(b'foo')
   <class 'bytes'>
   >>> isinstance('foo', six.text_type)
   True
   >>> isinstance(b'foo', six.text_type)
   False
   >>> bytes is str
   False
   >>> b'foo'[0]
   102

Python 3 of OpenStack Dependencies

Blocker Pointer: it's not yet possible to specify different list of dependencies for Python 2 and Python 3. For example, mox only works on Python 2, mox3 can be used on Python 3.

OpenStack Dependencies:

  • mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock)
  • eventlet: not available on Python 3 yet, alternatives: asyncio (Tulip for Python 3.3+/Trollius for Python 2), Tornado

Portage in progress

  • Oslo Messaging: Portage in Progress by Victor Stinner (dashboard)
  • glanceclient: Portage in Progress by Cyril Roelandt (dashboard)
  • heatclient: Portage in Progress by Cyril Roelandt (dashboard)
  • neutronclient: Portage in Progress by Cyril Roelandt (dashboard)
  • glanceclient: Portage in Progress by Cyril Roelandt (dashboard)

Portage done

  • keystoneclient: Portage in Progress by Cyril Roelandt (dashboard)

Python 3 Status of OpenStack projects

See also Python3Deps.

OpenStack clients

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
python-ceilometerclient Yes Voting In the Git repo, not on PyPI
python-cinderclient Yes Voting https://review.openstack.org/#/c/73844/
python-ganttclient  ?  ?  ?  ?
python-glanceclient In Progress Non-voting python-keystoneclient
python-heatclient In progress Non-voting https://review.openstack.org/#/dashboard/8122
python-ironicclient Yes Voting On PyPI
python-keystoneclient Yes Voting https://review.openstack.org/#/c/74543/
python-marconiclient Yes Voting No
python-melangeclient  ?  ?  ?
python-novaclient Yes Voting https://review.openstack.org/#/c/75964/
python-neutronclient In progress Non-voting Differences between Python 2 and 3
python-openstackclient In Progress Non-Voting Works with glanceclient HEAD
python-savannaclient In progress Non-voting python-keystoneclient https://review.openstack.org/#/c/73128/
python-swiftclient In progress Non-voting Differences between Python 2 and 3
python-tuskarclient Yes Voting No https://review.openstack.org/#/c/73130/
python-troveclient Yes Voting https://review.openstack.org/#/c/75075/

Core OpenStack projects

Project Python 3 compatibility CI tests running? Trove classifiers Blocked by Comment
ceilometer No No No
  • eventlet
  • hacking
  • mysql-python
  • oslo.config
  • oslosphinx
  • python-ceilometerclient
  • python-glanceclient
  • python-swiftclient
  • sphinxcontrib-docbookrestapi
  • sphinxcontrib-httpdomain
  • sphinxcontrib-pecanwsme
  • sqlalchemy-migrate
  • swift
  • thrift (which is blocking happybase)
cinder No No No
  • ecdsa (which is blocking paramiko)
  • eventlet
  • hacking
  • mysql-python
  • oslo.config
  • oslo.rootwrap
  • oslo.sphinx
  • paste
  • python-glanceclient
  • python-swiftclient
  • rtslib-fb
  • sqlalchemy-migrate
  • suds
  • taskflow
glance No No No
  • boto
  • eventlet
  • hacking
  • mysql-python
  • oslo.config
  • oslo.messaging
  • oslosphinx
  • paste
  • python-cinderclient
  • python-swiftclient
  • qpid-python
  • sqlalchemy-migrate
heat No No No
  • ecdsa (which is blocking paramiko)
  • eventlet
  • hacking
  • oslo.config
  • oslo.sphinx
  • python-ceilometerclient
  • python-cinderclient
  • python-glanceclient
  • python-heatclient
  • python-neutronclient
  • python-swiftclient
  • python-troveclient
  • qpid-python
  • sqlalchemy-migrate
horizon No No No
  • django-compressor
  • django-openstack-auth
  • eventlet
  • hacking
  • nose-exclude
  • nosehtmloutput
  • nosexcover
  • openstack.nose-plugin
  • oslo.sphinx
  • python-ceilometerclient
  • python-cinderclient
  • python-glanceclient
  • python-heatclient
  • python-neutronclient
  • python-swiftclient
  • python-troveclient
keystone No No No
  • eventlet
  • hacking
  • oslo.config
  • oslo.messaging
  • oslosphinx
  • pam
  • paste
  • pycadf
  • python-ldap
  • sqlalchemy-migrate
neutron No No No
  • eventlet
  • hacking
  • jsonrpclib
  • oslo.config
  • oslo.rootwrap
  • paste
  • python-neutronclient
nova No No No
  • boto
  • ecdsa (which is blocking paramiko)
  • eventlet
  • hacking
  • mysql-python
  • oslo.config
  • oslo.messaging
  • oslo.rootwrap
  • oslosphinx
  • paste
  • pycadf
  • python-cinderclient
  • python-glanceclient
  • python-neutronclient
  • sqlalchemy-migrate
  • suds
  • websockify
swift No No No
  • dnspython
  • eventlet
  • hacking
  • netifaces
  • nosehtmloutput
  • nosexcover
  • openstack.nose-plugin
  • python-swiftclient


Dependencies

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
boto No No No
django-compressor No No No https://github.com/django-compressor/django-compressor/issues/484
django-openstack-auth No No No
dnspython No No No https://github.com/rthalley/dnspython/issues/60
ecdsa No No No
eventlet No No No
hacking No No No
jsonrpclib No No No
mysql-python No No No
netifaces No No No
nose-exclude No No No
nosehtmloutput No No No
nosexcover No No No
openstack.nose-plugin No No No
oslo.config No No No
oslo.messaging No No No
oslo.rootwrap No No No
oslosphinx No No No
oslo.sphinx No No No
pam No No No
paste No No No
pycadf No No No
python-ldap No No No
qpid-python No No No
rtslib-fb No No No
sphinxcontrib-docbookrestapi No No No
sphinxcontrib-httpdomain No No No
sphinxcontrib-pecanwsme No No No
sqlalchemy-migrate No No No
suds No No No
taskflow No No No
thrift No No No
websockify No No No

Reports at OpenStack Summits

Dependencies

This is an attempt to document which OpenStack dependencies work under Python 3. We're starting with oslo, since it's the base of everything else, then gradually including other projects.

The caniusepython3 tool can be used to do quick yes/no Python 3 support checks.

oslo.config

pip-requires

  • argparse - included with Python 2.7+. The separate package is only needed for 2.6.

test-requires

  • mox - mox3 supports Python, or replace mox with mock (which is now included in python 3.3)
  • nose - supports Python 3
  • nose-exclude - 2.6-2.7, 3.1-3.3
  • testtools - 2.6-2.7, 3.2-3.3
  • coverage - supports 2.3-3.3
  • sphinx - supports Python 3

oslo-incubator

pip-requires

  • PasteDeploy: supports 2.5-3.3
  • WebOb: support Python 3
  • eventlet: NO (MAJOR PAIN POINT) (gevent doesn't either, though there are some old forks that tried)
  • greenlet: supports 2.4-3.2 (lack of 3.3 may be false negative)
  • lxml: supports 2.4-3.3
  • routes: supports 2.6-3.3
  • iso8601: NO (perhaps try python-dateutil, specifically the parser module?)
  • anyjson: 2.4-3.1 (lack of 3.2-3.3 may be false negatives)
  • kombu: supports Python 3 (exact version not given)
  • argparse: included in Python 2.7+
  • stevedore: supports 2.7, 3.2, and 3.3
  • SQLAlchemy: supports Python 3 (exact version not given)
  • qpid-python: NO

test-requires

  • distribute: 2.4-3.3
  • coverage: 2.3-3.3
  • fixtures: supports Python 3 (exact version not given)
  • mock: 2.5-3.3
  • mox: use mox3, or replace mox with mock (which is now included in python 3.3)
  • mysql-python: NO (maybe try pymysql instead?)
  • nose: Supports Python 3
  • nose-exclude: 2.6-2.7, 3.1-3.3
  • nosexcover: NO (bug in tracker from 2012-05; no comments)
  • nosehtmloutput: ?
  • pep8: Supports Python 3
  • pyflakes: Supports Python 3
  • pylint: Supports Python 3 (tested with Python 3.2)
  • pyzmq: Supports 2.6-2.7, 3.2+
  • redis: Supports 2.5-2.7, 3.2+
  • setuptools-git: 2.4-2.7, 3.1-3.3
  • sphinx: Supports Python 3
  • testtools: 2.6-2.7, 3.2-3.3
  • webtest: 2.6-2.7, 3.2-3.3