Jump to: navigation, search

Difference between revisions of "Python3"

m (Drop wiki link to page that just points back to here)
(Link to caniusepython3.com for global-requirements.txt)
Line 131: Line 131:
  
 
== Python 3 of OpenStack Dependencies ==
 
== Python 3 of OpenStack Dependencies ==
 +
 +
[https://caniusepython3.com/check/4fd5dda2-b1f1-4db4-a636-67cd3276cb6a Porting status] for [https://github.com/openstack/requirements/blob/master/global-requirements.txt global-requirement.txt].
  
 
'''Blocker Pointer:''' it's not yet possible to specify different list of dependencies for Python 2 and Python 3. For example, mox only works on Python 2, mox3 can be used on Python 3.
 
'''Blocker Pointer:''' it's not yet possible to specify different list of dependencies for Python 2 and Python 3. For example, mox only works on Python 2, mox3 can be used on Python 3.

Revision as of 13:33, 15 May 2014

This page tracks the progress of Python 3 effort porting for OpenStack.

Python 3

Why should OpenStack move to Python 3 right now?

Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.

Pycon Montreal 2014: Sprint Port OpenStack to Python 3

Enovance organizes a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday). See the page Python3/SprintPycon2014.

OpenStack Summit 2014 at Atlanta (Juno): The Future of Python Support

Doug Hellmann proposed a cross-project workshop: http://summit.openstack.org/cfp/details/316

Etherpad: https://etherpad.openstack.org/p/juno-cross-project-future-of-python

Port Python 2 code to Python 3

OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack must still support Python 2.6 for RHEL, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).

Common patterns

  • Replace dict.iteritems() with six.iteritems(dict)
  • Replace iterator.next() with next(iterator)
  • Replace basestring with six.string_types
  • Replace unicode with six.text_type

bytes.decode and unicode.encode

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write data.decode() or text.encode() without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: data.decode('utf-8') or text.encode('utf-8'). The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

safe_decode

Olso Incubator has a function safe_decode() which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_decode(data, 'utf-8'): decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:

  • safe_decode(data)

By default, the decoder is strict. You can specify a different error handler using the optional errors parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

safe_encode

Olso Incubator has a function safe_encode() which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.

  • safe_encode(text) encodes text to the output encoding
  • safe_encode(bytes) may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (incomding parameter) is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (encoding parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_encode(data, incoming='utf-8'): encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:

  • safe_encode(data)

Example:

  • safe_encode(b'\xe9', incoming='latin-1') returns b'\xc3\xa9'.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional errors parameter. Example: safe_encode(b'[\xff]', incoming='ascii', errors='ignore') returns b'[]'.

HTTP

The HTTP protocol is based on bytes:

  • HTTP body contains bytes. For example, use io.BytesIO for a stream storing an HTTP body.
  • HTTPConnection.getresponse().read() returns bytes (in Python 3, str which is bytes in Python 2)
  • On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
  • It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and HTTP header should use what character encoding?

References to port Python 2 code to Python 3

Common pitfalls

What is a string ?

You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:

   >>> type('foo')
   <type 'str'>
   >>> type(u'foo')
   <type 'unicode'>
   >>> type(b'foo')
   <type 'str'>
   >>> isinstance('foo', six.text_type)
   False
   >>> isinstance(u'foo', six.text_type)
   True
   >>> bytes is str
   True
   >>> b'foo'[0]
   'f'

Python 3:

   >>> type('foo')
   <class 'str'>
   >>> type(u'foo')
   <class 'str'>
   >>> type(b'foo')
   <class 'bytes'>
   >>> isinstance('foo', six.text_type)
   True
   >>> isinstance(b'foo', six.text_type)
   False
   >>> bytes is str
   False
   >>> b'foo'[0]
   102

Python 3 of OpenStack Dependencies

Porting status for global-requirement.txt.

Blocker Pointer: it's not yet possible to specify different list of dependencies for Python 2 and Python 3. For example, mox only works on Python 2, mox3 can be used on Python 3.

OpenStack Dependencies:

  • mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock)
  • eventlet: not available on Python 3 yet, alternatives: asyncio (Tulip for Python 3.3+/Trollius for Python 2), Tornado

Portage in progress

  • Oslo Messaging: Portage in Progress by Victor Stinner (dashboard)
  • glanceclient: Portage in Progress by Cyril Roelandt (dashboard)
  • heatclient: Portage in Progress by Cyril Roelandt (dashboard)
  • neutronclient: Portage in Progress by Cyril Roelandt (dashboard)
  • glanceclient: Portage in Progress by Cyril Roelandt (dashboard)

Portage done

  • keystoneclient: Portage in Progress by Cyril Roelandt (dashboard)

Python 3 Status of OpenStack projects

Oslo Incubator

BLOCKER BUG: Tests using testscenarios fail on Python 3 with nosetests because of this bug: https://bugs.launchpad.net/testscenarios/+bug/872887

Recently merged reviews:


Test (full path) Patches Comment
tests/unit/config/test_generator.py https://review.openstack.org/#/c/88087/
tests/unit/crypto/test_utils.py https://review.openstack.org/87413
tests/unit/db/sqlalchemy/test_migration_common.py
tests/unit/db/sqlalchemy/test_migrate.py
tests/unit/db/sqlalchemy/test_models.py https://review.openstack.org/#/c/80307/
tests/unit/db/sqlalchemy/test_options.py https://review.openstack.org/#/c/80627/
tests/unit/db/sqlalchemy/test_migrate_cli.py
tests/unit/db/sqlalchemy/test_utils.py
tests/unit/db/sqlalchemy/test_sqlalchemy.py
tests/unit/fixture/test_logging.py https://review.openstack.org/#/c/90318/
tests/unit/middleware/test_request_id.py https://review.openstack.org/#/c/80336/
tests/unit/middleware/test_sizelimit.py https://review.openstack.org/#/c/80450/
tests/unit/middleware/test_audit.py depends on pycadf
tests/unit/reports/test_guru_meditation_report.py https://review.openstack.org/#/c/87404/
tests/unit/reports/test_base_report.py https://review.openstack.org/#/c/87973/
tests/unit/reports/test_openstack_generators.py https://review.openstack.org/#/c/88124/
tests/unit/reports/test_views.py https://review.openstack.org/87376
tests/unit/rpc/test_common.py https://review.openstack.org/#/c/80533/ The RPC code in the incubator is deprecated in favor of oslo.messaging. --doug-hellmann (talk) 15:40, 14 April 2014 (UTC)
tests/unit/scheduler/test_base_filter.py https://review.openstack.org/#/c/80321/
tests/unit/scheduler/test_weights.py https://review.openstack.org/#/c/87336/
tests/unit/test_cliutils https://review.openstack.org/#/c/74433/
tests/unit/test_fileutils https://review.openstack.org/#/c/74728/
tests/unit/test_gettext.py https://review.openstack.org/#/c/80534/
tests/unit/test_imageutils.py https://review.openstack.org/#/c/90532/
tests/unit/test_jsonutils.py https://review.openstack.org/#/c/80370/
tests/unit/test_log.py depends on https://review.openstack.org/#/c/80534/
tests/unit/test_quota.py https://review.openstack.org/#/c/80564/
tests/unit/test_strutils.py

OpenStack clients

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
python-ceilometerclient Yes Voting On PyPI
python-cinderclient Yes Voting On PyPI
python-ganttclient  ?  ?  ?  ?
python-glanceclient In Progress Non-voting No
python-heatclient Yes Voting On PyPI
python-ironicclient Yes Voting On PyPI
python-keystoneclient Yes Voting On PyPI
python-marconiclient Yes Voting Yes
python-melangeclient  ?  ?  ?
python-novaclient Yes Voting In the Git repo, not on PyPI
python-neutronclient In progress Non-voting Differences between Python 2 and 3
python-openstackclient In Progress Non-Voting No Works with glanceclient HEAD
python-savannaclient In progress Non-voting https://review.openstack.org/#/c/73128/
python-swiftclient In progress Non-voting `No Differences between Python 2 and 3
python-tuskarclient Yes Voting On PyPI
python-troveclient Yes Voting On PyPI

Core OpenStack projects

Project Python 3 compatibility CI tests running? Trove classifiers Blocked by Comment
ceilometer No No No
  • eventlet
  • hacking (tests only)
  • mysql-python
  • oslosphinx (tests only)
  • python-ceilometerclient
  • python-glanceclient
  • python-swiftclient
  • sphinxcontrib-docbookrestapi (tests only)
  • sphinxcontrib-httpdomain (tests only)
  • sphinxcontrib-pecanwsme (tests only)
  • sqlalchemy-migrate
  • swift (tests only)
  • thrift (which is blocking happybase)
cinder No No No
  • eventlet
  • hacking (tests only)
  • mysql-python (tests only)
  • oslo.rootwrap
  • oslosphinx (tests only)
  • paste
  • python-glanceclient
  • python-swiftclient
  • rtslib-fb
  • sqlalchemy-migrate
  • suds
  • taskflow
glance No No No
  • boto
  • eventlet
  • hacking (tests only)
  • mysql-python (tests only)
  • nose-exclude (tests only)
  • nose-htmloutput (tests only)
  • openstack.nose_plugin (test only)
  • oslo.messaging
  • oslosphinx
  • paste
  • python-cinderclient
  • python-swiftclient
  • qpid-python (tests only)
  • sqlalchemy-migrate
heat No No No
  • eventlet
  • hacking (tests only)
  • oslosphinx (tests only)
  • python-ceilometerclient
  • python-cinderclient
  • python-glanceclient (tests only)
  • python-heatclient
  • python-neutronclient
  • python-swiftclient
  • python-troveclient
  • qpid-python
  • sqlalchemy-migrate
horizon No No No
  • django-compressor
  • django-openstack-auth
  • eventlet
  • hacking (tests)
  • nose-exclude (tests)
  • nosehtmloutput (tests)
  • openstack.nose-plugin (tests)
  • oslo.sphinx (tests)
  • python-ceilometerclient
  • python-cinderclient
  • python-glanceclient
  • python-heatclient
  • python-neutronclient
  • python-swiftclient
  • python-troveclient
keystone No No No
  • eventlet
  • hacking (tests only)
  • oslo.messaging
  • oslosphinx (tests only)
  • pam
  • paste
  • pycadf
  • python-ldap (tests only)
  • sqlalchemy-migrate
neutron No No No
  • eventlet
  • hacking (hacking)
  • jsonrpclib
  • oslo.rootwrap
  • paste
  • python-neutronclient
nova No No No
  • boto
  • eventlet
  • hacking (tests only)
  • mysql-python (tests only)
  • oslo.messaging
  • oslo.rootwrap
  • oslosphinx (tests only)
  • paste
  • pycadf
  • python-cinderclient
  • python-glanceclient
  • python-neutronclient
  • sqlalchemy-migrate
  • suds
  • websockify
swift No No No
  • dnspython
  • eventlet
  • hacking (tests only)
  • netifaces
  • nosehtmloutput (tests only)
  • openstack.nose-plugin (tests only)
  • python-swiftclient (tests only)

Dependencies

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
boto No N/A No See https://github.com/boto/boto3 (experimental)
django-compressor No No No https://github.com/django-compressor/django-compressor/issues/484
django-openstack-auth No No No Ported: https://review.openstack.org/#/dashboard/8122
dnspython Yes N/A Yes Must use the Python 3 version, see https://github.com/rthalley/dnspython/issues/60
ecdsa Yes N/A In the Git repo Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
eventlet No No No Victor Stinner is working on Trollius (asyncio for Python 2) which may replace eventlet: Use the new asyncio module and Trollius in OpenStack
hacking No No No Cyril Roelandt patch: Make hacking Python 3 compatible
jsonrpclib No N/A No The project seems dead :(
mysql-python No No No 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there.
netifaces No N/A No Patch sent by Victor Stinner (in private): netifaces_python3.patch, Debian has patches too
nose-exclude No No No https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
nosehtmloutput No No No
  • nose-exclude (tests only)
  • openstack.nose-plugin
nosexcover No N/A On PyPI Python 3 support since 1.0.9
openstack.nose-plugin No No No
oslo.vmware No Voting Yes suds
oslo.config Yes Voting In the git repo, not on PyPI
oslo.messaging No No No
oslo.rootwrap Yes Yes In the Git repo, not on PyPI (1.1.0)
oslosphinx Yes No tests :) In the git repo, not on PyPI https://review.openstack.org/#/c/79311/
oslo.sphinx No No No Must be replaced by oslosphinx (without the dot)
pam No No No The fork simplepam works on Python 2 and 3
paramiko Yes N/A On PyPI   Get https://review.openstack.org/#/c/81132/ merged
paste No No No https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
pycadf No No No
python-ldap No No No The project seems dead.
qpid-python No No No
rtslib-fb No No No
sphinxcontrib-docbookrestapi No No No
sphinxcontrib-httpdomain No No No
sphinxcontrib-pecanwsme No No No
sqlalchemy-migrate No No No
  • hacking
  • ibm-db-sa
  • scripttest
suds No No No Dead project
taskflow Yes Yes Yes
thrift No No No
websockify No No No

Reports at OpenStack Summits

Dependencies

This is an attempt to document which OpenStack dependencies work under Python 3. We're starting with oslo, since it's the base of everything else, then gradually including other projects.

The caniusepython3 tool can be used to do quick yes/no Python 3 support checks.

oslo.config

pip-requires

  • argparse - included with Python 2.7+. The separate package is only needed for 2.6.

test-requires

  • mox - mox3 supports Python, or replace mox with mock (which is now included in python 3.3)
  • nose - supports Python 3
  • nose-exclude - 2.6-2.7, 3.1-3.3
  • testtools - 2.6-2.7, 3.2-3.3
  • coverage - supports 2.3-3.3
  • sphinx - supports Python 3

oslo-incubator

pip-requires

  • PasteDeploy: supports 2.5-3.3
  • WebOb: support Python 3
  • eventlet: NO (MAJOR PAIN POINT) (gevent doesn't either, though there are some old forks that tried)
  • greenlet: supports 2.4-3.2 (lack of 3.3 may be false negative)
  • lxml: supports 2.4-3.3
  • routes: supports 2.6-3.3
  • iso8601: NO (perhaps try python-dateutil, specifically the parser module?)
  • anyjson: 2.4-3.1 (lack of 3.2-3.3 may be false negatives)
  • kombu: supports Python 3 (exact version not given)
  • argparse: included in Python 2.7+
  • stevedore: supports 2.7, 3.2, and 3.3
  • SQLAlchemy: supports Python 3 (exact version not given)
  • qpid-python: NO

test-requires

  • distribute: 2.4-3.3
  • coverage: 2.3-3.3
  • fixtures: supports Python 3 (exact version not given)
  • mock: 2.5-3.3
  • mox: use mox3, or replace mox with mock (which is now included in python 3.3)
  • mysql-python: NO (maybe try pymysql instead?)
  • nose: Supports Python 3
  • nose-exclude: 2.6-2.7, 3.1-3.3
  • nosehtmloutput: ?
  • pep8: Supports Python 3
  • pyflakes: Supports Python 3
  • pylint: Supports Python 3 (tested with Python 3.2)
  • pyzmq: Supports 2.6-2.7, 3.2+
  • redis: Supports 2.5-2.7, 3.2+
  • setuptools-git: 2.4-2.7, 3.1-3.3
  • sphinx: Supports Python 3
  • testtools: 2.6-2.7, 3.2-3.3
  • webtest: 2.6-2.7, 3.2-3.3