Jump to: navigation, search

Difference between revisions of "Python3"

(Pycon Montreal 2014: Sprint Port OpenStack to Python 3)
(Dependencies)
Line 605: Line 605:
 
|-
 
|-
  
| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: orange;" | In the Git repo || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
+
| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
 
|-
 
|-
  

Revision as of 13:49, 10 April 2015

This page tracks the progress of Python 3 effort porting for OpenStack.

Python 3

Why should OpenStack move to Python 3 right now?

Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.

Port Python 2 code to Python 3

OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).

Common patterns

  • Replace dict.iteritems() with six.iteritems(dict)
  • Replace iterator.next() with next(iterator)
  • Replace basestring with six.string_types
  • Replace unicode with six.text_type

bytes.decode and unicode.encode

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write data.decode() or text.encode() without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: data.decode('utf-8') or text.encode('utf-8'). The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

safe_decode

Olso Incubator has a function safe_decode() which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_decode(data, 'utf-8'): decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:

  • safe_decode(data)

By default, the decoder is strict. You can specify a different error handler using the optional errors parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

safe_encode

Olso Incubator has a function safe_encode() which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.

  • safe_encode(text) encodes text to the output encoding
  • safe_encode(bytes) may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (incomding parameter) is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (encoding parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_encode(data, incoming='utf-8'): encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:

  • safe_encode(data)

Example:

  • safe_encode(b'\xe9', incoming='latin-1') returns b'\xc3\xa9'.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional errors parameter. Example: safe_encode(b'[\xff]', incoming='ascii', errors='ignore') returns b'[]'.

logging module and format exceptions

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs b'hello' instead of 'hello'.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

  • str(exc): native string, so use bytes on Python 2
  • six.text_type(exc): always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: unicode(Exception("nonascii:\xe9")).
  • six.u(str(exc)): unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: unicode(str(Exception("\xff")))
  • LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using str(exc). Example: LOG.debug(str(exc)).

HTTP

The HTTP protocol is based on bytes:

  • HTTP body contains bytes. For example, use io.BytesIO for a stream storing an HTTP body.
  • HTTPConnection.getresponse().read() returns bytes (in Python 3, str which is bytes in Python 2)
  • On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
  • It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and HTTP header should use what character encoding?

References to port Python 2 code to Python 3

Common pitfalls

What is a string ?

You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:

   >>> type('foo')
   <type 'str'>
   >>> type(u'foo')
   <type 'unicode'>
   >>> type(b'foo')
   <type 'str'>
   >>> isinstance('foo', six.text_type)
   False
   >>> isinstance(u'foo', six.text_type)
   True
   >>> bytes is str
   True
   >>> b'foo'[0]
   'f'

Python 3:

   >>> type('foo')
   <class 'str'>
   >>> type(u'foo')
   <class 'str'>
   >>> type(b'foo')
   <class 'bytes'>
   >>> isinstance('foo', six.text_type)
   True
   >>> isinstance(b'foo', six.text_type)
   False
   >>> bytes is str
   False
   >>> b'foo'[0]
   102

tox/testr error: db type could not be determined

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/".

Python 3 Status of OpenStack projects

Oslo Incubator

BLOCKER BUG: Tests using testscenarios fail on Python 3 with nosetests because of this bug: https://bugs.launchpad.net/testscenarios/+bug/872887

Recently merged reviews:


Test (full path) Patches Comment
tests/unit/config/test_generator.py
tests/unit/crypto/test_utils.py
tests/unit/reports/test_guru_meditation_report.py
tests/unit/reports/test_base_report.py
tests/unit/reports/test_openstack_generators.py
tests/unit/reports/test_views.py
tests/unit/scheduler/test_base_filter.py
tests/unit/scheduler/test_weights.py
tests/unit/test_cliutils
tests/unit/test_deprecated.py
tests/unit/test_eventlet_backdoor.py
tests/unit/test_fileutils
tests/unit/test_imageutils.py
tests/unit/test_local.py
tests/unit/test_loopingcall.py https://review.openstack.org/172356
tests/unit/test_memorycache.py
tests/unit/test_periodic.py
tests/unit/test_quota.py
tests/unit/test_service.py
tests/unit/test_systemd.py
tests/unit/test_threadgroup.py https://review.openstack.org/172356
tests/unit/test_versionutils.py

Common Libraries (Oslo Projects)

For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160

Project Python 3 compatibility Comment
cliff Yes
oslo.concurrency Partial https://review.openstack.org/#/c/146984/
oslo.config Yes
oslo.db Partial No unit tests with MySQL because of DB driver
oslo.i18n Yes
oslo.log Yes
oslo.messaging Yes https://review.openstack.org/#/c/172135/ enables eventlet tests on Python 3
oslo.middleware Yes
oslo.rootwrap Yes
oslo.serialization Yes
oslosphinx  ? The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible?
oslotest Yes
oslo.versionedobjects Yes
oslo.vmware No Blocked suds dependency
oslo.utils Yes
oslo.version Yes not released on PyPI yet
pylockfile Yes related to https://pypi.python.org/pypi/lockfile ?
stevedore Yes
taskflow Yes

Development tools

Project Python 3 compatibility Comment
cookiecutter yes
hacking yes py33 gate is not voting
pbr yes
stackforge/python-jenkins yes py33 gate is voting
openstack-infra/jenkins-job-builder partial https://review.openstack.org/172238

OpenStack clients

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
python-barbicanclient Yes Voting On PyPI
python-ceilometerclient Yes Voting On PyPI
python-cinderclient Yes Voting On PyPI
python-ganttclient  ?  ?  ?  ?
python-glanceclient Yes Voting On PyPI
python-heatclient Yes Voting On PyPI
python-ironicclient Yes Voting On PyPI
python-keystoneclient Yes Voting On PyPI
python-marconiclient Yes Voting Yes
python-melangeclient  ?  ?  ?
python-novaclient Yes Voting On PyPII
python-neutronclient Yes Voting Yes
python-openstackclient OK Voting Yes As of 0.9
python-savannaaclient In progress Non-voting https://review.openstack.org/#/c/73128/
python-saharaclient OK Voting In the git repo
python-swiftclient Yes Voting On PyPI
python-tuskarclient Yes Voting On PyPI
python-troveclient Yes Voting On PyPI

Core OpenStack projects

Completely updated on Monday, September the 29th.

Project Python 3 compatibility CI tests running? Trove classifiers Blocked by Comment
ceilometer No No No

Requirements

  • croniter
  • thrift (which is blocking happybase)
  • oslo.db
  • oslo.messaging
  • sqlalchemy-migrate

Requirements for tests

  • mysql-python
  • sphinxcontrib-docbookrestapi
  • sphinxcontrib-httpdomain
  • sphinxcontrib-pecanwsme
cinder No No No

Requirements

  • oslo.db
  • oslo.messaging
  • paste
  • rtslib-fb
  • sqlalchemy-migrate
  • suds

Requirements for tests

  • mysql-python
glance No No No

Requirements

  • glance_store (partial py3 support, blocked by suds)
  • oslo.db
  • oslo.messaging
  • paste
  • sqlalchemy-migrate

Requirements for tests

  • mysql-python
  • qpid-python
heat No No No

Requirements

  • oslo.db
  • oslo.messaging
  • qpid-python
  • sqlalchemy-migrate

Requirements for tests

  • mysql-python
horizon No No No

Requirements

  • django-pyscss
  • xstatic
  • xstatic-angular
  • xstatic-angular-cookies
  • xstatic-angular-mock
  • xstatic-bootstrap-datepicker
  • xstatic-d3
  • xstatic-font-awesome
  • xstatic-hogan
  • xstatic-jasmine
  • xstatic-jquery
  • xstatic-jquery-migrate
  • xstatic-jquery.quicksearch
  • xstatic-jquery.tablesorter
  • xstatic-jsencrypt
  • xstatic-qunit
  • xstatic-rickshaw
  • xstatic-spin

Requirements for tests

  • nodeenv
  • nose-exclude
  • nosehtmloutput
  • openstack.nose_plugin
keystone No No No
  • oslo.db
  • oslo.messaging
  • paste
  • sqlalchemy-migrate

Requirements for tests

  • ldappool
  • paste (which is blocking pysaml2)
  • python-ldap
neutron No No No

Requirements

  • jsonrpclib
  • oslo.db
  • oslo.messaging
  • paste
nova No No No
  • oslo.db
  • oslo.messaging
  • paste
  • sqlalchemy-migrate
  • suds
  • websockify

Requirements for tests

  • libvirt-python
  • mysql-python
swift No No No

Requirements:

Requirements for tests:

  • nosehtmloutput
  • openstack.nose_plugin

Number of core OpenStack projetcs blocked by each dependency:

     7 oslo.messaging
     7 oslo.db
     6 sqlalchemy-migrate
     6 paste
     5 mysql-python
     2 suds
     2 qpid-python
     2 openstack.nose_plugin
     2 nosehtmloutput
     1 xstatic-spin
     1 xstatic-rickshaw
     1 xstatic-qunit
     1 xstatic-jsencrypt
     1 xstatic-jquery.tablesorter
     1 xstatic-jquery.quicksearch
     1 xstatic-jquery-migrate
     1 xstatic-jquery
     1 xstatic-jasmine
     1 xstatic-hogan
     1 xstatic-font-awesome
     1 xstatic-d3
     1 xstatic-bootstrap-datepicker
     1 xstatic-angular-mock
     1 xstatic-angular-cookies
     1 xstatic-angular
     1 xstatic
     1 websockify
     1 thrift
     1 sphinxcontrib-pecanwsme
     1 sphinxcontrib-httpdomain
     1 sphinxcontrib-docbookrestapi
     1 rtslib-fb
     1 python-ldap
     1 nose-exclude
     1 nodeenv
     1 libvirt-python
     1 ldappool
     1 jsonrpclib
     1 glance_store
     1 django-pyscss
     1 croniter

eventlet and Python 3

eventlet 0.15 is the first release supporting Python 3, but monkey-patching did not work with Python 3. Monkey-patching partially works on Python 3 with eventlet 0.16. All monkey-patching should be fixed in the future eventlet 0.17.3 release.

Status of eventlet+Python 3 in OpenStack components.

Working:

  • oslo.db: tests running on Python 3.4 with eventlet (0.17.2)
  • oslo.concurrency: tests running on Python 3.4 with eventlet (0.17.1)

Don't work:

  • oslo.messaging: eventlet currently disabled in tests, tests fail even if the development version of eventlet

Unknown status:

  • oslo-incubator: tests not running with eventlet yet

Not impacted (don't use eventlet):

  • oslo.i18n
  • oslo.utils

Dependencies

Porting status for global-requirement.txt.

It's now possible to specify different dependencies for Python 2 and Python 3 using:

  • requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
  • requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
  • (same for test-requirements.txt)

You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:

...
[testenv:py33]
deps = -r{toxinidir}/requirements-py3.txt
       -r{toxinidir}/test-requirements-py3.txt
...

See also a patch to support markers in requirements (in pip): pip issue: Support markers in setup(install_requires)?; Victor Stinner's pull request: "parse requirements in markers"

OpenStack Dependencies:

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
boto Yes N/A Yes See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32).
django-compressor Yes N/A Yes Requirements upgraded: https://review.openstack.org/94357
django-openstack-auth Yes N/A Yes

As of 1.1.6

dnspython Yes N/A Yes Must use the Python 3 version, see https://github.com/rthalley/dnspython/issues/60
ecdsa Yes N/A Yes Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
eventlet Yes Yes Yes eventlet 0.17.3 now fully support Python 3
jsonrpclib No N/A No The project seems dead :(
mysql-python No No No 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there.
netifaces Yes N/A Yes Patch sent by Victor Stinner (in private): netifaces_python3.patch, Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 .
nose-exclude No No No https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
nosehtmloutput No No No
  • nose-exclude (tests only)
  • openstack.nose-plugin
nosexcover No N/A On PyPI Python 3 support since 1.0.9
openstack.nose-plugin No No No
oslo.vmware No Voting Yes suds
oslo.config Yes Voting On PyPI
pam No No No The fork simplepam works on Python 2 and 3
paramiko Yes N/A On PyPI   Requirements upgraded: https://review.openstack.org/#/c/81132/
paste No No No https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
pycadf Yes Yes Yes
python-ldap No No No The project seems dead.
python-memcached Partial in 1.54 No No Issue #52: Python 3.3 support? , Pull request #26: Python 3.3 Support -- Julien Danjou ported pymemcache to Python 3, another memcached client, he suggests to use this one instead
qpid-python No No No
rtslib-fb No No No
sphinxcontrib-docbookrestapi Yes Yes In the Git repo, not on PyPI
sphinxcontrib-httpdomain Yes N/A No
sphinxcontrib-pecanwsme No No No
sqlalchemy-migrate No No No
  • hacking
  • ibm-db-sa
  • scripttest
suds No No No "Lightweight SOAP client". Last commit 2 years ago: https://fedorahosted.org/suds/browser See also this fork which is promising: https://bitbucket.org/jurko/suds
taskflow Yes Yes Yes
thrift No No No
websockify No No No

Reports at OpenStack Summits

Pycon Montreal 2014: Sprint Port OpenStack to Python 3

Enovance organized a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday) 2014. See the page Python3/SprintPycon2014.