Difference between revisions of "Python3"
(→Common Libraries (Oslo Projects)) |
(→Common Libraries (Oslo Projects)) |
||
Line 264: | Line 264: | ||
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes || | | [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes || | ||
|- | |- | ||
− | | [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: | + | | [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: orange;" | Yes if suds if replaced with suds-jurko || Blocked suds dependency which can easily be replaced with suds-jurko |
|- | |- | ||
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes || | | [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes || |
Revision as of 15:58, 22 April 2015
This page tracks the progress of Python 3 effort porting for OpenStack.
Contents
Python 3
Why should OpenStack move to Python 3 right now?
- Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.
Port Python 2 code to Python 3
OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).
Common patterns
- Replace dict.iteritems() with six.iteritems(dict)
- Replace iterator.next() with next(iterator)
- Replace basestring with six.string_types
- Replace unicode with six.text_type
bytes.decode and unicode.encode
Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.
Don't write data.decode()
or text.encode()
without parameter, because you will use a different encoding on Python 2 and Python 3.
Use an explicit encoding instead. Example: data.decode('utf-8')
or text.encode('utf-8')
. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).
safe_decode
Olso Incubator has a function safe_decode() which can be used to decode a bytes string and pass text strings unchanged.
The default encoding is sys.stdin.encoding or sys.getdefaultencoding()
:
- Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
- Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
Safe usage:
-
safe_decode(data, 'utf-8')
: decode bytes from UTF-8 or returns data unchanged if it's already a text string
Unsafe usage:
-
safe_decode(data)
By default, the decoder is strict. You can specify a different error handler using the optional errors
parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.
safe_encode
Olso Incubator has a function safe_encode() which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
-
safe_encode(text)
encodes text to the output encoding -
safe_encode(bytes)
may decode the string and then reencode to a different encoding if input and output encodings are different
The default input encoding (incomding
parameter) is sys.stdin.encoding or sys.getdefaultencoding()
:
- Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
- Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
The default output encoding (encoding
parameter) is UTF-8.
It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
Safe usage:
-
safe_encode(data, incoming='utf-8')
: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)
Unsafe usage:
-
safe_encode(data)
Example:
-
safe_encode(b'\xe9', incoming='latin-1')
returnsb'\xc3\xa9'
.
By default, the encoder and the decoder are strict. You can specify a different error handler using the optional errors
parameter. Example: safe_encode(b'[\xff]', incoming='ascii', errors='ignore')
returns b'[]'
.
logging module and format exceptions
On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs b'hello'
instead of 'hello'
.
There is no clear rule for format exceptions yet. There are different choices depending on the project:
-
str(exc)
: native string, so use bytes on Python 2 -
six.text_type(exc)
: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2:unicode(Exception("nonascii:\xe9"))
. -
six.u(str(exc))
: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex:unicode(str(Exception("\xff")))
-
LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})
Since logging functions expect text strings on Python 3, logged exceptions should be formatted using str(exc)
. Example: LOG.debug(str(exc))
.
HTTP
The HTTP protocol is based on bytes:
- HTTP body contains bytes. For example, use io.BytesIO for a stream storing an HTTP body.
- HTTPConnection.getresponse().read() returns bytes (in Python 3, str which is bytes in Python 2)
- On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
- It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and HTTP header should use what character encoding?
References to port Python 2 code to Python 3
- Porting to Python 3 Book by Lennart Regebro, especially the Language differences and workarounds.
- HOWTO: Porting Python 2 Code to Python 3 by Brett Cannon
- Porting Python Code to 3.x
- python-incompatibility: Demonstrates incompatibilities between Python versions.
Common pitfalls
What is a string ?
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:
Python 2:
>>> type('foo') <type 'str'> >>> type(u'foo') <type 'unicode'> >>> type(b'foo') <type 'str'> >>> isinstance('foo', six.text_type) False >>> isinstance(u'foo', six.text_type) True >>> bytes is str True >>> b'foo'[0] 'f'
Python 3:
>>> type('foo') <class 'str'> >>> type(u'foo') <class 'str'> >>> type(b'foo') <class 'bytes'> >>> isinstance('foo', six.text_type) True >>> isinstance(b'foo', six.text_type) False >>> bytes is str False >>> b'foo'[0] 102
tox/testr error: db type could not be determined
The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.
Workaround: "rm -rf .testrepository/".
Python 3 Status of OpenStack projects
Oslo Incubator
BLOCKER BUG: Tests using testscenarios fail on Python 3 with nosetests because of this bug: https://bugs.launchpad.net/testscenarios/+bug/872887
Recently merged reviews:
Test (full path) | Patches | Comment |
---|---|---|
tests/unit/config/test_generator.py | ||
tests/unit/crypto/test_utils.py | ||
tests/unit/reports/test_guru_meditation_report.py | ||
tests/unit/reports/test_base_report.py | ||
tests/unit/reports/test_openstack_generators.py | ||
tests/unit/reports/test_views.py | ||
tests/unit/scheduler/test_base_filter.py | ||
tests/unit/scheduler/test_weights.py | ||
tests/unit/test_cliutils | ||
tests/unit/test_deprecated.py | ||
tests/unit/test_eventlet_backdoor.py | ||
tests/unit/test_fileutils | ||
tests/unit/test_imageutils.py | ||
tests/unit/test_local.py | ||
tests/unit/test_loopingcall.py | ||
tests/unit/test_memorycache.py | ||
tests/unit/test_periodic.py | ||
tests/unit/test_quota.py | ||
tests/unit/test_service.py | ||
tests/unit/test_systemd.py | ||
tests/unit/test_threadgroup.py | ||
tests/unit/test_versionutils.py |
Common Libraries (Oslo Projects)
For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160
Project | Python 3 compatibility | Comment |
---|---|---|
cliff | Yes | |
oslo.concurrency | Yes |
oslo.concurrency 1.9 supports Python 3, next version will support fully Python 3 in processutils (execute() and ssh_execute() will use Unicode). |
oslo.config | Yes | |
oslo.context | Yes | |
oslo.db | Yes | On Python 3, PyMySQL driver is used instead of MySQL-Python to test MySQL. The Python 3 classifier is missing: patch to add Python 3 classifier. |
oslo.i18n | Yes | |
oslo.log | Yes | |
oslo.messaging | Partial | Rabbit transport supports Python 3. The qpid and AMQP 1.0 transports are Python 2 only. Patches:
|
oslo.middleware | Yes | |
oslo.rootwrap | Yes | |
oslo.serialization | Yes | |
oslosphinx | ? | The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible? |
oslotest | Yes | |
oslo.versionedobjects | Yes | |
oslo.vmware | Yes if suds if replaced with suds-jurko | Blocked suds dependency which can easily be replaced with suds-jurko |
oslo.utils | Yes | |
oslo.version | Yes | not released on PyPI yet |
pylockfile | Yes | |
stevedore | Yes | |
taskflow | Yes |
Development tools
Project | Python 3 compatibility | Comment |
---|---|---|
cookiecutter | yes | |
hacking | yes | py33 gate is not voting |
pbr | yes | |
stackforge/python-jenkins | yes | py33 gate is voting |
openstack-infra/jenkins-job-builder | partial | https://review.openstack.org/172238 |
OpenStack clients
Project | Python 3 compatibility | CI tests running? | Python 3 classifiers ? | Blocked by | Comment |
---|---|---|---|---|---|
python-barbicanclient | Yes | Voting | On PyPI | ||
python-ceilometerclient | Yes | Voting | On PyPI | ||
python-cinderclient | Yes | Voting | On PyPI | ||
python-ganttclient | ? | ? | ? | ? | |
python-glanceclient | Yes | Voting | On PyPI | ||
python-heatclient | Yes | Voting | On PyPI | ||
python-ironicclient | Yes | Voting | On PyPI | ||
python-keystoneclient | Yes | Voting | On PyPI | ||
python-marconiclient | Yes | Voting | Yes | ||
python-melangeclient | ? | ? | ? | ||
python-novaclient | Yes | Voting | On PyPII | ||
python-neutronclient | Yes | Voting | Yes | ||
python-openstackclient | OK | Voting | Yes | As of 0.9 | |
python-savannaaclient | In progress | Non-voting | https://review.openstack.org/#/c/73128/ | ||
python-saharaclient | OK | Voting | In the git repo | ||
python-swiftclient | Yes | Voting | On PyPI | ||
python-tuskarclient | Yes | Voting | On PyPI | ||
python-troveclient | Yes | Voting | On PyPI |
eventlet and Python 3
eventlet 0.15 is the first release supporting Python 3, but monkey-patching did not work with Python 3. Monkey-patching partially works on Python 3 with eventlet 0.16. All monkey-patching should be fixed in the future eventlet 0.17.3 release.
Status of eventlet+Python 3 in OpenStack components.
Working:
- oslo.db: tests running on Python 3.4 with eventlet (0.17.2)
- oslo.concurrency: tests running on Python 3.4 with eventlet (0.17.1)
Don't work:
- oslo.messaging: eventlet currently disabled in tests, tests fail even if the development version of eventlet
Unknown status:
- oslo-incubator: tests not running with eventlet yet
Not impacted (don't use eventlet):
- oslo.i18n
- oslo.utils
Dependencies
Porting status for global-requirement.txt.
It's now possible to specify different dependencies for Python 2 and Python 3 using:
- requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
- requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
- (same for test-requirements.txt)
You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:
... [testenv:py33] deps = -r{toxinidir}/requirements-py3.txt -r{toxinidir}/test-requirements-py3.txt ...
pip supports environment markers in requirements since pip 6.0, example of requirement:
futures; python_version < '3.2'
See Handle PEP426 markers for pbr. Environment markers are not used yet in OpenStack.
OpenStack Dependencies:
- mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock). Examples:
- neutronclient: https://review.openstack.org/#/c/95786/
- Oslo Incubator: https://review.openstack.org/#/c/93729/
Project | Python 3 compatibility | CI tests running? | Python 3 classifiers ? | Blocked by | Comment | |
---|---|---|---|---|---|---|
boto | Yes | N/A | Yes | See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32). | ||
django-compressor | Yes | N/A | Yes | Requirements upgraded: https://review.openstack.org/94357 | ||
django-openstack-auth | Yes | N/A | Yes |
As of 1.1.6 | ||
dnspython | Yes | N/A | Yes | Must use the Python 3 version, see https://github.com/rthalley/dnspython/issues/60 | ||
ecdsa | Yes | N/A | Yes | Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master) | ||
eventlet | Yes | Yes | Yes | eventlet 0.17.3 now fully support Python 3 | ||
jsonrpclib | No | N/A | No | The project seems dead :( | ||
libvirt-python | Yes | N/A | No | Python 3 classifier is missing, but libvirt-python works on Python 3 | ||
mysql-python | No, but it may be replaced with PyMySQL | No | No | 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there. See https://wiki.openstack.org/wiki/PyMySQL_evaluation for PyMySQL which is py3 compatible. | ||
netifaces | Yes | N/A | Yes | Patch sent by Victor Stinner (in private): netifaces_python3.patch, Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 . | ||
nose-exclude | No | No | No | https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3 | ||
nosehtmloutput | No | No | No |
|
||
nosexcover | No | N/A | On PyPI | Python 3 support since 1.0.9 | ||
openstack.nose-plugin | No | No | No | |||
oslo.vmware | No | Voting | Yes | suds | ||
oslo.config | Yes | Voting | On PyPI | |||
pam | No | No | No | The fork simplepam works on Python 2 and 3 | ||
paramiko | Yes | N/A | On PyPI | Requirements upgraded: https://review.openstack.org/#/c/81132/ | ||
paste | Yes | N/A | Yes | Compatible with Python 3 since Paste 2.0 | ||
pycadf | Yes | Yes | Yes | |||
python-ldap | No | No | No | The project seems dead. See ldap3 which is Python 3 compatible (license: LGPL, one of the optional deps is BSD-4-Clause). | ||
python-memcached | Partial in 1.54 | No | No | Issue #52: Python 3.3 support? , Pull request #26: Python 3.3 Support -- Julien Danjou ported pymemcache to Python 3, another memcached client, he suggests to use this one instead | ||
qpid-python | No | No | No | Required if using oslo.messaging's qpid backend. That is the old pure python amqp 0-10 client library. That only speaks 0-10 and works with qpidd broker. It is used by the impl_qpid.py driver in oslo.messaging. That client is legacy - it isn't being ported to use AMQP 1.0 | ||
pyngus | No | No | No |
pyngus is required if using oslo.messaging's AMQP 1.0 driver. pyngus is a wrapper for proton and proton has not been ported to python 3 yet: see Proton issue #490. Proton is a C library with various bindings in other languages. The language bindings are auto generated via the SWIG tool, plus a the python wrapper bits. All the python stuff is found under proton-c/bindings/python in the git tree. The first bit of work is to refactor the SWIG file cproton.i to work for both Python 2 and 3. | ||
rtslib-fb | No | No | No | |||
Routes | Yes using 2to3 | Yes | Yes | Routes doesn't work on Python 3 when building wheel packages using Python 2, see: Port Routes to Python 3 | ||
sphinxcontrib-docbookrestapi | Yes | Yes | In the Git repo, not on PyPI | |||
sphinxcontrib-httpdomain | Yes | N/A | No | |||
sphinxcontrib-pecanwsme | No | No | No | |||
sqlalchemy-migrate | Yes | N/A | Patch sent |
MySQL is not supported yet on Python 3 (MySQL-python is still used). Patches: |
||
suds | Yes for the fork suds-jurko | No | No | "Lightweight SOAP client". Last commit 2 years ago: https://fedorahosted.org/suds/browser See also this fork which is promising: jurko/suds. Note in global requirements: "suds is not python 3.x compatible, suds-jurko is a fork that works with py3x. oslo.vmware would convert to suds-jurko first then nova and cinder would follow. suds should be remove immediately once those projects move to suds-jurko for all jobs." | ||
taskflow | Yes | Yes | Yes | |||
thrift | No | No | No | |||
websockify | No | No | No |
OpenStack applications
Completely updated on Monday, September the 29th, 2014. Partially updated later.
Project | Python 3 compatibility | CI tests running? | Trove classifiers | Blocked by | Comment |
---|---|---|---|---|---|
ceilometer | No | No | No |
Requirements
Requirements for tests
|
|
cinder | No | No | No |
Requirements
Requirements for tests
|
|
glance | No | No | No |
Requirements
Requirements for tests
|
|
heat | No | No | No |
Requirements
Requirements for tests
|
Sirushti Murugesan proposed Add Python 3.4 support spec for Liberty. |
horizon | No | No | No |
Requirements
Requirements for tests
|
|
keystone | No | No | No |
Requirements for tests
|
|
neutron | No | No | No |
Requirements
|
Cyril Roelandt wrote Porting to Python 3 spec for Liberty with the support of Ihar Hrachyshka. |
nova | No | No | No |
Requirements for tests
|
Victor Stinner is working on a spec for Liberty. |
swift | No | No | No |
Requirements: Requirements for tests:
|
Number of core OpenStack projetcs blocked by each dependency:
7 oslo.messaging 5 MySQL-python 2 suds 2 qpid-python 2 openstack.nose_plugin 2 nosehtmloutput 1 xstatic-spin 1 xstatic-rickshaw 1 xstatic-qunit 1 xstatic-jsencrypt 1 xstatic-jquery.tablesorter 1 xstatic-jquery.quicksearch 1 xstatic-jquery-migrate 1 xstatic-jquery 1 xstatic-jasmine 1 xstatic-hogan 1 xstatic-font-awesome 1 xstatic-d3 1 xstatic-bootstrap-datepicker 1 xstatic-angular-mock 1 xstatic-angular-cookies 1 xstatic-angular 1 xstatic 1 websockify 1 thrift 1 sphinxcontrib-pecanwsme 1 sphinxcontrib-httpdomain 1 sphinxcontrib-docbookrestapi 1 rtslib-fb 1 python-ldap 1 nose-exclude 1 nodeenv 1 ldappool 1 jsonrpclib 1 glance_store 1 django-pyscss 1 croniter
Reports at OpenStack Summits
- Kilo Summit
- Doug Hellmann: "Moving our applications to Python 3" cross-project session
- Victor Stinner: "Port nova to Python 3", https://etherpad.openstack.org/p/liberty-nova-summit-ideas
- Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)
- Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
- Havana summit notes: https://etherpad.openstack.org/p/havana-python3
Pycon Montreal 2014: Sprint Port OpenStack to Python 3
Enovance organized a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday) 2014. See the page Python3/SprintPycon2014.