Difference between revisions of "Python3"
(→OpenStack applications (tc-approved)) |
(→Dependencies status) |
||
Line 525: | Line 525: | ||
|- | |- | ||
− | | [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || | + | | [https://pypi.python.org/pypi/pyldap pyldap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || || Compatible with Python 3 (drop in replacement for python-ldap). |
+ | |- | ||
+ | |||
+ | | [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || python-ldap has been dropped in favour of [htps://pypi.python.org/pypi/pyldap pyldap]. No longer part of global-requirements as of Newton. | ||
|- | |- | ||
Revision as of 20:49, 22 May 2016
This page tracks the progress of Python 3 effort porting for OpenStack.
Contents
- 1 IRC: #openstack-python3
- 2 Python 3
- 3 Python 2: Python 2.6 support dropped, Python 2.7 only
- 4 Port Python 2 code to Python 3
- 4.1 Before you begin
- 4.2 sixer tool
- 4.3 Common patterns
- 4.4 Serialization: base64, JSON, etc.
- 4.5 contextlib.nested
- 4.6 oslo_utils.encodeutils.to_utf8
- 4.7 bytes.decode and unicode.encode
- 4.8 safe_decode
- 4.9 safe_encode
- 4.10 logging module and format exceptions
- 4.11 HTTP
- 4.12 References to port Python 2 code to Python 3
- 4.13 Common pitfalls
- 5 Python 3 Status of OpenStack projects
- 6 Status of Python 3 in Linux distributions
- 7 Reports at OpenStack Summits
- 8 Articles
- 9 Pycon Montreal 2014: Sprint Port OpenStack to Python 3
IRC: #openstack-python3
Join the #openstack-python3 IRC channel on the Freenode network to discuss Python 3.
Python 3
Why should OpenStack move to Python 3 right now?
- Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.
Python 2: Python 2.6 support dropped, Python 2.7 only
OpenStack Liberty targets Python 2.7 and 3.4.
Python 2.6 support is being dropped in OpenStack since OpenStack Juno for servers. Python 2.6 support is currently kept in Oslo libraries and clients. See juno-cross-project-future-of-python etherpad.
Python 3.3 support is being dropped since OpenStack Liberty.
Python 2.6 is slowly being dropped in the whole OpenStack project, ex: Remove p26 job from DIB.
Python 2.6 support will be removed in Oslo and Clients clients for OpenStack Mitaka: Oslo libraries dropping python 2.6 compatability.
Port Python 2 code to Python 3
OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. As we are targeting Python 3.4 and up, there is no need to avoid u'unicode' syntax. Do not use six.u('unicode').
Before you begin
If you're doing development with Ubuntu/Debian (and not using devstack with the USE_PYTHON flag set), then you'll need the following packages installed to run the py34 tox unit test targets in the projects:
sudo apt-get install python3.4 python3.4-dev
sixer tool
The sixer tool helps to replace most basic patterns to add Python 3 compatibility and it respects OpenStack coding style.
Common patterns
- Replace "for key in dict.iterkeys()" with "for key in dict"
- Replace dict.iteritems() with dict.items()
- Replace dict.itervalues() with dict.values()
Note: Replacing dict.iteritems()/.itervalues() with six.iteritems(dict)/six.itervalues(dict) was preferred in the past, but there was a discussion suggesting to avoid six for this. The overhead of creating a temporary list on Python 2 is negligible.
- Replace iterator.next() with next(iterator)
- Replace basestring with six.string_types
- Replace unicode with six.text_type
- Replace (str, unicode) with six.string_types
- Replace (int, long) with six.integer_types
- Replace func.func_name with func.__name__
- Replace exceptions.OSError with OSError and remove "import exceptions"
- map() and filter() if a list is needed on Python 3:
- Replace map(func, data) with [func(item) for item in data]
- Replace filter(lambda obj: test(obj), data) with [obj for obj in data if test(obj)]
- Note: Usually, tests fail because map() or filter() objects have no length
Serialization: base64, JSON, etc.
- [with oslo.serialization 1.10 or newer] Get the base64 module from oslo_serialization (from oslo_serialization import base64) to get functions:
- oslo_serialization.base64.decode_as_bytes(encoded)
- oslo_serialization.base64.decode_as_text(encoded, encoding='utf-8')
- oslo_serialization.base64.encode_as_bytes(s, encoding='utf-8')
- oslo_serialization.base64.encode_as_text(s, encoding='utf-8')
- Replace text.encode('base64') and base64.b64encode(text) with:
- base64.b64encode(text): only accept bytes and returns bytes,
- or: oslo_serialization.base64.encode_as_bytes(text): accept bytes or Unicode and returns bytes
- or: oslo_serialization.base64.encode_as_text(text): accept bytes and Unicode and returns Unicode
- Warning: base64.encodestring(raw) adds a newline ("\n"), whereas encode_as_bytes() and encode_as_text() don't.
- Replace raw.decode('base64') with:
- base64.b64decode(raw): return bytes
- oslo_serialization.base64.decode_as_bytes(encoded): accept bytes and Unicode, returns bytes
- oslo_serialization.base64.decode_as_text(encoded): accept bytes and Unicode, returns Unicode
Hexadecimal:
- Replace raw.decode('hex') with binascii.unhexlify(raw)
- Replace bytes.encode('hex') with binascii.hexlify(bytes)
JSON:
- [with oslo.serialization 1.10 or newer] Replace json.dumps(obj) with oslo_serialization.jsoutils.dump_as_bytes(obj)
- Replace json.loads(obj) with oslo_serialization.jsoutils.loads(obj): it accepts bytes and Unicode, bytes is decoded from UTF-8. It avoids "if isinstance(obj, bytes): obj = obj.decode('utf-8')" which may require a second temporary variable.
contextlib.nested
To replaced contextlib.nested is to use contextlib.ExitStack. It's available on Python 2 using contextlib2.ExitStack. For contextlib.nested, nova/test.py defines:
if six.PY3: @contextlib.contextmanager def nested(*contexts): with contextlib.ExitStack() as stack: yield [stack.enter_context(c) for c in contexts] else: nested = contextlib.nested
Another option to replaced contextlib.nested is to use the @mock.patch decorator. Example with nested function:
def test_thing(self): @mock.patch(...) @mock.patch(...) @mock.patch(...) def do_test(...): ...
do_test()
More options:
- Mock provides a context that patches multiple things so that no nesting is needed: mock.patch.multiple()
- oslotest provides fixtures for mock, so you don't need a context: oslotest.mockpatch.
oslo_utils.encodeutils.to_utf8
oslo.utils 3.5 has an oslo_utils.encodeutils.to_utf8() function to encode Unicode to UTF-8 and return bytes unchanged.
bytes.decode and unicode.encode
Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.
Don't write data.decode()
or text.encode()
without parameter, because you will use a different encoding on Python 2 and Python 3.
Use an explicit encoding instead. Example: data.decode('utf-8')
or text.encode('utf-8')
. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).
safe_decode
Olso Incubator has a function safe_decode() which can be used to decode a bytes string and pass text strings unchanged.
The default encoding is sys.stdin.encoding or sys.getdefaultencoding()
:
- Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
- Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
Safe usage:
-
safe_decode(data, 'utf-8')
: decode bytes from UTF-8 or returns data unchanged if it's already a text string
Unsafe usage:
-
safe_decode(data)
By default, the decoder is strict. You can specify a different error handler using the optional errors
parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.
safe_encode
Olso Incubator has a function safe_encode() which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
-
safe_encode(text)
encodes text to the output encoding -
safe_encode(bytes)
may decode the string and then reencode to a different encoding if input and output encodings are different
The default input encoding (incomding
parameter) is sys.stdin.encoding or sys.getdefaultencoding()
:
- Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
- Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
The default output encoding (encoding
parameter) is UTF-8.
It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
Safe usage:
-
safe_encode(data, incoming='utf-8')
: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)
Unsafe usage:
-
safe_encode(data)
Example:
-
safe_encode(b'\xe9', incoming='latin-1')
returnsb'\xc3\xa9'
.
By default, the encoder and the decoder are strict. You can specify a different error handler using the optional errors
parameter. Example: safe_encode(b'[\xff]', incoming='ascii', errors='ignore')
returns b'[]'
.
logging module and format exceptions
The exception_to_unicode(exc)
function of oslo_utils.encodeutils is the recommanded way to format an exception to Unicode. This function works on Python 2 and Python 3 and it should avoid mojibake is most cases.
On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs b'hello'
instead of 'hello'
.
There is no clear rule for format exceptions yet. There are different choices depending on the project:
-
str(exc)
: native string, so use bytes on Python 2 -
six.text_type(exc)
: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2:unicode(Exception("nonascii:\xe9"))
. -
six.u(str(exc))
: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex:unicode(str(Exception("\xff")))
-
LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})
Since logging functions expect text strings on Python 3, logged exceptions should be formatted using str(exc)
. Example: LOG.debug(str(exc))
.
HTTP
The HTTP protocol is based on bytes:
- HTTP body contains bytes. For example, use io.BytesIO for a stream storing an HTTP body.
- HTTPConnection.getresponse().read() returns bytes (in Python 3, str which is bytes in Python 2)
- On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
- It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and HTTP header should use what character encoding?
References to port Python 2 code to Python 3
- Porting to Python 3 Book by Lennart Regebro, especially the Language differences and workarounds.
- HOWTO: Porting Python 2 Code to Python 3 by Brett Cannon
- Porting Python Code to 3.x
- python-incompatibility: Demonstrates incompatibilities between Python versions.
Common pitfalls
What is a string ?
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:
Python 2:
>>> type('foo') <type 'str'> >>> type(u'foo') <type 'unicode'> >>> type(b'foo') <type 'str'> >>> isinstance('foo', six.text_type) False >>> isinstance(u'foo', six.text_type) True >>> bytes is str True >>> b'foo'[0] 'f'
Python 3:
>>> type('foo') <class 'str'> >>> type(u'foo') <class 'str'> >>> type(b'foo') <class 'bytes'> >>> isinstance('foo', six.text_type) True >>> isinstance(b'foo', six.text_type) False >>> bytes is str False >>> b'foo'[0] 102
tox/testr error: db type could not be determined
The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.
Workaround: "rm -rf .testrepository/" and then run "tox -e py34" before running "tox -e py27". You only have to do this once. The problem only occurs with "tox -e py34" when .testrepository/ was created by Python 2.
Python 3 Status of OpenStack projects
Common Libraries (Oslo Projects)
See programs.yaml for the list of Common Libraries.
Project | Python 3 compatibility | Comment |
---|---|---|
cliff | Yes | |
oslo.concurrency | Yes | |
oslo-incubator | Yes | py34 gate is voting |
oslo.config | Yes | |
oslo.context | Yes | |
oslo.db | Yes |
PyMySQL driver is now used by default for MySQL. setup.cfg contains the Python 3 classifier. |
oslo.i18n | Yes | |
oslo.log | Yes | |
oslo.messaging | Yes |
oslo.messaging is now fully compatible with Python 3. The old (and now removed) Qpid transport was not compatible, the AMQP driver was ported recently. The py34 check job is not more voting because of ubuntu still uses Python 3.4.0 which has a severe bug. |
oslo.middleware | Yes | |
oslo.rootwrap | Yes |
oslo.rootwrap 2.4 works on Python 3, but eventlet tests hangs on Python 3. It's now fixed in the development version. oslo.rootwrap has a non-voting python34 check job. |
oslo.serialization | Yes | |
oslosphinx | ? | The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible? |
oslotest | Yes | |
oslo.versionedobjects | Yes | |
oslo.vmware | Yes | Supports Python 3 since oslo.vmware 0.13.0 |
oslo.utils | Yes | |
pylockfile | Yes | |
stevedore | Yes | |
taskflow | Yes |
Development tools
Project | Python 3 compatibility | Comment |
---|---|---|
cookiecutter | yes | |
hacking | yes | py33 gate is not voting |
pbr | yes | |
stackforge/python-jenkins | yes | py33 gate is voting |
openstack-infra/jenkins-job-builder | partial | https://review.openstack.org/172238 |
OpenStack clients
Project | Python 3 compatibility | CI tests running? | Python 3 classifiers ? | Blocked by | Comment |
---|---|---|---|---|---|
keystonemiddleware | Yes | Voting | On PyPI | python-memcached | keystonemiddleware 1.6.1 supports Python 3. |
python-barbicanclient | Yes | Voting | On PyPI | ||
python-ceilometerclient | Yes | Voting | On PyPI | ||
python-cinderclient | Yes | Voting | On PyPI | ||
python-fuelclient | Yes | voting | Yes | ||
python-glanceclient | Yes | Voting | On PyPI | ||
python-heatclient | Yes | Voting | On PyPI | ||
python-ironicclient | Yes | Voting | On PyPI | ||
python-keystoneclient | Yes | Voting | On PyPI | ||
python-manilaclient | Yes | Voting | On PyPI | ||
python-marconiclient | Yes | Voting | On PyPI | ||
python-novaclient | Yes | Voting | On PyPII | ||
python-neutronclient | Yes | Voting | On PyPI | ||
python-openstackclient | Yes | Voting | On PyPI | As of 0.9 | |
python-saharaclient | Yes | Voting | On PyPI | ||
python-swiftclient | Yes | Voting | On PyPI | ||
python-tuskarclient | Yes | Voting | On PyPI | ||
python-troveclient | Yes | Voting | On PyPI | ||
python-watcherclient | Yes | Voting | On PyPI | ||
python-designateclient | work in progress | Voting | No |
OpenStack Libraries
Project | Python 3 compatibility | CI tests running? | Python 3 classifiers ? | Comment |
---|---|---|---|---|
django_openstack_auth | Yes | voting | Yes |
django_openstack_auth 2.0 is fully compatible with Python 3. |
glance_store | Yes | voting | Yes | glance_store 0.7.0 added the Python 3 support. |
neutron-lib | Yes | voting | Yes | |
os-brick | Yes | voting | Yes | os-brick 0.3.2 added Python 3 support. |
sqlalchemy-migrate | Yes | voting | Yes | |
taskflow | Yes | voting | Yes |
Dependencies
There are two ways to express dependencies that should differ between Python 2.x and 3.x. We currently support using multiple requirements files, and will shortly be supporting and recommending the use of environment markers. Recent releases of all the tools in our ecosystem are needed for environment markers.
requirements-py3.txt
This approach generates a different requirements list depending on what Python version is running when the egg info is created (which goes into wheels and sdists). As a consequence this can't work with tox, publishing sdists to PyPI, or wheels. It is however useful for anything installing from git. To use it, create two requirements files:
- requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
- requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
- (same for test-requirements.txt)
You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:
... [testenv:py33] deps = -r{toxinidir}/requirements-py3.txt -r{toxinidir}/test-requirements-py3.txt ...
Environment markers
Environment markers provide a mini-language to express when a dependency is relevant. We hope the next release of pbr will support them. Recent pip, setuptools and wheel versions do support them, but until pbr exports the requirements metadata with them intact and structured correctly for setuptools, we can't use them anywhere.
- PEP 0345 -- Metadata for Python Software Packages 1.2 (accepted): Environment markers
- PEP 0426 -- Metadata for Python Software Packages 2.0 (draft): Environment markers
When placed in a requirements.txt file, the markers are interpreted directly by pip in `pip install -r requirements.txt`, and indirectly via setuptools when installing an sdist or wheel (including `pip install .`. For instance:
six futures; python_version < '3.2'
See also:
- oslo.db: Refactor deps to use extras and env markers (require pip 7, not released yet)
- pip: Fix issue #1433: parse requirements in markers #1472
- setuptools: pkg_resources doesn't understand environment markers: pkg_resources.parse_requirements() raises ValueError
- pbr: Teach pbr to read extras and env markers, Handle PEP426 markers (abandoned)
- wheel: wheel should support PEP 426 environment markers (closed), wheel supports environment markers in extra requirements
- tox: tox must create the source distribution with the Python of the virtual environment, Issue #236: subcommand_test() rebuilds sdist with venv python (declined)
Oslo Messaging issue:
- futures is not needed on Python 3, since concurrent.futures is now part of the Pyhon stdlib since Python 3.2
- futures was added to requirements-py3.txt because tox creates the source tarball using Python 2 which reads requirements.txt
- The correct fix would be to use environment markers on futures
Dependencies status
Porting status for global-requirement.txt.
Project | Python 3 compatibility | CI tests running? | Python 3 classifiers ? | Blocked by | Comment |
---|---|---|---|---|---|
boto | Yes | N/A | Yes | See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32). | |
django-compressor | Yes | N/A | Yes | Requirements upgraded: https://review.openstack.org/94357 | |
dnspython | Yes | N/A | Yes | Must use the Python 3 version, see https://github.com/rthalley/dnspython/issues/60 | |
ecdsa | Yes | N/A | Yes | Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master) | |
eventlet | Yes | Yes | Yes | eventlet 0.17.3 now fully support Python 3 | |
jsonrpclib | No | N/A | No | The project seems dead :( | |
libvirt-python | Yes | N/A | Yes | ||
MySQL-Python | No, but mysqlclient or PyMySQL are compatible | N/A | No |
Patches: MySQL-Python: 2 pending pull requests for Python 3. INADA Naoki, the maintainer of PyMySQL, wrote mysqlclient: fork of MySQL-Python using libmysqlclient.so which is Python 3 compatible. He recommends to use mysqlclient instead of MySQL-python even on Python 2. mysqlclient fixes some bugs, build issues and it support Python 3. For example: support MariaDB's libmysqlclient.so, support microsecond in TIME column. PyMySQL is compatible with Python 3: see PyMySQL evaluation discussion to eventually replace mysql-python with PyMySQL in OpenStack. | |
netifaces | Yes | N/A | Yes | Patch sent by Victor Stinner (in private): netifaces_python3.patch, Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 . | |
nose-exclude | Yes | N/A | No | Python 3 is supported since nose-exclude 0.4. | |
nosehtmloutput | No | No | No |
|
|
nosexcover | No | N/A | On PyPI | Python 3 support since 1.0.9 | |
openstack.nose-plugin | No | No | No | ||
pam | No for pam, Yes for simplepam | NA | No | The fork simplepam works on Python 2 and 3 | |
paramiko | Yes | N/A | On PyPI | Requirements upgraded: https://review.openstack.org/#/c/81132/ | |
paste | Yes | N/A | Yes | Compatible with Python 3 since Paste 2.0 | |
pycadf | Yes | Yes | Yes | ||
pyldap | Yes | Yes | Yes | Compatible with Python 3 (drop in replacement for python-ldap). | |
python-ldap | No | No | No | python-ldap has been dropped in favour of [htps://pypi.python.org/pypi/pyldap pyldap]. No longer part of global-requirements as of Newton. | |
python-memcached | Yes | N/A | Yes |
python-memcached 1.56 is compatible with Python 3, but there are still some bugs: More Python 3 fixes (not part of python-memcached 1.57). These bugs impact running keystonemiddleware tests when the memcached server is not running. It can also impact a setup with multiple memcached servers. | |
qpid-python | No | No | No | Required if using oslo.messaging's qpid backend. That is the old pure python amqp 0-10 client library. That only speaks 0-10 and works with qpidd broker. It is used by the impl_qpid.py driver in oslo.messaging. That client is legacy - it isn't being ported to use AMQP 1.0. | |
pyngus | Yes | N/A | No |
pyngus 2.0 supports Python 3. pyngus is required if using oslo.messaging's AMQP 1.0 driver. pyngus is a wrapper for proton and proton has not been ported to python 3 yet: see Proton issue #490. Proton is a C library with various bindings in other languages. The language bindings are auto generated via the SWIG tool, plus a the python wrapper bits. All the python stuff is found under proton-c/bindings/python in the git tree. The first bit of work is to refactor the SWIG file cproton.i to work for both Python 2 and 3. | |
rtslib-fb | No | No | No | ||
Routes | Yes | Yes | Yes | Routes 2.2 works Python 2 and 3 using a single code base. | |
sphinxcontrib-docbookrestapi | Yes | Yes | In the Git repo, not on PyPI | ||
sphinxcontrib-httpdomain | Yes | N/A | No | ||
sphinxcontrib-pecanwsme | No | No | No | ||
thrift | No | No | No | ||
websockify | Yes | N/A | Yes | python3 works with the 0.7.0 version released on Jul 23, 2015 |
OpenStack applications (tc-approved)
OpenStack applications approved by the OpenStack Technical Committee.
Project | Unit tests pass on Python 3 | py34 gate | Comments |
---|---|---|---|
ceilometer | Yes | voting | |
cinder | Yes | voting |
cinder-python3 blueprint (written by Victor Stinner) has been accepted for Liberty. Patches: bp/cinder-python3 topic. Note: Commit message must use "Partial-Implements: blueprint cinder-python3" to be linked to the blueprint. Kendall Nelson, Ivan Kolodyazhny (core) and Eric Harney (core) prefer this syntax: see COMMIT_MSG. |
glance | Yes | voting |
Victor Stinner is working on porting Glance to Python3. Patches: Patches for glance (topic: py3). |
ironic | Yes | voting |
Python 3.4 unit tests are now being run for openstack/ironic. The unit tests are a voting job. Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4: Run tests in py34 environment |
heat | Yes | voting |
Sirushti Murugesan wrote and implemented the spec Python34 Support (which was accepted for Liberty). Mitaka version will fully support Python 3. Great job Sirushti Murugesan who did almost all the work! |
horizon | Yes | voting |
George Peristerakis wrote Make Horizon python3.4 compatible in May 2015, the blueprint was approved. Patches: bp/porting-python3 topic. |
neutron | Yes | voting |
Porting to Python 3 spec (by Cyril Roelandt) accepted for Liberty, port in progress. Patches of the blueprint neutron-python3 |
neutron-lbaas | Yes | voting |
Patches: topic:bp/neutron-python3. |
neutron-fwaas | Yes | voting | |
neutron-vpnaas | Yes | voting | |
sahara | Yes | voting | |
keystone | Yes | voting |
The move to pyldap and new ldappool were major hurdles solved in Newton. Great work by the Keystone team to get the last things through the gate in Newton. |
nova | work in progress | voting |
Adding Python 3.4 support to Nova spec (by Victor Stinner) accepted for Liberty. Port in progress. Patches:
|
trove | work in progress | voting |
Update 2016-05-02: 115 tests/398. Victor Stinner proposed the trove-python3 blueprint for Mitaka. From Victor (3/17/2016): "I propose to discuss Python 3 before the summit. For example, prepare a concrete plan to port Trove to Python 3, list technical issues like MySQL-Python, etc." Patches: search for the bp/trove-python3 topic. |
swift | blocked | voting |
BLOCKED: Swift developers are not reviewing Python 3 patches. Plan to add Python 3 support to Swift. Patches: Patches with the topic py3 Python 3 unit tests require liberasurecode >= 1.0.9 and PyEClib >= 1.0.9. To install liberasurecode (in /usr): git clone https://bitbucket.org/tsg-/liberasurecode/ cd liberasurecode /autogen.sh && /configure --prefix=/usr && make && sudo make install liberasurecode >= 1.0.9 is available in Ubuntu Precise, Ubuntu Trusty, Fedora 22, Fedora 23 and CentoOS 7. |
Other OpenStack Applications and Projects
List of all OpenStack projects: projects.yaml.
Project | Unit tests pass on Python 3 | py34 gate | Comments |
---|---|---|---|
aodh | Yes | voting | |
cue | Yes | voting |
Patches: search for topic py3. |
congress | Yes | voting |
Random Stacker wrote a support-python3 blueprint. Patches: search for the topic bp/support-python3. |
designate | Yes | voting |
Pradeep Kumar Singh proposed the blueprint designate-py3 which was accepted for Liberty. Patches: topic:bp/designate-py3. |
gnocchi | Yes | voting | |
magnum | Yes | voting | |
manila | Yes but... | voting |
Valeriy Ponomaryov wrote the py3-compatibility blueprint. SSL tests are skipped because of the bug requests to SSL wrapped sockets hang while reading using py3. |
octavia | Yes | voting | |
rally | Yes | voting | A big thank to Andrey Kurilin for a lot of work in this direction |
searchlight | Yes | voting | |
solum | Yes | voting |
Victor Stinner proposed the blueprint solum-python3 which was approved. Patches: search for bp/solum-python3 topic (open reviews). |
zaqar | Yes | voting | |
watcher | Yes | voting | |
barbican | work in progress | voting | Pradeep Kumar Singh proposed the blueprint barbican-py3 for Liberty.
Patches: search for the bp/barbican-py3 topic. |
mistral | work in progress | voting |
Update 2015-10-05: Renat Akhmerov wrote "Currently, we have 70-80 tests that fail on Python 3 which is the way better than it was a few weeks ago (it used to be ~500). (...) we're actively working on fixing all the tests for Python 3". |
murano | work in progress | voting |
Blueprint: murano-python-3-support. python-muranoclient and murano-dashboard fully support Python 3 and have jobs voting. murano and murano-agent are a work in progress, and have their jobs non-voting. Patches: topic:py3. Many unit tests fail with the same error in YAQL, see YAQL bug: Need a new release to get full Python 3 support. |
Unsorted projects:
- App Catalog
- Chef OpenStack (Ruby)
- Kolla: ~2100 lines of Python, but not "tox -e py27" target
- OpenStack Ansible (Python): Ansible playbooks and roles for deployment
- Puppet OpenStack (Ruby)
- TripleO
Enable Python 3 in DevStack
- Fedora::
sudo dnf install python3-devel sudo python3 -m pip install python-memcached # needed by keystonemiddleware
- localrc: add the following lines
USE_PYTHON3=True PYTHON3_VERSION=3.4
Functional and Integration Tests
Doug Hellmann wrote a cross-project specification to run integration tests on Python 3: Enabling Python 3 for Application Integration Tests (spec approved for Liberty).
Patch for DevStack: Enable optional Python 3 support.
Projects wanting to run their existing functional test suite against their services on Python 3 should:-
- Add the Python3.4 trove classifier to the setup.cfg configuration file of the respective service.
- set USE_PYTHON3=True in Devstack's localrc config file before running stack.sh after the DevStack change is merged or cherry-pick until then.
- set basepython = python3.4 in tox for the necessary functional/integration tests environment.
Then run tox -e<your functional test suite name>. Example:- tox -efunctional to run the functional the tests of a certain project.
Once the functional test suite is reliably passing against the service in a python 3.4 env, the trove classifiers must be updated along with a new gate job being added to the project.
Status of Python 3 in Linux distributions
- ArchLinux already switched to Python 3 by default in 2011
- Ubuntu wants to remove Python 2 from the default installation for Ubuntu 16.04 LTS: see (Python) Plans for 16.04.
- Fedora schedules the switch in Fedora 23 (october 2015): Python 3 as the Default Implementation
- RHEL: Python 3.3 and Python 3.4 are available on RHEL6 and RHEL7 using SCL.
- CentOS: Python 3.3 and Python 3.4 are available on CentOS 6 and 7 using SCL (no Red Hat subscription needed, CentOS has its own flavor of SCL)
- In Debian, the plan is to deprecate Python 2 in Stretch (aka: Debian 9, the next Stable Debian after Jessie) and to completely remove Python 2 for the Buster release of Debian (aka: Debian 10, to be release in approximatively 2019).
Reports at OpenStack Summits
- Mitaka Summit
- Cross-project session: Python 3 was proposed, https://etherpad.openstack.org/p/mitaka-cross-project-session-planning
- Liberty Summit
- Cross-project session on enabling app integration testing: https://etherpad.openstack.org/p/liberty-cross-project-python3
- Kilo Summit
- Doug Hellmann: "Moving our applications to Python 3" cross-project session
- Victor Stinner: "Port nova to Python 3", https://etherpad.openstack.org/p/liberty-nova-summit-ideas
- Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)
- Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
- Havana summit notes: https://etherpad.openstack.org/p/havana-python3
Articles
- Status of Python 3 in OpenStack Mitaka (Victor Stinner, March 2016)
- Python 3 Status in OpenStack Liberty (Cyril Roelandt and Victor Stinner, September 2015)
- Status of the OpenStack port to Python 3 (Cyril Roelandt, February 2014)
- Why should OpenStack move to Python 3 right now? (Victor Stinner, December 2013)
Pycon Montreal 2014: Sprint Port OpenStack to Python 3
Enovance organized a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday) 2014. See the page Python3/SprintPycon2014.