Jump to: navigation, search

Difference between revisions of "Python3"

(Dependencies)
m (Update link for oslo_serialization.jsonutils.dump_as_bytes(obj))
 
(662 intermediate revisions by 59 users not shown)
Line 1: Line 1:
 
This page tracks the progress of Python 3 effort porting for OpenStack.
 
This page tracks the progress of Python 3 effort porting for OpenStack.
 +
 +
== IRC: #openstack-dev ==
 +
 +
Join the #openstack-dev IRC channel on the Freenode network to discuss Python 3.
  
 
== Python 3 ==
 
== Python 3 ==
Line 6: Line 10:
 
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''
 
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''
  
== Pycon Montreal 2014: Sprint Port OpenStack to Python 3 ==
+
== Python 2: Python 2.6 support dropped, Python 2.7 only ==
 +
 
 +
OpenStack Liberty targets Python 2.7 and 3.4.
 +
 
 +
Python 2.6 support is being dropped in OpenStack since OpenStack Juno for servers. Python 2.6 support is currently kept in Oslo libraries and clients. See [https://etherpad.openstack.org/p/juno-cross-project-future-of-python juno-cross-project-future-of-python] etherpad.
 +
 
 +
Python 3.3 support is being dropped since OpenStack Liberty.
 +
 
 +
Python 2.6 is slowly being dropped in the whole OpenStack project, ex: [https://review.openstack.org/#/c/201295/ Remove p26 job from DIB].
  
Enovance organizes a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday). See the page [[Python3/SprintPycon2014]].
+
Python 2.6 support will be removed in Oslo and Clients clients for OpenStack Mitaka: [http://markmail.org/message/oe6wewgvbvvyts73 Oslo libraries dropping python 2.6 compatability].
  
 
== Port Python 2 code to Python 3 ==
 
== Port Python 2 code to Python 3 ==
  
OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack must still support Python 2.6 for RHEL, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).
+
OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. As we are targeting Python 3.4 and up, there is no need to avoid u'unicode' syntax. Do not use six.u('unicode').
 +
 
 +
=== Before you begin ===
 +
 
 +
If you're doing development with Ubuntu/Debian (and not using devstack with the USE_PYTHON flag set), then you'll need the following packages installed to run the py34 tox unit test targets in the projects:
 +
 
 +
    sudo apt-get install python3.4 python3.4-dev
 +
 
 +
=== sixer tool ===
 +
 
 +
The [https://pypi.python.org/pypi/sixer sixer] tool helps to replace most basic patterns to add Python 3 compatibility and it respects OpenStack coding style.
  
 
=== Common patterns ===
 
=== Common patterns ===
  
* Replace dict.iteritems() with six.iteritems(dict)
+
* Replace "for key in dict.iterkeys()" with "for key in dict"
 +
* Replace dict.iteritems() with dict.items()
 +
* Replace dict.itervalues() with dict.values()
 +
 
 +
Note: Replacing dict.iteritems()/.itervalues() with six.iteritems(dict)/six.itervalues(dict) was preferred in the past, but there was [http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html a discussion suggesting to avoid six for this]. The overhead of creating a temporary list on Python 2 is negligible.
 +
 
 
* Replace iterator.next() with next(iterator)
 
* Replace iterator.next() with next(iterator)
 
* Replace basestring with six.string_types
 
* Replace basestring with six.string_types
 
* Replace unicode with six.text_type
 
* Replace unicode with six.text_type
 +
* Replace (str, unicode) with six.string_types
 +
* Replace (int, long) with six.integer_types
 +
* Replace func.func_name with func.__name__
 +
* Replace exceptions.OSError with OSError and remove "import exceptions"
 +
* map() and filter() if a list is needed on Python 3:
 +
** Replace map(func, data) with [func(item) for item in data]
 +
** Replace filter(lambda obj: test(obj), data) with [obj for obj in data if test(obj)]
 +
** Note: Usually, tests fail because map() or filter() objects have no length
 +
 +
=== Serialization: base64, JSON, etc. ===
 +
 +
* [with oslo.serialization 1.10 or newer] Get the [http://docs.openstack.org/developer/oslo.serialization/api.html#module-oslo_serialization.base64 base64 module] from oslo_serialization (from oslo_serialization import base64) to get functions:
 +
** oslo_serialization.base64.decode_as_bytes(encoded)
 +
** oslo_serialization.base64.decode_as_text(encoded, encoding='utf-8')
 +
** oslo_serialization.base64.encode_as_bytes(s, encoding='utf-8')
 +
**  oslo_serialization.base64.encode_as_text(s, encoding='utf-8')
 +
* Replace text.encode('base64') and base64.b64encode(text) with:
 +
** base64.b64encode(text): only accept bytes and returns bytes,
 +
** or: oslo_serialization.base64.encode_as_bytes(text): accept bytes or Unicode and returns bytes
 +
** or: oslo_serialization.base64.encode_as_text(text): accept bytes and Unicode and returns Unicode
 +
** '''Warning:''' base64.encodestring(raw) adds a newline ("\n"), whereas encode_as_bytes() and encode_as_text() don't.
 +
* Replace raw.decode('base64') with:
 +
** base64.b64decode(raw): return bytes
 +
** oslo_serialization.base64.decode_as_bytes(encoded): accept bytes and Unicode, returns bytes
 +
** oslo_serialization.base64.decode_as_text(encoded): accept bytes and Unicode, returns Unicode
 +
 +
Hexadecimal:
 +
 +
* Replace raw.decode('hex') with binascii.unhexlify(raw)
 +
* Replace bytes.encode('hex') with binascii.hexlify(bytes)
 +
 +
JSON:
 +
 +
* [with oslo.serialization 1.10 or newer] Replace json.dumps(obj) with [https://docs.openstack.org/oslo.serialization/latest/reference/index.html#oslo_serialization.jsonutils.dump_as_bytes oslo_serialization.jsonutils.dump_as_bytes(obj)]
 +
* Replace json.loads(obj) with oslo_serialization.jsonutils.loads(obj): it accepts bytes and Unicode, bytes is decoded from UTF-8. It avoids "if isinstance(obj, bytes): obj = obj.decode('utf-8')" which may require a second temporary variable.
 +
 +
=== contextlib.nested ===
 +
 +
To replaced contextlib.nested is to use [https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack contextlib.ExitStack]. It's available on Python 2 using [https://contextlib2.readthedocs.org/en/latest/#contextlib2.ExitStack contextlib2.ExitStack]. For contextlib.nested, nova/test.py defines:
 +
 +
if six.PY3:
 +
    @contextlib.contextmanager
 +
    def nested(*contexts):
 +
        with contextlib.ExitStack() as stack:
 +
            yield [stack.enter_context(c) for c in contexts]
 +
else:
 +
    nested = contextlib.nested
 +
 +
Another option to replaced contextlib.nested is to use the @mock.patch decorator. Example with nested function:
 +
 +
def test_thing(self):
 +
    @mock.patch(...)
 +
    @mock.patch(...)
 +
    @mock.patch(...)
 +
    def do_test(...):
 +
        ...
 +
 +
    do_test()
 +
 +
More options:
 +
* Mock provides a context that patches multiple things so that no nesting is needed: [https://docs.python.org/dev/library/unittest.mock.html#patch-multiple mock.patch.multiple()]
 +
* oslotest provides fixtures for mock, so you don't need a context: [http://docs.openstack.org/developer/oslotest/api.html#module-oslotest.mockpatch oslotest.mockpatch].
 +
 +
=== oslo_utils.encodeutils.to_utf8 ===
 +
 +
oslo.utils 3.5 has an oslo_utils.encodeutils.to_utf8() function to encode Unicode to UTF-8 and return bytes unchanged.
 +
 +
=== bytes.decode and unicode.encode ===
 +
 +
Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.
 +
 +
Don't write <code>data.decode()</code> or <code>text.encode()</code> without parameter, because you will use a different encoding on Python 2 and Python 3.
 +
 +
Use an explicit encoding instead. Example: <code>data.decode('utf-8')</code> or <code>text.encode('utf-8')</code>. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).
 +
 +
=== safe_decode ===
 +
 +
Olso Incubator has a function '''safe_decode()''' which can be used to decode a bytes string and pass text strings unchanged.
 +
 +
The default encoding is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
 +
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
 +
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
 +
 +
It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
 +
 +
Safe usage:
 +
* <code>safe_decode(data, 'utf-8')</code>: decode bytes from UTF-8 or returns data unchanged if it's already a text string
 +
 +
Unsafe usage:
 +
* <code>safe_decode(data)</code>
 +
 +
By default, the decoder is strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.
 +
 +
=== safe_encode ===
 +
 +
Olso Incubator has a function '''safe_encode()''' which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
 +
* <code>safe_encode(text)</code> encodes text to the output encoding
 +
* <code>safe_encode(bytes)</code> may decode the string and then reencode to a different encoding if input and output encodings are different
 +
 +
The default input encoding (<code>incomding</code> parameter) is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
 +
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
 +
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
 +
 +
The default output encoding (<code>encoding</code> parameter) is UTF-8.
 +
 +
It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
 +
 +
Safe usage:
 +
* <code>safe_encode(data, incoming='utf-8')</code>: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)
 +
 +
Unsafe usage:
 +
* <code>safe_encode(data)</code>
 +
 +
Example:
 +
* <code>safe_encode(b'\xe9', incoming='latin-1')</code> returns <code>b'\xc3\xa9'</code>.
 +
 +
By default, the encoder and the decoder are strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: <code>safe_encode(b'[\xff]', incoming='ascii', errors='ignore')</code> returns <code>b'[]'</code>.
 +
 +
=== logging module and format exceptions ===
 +
 +
The <code>exception_to_unicode(exc)</code> function of oslo_utils.encodeutils is the recommanded way to format an exception to Unicode. This function works on Python 2 and Python 3 and it should avoid mojibake is most cases.
 +
 +
On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs <code>b'hello'</code> instead of <code>'hello'</code>.
 +
 +
There is no clear rule for format exceptions yet. There are different choices depending on the project:
 +
 +
* <code>str(exc)</code>: native string, so use bytes on Python 2
 +
* <code>six.text_type(exc)</code>: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: <code>unicode(Exception("nonascii:\xe9"))</code>.
 +
* <code>six.u(str(exc))</code>: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: <code>unicode(str(Exception("\xff")))</code>
 +
* <code>LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})</code>
 +
 +
Since logging functions expect text strings on Python 3, logged exceptions should be formatted using <code>str(exc)</code>. Example: <code>LOG.debug(str(exc))</code>.
 +
 +
=== HTTP ===
 +
 +
The HTTP protocol is based on '''bytes''':
 +
 +
* HTTP body contains '''bytes'''. For example, use io.BytesIO for a stream storing an HTTP body.
 +
* HTTPConnection.getresponse().read() returns '''bytes''' (in Python 3, '''str''' which is bytes in Python 2)
 +
* On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
 +
* It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and [http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding  HTTP header should use what character encoding?]
  
 
=== References to port Python 2 code to Python 3 ===
 
=== References to port Python 2 code to Python 3 ===
Line 64: Line 232:
 
     102
 
     102
  
== Python 3 of OpenStack Dependencies ==
+
==== tox/testr error: db type could not be determined ====
  
'''Blocker Pointer:''' it's not yet possible to specify different list of dependencies for Python 2 and Python 3. For example, mox only works on Python 2, mox3 can be used on Python 3.
+
The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.
  
* Julien Danjou proposed to add requirements-py3.txt: [https://review.openstack.org/#/c/58770/ openstack/requirements patch] and [https://review.openstack.org/#/c/63236/ openstack-dev/pbr patch]
+
Workaround: "rm -rf .testrepository/" and then run "tox -e py34" before running "tox -e py27". You only have to do this once. The problem only occurs with "tox -e py34" when .testrepository/ was created by Python 2.
* An alternative is to support markers in requirements (in pip): [https://github.com/pypa/pip/issues/1433 pip issue: Support markers in setup(install_requires)?]; [https://github.com/pypa/pip/pull/1472 Victor Stinner's pull request: "parse requirements in markers"]
 
  
OpenStack Dependencies:
+
== Python 3 Status of OpenStack projects ==
  
* mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock)
+
=== Common Libraries (Oslo Projects) ===
* eventlet: not available on Python 3 yet, alternatives: [http://docs.python.org/dev/library/asyncio.html asyncio] ([http://code.google.com/p/tulip/ Tulip] for Python 3.3+/[https://bitbucket.org/haypo/trollius Trollius] for Python 2), [http://www.tornadoweb.org/ Tornado]
 
  
== Portage in progress ==
+
See [http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml programs.yaml] for the list of Common Libraries.
  
* Oslo Messaging: Portage in Progress by Victor Stinner ([https://review.openstack.org/#/dashboard/9107 dashboard])
+
{|class="wikitable sortable"
* glanceclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])
+
|-
* heatclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])
+
! Project !! Python 3 compatibility !! Comment
* neutronclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])
+
|-
* glanceclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])
+
| [https://pypi.python.org/pypi/automaton automaton] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/castellan castellan] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/cliff cliff] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/debtcollector debtcollector] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/futurist futurist] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.cache oslo.cache] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.concurrency oslo.concurrency] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.context oslo.context] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.db oslo.db] || style="background-color: lightgreen;" | Yes ||
 +
PyMySQL driver is now used by default for MySQL. setup.cfg contains the Python 3 classifier.
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.i18n oslo.i18n] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.log oslo.log] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color: lightgreen;" | Yes ||
 +
oslo.messaging is now fully compatible with Python 3. The old (and now removed) Qpid transport was not compatible, the AMQP driver was ported recently.
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.middleware oslo.middleware] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.policy oslo.policy] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.privsep oslo.privsep] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.reports oslo.reports] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.serialization oslo.serialization] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.service oslo.service] || style="background-color: orange;" | Partial ||
 +
WSGI module SSL support is broken under Python 3. We want projects to support deployment without relying on that code, but not all do (see glance and designate below).
 +
|-
 +
| [https://pypi.python.org/pypi/oslotest oslotest] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: lightgreen;" | Yes  || Supports Python 3 since oslo.vmware 0.13.0
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| pylockfile || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/stevedore stevedore] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/tooz tooz] || style="background-color: lightgreen;" | Yes ||
 +
|}
  
== Portage done ==
+
=== Development tools ===
* keystoneclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])
 
  
== Python 3 Status of OpenStack projects ==
+
{|class="wikitable sortable"
 +
|-
 +
! Project !! Python 3 compatibility !! Comment
 +
|-
 +
| [https://pypi.python.org/pypi/cookiecutter cookiecutter] || style="background-color: lightgreen;" | yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/hacking hacking] || style="background-color: lightgreen;" | yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/pbr pbr] || style="background-color: lightgreen;" | yes ||
 +
|-
 +
| stackforge/python-jenkins || style="background-color: lightgreen;" | yes || py33 gate is voting
 +
|-
 +
| openstack-infra/jenkins-job-builder || style="background-color: orange;" | partial || https://review.openstack.org/172238
 +
|}
  
See also [[Python3Deps]].
+
=== OpenStack clients ===
  
=== Oslo Incubator ===
+
{|class="wikitable sortable"
 +
|-
 +
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
 +
|-
  
'''BLOCKER BUG:''' Tests using testscenarios fail on Python 3 with nosetests because of this bug:
+
| [https://pypi.python.org/pypi/keystonemiddleware keystonemiddleware] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || python-memcached || keystonemiddleware 1.6.1 supports Python 3.
https://bugs.launchpad.net/testscenarios/+bug/872887
+
|-
  
Recently merged reviews:
+
| [https://pypi.python.org/pypi/python-barbicanclient python-barbicanclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
 +
|-
  
* https://review.openstack.org/#/c/79781/
 
  
 +
| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
 +
|-
  
{|class="wikitable sortable"
+
| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
 
|-
 
|-
! Test (full path) !! Patches !! Comment
+
 
 +
| [https://pypi.python.org/pypi/python-fuelclient python-fuelclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/config/test_generator.py  || ||
+
| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color:lightgreen" | On PyPI || ||
|-  
+
|-
  
| style="background-color: red" | tests/unit/crypto/test_utils.py || ||
+
| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/db/sqlalchemy/test_migration_common.py || ||
+
| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient]  || style="background-color: lightgreen;" | Yes||  style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI |||  
 
|-
 
|-
  
| style="background-color: red" | tests/unit/db/sqlalchemy/test_migrate.py  || ||
+
| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/db/sqlalchemy/test_models.py || https://review.openstack.org/#/c/80307/ ||
+
| [https://pypi.python.org/pypi/python-manilaclient python-manilaclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_options.py || https://review.openstack.org/#/c/80627/  ||
+
| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/db/sqlalchemy/test_migrate_cli.py  || ||
+
| [https://pypi.python.org/pypi/python-masakariclient python-masakariiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/db/sqlalchemy/test_utils.py  || ||
+
| [https://pypi.python.org/pypi/python-monascaclient python-monascaclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/db/sqlalchemy/test_sqlalchemy.py  || ||
+
| [https://pypi.python.org/pypi/python-novaclient python-novaclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-  
+
|-
  
| style="background-color: red" | tests/unit/fixture/test_logging.py || ||
+
| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI ||  ||
 
|-
 
|-
  
| style="background-color: lightgreen" | tests/unit/middleware/test_request_id.py || https://review.openstack.org/#/c/80336/ ||
+
| [https://pypi.python.org/pypi/python-octaviaclient python-octaviaclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI ||  ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/middleware/test_sizelimit.py || https://review.openstack.org/#/c/80450/  ||
+
| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient]      || style="background-color: lightgreen" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || || As of 0.9
 
|-
 
|-
  
| style="background-color: red" | tests/unit/middleware/test_audit.py  || ||  
+
| [https://pypi.python.org/pypi/python-saharaclient python-saharaclient]    || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/reports/test_guru_meditation_report.py  || ||
+
| [https://pypi.python.org/pypi/python-searchlightclient python-searchlightclient]    || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/reports/test_base_report.py  || ||
+
| [https://pypi.python.org/pypi/python-senlinclient python-senlinclient]    || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/reports/test_openstack_generators.py  || ||
+
| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/reports/test_views.py  || ||
+
| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/rpc/test_common.py || https://review.openstack.org/#/c/80533/  ||
+
| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: lightgreen" | tests/unit/scheduler/test_base_filter.py || https://review.openstack.org/#/c/80321/  ||
+
| [https://pypi.python.org/pypi/python-watcherclient python-watcherclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/scheduler/test_weights.py  || ||
+
| [https://pypi.python.org/pypi/python-designateclient python-designateclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
 +
 
 +
|}
 +
 
 +
=== OpenStack Libraries ===
 +
 
 +
{|class="wikitable sortable"
 +
|-
 +
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Comment
 +
|-
 +
| blazar-nova || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| ceilometermiddleware || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/test_cliutils || https://review.openstack.org/#/c/74433/  ||
+
| [https://pypi.python.org/pypi/django_openstack_auth django_openstack_auth] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
django_openstack_auth 2.0 is fully compatible with Python 3.
 
|-
 
|-
  
| style="background-color: red" | tests/unit/test_fileutils || https://review.openstack.org/#/c/74728/  ||
+
| [https://pypi.python.org/pypi/glance_store glance_store] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes || glance_store 0.7.0 added the Python 3 support.
|-  
+
|-
  
| style="background-color: lightgreen" | tests/unit/test_gettext.py || https://review.openstack.org/#/c/80534/ || (issue with testscenarios)
+
| heat-translator || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/test_imageutils.py || || nosetests + testscenarios failure
+
| ironic-lib || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| style="background-color: lightgreen" | tests/unit/test_jsonutils.py || https://review.openstack.org/#/c/80370/  ||
+
| keystoneauth || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
|-  
+
|-
  
| style="background-color: red" | tests/unit/test_log.py || || depends on https://review.openstack.org/#/c/80534/
+
| keystonemiddleware || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| style="background-color: lightgreen" | tests/unit/test_quota.py || https://review.openstack.org/#/c/80564/  ||
+
| ldappool || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| style="background-color: red" | tests/unit/test_strutils.py ||  
+
| monasca-statsd || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
*  https://review.openstack.org/#/c/80571/ (safe_encode)
 
*  https://review.openstack.org/#/c/80573/
 
||
 
 
|-
 
|-
  
|}
+
| mistral-lib || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
=== OpenStack clients ===
+
| neutron-lib || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
{|class="wikitable sortable"
 
|-
 
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
 
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
+
| octavia-lib || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting ||https://review.openstack.org/#/c/73844/ || ||
+
| [https://pypi.python.org/pypi/os-brick os-brick] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes || os-brick 0.3.2 added Python 3 support.
|-
 
  
| [https://pypi.python.org/pypi/python-ganttclient python-ganttclient] ||  ? || ? || ? || ? ||
 
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient]  || style="background-color: orange;" | In Progress || style="background-color: orange" | Non-voting || style="background-color:red" | No || ||
+
| os-client-config || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: orange" | In the git repo, not on PyPI || ||
+
| os-traits || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| os-vif || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| os-win || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: red;" | No || ||
+
| osc-lib || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-melangeclient python-melangeclient] || ? || ? || ? || ||
+
| osc-placement || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-novaclient python-novaclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: orange;" | In the Git repo, not on PyPI || ||
+
| pycadf || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: orange;" | In progress || style="background-color: orange;" | Non-voting   || || Differences between Python 2 and 3 ||
+
| shade || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient]     || style="background-color: orange" | In Progress || style="background-color: orange" | Non-Voting || style="background-color: red" | No || || Works with glanceclient HEAD
+
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color:lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-savannaclient python-savannaclient]    || style="background-color: orange;" | In progress || style="background-color: orange" | Non-voting || ||| https://review.openstack.org/#/c/73128/
+
| sushy || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient]   || style="background-color: orange;" | In progress || style="background-color: orange;" | Non-voting   || style="background-color:red" |`No || Differences between Python 2 and 3  ||  
+
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| tosca-parser || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
 
 
|}
 
|}
  
=== Core OpenStack projects ===
+
=== Dependencies: Environment markers ===
 +
 
 +
Example to not install the "futures" dependency on Python 3.3 and newer:
 +
 
 +
<pre>
 +
futures; python_version < '3.2'
 +
</pre>
 +
 
 +
=== OpenStack applications (tc:approved-release) ===
 +
 
 +
OpenStack applications [https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-tc-approved-release approved by the OpenStack Technical Committee].
 +
 
 
{| class="wikitable sortable"
 
{| class="wikitable sortable"
 
|-
 
|-
! Project !! Python 3 compatibility !! CI tests running? !! Trove classifiers !! Blocked by !! Comment
+
! style="width: 10%;" | Project !! style="width: 10%;" | Unit tests pass on Python 3 !! style="width: 10%;" | py35 gate !! style="width: 20%;" | functional tests !! Comments
 +
|-
 +
 
 +
| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| [https://pypi.python.org/pypi/cinder cinder] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 ||
*  eventlet
+
[https://blueprints.launchpad.net/cinder/+spec/cinder-python3 cinder-python3] blueprint (written by Victor Stinner) has been accepted for Liberty. Patches: [https://review.openstack.org/#/q/project:openstack/cinder+topic:bp/cinder-python3,n,z bp/cinder-python3 topic]. Note: Commit message must use "Partial-Implements: blueprint cinder-python3" to be linked to the blueprint. Kendall Nelson, Ivan Kolodyazhny (core) and Eric Harney (core) prefer this syntax: see [https://review.openstack.org/#/c/229991/2//COMMIT_MSG COMMIT_MSG].
*  hacking (tests only)
 
*  mysql-python
 
*  oslosphinx (tests only)
 
*  python-ceilometerclient
 
*  python-glanceclient
 
*  python-swiftclient
 
*  sphinxcontrib-docbookrestapi (tests only)
 
*  sphinxcontrib-httpdomain (tests only)
 
*  sphinxcontrib-pecanwsme (tests only)
 
*  sqlalchemy-migrate
 
*  swift (tests only)
 
*  thrift (which is blocking happybase)
 
||
 
 
|-
 
|-
  
 +
| designate || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | designate-bind9-py36 designate-bind9-py35 || ! style="background-color: orange;" | Relies on oslo.service WSGI code that does not work under Python 3 with SSL. Functional tempest tests are ran on both of these jobs. [http://lists.openstack.org/pipermail/openstack-dev/2018-May/130277.html]
 +
|-
  
| [https://pypi.python.org/pypi/cinder cinder] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| [https://pypi.python.org/pypi/glance glance] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 || ! style="background-color: orange;" | Relies on oslo.service WSGI code that does not work under Python 3. [http://lists.openstack.org/pipermail/openstack-dev/2018-May/130277.html]
*  eventlet
+
Victor Stinner is working on porting Glance to Python3. Patches: [https://review.openstack.org/#/q/status:open+project:openstack/glance+branch:master+topic:py3,n,z Patches for glance (topic: py3)].
*  hacking (tests only)
 
*  mysql-python (tests only)
 
oslo.rootwrap
 
*  oslosphinx (tests only)
 
*  paste
 
*  python-glanceclient
 
*  python-swiftclient
 
*  rtslib-fb
 
*  sqlalchemy-migrate
 
*  suds
 
*  taskflow
 
||
 
 
|-
 
|-
  
 +
| [https://pypi.python.org/pypi/heat heat] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | heat-functional-convg-mysql-lbaasv2-py35 ||
 +
Sirushti Murugesan wrote and implemented the spec [http://specs.openstack.org/openstack/heat-specs/specs/liberty/heat-python34-support.html Python34 Support] (which was accepted for Liberty). Mitaka version will fully support Python 3. Great job Sirushti Murugesan who did almost all the work!
 +
|-
  
| [https://pypi.python.org/pypi/glance glance] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| [https://pypi.python.org/pypi/horizon horizon] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | horizon-openstack-tox-python3-django111,horizon-integration-tests, horizon-dsvm-tempest-plugin ||
*  boto
+
George Peristerakis wrote [https://blueprints.launchpad.net/horizon/+spec/porting-python3 Make Horizon python3.4 compatible] in May 2015, the blueprint was approved. Patches: [https://review.openstack.org/#/q/topic:bp/porting-python3+project:openstack/horizon,n,z bp/porting-python3 topic].
*  eventlet
 
*  hacking (tests only)
 
*  mysql-python (tests only)
 
*  nose-exclude (tests only)
 
*  nose-htmloutput (tests only)
 
*  openstack.nose_plugin (test only)
 
*  oslo.messaging
 
*  oslosphinx
 
*  paste
 
*  python-cinderclient
 
*  python-swiftclient
 
*  qpid-python (tests only)
 
*  sqlalchemy-migrate
 
||
 
 
|-
 
|-
  
| [https://pypi.python.org/pypi/heat heat] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| [https://pypi.python.org/pypi/ironic ironic] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ironic-tempest-dsvm-ipa-partition-pxe_ipmitool-tinyipa-python3 ||
*  eventlet
+
[http://lists.openstack.org/pipermail/openstack-dev/2015-May/064049.html Python 3.4 unit tests are now being run for openstack/ironic. The unit tests are a voting job.]
*  hacking (tests only)
+
Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4: [https://review.openstack.org/#/c/156192/ Run tests in py34 environment]
*  oslosphinx (tests only)
 
*  python-ceilometerclient
 
*  python-cinderclient
 
*  python-glanceclient (tests only)
 
*  python-heatclient
 
*  python-neutronclient
 
*  python-swiftclient
 
*  python-troveclient
 
*  qpid-python
 
* sqlalchemy-migrate
 
||
 
 
|-
 
|-
  
| [https://pypi.python.org/pypi/horizon horizon] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| [https://pypi.python.org/pypi/keystone keystone] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 ||
*  django-compressor
+
The move to pyldap and new ldappool were major hurdles solved in Newton. Great work by the Keystone team to get the last things through the gate in Newton.
*  django-openstack-auth
 
*  eventlet
 
*  hacking (tests)
 
*  nose-exclude (tests)
 
*  nosehtmloutput (tests)
 
*  openstack.nose-plugin (tests)
 
*  oslo.sphinx (tests)
 
*  python-ceilometerclient
 
*  python-cinderclient
 
*  python-glanceclient
 
*  python-heatclient
 
*  python-neutronclient
 
*  python-swiftclient
 
*  python-troveclient
 
||
 
 
|-
 
|-
  
 +
| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 ||
 +
[https://review.openstack.org/#/c/172962/ Porting to Python 3] spec (by Cyril Roelandt) accepted for Liberty, port in progress. [https://review.openstack.org/#/q/project:openstack/neutron+branch:master+topic:bp/neutron-python3,n,z Patches of the blueprint neutron-python3]
 +
|-
  
 +
| neutron-lbaas || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | neutron-lbaasv2-dsvm-py3x-api and neutron-lbaasv2-dsvm-py3x-api-namespace ||
 +
Patches: [https://review.openstack.org/#/q/topic:bp/neutron-python3 topic:bp/neutron-python3].
 +
|-
 +
 +
| neutron-fwaas || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: orange;" | neutron-fwaas-networking-midonet-cross-py35 (non-voting) ||
 +
|-
  
| [https://pypi.python.org/pypi/keystone keystone] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| neutron-vpnaas || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
*  eventlet
 
*  hacking (tests only)
 
*  oslo.messaging
 
*  oslosphinx (tests only)
 
*  pam
 
*  paste
 
*  pycadf
 
*  python-ldap (tests only)
 
*  sqlalchemy-migrate
 
||
 
 
|-
 
|-
  
 +
| [https://pypi.python.org/pypi/nova nova] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 ||
 +
Status at 2019-01-24: Unit tests all running and voting under Python 3 via the use of the mox3 package. There are only 3 unit test files left that are using mox3 (nova/tests/unit/cells/test_cells_messaging.py, nova/tests/unit/network/test_neutronv2.py, nova/tests/unit/network/test_manager.py).
 +
 +
We advised contributors not to spend time replacing mox in files testing cells v1 and nova-network (test_cells_messaging.py and test_manager.py), as both are slated for removal as soon as we're able. Progress is being tracked at: https://blueprints.launchpad.net/nova/+spec/mox-removal-stein
 +
|-
  
| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| [https://pypi.python.org//pypi/sahara sahara] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: orange;" | sahara-tests-scenario-py3 (experimental) || Status at 2018-02-23: the job fails during devstack deployment, because swift does not start. The list of its dependencies is minimal and components like keystone are optional. This was partially addressed with an extra dependency: https://review.openstack.org/#/c/544121/ but this requires a) support in devstack (tentative support in https://review.openstack.org/#/c/544383/) b) at least another extra, or a more generic way of doing this; even with an additional extra explicitly required in devstack, the extras would be installed from pip breaking the test of depending patches.
* eventlet
+
We are facing an issue with pickle library and python 3, once that gets fixed we can try to make the python 3 gate voting.
*  hacking (hacking)
 
*  jsonrpclib
 
*  oslo.rootwrap
 
*  paste
 
python-neutronclient
 
||
 
 
|-
 
|-
  
 +
| [https://pypi.python.org/pypi/swift swift] || style="background-color: red;" | blocked || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 +
Status at 2016-06-22: 3/4,498 unit tests (0%), TODO: 4,495 tests
 +
 +
Update (2016-06-27): [http://lists.openstack.org/pipermail/openstack-dev/2016-June/097102.html master branch is currently in a soft-freeze in order to get encryption middleware merged]
 +
 +
Python 3 bugs: https://bugs.launchpad.net/swift/+bug/1614289
 +
 +
[http://lists.openstack.org/pipermail/openstack-dev/2015-October/078058.html Plan to add Python 3 support to Swift]. Patches: [https://review.openstack.org/#/q/project:openstack/swift+branch:master+topic:py3,n,z Patches with the topic py3]
 +
 +
Python 3 unit tests require liberasurecode >= 1.0.9 and PyEClib >= 1.0.9. To install liberasurecode (in /usr):
 +
 +
git clone https://bitbucket.org/tsg-/liberasurecode/
 +
cd liberasurecode
 +
/autogen.sh  && /configure --prefix=/usr && make && sudo make install
 +
 +
liberasurecode >= 1.0.9 is available in Ubuntu Precise, Ubuntu Trusty, Fedora 22, Fedora 23 and CentoOS 7.
 +
 +
Update 2018-02-22: A list of current blockers: https://gist.github.com/tipabu/833b03a865dba96e9fa2230b82f5d075
 +
 +
Update 2018-03-06: Current list of blockers with progress notes: https://etherpad.openstack.org/p/Swift_py3
  
| [https://pypi.python.org/pypi/nova nova] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
 
*  boto
 
*  eventlet
 
*  hacking (tests only)
 
*  mysql-python (tests only)
 
*  oslo.messaging
 
*  oslo.rootwrap
 
*  oslosphinx (tests only)
 
*  paste
 
*  pycadf
 
*  python-cinderclient
 
*  python-glanceclient
 
*  python-neutronclient
 
*  sqlalchemy-migrate
 
*  suds
 
*  websockify
 
||
 
 
|-
 
|-
  
| [https://pypi.python.org/pypi/swift swift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| [https://pypi.python.org/pypi/trove trove] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | trove-devstack-base ||
*  dnspython
+
Status at 2019-01-06:
*  eventlet
+
Unit tests - 1708 tests, 1707 passed, 1 skipped.
*  hacking (tests only)
+
 
*  netifaces
+
Integration tests are based on trove-devstack-base job.
*  nosehtmloutput (tests only)
+
 
*  openstack.nose-plugin (tests only)
+
Status at 2016-07-25: 1,155 / 2,014 (run_tests.py: 398; testr: 1,521; generate_examples: 95) unit tests (57%), TODO: 859 tests
*  python-swiftclient (tests only)
+
 
||
+
Status at 2016-06-22: 839 (testtools) / 1,992 (run_tests.py: 398; testr: 1,499; generate_examples: 95) unit tests (42%), TODO: 1,153 tests
 +
 
 +
Status at 2018-02-22: It appears that only one test is in the blacklist now: guestagent.test_operating_system
 +
 
 +
run_tests.py (proboscis) failures: https://review.openstack.org/#/c/346905/1
 +
 
 +
Victor Stinner proposed the [https://blueprints.launchpad.net/trove/+spec/trove-python3 trove-python3 blueprint] for Mitaka.
 +
 
 +
From Victor (3/17/2016): "I propose to discuss Python 3 before the summit. For example, prepare a concrete plan to port Trove to Python 3, list technical issues like MySQL-Python, etc."
 +
 
 +
Patches: [https://review.openstack.org/#/q/topic:bp/trove-python3,n,z search for the bp/trove-python3 topic].
 
|-
 
|-
  
 
|}
 
|}
  
=== Dependencies ===
+
=== Other OpenStack Applications and Projects ===
{|class="wikitable sortable"
+
 
 +
List of all OpenStack projects: [http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml projects.yaml].
 +
 
 +
{| class="wikitable sortable"
 +
|-
 +
! style="width: 10%;" |  Project !! style="width: 10%;" |  Unit tests pass on Python 3 !! style="width: 10%;" |  py35 gate || style="width: 20%;" | functional tests !! Comments
 +
|-
 +
 
 +
| adjutant || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: orange;" | Partial* || (*) Adjutant's unit tests include a suite of API tests against Adjutant APIs using DRF's internal test classes. While not true functional tests, they do stand up an internal web server and process real workloads from Adjutant's perspective, with mocking for external APIs and services (Keystone, Nova, Neutron, etc) in the places we touch their clients. This tests 92% of the code, so in all likelihood covers most py3 vs py2 issues. Tempest tests are also planned for later.
 +
|-
 +
 
 +
| [https://pypi.python.org/pypi/aodh aodh] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 +
|-
 +
 
 +
| [https://pypi.python.org/pypi/barbican barbican] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||  style="background-color: lightgreen;" | unspecified ||
 +
Blueprint [https://blueprints.launchpad.net/barbican/+spec/barbican-py3 barbican-py3].
 +
|-
 +
 
 +
| blazar || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | blazar-devstack-dsvm-py35 ||
 
|-
 
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
+
 
 +
| cloudkitty || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | cloudkitty-tempest-full-python3 ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/boto boto] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || See https://github.com/boto/boto3 (experimental)
+
| [https://pypi.python.org/pypi/congress congress] || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | voting* || style="background-color: lightgreen;" | congress-devstack-py35-api-mysql ||
 +
(*)py35 support required tiny modification to antlr3 package, creating debian packaging difficulty
 +
 
 +
Random Stacker wrote a [https://blueprints.launchpad.net/congress/+spec/support-python3 support-python3 blueprint]. Patches: [https://review.openstack.org/#/q/project:openstack/congress+branch:master+topic:bp/support-python3,n,z search for the topic bp/support-python3].
 
|-
 
|-
  
| [https://pypi.python.org/pypi/django-compressor django-compressor] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://github.com/django-compressor/django-compressor/issues/484
+
| cyborg || style="background-color: red;" | No job || style="background-color: red;" | No job || style="background-color: red;" | no job ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/django-openstack-auth django-openstack-auth] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Ported: https://review.openstack.org/#/dashboard/8122
+
| [https://pypi.python.org/pypi/designate designate] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: orange;" | designate-devstack-pdns4-py35 (check queue only) ||
 +
Pradeep Kumar Singh proposed the [https://blueprints.launchpad.net/designate/+spec/designate-py3 blueprint designate-py3] which was accepted for Liberty. Patches: [https://review.openstack.org/#/q/status:open+project:openstack/designate+branch:master+topic:bp/designate-py3,n,z topic:bp/designate-py3].
 
|-
 
|-
  
| [https://pypi.python.org/pypi/dnspython dnspython] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Must use the [https://pypi.python.org/pypi/dnspython3/ Python 3 version], see https://github.com/rthalley/dnspython/issues/60
+
| [https://pypi.python.org/pypi/ec2-api ec2-api] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | openstack-tox-py35 ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: orange;" | In the Git repo || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
+
| [https://pypi.python.org/pypi/freezer freezer] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/eventlet eventlet] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Victor Stinner is working on Trollius (asyncio for Python 2) which may replace eventlet: [http://techs.enovance.com/6562/asyncio-openstack-python3 Use the new asyncio module and Trollius in OpenStack]
+
| karbor || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/hacking hacking] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Cyril Roelandt patch: [https://review.openstack.org/#/c/77585/ Make hacking Python 3 compatible]
+
| [https://pypi.python.org/pypi/kuryr-kubernetes kuryr] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | openstack-tox-py35 ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/jsonrpclib jsonrpclib] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || The project seems dead :(
+
| [https://pypi.python.org/pypi/magnum magnum] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | (voting: magnum-functional-api) AND (non-voting: magnum-functional-k8s AND magnum-functional-swarm-mode) ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/mysql-python mysql-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there.
+
| [https://github.com/openstack/manila manila] || style="background-color: lightgreen;" | Yes  || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | manila-tempest-minimal-dsvm-dummy || ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/netifaces netifaces] || style="background-color:red;" | No || N/A|| style="background-color: red;" | No || || Patch sent by Victor Stinner (in private): [https://bitbucket.org/haypo/misc/src/tip/openstack/netifaces_python3.patch netifaces_python3.patch], Debian has patches too
+
| [https://github.com/openstack/manila-ui manila-ui] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | horizon-openstack-tox-python3-django111 ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/nose-exclude nose-exclude] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
+
| masakari || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | openstack-tox-py35 ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/nosehtmloutput nosehtmloutput] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
| [https://pypi.python.org/pypi/mistral mistral] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
*  nose-exclude (tests only)
 
*  openstack.nose-plugin
 
||
 
*  https://bugs.launchpad.net/ubuntu/+source/python-nosehtmloutput/+bug/1287247
 
*  https://review.openstack.org/#/c/80956/
 
 
|-
 
|-
| [https://pypi.python.org/pypi/nosexcover nosexcover] || style="background-color:lightgreen;" | No || N/A || style="background-color: lightgreen;" | On PyPI || || Python 3 support since 1.0.9
+
 
 +
| monasca-agent || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | monasca-tempest-python3-influxdb || https://storyboard.openstack.org/#!/story/2000975 ||
 
|-
 
|-
| [https://pypi.python.org/pypi/openstack.nose-plugin openstack.nose-plugin] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
 
 +
| monasca-api || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | monasca-tempest-python3-influxdb || https://storyboard.openstack.org/#!/story/2000975 ||
 
|-
 
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color:red;" | No || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || suds ||  
+
 
 +
| monasca-log-api || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | monascalog-python3-tempest ||
 
|-
 
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: orange;" | In the git repo, not on PyPI || ||
+
 
 +
| monasca-ui || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job || https://storyboard.openstack.org/#!/story/2000975 ||
 
|-
 
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
 
 +
| [https://pypi.python.org/pypi/murano murano] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 +
Blueprint: [https://blueprints.launchpad.net/murano/+spec/murano-python-3-support murano-python-3-support].
 
|-
 
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI (1.1.0) || ||
+
 
 +
| murano-agent || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 +
Blueprint: [https://blueprints.launchpad.net/murano/+spec/murano-python-3-support murano-python-3-support].
 
|-
 
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://review.openstack.org/#/c/79311/
+
 
 +
| [https://wiki.openstack.org/wiki/Octavia octavia] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | octavia-tox-functional-py35 ||
 
|-
 
|-
| [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Must be replaced by oslosphinx (without the dot)
+
 
 +
| panko || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | panko-tox-py35-mysql and panko-tox-py35-postgresql ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/pam pam] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The fork [https://pypi.python.org/pypi/simplepam simplepam] works on Python 2 and 3
+
| qinling || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/paramiko paramiko] ||  style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | On PyPI || || Get https://review.openstack.org/#/c/81132/ merged
+
| [https://pypi.python.org/pypi/rally rally] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | rally-tox-functional-py3 and rally-tox-self ||
 +
A big thank to Andrey Kurilin for a lot of work in this direction
 
|-
 
|-
  
| [https://pypi.python.org/pypi/paste paste] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
+
| [https://wiki.openstack.org/wiki/Searchlight searchlight] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | openstack-tox-py36 ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/pycadf pycadf] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
| [https://wiki.openstack.org/wiki/Senlin senlin] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: orange;" | senlin-dsvm-tempest-py35-api, senlin-dsvm-tempest-py35-functional, senlin-dsvm-tempest-py35-integration (check only, non-voting) ||
 
|-
 
|-
  
| [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The project seems dead.
+
| [https://wiki.openstack.org/wiki/Solum solum] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: orange;" | solum-devstack-py35 (check only) ||
 +
Victor Stinner proposed the [https://blueprints.launchpad.net/solum/+spec/solum-python3 blueprint solum-python3] which was approved. Patches: [https://review.openstack.org/#/q/status:open+topic:bp/solum-python3,n,z search for bp/solum-python3 topic] (open reviews).
 
|-
 
|-
  
| [https://pypi.python.org/pypi/qpid-python qpid-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
| storlets || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job (likely blocked on swift) ||
 
|-
 
|-
| [https://pypi.python.org/pypi/rtslib-fb rtslib-fb] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
 
 +
| tacker || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: orange;" | tacker-functional-devstack-python3 (check only, non-voting) ||
 
|-
 
|-
| [https://pypi.python.org/pypi/sphinxcontrib-docbookrestapi sphinxcontrib-docbookrestapi] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
 
 +
| tricircle || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 
|-
 
|-
| [https://pypi.python.org/pypi/sphinxcontrib-httpdomain sphinxcontrib-httpdomain] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
 
 +
| vitrage || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | vitrage-dsvm-api-py35 and vitrage-dsvm-datasources-py35 ||
 
|-
 
|-
| [https://pypi.python.org/pypi/sphinxcontrib-pecanwsme sphinxcontrib-pecanwsme] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
 
 +
| [https://wiki.openstack.org/wiki/Watcher watcher] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 
|-
 
|-
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No||
+
 
*  hacking
+
| [https://wiki.openstack.org/wiki/Zaqar zaqar] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
*  ibm-db-sa
 
*  scripttest
 
||
 
 
|-
 
|-
| [https://pypi.python.org/pypi/suds suds] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Dead project
+
 
 +
| [https://github.com/openstack/zun zun] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | zun-tempest-py35-docker-sql ||
 
|-
 
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI || ||
+
 
 +
| networking-l2gw || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | no job ||
 
|-
 
|-
| [https://pypi.python.org/pypi/thrift thrift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
 
 +
| [https://pypi.org/project/vmware-nsx vmware-nsx] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | All 3rd part CI jobs are running python 3  ||
 
|-
 
|-
| [https://pypi.python.org/pypi/websockify websockify] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
 
 +
| [https://pypi.org/project/vmware-nsxlib vmware-nsxlib] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | tempest-api-vmware-nsxv3 is running python 3.5 and is voting ||
 
|-
 
|-
  
 +
|-
 
|}
 
|}
  
== Reports at OpenStack Summits ==
+
Unsorted projects:
 +
* [https://wiki.openstack.org/wiki/App-Catalog App Catalog]
 +
* Chef OpenStack (Ruby)
 +
* [https://github.com/openstack/kolla/ Kolla]: ~2100 lines of Python, but not "tox -e py27" target
 +
* OpenStack Ansible (Python): Ansible playbooks and roles for deployment
 +
* Puppet OpenStack (Ruby)
 +
* TripleO
 +
 
 +
=== Enable Python 3 in DevStack ===
 +
 
 +
* Fedora::
 +
 
 +
sudo dnf install python3-devel
 +
sudo python3 -m pip install python-memcached  # needed by keystonemiddleware
 +
 
 +
* localrc: add the following line
 +
 
 +
USE_PYTHON3=True
 +
 
 +
See [https://review.openstack.org/#/c/414176/ Switch to python 3.5].
 +
 
 +
=== Functional and Integration Tests ===
 +
 
 +
Etherpad: https://etherpad.openstack.org/p/support-python3.5-functional-tests
 +
 
 +
Doug Hellmann wrote a cross-project specification to run integration tests on Python 3: [http://specs.openstack.org/openstack/openstack-specs/specs/enable-python-3-int-func-tests.html Enabling Python 3 for Application Integration Tests] (spec approved for Liberty).
 +
 
 +
Patch for DevStack: [https://review.openstack.org/#/c/181165/ Enable optional Python 3 support].
 +
 
 +
Projects wanting to run their existing functional test suite against their services on Python 3 should:-
 +
* Add the Python3.4 trove classifier to the setup.cfg configuration file of the respective service.
 +
* set USE_PYTHON3=True in Devstack's localrc config file before running stack.sh after [https://review.openstack.org/#/c/181165/ the DevStack change] is merged or cherry-pick until then.
 +
* set basepython = python3.5 in tox for the necessary functional/integration tests environment.
 +
 
 +
Then run tox -e<your functional test suite name>. Example:- tox -efunctional to run the functional the tests of a certain project.
 +
 
 +
Once the functional test suite is reliably passing against the service in a python 3.5 env, the trove classifiers must be updated along with a new gate job being added to the project.
 +
 
 +
Sirushti Murugesan's attempt on Heat: https://review.openstack.org/#/c/188033/
 +
 
 +
=== Python 3.5 ===
 +
 
 +
* project-config: [https://review.openstack.org/#/c/336272/ Add python35 jobs]
 +
* nova: [https://review.openstack.org/#/c/336432/ Add a py35 environment to tox]
 +
* Neutron issues: https://bugs.launchpad.net/neutron/+bug/1559191
 +
* There is no Jenkins job using Python 3.5 yet
 +
* /usr/bin/python3 is Python 3.5 in Fedora 24, Ubuntu 16.04 LTS (Xenial Xerus) and Debian Stretch (Testing)
 +
 
 +
=== Python 3.6 status ===
 +
 
 +
Python 3.6 is scheduled to be the default/only version of Python available in Ubuntu 18.04 LTS (Bionic Beaver) and Debian 10 (Buster).
  
* Havana summit notes: https://etherpad.openstack.org/p/havana-python3
+
Known issues:
* Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
 
  
== Dependencies ==
+
* eventlet<0.21.0 is broken with Python 3.6, see https://github.com/eventlet/eventlet/issues/371. There is also https://github.com/eventlet/eventlet/issues/401 which is fixed in 0.22.0, so it might seem feasible to upgrade to eventlet-0.22.1 once we can do proper testing.
 +
* Horizon explicitly only claims support for Python 3.5, when running devstack with Python 3.6 one needs to set:
 +
    ENABLED_PYTHON3_PACKAGES=horizon
  
This is an attempt to document which OpenStack dependencies work under Python 3.  We're starting with oslo, since it's the base of everything else, then gradually including other projects.
+
== Status of Python 3 in Linux distributions ==
  
The [https://pypi.python.org/pypi/caniusepython3 caniusepython3] tool can be used to do quick yes/no Python 3 support checks.
+
* ArchLinux already [https://www.archlinux.org/news/python-is-now-python-3 switched to Python 3 by default] in 2011
 +
* Ubuntu wants to remove Python 2 from the default installation for Ubuntu 18.04 LTS: see [https://wiki.ubuntu.com/Python (Python) Plans for 18.04].
 +
* Fedora schedules the switch in Fedora 23 (october 2015): [https://fedoraproject.org/wiki/Changes/Python_3_as_Default Python 3 as the Default Implementation]
 +
** [http://fedora.portingdb.xyz/ Python 3 Porting Database for Fedora] and [http://fedora.portingdb.xyz/history/ history of packages tracked by the Fedora portingdb]
 +
** [https://lwn.net/Articles/690676/ Python 3 in Fedora] (Python 3 in Fedora)
 +
* RHEL: [https://www.softwarecollections.org/en/scls/rhscl/python33/ Python 3.3] and [https://www.softwarecollections.org/en/scls/rhscl/rh-python34/ Python 3.4] are available on RHEL6 and RHEL7 using SCL.
 +
* CentOS: [https://www.softwarecollections.org/en/scls/rhscl/python33/ Python 3.3] and [https://www.softwarecollections.org/en/scls/rhscl/rh-python34/ Python 3.4] are available on CentOS 6 and 7 using SCL (no Red Hat subscription needed, CentOS has its own flavor of SCL)
 +
* In Debian, the plan is to deprecate Python 2 in Stretch (aka: Debian 9, the next Stable Debian after Jessie) and to completely remove Python 2 for the Buster release of Debian (aka: Debian 10, to be released in approximatively 2019).
  
=== oslo.config ===
+
Check the [https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions Project Testing Interface] for updates about the platforms each release is tested on.
  
==== pip-requires ====
+
== Status of Python 2 Support Being Dropped in Dependencies ==
* <span style="color: green;">argparse</span> - included with Python 2.7+.  The separate package is only needed for 2.6.
 
  
==== test-requires ====
+
* Django 2.0 will not include Python 2 support (23 Jan 2017 we use Django>=1.8,<1.9)
* <span style="color: orange;">mox</span> - mox3 supports Python, or replace mox with mock (which is now included in python 3.3)
+
** https://github.com/django/django/pull/7867/files
* <span style="color: green;">nose</span> - supports Python 3
+
** https://github.com/django/django/pull/7871/files
* <span style="color: green;">nose-exclude</span> - 2.6-2.7, 3.1-3.3
+
* Pylons/Pyramid are also considering dropping Python 2
* <span style="color: green;">testtools</span> - 2.6-2.7, 3.2-3.3
+
** https://github.com/Pylons/pyramid/issues/2903
* <span style="color: green;">coverage</span> - supports 2.3-3.3
+
** This is the same team that maintains webob, but it isn't clear if they intend to also drop support there.
* <span style="color: green;">sphinx</span> - supports Python 3
 
  
=== oslo-incubator ===
+
[https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html OpenStack plans to start dropping Python 2 support at the start of the U cycle]
  
==== pip-requires ====
+
== Progress reports, meetings and sprints ==
  
* <span style="color: green;">PasteDeploy</span>: supports 2.5-3.3
+
* Pike PTG: https://etherpad.openstack.org/p/ptg-pike-python35
* <span style="color: green;">WebOb</span>: support Python 3
+
* Ocata Summit
* <span style="color: red;">eventlet</span>: NO (MAJOR PAIN POINT) (gevent doesn't either, though there are some old forks that tried)
+
** https://etherpad.openstack.org/p/ocata-python-3
* <span style="color: orange;">greenlet</span>: supports 2.4-3.2 (lack of 3.3 may be false negative)
+
* openstack-dev: [http://lists.openstack.org/pipermail/openstack-dev/2016-June/097930.html Status of the OpenStack port to Python 3] (Jun 22, 2016). 80 projects/83 (96%) have unit tests passing on Python 3, TODO (3): Nova, Swift and Trove.
* <span style="color: green;">lxml</span>: supports 2.4-3.3
+
* Mitaka Summit
* <span style="color: green;">routes</span>: supports 2.6-3.3
+
** Cross-project session: Python 3 was proposed, https://etherpad.openstack.org/p/mitaka-cross-project-session-planning
* <span style="color: green;">iso8601</span>: NO (perhaps try [http://labix.org/python-dateutil python-dateutil], specifically the parser module?)
+
* Liberty Summit
* <span style="color: orange;">anyjson</span>: 2.4-3.1 (lack of 3.2-3.3 may be false negatives)
+
** Cross-project session on enabling app integration testing: https://etherpad.openstack.org/p/liberty-cross-project-python3
* <span style="color: green;">kombu</span>: supports Python 3 (exact version not given)
+
* Kilo Summit
* <span style="color: green;">argparse</span>: included in Python 2.7+
+
** Doug Hellmann: "Moving our applications to Python 3" cross-project session
* <span style="color: green;">stevedore</span>: supports 2.7, 3.2, and 3.3
+
** Victor Stinner: "Port nova to Python 3", https://etherpad.openstack.org/p/liberty-nova-summit-ideas
* <span style="color: green;">SQLAlchemy</span>: supports Python 3 (exact version not given)
+
* Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)
* <span style="color: red;">qpid-python</span>: NO
+
* Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
 +
* Havana summit notes: https://etherpad.openstack.org/p/havana-python3
  
==== test-requires ====
+
== Articles ==
 +
 
 +
* [http://blogs.rdoproject.org/7894/status-of-python-3-in-openstack-mitaka Status of Python 3 in OpenStack Mitaka] (Victor Stinner, March 2016)
 +
* [http://techs.enovance.com/7807/python-3-status-openstack-liberty Python 3 Status in OpenStack Liberty] (Cyril Roelandt and Victor Stinner, September 2015)
 +
* [http://techs.enovance.com/6722/status-of-the-openstack-port-to-python-3-2 Status of the OpenStack port to Python 3] (Cyril Roelandt, February 2014)
 +
* [http://techs.enovance.com/6521/openstack_python3 Why should OpenStack move to Python 3 right now?] (Victor Stinner, December 2013)
 +
 
 +
== Pycon Montreal 2014: Sprint Port OpenStack to Python 3 ==
  
* <span style="color: green;">distribute</span>: 2.4-3.3
+
Enovance organized a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday) 2014. See the page [[Python3/SprintPycon2014]].
* <span style="color: green;">coverage</span>: 2.3-3.3
 
* <span style="color: green;">fixtures</span>: supports Python 3 (exact version not given)
 
* <span style="color: green;">mock</span>: 2.5-3.3
 
* <span style="color: orange;">mox</span>: use mox3, or replace mox with mock (which is now included in python 3.3)
 
* <span style="color: red;">mysql-python</span>: NO (maybe try pymysql instead?)
 
* <span style="color: green;">nose</span>: Supports Python 3
 
* <span style="color: green;">nose-exclude</span>: 2.6-2.7, 3.1-3.3
 
* <span style="color: red;">nosehtmloutput</span>: ?
 
* <span style="color: green;">pep8</span>: Supports Python 3
 
* <span style="color: green;">pyflakes</span>: Supports Python 3
 
* <span style="color: green;">pylint</span>: Supports Python 3 (tested with Python 3.2)
 
* <span style="color: green;">pyzmq</span>: Supports 2.6-2.7, 3.2+
 
* <span style="color: green;">redis</span>: Supports 2.5-2.7, 3.2+
 
* <span style="color: green;">setuptools-git</span>: 2.4-2.7, 3.1-3.3
 
* <span style="color: green;">sphinx</span>: Supports Python 3
 
* <span style="color: green;">testtools</span>: 2.6-2.7, 3.2-3.3
 
* <span style="color: green;">webtest</span>: 2.6-2.7, 3.2-3.3
 

Latest revision as of 14:17, 7 March 2019

This page tracks the progress of Python 3 effort porting for OpenStack.

IRC: #openstack-dev

Join the #openstack-dev IRC channel on the Freenode network to discuss Python 3.

Python 3

Why should OpenStack move to Python 3 right now?

Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.

Python 2: Python 2.6 support dropped, Python 2.7 only

OpenStack Liberty targets Python 2.7 and 3.4.

Python 2.6 support is being dropped in OpenStack since OpenStack Juno for servers. Python 2.6 support is currently kept in Oslo libraries and clients. See juno-cross-project-future-of-python etherpad.

Python 3.3 support is being dropped since OpenStack Liberty.

Python 2.6 is slowly being dropped in the whole OpenStack project, ex: Remove p26 job from DIB.

Python 2.6 support will be removed in Oslo and Clients clients for OpenStack Mitaka: Oslo libraries dropping python 2.6 compatability.

Port Python 2 code to Python 3

OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. As we are targeting Python 3.4 and up, there is no need to avoid u'unicode' syntax. Do not use six.u('unicode').

Before you begin

If you're doing development with Ubuntu/Debian (and not using devstack with the USE_PYTHON flag set), then you'll need the following packages installed to run the py34 tox unit test targets in the projects:

   sudo apt-get install python3.4 python3.4-dev

sixer tool

The sixer tool helps to replace most basic patterns to add Python 3 compatibility and it respects OpenStack coding style.

Common patterns

  • Replace "for key in dict.iterkeys()" with "for key in dict"
  • Replace dict.iteritems() with dict.items()
  • Replace dict.itervalues() with dict.values()

Note: Replacing dict.iteritems()/.itervalues() with six.iteritems(dict)/six.itervalues(dict) was preferred in the past, but there was a discussion suggesting to avoid six for this. The overhead of creating a temporary list on Python 2 is negligible.

  • Replace iterator.next() with next(iterator)
  • Replace basestring with six.string_types
  • Replace unicode with six.text_type
  • Replace (str, unicode) with six.string_types
  • Replace (int, long) with six.integer_types
  • Replace func.func_name with func.__name__
  • Replace exceptions.OSError with OSError and remove "import exceptions"
  • map() and filter() if a list is needed on Python 3:
    • Replace map(func, data) with [func(item) for item in data]
    • Replace filter(lambda obj: test(obj), data) with [obj for obj in data if test(obj)]
    • Note: Usually, tests fail because map() or filter() objects have no length

Serialization: base64, JSON, etc.

  • [with oslo.serialization 1.10 or newer] Get the base64 module from oslo_serialization (from oslo_serialization import base64) to get functions:
    • oslo_serialization.base64.decode_as_bytes(encoded)
    • oslo_serialization.base64.decode_as_text(encoded, encoding='utf-8')
    • oslo_serialization.base64.encode_as_bytes(s, encoding='utf-8')
    • oslo_serialization.base64.encode_as_text(s, encoding='utf-8')
  • Replace text.encode('base64') and base64.b64encode(text) with:
    • base64.b64encode(text): only accept bytes and returns bytes,
    • or: oslo_serialization.base64.encode_as_bytes(text): accept bytes or Unicode and returns bytes
    • or: oslo_serialization.base64.encode_as_text(text): accept bytes and Unicode and returns Unicode
    • Warning: base64.encodestring(raw) adds a newline ("\n"), whereas encode_as_bytes() and encode_as_text() don't.
  • Replace raw.decode('base64') with:
    • base64.b64decode(raw): return bytes
    • oslo_serialization.base64.decode_as_bytes(encoded): accept bytes and Unicode, returns bytes
    • oslo_serialization.base64.decode_as_text(encoded): accept bytes and Unicode, returns Unicode

Hexadecimal:

  • Replace raw.decode('hex') with binascii.unhexlify(raw)
  • Replace bytes.encode('hex') with binascii.hexlify(bytes)

JSON:

  • [with oslo.serialization 1.10 or newer] Replace json.dumps(obj) with oslo_serialization.jsonutils.dump_as_bytes(obj)
  • Replace json.loads(obj) with oslo_serialization.jsonutils.loads(obj): it accepts bytes and Unicode, bytes is decoded from UTF-8. It avoids "if isinstance(obj, bytes): obj = obj.decode('utf-8')" which may require a second temporary variable.

contextlib.nested

To replaced contextlib.nested is to use contextlib.ExitStack. It's available on Python 2 using contextlib2.ExitStack. For contextlib.nested, nova/test.py defines:

if six.PY3:
    @contextlib.contextmanager
    def nested(*contexts):
        with contextlib.ExitStack() as stack:
            yield [stack.enter_context(c) for c in contexts]
else:
    nested = contextlib.nested

Another option to replaced contextlib.nested is to use the @mock.patch decorator. Example with nested function:

def test_thing(self):
   @mock.patch(...)
   @mock.patch(...)
   @mock.patch(...)
   def do_test(...):
       ...
   do_test()

More options:

oslo_utils.encodeutils.to_utf8

oslo.utils 3.5 has an oslo_utils.encodeutils.to_utf8() function to encode Unicode to UTF-8 and return bytes unchanged.

bytes.decode and unicode.encode

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write data.decode() or text.encode() without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: data.decode('utf-8') or text.encode('utf-8'). The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

safe_decode

Olso Incubator has a function safe_decode() which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_decode(data, 'utf-8'): decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:

  • safe_decode(data)

By default, the decoder is strict. You can specify a different error handler using the optional errors parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

safe_encode

Olso Incubator has a function safe_encode() which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.

  • safe_encode(text) encodes text to the output encoding
  • safe_encode(bytes) may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (incomding parameter) is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (encoding parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_encode(data, incoming='utf-8'): encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:

  • safe_encode(data)

Example:

  • safe_encode(b'\xe9', incoming='latin-1') returns b'\xc3\xa9'.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional errors parameter. Example: safe_encode(b'[\xff]', incoming='ascii', errors='ignore') returns b'[]'.

logging module and format exceptions

The exception_to_unicode(exc) function of oslo_utils.encodeutils is the recommanded way to format an exception to Unicode. This function works on Python 2 and Python 3 and it should avoid mojibake is most cases.

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs b'hello' instead of 'hello'.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

  • str(exc): native string, so use bytes on Python 2
  • six.text_type(exc): always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: unicode(Exception("nonascii:\xe9")).
  • six.u(str(exc)): unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: unicode(str(Exception("\xff")))
  • LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using str(exc). Example: LOG.debug(str(exc)).

HTTP

The HTTP protocol is based on bytes:

  • HTTP body contains bytes. For example, use io.BytesIO for a stream storing an HTTP body.
  • HTTPConnection.getresponse().read() returns bytes (in Python 3, str which is bytes in Python 2)
  • On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
  • It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and HTTP header should use what character encoding?

References to port Python 2 code to Python 3

Common pitfalls

What is a string ?

You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:

   >>> type('foo')
   <type 'str'>
   >>> type(u'foo')
   <type 'unicode'>
   >>> type(b'foo')
   <type 'str'>
   >>> isinstance('foo', six.text_type)
   False
   >>> isinstance(u'foo', six.text_type)
   True
   >>> bytes is str
   True
   >>> b'foo'[0]
   'f'

Python 3:

   >>> type('foo')
   <class 'str'>
   >>> type(u'foo')
   <class 'str'>
   >>> type(b'foo')
   <class 'bytes'>
   >>> isinstance('foo', six.text_type)
   True
   >>> isinstance(b'foo', six.text_type)
   False
   >>> bytes is str
   False
   >>> b'foo'[0]
   102

tox/testr error: db type could not be determined

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/" and then run "tox -e py34" before running "tox -e py27". You only have to do this once. The problem only occurs with "tox -e py34" when .testrepository/ was created by Python 2.

Python 3 Status of OpenStack projects

Common Libraries (Oslo Projects)

See programs.yaml for the list of Common Libraries.

Project Python 3 compatibility Comment
automaton Yes
castellan Yes
cliff Yes
debtcollector Yes
futurist Yes
oslo.cache Yes
oslo.concurrency Yes
oslo.config Yes
oslo.context Yes
oslo.db Yes

PyMySQL driver is now used by default for MySQL. setup.cfg contains the Python 3 classifier.

oslo.i18n Yes
oslo.log Yes
oslo.messaging Yes

oslo.messaging is now fully compatible with Python 3. The old (and now removed) Qpid transport was not compatible, the AMQP driver was ported recently.

oslo.middleware Yes
oslo.policy Yes
oslo.privsep Yes
oslo.reports Yes
oslo.rootwrap Yes
oslo.serialization Yes
oslo.service Partial

WSGI module SSL support is broken under Python 3. We want projects to support deployment without relying on that code, but not all do (see glance and designate below).

oslotest Yes
oslo.versionedobjects Yes
oslo.vmware Yes Supports Python 3 since oslo.vmware 0.13.0
oslo.utils Yes
pylockfile Yes
stevedore Yes
taskflow Yes
tooz Yes

Development tools

Project Python 3 compatibility Comment
cookiecutter yes
hacking yes
pbr yes
stackforge/python-jenkins yes py33 gate is voting
openstack-infra/jenkins-job-builder partial https://review.openstack.org/172238

OpenStack clients

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
keystonemiddleware Yes Voting On PyPI python-memcached keystonemiddleware 1.6.1 supports Python 3.
python-barbicanclient Yes Voting On PyPI
python-ceilometerclient Yes Voting On PyPI
python-cinderclient Yes Voting On PyPI
python-fuelclient Yes voting On PyPI
python-glanceclient Yes Voting On PyPI
python-heatclient Yes Voting On PyPI
python-ironicclient Yes Voting On PyPI
python-keystoneclient Yes Voting On PyPI
python-manilaclient Yes Voting On PyPI
python-marconiclient Yes Voting On PyPI
python-masakariiclient Yes Voting On PyPI
python-monascaclient Yes Voting On PyPI
python-novaclient Yes Voting On PyPI
python-neutronclient Yes Voting On PyPI
python-octaviaclient Yes Voting On PyPI
python-openstackclient Yes Voting On PyPI As of 0.9
python-saharaclient Yes Voting On PyPI
python-searchlightclient Yes Voting On PyPI
python-senlinclient Yes Voting On PyPI
python-swiftclient Yes Voting On PyPI
python-tuskarclient Yes Voting On PyPI
python-troveclient Yes Voting On PyPI
python-watcherclient Yes Voting On PyPI
python-designateclient Yes Voting On PyPI

OpenStack Libraries

Project Python 3 compatibility CI tests running? Python 3 classifiers ? Comment
blazar-nova Yes voting Yes
ceilometermiddleware Yes voting Yes
django_openstack_auth Yes voting Yes

django_openstack_auth 2.0 is fully compatible with Python 3.

glance_store Yes voting Yes glance_store 0.7.0 added the Python 3 support.
heat-translator Yes voting Yes
ironic-lib Yes voting Yes
keystoneauth Yes voting Yes
keystonemiddleware Yes voting Yes
ldappool Yes voting Yes
monasca-statsd Yes voting Yes
mistral-lib Yes voting Yes
neutron-lib Yes voting Yes
octavia-lib Yes voting Yes
os-brick Yes voting Yes os-brick 0.3.2 added Python 3 support.
os-client-config Yes voting Yes
os-traits Yes voting Yes
os-vif Yes voting Yes
os-win Yes voting Yes
osc-lib Yes voting Yes
osc-placement Yes voting Yes
pycadf Yes voting Yes
shade Yes voting Yes
sqlalchemy-migrate Yes voting Yes
sushy Yes voting Yes
taskflow Yes voting Yes
tosca-parser Yes voting Yes

Dependencies: Environment markers

Example to not install the "futures" dependency on Python 3.3 and newer:

futures; python_version < '3.2'

OpenStack applications (tc:approved-release)

OpenStack applications approved by the OpenStack Technical Committee.

Project Unit tests pass on Python 3 py35 gate functional tests Comments
ceilometer Yes voting no job
cinder Yes voting integrated-gate-py35

cinder-python3 blueprint (written by Victor Stinner) has been accepted for Liberty. Patches: bp/cinder-python3 topic. Note: Commit message must use "Partial-Implements: blueprint cinder-python3" to be linked to the blueprint. Kendall Nelson, Ivan Kolodyazhny (core) and Eric Harney (core) prefer this syntax: see COMMIT_MSG.

designate Yes voting designate-bind9-py36 designate-bind9-py35 Relies on oslo.service WSGI code that does not work under Python 3 with SSL. Functional tempest tests are ran on both of these jobs. [1]
glance Yes voting integrated-gate-py35 Relies on oslo.service WSGI code that does not work under Python 3. [2]

Victor Stinner is working on porting Glance to Python3. Patches: Patches for glance (topic: py3).

heat Yes voting heat-functional-convg-mysql-lbaasv2-py35

Sirushti Murugesan wrote and implemented the spec Python34 Support (which was accepted for Liberty). Mitaka version will fully support Python 3. Great job Sirushti Murugesan who did almost all the work!

horizon Yes voting horizon-openstack-tox-python3-django111,horizon-integration-tests, horizon-dsvm-tempest-plugin

George Peristerakis wrote Make Horizon python3.4 compatible in May 2015, the blueprint was approved. Patches: bp/porting-python3 topic.

ironic Yes voting ironic-tempest-dsvm-ipa-partition-pxe_ipmitool-tinyipa-python3

Python 3.4 unit tests are now being run for openstack/ironic. The unit tests are a voting job. Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4: Run tests in py34 environment

keystone Yes voting integrated-gate-py35

The move to pyldap and new ldappool were major hurdles solved in Newton. Great work by the Keystone team to get the last things through the gate in Newton.

neutron Yes voting integrated-gate-py35

Porting to Python 3 spec (by Cyril Roelandt) accepted for Liberty, port in progress. Patches of the blueprint neutron-python3

neutron-lbaas Yes voting neutron-lbaasv2-dsvm-py3x-api and neutron-lbaasv2-dsvm-py3x-api-namespace

Patches: topic:bp/neutron-python3.

neutron-fwaas Yes voting neutron-fwaas-networking-midonet-cross-py35 (non-voting)
neutron-vpnaas Yes voting no job
nova Yes voting integrated-gate-py35

Status at 2019-01-24: Unit tests all running and voting under Python 3 via the use of the mox3 package. There are only 3 unit test files left that are using mox3 (nova/tests/unit/cells/test_cells_messaging.py, nova/tests/unit/network/test_neutronv2.py, nova/tests/unit/network/test_manager.py).

We advised contributors not to spend time replacing mox in files testing cells v1 and nova-network (test_cells_messaging.py and test_manager.py), as both are slated for removal as soon as we're able. Progress is being tracked at: https://blueprints.launchpad.net/nova/+spec/mox-removal-stein

sahara Yes voting sahara-tests-scenario-py3 (experimental) Status at 2018-02-23: the job fails during devstack deployment, because swift does not start. The list of its dependencies is minimal and components like keystone are optional. This was partially addressed with an extra dependency: https://review.openstack.org/#/c/544121/ but this requires a) support in devstack (tentative support in https://review.openstack.org/#/c/544383/) b) at least another extra, or a more generic way of doing this; even with an additional extra explicitly required in devstack, the extras would be installed from pip breaking the test of depending patches.

We are facing an issue with pickle library and python 3, once that gets fixed we can try to make the python 3 gate voting.

swift blocked voting no job

Status at 2016-06-22: 3/4,498 unit tests (0%), TODO: 4,495 tests

Update (2016-06-27): master branch is currently in a soft-freeze in order to get encryption middleware merged

Python 3 bugs: https://bugs.launchpad.net/swift/+bug/1614289

Plan to add Python 3 support to Swift. Patches: Patches with the topic py3

Python 3 unit tests require liberasurecode >= 1.0.9 and PyEClib >= 1.0.9. To install liberasurecode (in /usr):

git clone https://bitbucket.org/tsg-/liberasurecode/
cd liberasurecode
/autogen.sh  && /configure --prefix=/usr && make && sudo make install

liberasurecode >= 1.0.9 is available in Ubuntu Precise, Ubuntu Trusty, Fedora 22, Fedora 23 and CentoOS 7.

Update 2018-02-22: A list of current blockers: https://gist.github.com/tipabu/833b03a865dba96e9fa2230b82f5d075

Update 2018-03-06: Current list of blockers with progress notes: https://etherpad.openstack.org/p/Swift_py3

trove Yes voting trove-devstack-base

Status at 2019-01-06: Unit tests - 1708 tests, 1707 passed, 1 skipped.

Integration tests are based on trove-devstack-base job.

Status at 2016-07-25: 1,155 / 2,014 (run_tests.py: 398; testr: 1,521; generate_examples: 95) unit tests (57%), TODO: 859 tests

Status at 2016-06-22: 839 (testtools) / 1,992 (run_tests.py: 398; testr: 1,499; generate_examples: 95) unit tests (42%), TODO: 1,153 tests

Status at 2018-02-22: It appears that only one test is in the blacklist now: guestagent.test_operating_system

run_tests.py (proboscis) failures: https://review.openstack.org/#/c/346905/1

Victor Stinner proposed the trove-python3 blueprint for Mitaka.

From Victor (3/17/2016): "I propose to discuss Python 3 before the summit. For example, prepare a concrete plan to port Trove to Python 3, list technical issues like MySQL-Python, etc."

Patches: search for the bp/trove-python3 topic.

Other OpenStack Applications and Projects

List of all OpenStack projects: projects.yaml.

Project Unit tests pass on Python 3 py35 gate functional tests Comments
adjutant Yes voting Partial* (*) Adjutant's unit tests include a suite of API tests against Adjutant APIs using DRF's internal test classes. While not true functional tests, they do stand up an internal web server and process real workloads from Adjutant's perspective, with mocking for external APIs and services (Keystone, Nova, Neutron, etc) in the places we touch their clients. This tests 92% of the code, so in all likelihood covers most py3 vs py2 issues. Tempest tests are also planned for later.
aodh Yes voting no job
barbican Yes voting unspecified

Blueprint barbican-py3.

blazar Yes voting blazar-devstack-dsvm-py35
cloudkitty Yes voting cloudkitty-tempest-full-python3
congress Yes voting* congress-devstack-py35-api-mysql

(*)py35 support required tiny modification to antlr3 package, creating debian packaging difficulty

Random Stacker wrote a support-python3 blueprint. Patches: search for the topic bp/support-python3.

cyborg No job No job no job
designate Yes voting designate-devstack-pdns4-py35 (check queue only)

Pradeep Kumar Singh proposed the blueprint designate-py3 which was accepted for Liberty. Patches: topic:bp/designate-py3.

ec2-api Yes voting openstack-tox-py35
freezer Yes voting no job
karbor Yes voting no job
kuryr Yes voting openstack-tox-py35
magnum Yes voting (voting: magnum-functional-api) AND (non-voting: magnum-functional-k8s AND magnum-functional-swarm-mode)
manila Yes voting manila-tempest-minimal-dsvm-dummy
manila-ui Yes voting horizon-openstack-tox-python3-django111
masakari Yes voting openstack-tox-py35
mistral Yes voting no job
monasca-agent Yes voting monasca-tempest-python3-influxdb https://storyboard.openstack.org/#!/story/2000975
monasca-api Yes voting monasca-tempest-python3-influxdb https://storyboard.openstack.org/#!/story/2000975
monasca-log-api Yes voting monascalog-python3-tempest
monasca-ui Yes voting no job https://storyboard.openstack.org/#!/story/2000975
murano Yes voting no job

Blueprint: murano-python-3-support.

murano-agent Yes voting no job

Blueprint: murano-python-3-support.

octavia Yes voting octavia-tox-functional-py35
panko Yes voting panko-tox-py35-mysql and panko-tox-py35-postgresql
qinling Yes voting no job
rally Yes voting rally-tox-functional-py3 and rally-tox-self

A big thank to Andrey Kurilin for a lot of work in this direction

searchlight Yes voting openstack-tox-py36
senlin Yes voting senlin-dsvm-tempest-py35-api, senlin-dsvm-tempest-py35-functional, senlin-dsvm-tempest-py35-integration (check only, non-voting)
solum Yes voting solum-devstack-py35 (check only)

Victor Stinner proposed the blueprint solum-python3 which was approved. Patches: search for bp/solum-python3 topic (open reviews).

storlets Yes voting no job (likely blocked on swift)
tacker Yes voting tacker-functional-devstack-python3 (check only, non-voting)
tricircle Yes voting no job
vitrage Yes voting vitrage-dsvm-api-py35 and vitrage-dsvm-datasources-py35
watcher Yes voting no job
zaqar Yes voting no job
zun Yes voting zun-tempest-py35-docker-sql
networking-l2gw Yes voting no job
vmware-nsx Yes voting All 3rd part CI jobs are running python 3
vmware-nsxlib Yes voting tempest-api-vmware-nsxv3 is running python 3.5 and is voting

Unsorted projects:

  • App Catalog
  • Chef OpenStack (Ruby)
  • Kolla: ~2100 lines of Python, but not "tox -e py27" target
  • OpenStack Ansible (Python): Ansible playbooks and roles for deployment
  • Puppet OpenStack (Ruby)
  • TripleO

Enable Python 3 in DevStack

  • Fedora::
sudo dnf install python3-devel
sudo python3 -m pip install python-memcached   # needed by keystonemiddleware
  • localrc: add the following line
USE_PYTHON3=True

See Switch to python 3.5.

Functional and Integration Tests

Etherpad: https://etherpad.openstack.org/p/support-python3.5-functional-tests

Doug Hellmann wrote a cross-project specification to run integration tests on Python 3: Enabling Python 3 for Application Integration Tests (spec approved for Liberty).

Patch for DevStack: Enable optional Python 3 support.

Projects wanting to run their existing functional test suite against their services on Python 3 should:-

  • Add the Python3.4 trove classifier to the setup.cfg configuration file of the respective service.
  • set USE_PYTHON3=True in Devstack's localrc config file before running stack.sh after the DevStack change is merged or cherry-pick until then.
  • set basepython = python3.5 in tox for the necessary functional/integration tests environment.

Then run tox -e<your functional test suite name>. Example:- tox -efunctional to run the functional the tests of a certain project.

Once the functional test suite is reliably passing against the service in a python 3.5 env, the trove classifiers must be updated along with a new gate job being added to the project.

Sirushti Murugesan's attempt on Heat: https://review.openstack.org/#/c/188033/

Python 3.5

Python 3.6 status

Python 3.6 is scheduled to be the default/only version of Python available in Ubuntu 18.04 LTS (Bionic Beaver) and Debian 10 (Buster).

Known issues:

   ENABLED_PYTHON3_PACKAGES=horizon

Status of Python 3 in Linux distributions

Check the Project Testing Interface for updates about the platforms each release is tested on.

Status of Python 2 Support Being Dropped in Dependencies

OpenStack plans to start dropping Python 2 support at the start of the U cycle

Progress reports, meetings and sprints

Articles

Pycon Montreal 2014: Sprint Port OpenStack to Python 3

Enovance organized a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday) 2014. See the page Python3/SprintPycon2014.