Jump to: navigation, search

Difference between revisions of "Python3"

(→‎OpenStack applications)
 
(265 intermediate revisions by 40 users not shown)
Line 1: Line 1:
 
This page tracks the progress of Python 3 effort porting for OpenStack.
 
This page tracks the progress of Python 3 effort porting for OpenStack.
 +
 +
== IRC: #openstack-dev ==
 +
 +
Join the #openstack-dev IRC channel on the Freenode network to discuss Python 3.
 +
 +
== Current Status ==
 +
 +
Since Ussuri, OpenStack is Python3 only except Swift and Storlets. Python2.7 support and testing has been removed from all other projects. For more details, please refer to [https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html Drop py2.7 community goal]
 +
 +
{| class="wikitable"
 +
|-
 +
! Release !! Python Versions !! Notes
 +
|-
 +
| V ||  <code>python3.6</code> <code>python3.7</code> <code>python3.8</code> [https://review.opendev.org/#/c/693743/]||
 +
|-
 +
| Ussuri (Dropped Python 2.7) || <code>python3.6</code> <code>python3.7</code> [https://governance.openstack.org/tc/reference/runtimes/ussuri.html] || In Ussuri, OpenStack is python3-only except Swift & Storlets. Refer to the [https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html drop Python 2.7 goal]
 +
|-
 +
| Train || <code>python2.7</code> <code>python3.6</code> <code>python3.7</code> [https://governance.openstack.org/tc/reference/runtimes/train.html] || Train is the last release where <code>python2.7</code> testing was required.
 +
|}
  
 
== Python 3 ==
 
== Python 3 ==
Line 6: Line 25:
 
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''
 
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''
  
−
== Python 2: Python 2.6 support dropped, Python 2.7 only ==
 
  
−
OpenStack Liberty targets Python 2.7 and 3.4.
 
  
−
Python 2.6 support is being dropped in OpenStack since OpenStack Juno for servers. Python 2.6 support is currently kept in Oslo libraries and clients. See [https://etherpad.openstack.org/p/juno-cross-project-future-of-python juno-cross-project-future-of-python] etherpad.
+
== Python 3 Status of OpenStack projects ==
  
−
Python 3.3 support is being dropped since OpenStack Liberty.
+
=== Common Libraries (Oslo Projects) ===
  
−
Python 2.6 is slowly being dropped in the whole OpenStack project, ex: [https://review.openstack.org/#/c/201295/ Remove p26 job from DIB].
+
See [http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml programs.yaml] for the list of Common Libraries.
  
−
== Port Python 2 code to Python 3 ==
+
{|class="wikitable sortable"
 +
|-
 +
! Project !! Python 3 Only (Ussuri onwards) !! Python 3 compatibility !! Comment
 +
|-
 +
| [https://pypi.python.org/pypi/automaton automaton] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/castellan castellan] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/cliff cliff] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/debtcollector debtcollector] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/futurist futurist] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.cache oslo.cache] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.concurrency oslo.concurrency] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.context oslo.context] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.db oslo.db] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.i18n oslo.i18n] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.log oslo.log] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.middleware oslo.middleware] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.policy oslo.policy] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.privsep oslo.privsep] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.reports oslo.reports] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.serialization oslo.serialization] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.service oslo.service] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslotest oslotest] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes  ||
 +
|-
 +
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| pylockfile || style="background-color: orange;color: white;" | '''NA''' || style="background-color: lightgreen;" | Yes || Project is deprecated.
 +
|-
 +
| [https://pypi.python.org/pypi/stevedore stevedore] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/tooz tooz] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes ||
 +
|}
  
−
OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. As we are targeting Python 3.4 and up, there is no need to avoid u'unicode' syntax. Do not use six.u('unicode').
+
=== Development tools ===
  
−
=== sixer tool ===
+
{|class="wikitable sortable"
 +
|-
 +
! Project !! Python 3 Only (Ussuri onwards) !! Python 3 compatibility !! Comment
 +
|-
 +
| [https://pypi.python.org/pypi/cookiecutter cookiecutter] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/hacking hacking] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | yes ||
 +
|-
 +
| [https://pypi.python.org/pypi/pbr pbr] || style="background-color: red;color: white;" | '''No''' || style="background-color: lightgreen;" | yes || pbr is not capped so we need to keep support for py2.7 code on safer side.
 +
|-
 +
| stackforge/python-jenkins || style="background-color: red;color: white;" | '''No''' || style="background-color: lightgreen;" | yes ||
 +
|-
 +
| openstack-infra/jenkins-job-builder || style="background-color: red;color: white;" | '''No''' || style="background-color: orange;" | partial || https://review.openstack.org/172238
 +
|}
  
−
The [https://pypi.python.org/pypi/sixer sixer] tool helps to replace most basic patterns to add Python 3 compatibility and it respects OpenStack coding style.
+
=== OpenStack clients ===
  
−
=== Common patterns ===
+
{|class="wikitable sortable"
 +
|-
 +
! Project !! Python 3 Only (Ussuri onwards) !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
 +
|-
  
−
* Replace dict.iteritems() with dict.items()
+
| [https://pypi.python.org/pypi/keystonemiddleware keystonemiddleware] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI ||  ||
−
* Replace dict.itervalues() with dict.values()
+
|-
−
* Replace "for key in dict.iterkeys()" with "for key in dict"
 
−
* Replace iterator.next() with next(iterator)
 
−
* Replace basestring with six.string_types
 
−
* Replace unicode with six.text_type
 
−
* Replace (str, unicode) with six.string_types
 
−
* Replace (int, long) with six.integer_types
 
−
* Replace text.encode('base64') with base64.b64encode(text) which requires to encode (Unicode) text to UTF-8 on Python 3
 
  
−
if six.PY3:
+
| [https://pypi.python.org/pypi/python-barbicanclient python-barbicanclient] || style="background-color: red;color: white;" | '''No''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || || https://review.opendev.org/#/c/699096/
−
    text = text.encode('utf-8')
+
|-
−
text_b64 = base64.b64encode(text)
 
  
−
* Replace raw.decode('base64') with base64.b64decode(raw). The output may be decoded from UTF-8 to get back text (Unicode) on Python 3.
 
  
−
text = base64.b64decode(raw)
+
| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
−
if six.PY3:
+
|-
−
    text = text.decode('utf-8')
 
  
−
* Replace raw.decode('hex') with binascii.unhexlify(raw)
+
| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
−
* Replace bytes.encode('hex') with binascii.hexlify(bytes)
+
|-
  
−
Note: Replacing dict.iteritems() with six.iteritems(dict) was preferred in the past, but there was [http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html a discussion suggesting to avoid six for this].
+
| [https://pypi.python.org/pypi/python-fuelclient python-fuelclient] || style="background-color: orange;color: white;" | '''N/A''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" |  On PyPI || || This project is no longer maintained.
 +
|-
  
−
=== contextlib.nested ===
+
| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient]  || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color:lightgreen" | On PyPI || ||
 +
|-
  
−
To replaced contextlib.nested is to use [https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack contextlib.ExitStack]. It's available on Python 2 using [https://contextlib2.readthedocs.org/en/latest/#contextlib2.ExitStack contextlib2.ExitStack]. For contextlib.nested, nova/test.py defines:
+
| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes  || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
 +
|-
  
−
  if six.PY3:
+
| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes|| style="background-color: lightgreen" | Voting || style="background-color: lightgreen;"  | On PyPI ||  ||
−
    @contextlib.contextmanager
+
|-
−
    def nested(*contexts):
 
−
        with contextlib.ExitStack() as stack:
 
−
            yield [stack.enter_context(c) for c in contexts]
 
−
  else:
 
−
    nested = contextlib.nested
 
  
−
Another option to replaced contextlib.nested is to use the @mock.patch decorator. Example with nested function:
+
| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI ||  ||
 +
|-
  
−
def test_thing(self):
+
| [https://pypi.python.org/pypi/python-manilaclient python-manilaclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
−
    @mock.patch(...)
+
|-
−
    @mock.patch(...)
 
−
    @mock.patch(...)
 
−
    def do_test(...):
 
−
        ...
 
−
   
 
−
    do_test()
 
  
−
More options:
+
| [https://pypi.python.org/pypi/python-marconiclient python-zaqarclient]|| style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
−
* Mock provides a context that patches multiple things so that no nesting is needed: [https://docs.python.org/dev/library/unittest.mock.html#patch-multiple mock.patch.multiple()]
+
|-
−
* oslotest provides fixtures for mock, so you don't need a context: [http://docs.openstack.org/developer/oslotest/api.html#module-oslotest.mockpatch oslotest.mockpatch].
 
  
−
=== bytes.decode and unicode.encode ===
+
| [https://pypi.python.org/pypi/python-masakariclient python-masakariiclient]|| style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 +
|-
  
−
Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.
+
| [https://pypi.python.org/pypi/python-monascaclient python-monascaclient]  || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 +
|-
  
−
Don't write <code>data.decode()</code> or <code>text.encode()</code> without parameter, because you will use a different encoding on Python 2 and Python 3.
+
| [https://pypi.python.org/pypi/python-novaclient python-novaclient]  || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
 +
|-
  
−
Use an explicit encoding instead. Example: <code>data.decode('utf-8')</code> or <code>text.encode('utf-8')</code>. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).
+
| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting  || style="background-color: lightgreen;" | On PyPI ||  ||
 +
|-
  
−
=== safe_decode ===
+
| [https://pypi.python.org/pypi/python-octaviaclient python-octaviaclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting  || style="background-color: lightgreen;" | On PyPI ||  ||
 +
|-
  
−
Olso Incubator has a function '''safe_decode()''' which can be used to decode a bytes string and pass text strings unchanged.
+
| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient]      || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || || As of 0.9
 +
|-
  
−
The default encoding is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
+
| [https://pypi.python.org/pypi/python-saharaclient python-saharaclient]    || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI ||  ||
−
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
+
|-
−
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
 
  
−
It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
+
| [https://pypi.python.org/pypi/python-searchlightclient python-searchlightclient]    || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI ||  ||
 +
|-
  
−
Safe usage:
+
| [https://pypi.python.org/pypi/python-senlinclient python-senlinclient]    || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI ||  ||
−
* <code>safe_decode(data, 'utf-8')</code>: decode bytes from UTF-8 or returns data unchanged if it's already a text string
+
|-
  
−
Unsafe usage:
+
| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient]  || style="background-color: red;color: white;" | '''No''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || || Swift is keeping python2 support
−
* <code>safe_decode(data)</code>
+
|-
  
−
By default, the decoder is strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.
+
| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient]  || style="background-color: orange;" | N/A || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || || This project is no longer maintained.
 +
|-
  
−
=== safe_encode ===
+
| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
 +
|-
  
−
Olso Incubator has a function '''safe_encode()''' which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
+
| [https://pypi.python.org/pypi/python-watcherclient python-watcherclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
−
* <code>safe_encode(text)</code> encodes text to the output encoding
+
|-
−
* <code>safe_encode(bytes)</code> may decode the string and then reencode to a different encoding if input and output encodings are different
 
  
−
The default input encoding (<code>incomding</code> parameter) is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
+
| [https://pypi.python.org/pypi/python-designateclient python-designateclient] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
−
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
 
−
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
 
  
−
The default output encoding (<code>encoding</code> parameter) is UTF-8.
+
|}
  
−
It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
+
=== OpenStack Libraries ===
  
−
Safe usage:
+
{|class="wikitable sortable"
−
* <code>safe_encode(data, incoming='utf-8')</code>: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)
+
|-
 +
! Project !! Python 3 Only (Ussuri onwards) !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Comment
 +
|-
 +
| blazar-nova || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
 +
| ceilometermiddleware || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
Unsafe usage:
+
| [https://pypi.python.org/pypi/django_openstack_auth django_openstack_auth] || style="background-color: orange;color: white;" | '''N/A''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
−
* <code>safe_encode(data)</code>
+
The code has been merged into openstack/horizon.
 +
|-
  
−
Example:
+
| [https://pypi.python.org/pypi/glance_store glance_store] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
−
* <code>safe_encode(b'\xe9', incoming='latin-1')</code> returns <code>b'\xc3\xa9'</code>.
+
|-
  
−
By default, the encoder and the decoder are strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: <code>safe_encode(b'[\xff]', incoming='ascii', errors='ignore')</code> returns <code>b'[]'</code>.
+
| heat-translator || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
=== logging module and format exceptions ===
+
| ironic-lib || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
The <code>exception_to_unicode(exc)</code> function of oslo_utils.encodeutils is the recommanded way to format an exception to Unicode. This function works on Python 2 and Python 3 and it should avoid mojibake is most cases.
+
| keystoneauth || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs <code>b'hello'</code> instead of <code>'hello'</code>.
+
| keystonemiddleware || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
There is no clear rule for format exceptions yet. There are different choices depending on the project:
+
| ldappool || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
* <code>str(exc)</code>: native string, so use bytes on Python 2
+
| monasca-statsd || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
−
* <code>six.text_type(exc)</code>: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: <code>unicode(Exception("nonascii:\xe9"))</code>.
+
|-
−
* <code>six.u(str(exc))</code>: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: <code>unicode(str(Exception("\xff")))</code>
 
−
* <code>LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})</code>
 
  
−
Since logging functions expect text strings on Python 3, logged exceptions should be formatted using <code>str(exc)</code>. Example: <code>LOG.debug(str(exc))</code>.
+
| mistral-lib || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
=== HTTP ===
+
| neutron-lib || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
The HTTP protocol is based on '''bytes''':
+
| octavia-lib || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
* HTTP body contains '''bytes'''. For example, use io.BytesIO for a stream storing an HTTP body.
+
| [https://pypi.python.org/pypi/os-brick os-brick] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
−
* HTTPConnection.getresponse().read() returns '''bytes''' (in Python 3, '''str''' which is bytes in Python 2)
 
−
* On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
 
−
* It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and [http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding  HTTP header should use what character encoding?]
 
  
−
=== References to port Python 2 code to Python 3 ===
+
|-
−
* [http://python3porting.com/ Porting to Python 3 Book] by Lennart Regebro, especially the [http://python3porting.com/differences.html Language differences and workarounds].
 
−
* [http://docs.python.org/dev/howto/pyporting.html HOWTO: Porting Python 2 Code to Python 3] by Brett Cannon
 
−
* [https://wiki.python.org/moin/PortingPythonToPy3k Porting Python Code to 3.x]
 
−
* [http://code.google.com/p/python-incompatibility/  python-incompatibility]: Demonstrates incompatibilities between Python versions.
 
  
−
=== Common pitfalls ===
+
| os-client-config || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
==== What is a string ? ====
+
| os-traits || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
−
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:
+
|-
  
−
Python 2:
+
| os-vif || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
−
    >>> type('foo')
+
|-
−
    <type 'str'>
 
−
    >>> type(u'foo')
 
−
    <type 'unicode'>
 
−
    >>> type(b'foo')
 
−
    <type 'str'>
 
−
    >>> isinstance('foo', six.text_type)
 
−
    False
 
−
    >>> isinstance(u'foo', six.text_type)
 
−
    True
 
−
    >>> bytes is str
 
−
    True
 
−
    >>> b'foo'[0]
 
−
    'f'
 
  
−
Python 3:
+
| os-win || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
−
    >>> type('foo')
+
|-
−
    <class 'str'>
 
−
    >>> type(u'foo')
 
−
    <class 'str'>
 
−
    >>> type(b'foo')
 
−
    <class 'bytes'>
 
−
    >>> isinstance('foo', six.text_type)
 
−
    True
 
−
    >>> isinstance(b'foo', six.text_type)
 
−
    False
 
−
    >>> bytes is str
 
−
    False
 
−
    >>> b'foo'[0]
 
−
    102
 
  
−
==== tox/testr error: db type could not be determined ====
+
| osc-lib || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.
+
| osc-placement || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
Workaround: "rm -rf .testrepository/" and then run "tox -e py34" before running "tox -e py27". You only have to do this once. The problem only occurs with "tox -e py34" when .testrepository/ was created by Python 2.
+
| pycadf || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
== Python 3 Status of OpenStack projects ==
+
| shade || style="background-color: orange;color: white;" | '''No''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes || shade is on extended maintainance.
 +
|-
  
−
=== Common Libraries (Oslo Projects) ===
+
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color: orange;color: white;" | '''N/A''' || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color:lightgreen;" | Yes || This project is not maintained under OpenStack
 +
|-
  
−
For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160
+
| sushy || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 +
|-
  
−
{|class="wikitable sortable"
+
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: green;color: white;" | '''Yes''' || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
−
! Project !! Python 3 compatibility !! Comment
+
 
 +
| tosca-parser || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/cliff cliff] || style="background-color: lightgreen;" | Yes ||
+
 
 +
|}
 +
 
 +
=== Dependencies: Environment markers ===
 +
 
 +
Example to not install the "futures" dependency on Python 3.3 and newer:
 +
 
 +
<pre>
 +
futures; python_version < '3.2'
 +
</pre>
 +
 
 +
=== OpenStack applications (tc:approved-release) ===
 +
 
 +
OpenStack applications [https://governance.openstack.org/tc/reference/tags/tc_approved-release.html#tag-tc-approved-release approved by the OpenStack Technical Committee].
 +
 
 +
{| class="wikitable sortable"
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.concurrency oslo.concurrency] || style="background-color: lightgreen;" | Yes ||
+
! style="width: 10%;" | Project !! Python 3 Only (Ussuri onwards) !! style="width: 10%;" | Unit tests pass on Python 3 !! style="width: 10%;" | py35 gate !! style="width: 20%;" | functional tests !! Comments
−
|-/
 
−
| oslo-incubator || style="background-color: lightgreen;" | Yes || py34 gate is voting
 
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.context oslo.context] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| [https://pypi.python.org/pypi/cinder cinder] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 ||
 +
[https://blueprints.launchpad.net/cinder/+spec/cinder-python3 cinder-python3] blueprint (written by Victor Stinner) has been accepted for Liberty. Patches: [https://review.openstack.org/#/q/project:openstack/cinder+topic:bp/cinder-python3,n,z bp/cinder-python3 topic]. Note: Commit message must use "Partial-Implements: blueprint cinder-python3" to be linked to the blueprint. Kendall Nelson, Ivan Kolodyazhny (core) and Eric Harney (core) prefer this syntax: see [https://review.openstack.org/#/c/229991/2//COMMIT_MSG COMMIT_MSG].
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.db oslo.db] || style="background-color: lightgreen;" | Yes ||
+
 
−
PyMySQL driver is now used by default for MySQL. setup.cfg contains the Python 3 classifier.
+
| designate || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | designate-bind9-py36 designate-bind9-py35 || ! style="background-color: orange;" | Relies on oslo.service WSGI code that does not work under Python 3 with SSL. Functional tempest tests are ran on both of these jobs. [http://lists.openstack.org/pipermail/openstack-dev/2018-May/130277.html]
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.i18n oslo.i18n] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| [https://pypi.python.org/pypi/glance glance] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 || ! style="background-color: orange;" | Relies on oslo.service WSGI code that does not work under Python 3. [http://lists.openstack.org/pipermail/openstack-dev/2018-May/130277.html]
 +
Victor Stinner is working on porting Glance to Python3. Patches: [https://review.openstack.org/#/q/status:open+project:openstack/glance+branch:master+topic:py3,n,z Patches for glance (topic: py3)].
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.log oslo.log] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| [https://pypi.python.org/pypi/heat heat] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | heat-functional-convg-mysql-lbaasv2-py35 ||
 +
Sirushti Murugesan wrote and implemented the spec [http://specs.openstack.org/openstack/heat-specs/specs/liberty/heat-python34-support.html Python34 Support] (which was accepted for Liberty). Mitaka version will fully support Python 3. Great job Sirushti Murugesan who did almost all the work!
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color: orange;" | Yes, except ... ||
 
−
oslo.messaging 1.11.0 fully works on Python 3... except of Qpid and AMQP 1.0 transports which are Python 2 only.
 
  
−
Patch: [https://review.openstack.org/#/c/204696/ Port the AMQP 1.0 driver to Python 3].
+
| [https://pypi.python.org/pypi/horizon horizon] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | horizon-openstack-tox-python3-django111,horizon-integration-tests, horizon-dsvm-tempest-plugin ||
 +
George Peristerakis wrote [https://blueprints.launchpad.net/horizon/+spec/porting-python3 Make Horizon python3.4 compatible] in May 2015, the blueprint was approved. Patches: [https://review.openstack.org/#/q/topic:bp/porting-python3+project:openstack/horizon,n,z bp/porting-python3 topic].
 +
|-
  
−
Note: We care only about AMQP1.0 driver. The legacy Qpid driver is deprecated, buggy and unmaintened.
+
| [https://pypi.python.org/pypi/ironic ironic] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ironic-tempest-dsvm-ipa-partition-pxe_ipmitool-tinyipa-python3 ||
 +
[http://lists.openstack.org/pipermail/openstack-dev/2015-May/064049.html Python 3.4 unit tests are now being run for openstack/ironic.  The unit tests are a voting job.]
 +
Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4: [https://review.openstack.org/#/c/156192/ Run tests in py34 environment]
 +
|-
  
−
The py34 check job is not more voting because of [//bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1367907 ubuntu still uses Python 3.4.0 which has a severe bug].
+
| [https://pypi.python.org/pypi/keystone keystone] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 ||
 +
The move to pyldap and new ldappool were major hurdles solved in Newton. Great work by the Keystone team to get the last things through the gate in Newton.
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.middleware oslo.middleware] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| [https://pypi.python.org/pypi/neutron neutron] || style="background-color: green;color: white;" | '''Yes''' || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 ||
 +
[https://review.openstack.org/#/c/172962/ Porting to Python 3] spec (by Cyril Roelandt) accepted for Liberty, port in progress. [https://review.openstack.org/#/q/project:openstack/neutron+branch:master+topic:bp/neutron-python3,n,z Patches of the blueprint neutron-python3]
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| neutron-lbaas || style="background-color: orange;color: white;" | '''N/A''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | neutron-lbaasv2-dsvm-py3x-api and neutron-lbaasv2-dsvm-py3x-api-namespace ||This project is no longer maintained.
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.serialization oslo.serialization] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| neutron-fwaas || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | neutron-fwaas-networking-midonet-cross-py35 (non-voting) ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || ? || The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible?
+
 
 +
| neutron-vpnaas || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslotest oslotest] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| [https://pypi.python.org/pypi/nova nova] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | integrated-gate-py35 ||
 +
Status at 2019-01-24: Unit tests all running and voting under Python 3 via the use of the mox3 package. There are only 3 unit test files left that are using mox3 (nova/tests/unit/cells/test_cells_messaging.py, nova/|tests/unit/network/test_neutronv2.py, nova/tests/unit/network/test_manager.py).
 +
 
 +
We advised contributors not to spend time replacing mox in files testing cells v1 and nova-network (test_cells_messaging.py and test_manager.py), as both are slated for removal as soon as we're able. Progress is being tracked at: https://blueprints.launchpad.net/nova/+spec/mox-removal-stein
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| [https://pypi.python.org//pypi/sahara sahara] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | sahara-tests-scenario-py3 (experimental) || Status at 2018-02-23: the job fails during devstack deployment, because swift does not start. The list of its dependencies is minimal and components like keystone are optional. This was partially addressed with an extra dependency: https://review.openstack.org/#/c/544121/ but this requires  a) support in devstack (tentative support in https://review.openstack.org/#/c/544383/) b) at least another extra, or a more generic way of doing this; even with an additional extra explicitly required in devstack, the extras would be installed from pip breaking the test of depending patches.
 +
We are facing an issue with pickle library and python 3, once that gets fixed we can try to make the python 3 gate voting.
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: lightgreen;" | Yes  || Supports Python 3 since oslo.vmware 0.13.0
+
 
 +
| [https://pypi.python.org/pypi/swift swift] || style="background-color: red;color: white;" | '''No''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | || Swift will keep python2 support.
 +
 
 +
Update (2016-06-27): [http://lists.openstack.org/pipermail/openstack-dev/2016-June/097102.html master branch is currently in a soft-freeze in order to get encryption middleware merged]
 +
 
 +
Python 3 bugs: https://bugs.launchpad.net/swift/+bug/1614289
 +
 
 +
[http://lists.openstack.org/pipermail/openstack-dev/2015-October/078058.html Plan to add Python 3 support to Swift]. Patches: [https://review.openstack.org/#/q/project:openstack/swift+branch:master+topic:py3,n,z Patches with the topic py3]
 +
 
 +
Python 3 unit tests require liberasurecode >= 1.0.9 and PyEClib >= 1.0.9. To install liberasurecode (in /usr):
 +
 
 +
git clone https://bitbucket.org/tsg-/liberasurecode/
 +
cd liberasurecode
 +
/autogen.sh  && /configure --prefix=/usr && make && sudo make install
 +
 
 +
liberasurecode >= 1.0.9 is available in Ubuntu Precise, Ubuntu Trusty, Fedora 22, Fedora 23 and CentoOS 7.
 +
 
 +
Update 2018-02-22: A list of current blockers: https://gist.github.com/tipabu/833b03a865dba96e9fa2230b82f5d075
 +
 
 +
Update 2018-03-06: Current list of blockers with progress notes: https://etherpad.openstack.org/p/Swift_py3
 +
 
 
|-
 
|-
−
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| [https://pypi.python.org/pypi/trove trove] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | trove-devstack-base ||
 +
Status at 2019-01-06:
 +
Unit tests - 1708 tests, 1707 passed, 1 skipped.
 +
 
 +
Integration tests are based on trove-devstack-base job.
 +
 
 +
Status at 2016-07-25: 1,155 / 2,014 (run_tests.py: 398; testr: 1,521; generate_examples: 95) unit tests (57%), TODO: 859 tests
 +
 
 +
Status at 2016-06-22: 839 (testtools) / 1,992 (run_tests.py: 398; testr: 1,499; generate_examples: 95) unit tests (42%), TODO: 1,153 tests
 +
 
 +
Status at 2018-02-22: It appears that only one test is in the blacklist now: guestagent.test_operating_system
 +
 
 +
run_tests.py (proboscis) failures: https://review.openstack.org/#/c/346905/1
 +
 
 +
Victor Stinner proposed the [https://blueprints.launchpad.net/trove/+spec/trove-python3 trove-python3 blueprint] for Mitaka.
 +
 
 +
From Victor (3/17/2016): "I propose to discuss Python 3 before the summit. For example, prepare a concrete plan to port Trove to Python 3, list technical issues like MySQL-Python, etc."
 +
 
 +
Patches: [https://review.openstack.org/#/q/topic:bp/trove-python3,n,z search for the bp/trove-python3 topic].
 
|-
 
|-
−
| pylockfile || style="background-color: lightgreen;" | Yes ||
+
 
 +
|}
 +
 
 +
=== Other OpenStack Applications and Projects ===
 +
 
 +
List of all OpenStack projects: [http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml projects.yaml].
 +
 
 +
{| class="wikitable sortable"
 +
|-
 +
! style="width: 10%;" | Project !! Python 3 Only (Ussuri onwards) !! style="width: 10%;" | Unit tests pass on Python 3 !! style="width: 10%;" | py35 gate || style="width: 20%;" | functional tests !! Comments
 
|-
 
|-
−
| [https://pypi.python.org/pypi/stevedore stevedore] || style="background-color: lightgreen;" | Yes ||
+
 
 +
| adjutant || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Partial* || (*) Adjutant's unit tests include a suite of API tests against Adjutant APIs using DRF's internal test classes. While not true functional tests, they do stand up an internal web server and process real workloads from Adjutant's perspective, with mocking for external APIs and services (Keystone, Nova, Neutron, etc) in the places we touch their clients. This tests 92% of the code, so in all likelihood covers most py3 vs py2 issues. Tempest tests are also planned for later.
 
|-
 
|-
−
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: lightgreen;" | Yes ||
 
−
|}
 
  
−
=== Development tools ===
+
| [https://pypi.python.org/pypi/aodh aodh] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ||
 +
|-
  
−
{|class="wikitable sortable"
+
| [https://pypi.python.org/pypi/barbican barbican] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||  style="background-color: lightgreen;" | unspecified ||
 +
Blueprint [https://blueprints.launchpad.net/barbican/+spec/barbican-py3 barbican-py3].
 
|-
 
|-
−
! Project !! Python 3 compatibility !! Comment
+
 
 +
| blazar || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | blazar-devstack-dsvm-py35 ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/cookiecutter cookiecutter] || style="background-color: lightgreen;" | yes ||
+
 
 +
| cloudkitty || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | cloudkitty-tempest-full-python3 ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/hacking hacking] || style="background-color: lightgreen;" | yes || py33 gate is not voting
+
 
 +
| [https://pypi.python.org/pypi/congress congress] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting* || style="background-color: lightgreen;" | congress-devstack-py35-api-mysql ||
 +
(*)py35 support required tiny modification to antlr3 package, creating debian packaging difficulty
 +
 
 +
Random Stacker wrote a [https://blueprints.launchpad.net/congress/+spec/support-python3 support-python3 blueprint]. Patches: [https://review.openstack.org/#/q/project:openstack/congress+branch:master+topic:bp/support-python3,n,z search for the topic bp/support-python3].
 
|-
 
|-
−
| [https://pypi.python.org/pypi/pbr pbr] || style="background-color: lightgreen;" | yes ||
+
 
 +
| cyborg || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | No job || style="background-color: lightgreen;" | No job || style="background-color: lightgreen;" | ||
 
|-
 
|-
−
| stackforge/python-jenkins || style="background-color: lightgreen;" | yes || py33 gate is voting
+
 
 +
| [https://pypi.python.org/pypi/designate designate] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | designate-devstack-pdns4-py35 (check queue only) ||
 +
Pradeep Kumar Singh proposed the [https://blueprints.launchpad.net/designate/+spec/designate-py3 blueprint designate-py3] which was accepted for Liberty. Patches: [https://review.openstack.org/#/q/status:open+project:openstack/designate+branch:master+topic:bp/designate-py3,n,z topic:bp/designate-py3].
 
|-
 
|-
−
| openstack-infra/jenkins-job-builder || style="background-color: orange;" | partial || https://review.openstack.org/172238
 
−
|}
 
  
−
=== OpenStack clients ===
+
| [https://pypi.python.org/pypi/ec2-api ec2-api] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | openstack-tox-py35 ||
 +
|-
  
−
{|class="wikitable sortable"
+
| [https://pypi.python.org/pypi/freezer freezer] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" |  ||
 
|-
 
|-
−
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
+
 
 +
| karbor || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" |  ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/keystonemiddleware keystonemiddleware] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || python-memcached || keystonemiddleware 1.6.1 supports Python 3.
+
| [https://pypi.python.org/pypi/kuryr-kubernetes kuryr] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | openstack-tox-py35 ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-barbicanclient python-barbicanclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
+
| [https://pypi.python.org/pypi/magnum magnum] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | (voting: magnum-functional-api) AND (non-voting: magnum-functional-k8s AND magnum-functional-swarm-mode) ||
 
|-
 
|-
  
 +
| [https://github.com/openstack/manila manila] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes  || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" |  manila-tempest-minimal-dsvm-dummy || ||
 +
|-
  
−
| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
+
| [https://github.com/openstack/manila-ui manila-ui] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | horizon-openstack-tox-python3-django111 ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
+
| masakari || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | openstack-tox-py35 ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-fuelclient python-fuelclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: red;" | [https://review.openstack.org/#/c/225103/ Not yet] || ||
+
| [https://pypi.python.org/pypi/mistral mistral] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ||
−
Patches: [https://review.openstack.org/#/q/status:open+project:stackforge/python-fuelclient+branch:master+topic:py3,n,z topic:py3].
 
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color:lightgreen" | On PyPI || ||  
+
| monasca-agent || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | monasca-tempest-python3-influxdb || https://storyboard.openstack.org/#!/story/2000975 ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
+
| monasca-api || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | monasca-tempest-python3-influxdb || https://storyboard.openstack.org/#!/story/2000975 ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient]  || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| monasca-log-api || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | monascalog-python3-tempest ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| monasca-ui || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | || https://storyboard.openstack.org/#!/story/2000975 ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-manilaclient python-manilaclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| [https://pypi.python.org/pypi/murano murano] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" |  ||
 +
Blueprint: [https://blueprints.launchpad.net/murano/+spec/murano-python-3-support murano-python-3-support].
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| murano-agent || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ||
 +
Blueprint: [https://blueprints.launchpad.net/murano/+spec/murano-python-3-support murano-python-3-support].
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-novaclient python-novaclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPII || ||
+
| [https://wiki.openstack.org/wiki/Octavia octavia] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | octavia-tox-functional-py35 ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting  || style="background-color: lightgreen;" | On PyPI ||  ||  
+
| panko || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | panko-tox-py35-mysql and panko-tox-py35-postgresql ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient]      || style="background-color: lightgreen" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || || As of 0.9
+
| qinling || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-saharaclient python-saharaclient]   || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI ||  ||  
+
| [https://pypi.python.org/pypi/rally rally] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||  style="background-color: lightgreen;" | rally-tox-functional-py3 and rally-tox-self ||
 +
A big thank to Andrey Kurilin for a lot of work in this direction
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient]   || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| [https://wiki.openstack.org/wiki/Searchlight searchlight] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | openstack-tox-py36 ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient]   || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
+
| [https://wiki.openstack.org/wiki/Senlin senlin] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | senlin-dsvm-tempest-py35-api, senlin-dsvm-tempest-py35-functional, senlin-dsvm-tempest-py35-integration (check only, non-voting) ||
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
+
| [https://wiki.openstack.org/wiki/Solum solum] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | solum-devstack-py35 (check only) ||
 +
Victor Stinner proposed the [https://blueprints.launchpad.net/solum/+spec/solum-python3 blueprint solum-python3] which was approved. Patches: [https://review.openstack.org/#/q/status:open+topic:bp/solum-python3,n,z search for bp/solum-python3 topic] (open reviews).
 +
|-
  
 +
| storlets || style="background-color: red;color: white;" | '''No''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" |  || Storlets will keep python2 support.
 
|-
 
|-
  
−
| [https://pypi.python.org/pypi/python-designateclient python-designateclient] || style="background-color: orange;" | work in progress || style="background-color: lightgreen;" | Voting || style="background-color: red;" | No || ||
+
| tacker || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | tacker-functional-devstack-python3 (check only, non-voting) ||
 +
|-
  
−
|}
+
| tricircle || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ||
 +
|-
  
−
=== OpenStack Libraries ===
+
| vitrage || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | vitrage-dsvm-api-py35 and vitrage-dsvm-datasources-py35 ||
 +
|-
  
−
{|class="wikitable sortable"
+
| [https://wiki.openstack.org/wiki/Watcher watcher] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" |  ||
 
|-
 
|-
−
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Comment
+
 
 +
| [https://wiki.openstack.org/wiki/Zaqar zaqar] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" |  ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/django_openstack_auth django_openstack_auth] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
 
−
django_openstack_auth 2.0 is fully compatible with Python 3.
 
  
 +
| [https://github.com/openstack/zun zun] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | zun-tempest-py35-docker-sql ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/glance_store glance_store] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes || glance_store 0.7.0 added the Python 3 support.
+
 
 +
| networking-l2gw || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/os-brick os-brick] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes || os-brick 0.3.2 added Python 3 support.
 
  
 +
| [https://pypi.org/project/vmware-nsx vmware-nsx] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||  style="background-color: lightgreen;" | All 3rd part CI jobs are running python 3  ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color:lightgreen;" | Yes ||
+
 
 +
| [https://pypi.org/project/vmware-nsxlib vmware-nsxlib] || style="background-color: green;color: white;" | '''Yes''' || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | tempest-api-vmware-nsxv3 is running python 3.5 and is voting ||
 
|-
 
|-
−
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | voting || style="background-color: lightgreen;" | Yes ||
+
 
 
|-
 
|-
 
|}
 
|}
  
−
=== Dependencies ===
+
Unsorted projects:
 +
* [https://wiki.openstack.org/wiki/App-Catalog App Catalog]
 +
* Chef OpenStack (Ruby)
 +
* [https://github.com/openstack/kolla/ Kolla]: ~2100 lines of Python, but not "tox -e py27" target
 +
* OpenStack Ansible (Python): Ansible playbooks and roles for deployment
 +
* Puppet OpenStack (Ruby)
 +
* TripleO
  
−
There are two ways to express dependencies that should differ between Python 2.x and 3.x. We currently support using multiple requirements files, and will shortly be supporting and recommending the use of environment markers. Recent releases of all the tools in our ecosystem are needed for environment markers.
+
== Python 2: Python 2.6 support dropped, Python 2.7 only ==
  
−
==== requirements-py3.txt ====
+
OpenStack Liberty targets Python 2.7 and 3.4.
 +
 
 +
Python 2.6 support is being dropped in OpenStack since OpenStack Juno for servers. Python 2.6 support is currently kept in Oslo libraries and clients. See [https://etherpad.openstack.org/p/juno-cross-project-future-of-python juno-cross-project-future-of-python] etherpad.
  
−
This approach generates a different requirements list depending on what Python version is running when the egg info is created (which goes into wheels and sdists). As a consequence this can't work with tox, publishing sdists to PyPI, or wheels. It is however useful for anything installing from git. To use it, create two requirements files:
+
Python 3.3 support is being dropped since OpenStack Liberty.
  
−
* requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
+
Python 2.6 is slowly being dropped in the whole OpenStack project, ex: [https://review.openstack.org/#/c/201295/ Remove p26 job from DIB].
−
* requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
 
−
* (same for test-requirements.txt)
 
  
−
You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:
+
Python 2.6 support will be removed in Oslo and Clients clients for OpenStack Mitaka: [http://markmail.org/message/oe6wewgvbvvyts73 Oslo libraries dropping python 2.6 compatability].
  
−
<pre>
+
== Port Python 2 code to Python 3 ==
−
...
 
−
[testenv:py33]
 
−
deps = -r{toxinidir}/requirements-py3.txt
 
−
      -r{toxinidir}/test-requirements-py3.txt
 
−
...
 
−
</pre>
 
  
−
==== Environment markers ====
+
OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. As we are targeting Python 3.4 and up, there is no need to avoid u'unicode' syntax. Do not use six.u('unicode').
  
−
Environment markers provide a mini-language to express when a dependency is relevant. We hope the next release of pbr will support them. Recent pip, setuptools and wheel versions do support them, but until pbr exports the requirements metadata with them intact and structured correctly for setuptools, we can't use them anywhere.
+
=== Before you begin ===
  
−
* PEP 0345 -- Metadata for Python Software Packages 1.2 (accepted): [https://www.python.org/dev/peps/pep-0345/#environment-markers Environment markers]
+
If you're doing development with Ubuntu/Debian (and not using devstack with the USE_PYTHON flag set), then you'll need the following packages installed to run the py34 tox unit test targets in the projects:
−
* PEP 0426 -- Metadata for Python Software Packages 2.0 (draft): [https://www.python.org/dev/peps/pep-0426/#environment-markers  Environment markers]
 
  
−
When placed in a requirements.txt file, the markers are interpreted directly by pip in `pip install -r requirements.txt`, and indirectly via setuptools when installing an sdist or wheel (including `pip install .`. For instance:
+
    sudo apt-get install python3.4 python3.4-dev
−
<pre>
 
−
six
 
−
futures; python_version < '3.2'
 
−
</pre>
 
  
−
See also:
+
=== sixer tool ===
−
* [https://review.openstack.org/#/c/184328/ oslo.db: Refactor deps to use extras and env markers] (require pip 7, not released yet)
 
−
* pip: [https://github.com/pypa/pip/pull/1472  Fix issue #1433: parse requirements in markers #1472]
 
−
* setuptools: pkg_resources doesn't understand environment markers: [https://bitbucket.org/pypa/wheel/issue/139/wheel-should-support-pep-426-environment pkg_resources.parse_requirements() raises ValueError]
 
−
* pbr: [https://review.openstack.org/#/c/181795/ Teach pbr to read extras and env markers], [https://review.openstack.org/#/c/165015/ Handle PEP426 markers] (abandoned)
 
−
* wheel: [https://bitbucket.org/pypa/wheel/issue/139/wheel-should-support-pep-426-environment wheel should support PEP 426 environment markers] (closed), wheel supports environment markers in extra requirements
 
−
* tox: [https://bitbucket.org/hpk42/tox/issue/236/tox-must-create-the-source-distribution tox must create the source distribution with the Python of the virtual environment], [https://bitbucket.org/hpk42/tox/pull-request/142/issue-236-subcommand_test-rebuilds-sdist/ Issue #236: subcommand_test() rebuilds sdist with venv python] (declined)
 
  
−
Oslo Messaging issue:
+
The [https://pypi.python.org/pypi/sixer sixer] tool helps to replace most basic patterns to add Python 3 compatibility and it respects OpenStack coding style.
−
* futures is not needed on Python 3, since concurrent.futures is now part of the Pyhon stdlib since Python 3.2
 
−
* [https://review.openstack.org/#/c/172135/ futures was added to requirements-py3.txt] because tox creates the source tarball using Python 2 which reads requirements.txt
 
−
* The correct fix would be to use environment markers on futures
 
  
−
==== Dependencies status ====
+
=== Common patterns ===
  
−
[https://caniusepython3.com/check/4fd5dda2-b1f1-4db4-a636-67cd3276cb6a Porting status] for [https://github.com/openstack/requirements/blob/master/global-requirements.txt global-requirement.txt].
+
* Replace "for key in dict.iterkeys()" with "for key in dict"
 +
* Replace dict.iteritems() with dict.items()
 +
* Replace dict.itervalues() with dict.values()
  
−
{|class="wikitable sortable"
+
Note: Replacing dict.iteritems()/.itervalues() with six.iteritems(dict)/six.itervalues(dict) was preferred in the past, but there was [http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html a discussion suggesting to avoid six for this]. The overhead of creating a temporary list on Python 2 is negligible.
−
|-
 
−
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
 
−
|-
 
  
−
| [https://pypi.python.org/pypi/boto boto] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32).
+
* Replace iterator.next() with next(iterator)
−
|-
+
* Replace basestring with six.string_types
 +
* Replace unicode with six.text_type
 +
* Replace (str, unicode) with six.string_types
 +
* Replace (int, long) with six.integer_types
 +
* Replace func.func_name with func.__name__
 +
* Replace exceptions.OSError with OSError and remove "import exceptions"
 +
* map() and filter() if a list is needed on Python 3:
 +
** Replace map(func, data) with [func(item) for item in data]
 +
** Replace filter(lambda obj: test(obj), data) with [obj for obj in data if test(obj)]
 +
** Note: Usually, tests fail because map() or filter() objects have no length
  
−
| [https://pypi.python.org/pypi/django-compressor django-compressor] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || Requirements upgraded: https://review.openstack.org/94357
+
=== Serialization: base64, JSON, etc. ===
−
|-
 
  
−
| [https://pypi.python.org/pypi/tablib tablib ] || style="background-color: orange;" | Yes but... || N/A || style="background-color: lightgreen;" | Yes || || setup.py installs tablib.packages.markup which is not compatible with Python 2. setup.py displays a warning on compiling the bytecode of markup.py and many others in the tablib/packages folder. Overall, the package cannot be built for Python 3 in distributions (zigo has issues to package it on Debian).
+
* [with oslo.serialization 1.10 or newer] Get the [http://docs.openstack.org/developer/oslo.serialization/api.html#module-oslo_serialization.base64 base64 module] from oslo_serialization (from oslo_serialization import base64) to get functions:
 +
** oslo_serialization.base64.decode_as_bytes(encoded)
 +
** oslo_serialization.base64.decode_as_text(encoded, encoding='utf-8')
 +
** oslo_serialization.base64.encode_as_bytes(s, encoding='utf-8')
 +
**  oslo_serialization.base64.encode_as_text(s, encoding='utf-8')
 +
* Replace text.encode('base64') and base64.b64encode(text) with:
 +
** base64.b64encode(text): only accept bytes and returns bytes,
 +
** or: oslo_serialization.base64.encode_as_bytes(text): accept bytes or Unicode and returns bytes
 +
** or: oslo_serialization.base64.encode_as_text(text): accept bytes and Unicode and returns Unicode
 +
** '''Warning:''' base64.encodestring(raw) adds a newline ("\n"), whereas encode_as_bytes() and encode_as_text() don't.
 +
* Replace raw.decode('base64') with:
 +
** base64.b64decode(raw): return bytes
 +
** oslo_serialization.base64.decode_as_bytes(encoded): accept bytes and Unicode, returns bytes
 +
** oslo_serialization.base64.decode_as_text(encoded): accept bytes and Unicode, returns Unicode
  
−
tablib is needed by cliff-tablib, itself needed by python-neutronclient, then neutronclient is itself needed by many other libs.
+
Hexadecimal:
−
|-
 
  
−
| [https://pypi.python.org/pypi/dnspython dnspython] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Must use the [https://pypi.python.org/pypi/dnspython3/ Python 3 version], see https://github.com/rthalley/dnspython/issues/60
+
* Replace raw.decode('hex') with binascii.unhexlify(raw)
−
|-
+
* Replace bytes.encode('hex') with binascii.hexlify(bytes)
  
−
| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
+
JSON:
−
|-
 
  
−
| [https://pypi.python.org/pypi/eventlet eventlet] || style="background-color:lightgreen;" | Yes || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || || eventlet 0.17.3 now fully support Python 3
+
* [with oslo.serialization 1.10 or newer] Replace json.dumps(obj) with [https://docs.openstack.org/oslo.serialization/latest/reference/index.html#oslo_serialization.jsonutils.dump_as_bytes oslo_serialization.jsonutils.dump_as_bytes(obj)]
−
|-
+
* Replace json.loads(obj) with oslo_serialization.jsonutils.loads(obj): it accepts bytes and Unicode, bytes is decoded from UTF-8. It avoids "if isinstance(obj, bytes): obj = obj.decode('utf-8')" which may require a second temporary variable.
  
−
| [https://pypi.python.org/pypi/jsonrpclib jsonrpclib] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || The project seems dead :(
+
=== contextlib.nested ===
−
|-
 
  
−
| [https://pypi.python.org/pypi/libvirt-python libvirt-python] || style="background-color: lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || ||
+
To replaced contextlib.nested is to use [https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack contextlib.ExitStack]. It's available on Python 2 using [https://contextlib2.readthedocs.org/en/latest/#contextlib2.ExitStack contextlib2.ExitStack]. For contextlib.nested, nova/test.py defines:
−
|-
 
  
−
| [https://pypi.python.org/pypi/mysql-python MySQL-Python] || style="background-color:orange;" | No, but mysqlclient or PyMySQL are compatible || N/A || style="background-color: orange;" | No || ||
+
if six.PY3:
−
Patches:
+
    @contextlib.contextmanager
−
* [https://review.openstack.org/#/c/179745/ Add mysqlclient dependency]
+
    def nested(*contexts):
 +
        with contextlib.ExitStack() as stack:
 +
            yield [stack.enter_context(c) for c in contexts]
 +
else:
 +
    nested = contextlib.nested
  
−
MySQL-Python: [https://github.com/farcepest/MySQLdb1/pulls 2 pending pull requests for Python 3].
+
Another option to replaced contextlib.nested is to use the @mock.patch decorator. Example with nested function:
  
−
INADA Naoki, the maintainer of PyMySQL, wrote [https://pypi.python.org/pypi/mysqlclient mysqlclient]: fork of MySQL-Python using libmysqlclient.so which is Python 3 compatible. He recommends to use mysqlclient instead of MySQL-python even on Python 2. mysqlclient fixes some bugs, build issues and it support Python 3. For example: support MariaDB's libmysqlclient.so, support microsecond in TIME column.
+
def test_thing(self):
 +
    @mock.patch(...)
 +
    @mock.patch(...)
 +
    @mock.patch(...)
 +
    def do_test(...):
 +
        ...
  
−
PyMySQL is compatible with Python 3: see [https://wiki.openstack.org/wiki/PyMySQL_evaluation PyMySQL evaluation] discussion to eventually replace mysql-python with PyMySQL in OpenStack.
+
    do_test()
−
|-
 
  
−
| [https://pypi.python.org/pypi/netifaces netifaces] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Patch sent by Victor Stinner (in private): [https://bitbucket.org/haypo/misc/src/tip/openstack/netifaces_python3.patch netifaces_python3.patch], Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 .
+
More options:
−
|-
+
* Mock provides a context that patches multiple things so that no nesting is needed: [https://docs.python.org/dev/library/unittest.mock.html#patch-multiple mock.patch.multiple()]
 +
* oslotest provides fixtures for mock, so you don't need a context: [http://docs.openstack.org/developer/oslotest/api.html#module-oslotest.mockpatch oslotest.mockpatch].
  
−
| [https://pypi.python.org/pypi/nose-exclude nose-exclude] || style="background-color: lightgreen;" | Yes || N/A || style="background-color: red;" | No || || Python 3 is supported since nose-exclude 0.4.
+
=== oslo_utils.encodeutils.to_utf8 ===
−
|-
 
  
−
| [https://pypi.python.org/pypi/nosehtmloutput nosehtmloutput] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
+
oslo.utils 3.5 has an oslo_utils.encodeutils.to_utf8() function to encode Unicode to UTF-8 and return bytes unchanged.
−
*  nose-exclude (tests only)
 
−
*  openstack.nose-plugin
 
−
||
 
−
*  https://bugs.launchpad.net/ubuntu/+source/python-nosehtmloutput/+bug/1287247
 
−
*  https://review.openstack.org/#/c/80956/
 
−
|-
 
−
| [https://pypi.python.org/pypi/nosexcover nosexcover] || style="background-color:lightgreen;" | No || N/A || style="background-color: lightgreen;" | On PyPI || || Python 3 support since 1.0.9
 
−
|-
 
−
| [https://pypi.python.org/pypi/openstack.nose-plugin openstack.nose-plugin] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
 
−
|-
 
  
−
| [https://pypi.python.org/pypi/pam pam] || style="background-color:orange;" | No for pam, Yes for simplepam || NA || style="background-color: red;" | No || || The fork [https://pypi.python.org/pypi/simplepam simplepam] works on Python 2 and 3
+
=== bytes.decode and unicode.encode ===
−
|-
 
  
−
| [https://pypi.python.org/pypi/paramiko paramiko] ||  style="background-color:lightgreen;" | Yes  || N/A || style="background-color: lightgreen;" | On PyPI || || Requirements upgraded: https://review.openstack.org/#/c/81132/
+
Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.
−
|-
 
  
−
| [https://pypi.python.org/pypi/paste paste] || style="background-color:lightgreen;" | Yes || style="background-color: lightgrey;" | N/A || style="background-color: lightgreen;" | Yes || || Compatible with Python 3 since Paste 2.0
+
Don't write <code>data.decode()</code> or <code>text.encode()</code> without parameter, because you will use a different encoding on Python 2 and Python 3.
−
|-
 
  
−
| [https://pypi.python.org/pypi/pycadf pycadf] || style="background-color:lightgreen;" | Yes || style="background-color:lightgreen;" | Yes || style="background-color:lightgreen;" | Yes || ||
+
Use an explicit encoding instead. Example: <code>data.decode('utf-8')</code> or <code>text.encode('utf-8')</code>. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).
−
|-
 
  
−
| [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The project seems dead. See [https://pypi.python.org/pypi/ldap3 ldap3] which is Python 3 compatible (license: LGPL, one of the optional deps is BSD-4-Clause).
+
=== safe_decode ===
−
|-
 
  
−
| [https://pypi.python.org/pypi/python-memcached python-memcached] || style="background-color:lightgreen;" | Yes || style="background-color:lightgreen;" | N/A || style="background-color:lightgreen;" | Yes || ||
+
Olso Incubator has a function '''safe_decode()''' which can be used to decode a bytes string and pass text strings unchanged.
−
python-memcached 1.56 is compatible with Python 3, but there are still some bugs: [https://github.com/linsomniac/python-memcached/pull/73  More Python 3 fixes] (not part of python-memcached 1.57). These bugs impact running keystonemiddleware tests when the memcached server is not running. It can also impact a setup with multiple memcached servers.
 
−
|-
 
  
−
| [https://pypi.python.org/pypi/qpid-python qpid-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||  || Required if using oslo.messaging's qpid backend. That is the old pure python amqp 0-10 client library.  That only speaks 0-10 and works with qpidd broker.  It is used by the impl_qpid.py driver in oslo.messaging. That client is legacy - it isn't being ported to use AMQP 1.0.
+
The default encoding is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
−
|-
+
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
 +
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
  
−
| [https://pypi.python.org/pypi/pyngus pyngus] || style="background-color: lightgreen;" | Yes || style="background-color: lightgray;" | N/A || style="background-color: red;" | No ||  ||
+
It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
−
pyngus 2.0 supports Python 3.
 
  
−
pyngus is required if using oslo.messaging's AMQP 1.0 driver. pyngus is a wrapper for proton and proton has not been ported to python 3 yet: see [https://issues.apache.org/jira/browse/PROTON-490 Proton issue #490].
+
Safe usage:
 +
* <code>safe_decode(data, 'utf-8')</code>: decode bytes from UTF-8 or returns data unchanged if it's already a text string
  
−
[https://pypi.python.org/pypi/python-qpid-proton Proton] is a C library with various bindings in other languages. The language bindings are auto generated via the SWIG tool, plus a the python wrapper bits. All the python stuff is found under proton-c/bindings/python in the [https://git-wip-us.apache.org/repos/asf/qpid-proton.git git tree]. The first bit of work is to refactor the SWIG file cproton.i to work for both Python 2 and 3.
+
Unsafe usage:
−
|-
+
* <code>safe_decode(data)</code>
  
−
| [https://pypi.python.org/pypi/rtslib-fb rtslib-fb] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
+
By default, the decoder is strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.
−
|-
 
−
| [https://pypi.python.org/pypi/Routes Routes] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || || Routes 2.2 works Python 2 and 3 using a single code base.
 
−
|-
 
−
| [https://pypi.python.org/pypi/sphinxcontrib-docbookrestapi sphinxcontrib-docbookrestapi] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI || ||
 
−
|-
 
−
| [https://pypi.python.org/pypi/sphinxcontrib-httpdomain sphinxcontrib-httpdomain] || style="background-color:lightgreen;" | Yes  || N/A || style="background-color: red;" | No || ||
 
−
|-
 
−
| [https://pypi.python.org/pypi/sphinxcontrib-pecanwsme sphinxcontrib-pecanwsme] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
 
−
|-
 
−
| [https://pypi.python.org/pypi/thrift thrift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
 
−
|-
 
−
| [https://pypi.python.org/pypi/websockify websockify] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | N/A || style="background-color: lightgreen;" | Yes || || python3 works with the 0.7.0 version released on Jul 23, 2015
 
−
|-
 
  
−
|}
+
=== safe_encode ===
  
−
=== OpenStack applications ===
+
Olso Incubator has a function '''safe_encode()''' which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
 +
* <code>safe_encode(text)</code> encodes text to the output encoding
 +
* <code>safe_encode(bytes)</code> may decode the string and then reencode to a different encoding if input and output encodings are different
  
−
List of "official" OpenStack applications: [http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml projects.yaml].
+
The default input encoding (<code>incomding</code> parameter) is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
 +
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
 +
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)
  
−
{| class="wikitable sortable"
+
The default output encoding (<code>encoding</code> parameter) is UTF-8.
−
|-
 
−
! Project !! Python 3 compatibility !! py34 gate !! Comments
 
−
|-
 
  
−
| [https://pypi.python.org/pypi/aodh aodh] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||
+
It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".
−
|-
 
  
−
| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||
+
Safe usage:
−
|-
+
* <code>safe_encode(data, incoming='utf-8')</code>: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)
  
−
| [https://pypi.python.org/pypi/gnocchi gnocchi] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||
+
Unsafe usage:
−
|-
+
* <code>safe_encode(data)</code>
−
| [https://pypi.python.org/pypi/ironic ironic] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||
 
−
[http://lists.openstack.org/pipermail/openstack-dev/2015-May/064049.html Python 3.4 unit tests are now being run for openstack/ironic.  The unit tests are a voting job.]
 
−
Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4: [https://review.openstack.org/#/c/156192/ Run tests in py34 environment]
 
−
|-
 
  
−
| [https://pypi.python.org/pypi/rally Rally] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting || A big thank to Andrey Kurilin for a lot of work in this direction
+
Example:
−
|-
+
* <code>safe_encode(b'\xe9', incoming='latin-1')</code> returns <code>b'\xc3\xa9'</code>.
  
−
| [https://pypi.python.org//pypi/sahara sahara] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | voting ||
+
By default, the encoder and the decoder are strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: <code>safe_encode(b'[\xff]', incoming='ascii', errors='ignore')</code> returns <code>b'[]'</code>.
−
|-
 
  
−
| [https://pypi.python.org/pypi/cinder cinder] || style="background-color: orange;" | work in progress || style="background-color: lightgreen;" | voting ||
+
=== logging module and format exceptions ===
−
''Update: We have passed feature freeze, should hold this for Mitaka.''
 
  
−
[https://blueprints.launchpad.net/cinder/+spec/cinder-python3 cinder-python3] blueprint (written by Victor Stinner) has been accepted for Liberty.
+
The <code>exception_to_unicode(exc)</code> function of oslo_utils.encodeutils is the recommanded way to format an exception to Unicode. This function works on Python 2 and Python 3 and it should avoid mojibake is most cases.
  
−
Patches: [https://review.openstack.org/#/q/project:openstack/cinder+topic:bp/cinder-python3,n,z bp/cinder-python3 topic].
+
On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs <code>b'hello'</code> instead of <code>'hello'</code>.
−
|-
 
  
 +
There is no clear rule for format exceptions yet. There are different choices depending on the project:
  
−
| [https://pypi.python.org/pypi/glance glance] || style="background-color: orange;" | work in progress || style="background-color: lightgreen;" | voting ||
+
* <code>str(exc)</code>: native string, so use bytes on Python 2
−
Victor Stinner is working on porting Glance to Python3.
+
* <code>six.text_type(exc)</code>: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: <code>unicode(Exception("nonascii:\xe9"))</code>.
 +
* <code>six.u(str(exc))</code>: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: <code>unicode(str(Exception("\xff")))</code>
 +
* <code>LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})</code>
  
−
Patches: [https://review.openstack.org/#/q/status:open+project:openstack/glance+branch:master+topic:py3,n,z Patches for glance (topic: py3)].
+
Since logging functions expect text strings on Python 3, logged exceptions should be formatted using <code>str(exc)</code>. Example: <code>LOG.debug(str(exc))</code>.
−
|-
 
  
−
| [https://pypi.python.org/pypi/horizon horizon] || style="background-color: orange;" | work in progress || style="background-color: lightgreen;" | voting ||
+
=== HTTP ===
−
'''Update''': David Lyle blocked Python 3 patches until the next M cycle to focus on stabilize
 
  
−
George Peristerakis wrote [https://blueprints.launchpad.net/horizon/+spec/porting-python3 Make Horizon python3.4 compatible] in May 2015, the blueprint was approved.
+
The HTTP protocol is based on '''bytes''':
  
−
Patches: [https://review.openstack.org/#/q/topic:bp/porting-python3+project:openstack/horizon,n,z bp/porting-python3 topic].
+
* HTTP body contains '''bytes'''. For example, use io.BytesIO for a stream storing an HTTP body.
 +
* HTTPConnection.getresponse().read() returns '''bytes''' (in Python 3, '''str''' which is bytes in Python 2)
 +
* On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
 +
* It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and [http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding  HTTP header should use what character encoding?]
  
−
Some tests are still blocked a [https://review.openstack.org/#/c/217687/ Python 3 bug] in django_openstack_auth 1.4.0. The fix is already merged, but a release is still needed.
+
=== References to port Python 2 code to Python 3 ===
 +
* [http://python3porting.com/ Porting to Python 3 Book] by Lennart Regebro, especially the [http://python3porting.com/differences.html Language differences and workarounds].
 +
* [http://docs.python.org/dev/howto/pyporting.html HOWTO: Porting Python 2 Code to Python 3] by Brett Cannon
 +
* [https://wiki.python.org/moin/PortingPythonToPy3k Porting Python Code to 3.x]
 +
* [http://code.google.com/p/python-incompatibility/  python-incompatibility]: Demonstrates incompatibilities between Python versions.
  
−
babel 2.0: openstack_dashboard.dashboards.settings.user.tests.UserSettingsTest fails with a TypeError on babel.core.get_global(): it's a bug in babel, see [https://github.com/mitsuhiko/babel/issues/174  Locale.parse('zh_CN') raises a TypeError on Babel 2.0 with Python 3.3 and 3.4] and the fix [https://github.com/mitsuhiko/babel/pull/188  Avoid incompatible *.dat files between Python 2 and 3]. See the bug [https://github.com/mitsuhiko/babel/issues/228  Need a new release to fix Python 3 compatibility].
+
=== Common pitfalls ===
−
|-
 
  
−
| [https://pypi.python.org/pypi/heat heat] || style="background-color: orange;" | work in progress || style="background-color: lightgreen;" | voting ||
+
==== What is a string ? ====
−
[http://specs.openstack.org/openstack/heat-specs/specs/liberty/heat-python34-support.html Python34 Support] spec (by Sirushti Murugesan) accepted for Liberty.
+
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:
  
−
Patches: [https://review.openstack.org/#/q/topic:bp/heat-python34-support,n,z bp/heat-python34-support topic].
+
Python 2:
−
|-
+
    >>> type('foo')
 +
    <type 'str'>
 +
    >>> type(u'foo')
 +
    <type 'unicode'>
 +
    >>> type(b'foo')
 +
    <type 'str'>
 +
    >>> isinstance('foo', six.text_type)
 +
    False
 +
    >>> isinstance(u'foo', six.text_type)
 +
    True
 +
    >>> bytes is str
 +
    True
 +
    >>> b'foo'[0]
 +
    'f'
  
−
| [https://pypi.python.org/pypi/keystone keystone] || style="background-color: orange;" | work in progress || style="background-color: lightgreen;" | voting ||
+
Python 3:
−
The spec [http://specs.openstack.org/openstack/keystone-specs/specs/liberty/python3.html Add spec for python-3 compatibility] (by Morgan Fainberg) was accepted for Liberty. See also the blueprint [https://launchpad.net/keystone/+spec/python3 python3] started by Dolph Mathews on 2014-07-22; David Stanek is working on it.
+
    >>> type('foo')
 +
    <class 'str'>
 +
    >>> type(u'foo')
 +
    <class 'str'>
 +
    >>> type(b'foo')
 +
    <class 'bytes'>
 +
    >>> isinstance('foo', six.text_type)
 +
    True
 +
    >>> isinstance(b'foo', six.text_type)
 +
    False
 +
    >>> bytes is str
 +
    False
 +
    >>> b'foo'[0]
 +
    102
  
−
Patches: [https://review.openstack.org/#/q/project:openstack/keystone+branch:master+topic:bp/python3,n,z bp/python3 topic].
+
==== tox/testr error: db type could not be determined ====
  
−
[https://github.com/rohe/pysaml2/ pysaml2] 3.0 now supports Python 3.
+
The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.
  
−
Aslo blocked by [https://pypi.python.org/pypi/ldappool ldappool] and [https://pypi.python.org/pypi/python-ldap python-ldap]. [https://github.com/pyldap/pyldap/ pyldap] is a fork of python-ldap which adds Python 3 support. python-ldap may be replaced with [https://pypi.python.org/pypi/ldap3 ldap3], but it requires more work.
+
Workaround: "rm -rf .testrepository/" and then run "tox -e py34" before running "tox -e py27". You only have to do this once. The problem only occurs with "tox -e py34" when .testrepository/ was created by Python 2.
−
|-
 
  
−
| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:orange;" | work in progress || style="background-color: lightgreen;" | voting ||
+
=== Enable Python 3 in DevStack ===
−
[https://review.openstack.org/#/c/172962/ Porting to Python 3] spec (by Cyril Roelandt) accepted for Liberty, port in progress. Patches:
 
−
* [https://review.openstack.org/#/q/project:openstack/neutron+branch:master+topic:bp/neutron-python3,n,z Patches of the blueprint neutron-python3]
 
−
|-
 
  
 +
* Fedora::
  
−
| [https://pypi.python.org/pypi/nova nova] || style="background-color: orange;" | work in progress || style="background-color: lightgreen;" | voting ||
+
sudo dnf install python3-devel
−
''Update:"Given all the current patches have merged, and the agreement to timebox this effort, lets call this the end of things for liberty.
+
sudo python3 -m pip install python-memcached  # needed by keystonemiddleware
−
--johnthetubaguy 5th August 2015".''
 
  
−
[http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/adding-python34-support-to-nova.html Adding Python 3.4 support to Nova] spec (by Victor Stinner) accepted for Liberty. Port in progress.
+
* localrc: add the following line
  
−
Patches:
+
USE_PYTHON3=True
−
* [https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/nova-python3,n,z master (bp/nova-python3)].
 
  
−
See also [https://etherpad.openstack.org/p/liberty-nova-priorities-tracking Py3.4 Compatibility in Liberty nova priorities tracking].
+
See [https://review.openstack.org/#/c/414176/ Switch to python 3.5].
−
|-
 
  
−
| [https://github.com/openstack/manila manila] || style="background-color: orange;" | work in progress || style="background-color: orange;" | non-voting ||
+
=== Functional and Integration Tests ===
−
[https://blueprints.launchpad.net/manila/+spec/py3-compatibility py3-compatibility] blueprint.
 
  
−
non-voting py34 CI job runs in 'check' pipeline for each Manila gerrit commit. It fails because py3 support fixes are in progress [L-2].
+
Etherpad: https://etherpad.openstack.org/p/support-python3.5-functional-tests
−
Patches: [https://review.openstack.org/#/q/project:openstack/manila+topic:bp/py3-compatibility,n,z bp/py3-compatibility topic].
 
−
|-
 
  
−
| [https://pypi.python.org/pypi/swift swift] || style="background-color: orange;" | work in progress || style="background-color: orange;" | non voting ||
+
Doug Hellmann wrote a cross-project specification to run integration tests on Python 3: [http://specs.openstack.org/openstack/openstack-specs/specs/enable-python-3-int-func-tests.html Enabling Python 3 for Application Integration Tests] (spec approved for Liberty).
−
September 2015: Development now focused on fixing last erasure code (EC) bugs, Python 3 is out of the scope of this development cycle (september).
 
  
−
swift now depends on six. (Jaivish Kothari is also working on this)
+
Patch for DevStack: [https://review.openstack.org/#/c/181165/ Enable optional Python 3 support].
  
−
Fixing "tox -e py34" requires:
+
Projects wanting to run their existing functional test suite against their services on Python 3 should:-
−
* pbr >= 1.0 but Swift currently uses hacking 0.8 which is incompatible with pbr 1.0
+
* Add the Python3.4 trove classifier to the setup.cfg configuration file of the respective service.
−
* [https://review.openstack.org/#/c/205977/ Update hacking to 0.10.0]: hacking 0.10 is compatible with pbr 1.0
+
* set USE_PYTHON3=True in Devstack's localrc config file before running stack.sh after [https://review.openstack.org/#/c/181165/ the DevStack change] is merged or cherry-pick until then.
−
* PyEClib 1.0.8: [https://bitbucket.org/kmgreen2/pyeclib/issues/69/need-a-release-108-with-python-3-fixes waiting for the release 1.0.8]
+
* set basepython = python3.5 in tox for the necessary functional/integration tests environment.
−
* [https://review.openstack.org/#/c/196633/ dnspython3 dependency] in global requirements (merged!)
 
−
* patch: [https://review.openstack.org/#/c/199034/ fix tox -e py34]
 
  
−
Patches:
+
Then run tox -e<your functional test suite name>. Example:- tox -efunctional to run the functional the tests of a certain project.
−
* [https://review.openstack.org/#/q/project:openstack/swift+branch:master+topic:py3,n,z Patches with the topic py3]
 
  
−
Dependencies:
+
Once the functional test suite is reliably passing against the service in a python 3.5 env, the trove classifiers must be updated along with a new gate job being added to the project.
−
* dnspython: use dnspython3 on Python 3 ([https://review.openstack.org/#/c/200416/ not in global requirements yet])
 
−
|-
 
  
−
| [https://pypi.python.org/pypi/designate designate] || style="background-color: orange;" | work in progress || style="background-color: red;" | No || Pradeep Kumar Singh is working on the Python 3 port.
+
Sirushti Murugesan's attempt on Heat: https://review.openstack.org/#/c/188033/
−
|-
 
  
−
| [https://pypi.python.org/pypi/barbican barbican] || style="background-color: orange;" | work in progress || style="background-color: red;" | No || Pradeep Kumar Singh proposed the blueprint [https://blueprints.launchpad.net/barbican/+spec/barbican-py3 barbican-py3] for Liberty.
+
=== Python 3.5 ===
  
−
Patches: [https://review.openstack.org/#/q/status:open+project:openstack/barbican+branch:master+topic:bp/barbican-py3,n,z search for the bp/barbican-py3 topic].
+
* project-config: [https://review.openstack.org/#/c/336272/ Add python35 jobs]
−
|-
+
* nova: [https://review.openstack.org/#/c/336432/ Add a py35 environment to tox]
 +
* Neutron issues: https://bugs.launchpad.net/neutron/+bug/1559191
 +
* There is no Jenkins job using Python 3.5 yet
 +
* /usr/bin/python3 is Python 3.5 in Fedora 24, Ubuntu 16.04 LTS (Xenial Xerus) and Debian Stretch (Testing)
  
−
|}
+
=== Python 3.6 status ===
  
−
=== Functional tests ===
+
Python 3.6 is scheduled to be the default/only version of Python available in Ubuntu 18.04 LTS (Bionic Beaver) and Debian 10 (Buster).
  
−
Doug Hellmann wrote a cross-project specification to run integration tests on Python 3: [https://review.openstack.org/#/c/177375/ Enabling Python 3 for Application Integration Tests] (spec approved for Liberty).
+
Known issues:
  
−
Patch for DevStack: [https://review.openstack.org/#/c/181165/ Enable optional Python 3 support].
+
* eventlet<0.21.0 is broken with Python 3.6, see https://github.com/eventlet/eventlet/issues/371. There is also https://github.com/eventlet/eventlet/issues/401 which is fixed in 0.22.0, so it might seem feasible to upgrade to eventlet-0.22.1 once we can do proper testing.
 +
* Horizon explicitly only claims support for Python 3.5, when running devstack with Python 3.6 one needs to set:
 +
    ENABLED_PYTHON3_PACKAGES=horizon
  
 
== Status of Python 3 in Linux distributions ==
 
== Status of Python 3 in Linux distributions ==
  
 
* ArchLinux already [https://www.archlinux.org/news/python-is-now-python-3 switched to Python 3 by default] in 2011
 
* ArchLinux already [https://www.archlinux.org/news/python-is-now-python-3 switched to Python 3 by default] in 2011
−
* Ubuntu wants to remove Python 2 from the default installation: see [http://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-16.04-Python-Plans Ubuntu Plans For Python 3 By Default For Ubuntu 16.04 LTS]
+
* Ubuntu wants to remove Python 2 from the default installation for Ubuntu 18.04 LTS: see [https://wiki.ubuntu.com/Python (Python) Plans for 18.04].
 
* Fedora schedules the switch in Fedora 23 (october 2015): [https://fedoraproject.org/wiki/Changes/Python_3_as_Default Python 3 as the Default Implementation]
 
* Fedora schedules the switch in Fedora 23 (october 2015): [https://fedoraproject.org/wiki/Changes/Python_3_as_Default Python 3 as the Default Implementation]
 +
** [http://fedora.portingdb.xyz/ Python 3 Porting Database for Fedora] and [http://fedora.portingdb.xyz/history/ history of packages tracked by the Fedora portingdb]
 +
** [https://lwn.net/Articles/690676/ Python 3 in Fedora] (Python 3 in Fedora)
 
* RHEL: [https://www.softwarecollections.org/en/scls/rhscl/python33/ Python 3.3] and [https://www.softwarecollections.org/en/scls/rhscl/rh-python34/ Python 3.4] are available on RHEL6 and RHEL7 using SCL.
 
* RHEL: [https://www.softwarecollections.org/en/scls/rhscl/python33/ Python 3.3] and [https://www.softwarecollections.org/en/scls/rhscl/rh-python34/ Python 3.4] are available on RHEL6 and RHEL7 using SCL.
 
* CentOS: [https://www.softwarecollections.org/en/scls/rhscl/python33/ Python 3.3] and [https://www.softwarecollections.org/en/scls/rhscl/rh-python34/ Python 3.4] are available on CentOS 6 and 7 using SCL (no Red Hat subscription needed, CentOS has its own flavor of SCL)
 
* CentOS: [https://www.softwarecollections.org/en/scls/rhscl/python33/ Python 3.3] and [https://www.softwarecollections.org/en/scls/rhscl/rh-python34/ Python 3.4] are available on CentOS 6 and 7 using SCL (no Red Hat subscription needed, CentOS has its own flavor of SCL)
−
* In Debian, the plan is to deprecate Python 2 in Stretch (aka: Debian 9, the next Stable Debian after Jessie) and to completely remove Python 2 for the Buster release of Debian (aka: Debian 10, to be release in approximatively 2019).
+
* In Debian, the plan is to deprecate Python 2 in Stretch (aka: Debian 9, the next Stable Debian after Jessie) and to completely remove Python 2 for the Buster release of Debian (aka: Debian 10, to be released in approximatively 2019).
 +
 
 +
Check the [https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions Project Testing Interface] for updates about the platforms each release is tested on.
 +
 
 +
== Status of Python 2 Support Being Dropped in Dependencies ==
 +
 
 +
* Django 2.0 will not include Python 2 support (23 Jan 2017 we use Django>=1.8,<1.9)
 +
** https://github.com/django/django/pull/7867/files
 +
** https://github.com/django/django/pull/7871/files
 +
* Pylons/Pyramid are also considering dropping Python 2
 +
** https://github.com/Pylons/pyramid/issues/2903
 +
** This is the same team that maintains webob, but it isn't clear if they intend to also drop support there.
 +
 
 +
[https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html OpenStack plans to start dropping Python 2 support at the start of the U cycle]
  
−
== Reports at OpenStack Summits ==
+
== Progress reports, meetings and sprints ==
  
 +
* Pike PTG: https://etherpad.openstack.org/p/ptg-pike-python35
 +
* Ocata Summit
 +
** https://etherpad.openstack.org/p/ocata-python-3
 +
* openstack-dev: [http://lists.openstack.org/pipermail/openstack-dev/2016-June/097930.html Status of the OpenStack port to Python 3] (Jun 22, 2016). 80 projects/83 (96%) have unit tests passing on Python 3, TODO (3): Nova, Swift and Trove.
 +
* Mitaka Summit
 +
** Cross-project session: Python 3 was proposed, https://etherpad.openstack.org/p/mitaka-cross-project-session-planning
 
* Liberty Summit
 
* Liberty Summit
 
** Cross-project session on enabling app integration testing: https://etherpad.openstack.org/p/liberty-cross-project-python3
 
** Cross-project session on enabling app integration testing: https://etherpad.openstack.org/p/liberty-cross-project-python3
Line 679: Line 868:
 
== Articles ==
 
== Articles ==
  
 +
* [http://blogs.rdoproject.org/7894/status-of-python-3-in-openstack-mitaka Status of Python 3 in OpenStack Mitaka] (Victor Stinner, March 2016)
 
* [http://techs.enovance.com/7807/python-3-status-openstack-liberty Python 3 Status in OpenStack Liberty] (Cyril Roelandt and Victor Stinner, September 2015)
 
* [http://techs.enovance.com/7807/python-3-status-openstack-liberty Python 3 Status in OpenStack Liberty] (Cyril Roelandt and Victor Stinner, September 2015)
 
* [http://techs.enovance.com/6722/status-of-the-openstack-port-to-python-3-2 Status of the OpenStack port to Python 3] (Cyril Roelandt, February 2014)
 
* [http://techs.enovance.com/6722/status-of-the-openstack-port-to-python-3-2 Status of the OpenStack port to Python 3] (Cyril Roelandt, February 2014)

Latest revision as of 02:42, 1 May 2020

This page tracks the progress of Python 3 effort porting for OpenStack.

IRC: #openstack-dev

Join the #openstack-dev IRC channel on the Freenode network to discuss Python 3.

Current Status

Since Ussuri, OpenStack is Python3 only except Swift and Storlets. Python2.7 support and testing has been removed from all other projects. For more details, please refer to Drop py2.7 community goal

Release Python Versions Notes
V python3.6 python3.7 python3.8 [1]
Ussuri (Dropped Python 2.7) python3.6 python3.7 [2] In Ussuri, OpenStack is python3-only except Swift & Storlets. Refer to the drop Python 2.7 goal
Train python2.7 python3.6 python3.7 [3] Train is the last release where python2.7 testing was required.

Python 3

Why should OpenStack move to Python 3 right now?

Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.


Python 3 Status of OpenStack projects

Common Libraries (Oslo Projects)

See programs.yaml for the list of Common Libraries.

Project Python 3 Only (Ussuri onwards) Python 3 compatibility Comment
automaton Yes Yes
castellan Yes Yes
cliff Yes Yes
debtcollector Yes Yes
futurist Yes Yes
oslo.cache Yes Yes
oslo.concurrency Yes Yes
oslo.config Yes Yes
oslo.context Yes Yes
oslo.db Yes Yes
oslo.i18n Yes Yes
oslo.log Yes Yes
oslo.messaging Yes Yes
oslo.middleware Yes Yes
oslo.policy Yes Yes
oslo.privsep Yes Yes
oslo.reports Yes Yes
oslo.rootwrap Yes Yes
oslo.serialization Yes Yes
oslo.service Yes Yes
oslotest Yes Yes
oslo.versionedobjects Yes Yes
oslo.vmware Yes Yes
oslo.utils Yes Yes
pylockfile NA Yes Project is deprecated.
stevedore Yes Yes
taskflow Yes Yes
tooz Yes Yes

Development tools

Project Python 3 Only (Ussuri onwards) Python 3 compatibility Comment
cookiecutter Yes yes
hacking Yes yes
pbr No yes pbr is not capped so we need to keep support for py2.7 code on safer side.
stackforge/python-jenkins No yes
openstack-infra/jenkins-job-builder No partial https://review.openstack.org/172238

OpenStack clients

Project Python 3 Only (Ussuri onwards) Python 3 compatibility CI tests running? Python 3 classifiers ? Blocked by Comment
keystonemiddleware Yes Yes Voting On PyPI
python-barbicanclient No Yes Voting On PyPI https://review.opendev.org/#/c/699096/
python-ceilometerclient Yes Yes Voting On PyPI
python-cinderclient Yes Yes Voting On PyPI
python-fuelclient N/A Yes voting On PyPI This project is no longer maintained.
python-glanceclient Yes Yes Voting On PyPI
python-heatclient Yes Yes Voting On PyPI
python-ironicclient Yes Yes Voting On PyPI
python-keystoneclient Yes Yes Voting On PyPI
python-manilaclient Yes Yes Voting On PyPI
python-zaqarclient Yes Yes Voting On PyPI
python-masakariiclient Yes Yes Voting On PyPI
python-monascaclient Yes Yes Voting On PyPI
python-novaclient Yes Yes Voting On PyPI
python-neutronclient Yes Yes Voting On PyPI
python-octaviaclient Yes Yes Voting On PyPI
python-openstackclient Yes Yes Voting On PyPI As of 0.9
python-saharaclient Yes Yes Voting On PyPI
python-searchlightclient Yes Yes Voting On PyPI
python-senlinclient Yes Yes Voting On PyPI
python-swiftclient No Yes Voting On PyPI Swift is keeping python2 support
python-tuskarclient N/A Yes Voting On PyPI This project is no longer maintained.
python-troveclient Yes Yes Voting On PyPI
python-watcherclient Yes Yes Voting On PyPI
python-designateclient Yes Yes Voting On PyPI

OpenStack Libraries

Project Python 3 Only (Ussuri onwards) Python 3 compatibility CI tests running? Python 3 classifiers ? Comment
blazar-nova Yes Yes voting Yes
ceilometermiddleware Yes Yes voting Yes
django_openstack_auth N/A Yes voting Yes

The code has been merged into openstack/horizon.

glance_store Yes Yes voting Yes
heat-translator Yes Yes voting Yes
ironic-lib Yes Yes voting Yes
keystoneauth Yes Yes voting Yes
keystonemiddleware Yes Yes voting Yes
ldappool Yes Yes voting Yes
monasca-statsd Yes Yes voting Yes
mistral-lib Yes Yes voting Yes
neutron-lib Yes Yes voting Yes
octavia-lib Yes Yes voting Yes
os-brick Yes Yes voting Yes
os-client-config Yes Yes voting Yes
os-traits Yes Yes voting Yes
os-vif Yes Yes voting Yes
os-win Yes Yes voting Yes
osc-lib Yes Yes voting Yes
osc-placement Yes Yes voting Yes
pycadf Yes Yes voting Yes
shade No Yes voting Yes shade is on extended maintainance.
sqlalchemy-migrate N/A Yes voting Yes This project is not maintained under OpenStack
sushy Yes Yes voting Yes
taskflow Yes Yes voting Yes
tosca-parser Yes Yes voting Yes

Dependencies: Environment markers

Example to not install the "futures" dependency on Python 3.3 and newer: 

futures; python_version < '3.2'

OpenStack applications (tc:approved-release)

OpenStack applications approved by the OpenStack Technical Committee.

Project Python 3 Only (Ussuri onwards) Unit tests pass on Python 3 py35 gate functional tests Comments
ceilometer Yes Yes voting
cinder Yes Yes voting integrated-gate-py35

cinder-python3 blueprint (written by Victor Stinner) has been accepted for Liberty. Patches: bp/cinder-python3 topic. Note: Commit message must use "Partial-Implements: blueprint cinder-python3" to be linked to the blueprint. Kendall Nelson, Ivan Kolodyazhny (core) and Eric Harney (core) prefer this syntax: see COMMIT_MSG.

designate Yes Yes voting designate-bind9-py36 designate-bind9-py35 Relies on oslo.service WSGI code that does not work under Python 3 with SSL. Functional tempest tests are ran on both of these jobs. [4]
glance Yes Yes voting integrated-gate-py35 Relies on oslo.service WSGI code that does not work under Python 3. [5]

Victor Stinner is working on porting Glance to Python3. Patches: Patches for glance (topic: py3).

heat Yes Yes voting heat-functional-convg-mysql-lbaasv2-py35

Sirushti Murugesan wrote and implemented the spec Python34 Support (which was accepted for Liberty). Mitaka version will fully support Python 3. Great job Sirushti Murugesan who did almost all the work!

horizon Yes Yes voting horizon-openstack-tox-python3-django111,horizon-integration-tests, horizon-dsvm-tempest-plugin

George Peristerakis wrote Make Horizon python3.4 compatible in May 2015, the blueprint was approved. Patches: bp/porting-python3 topic.

ironic Yes Yes voting ironic-tempest-dsvm-ipa-partition-pxe_ipmitool-tinyipa-python3

Python 3.4 unit tests are now being run for openstack/ironic. The unit tests are a voting job. Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4: Run tests in py34 environment

keystone Yes Yes voting integrated-gate-py35

The move to pyldap and new ldappool were major hurdles solved in Newton. Great work by the Keystone team to get the last things through the gate in Newton.

neutron Yes Yes voting integrated-gate-py35

Porting to Python 3 spec (by Cyril Roelandt) accepted for Liberty, port in progress. Patches of the blueprint neutron-python3

neutron-lbaas N/A Yes voting neutron-lbaasv2-dsvm-py3x-api and neutron-lbaasv2-dsvm-py3x-api-namespace This project is no longer maintained.
neutron-fwaas Yes Yes voting neutron-fwaas-networking-midonet-cross-py35 (non-voting)
neutron-vpnaas Yes Yes voting
nova Yes Yes voting integrated-gate-py35

Status at 2019-01-24: Unit tests all running and voting under Python 3 via the use of the mox3 package. There are only 3 unit test files left that are using mox3 (nova/tests/unit/cells/test_cells_messaging.py, nova/|tests/unit/network/test_neutronv2.py, nova/tests/unit/network/test_manager.py).

We advised contributors not to spend time replacing mox in files testing cells v1 and nova-network (test_cells_messaging.py and test_manager.py), as both are slated for removal as soon as we're able. Progress is being tracked at: https://blueprints.launchpad.net/nova/+spec/mox-removal-stein

sahara Yes Yes voting sahara-tests-scenario-py3 (experimental) Status at 2018-02-23: the job fails during devstack deployment, because swift does not start. The list of its dependencies is minimal and components like keystone are optional. This was partially addressed with an extra dependency: https://review.openstack.org/#/c/544121/ but this requires a) support in devstack (tentative support in https://review.openstack.org/#/c/544383/) b) at least another extra, or a more generic way of doing this; even with an additional extra explicitly required in devstack, the extras would be installed from pip breaking the test of depending patches.

We are facing an issue with pickle library and python 3, once that gets fixed we can try to make the python 3 gate voting.

swift No Yes voting Swift will keep python2 support.

Update (2016-06-27): master branch is currently in a soft-freeze in order to get encryption middleware merged

Python 3 bugs: https://bugs.launchpad.net/swift/+bug/1614289

Plan to add Python 3 support to Swift. Patches: Patches with the topic py3

Python 3 unit tests require liberasurecode >= 1.0.9 and PyEClib >= 1.0.9. To install liberasurecode (in /usr): 

git clone https://bitbucket.org/tsg-/liberasurecode/
cd liberasurecode
/autogen.sh  && /configure --prefix=/usr && make && sudo make install

liberasurecode >= 1.0.9 is available in Ubuntu Precise, Ubuntu Trusty, Fedora 22, Fedora 23 and CentoOS 7.

Update 2018-02-22: A list of current blockers: https://gist.github.com/tipabu/833b03a865dba96e9fa2230b82f5d075

Update 2018-03-06: Current list of blockers with progress notes: https://etherpad.openstack.org/p/Swift_py3

trove Yes Yes voting trove-devstack-base

Status at 2019-01-06: Unit tests - 1708 tests, 1707 passed, 1 skipped.

Integration tests are based on trove-devstack-base job.

Status at 2016-07-25: 1,155 / 2,014 (run_tests.py: 398; testr: 1,521; generate_examples: 95) unit tests (57%), TODO: 859 tests

Status at 2016-06-22: 839 (testtools) / 1,992 (run_tests.py: 398; testr: 1,499; generate_examples: 95) unit tests (42%), TODO: 1,153 tests

Status at 2018-02-22: It appears that only one test is in the blacklist now: guestagent.test_operating_system

run_tests.py (proboscis) failures: https://review.openstack.org/#/c/346905/1

Victor Stinner proposed the trove-python3 blueprint for Mitaka.

From Victor (3/17/2016): "I propose to discuss Python 3 before the summit. For example, prepare a concrete plan to port Trove to Python 3, list technical issues like MySQL-Python, etc."

Patches: search for the bp/trove-python3 topic.

Other OpenStack Applications and Projects

List of all OpenStack projects: projects.yaml.

Project Python 3 Only (Ussuri onwards) Unit tests pass on Python 3 py35 gate functional tests Comments
adjutant Yes Yes voting Partial* (*) Adjutant's unit tests include a suite of API tests against Adjutant APIs using DRF's internal test classes. While not true functional tests, they do stand up an internal web server and process real workloads from Adjutant's perspective, with mocking for external APIs and services (Keystone, Nova, Neutron, etc) in the places we touch their clients. This tests 92% of the code, so in all likelihood covers most py3 vs py2 issues. Tempest tests are also planned for later.
aodh Yes Yes voting
barbican Yes Yes voting unspecified

Blueprint barbican-py3.

blazar Yes Yes voting blazar-devstack-dsvm-py35
cloudkitty Yes Yes voting cloudkitty-tempest-full-python3
congress Yes Yes voting* congress-devstack-py35-api-mysql

(*)py35 support required tiny modification to antlr3 package, creating debian packaging difficulty

Random Stacker wrote a support-python3 blueprint. Patches: search for the topic bp/support-python3.

cyborg Yes No job No job
designate Yes Yes voting designate-devstack-pdns4-py35 (check queue only)

Pradeep Kumar Singh proposed the blueprint designate-py3 which was accepted for Liberty. Patches: topic:bp/designate-py3.

ec2-api Yes Yes voting openstack-tox-py35
freezer Yes Yes voting
karbor Yes Yes voting
kuryr Yes Yes voting openstack-tox-py35
magnum Yes Yes voting (voting: magnum-functional-api) AND (non-voting: magnum-functional-k8s AND magnum-functional-swarm-mode)
manila Yes Yes voting manila-tempest-minimal-dsvm-dummy
manila-ui Yes Yes voting horizon-openstack-tox-python3-django111
masakari Yes Yes voting openstack-tox-py35
mistral Yes Yes voting
monasca-agent Yes Yes voting monasca-tempest-python3-influxdb https://storyboard.openstack.org/#!/story/2000975
monasca-api Yes Yes voting monasca-tempest-python3-influxdb https://storyboard.openstack.org/#!/story/2000975
monasca-log-api Yes Yes voting monascalog-python3-tempest
monasca-ui Yes Yes voting https://storyboard.openstack.org/#!/story/2000975
murano Yes Yes voting

Blueprint: murano-python-3-support.

murano-agent Yes Yes voting

Blueprint: murano-python-3-support.

octavia Yes Yes voting octavia-tox-functional-py35
panko Yes Yes voting panko-tox-py35-mysql and panko-tox-py35-postgresql
qinling Yes Yes voting
rally Yes Yes voting rally-tox-functional-py3 and rally-tox-self

A big thank to Andrey Kurilin for a lot of work in this direction

searchlight Yes Yes voting openstack-tox-py36
senlin Yes Yes voting senlin-dsvm-tempest-py35-api, senlin-dsvm-tempest-py35-functional, senlin-dsvm-tempest-py35-integration (check only, non-voting)
solum Yes Yes voting solum-devstack-py35 (check only)

Victor Stinner proposed the blueprint solum-python3 which was approved. Patches: search for bp/solum-python3 topic (open reviews).

storlets No Yes voting Storlets will keep python2 support.
tacker Yes Yes voting tacker-functional-devstack-python3 (check only, non-voting)
tricircle Yes Yes voting
vitrage Yes Yes voting vitrage-dsvm-api-py35 and vitrage-dsvm-datasources-py35
watcher Yes Yes voting
zaqar Yes Yes voting
zun Yes Yes voting zun-tempest-py35-docker-sql
networking-l2gw Yes Yes voting
vmware-nsx Yes Yes voting All 3rd part CI jobs are running python 3
vmware-nsxlib Yes Yes voting tempest-api-vmware-nsxv3 is running python 3.5 and is voting

Unsorted projects:

  • App Catalog
  • Chef OpenStack (Ruby)
  • Kolla: ~2100 lines of Python, but not "tox -e py27" target
  • OpenStack Ansible (Python): Ansible playbooks and roles for deployment
  • Puppet OpenStack (Ruby)
  • TripleO

Python 2: Python 2.6 support dropped, Python 2.7 only

OpenStack Liberty targets Python 2.7 and 3.4.

Python 2.6 support is being dropped in OpenStack since OpenStack Juno for servers. Python 2.6 support is currently kept in Oslo libraries and clients. See juno-cross-project-future-of-python etherpad.

Python 3.3 support is being dropped since OpenStack Liberty.

Python 2.6 is slowly being dropped in the whole OpenStack project, ex: Remove p26 job from DIB.

Python 2.6 support will be removed in Oslo and Clients clients for OpenStack Mitaka: Oslo libraries dropping python 2.6 compatability.

Port Python 2 code to Python 3

OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. As we are targeting Python 3.4 and up, there is no need to avoid u'unicode' syntax. Do not use six.u('unicode').

Before you begin

If you're doing development with Ubuntu/Debian (and not using devstack with the USE_PYTHON flag set), then you'll need the following packages installed to run the py34 tox unit test targets in the projects: 

   sudo apt-get install python3.4 python3.4-dev

sixer tool

The sixer tool helps to replace most basic patterns to add Python 3 compatibility and it respects OpenStack coding style.

Common patterns

  • Replace "for key in dict.iterkeys()" with "for key in dict"
  • Replace dict.iteritems() with dict.items()
  • Replace dict.itervalues() with dict.values()

Note: Replacing dict.iteritems()/.itervalues() with six.iteritems(dict)/six.itervalues(dict) was preferred in the past, but there was a discussion suggesting to avoid six for this. The overhead of creating a temporary list on Python 2 is negligible.

  • Replace iterator.next() with next(iterator)
  • Replace basestring with six.string_types
  • Replace unicode with six.text_type
  • Replace (str, unicode) with six.string_types
  • Replace (int, long) with six.integer_types
  • Replace func.func_name with func.__name__
  • Replace exceptions.OSError with OSError and remove "import exceptions"
  • map() and filter() if a list is needed on Python 3:
    • Replace map(func, data) with [func(item) for item in data]
    • Replace filter(lambda obj: test(obj), data) with [obj for obj in data if test(obj)]
    • Note: Usually, tests fail because map() or filter() objects have no length

Serialization: base64, JSON, etc.

  • [with oslo.serialization 1.10 or newer] Get the base64 module from oslo_serialization (from oslo_serialization import base64) to get functions:
    • oslo_serialization.base64.decode_as_bytes(encoded)
    • oslo_serialization.base64.decode_as_text(encoded, encoding='utf-8')
    • oslo_serialization.base64.encode_as_bytes(s, encoding='utf-8')
    • oslo_serialization.base64.encode_as_text(s, encoding='utf-8')
  • Replace text.encode('base64') and base64.b64encode(text) with:
    • base64.b64encode(text): only accept bytes and returns bytes,
    • or: oslo_serialization.base64.encode_as_bytes(text): accept bytes or Unicode and returns bytes
    • or: oslo_serialization.base64.encode_as_text(text): accept bytes and Unicode and returns Unicode
    • Warning: base64.encodestring(raw) adds a newline ("\n"), whereas encode_as_bytes() and encode_as_text() don't.
  • Replace raw.decode('base64') with:
    • base64.b64decode(raw): return bytes
    • oslo_serialization.base64.decode_as_bytes(encoded): accept bytes and Unicode, returns bytes
    • oslo_serialization.base64.decode_as_text(encoded): accept bytes and Unicode, returns Unicode

Hexadecimal:

  • Replace raw.decode('hex') with binascii.unhexlify(raw)
  • Replace bytes.encode('hex') with binascii.hexlify(bytes)

JSON:

  • [with oslo.serialization 1.10 or newer] Replace json.dumps(obj) with oslo_serialization.jsonutils.dump_as_bytes(obj)
  • Replace json.loads(obj) with oslo_serialization.jsonutils.loads(obj): it accepts bytes and Unicode, bytes is decoded from UTF-8. It avoids "if isinstance(obj, bytes): obj = obj.decode('utf-8')" which may require a second temporary variable.

contextlib.nested

To replaced contextlib.nested is to use contextlib.ExitStack. It's available on Python 2 using contextlib2.ExitStack. For contextlib.nested, nova/test.py defines: 

if six.PY3:
    @contextlib.contextmanager
    def nested(*contexts):
        with contextlib.ExitStack() as stack:
            yield [stack.enter_context(c) for c in contexts]
else:
    nested = contextlib.nested

Another option to replaced contextlib.nested is to use the @mock.patch decorator. Example with nested function: 

def test_thing(self):
   @mock.patch(...)
   @mock.patch(...)
   @mock.patch(...)
   def do_test(...):
       ...

   do_test()

More options:

oslo_utils.encodeutils.to_utf8

oslo.utils 3.5 has an oslo_utils.encodeutils.to_utf8() function to encode Unicode to UTF-8 and return bytes unchanged.

bytes.decode and unicode.encode

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write data.decode() or text.encode() without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: data.decode('utf-8') or text.encode('utf-8'). The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

safe_decode

Olso Incubator has a function safe_decode() which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_decode(data, 'utf-8'): decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:

  • safe_decode(data)

By default, the decoder is strict. You can specify a different error handler using the optional errors parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

safe_encode

Olso Incubator has a function safe_encode() which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.

  • safe_encode(text) encodes text to the output encoding
  • safe_encode(bytes) may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (incomding parameter) is sys.stdin.encoding or sys.getdefaultencoding():

  • Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
  • Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (encoding parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:

  • safe_encode(data, incoming='utf-8'): encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:

  • safe_encode(data)

Example:

  • safe_encode(b'\xe9', incoming='latin-1') returns b'\xc3\xa9'.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional errors parameter. Example: safe_encode(b'[\xff]', incoming='ascii', errors='ignore') returns b'[]'.

logging module and format exceptions

The exception_to_unicode(exc) function of oslo_utils.encodeutils is the recommanded way to format an exception to Unicode. This function works on Python 2 and Python 3 and it should avoid mojibake is most cases.

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs b'hello' instead of 'hello'.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

  • str(exc): native string, so use bytes on Python 2
  • six.text_type(exc): always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: unicode(Exception("nonascii:\xe9")).
  • six.u(str(exc)): unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: unicode(str(Exception("\xff")))
  • LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using str(exc). Example: LOG.debug(str(exc)).

HTTP

The HTTP protocol is based on bytes:

  • HTTP body contains bytes. For example, use io.BytesIO for a stream storing an HTTP body.
  • HTTPConnection.getresponse().read() returns bytes (in Python 3, str which is bytes in Python 2)
  • On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
  • It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and HTTP header should use what character encoding?

References to port Python 2 code to Python 3

Common pitfalls

What is a string ?

You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2: 

   >>> type('foo')
   <type 'str'>
   >>> type(u'foo')
   <type 'unicode'>
   >>> type(b'foo')
   <type 'str'>
   >>> isinstance('foo', six.text_type)
   False
   >>> isinstance(u'foo', six.text_type)
   True
   >>> bytes is str
   True
   >>> b'foo'[0]
   'f'

Python 3: 

   >>> type('foo')
   <class 'str'>
   >>> type(u'foo')
   <class 'str'>
   >>> type(b'foo')
   <class 'bytes'>
   >>> isinstance('foo', six.text_type)
   True
   >>> isinstance(b'foo', six.text_type)
   False
   >>> bytes is str
   False
   >>> b'foo'[0]
   102

tox/testr error: db type could not be determined

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/" and then run "tox -e py34" before running "tox -e py27". You only have to do this once. The problem only occurs with "tox -e py34" when .testrepository/ was created by Python 2.

Enable Python 3 in DevStack

  • Fedora::
sudo dnf install python3-devel
sudo python3 -m pip install python-memcached   # needed by keystonemiddleware
  • localrc: add the following line
USE_PYTHON3=True

See Switch to python 3.5.

Functional and Integration Tests

Etherpad: https://etherpad.openstack.org/p/support-python3.5-functional-tests

Doug Hellmann wrote a cross-project specification to run integration tests on Python 3: Enabling Python 3 for Application Integration Tests (spec approved for Liberty).

Patch for DevStack: Enable optional Python 3 support.

Projects wanting to run their existing functional test suite against their services on Python 3 should:-

  • Add the Python3.4 trove classifier to the setup.cfg configuration file of the respective service.
  • set USE_PYTHON3=True in Devstack's localrc config file before running stack.sh after the DevStack change is merged or cherry-pick until then.
  • set basepython = python3.5 in tox for the necessary functional/integration tests environment.

Then run tox -e<your functional test suite name>. Example:- tox -efunctional to run the functional the tests of a certain project.

Once the functional test suite is reliably passing against the service in a python 3.5 env, the trove classifiers must be updated along with a new gate job being added to the project.

Sirushti Murugesan's attempt on Heat: https://review.openstack.org/#/c/188033/

Python 3.5

Python 3.6 status

Python 3.6 is scheduled to be the default/only version of Python available in Ubuntu 18.04 LTS (Bionic Beaver) and Debian 10 (Buster).

Known issues: 

   ENABLED_PYTHON3_PACKAGES=horizon

Status of Python 3 in Linux distributions

Check the Project Testing Interface for updates about the platforms each release is tested on.

Status of Python 2 Support Being Dropped in Dependencies

OpenStack plans to start dropping Python 2 support at the start of the U cycle

Progress reports, meetings and sprints

Articles

Pycon Montreal 2014: Sprint Port OpenStack to Python 3

Enovance organized a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday) 2014. See the page Python3/SprintPycon2014.

Retrieved from "https://wiki.openstack.org/w/index.php?title=Python3&oldid=174730"