OpenStack - User contributions [en]

Python3

2015-03-06T11:55:24Z

Lennart Regebro: /* Dependencies */ updated status for sphinxcontrib-httpdomain

This page tracks the progress of Python 3 effort porting for OpenStack.

== Python 3 ==

[http://techs.enovance.com/6521/openstack_python3 Why should OpenStack move to Python 3 right now?]
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''

== Port Python 2 code to Python 3 ==

OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).

=== Common patterns ===

* Replace dict.iteritems() with six.iteritems(dict)
* Replace iterator.next() with next(iterator)
* Replace basestring with six.string_types
* Replace unicode with six.text_type

=== bytes.decode and unicode.encode ===

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write <code>data.decode()</code> or <code>text.encode()</code> without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: <code>data.decode('utf-8')</code> or <code>text.encode('utf-8')</code>. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

=== safe_decode ===

Olso Incubator has a function '''safe_decode()''' which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_decode(data, 'utf-8')</code>: decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:
* <code>safe_decode(data)</code>

By default, the decoder is strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

=== safe_encode ===

Olso Incubator has a function '''safe_encode()''' which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
* <code>safe_encode(text)</code> encodes text to the output encoding
* <code>safe_encode(bytes)</code> may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (<code>incomding</code> parameter) is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (<code>encoding</code> parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_encode(data, incoming='utf-8')</code>: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:
* <code>safe_encode(data)</code>

Example:
* <code>safe_encode(b'\xe9', incoming='latin-1')</code> returns <code>b'\xc3\xa9'</code>.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: <code>safe_encode(b'[\xff]', incoming='ascii', errors='ignore')</code> returns <code>b'[]'</code>.

=== logging module and format exceptions ===

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs <code>b'hello'</code> instead of <code>'hello'</code>.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

* <code>str(exc)</code>: native string, so use bytes on Python 2
* <code>six.text_type(exc)</code>: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: <code>unicode(Exception("nonascii:\xe9"))</code>.
* <code>six.u(str(exc))</code>: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: <code>unicode(str(Exception("\xff")))</code>
* <code>LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})</code>

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using <code>str(exc)</code>. Example: <code>LOG.debug(str(exc))</code>.

=== HTTP ===

The HTTP protocol is based on '''bytes''':

* HTTP body contains '''bytes'''. For example, use io.BytesIO for a stream storing an HTTP body.
* HTTPConnection.getresponse().read() returns '''bytes''' (in Python 3, '''str''' which is bytes in Python 2)
* On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
* It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and [http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding HTTP header should use what character encoding?]

=== References to port Python 2 code to Python 3 ===
* [http://python3porting.com/ Porting to Python 3 Book] by Lennart Regebro, especially the [http://python3porting.com/differences.html Language differences and workarounds].
* [http://docs.python.org/dev/howto/pyporting.html HOWTO: Porting Python 2 Code to Python 3] by Brett Cannon
* [https://wiki.python.org/moin/PortingPythonToPy3k Porting Python Code to 3.x]
* [http://code.google.com/p/python-incompatibility/ python-incompatibility]: Demonstrates incompatibilities between Python versions.

=== Common pitfalls ===

==== What is a string ? ====
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:
>>> type('foo')
<type 'str'>
>>> type(u'foo')
<type 'unicode'>
>>> type(b'foo')
<type 'str'>
>>> isinstance('foo', six.text_type)
False
>>> isinstance(u'foo', six.text_type)
True
>>> bytes is str
True
>>> b'foo'[0]
'f'

Python 3:
>>> type('foo')
<class 'str'>
>>> type(u'foo')
<class 'str'>
>>> type(b'foo')
<class 'bytes'>
>>> isinstance('foo', six.text_type)
True
>>> isinstance(b'foo', six.text_type)
False
>>> bytes is str
False
>>> b'foo'[0]
102

==== tox/testr error: db type could not be determined ====

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/".

== Python 3 Status of OpenStack projects ==

=== Oslo Incubator ===

'''BLOCKER BUG:''' Tests using testscenarios fail on Python 3 with nosetests because of this bug:
https://bugs.launchpad.net/testscenarios/+bug/872887

Recently merged reviews:

* https://review.openstack.org/#/c/79781/

{|class="wikitable sortable"
|-
! Test (full path) !! Patches !! Comment
|-

| style="background-color: lightgreen" | tests/unit/config/test_generator.py || https://review.openstack.org/#/c/88087/ ||
|-

| style="background-color: lightgreen" | tests/unit/crypto/test_utils.py || https://review.openstack.org/87413 ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migration_common.py || https://review.openstack.org/#/c/107752/ ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migrate.py || https://review.openstack.org/#/c/107921/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_models.py || https://review.openstack.org/#/c/80307/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_options.py || https://review.openstack.org/#/c/80627/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_migrate_cli.py || https://review.openstack.org/107988 ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_utils.py || ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_sqlalchemy.py || ||
|-

| style="background-color: lightgreen" | tests/unit/fixture/test_logging.py || https://review.openstack.org/#/c/90318/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_request_id.py || https://review.openstack.org/#/c/80336/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_sizelimit.py || https://review.openstack.org/#/c/80450/ ||
|-

| style="background-color: red" | tests/unit/middleware/test_audit.py || || depends on pycadf ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_guru_meditation_report.py || https://review.openstack.org/#/c/87404/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_base_report.py || https://review.openstack.org/#/c/87973/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_openstack_generators.py || https://review.openstack.org/#/c/88124/ || ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_views.py || https://review.openstack.org/87376 ||
|-

| style="background-color: lightblue" | tests/unit/rpc/test_common.py || https://review.openstack.org/#/c/80533/ || The RPC code in the incubator is deprecated in favor of oslo.messaging. --[[User:Doug-hellmann|doug-hellmann]] ([[User talk:Doug-hellmann|talk]]) 15:40, 14 April 2014 (UTC)
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_base_filter.py || https://review.openstack.org/#/c/80321/ ||
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_weights.py || https://review.openstack.org/#/c/87336/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_cliutils || https://review.openstack.org/#/c/74433/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_fileutils || https://review.openstack.org/#/c/74728/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_gettext.py || https://review.openstack.org/#/c/80534/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_imageutils.py || https://review.openstack.org/#/c/90532/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_jsonutils.py || https://review.openstack.org/#/c/80370/ ||
|-

| style="background-color: orange" | tests/unit/test_log.py || https://review.openstack.org/104890 ||
|-

| style="background-color: lightblue" | tests/unit/test_processutils.py || || processutils was moved to the new oslo.concurrency project ||
|-

| style="background-color: lightgreen" | tests/unit/test_quota.py || https://review.openstack.org/#/c/80564/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_strutils.py ||
* https://review.openstack.org/#/c/80571/ (safe_encode)
* https://review.openstack.org/#/c/80573/
||
|-

|}

=== Common Libraries (Oslo Projects) ===

For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cliff cliff] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.concurrency oslo.concurrency] || style="background-color: orange;" | Partial || https://review.openstack.org/141206
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.db oslo.db] || style="background-color: orange;" | Partial || No unit tests with MySQL because of DB driver
|-
| [https://pypi.python.org/pypi/oslo.i18n oslo.i18n] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.log oslo.log] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color: lightgreen;" | Yes || eventlet executor and qpid driver are not supported on Python 3, greenio
executor may help
|-
| [https://pypi.python.org/pypi/oslo.middleware oslo.middleware] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.serialization oslo.serialization] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || ? || The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible?
|-
| [https://pypi.python.org/pypi/oslotest oslotest] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: red;" | No || Blocked suds dependency
|-
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes ||
|-
| oslo.version || style="background-color: lightgreen;" | Yes || not released on PyPI yet
|-
| pylockfile || style="background-color: lightgreen;" | Yes || related to https://pypi.python.org/pypi/lockfile ?
|-
| [https://pypi.python.org/pypi/stevedore stevedore] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: lightgreen;" | Yes ||
|}

Development tools:

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cookiecutter cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/oslo-cookiecutter oslo-cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/hacking hacking] || ? ||
|-
| [https://pypi.python.org/pypi/pbr pbr] || ? ||
|}

=== OpenStack clients ===

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/python-barbicanclient python-barbicanclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:orange;" | In the git repo, not on PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ganttclient python-ganttclient] || ? || ? || ? || ? ||
|-

| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color:lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-melangeclient python-melangeclient] || ? || ? || ? || ||
|-

| [https://pypi.python.org/pypi/python-novaclient python-novaclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPII || ||
|-

| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: orange;" | Not released yet || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient] || style="background-color: lightgreen" | OK || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | Yes || || As of 0.9
|-

| [https://pypi.python.org/pypi/python-saharaclient python-saharaclient] || style="background-color: orange;" | In progress || style="background-color: orange" | Non-voting || || || https://review.openstack.org/#/c/73128/
|-

| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | [https://pypi.python.org/pypi/python-swiftclient/ On PyPI] || ||
|-

| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|}

=== Core OpenStack projects ===

Completely updated on Monday, September the 29th.

{| class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Trove classifiers !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* croniter
* eventlet
* thrift (which is blocking happybase)
* oslo.db
* oslo.messaging
* python-neutronclient
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* sphinxcontrib-docbookrestapi
* sphinxcontrib-httpdomain
* sphinxcontrib-pecanwsme
||
|-

| [https://pypi.python.org/pypi/cinder cinder] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* paste
* python-barbicanclient
* rtslib-fb
* sqlalchemy-migrate
* suds
Requirements for tests
* mysql-python

||
|-

| [https://pypi.python.org/pypi/glance glance] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* glance_store
* oslo.db
* oslo.messaging
* paste
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* qpid-python

||
|-

| [https://pypi.python.org/pypi/heat heat] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* python-neutronclient
* python-saharaclient
* qpid-python
* sqlalchemy-migrate
Requirements for tests
* mysql-python
||
|-

| [https://pypi.python.org/pypi/horizon horizon] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* django-pyscss
* eventlet
* python-neutronclient
* python-saharaclient
* xstatic
* xstatic-angular
* xstatic-angular-cookies
* xstatic-angular-mock
* xstatic-bootstrap-datepicker
* xstatic-d3
* xstatic-font-awesome
* xstatic-hogan
* xstatic-jasmine
* xstatic-jquery
* xstatic-jquery-migrate
* xstatic-jquery.quicksearch
* xstatic-jquery.tablesorter
* xstatic-jsencrypt
* xstatic-qunit
* xstatic-rickshaw
* xstatic-spin
Requirements for tests
* nodeenv
* nose-exclude
* nosehtmloutput
* openstack.nose_plugin
||
|-

| [https://pypi.python.org/pypi/keystone keystone] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* sqlalchemy-migrate
Requirements for tests
* ldappool
* paste (which is blocking pysaml2)
* python-ldap
||
|-

| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* jsonrpclib
* oslo.db
* oslo.messaging
* paste
* python-neutronclient
||
|-

| [https://pypi.python.org/pypi/nova nova] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* python-neutronclient
* sqlalchemy-migrate
* suds
* websockify
Requirements for tests
* libvirt-python
* mysql-python
||
|-

| [https://pypi.python.org/pypi/swift swift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
Requirements for tests
* nosehtmloutput
* openstack.nose_plugin

||
|-

|}

Number of core OpenStack projetcs blocked by each dependency:
9 eventlet
7 oslo.messaging
7 oslo.db
6 sqlalchemy-migrate
6 paste
5 python-neutronclient
5 mysql-python
2 suds
2 qpid-python
2 python-saharaclient
2 pycadf
2 openstack.nose_plugin
2 nosehtmloutput
1 xstatic-spin
1 xstatic-rickshaw
1 xstatic-qunit
1 xstatic-jsencrypt
1 xstatic-jquery.tablesorter
1 xstatic-jquery.quicksearch
1 xstatic-jquery-migrate
1 xstatic-jquery
1 xstatic-jasmine
1 xstatic-hogan
1 xstatic-font-awesome
1 xstatic-d3
1 xstatic-bootstrap-datepicker
1 xstatic-angular-mock
1 xstatic-angular-cookies
1 xstatic-angular
1 xstatic
1 websockify
1 thrift
1 sphinxcontrib-pecanwsme
1 sphinxcontrib-httpdomain
1 sphinxcontrib-docbookrestapi
1 rtslib-fb
1 python-ldap
1 python-barbicanclient
1 nose-exclude
1 nodeenv
1 libvirt-python
1 ldappool
1 jsonrpclib
1 glance_store
1 django-pyscss
1 croniter

=== Dependencies ===

[https://caniusepython3.com/check/4fd5dda2-b1f1-4db4-a636-67cd3276cb6a Porting status] for [https://github.com/openstack/requirements/blob/master/global-requirements.txt global-requirement.txt].

It's now possible to specify different dependencies for Python 2 and Python 3 using:
* requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
* requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
* (same for test-requirements.txt)

You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:

<pre>
...
[testenv:py33]
deps = -r{toxinidir}/requirements-py3.txt
-r{toxinidir}/test-requirements-py3.txt
...
</pre>

See also a patch to support markers in requirements (in pip): [https://github.com/pypa/pip/issues/1433 pip issue: Support markers in setup(install_requires)?]; [https://github.com/pypa/pip/pull/1472 Victor Stinner's pull request: "parse requirements in markers"]

OpenStack Dependencies:

* mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock). Examples:
** neutronclient: https://review.openstack.org/#/c/95786/
** Oslo Incubator: https://review.openstack.org/#/c/93729/

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/boto boto] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32).
|-

| [https://pypi.python.org/pypi/django-compressor django-compressor] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || Requirements upgraded: https://review.openstack.org/94357
|-

| [https://pypi.python.org/pypi/django-openstack-auth django-openstack-auth] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || ||
As of 1.1.6
|-

| [https://pypi.python.org/pypi/dnspython dnspython] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Must use the [https://pypi.python.org/pypi/dnspython3/ Python 3 version], see https://github.com/rthalley/dnspython/issues/60
|-

| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: orange;" | In the Git repo || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
|-

| [https://pypi.python.org/pypi/eventlet eventlet] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || eventlet portage to Python 3 in progress: [https://github.com/eventlet/eventlet/issues/6 eventlet issue #6: Support Python 3.3] and [https://github.com/eventlet/eventlet/pull/99 Pull request #99: Fix several issues with python3 thread patching]. Victor Stinner is working on [http://trollius.readthedocs.org/ Trollius] (asyncio for Python 2) which may replace eventlet: [http://techs.enovance.com/6562/asyncio-openstack-python3 Use the new asyncio module and Trollius in OpenStack]
|-

| [https://pypi.python.org/pypi/hacking hacking] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Cyril Roelandt patch: [https://review.openstack.org/#/c/77585/ Make hacking Python 3 compatible]
|-

| [https://pypi.python.org/pypi/jsonrpclib jsonrpclib] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || The project seems dead :(
|-

| [https://pypi.python.org/pypi/mysql-python mysql-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there.
|-

| [https://pypi.python.org/pypi/netifaces netifaces] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Patch sent by Victor Stinner (in private): [https://bitbucket.org/haypo/misc/src/tip/openstack/netifaces_python3.patch netifaces_python3.patch], Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 .
|-

| [https://pypi.python.org/pypi/nose-exclude nose-exclude] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
|-

| [https://pypi.python.org/pypi/nosehtmloutput nosehtmloutput] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* nose-exclude (tests only)
* openstack.nose-plugin
||
* https://bugs.launchpad.net/ubuntu/+source/python-nosehtmloutput/+bug/1287247
* https://review.openstack.org/#/c/80956/
|-
| [https://pypi.python.org/pypi/nosexcover nosexcover] || style="background-color:lightgreen;" | No || N/A || style="background-color: lightgreen;" | On PyPI || || Python 3 support since 1.0.9
|-
| [https://pypi.python.org/pypi/openstack.nose-plugin openstack.nose-plugin] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color:red;" | No || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || suds ||
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Portage in Progress by Victor Stinner ([https://review.openstack.org/#/dashboard/9107 dashboard])
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI (1.1.0) || ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color:lightgreen;" | Yes || No tests :) || style="background-color: lightgreen;" | In the git repo, not on PyPI || || https://review.openstack.org/#/c/79311/
|-
| [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Must be replaced by oslosphinx (without the dot)
|-

| [https://pypi.python.org/pypi/pam pam] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The fork [https://pypi.python.org/pypi/simplepam simplepam] works on Python 2 and 3
|-

| [https://pypi.python.org/pypi/paramiko paramiko] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | On PyPI || || Requirements upgraded: https://review.openstack.org/#/c/81132/
|-

| [https://pypi.python.org/pypi/paste paste] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
|-

| [https://pypi.python.org/pypi/pycadf pycadf] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Classifiers added https://review.openstack.org/#/c/133088/
|-

| [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The project seems dead.
|-

| [https://pypi.python.org/pypi/python-memcached python-memcached] || style="background-color:green;" | Not released, but last commit on git has it || style="background-color: red;" | No || style="background-color: red;" | No || || [https://github.com/jbalogh/django-cache-machine/issues/52 Issue #52: Python 3.3 support? ], [https://github.com/linsomniac/python-memcached/pull/26 Pull request #26: Python 3.3 Support] -- Julien Danjou ported [https://pypi.python.org/pypi/pymemcache pymemcache] to Python 3, another memcached client, he suggests to use this one instead
|-

| [https://pypi.python.org/pypi/qpid-python qpid-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/rtslib-fb rtslib-fb] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-docbookrestapi sphinxcontrib-docbookrestapi] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-httpdomain sphinxcontrib-httpdomain] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-pecanwsme sphinxcontrib-pecanwsme] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No||
* hacking
* ibm-db-sa
* scripttest
||
|-
| [https://pypi.python.org/pypi/suds suds] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || "Lightweight SOAP client". Last commit 2 years ago: https://fedorahosted.org/suds/browser See also this fork which is promising: https://bitbucket.org/jurko/suds
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || ||
|-
| [https://pypi.python.org/pypi/thrift thrift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/websockify websockify] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

|}

=== Other OpenStack projects ===

Python 3 compatible:

* stackforge/python-jenkins : https://review.openstack.org/#/c/95004/ (py33 gate is voting)
* openstack-infra/jenkins-job-builder : https://review.openstack.org/87810 (Python 3 support finally merged in, Sept. 2014)

== Reports at OpenStack Summits ==

* Havana summit notes: https://etherpad.openstack.org/p/havana-python3
* Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
* Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)

== Pycon Montreal 2014: Sprint Port OpenStack to Python 3 ==

Enovance organizes a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday). See the page [[Python3/SprintPycon2014]].

Python3

2015-03-06T11:54:15Z

Lennart Regebro: /* Dependencies */ updated status for sphinxcontrib-httpdomain

This page tracks the progress of Python 3 effort porting for OpenStack.

== Python 3 ==

[http://techs.enovance.com/6521/openstack_python3 Why should OpenStack move to Python 3 right now?]
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''

== Port Python 2 code to Python 3 ==

OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).

=== Common patterns ===

* Replace dict.iteritems() with six.iteritems(dict)
* Replace iterator.next() with next(iterator)
* Replace basestring with six.string_types
* Replace unicode with six.text_type

=== bytes.decode and unicode.encode ===

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write <code>data.decode()</code> or <code>text.encode()</code> without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: <code>data.decode('utf-8')</code> or <code>text.encode('utf-8')</code>. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

=== safe_decode ===

Olso Incubator has a function '''safe_decode()''' which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_decode(data, 'utf-8')</code>: decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:
* <code>safe_decode(data)</code>

By default, the decoder is strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

=== safe_encode ===

Olso Incubator has a function '''safe_encode()''' which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
* <code>safe_encode(text)</code> encodes text to the output encoding
* <code>safe_encode(bytes)</code> may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (<code>incomding</code> parameter) is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (<code>encoding</code> parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_encode(data, incoming='utf-8')</code>: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:
* <code>safe_encode(data)</code>

Example:
* <code>safe_encode(b'\xe9', incoming='latin-1')</code> returns <code>b'\xc3\xa9'</code>.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: <code>safe_encode(b'[\xff]', incoming='ascii', errors='ignore')</code> returns <code>b'[]'</code>.

=== logging module and format exceptions ===

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs <code>b'hello'</code> instead of <code>'hello'</code>.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

* <code>str(exc)</code>: native string, so use bytes on Python 2
* <code>six.text_type(exc)</code>: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: <code>unicode(Exception("nonascii:\xe9"))</code>.
* <code>six.u(str(exc))</code>: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: <code>unicode(str(Exception("\xff")))</code>
* <code>LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})</code>

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using <code>str(exc)</code>. Example: <code>LOG.debug(str(exc))</code>.

=== HTTP ===

The HTTP protocol is based on '''bytes''':

* HTTP body contains '''bytes'''. For example, use io.BytesIO for a stream storing an HTTP body.
* HTTPConnection.getresponse().read() returns '''bytes''' (in Python 3, '''str''' which is bytes in Python 2)
* On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
* It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and [http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding HTTP header should use what character encoding?]

=== References to port Python 2 code to Python 3 ===
* [http://python3porting.com/ Porting to Python 3 Book] by Lennart Regebro, especially the [http://python3porting.com/differences.html Language differences and workarounds].
* [http://docs.python.org/dev/howto/pyporting.html HOWTO: Porting Python 2 Code to Python 3] by Brett Cannon
* [https://wiki.python.org/moin/PortingPythonToPy3k Porting Python Code to 3.x]
* [http://code.google.com/p/python-incompatibility/ python-incompatibility]: Demonstrates incompatibilities between Python versions.

=== Common pitfalls ===

==== What is a string ? ====
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:
>>> type('foo')
<type 'str'>
>>> type(u'foo')
<type 'unicode'>
>>> type(b'foo')
<type 'str'>
>>> isinstance('foo', six.text_type)
False
>>> isinstance(u'foo', six.text_type)
True
>>> bytes is str
True
>>> b'foo'[0]
'f'

Python 3:
>>> type('foo')
<class 'str'>
>>> type(u'foo')
<class 'str'>
>>> type(b'foo')
<class 'bytes'>
>>> isinstance('foo', six.text_type)
True
>>> isinstance(b'foo', six.text_type)
False
>>> bytes is str
False
>>> b'foo'[0]
102

==== tox/testr error: db type could not be determined ====

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/".

== Python 3 Status of OpenStack projects ==

=== Oslo Incubator ===

'''BLOCKER BUG:''' Tests using testscenarios fail on Python 3 with nosetests because of this bug:
https://bugs.launchpad.net/testscenarios/+bug/872887

Recently merged reviews:

* https://review.openstack.org/#/c/79781/

{|class="wikitable sortable"
|-
! Test (full path) !! Patches !! Comment
|-

| style="background-color: lightgreen" | tests/unit/config/test_generator.py || https://review.openstack.org/#/c/88087/ ||
|-

| style="background-color: lightgreen" | tests/unit/crypto/test_utils.py || https://review.openstack.org/87413 ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migration_common.py || https://review.openstack.org/#/c/107752/ ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migrate.py || https://review.openstack.org/#/c/107921/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_models.py || https://review.openstack.org/#/c/80307/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_options.py || https://review.openstack.org/#/c/80627/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_migrate_cli.py || https://review.openstack.org/107988 ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_utils.py || ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_sqlalchemy.py || ||
|-

| style="background-color: lightgreen" | tests/unit/fixture/test_logging.py || https://review.openstack.org/#/c/90318/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_request_id.py || https://review.openstack.org/#/c/80336/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_sizelimit.py || https://review.openstack.org/#/c/80450/ ||
|-

| style="background-color: red" | tests/unit/middleware/test_audit.py || || depends on pycadf ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_guru_meditation_report.py || https://review.openstack.org/#/c/87404/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_base_report.py || https://review.openstack.org/#/c/87973/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_openstack_generators.py || https://review.openstack.org/#/c/88124/ || ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_views.py || https://review.openstack.org/87376 ||
|-

| style="background-color: lightblue" | tests/unit/rpc/test_common.py || https://review.openstack.org/#/c/80533/ || The RPC code in the incubator is deprecated in favor of oslo.messaging. --[[User:Doug-hellmann|doug-hellmann]] ([[User talk:Doug-hellmann|talk]]) 15:40, 14 April 2014 (UTC)
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_base_filter.py || https://review.openstack.org/#/c/80321/ ||
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_weights.py || https://review.openstack.org/#/c/87336/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_cliutils || https://review.openstack.org/#/c/74433/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_fileutils || https://review.openstack.org/#/c/74728/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_gettext.py || https://review.openstack.org/#/c/80534/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_imageutils.py || https://review.openstack.org/#/c/90532/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_jsonutils.py || https://review.openstack.org/#/c/80370/ ||
|-

| style="background-color: orange" | tests/unit/test_log.py || https://review.openstack.org/104890 ||
|-

| style="background-color: lightblue" | tests/unit/test_processutils.py || || processutils was moved to the new oslo.concurrency project ||
|-

| style="background-color: lightgreen" | tests/unit/test_quota.py || https://review.openstack.org/#/c/80564/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_strutils.py ||
* https://review.openstack.org/#/c/80571/ (safe_encode)
* https://review.openstack.org/#/c/80573/
||
|-

|}

=== Common Libraries (Oslo Projects) ===

For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cliff cliff] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.concurrency oslo.concurrency] || style="background-color: orange;" | Partial || https://review.openstack.org/141206
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.db oslo.db] || style="background-color: orange;" | Partial || No unit tests with MySQL because of DB driver
|-
| [https://pypi.python.org/pypi/oslo.i18n oslo.i18n] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.log oslo.log] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color: lightgreen;" | Yes || eventlet executor and qpid driver are not supported on Python 3, greenio
executor may help
|-
| [https://pypi.python.org/pypi/oslo.middleware oslo.middleware] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.serialization oslo.serialization] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || ? || The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible?
|-
| [https://pypi.python.org/pypi/oslotest oslotest] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: red;" | No || Blocked suds dependency
|-
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes ||
|-
| oslo.version || style="background-color: lightgreen;" | Yes || not released on PyPI yet
|-
| pylockfile || style="background-color: lightgreen;" | Yes || related to https://pypi.python.org/pypi/lockfile ?
|-
| [https://pypi.python.org/pypi/stevedore stevedore] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: lightgreen;" | Yes ||
|}

Development tools:

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cookiecutter cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/oslo-cookiecutter oslo-cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/hacking hacking] || ? ||
|-
| [https://pypi.python.org/pypi/pbr pbr] || ? ||
|}

=== OpenStack clients ===

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/python-barbicanclient python-barbicanclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:orange;" | In the git repo, not on PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ganttclient python-ganttclient] || ? || ? || ? || ? ||
|-

| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color:lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-melangeclient python-melangeclient] || ? || ? || ? || ||
|-

| [https://pypi.python.org/pypi/python-novaclient python-novaclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPII || ||
|-

| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: orange;" | Not released yet || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient] || style="background-color: lightgreen" | OK || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | Yes || || As of 0.9
|-

| [https://pypi.python.org/pypi/python-saharaclient python-saharaclient] || style="background-color: orange;" | In progress || style="background-color: orange" | Non-voting || || || https://review.openstack.org/#/c/73128/
|-

| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | [https://pypi.python.org/pypi/python-swiftclient/ On PyPI] || ||
|-

| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|}

=== Core OpenStack projects ===

Completely updated on Monday, September the 29th.

{| class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Trove classifiers !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* croniter
* eventlet
* thrift (which is blocking happybase)
* oslo.db
* oslo.messaging
* python-neutronclient
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* sphinxcontrib-docbookrestapi
* sphinxcontrib-httpdomain
* sphinxcontrib-pecanwsme
||
|-

| [https://pypi.python.org/pypi/cinder cinder] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* paste
* python-barbicanclient
* rtslib-fb
* sqlalchemy-migrate
* suds
Requirements for tests
* mysql-python

||
|-

| [https://pypi.python.org/pypi/glance glance] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* glance_store
* oslo.db
* oslo.messaging
* paste
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* qpid-python

||
|-

| [https://pypi.python.org/pypi/heat heat] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* python-neutronclient
* python-saharaclient
* qpid-python
* sqlalchemy-migrate
Requirements for tests
* mysql-python
||
|-

| [https://pypi.python.org/pypi/horizon horizon] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* django-pyscss
* eventlet
* python-neutronclient
* python-saharaclient
* xstatic
* xstatic-angular
* xstatic-angular-cookies
* xstatic-angular-mock
* xstatic-bootstrap-datepicker
* xstatic-d3
* xstatic-font-awesome
* xstatic-hogan
* xstatic-jasmine
* xstatic-jquery
* xstatic-jquery-migrate
* xstatic-jquery.quicksearch
* xstatic-jquery.tablesorter
* xstatic-jsencrypt
* xstatic-qunit
* xstatic-rickshaw
* xstatic-spin
Requirements for tests
* nodeenv
* nose-exclude
* nosehtmloutput
* openstack.nose_plugin
||
|-

| [https://pypi.python.org/pypi/keystone keystone] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* sqlalchemy-migrate
Requirements for tests
* ldappool
* paste (which is blocking pysaml2)
* python-ldap
||
|-

| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* jsonrpclib
* oslo.db
* oslo.messaging
* paste
* python-neutronclient
||
|-

| [https://pypi.python.org/pypi/nova nova] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* python-neutronclient
* sqlalchemy-migrate
* suds
* websockify
Requirements for tests
* libvirt-python
* mysql-python
||
|-

| [https://pypi.python.org/pypi/swift swift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
Requirements for tests
* nosehtmloutput
* openstack.nose_plugin

||
|-

|}

Number of core OpenStack projetcs blocked by each dependency:
9 eventlet
7 oslo.messaging
7 oslo.db
6 sqlalchemy-migrate
6 paste
5 python-neutronclient
5 mysql-python
2 suds
2 qpid-python
2 python-saharaclient
2 pycadf
2 openstack.nose_plugin
2 nosehtmloutput
1 xstatic-spin
1 xstatic-rickshaw
1 xstatic-qunit
1 xstatic-jsencrypt
1 xstatic-jquery.tablesorter
1 xstatic-jquery.quicksearch
1 xstatic-jquery-migrate
1 xstatic-jquery
1 xstatic-jasmine
1 xstatic-hogan
1 xstatic-font-awesome
1 xstatic-d3
1 xstatic-bootstrap-datepicker
1 xstatic-angular-mock
1 xstatic-angular-cookies
1 xstatic-angular
1 xstatic
1 websockify
1 thrift
1 sphinxcontrib-pecanwsme
1 sphinxcontrib-httpdomain
1 sphinxcontrib-docbookrestapi
1 rtslib-fb
1 python-ldap
1 python-barbicanclient
1 nose-exclude
1 nodeenv
1 libvirt-python
1 ldappool
1 jsonrpclib
1 glance_store
1 django-pyscss
1 croniter

=== Dependencies ===

[https://caniusepython3.com/check/4fd5dda2-b1f1-4db4-a636-67cd3276cb6a Porting status] for [https://github.com/openstack/requirements/blob/master/global-requirements.txt global-requirement.txt].

It's now possible to specify different dependencies for Python 2 and Python 3 using:
* requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
* requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
* (same for test-requirements.txt)

You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:

<pre>
...
[testenv:py33]
deps = -r{toxinidir}/requirements-py3.txt
-r{toxinidir}/test-requirements-py3.txt
...
</pre>

See also a patch to support markers in requirements (in pip): [https://github.com/pypa/pip/issues/1433 pip issue: Support markers in setup(install_requires)?]; [https://github.com/pypa/pip/pull/1472 Victor Stinner's pull request: "parse requirements in markers"]

OpenStack Dependencies:

* mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock). Examples:
** neutronclient: https://review.openstack.org/#/c/95786/
** Oslo Incubator: https://review.openstack.org/#/c/93729/

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/boto boto] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32).
|-

| [https://pypi.python.org/pypi/django-compressor django-compressor] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || Requirements upgraded: https://review.openstack.org/94357
|-

| [https://pypi.python.org/pypi/django-openstack-auth django-openstack-auth] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || ||
As of 1.1.6
|-

| [https://pypi.python.org/pypi/dnspython dnspython] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Must use the [https://pypi.python.org/pypi/dnspython3/ Python 3 version], see https://github.com/rthalley/dnspython/issues/60
|-

| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: orange;" | In the Git repo || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
|-

| [https://pypi.python.org/pypi/eventlet eventlet] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || eventlet portage to Python 3 in progress: [https://github.com/eventlet/eventlet/issues/6 eventlet issue #6: Support Python 3.3] and [https://github.com/eventlet/eventlet/pull/99 Pull request #99: Fix several issues with python3 thread patching]. Victor Stinner is working on [http://trollius.readthedocs.org/ Trollius] (asyncio for Python 2) which may replace eventlet: [http://techs.enovance.com/6562/asyncio-openstack-python3 Use the new asyncio module and Trollius in OpenStack]
|-

| [https://pypi.python.org/pypi/hacking hacking] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Cyril Roelandt patch: [https://review.openstack.org/#/c/77585/ Make hacking Python 3 compatible]
|-

| [https://pypi.python.org/pypi/jsonrpclib jsonrpclib] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || The project seems dead :(
|-

| [https://pypi.python.org/pypi/mysql-python mysql-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there.
|-

| [https://pypi.python.org/pypi/netifaces netifaces] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Patch sent by Victor Stinner (in private): [https://bitbucket.org/haypo/misc/src/tip/openstack/netifaces_python3.patch netifaces_python3.patch], Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 .
|-

| [https://pypi.python.org/pypi/nose-exclude nose-exclude] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
|-

| [https://pypi.python.org/pypi/nosehtmloutput nosehtmloutput] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* nose-exclude (tests only)
* openstack.nose-plugin
||
* https://bugs.launchpad.net/ubuntu/+source/python-nosehtmloutput/+bug/1287247
* https://review.openstack.org/#/c/80956/
|-
| [https://pypi.python.org/pypi/nosexcover nosexcover] || style="background-color:lightgreen;" | No || N/A || style="background-color: lightgreen;" | On PyPI || || Python 3 support since 1.0.9
|-
| [https://pypi.python.org/pypi/openstack.nose-plugin openstack.nose-plugin] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color:red;" | No || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || suds ||
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Portage in Progress by Victor Stinner ([https://review.openstack.org/#/dashboard/9107 dashboard])
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI (1.1.0) || ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color:lightgreen;" | Yes || No tests :) || style="background-color: lightgreen;" | In the git repo, not on PyPI || || https://review.openstack.org/#/c/79311/
|-
| [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Must be replaced by oslosphinx (without the dot)
|-

| [https://pypi.python.org/pypi/pam pam] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The fork [https://pypi.python.org/pypi/simplepam simplepam] works on Python 2 and 3
|-

| [https://pypi.python.org/pypi/paramiko paramiko] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | On PyPI || || Requirements upgraded: https://review.openstack.org/#/c/81132/
|-

| [https://pypi.python.org/pypi/paste paste] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
|-

| [https://pypi.python.org/pypi/pycadf pycadf] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Classifiers added https://review.openstack.org/#/c/133088/
|-

| [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The project seems dead.
|-

| [https://pypi.python.org/pypi/python-memcached python-memcached] || style="background-color:green;" | Not released, but last commit on git has it || style="background-color: red;" | No || style="background-color: red;" | No || || [https://github.com/jbalogh/django-cache-machine/issues/52 Issue #52: Python 3.3 support? ], [https://github.com/linsomniac/python-memcached/pull/26 Pull request #26: Python 3.3 Support] -- Julien Danjou ported [https://pypi.python.org/pypi/pymemcache pymemcache] to Python 3, another memcached client, he suggests to use this one instead
|-

| [https://pypi.python.org/pypi/qpid-python qpid-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/rtslib-fb rtslib-fb] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-docbookrestapi sphinxcontrib-docbookrestapi] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-httpdomain sphinxcontrib-httpdomain] || N/A || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-pecanwsme sphinxcontrib-pecanwsme] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No||
* hacking
* ibm-db-sa
* scripttest
||
|-
| [https://pypi.python.org/pypi/suds suds] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || "Lightweight SOAP client". Last commit 2 years ago: https://fedorahosted.org/suds/browser See also this fork which is promising: https://bitbucket.org/jurko/suds
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || ||
|-
| [https://pypi.python.org/pypi/thrift thrift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/websockify websockify] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

|}

=== Other OpenStack projects ===

Python 3 compatible:

* stackforge/python-jenkins : https://review.openstack.org/#/c/95004/ (py33 gate is voting)
* openstack-infra/jenkins-job-builder : https://review.openstack.org/87810 (Python 3 support finally merged in, Sept. 2014)

== Reports at OpenStack Summits ==

* Havana summit notes: https://etherpad.openstack.org/p/havana-python3
* Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
* Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)

== Pycon Montreal 2014: Sprint Port OpenStack to Python 3 ==

Enovance organizes a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday). See the page [[Python3/SprintPycon2014]].

Python3

2015-03-06T11:51:45Z

Lennart Regebro: /* Dependencies */ Even better

This page tracks the progress of Python 3 effort porting for OpenStack.

== Python 3 ==

[http://techs.enovance.com/6521/openstack_python3 Why should OpenStack move to Python 3 right now?]
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''

== Port Python 2 code to Python 3 ==

OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).

=== Common patterns ===

* Replace dict.iteritems() with six.iteritems(dict)
* Replace iterator.next() with next(iterator)
* Replace basestring with six.string_types
* Replace unicode with six.text_type

=== bytes.decode and unicode.encode ===

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write <code>data.decode()</code> or <code>text.encode()</code> without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: <code>data.decode('utf-8')</code> or <code>text.encode('utf-8')</code>. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

=== safe_decode ===

Olso Incubator has a function '''safe_decode()''' which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_decode(data, 'utf-8')</code>: decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:
* <code>safe_decode(data)</code>

By default, the decoder is strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

=== safe_encode ===

Olso Incubator has a function '''safe_encode()''' which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
* <code>safe_encode(text)</code> encodes text to the output encoding
* <code>safe_encode(bytes)</code> may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (<code>incomding</code> parameter) is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (<code>encoding</code> parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_encode(data, incoming='utf-8')</code>: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:
* <code>safe_encode(data)</code>

Example:
* <code>safe_encode(b'\xe9', incoming='latin-1')</code> returns <code>b'\xc3\xa9'</code>.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: <code>safe_encode(b'[\xff]', incoming='ascii', errors='ignore')</code> returns <code>b'[]'</code>.

=== logging module and format exceptions ===

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs <code>b'hello'</code> instead of <code>'hello'</code>.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

* <code>str(exc)</code>: native string, so use bytes on Python 2
* <code>six.text_type(exc)</code>: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: <code>unicode(Exception("nonascii:\xe9"))</code>.
* <code>six.u(str(exc))</code>: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: <code>unicode(str(Exception("\xff")))</code>
* <code>LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})</code>

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using <code>str(exc)</code>. Example: <code>LOG.debug(str(exc))</code>.

=== HTTP ===

The HTTP protocol is based on '''bytes''':

* HTTP body contains '''bytes'''. For example, use io.BytesIO for a stream storing an HTTP body.
* HTTPConnection.getresponse().read() returns '''bytes''' (in Python 3, '''str''' which is bytes in Python 2)
* On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
* It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and [http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding HTTP header should use what character encoding?]

=== References to port Python 2 code to Python 3 ===
* [http://python3porting.com/ Porting to Python 3 Book] by Lennart Regebro, especially the [http://python3porting.com/differences.html Language differences and workarounds].
* [http://docs.python.org/dev/howto/pyporting.html HOWTO: Porting Python 2 Code to Python 3] by Brett Cannon
* [https://wiki.python.org/moin/PortingPythonToPy3k Porting Python Code to 3.x]
* [http://code.google.com/p/python-incompatibility/ python-incompatibility]: Demonstrates incompatibilities between Python versions.

=== Common pitfalls ===

==== What is a string ? ====
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:
>>> type('foo')
<type 'str'>
>>> type(u'foo')
<type 'unicode'>
>>> type(b'foo')
<type 'str'>
>>> isinstance('foo', six.text_type)
False
>>> isinstance(u'foo', six.text_type)
True
>>> bytes is str
True
>>> b'foo'[0]
'f'

Python 3:
>>> type('foo')
<class 'str'>
>>> type(u'foo')
<class 'str'>
>>> type(b'foo')
<class 'bytes'>
>>> isinstance('foo', six.text_type)
True
>>> isinstance(b'foo', six.text_type)
False
>>> bytes is str
False
>>> b'foo'[0]
102

==== tox/testr error: db type could not be determined ====

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/".

== Python 3 Status of OpenStack projects ==

=== Oslo Incubator ===

'''BLOCKER BUG:''' Tests using testscenarios fail on Python 3 with nosetests because of this bug:
https://bugs.launchpad.net/testscenarios/+bug/872887

Recently merged reviews:

* https://review.openstack.org/#/c/79781/

{|class="wikitable sortable"
|-
! Test (full path) !! Patches !! Comment
|-

| style="background-color: lightgreen" | tests/unit/config/test_generator.py || https://review.openstack.org/#/c/88087/ ||
|-

| style="background-color: lightgreen" | tests/unit/crypto/test_utils.py || https://review.openstack.org/87413 ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migration_common.py || https://review.openstack.org/#/c/107752/ ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migrate.py || https://review.openstack.org/#/c/107921/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_models.py || https://review.openstack.org/#/c/80307/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_options.py || https://review.openstack.org/#/c/80627/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_migrate_cli.py || https://review.openstack.org/107988 ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_utils.py || ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_sqlalchemy.py || ||
|-

| style="background-color: lightgreen" | tests/unit/fixture/test_logging.py || https://review.openstack.org/#/c/90318/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_request_id.py || https://review.openstack.org/#/c/80336/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_sizelimit.py || https://review.openstack.org/#/c/80450/ ||
|-

| style="background-color: red" | tests/unit/middleware/test_audit.py || || depends on pycadf ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_guru_meditation_report.py || https://review.openstack.org/#/c/87404/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_base_report.py || https://review.openstack.org/#/c/87973/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_openstack_generators.py || https://review.openstack.org/#/c/88124/ || ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_views.py || https://review.openstack.org/87376 ||
|-

| style="background-color: lightblue" | tests/unit/rpc/test_common.py || https://review.openstack.org/#/c/80533/ || The RPC code in the incubator is deprecated in favor of oslo.messaging. --[[User:Doug-hellmann|doug-hellmann]] ([[User talk:Doug-hellmann|talk]]) 15:40, 14 April 2014 (UTC)
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_base_filter.py || https://review.openstack.org/#/c/80321/ ||
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_weights.py || https://review.openstack.org/#/c/87336/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_cliutils || https://review.openstack.org/#/c/74433/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_fileutils || https://review.openstack.org/#/c/74728/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_gettext.py || https://review.openstack.org/#/c/80534/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_imageutils.py || https://review.openstack.org/#/c/90532/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_jsonutils.py || https://review.openstack.org/#/c/80370/ ||
|-

| style="background-color: orange" | tests/unit/test_log.py || https://review.openstack.org/104890 ||
|-

| style="background-color: lightblue" | tests/unit/test_processutils.py || || processutils was moved to the new oslo.concurrency project ||
|-

| style="background-color: lightgreen" | tests/unit/test_quota.py || https://review.openstack.org/#/c/80564/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_strutils.py ||
* https://review.openstack.org/#/c/80571/ (safe_encode)
* https://review.openstack.org/#/c/80573/
||
|-

|}

=== Common Libraries (Oslo Projects) ===

For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cliff cliff] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.concurrency oslo.concurrency] || style="background-color: orange;" | Partial || https://review.openstack.org/141206
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.db oslo.db] || style="background-color: orange;" | Partial || No unit tests with MySQL because of DB driver
|-
| [https://pypi.python.org/pypi/oslo.i18n oslo.i18n] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.log oslo.log] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color: lightgreen;" | Yes || eventlet executor and qpid driver are not supported on Python 3, greenio
executor may help
|-
| [https://pypi.python.org/pypi/oslo.middleware oslo.middleware] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.serialization oslo.serialization] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || ? || The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible?
|-
| [https://pypi.python.org/pypi/oslotest oslotest] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: red;" | No || Blocked suds dependency
|-
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes ||
|-
| oslo.version || style="background-color: lightgreen;" | Yes || not released on PyPI yet
|-
| pylockfile || style="background-color: lightgreen;" | Yes || related to https://pypi.python.org/pypi/lockfile ?
|-
| [https://pypi.python.org/pypi/stevedore stevedore] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: lightgreen;" | Yes ||
|}

Development tools:

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cookiecutter cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/oslo-cookiecutter oslo-cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/hacking hacking] || ? ||
|-
| [https://pypi.python.org/pypi/pbr pbr] || ? ||
|}

=== OpenStack clients ===

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/python-barbicanclient python-barbicanclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:orange;" | In the git repo, not on PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ganttclient python-ganttclient] || ? || ? || ? || ? ||
|-

| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color:lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-melangeclient python-melangeclient] || ? || ? || ? || ||
|-

| [https://pypi.python.org/pypi/python-novaclient python-novaclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPII || ||
|-

| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: orange;" | Not released yet || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient] || style="background-color: lightgreen" | OK || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | Yes || || As of 0.9
|-

| [https://pypi.python.org/pypi/python-saharaclient python-saharaclient] || style="background-color: orange;" | In progress || style="background-color: orange" | Non-voting || || || https://review.openstack.org/#/c/73128/
|-

| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | [https://pypi.python.org/pypi/python-swiftclient/ On PyPI] || ||
|-

| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|}

=== Core OpenStack projects ===

Completely updated on Monday, September the 29th.

{| class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Trove classifiers !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* croniter
* eventlet
* thrift (which is blocking happybase)
* oslo.db
* oslo.messaging
* python-neutronclient
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* sphinxcontrib-docbookrestapi
* sphinxcontrib-httpdomain
* sphinxcontrib-pecanwsme
||
|-

| [https://pypi.python.org/pypi/cinder cinder] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* paste
* python-barbicanclient
* rtslib-fb
* sqlalchemy-migrate
* suds
Requirements for tests
* mysql-python

||
|-

| [https://pypi.python.org/pypi/glance glance] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* glance_store
* oslo.db
* oslo.messaging
* paste
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* qpid-python

||
|-

| [https://pypi.python.org/pypi/heat heat] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* python-neutronclient
* python-saharaclient
* qpid-python
* sqlalchemy-migrate
Requirements for tests
* mysql-python
||
|-

| [https://pypi.python.org/pypi/horizon horizon] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* django-pyscss
* eventlet
* python-neutronclient
* python-saharaclient
* xstatic
* xstatic-angular
* xstatic-angular-cookies
* xstatic-angular-mock
* xstatic-bootstrap-datepicker
* xstatic-d3
* xstatic-font-awesome
* xstatic-hogan
* xstatic-jasmine
* xstatic-jquery
* xstatic-jquery-migrate
* xstatic-jquery.quicksearch
* xstatic-jquery.tablesorter
* xstatic-jsencrypt
* xstatic-qunit
* xstatic-rickshaw
* xstatic-spin
Requirements for tests
* nodeenv
* nose-exclude
* nosehtmloutput
* openstack.nose_plugin
||
|-

| [https://pypi.python.org/pypi/keystone keystone] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* sqlalchemy-migrate
Requirements for tests
* ldappool
* paste (which is blocking pysaml2)
* python-ldap
||
|-

| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* jsonrpclib
* oslo.db
* oslo.messaging
* paste
* python-neutronclient
||
|-

| [https://pypi.python.org/pypi/nova nova] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* python-neutronclient
* sqlalchemy-migrate
* suds
* websockify
Requirements for tests
* libvirt-python
* mysql-python
||
|-

| [https://pypi.python.org/pypi/swift swift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
Requirements for tests
* nosehtmloutput
* openstack.nose_plugin

||
|-

|}

Number of core OpenStack projetcs blocked by each dependency:
9 eventlet
7 oslo.messaging
7 oslo.db
6 sqlalchemy-migrate
6 paste
5 python-neutronclient
5 mysql-python
2 suds
2 qpid-python
2 python-saharaclient
2 pycadf
2 openstack.nose_plugin
2 nosehtmloutput
1 xstatic-spin
1 xstatic-rickshaw
1 xstatic-qunit
1 xstatic-jsencrypt
1 xstatic-jquery.tablesorter
1 xstatic-jquery.quicksearch
1 xstatic-jquery-migrate
1 xstatic-jquery
1 xstatic-jasmine
1 xstatic-hogan
1 xstatic-font-awesome
1 xstatic-d3
1 xstatic-bootstrap-datepicker
1 xstatic-angular-mock
1 xstatic-angular-cookies
1 xstatic-angular
1 xstatic
1 websockify
1 thrift
1 sphinxcontrib-pecanwsme
1 sphinxcontrib-httpdomain
1 sphinxcontrib-docbookrestapi
1 rtslib-fb
1 python-ldap
1 python-barbicanclient
1 nose-exclude
1 nodeenv
1 libvirt-python
1 ldappool
1 jsonrpclib
1 glance_store
1 django-pyscss
1 croniter

=== Dependencies ===

[https://caniusepython3.com/check/4fd5dda2-b1f1-4db4-a636-67cd3276cb6a Porting status] for [https://github.com/openstack/requirements/blob/master/global-requirements.txt global-requirement.txt].

It's now possible to specify different dependencies for Python 2 and Python 3 using:
* requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
* requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
* (same for test-requirements.txt)

You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:

<pre>
...
[testenv:py33]
deps = -r{toxinidir}/requirements-py3.txt
-r{toxinidir}/test-requirements-py3.txt
...
</pre>

See also a patch to support markers in requirements (in pip): [https://github.com/pypa/pip/issues/1433 pip issue: Support markers in setup(install_requires)?]; [https://github.com/pypa/pip/pull/1472 Victor Stinner's pull request: "parse requirements in markers"]

OpenStack Dependencies:

* mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock). Examples:
** neutronclient: https://review.openstack.org/#/c/95786/
** Oslo Incubator: https://review.openstack.org/#/c/93729/

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/boto boto] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32).
|-

| [https://pypi.python.org/pypi/django-compressor django-compressor] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || Requirements upgraded: https://review.openstack.org/94357
|-

| [https://pypi.python.org/pypi/django-openstack-auth django-openstack-auth] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || ||
As of 1.1.6
|-

| [https://pypi.python.org/pypi/dnspython dnspython] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Must use the [https://pypi.python.org/pypi/dnspython3/ Python 3 version], see https://github.com/rthalley/dnspython/issues/60
|-

| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: orange;" | In the Git repo || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
|-

| [https://pypi.python.org/pypi/eventlet eventlet] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || eventlet portage to Python 3 in progress: [https://github.com/eventlet/eventlet/issues/6 eventlet issue #6: Support Python 3.3] and [https://github.com/eventlet/eventlet/pull/99 Pull request #99: Fix several issues with python3 thread patching]. Victor Stinner is working on [http://trollius.readthedocs.org/ Trollius] (asyncio for Python 2) which may replace eventlet: [http://techs.enovance.com/6562/asyncio-openstack-python3 Use the new asyncio module and Trollius in OpenStack]
|-

| [https://pypi.python.org/pypi/hacking hacking] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Cyril Roelandt patch: [https://review.openstack.org/#/c/77585/ Make hacking Python 3 compatible]
|-

| [https://pypi.python.org/pypi/jsonrpclib jsonrpclib] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || The project seems dead :(
|-

| [https://pypi.python.org/pypi/mysql-python mysql-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there.
|-

| [https://pypi.python.org/pypi/netifaces netifaces] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Patch sent by Victor Stinner (in private): [https://bitbucket.org/haypo/misc/src/tip/openstack/netifaces_python3.patch netifaces_python3.patch], Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 .
|-

| [https://pypi.python.org/pypi/nose-exclude nose-exclude] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
|-

| [https://pypi.python.org/pypi/nosehtmloutput nosehtmloutput] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* nose-exclude (tests only)
* openstack.nose-plugin
||
* https://bugs.launchpad.net/ubuntu/+source/python-nosehtmloutput/+bug/1287247
* https://review.openstack.org/#/c/80956/
|-
| [https://pypi.python.org/pypi/nosexcover nosexcover] || style="background-color:lightgreen;" | No || N/A || style="background-color: lightgreen;" | On PyPI || || Python 3 support since 1.0.9
|-
| [https://pypi.python.org/pypi/openstack.nose-plugin openstack.nose-plugin] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color:red;" | No || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || suds ||
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Portage in Progress by Victor Stinner ([https://review.openstack.org/#/dashboard/9107 dashboard])
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI (1.1.0) || ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color:lightgreen;" | Yes || No tests :) || style="background-color: lightgreen;" | In the git repo, not on PyPI || || https://review.openstack.org/#/c/79311/
|-
| [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Must be replaced by oslosphinx (without the dot)
|-

| [https://pypi.python.org/pypi/pam pam] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The fork [https://pypi.python.org/pypi/simplepam simplepam] works on Python 2 and 3
|-

| [https://pypi.python.org/pypi/paramiko paramiko] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | On PyPI || || Requirements upgraded: https://review.openstack.org/#/c/81132/
|-

| [https://pypi.python.org/pypi/paste paste] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
|-

| [https://pypi.python.org/pypi/pycadf pycadf] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Classifiers added https://review.openstack.org/#/c/133088/
|-

| [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The project seems dead.
|-

| [https://pypi.python.org/pypi/python-memcached python-memcached] || style="background-color:green;" | Not released, but last commit on git has it || style="background-color: red;" | No || style="background-color: red;" | No || || [https://github.com/jbalogh/django-cache-machine/issues/52 Issue #52: Python 3.3 support? ], [https://github.com/linsomniac/python-memcached/pull/26 Pull request #26: Python 3.3 Support] -- Julien Danjou ported [https://pypi.python.org/pypi/pymemcache pymemcache] to Python 3, another memcached client, he suggests to use this one instead
|-

| [https://pypi.python.org/pypi/qpid-python qpid-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/rtslib-fb rtslib-fb] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-docbookrestapi sphinxcontrib-docbookrestapi] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-httpdomain sphinxcontrib-httpdomain] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-pecanwsme sphinxcontrib-pecanwsme] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No||
* hacking
* ibm-db-sa
* scripttest
||
|-
| [https://pypi.python.org/pypi/suds suds] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || "Lightweight SOAP client". Last commit 2 years ago: https://fedorahosted.org/suds/browser See also this fork which is promising: https://bitbucket.org/jurko/suds
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || ||
|-
| [https://pypi.python.org/pypi/thrift thrift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/websockify websockify] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

|}

=== Other OpenStack projects ===

Python 3 compatible:

* stackforge/python-jenkins : https://review.openstack.org/#/c/95004/ (py33 gate is voting)
* openstack-infra/jenkins-job-builder : https://review.openstack.org/87810 (Python 3 support finally merged in, Sept. 2014)

== Reports at OpenStack Summits ==

* Havana summit notes: https://etherpad.openstack.org/p/havana-python3
* Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
* Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)

== Pycon Montreal 2014: Sprint Port OpenStack to Python 3 ==

Enovance organizes a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday). See the page [[Python3/SprintPycon2014]].

Python3

2015-03-06T11:50:07Z

Lennart Regebro: /* Dependencies */ Wrong color

This page tracks the progress of Python 3 effort porting for OpenStack.

== Python 3 ==

[http://techs.enovance.com/6521/openstack_python3 Why should OpenStack move to Python 3 right now?]
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''

== Port Python 2 code to Python 3 ==

OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).

=== Common patterns ===

* Replace dict.iteritems() with six.iteritems(dict)
* Replace iterator.next() with next(iterator)
* Replace basestring with six.string_types
* Replace unicode with six.text_type

=== bytes.decode and unicode.encode ===

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write <code>data.decode()</code> or <code>text.encode()</code> without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: <code>data.decode('utf-8')</code> or <code>text.encode('utf-8')</code>. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

=== safe_decode ===

Olso Incubator has a function '''safe_decode()''' which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_decode(data, 'utf-8')</code>: decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:
* <code>safe_decode(data)</code>

By default, the decoder is strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

=== safe_encode ===

Olso Incubator has a function '''safe_encode()''' which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
* <code>safe_encode(text)</code> encodes text to the output encoding
* <code>safe_encode(bytes)</code> may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (<code>incomding</code> parameter) is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (<code>encoding</code> parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_encode(data, incoming='utf-8')</code>: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:
* <code>safe_encode(data)</code>

Example:
* <code>safe_encode(b'\xe9', incoming='latin-1')</code> returns <code>b'\xc3\xa9'</code>.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: <code>safe_encode(b'[\xff]', incoming='ascii', errors='ignore')</code> returns <code>b'[]'</code>.

=== logging module and format exceptions ===

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs <code>b'hello'</code> instead of <code>'hello'</code>.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

* <code>str(exc)</code>: native string, so use bytes on Python 2
* <code>six.text_type(exc)</code>: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: <code>unicode(Exception("nonascii:\xe9"))</code>.
* <code>six.u(str(exc))</code>: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: <code>unicode(str(Exception("\xff")))</code>
* <code>LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})</code>

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using <code>str(exc)</code>. Example: <code>LOG.debug(str(exc))</code>.

=== HTTP ===

The HTTP protocol is based on '''bytes''':

* HTTP body contains '''bytes'''. For example, use io.BytesIO for a stream storing an HTTP body.
* HTTPConnection.getresponse().read() returns '''bytes''' (in Python 3, '''str''' which is bytes in Python 2)
* On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
* It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and [http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding HTTP header should use what character encoding?]

=== References to port Python 2 code to Python 3 ===
* [http://python3porting.com/ Porting to Python 3 Book] by Lennart Regebro, especially the [http://python3porting.com/differences.html Language differences and workarounds].
* [http://docs.python.org/dev/howto/pyporting.html HOWTO: Porting Python 2 Code to Python 3] by Brett Cannon
* [https://wiki.python.org/moin/PortingPythonToPy3k Porting Python Code to 3.x]
* [http://code.google.com/p/python-incompatibility/ python-incompatibility]: Demonstrates incompatibilities between Python versions.

=== Common pitfalls ===

==== What is a string ? ====
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:
>>> type('foo')
<type 'str'>
>>> type(u'foo')
<type 'unicode'>
>>> type(b'foo')
<type 'str'>
>>> isinstance('foo', six.text_type)
False
>>> isinstance(u'foo', six.text_type)
True
>>> bytes is str
True
>>> b'foo'[0]
'f'

Python 3:
>>> type('foo')
<class 'str'>
>>> type(u'foo')
<class 'str'>
>>> type(b'foo')
<class 'bytes'>
>>> isinstance('foo', six.text_type)
True
>>> isinstance(b'foo', six.text_type)
False
>>> bytes is str
False
>>> b'foo'[0]
102

==== tox/testr error: db type could not be determined ====

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/".

== Python 3 Status of OpenStack projects ==

=== Oslo Incubator ===

'''BLOCKER BUG:''' Tests using testscenarios fail on Python 3 with nosetests because of this bug:
https://bugs.launchpad.net/testscenarios/+bug/872887

Recently merged reviews:

* https://review.openstack.org/#/c/79781/

{|class="wikitable sortable"
|-
! Test (full path) !! Patches !! Comment
|-

| style="background-color: lightgreen" | tests/unit/config/test_generator.py || https://review.openstack.org/#/c/88087/ ||
|-

| style="background-color: lightgreen" | tests/unit/crypto/test_utils.py || https://review.openstack.org/87413 ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migration_common.py || https://review.openstack.org/#/c/107752/ ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migrate.py || https://review.openstack.org/#/c/107921/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_models.py || https://review.openstack.org/#/c/80307/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_options.py || https://review.openstack.org/#/c/80627/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_migrate_cli.py || https://review.openstack.org/107988 ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_utils.py || ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_sqlalchemy.py || ||
|-

| style="background-color: lightgreen" | tests/unit/fixture/test_logging.py || https://review.openstack.org/#/c/90318/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_request_id.py || https://review.openstack.org/#/c/80336/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_sizelimit.py || https://review.openstack.org/#/c/80450/ ||
|-

| style="background-color: red" | tests/unit/middleware/test_audit.py || || depends on pycadf ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_guru_meditation_report.py || https://review.openstack.org/#/c/87404/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_base_report.py || https://review.openstack.org/#/c/87973/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_openstack_generators.py || https://review.openstack.org/#/c/88124/ || ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_views.py || https://review.openstack.org/87376 ||
|-

| style="background-color: lightblue" | tests/unit/rpc/test_common.py || https://review.openstack.org/#/c/80533/ || The RPC code in the incubator is deprecated in favor of oslo.messaging. --[[User:Doug-hellmann|doug-hellmann]] ([[User talk:Doug-hellmann|talk]]) 15:40, 14 April 2014 (UTC)
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_base_filter.py || https://review.openstack.org/#/c/80321/ ||
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_weights.py || https://review.openstack.org/#/c/87336/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_cliutils || https://review.openstack.org/#/c/74433/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_fileutils || https://review.openstack.org/#/c/74728/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_gettext.py || https://review.openstack.org/#/c/80534/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_imageutils.py || https://review.openstack.org/#/c/90532/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_jsonutils.py || https://review.openstack.org/#/c/80370/ ||
|-

| style="background-color: orange" | tests/unit/test_log.py || https://review.openstack.org/104890 ||
|-

| style="background-color: lightblue" | tests/unit/test_processutils.py || || processutils was moved to the new oslo.concurrency project ||
|-

| style="background-color: lightgreen" | tests/unit/test_quota.py || https://review.openstack.org/#/c/80564/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_strutils.py ||
* https://review.openstack.org/#/c/80571/ (safe_encode)
* https://review.openstack.org/#/c/80573/
||
|-

|}

=== Common Libraries (Oslo Projects) ===

For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cliff cliff] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.concurrency oslo.concurrency] || style="background-color: orange;" | Partial || https://review.openstack.org/141206
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.db oslo.db] || style="background-color: orange;" | Partial || No unit tests with MySQL because of DB driver
|-
| [https://pypi.python.org/pypi/oslo.i18n oslo.i18n] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.log oslo.log] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color: lightgreen;" | Yes || eventlet executor and qpid driver are not supported on Python 3, greenio
executor may help
|-
| [https://pypi.python.org/pypi/oslo.middleware oslo.middleware] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.serialization oslo.serialization] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || ? || The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible?
|-
| [https://pypi.python.org/pypi/oslotest oslotest] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: red;" | No || Blocked suds dependency
|-
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes ||
|-
| oslo.version || style="background-color: lightgreen;" | Yes || not released on PyPI yet
|-
| pylockfile || style="background-color: lightgreen;" | Yes || related to https://pypi.python.org/pypi/lockfile ?
|-
| [https://pypi.python.org/pypi/stevedore stevedore] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: lightgreen;" | Yes ||
|}

Development tools:

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cookiecutter cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/oslo-cookiecutter oslo-cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/hacking hacking] || ? ||
|-
| [https://pypi.python.org/pypi/pbr pbr] || ? ||
|}

=== OpenStack clients ===

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/python-barbicanclient python-barbicanclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:orange;" | In the git repo, not on PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ganttclient python-ganttclient] || ? || ? || ? || ? ||
|-

| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color:lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-melangeclient python-melangeclient] || ? || ? || ? || ||
|-

| [https://pypi.python.org/pypi/python-novaclient python-novaclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPII || ||
|-

| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: orange;" | Not released yet || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient] || style="background-color: lightgreen" | OK || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | Yes || || As of 0.9
|-

| [https://pypi.python.org/pypi/python-saharaclient python-saharaclient] || style="background-color: orange;" | In progress || style="background-color: orange" | Non-voting || || || https://review.openstack.org/#/c/73128/
|-

| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | [https://pypi.python.org/pypi/python-swiftclient/ On PyPI] || ||
|-

| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|}

=== Core OpenStack projects ===

Completely updated on Monday, September the 29th.

{| class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Trove classifiers !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* croniter
* eventlet
* thrift (which is blocking happybase)
* oslo.db
* oslo.messaging
* python-neutronclient
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* sphinxcontrib-docbookrestapi
* sphinxcontrib-httpdomain
* sphinxcontrib-pecanwsme
||
|-

| [https://pypi.python.org/pypi/cinder cinder] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* paste
* python-barbicanclient
* rtslib-fb
* sqlalchemy-migrate
* suds
Requirements for tests
* mysql-python

||
|-

| [https://pypi.python.org/pypi/glance glance] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* glance_store
* oslo.db
* oslo.messaging
* paste
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* qpid-python

||
|-

| [https://pypi.python.org/pypi/heat heat] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* python-neutronclient
* python-saharaclient
* qpid-python
* sqlalchemy-migrate
Requirements for tests
* mysql-python
||
|-

| [https://pypi.python.org/pypi/horizon horizon] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* django-pyscss
* eventlet
* python-neutronclient
* python-saharaclient
* xstatic
* xstatic-angular
* xstatic-angular-cookies
* xstatic-angular-mock
* xstatic-bootstrap-datepicker
* xstatic-d3
* xstatic-font-awesome
* xstatic-hogan
* xstatic-jasmine
* xstatic-jquery
* xstatic-jquery-migrate
* xstatic-jquery.quicksearch
* xstatic-jquery.tablesorter
* xstatic-jsencrypt
* xstatic-qunit
* xstatic-rickshaw
* xstatic-spin
Requirements for tests
* nodeenv
* nose-exclude
* nosehtmloutput
* openstack.nose_plugin
||
|-

| [https://pypi.python.org/pypi/keystone keystone] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* sqlalchemy-migrate
Requirements for tests
* ldappool
* paste (which is blocking pysaml2)
* python-ldap
||
|-

| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* jsonrpclib
* oslo.db
* oslo.messaging
* paste
* python-neutronclient
||
|-

| [https://pypi.python.org/pypi/nova nova] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* python-neutronclient
* sqlalchemy-migrate
* suds
* websockify
Requirements for tests
* libvirt-python
* mysql-python
||
|-

| [https://pypi.python.org/pypi/swift swift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
Requirements for tests
* nosehtmloutput
* openstack.nose_plugin

||
|-

|}

Number of core OpenStack projetcs blocked by each dependency:
9 eventlet
7 oslo.messaging
7 oslo.db
6 sqlalchemy-migrate
6 paste
5 python-neutronclient
5 mysql-python
2 suds
2 qpid-python
2 python-saharaclient
2 pycadf
2 openstack.nose_plugin
2 nosehtmloutput
1 xstatic-spin
1 xstatic-rickshaw
1 xstatic-qunit
1 xstatic-jsencrypt
1 xstatic-jquery.tablesorter
1 xstatic-jquery.quicksearch
1 xstatic-jquery-migrate
1 xstatic-jquery
1 xstatic-jasmine
1 xstatic-hogan
1 xstatic-font-awesome
1 xstatic-d3
1 xstatic-bootstrap-datepicker
1 xstatic-angular-mock
1 xstatic-angular-cookies
1 xstatic-angular
1 xstatic
1 websockify
1 thrift
1 sphinxcontrib-pecanwsme
1 sphinxcontrib-httpdomain
1 sphinxcontrib-docbookrestapi
1 rtslib-fb
1 python-ldap
1 python-barbicanclient
1 nose-exclude
1 nodeenv
1 libvirt-python
1 ldappool
1 jsonrpclib
1 glance_store
1 django-pyscss
1 croniter

=== Dependencies ===

[https://caniusepython3.com/check/4fd5dda2-b1f1-4db4-a636-67cd3276cb6a Porting status] for [https://github.com/openstack/requirements/blob/master/global-requirements.txt global-requirement.txt].

It's now possible to specify different dependencies for Python 2 and Python 3 using:
* requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
* requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
* (same for test-requirements.txt)

You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:

<pre>
...
[testenv:py33]
deps = -r{toxinidir}/requirements-py3.txt
-r{toxinidir}/test-requirements-py3.txt
...
</pre>

See also a patch to support markers in requirements (in pip): [https://github.com/pypa/pip/issues/1433 pip issue: Support markers in setup(install_requires)?]; [https://github.com/pypa/pip/pull/1472 Victor Stinner's pull request: "parse requirements in markers"]

OpenStack Dependencies:

* mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock). Examples:
** neutronclient: https://review.openstack.org/#/c/95786/
** Oslo Incubator: https://review.openstack.org/#/c/93729/

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/boto boto] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32).
|-

| [https://pypi.python.org/pypi/django-compressor django-compressor] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || Requirements upgraded: https://review.openstack.org/94357
|-

| [https://pypi.python.org/pypi/django-openstack-auth django-openstack-auth] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || ||
As of 1.1.6
|-

| [https://pypi.python.org/pypi/dnspython dnspython] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Must use the [https://pypi.python.org/pypi/dnspython3/ Python 3 version], see https://github.com/rthalley/dnspython/issues/60
|-

| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: orange;" | In the Git repo || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
|-

| [https://pypi.python.org/pypi/eventlet eventlet] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || eventlet portage to Python 3 in progress: [https://github.com/eventlet/eventlet/issues/6 eventlet issue #6: Support Python 3.3] and [https://github.com/eventlet/eventlet/pull/99 Pull request #99: Fix several issues with python3 thread patching]. Victor Stinner is working on [http://trollius.readthedocs.org/ Trollius] (asyncio for Python 2) which may replace eventlet: [http://techs.enovance.com/6562/asyncio-openstack-python3 Use the new asyncio module and Trollius in OpenStack]
|-

| [https://pypi.python.org/pypi/hacking hacking] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Cyril Roelandt patch: [https://review.openstack.org/#/c/77585/ Make hacking Python 3 compatible]
|-

| [https://pypi.python.org/pypi/jsonrpclib jsonrpclib] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || The project seems dead :(
|-

| [https://pypi.python.org/pypi/mysql-python mysql-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there.
|-

| [https://pypi.python.org/pypi/netifaces netifaces] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Patch sent by Victor Stinner (in private): [https://bitbucket.org/haypo/misc/src/tip/openstack/netifaces_python3.patch netifaces_python3.patch], Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 .
|-

| [https://pypi.python.org/pypi/nose-exclude nose-exclude] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
|-

| [https://pypi.python.org/pypi/nosehtmloutput nosehtmloutput] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* nose-exclude (tests only)
* openstack.nose-plugin
||
* https://bugs.launchpad.net/ubuntu/+source/python-nosehtmloutput/+bug/1287247
* https://review.openstack.org/#/c/80956/
|-
| [https://pypi.python.org/pypi/nosexcover nosexcover] || style="background-color:lightgreen;" | No || N/A || style="background-color: lightgreen;" | On PyPI || || Python 3 support since 1.0.9
|-
| [https://pypi.python.org/pypi/openstack.nose-plugin openstack.nose-plugin] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color:red;" | No || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || suds ||
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Portage in Progress by Victor Stinner ([https://review.openstack.org/#/dashboard/9107 dashboard])
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI (1.1.0) || ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color:lightgreen;" | Yes || No tests :) || style="background-color: lightgreen;" | In the git repo, not on PyPI || || https://review.openstack.org/#/c/79311/
|-
| [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Must be replaced by oslosphinx (without the dot)
|-

| [https://pypi.python.org/pypi/pam pam] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The fork [https://pypi.python.org/pypi/simplepam simplepam] works on Python 2 and 3
|-

| [https://pypi.python.org/pypi/paramiko paramiko] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | On PyPI || || Requirements upgraded: https://review.openstack.org/#/c/81132/
|-

| [https://pypi.python.org/pypi/paste paste] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
|-

| [https://pypi.python.org/pypi/pycadf pycadf] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Classifiers added https://review.openstack.org/#/c/133088/
|-

| [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The project seems dead.
|-

| [https://pypi.python.org/pypi/python-memcached python-memcached] || style="background-color:green;" | Not released, but last commit on git has it || style="background-color: red;" | No || style="background-color: red;" | No || || [https://github.com/jbalogh/django-cache-machine/issues/52 Issue #52: Python 3.3 support? ], [https://github.com/linsomniac/python-memcached/pull/26 Pull request #26: Python 3.3 Support] -- Julien Danjou ported [https://pypi.python.org/pypi/pymemcache pymemcache] to Python 3, another memcached client, he suggests to use this one instead
|-

| [https://pypi.python.org/pypi/qpid-python qpid-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/rtslib-fb rtslib-fb] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-docbookrestapi sphinxcontrib-docbookrestapi] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || || Support for 3.3, but not for 3.4, PyPI not updated.
|-
| [https://pypi.python.org/pypi/sphinxcontrib-httpdomain sphinxcontrib-httpdomain] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-pecanwsme sphinxcontrib-pecanwsme] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No||
* hacking
* ibm-db-sa
* scripttest
||
|-
| [https://pypi.python.org/pypi/suds suds] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || "Lightweight SOAP client". Last commit 2 years ago: https://fedorahosted.org/suds/browser See also this fork which is promising: https://bitbucket.org/jurko/suds
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || ||
|-
| [https://pypi.python.org/pypi/thrift thrift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/websockify websockify] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

|}

=== Other OpenStack projects ===

Python 3 compatible:

* stackforge/python-jenkins : https://review.openstack.org/#/c/95004/ (py33 gate is voting)
* openstack-infra/jenkins-job-builder : https://review.openstack.org/87810 (Python 3 support finally merged in, Sept. 2014)

== Reports at OpenStack Summits ==

* Havana summit notes: https://etherpad.openstack.org/p/havana-python3
* Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
* Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)

== Pycon Montreal 2014: Sprint Port OpenStack to Python 3 ==

Enovance organizes a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday). See the page [[Python3/SprintPycon2014]].

Python3

2015-03-06T11:49:27Z

Lennart Regebro: /* Dependencies */

This page tracks the progress of Python 3 effort porting for OpenStack.

== Python 3 ==

[http://techs.enovance.com/6521/openstack_python3 Why should OpenStack move to Python 3 right now?]
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''

== Port Python 2 code to Python 3 ==

OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions. OpenStack supported Python 2.6 for RHEL up to Juno, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).

=== Common patterns ===

* Replace dict.iteritems() with six.iteritems(dict)
* Replace iterator.next() with next(iterator)
* Replace basestring with six.string_types
* Replace unicode with six.text_type

=== bytes.decode and unicode.encode ===

Python has a notion of "default encoding": sys.getdefaultencoding(). On Python 2, the default encoding is ASCII, whereas it is UTF-8 on Python 3.

Don't write <code>data.decode()</code> or <code>text.encode()</code> without parameter, because you will use a different encoding on Python 2 and Python 3.

Use an explicit encoding instead. Example: <code>data.decode('utf-8')</code> or <code>text.encode('utf-8')</code>. The right encoding depends on the use case, but UTF-8 is usually a good candidate (it is a superset of ASCII).

=== safe_decode ===

Olso Incubator has a function '''safe_decode()''' which can be used to decode a bytes string and pass text strings unchanged.

The default encoding is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

It's safer to explicit the encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_decode(data, 'utf-8')</code>: decode bytes from UTF-8 or returns data unchanged if it's already a text string

Unsafe usage:
* <code>safe_decode(data)</code>

By default, the decoder is strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: safe_decode(b'[\xff]', 'ascii', 'ignore') returns '[]'.

=== safe_encode ===

Olso Incubator has a function '''safe_encode()''' which can be used to encode a string. Its usage is tricky and you should understand how it works and which encodings are used.
* <code>safe_encode(text)</code> encodes text to the output encoding
* <code>safe_encode(bytes)</code> may decode the string and then reencode to a different encoding if input and output encodings are different

The default input encoding (<code>incomding</code> parameter) is <code>sys.stdin.encoding or sys.getdefaultencoding()</code>:
* Python 3: the locale encoding, or UTF-8 if sys.stdin is "mocked" (io.StringIO instance)
* Python 2: the locale encoding, or ASCII if stdin is not a TTY or if sys.stdin is "mocked" (StringIO.StringIO instance)

The default output encoding (<code>encoding</code> parameter) is UTF-8.

It's safer to explicit the input encoding to not rely on the locale encoding and have the same behaviour even if sys.stdin is "mocked".

Safe usage:
* <code>safe_encode(data, incoming='utf-8')</code>: encode text to UTF-8 or returns data unchanged if it's already a bytes string (since the input and output encoding are UTF-8)

Unsafe usage:
* <code>safe_encode(data)</code>

Example:
* <code>safe_encode(b'\xe9', incoming='latin-1')</code> returns <code>b'\xc3\xa9'</code>.

By default, the encoder and the decoder are strict. You can specify a different error handler using the optional <code>errors</code> parameter. Example: <code>safe_encode(b'[\xff]', incoming='ascii', errors='ignore')</code> returns <code>b'[]'</code>.

=== logging module and format exceptions ===

On Python 2, the logging module accepts bytes and text strings. On Python 3, it only accepts text strings. For example, logging.error(b'hello') logs <code>b'hello'</code> instead of <code>'hello'</code>.

There is no clear rule for format exceptions yet. There are different choices depending on the project:

* <code>str(exc)</code>: native string, so use bytes on Python 2
* <code>six.text_type(exc)</code>: always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: <code>unicode(Exception("nonascii:\xe9"))</code>.
* <code>six.u(str(exc))</code>: unsafe on Python 2 if str(exc) contains non-ASCII bytes, ex: <code>unicode(str(Exception("\xff")))</code>
* <code>LOG.exception(_LE("... %(exc)s ..."), {"exc": exc, ...})</code>

Since logging functions expect text strings on Python 3, logged exceptions should be formatted using <code>str(exc)</code>. Example: <code>LOG.debug(str(exc))</code>.

=== HTTP ===

The HTTP protocol is based on '''bytes''':

* HTTP body contains '''bytes'''. For example, use io.BytesIO for a stream storing an HTTP body.
* HTTPConnection.getresponse().read() returns '''bytes''' (in Python 3, '''str''' which is bytes in Python 2)
* On Python 3, the http.client accepts text for HTTP headers: keys are encoded to ASCII and values to ISO 8859-1 (which is only a small subset of the Unicode charset)
* It looks like Swift encodes internally HTTP headers to UTF-8 (directly using the UTF-8 encoding, not using a MIME encoding like =?UTF-8?Q?...?=. See the HTTP [RFC 2047 http://www.ietf.org/rfc/rfc2047.txt] and [http://stackoverflow.com/questions/4400678/http-header-should-use-what-character-encoding HTTP header should use what character encoding?]

=== References to port Python 2 code to Python 3 ===
* [http://python3porting.com/ Porting to Python 3 Book] by Lennart Regebro, especially the [http://python3porting.com/differences.html Language differences and workarounds].
* [http://docs.python.org/dev/howto/pyporting.html HOWTO: Porting Python 2 Code to Python 3] by Brett Cannon
* [https://wiki.python.org/moin/PortingPythonToPy3k Porting Python Code to 3.x]
* [http://code.google.com/p/python-incompatibility/ python-incompatibility]: Demonstrates incompatibilities between Python versions.

=== Common pitfalls ===

==== What is a string ? ====
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:
>>> type('foo')
<type 'str'>
>>> type(u'foo')
<type 'unicode'>
>>> type(b'foo')
<type 'str'>
>>> isinstance('foo', six.text_type)
False
>>> isinstance(u'foo', six.text_type)
True
>>> bytes is str
True
>>> b'foo'[0]
'f'

Python 3:
>>> type('foo')
<class 'str'>
>>> type(u'foo')
<class 'str'>
>>> type(b'foo')
<class 'bytes'>
>>> isinstance('foo', six.text_type)
True
>>> isinstance(b'foo', six.text_type)
False
>>> bytes is str
False
>>> b'foo'[0]
102

==== tox/testr error: db type could not be determined ====

The "db type could not be determined" error comes from .testrepository/times.dbm used by testr.

Workaround: "rm -rf .testrepository/".

== Python 3 Status of OpenStack projects ==

=== Oslo Incubator ===

'''BLOCKER BUG:''' Tests using testscenarios fail on Python 3 with nosetests because of this bug:
https://bugs.launchpad.net/testscenarios/+bug/872887

Recently merged reviews:

* https://review.openstack.org/#/c/79781/

{|class="wikitable sortable"
|-
! Test (full path) !! Patches !! Comment
|-

| style="background-color: lightgreen" | tests/unit/config/test_generator.py || https://review.openstack.org/#/c/88087/ ||
|-

| style="background-color: lightgreen" | tests/unit/crypto/test_utils.py || https://review.openstack.org/87413 ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migration_common.py || https://review.openstack.org/#/c/107752/ ||
|-

| style="background-color: orange" | tests/unit/db/sqlalchemy/test_migrate.py || https://review.openstack.org/#/c/107921/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_models.py || https://review.openstack.org/#/c/80307/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_options.py || https://review.openstack.org/#/c/80627/ ||
|-

| style="background-color: lightgreen" | tests/unit/db/sqlalchemy/test_migrate_cli.py || https://review.openstack.org/107988 ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_utils.py || ||
|-

| style="background-color: red" | tests/unit/db/sqlalchemy/test_sqlalchemy.py || ||
|-

| style="background-color: lightgreen" | tests/unit/fixture/test_logging.py || https://review.openstack.org/#/c/90318/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_request_id.py || https://review.openstack.org/#/c/80336/ ||
|-

| style="background-color: lightgreen" | tests/unit/middleware/test_sizelimit.py || https://review.openstack.org/#/c/80450/ ||
|-

| style="background-color: red" | tests/unit/middleware/test_audit.py || || depends on pycadf ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_guru_meditation_report.py || https://review.openstack.org/#/c/87404/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_base_report.py || https://review.openstack.org/#/c/87973/ ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_openstack_generators.py || https://review.openstack.org/#/c/88124/ || ||
|-

| style="background-color: lightgreen" | tests/unit/reports/test_views.py || https://review.openstack.org/87376 ||
|-

| style="background-color: lightblue" | tests/unit/rpc/test_common.py || https://review.openstack.org/#/c/80533/ || The RPC code in the incubator is deprecated in favor of oslo.messaging. --[[User:Doug-hellmann|doug-hellmann]] ([[User talk:Doug-hellmann|talk]]) 15:40, 14 April 2014 (UTC)
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_base_filter.py || https://review.openstack.org/#/c/80321/ ||
|-

| style="background-color: lightgreen" | tests/unit/scheduler/test_weights.py || https://review.openstack.org/#/c/87336/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_cliutils || https://review.openstack.org/#/c/74433/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_fileutils || https://review.openstack.org/#/c/74728/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_gettext.py || https://review.openstack.org/#/c/80534/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_imageutils.py || https://review.openstack.org/#/c/90532/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_jsonutils.py || https://review.openstack.org/#/c/80370/ ||
|-

| style="background-color: orange" | tests/unit/test_log.py || https://review.openstack.org/104890 ||
|-

| style="background-color: lightblue" | tests/unit/test_processutils.py || || processutils was moved to the new oslo.concurrency project ||
|-

| style="background-color: lightgreen" | tests/unit/test_quota.py || https://review.openstack.org/#/c/80564/ ||
|-

| style="background-color: lightgreen" | tests/unit/test_strutils.py ||
* https://review.openstack.org/#/c/80571/ (safe_encode)
* https://review.openstack.org/#/c/80573/
||
|-

|}

=== Common Libraries (Oslo Projects) ===

For the list of Common Libraries, see http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n160

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cliff cliff] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.concurrency oslo.concurrency] || style="background-color: orange;" | Partial || https://review.openstack.org/141206
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.db oslo.db] || style="background-color: orange;" | Partial || No unit tests with MySQL because of DB driver
|-
| [https://pypi.python.org/pypi/oslo.i18n oslo.i18n] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.log oslo.log] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color: lightgreen;" | Yes || eventlet executor and qpid driver are not supported on Python 3, greenio
executor may help
|-
| [https://pypi.python.org/pypi/oslo.middleware oslo.middleware] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.serialization oslo.serialization] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || ? || The project only contains two short .py files, it looks to be Python 3 compatible. Is Sphinx Python 3 compatible?
|-
| [https://pypi.python.org/pypi/oslotest oslotest] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.versionedobjects oslo.versionedobjects] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color: red;" | No || Blocked suds dependency
|-
| [https://pypi.python.org/pypi/oslo.utils oslo.utils] || style="background-color: lightgreen;" | Yes ||
|-
| oslo.version || style="background-color: lightgreen;" | Yes || not released on PyPI yet
|-
| pylockfile || style="background-color: lightgreen;" | Yes || related to https://pypi.python.org/pypi/lockfile ?
|-
| [https://pypi.python.org/pypi/stevedore stevedore] || style="background-color: lightgreen;" | Yes ||
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color: lightgreen;" | Yes ||
|}

Development tools:

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! Comment
|-
| [https://pypi.python.org/pypi/cookiecutter cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/oslo-cookiecutter oslo-cookiecutter] || ? ||
|-
| [https://pypi.python.org/pypi/hacking hacking] || ? ||
|-
| [https://pypi.python.org/pypi/pbr pbr] || ? ||
|}

=== OpenStack clients ===

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/python-barbicanclient python-barbicanclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:orange;" | In the git repo, not on PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ganttclient python-ganttclient] || ? || ? || ? || ? ||
|-

| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color:lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-melangeclient python-melangeclient] || ? || ? || ? || ||
|-

| [https://pypi.python.org/pypi/python-novaclient python-novaclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPII || ||
|-

| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: orange;" | Not released yet || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || ||
|-

| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient] || style="background-color: lightgreen" | OK || style="background-color: lightgreen" | Voting || style="background-color: lightgreen" | Yes || || As of 0.9
|-

| [https://pypi.python.org/pypi/python-saharaclient python-saharaclient] || style="background-color: orange;" | In progress || style="background-color: orange" | Non-voting || || || https://review.openstack.org/#/c/73128/
|-

| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | [https://pypi.python.org/pypi/python-swiftclient/ On PyPI] || ||
|-

| [https://pypi.python.org/pypi/python-tuskarclient python-tuskarclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-

| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen" | On PyPI || ||
|}

=== Core OpenStack projects ===

Completely updated on Monday, September the 29th.

{| class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Trove classifiers !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* croniter
* eventlet
* thrift (which is blocking happybase)
* oslo.db
* oslo.messaging
* python-neutronclient
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* sphinxcontrib-docbookrestapi
* sphinxcontrib-httpdomain
* sphinxcontrib-pecanwsme
||
|-

| [https://pypi.python.org/pypi/cinder cinder] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* paste
* python-barbicanclient
* rtslib-fb
* sqlalchemy-migrate
* suds
Requirements for tests
* mysql-python

||
|-

| [https://pypi.python.org/pypi/glance glance] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* glance_store
* oslo.db
* oslo.messaging
* paste
* sqlalchemy-migrate
Requirements for tests
* mysql-python
* qpid-python

||
|-

| [https://pypi.python.org/pypi/heat heat] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* oslo.db
* oslo.messaging
* python-neutronclient
* python-saharaclient
* qpid-python
* sqlalchemy-migrate
Requirements for tests
* mysql-python
||
|-

| [https://pypi.python.org/pypi/horizon horizon] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* django-pyscss
* eventlet
* python-neutronclient
* python-saharaclient
* xstatic
* xstatic-angular
* xstatic-angular-cookies
* xstatic-angular-mock
* xstatic-bootstrap-datepicker
* xstatic-d3
* xstatic-font-awesome
* xstatic-hogan
* xstatic-jasmine
* xstatic-jquery
* xstatic-jquery-migrate
* xstatic-jquery.quicksearch
* xstatic-jquery.tablesorter
* xstatic-jsencrypt
* xstatic-qunit
* xstatic-rickshaw
* xstatic-spin
Requirements for tests
* nodeenv
* nose-exclude
* nosehtmloutput
* openstack.nose_plugin
||
|-

| [https://pypi.python.org/pypi/keystone keystone] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* sqlalchemy-migrate
Requirements for tests
* ldappool
* paste (which is blocking pysaml2)
* python-ldap
||
|-

| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
* jsonrpclib
* oslo.db
* oslo.messaging
* paste
* python-neutronclient
||
|-

| [https://pypi.python.org/pypi/nova nova] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* oslo.db
* oslo.messaging
* paste
* pycadf
* python-neutronclient
* sqlalchemy-migrate
* suds
* websockify
Requirements for tests
* libvirt-python
* mysql-python
||
|-

| [https://pypi.python.org/pypi/swift swift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
Requirements
* eventlet
Requirements for tests
* nosehtmloutput
* openstack.nose_plugin

||
|-

|}

Number of core OpenStack projetcs blocked by each dependency:
9 eventlet
7 oslo.messaging
7 oslo.db
6 sqlalchemy-migrate
6 paste
5 python-neutronclient
5 mysql-python
2 suds
2 qpid-python
2 python-saharaclient
2 pycadf
2 openstack.nose_plugin
2 nosehtmloutput
1 xstatic-spin
1 xstatic-rickshaw
1 xstatic-qunit
1 xstatic-jsencrypt
1 xstatic-jquery.tablesorter
1 xstatic-jquery.quicksearch
1 xstatic-jquery-migrate
1 xstatic-jquery
1 xstatic-jasmine
1 xstatic-hogan
1 xstatic-font-awesome
1 xstatic-d3
1 xstatic-bootstrap-datepicker
1 xstatic-angular-mock
1 xstatic-angular-cookies
1 xstatic-angular
1 xstatic
1 websockify
1 thrift
1 sphinxcontrib-pecanwsme
1 sphinxcontrib-httpdomain
1 sphinxcontrib-docbookrestapi
1 rtslib-fb
1 python-ldap
1 python-barbicanclient
1 nose-exclude
1 nodeenv
1 libvirt-python
1 ldappool
1 jsonrpclib
1 glance_store
1 django-pyscss
1 croniter

=== Dependencies ===

[https://caniusepython3.com/check/4fd5dda2-b1f1-4db4-a636-67cd3276cb6a Porting status] for [https://github.com/openstack/requirements/blob/master/global-requirements.txt global-requirement.txt].

It's now possible to specify different dependencies for Python 2 and Python 3 using:
* requirements-py2.txt: all dependencies for Python 2 (not only dependencies specific to Python 2)
* requirements-py3.txt: all dependencies for Python 3 (not only dependencies specific to Python 3)
* (same for test-requirements.txt)

You have to edit tox.ini to specify the right requirements file. Extract of a tox.ini file:

<pre>
...
[testenv:py33]
deps = -r{toxinidir}/requirements-py3.txt
-r{toxinidir}/test-requirements-py3.txt
...
</pre>

See also a patch to support markers in requirements (in pip): [https://github.com/pypa/pip/issues/1433 pip issue: Support markers in setup(install_requires)?]; [https://github.com/pypa/pip/pull/1472 Victor Stinner's pull request: "parse requirements in markers"]

OpenStack Dependencies:

* mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock). Examples:
** neutronclient: https://review.openstack.org/#/c/95786/
** Oslo Incubator: https://review.openstack.org/#/c/93729/

{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/boto boto] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || See https://github.com/boto/boto3 (experimental) <- This seems dead, and https://github.com/boto/boto works with Python 3.x (since 2.32).
|-

| [https://pypi.python.org/pypi/django-compressor django-compressor] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || || Requirements upgraded: https://review.openstack.org/94357
|-

| [https://pypi.python.org/pypi/django-openstack-auth django-openstack-auth] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | Yes || ||
As of 1.1.6
|-

| [https://pypi.python.org/pypi/dnspython dnspython] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Must use the [https://pypi.python.org/pypi/dnspython3/ Python 3 version], see https://github.com/rthalley/dnspython/issues/60
|-

| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: orange;" | In the Git repo || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
|-

| [https://pypi.python.org/pypi/eventlet eventlet] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || eventlet portage to Python 3 in progress: [https://github.com/eventlet/eventlet/issues/6 eventlet issue #6: Support Python 3.3] and [https://github.com/eventlet/eventlet/pull/99 Pull request #99: Fix several issues with python3 thread patching]. Victor Stinner is working on [http://trollius.readthedocs.org/ Trollius] (asyncio for Python 2) which may replace eventlet: [http://techs.enovance.com/6562/asyncio-openstack-python3 Use the new asyncio module and Trollius in OpenStack]
|-

| [https://pypi.python.org/pypi/hacking hacking] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Cyril Roelandt patch: [https://review.openstack.org/#/c/77585/ Make hacking Python 3 compatible]
|-

| [https://pypi.python.org/pypi/jsonrpclib jsonrpclib] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || The project seems dead :(
|-

| [https://pypi.python.org/pypi/mysql-python mysql-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there.
|-

| [https://pypi.python.org/pypi/netifaces netifaces] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Patch sent by Victor Stinner (in private): [https://bitbucket.org/haypo/misc/src/tip/openstack/netifaces_python3.patch netifaces_python3.patch], Debian has patches too. Python 3 support as of 0.10.4. Pushed to requirements: https://review.openstack.org/94358 .
|-

| [https://pypi.python.org/pypi/nose-exclude nose-exclude] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
|-

| [https://pypi.python.org/pypi/nosehtmloutput nosehtmloutput] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* nose-exclude (tests only)
* openstack.nose-plugin
||
* https://bugs.launchpad.net/ubuntu/+source/python-nosehtmloutput/+bug/1287247
* https://review.openstack.org/#/c/80956/
|-
| [https://pypi.python.org/pypi/nosexcover nosexcover] || style="background-color:lightgreen;" | No || N/A || style="background-color: lightgreen;" | On PyPI || || Python 3 support since 1.0.9
|-
| [https://pypi.python.org/pypi/openstack.nose-plugin openstack.nose-plugin] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color:red;" | No || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || suds ||
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Portage in Progress by Victor Stinner ([https://review.openstack.org/#/dashboard/9107 dashboard])
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI (1.1.0) || ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color:lightgreen;" | Yes || No tests :) || style="background-color: lightgreen;" | In the git repo, not on PyPI || || https://review.openstack.org/#/c/79311/
|-
| [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Must be replaced by oslosphinx (without the dot)
|-

| [https://pypi.python.org/pypi/pam pam] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The fork [https://pypi.python.org/pypi/simplepam simplepam] works on Python 2 and 3
|-

| [https://pypi.python.org/pypi/paramiko paramiko] || style="background-color:lightgreen;" | Yes || N/A || style="background-color: lightgreen;" | On PyPI || || Requirements upgraded: https://review.openstack.org/#/c/81132/
|-

| [https://pypi.python.org/pypi/paste paste] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
|-

| [https://pypi.python.org/pypi/pycadf pycadf] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Classifiers added https://review.openstack.org/#/c/133088/
|-

| [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The project seems dead.
|-

| [https://pypi.python.org/pypi/python-memcached python-memcached] || style="background-color:green;" | Not released, but last commit on git has it || style="background-color: red;" | No || style="background-color: red;" | No || || [https://github.com/jbalogh/django-cache-machine/issues/52 Issue #52: Python 3.3 support? ], [https://github.com/linsomniac/python-memcached/pull/26 Pull request #26: Python 3.3 Support] -- Julien Danjou ported [https://pypi.python.org/pypi/pymemcache pymemcache] to Python 3, another memcached client, he suggests to use this one instead
|-

| [https://pypi.python.org/pypi/qpid-python qpid-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/rtslib-fb rtslib-fb] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-docbookrestapi sphinxcontrib-docbookrestapi] || style="background-color:green;" | Yes || style="background-color: green;" | Yes || style="background-color: green;" | Yes || || Support for 3.3, but not for 3.4, PyPI not updated.
|-
| [https://pypi.python.org/pypi/sphinxcontrib-httpdomain sphinxcontrib-httpdomain] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-pecanwsme sphinxcontrib-pecanwsme] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No||
* hacking
* ibm-db-sa
* scripttest
||
|-
| [https://pypi.python.org/pypi/suds suds] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || "Lightweight SOAP client". Last commit 2 years ago: https://fedorahosted.org/suds/browser See also this fork which is promising: https://bitbucket.org/jurko/suds
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || ||
|-
| [https://pypi.python.org/pypi/thrift thrift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/websockify websockify] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

|}

=== Other OpenStack projects ===

Python 3 compatible:

* stackforge/python-jenkins : https://review.openstack.org/#/c/95004/ (py33 gate is voting)
* openstack-infra/jenkins-job-builder : https://review.openstack.org/87810 (Python 3 support finally merged in, Sept. 2014)

== Reports at OpenStack Summits ==

* Havana summit notes: https://etherpad.openstack.org/p/havana-python3
* Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
* Juno summit notes: https://etherpad.openstack.org/p/juno-cross-project-future-of-python (Oslo) and https://etherpad.openstack.org/p/juno_swift_python3 (Swift)

== Pycon Montreal 2014: Sprint Port OpenStack to Python 3 ==

Enovance organizes a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday). See the page [[Python3/SprintPycon2014]].

Tuskar/Devtest

2014-10-06T09:42:19Z

Lennart Regebro: /* Install Tuskar */

== Overview ==
Devtest is a script that creates a development environment for TripleO / Tuskar / TuskarUI development. Below you can find step-by-step instructions for setting up the environment using devtest. If you would like more information about how devtest works as well as the script itself, see the following links
* http://docs.openstack.org/developer/tripleo-incubator/index.html
* https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest.sh
However, following the steps below should lead to a complete functioning setup.

== Instructions ==

=== Setup devtest ===
All commands (unless otherwise specified) should be run as root, in a dedicated machine you will use for TripleO development. We will call this machine "the lab machine" or "the lab server". The setup has been tested with lab machines running Fedora 19.

Note: You might want to move the location of the default libvirt storage pool. The default location is /var/lib/libvirt/images. Depending on your file system / partitions setup, there might not be enough space for images at this default location. To be on the safe side, you can move it to another location which has plenty of space. Below is an example how to move the default storage pool location to /home/images. If you are sure that there is enough space at the default location, you can skip this step.

mkdir /home/images
yum install -y libvirt
service libvirtd start
virsh pool-destroy default
virsh pool-undefine default
virsh pool-define-as default dir --target /home/images
virsh pool-start default
virsh pool-autostart default
service libvirtd restart

Install git and kvm. Installing kvm at this point sets the ownership of the /dev/kvm correctly (it should be owned by the root user and kvm group). If you don't do this, devtest will fail due to incorrectly set ownership of /dev/kvm.
yum -y install git kvm

Check access rights on /dev/kvm afterwards. If group is not kvm or group doesn't have rw on this file, execute the following commands and reboot:
chgrp kvm /dev/kvm
chmod g+rw /dev/kvm

Create devtest configuration file:
cat <<'EOF' > ~/.devtestrc
export NODE_CPU=1 NODE_MEM=4096 NODE_DISK=60 NODE_ARCH=amd64
export NODE_DIST="fedora selinux-permissive"
export DIB_RELEASE=20
export UNDERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_COMPUTESCALE=1
export TRIPLEO_CLEANUP=1
export USE_IRONIC=1
EOF

Clone the tripleo-incubator repo and run the devtest script:
export TRIPLEO_ROOT=~/tripleo
mkdir -p $TRIPLEO_ROOT
cd $TRIPLEO_ROOT
git clone https://git.openstack.org/openstack/tripleo-incubator
$TRIPLEO_ROOT/tripleo-incubator/scripts/devtest.sh --trash-my-machine
# wait around 2+ hours
# mawagner: if openvswitch fails to start, see https://bugzilla.redhat.com/show_bug.cgi?id=1006412 (or just mkdir /var/lock/subsys)

If you want to e.g. redeploy Overcloud for testing purposes, you will need to source following
source tripleo-incubator/scripts/devtest_variables.sh
source ./tripleo-undercloud-passwords
source ./tripleo-incubator/undercloudrc
source $TRIPLEO_ROOT/tripleo-incubator/cloudprompt
source ~/.devtestrc

Build the block storage image
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-cinder-volume hosts \
baremetal cinder-volume neutron-openvswitch-agent os-collect-config heat-cfntools pip-cache \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-cinder-volume.log

load-image overcloud-cinder-volume.qcow2

To be able to deploy object storage node, you need this
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-swift-storage hosts \
baremetal swift-storage neutron-openvswitch-agent os-collect-config \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-swift-storage.log

load-image overcloud-swift-storage.qcow2

To list MAC's for virsh listed machines run
virsh list --all
virsh dumpxml baremetal_0 | grep 'mac address' | cut -d\' -f2

At this point, your devtest environment is configured. To proceed, you need to install Tuskar, Horizon and Tuskar-UI, since they are not installed by devtest.

=== Install Tuskar ===

mkdir -p /opt/stack && cd /opt/stack
git clone https://git.openstack.org/openstack/tuskar

Follow the instructions here to complete the install: https://github.com/openstack/tuskar/blob/master/doc/source/install.rst. Make sure that you create the initial data.

To be able to deploy Overcloud via Tuskar, you need to delete the Overcloud created by Devtest first.
Go to the Undercloud machine and delete the Overcloud:
ssh heat-admin@192.0.2.3
sudo -i
source stackrc
heat stack-delete overcloud

Devtest will also set up Tuskar endpoints, but pointing to the undercloud machine. You need them point to your machine. This can be done by deleting and readding endpoints, but it's quicker to just modify them in mysql:
ssh heat-admin@192.0.2.3
sudo -i
mysql -u keystone -p keystone
Enter password: unset
update endpoint, service set endpoint.url="http://<yourlocalip>:8585/" where service.type="management" and endpoint.service_id=service.id;

Next, you create a plan:
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' -d '{"name": "overcloud", "description": "overcloud"}' http://127.0.0.1:8585/v2/plans

Then you need to check out the role templates:

# use templates from this patch https://review.openstack.org/#/c/123100/
cd /opt/stack
git clone https://git.openstack.org/openstack/tripleo-heat-templates
cd tripleo-heat-templates
git fetch https://review.openstack.org/openstack/tripleo-heat-templates refs/changes/00/123100/4 && git checkout FETCH_HEAD

And load the roles:

cd /opt/stack/tuskar
.tox/py27/bin/tuskar-load-roles --config-file etc/tuskar/tuskar.conf --master-seed /opt/stack/tuskar_templates/overcloud.yaml --resource-registry /opt/stack/tuskar_templates/overcloud-resource-registry.yaml --role /opt/stack/tuskar_templates/controller.yaml --role /opt/stack/tuskar_templates/compute.yaml

Finally, you need to associate the roles with the plan:

# list the plan to get the ID
curl -i -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' http://127.0.0.1:8585/v2/plans
# list roles to get their uuids
curl -i -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' http://127.0.0.1:8585/v2/roles
# add each role to the plan (you'll have to run this once for each role)
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' -d '{"uuid": "<role_uuid>"}' http://127.0.0.1:8585/v2/plans/<plan_uuid>roles

=== Add Horizon and Tuskar-UI ===
Install and configure Horizon and Tuskar-UI using the instructions at: http://tuskar-ui.readthedocs.org/en/latest/install.html. Make sure to clone the horizon and tuskar-ui repositories in /opt/stack.

.Afterwards, copy /root/stackrc from the undercloud to the lab machine, change the OS_AUTH_URL to point to the undercloud, and source it:
ssh heat-admin@192.0.2.3 "sudo -i cat /root/stackrc" > /root/stackrc
sed -i 's/localhost:5000/192.0.2.3:5000/' /root/stackrc
source /root/stackrc

Configure OPENSTACK_HOST in horizon settings to point to the undercloud machine:
sed -i 's/OPENSTACK_HOST = "127.0.0.1"/OPENSTACK_HOST = "192.0.2.3"/' /opt/stack/horizon/openstack_dashboard/local/local_settings.py

Start the server:
tools/with_venv.sh ./manage.py runserver 0.0.0.0:8111

Now you can see the UI from your browser (you can find credentials for dashboard in undercloud machine in /root/stackrc):
lab-machine:8111

If you want to develop locally, you need only to mount the remote folder containing the code, in your local file system:
yum install -y sshfs && mkdir -p ~/devtest
sshfs root@lab-machine:/opt/stack/ ~/devtest

=== Script for installing Tuskar and Tuskar UI ===
There is a script to automatically do previous two steps: https://gist.github.com/Divius/9525147. It clones to the current directory, uses sudo to do privileged operations, so it can be used by non-root. The only requirement is that stackrc file is in the current directory. Also note that the script is mostly unsupported and can become outdated.

Tuskar/Devtest

2014-10-06T09:39:56Z

Lennart Regebro: /* Install Tuskar */

Tuskar/Devtest

2014-10-06T09:36:22Z

Lennart Regebro: /* Install Tuskar */

== Overview ==
Devtest is a script that creates a development environment for TripleO / Tuskar / TuskarUI development. Below you can find step-by-step instructions for setting up the environment using devtest. If you would like more information about how devtest works as well as the script itself, see the following links
* http://docs.openstack.org/developer/tripleo-incubator/index.html
* https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest.sh
However, following the steps below should lead to a complete functioning setup.

== Instructions ==

=== Setup devtest ===
All commands (unless otherwise specified) should be run as root, in a dedicated machine you will use for TripleO development. We will call this machine "the lab machine" or "the lab server". The setup has been tested with lab machines running Fedora 19.

Note: You might want to move the location of the default libvirt storage pool. The default location is /var/lib/libvirt/images. Depending on your file system / partitions setup, there might not be enough space for images at this default location. To be on the safe side, you can move it to another location which has plenty of space. Below is an example how to move the default storage pool location to /home/images. If you are sure that there is enough space at the default location, you can skip this step.

mkdir /home/images
yum install -y libvirt
service libvirtd start
virsh pool-destroy default
virsh pool-undefine default
virsh pool-define-as default dir --target /home/images
virsh pool-start default
virsh pool-autostart default
service libvirtd restart

Install git and kvm. Installing kvm at this point sets the ownership of the /dev/kvm correctly (it should be owned by the root user and kvm group). If you don't do this, devtest will fail due to incorrectly set ownership of /dev/kvm.
yum -y install git kvm

Check access rights on /dev/kvm afterwards. If group is not kvm or group doesn't have rw on this file, execute the following commands and reboot:
chgrp kvm /dev/kvm
chmod g+rw /dev/kvm

Create devtest configuration file:
cat <<'EOF' > ~/.devtestrc
export NODE_CPU=1 NODE_MEM=4096 NODE_DISK=60 NODE_ARCH=amd64
export NODE_DIST="fedora selinux-permissive"
export DIB_RELEASE=20
export UNDERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_COMPUTESCALE=1
export TRIPLEO_CLEANUP=1
export USE_IRONIC=1
EOF

Clone the tripleo-incubator repo and run the devtest script:
export TRIPLEO_ROOT=~/tripleo
mkdir -p $TRIPLEO_ROOT
cd $TRIPLEO_ROOT
git clone https://git.openstack.org/openstack/tripleo-incubator
$TRIPLEO_ROOT/tripleo-incubator/scripts/devtest.sh --trash-my-machine
# wait around 2+ hours
# mawagner: if openvswitch fails to start, see https://bugzilla.redhat.com/show_bug.cgi?id=1006412 (or just mkdir /var/lock/subsys)

If you want to e.g. redeploy Overcloud for testing purposes, you will need to source following
source tripleo-incubator/scripts/devtest_variables.sh
source ./tripleo-undercloud-passwords
source ./tripleo-incubator/undercloudrc
source $TRIPLEO_ROOT/tripleo-incubator/cloudprompt
source ~/.devtestrc

Build the block storage image
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-cinder-volume hosts \
baremetal cinder-volume neutron-openvswitch-agent os-collect-config heat-cfntools pip-cache \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-cinder-volume.log

load-image overcloud-cinder-volume.qcow2

To be able to deploy object storage node, you need this
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-swift-storage hosts \
baremetal swift-storage neutron-openvswitch-agent os-collect-config \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-swift-storage.log

load-image overcloud-swift-storage.qcow2

To list MAC's for virsh listed machines run
virsh list --all
virsh dumpxml baremetal_0 | grep 'mac address' | cut -d\' -f2

At this point, your devtest environment is configured. To proceed, you need to install Tuskar, Horizon and Tuskar-UI, since they are not installed by devtest.

=== Install Tuskar ===

mkdir -p /opt/stack && cd /opt/stack
git clone https://git.openstack.org/openstack/tuskar

Follow the instructions here to complete the install: https://github.com/openstack/tuskar/blob/master/doc/source/install.rst. Make sure that you create the initial data.

To be able to deploy Overcloud via Tuskar, you need to delete the Overcloud created by Devtest first.
Go to the Undercloud machine and delete the Overcloud:
ssh heat-admin@192.0.2.3
sudo -i
source stackrc
heat stack-delete overcloud

Devtest will also set up Tuskar endpoints, but pointing to the undercloud machine. You need them point to your machine. This can be done by deleting and readding endpoints, but it's quicker to just modify them in mysql:
ssh heat-admin@192.0.2.3
sudo -i
mysql -u keystone -p keystone
Enter password: unset
update endpoint, service set endpoint.url="http://<yourlocalip>:8585/" where service.type="management" and endpoint.service_id=service.id;

Finally, you'll want to set up an initial plan, some roles, and link the two:

#create a plan
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' -d '{"name": "overcloud", "description": "overcloud"}' http://127.0.0.1:8585/v2/plans
# check out templates
# use templates from this patch https://review.openstack.org/#/c/123100/
cd /opt/stack
git clone https://git.openstack.org/openstack/tripleo-heat-templates
cd tripleo-heat-templates
git fetch https://review.openstack.org/openstack/tripleo-heat-templates refs/changes/00/123100/4 && git checkout FETCH_HEAD
# load roles
cd /opt/stack/tuskar
.tox/py27/bin/tuskar-load-roles --config-file etc/tuskar/tuskar.conf --master-seed /opt/stack/tuskar_templates/overcloud.yaml --resource-registry /opt/stack/tuskar_templates/overcloud-resource-registry.yaml --role /opt/stack/tuskar_templates/controller.yaml --role /opt/stack/tuskar_templates/compute.yaml

# list roles to get their uuids
curl -i -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' http://127.0.0.1:8585/v2/roles
# add each role to the plan (you'll have to run this once for each role)
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' -d '{"uuid": "<role_uuid>"}' http://127.0.0.1:8585/v2/plans/<plan_uuid>roles

=== Add Horizon and Tuskar-UI ===
Install and configure Horizon and Tuskar-UI using the instructions at: http://tuskar-ui.readthedocs.org/en/latest/install.html. Make sure to clone the horizon and tuskar-ui repositories in /opt/stack.

.Afterwards, copy /root/stackrc from the undercloud to the lab machine, change the OS_AUTH_URL to point to the undercloud, and source it:
ssh heat-admin@192.0.2.3 "sudo -i cat /root/stackrc" > /root/stackrc
sed -i 's/localhost:5000/192.0.2.3:5000/' /root/stackrc
source /root/stackrc

Configure OPENSTACK_HOST in horizon settings to point to the undercloud machine:
sed -i 's/OPENSTACK_HOST = "127.0.0.1"/OPENSTACK_HOST = "192.0.2.3"/' /opt/stack/horizon/openstack_dashboard/local/local_settings.py

Start the server:
tools/with_venv.sh ./manage.py runserver 0.0.0.0:8111

Now you can see the UI from your browser (you can find credentials for dashboard in undercloud machine in /root/stackrc):
lab-machine:8111

If you want to develop locally, you need only to mount the remote folder containing the code, in your local file system:
yum install -y sshfs && mkdir -p ~/devtest
sshfs root@lab-machine:/opt/stack/ ~/devtest

=== Script for installing Tuskar and Tuskar UI ===
There is a script to automatically do previous two steps: https://gist.github.com/Divius/9525147. It clones to the current directory, uses sudo to do privileged operations, so it can be used by non-root. The only requirement is that stackrc file is in the current directory. Also note that the script is mostly unsupported and can become outdated.

Tuskar/Devtest

2014-10-06T09:35:27Z

Lennart Regebro: /* Install Tuskar */

== Overview ==
Devtest is a script that creates a development environment for TripleO / Tuskar / TuskarUI development. Below you can find step-by-step instructions for setting up the environment using devtest. If you would like more information about how devtest works as well as the script itself, see the following links
* http://docs.openstack.org/developer/tripleo-incubator/index.html
* https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest.sh
However, following the steps below should lead to a complete functioning setup.

== Instructions ==

=== Setup devtest ===
All commands (unless otherwise specified) should be run as root, in a dedicated machine you will use for TripleO development. We will call this machine "the lab machine" or "the lab server". The setup has been tested with lab machines running Fedora 19.

Note: You might want to move the location of the default libvirt storage pool. The default location is /var/lib/libvirt/images. Depending on your file system / partitions setup, there might not be enough space for images at this default location. To be on the safe side, you can move it to another location which has plenty of space. Below is an example how to move the default storage pool location to /home/images. If you are sure that there is enough space at the default location, you can skip this step.

mkdir /home/images
yum install -y libvirt
service libvirtd start
virsh pool-destroy default
virsh pool-undefine default
virsh pool-define-as default dir --target /home/images
virsh pool-start default
virsh pool-autostart default
service libvirtd restart

Install git and kvm. Installing kvm at this point sets the ownership of the /dev/kvm correctly (it should be owned by the root user and kvm group). If you don't do this, devtest will fail due to incorrectly set ownership of /dev/kvm.
yum -y install git kvm

Check access rights on /dev/kvm afterwards. If group is not kvm or group doesn't have rw on this file, execute the following commands and reboot:
chgrp kvm /dev/kvm
chmod g+rw /dev/kvm

Create devtest configuration file:
cat <<'EOF' > ~/.devtestrc
export NODE_CPU=1 NODE_MEM=4096 NODE_DISK=60 NODE_ARCH=amd64
export NODE_DIST="fedora selinux-permissive"
export DIB_RELEASE=20
export UNDERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_COMPUTESCALE=1
export TRIPLEO_CLEANUP=1
export USE_IRONIC=1
EOF

Clone the tripleo-incubator repo and run the devtest script:
export TRIPLEO_ROOT=~/tripleo
mkdir -p $TRIPLEO_ROOT
cd $TRIPLEO_ROOT
git clone https://git.openstack.org/openstack/tripleo-incubator
$TRIPLEO_ROOT/tripleo-incubator/scripts/devtest.sh --trash-my-machine
# wait around 2+ hours
# mawagner: if openvswitch fails to start, see https://bugzilla.redhat.com/show_bug.cgi?id=1006412 (or just mkdir /var/lock/subsys)

If you want to e.g. redeploy Overcloud for testing purposes, you will need to source following
source tripleo-incubator/scripts/devtest_variables.sh
source ./tripleo-undercloud-passwords
source ./tripleo-incubator/undercloudrc
source $TRIPLEO_ROOT/tripleo-incubator/cloudprompt
source ~/.devtestrc

Build the block storage image
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-cinder-volume hosts \
baremetal cinder-volume neutron-openvswitch-agent os-collect-config heat-cfntools pip-cache \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-cinder-volume.log

load-image overcloud-cinder-volume.qcow2

To be able to deploy object storage node, you need this
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-swift-storage hosts \
baremetal swift-storage neutron-openvswitch-agent os-collect-config \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-swift-storage.log

load-image overcloud-swift-storage.qcow2

To list MAC's for virsh listed machines run
virsh list --all
virsh dumpxml baremetal_0 | grep 'mac address' | cut -d\' -f2

At this point, your devtest environment is configured. To proceed, you need to install Tuskar, Horizon and Tuskar-UI, since they are not installed by devtest.

=== Install Tuskar ===

mkdir -p /opt/stack && cd /opt/stack
git clone https://git.openstack.org/openstack/tuskar

Follow the instructions here to complete the install: https://github.com/openstack/tuskar/blob/master/doc/source/install.rst. Make sure that you create the initial data.

To be able to deploy Overcloud via Tuskar, you need to delete the Overcloud created by Devtest first.
Go to the Undercloud machine and delete the Overcloud:
ssh heat-admin@192.0.2.3
sudo -i
source stackrc
heat stack-delete overcloud

Devtest will also set up Tuskar endpoints, but pointing to the undercloud machine. You need them point to your machine. This can be done by deleting and readding endpoints, but it's quicker to just modify them in mysql:
ssh heat-admin@192.0.2.3
sudo -i
mysql -u keystone -p keystone
Enter password: unset
update endpoint, service set endpoint.url="http://<yourlocalip>:8585/" where service.type="management" and endpoint.service_id=service.id;

Finally, you'll want to set up an initial plan, some roles, and link the two:

# create a plan
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' -d '{"name": "overcloud", "description": "overcloud"}' http://127.0.0.1:8585/v2/plans
# load roles
# use templates from this patch https://review.openstack.org/#/c/123100/
cd /opt/stack
git clone https://git.openstack.org/openstack/tripleo-heat-templates
cd tripleo-heat-templates
git fetch https://review.openstack.org/openstack/tripleo-heat-templates refs/changes/00/123100/4 && git checkout FETCH_HEAD

cd /opt/stack/tuskar
.tox/py27/bin/tuskar-load-roles --config-file etc/tuskar/tuskar.conf --master-seed /opt/stack/tuskar_templates/overcloud.yaml --resource-registry /opt/stack/tuskar_templates/overcloud-resource-registry.yaml --role /opt/stack/tuskar_templates/controller.yaml --role /opt/stack/tuskar_templates/compute.yaml

# list roles to get their uuids
curl -i -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' http://127.0.0.1:8585/v2/roles
# add each role to the plan (you'll have to run this once for each role)
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' -d '{"uuid": "<role_uuid>"}' http://127.0.0.1:8585/v2/plans/<plan_uuid>roles

=== Add Horizon and Tuskar-UI ===
Install and configure Horizon and Tuskar-UI using the instructions at: http://tuskar-ui.readthedocs.org/en/latest/install.html. Make sure to clone the horizon and tuskar-ui repositories in /opt/stack.

.Afterwards, copy /root/stackrc from the undercloud to the lab machine, change the OS_AUTH_URL to point to the undercloud, and source it:
ssh heat-admin@192.0.2.3 "sudo -i cat /root/stackrc" > /root/stackrc
sed -i 's/localhost:5000/192.0.2.3:5000/' /root/stackrc
source /root/stackrc

Configure OPENSTACK_HOST in horizon settings to point to the undercloud machine:
sed -i 's/OPENSTACK_HOST = "127.0.0.1"/OPENSTACK_HOST = "192.0.2.3"/' /opt/stack/horizon/openstack_dashboard/local/local_settings.py

Start the server:
tools/with_venv.sh ./manage.py runserver 0.0.0.0:8111

Now you can see the UI from your browser (you can find credentials for dashboard in undercloud machine in /root/stackrc):
lab-machine:8111

If you want to develop locally, you need only to mount the remote folder containing the code, in your local file system:
yum install -y sshfs && mkdir -p ~/devtest
sshfs root@lab-machine:/opt/stack/ ~/devtest

=== Script for installing Tuskar and Tuskar UI ===
There is a script to automatically do previous two steps: https://gist.github.com/Divius/9525147. It clones to the current directory, uses sudo to do privileged operations, so it can be used by non-root. The only requirement is that stackrc file is in the current directory. Also note that the script is mostly unsupported and can become outdated.

Tuskar/Devtest

2014-10-06T08:11:35Z

Lennart Regebro: /* Install Tuskar */

== Overview ==
Devtest is a script that creates a development environment for TripleO / Tuskar / TuskarUI development. Below you can find step-by-step instructions for setting up the environment using devtest. If you would like more information about how devtest works as well as the script itself, see the following links
* http://docs.openstack.org/developer/tripleo-incubator/index.html
* https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest.sh
However, following the steps below should lead to a complete functioning setup.

== Instructions ==

=== Setup devtest ===
All commands (unless otherwise specified) should be run as root, in a dedicated machine you will use for TripleO development. We will call this machine "the lab machine" or "the lab server". The setup has been tested with lab machines running Fedora 19.

Note: You might want to move the location of the default libvirt storage pool. The default location is /var/lib/libvirt/images. Depending on your file system / partitions setup, there might not be enough space for images at this default location. To be on the safe side, you can move it to another location which has plenty of space. Below is an example how to move the default storage pool location to /home/images. If you are sure that there is enough space at the default location, you can skip this step.

mkdir /home/images
yum install -y libvirt
service libvirtd start
virsh pool-destroy default
virsh pool-undefine default
virsh pool-define-as default dir --target /home/images
virsh pool-start default
virsh pool-autostart default
service libvirtd restart

Install git and kvm. Installing kvm at this point sets the ownership of the /dev/kvm correctly (it should be owned by the root user and kvm group). If you don't do this, devtest will fail due to incorrectly set ownership of /dev/kvm.
yum -y install git kvm

Check access rights on /dev/kvm afterwards. If group is not kvm or group doesn't have rw on this file, execute the following commands and reboot:
chgrp kvm /dev/kvm
chmod g+rw /dev/kvm

Create devtest configuration file:
cat <<'EOF' > ~/.devtestrc
export NODE_CPU=1 NODE_MEM=4096 NODE_DISK=60 NODE_ARCH=amd64
export NODE_DIST="fedora selinux-permissive"
export DIB_RELEASE=20
export UNDERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_COMPUTESCALE=1
export TRIPLEO_CLEANUP=1
export USE_IRONIC=1
EOF

Clone the tripleo-incubator repo and run the devtest script:
export TRIPLEO_ROOT=~/tripleo
mkdir -p $TRIPLEO_ROOT
cd $TRIPLEO_ROOT
git clone https://git.openstack.org/openstack/tripleo-incubator
$TRIPLEO_ROOT/tripleo-incubator/scripts/devtest.sh --trash-my-machine
# wait around 2+ hours
# mawagner: if openvswitch fails to start, see https://bugzilla.redhat.com/show_bug.cgi?id=1006412 (or just mkdir /var/lock/subsys)

If you want to e.g. redeploy Overcloud for testing purposes, you will need to source following
source tripleo-incubator/scripts/devtest_variables.sh
source ./tripleo-undercloud-passwords
source ./tripleo-incubator/undercloudrc
source $TRIPLEO_ROOT/tripleo-incubator/cloudprompt
source ~/.devtestrc

Build the block storage image
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-cinder-volume hosts \
baremetal cinder-volume neutron-openvswitch-agent os-collect-config heat-cfntools pip-cache \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-cinder-volume.log

load-image overcloud-cinder-volume.qcow2

To be able to deploy object storage node, you need this
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-swift-storage hosts \
baremetal swift-storage neutron-openvswitch-agent os-collect-config \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-swift-storage.log

load-image overcloud-swift-storage.qcow2

To list MAC's for virsh listed machines run
virsh list --all
virsh dumpxml baremetal_0 | grep 'mac address' | cut -d\' -f2

At this point, your devtest environment is configured. To proceed, you need to install Tuskar, Horizon and Tuskar-UI, since they are not installed by devtest.

=== Install Tuskar ===

mkdir -p /opt/stack && cd /opt/stack
git clone https://git.openstack.org/openstack/tuskar

Follow the instructions here to complete the install: https://github.com/openstack/tuskar/blob/master/doc/source/install.rst. Make sure that you create the initial data.

To be able to deploy Overcloud via Tuskar, you need to delete the Overcloud created by Devtest first.
Go to the Undercloud machine and delete the Overcloud:
ssh heat-admin@192.0.2.3
sudo -i
source stackrc
heat stack-delete overcloud

Devtest will also set up Tuskar endpoints, but pointing to the undercloud machine. You need them point to your machine. This can be done by deleting and readding endpoints, but it's quicker to just modify them in mysql:
ssh heat-admin@192.0.2.3
sudo -i
mysql -u keystone -p keystone
Enter password: unset
update endpoint, service set endpoint.url="http://<yourlocalip>:8585/" where service.type="management" and endpoint.service_id=service.id;

Finally, you'll want to set up an initial plan, some roles, and link the two:

# create a plan
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' -d '{"name": "overcloud", "description": "overcloud"}' http://127.0.0.1:8585/v2/plans
# load roles
# use templates from this patch https://review.openstack.org/#/c/123100/
# this script is creating roles from all templates in the tuskar_roles/ dir,
# so copy there only compute.yaml and controller.yaml
tuskar-load-roles --config-file etc/tuskar/tuskar.conf /opt/stack/tuskar_roles/ --master-seed /opt/stack/tuskar_templates/overcloud.yaml

# list roles to get their uuids
curl -i -X GET -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' http://127.0.0.1:8585/v2/roles
# add each role to the plan (you'll have to run this once for each role)
curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-tuskarclient' -d '{"uuid": "<role_uuid>"}' http://127.0.0.1:8585/v2/plans/<plan_uuid>roles

=== Add Horizon and Tuskar-UI ===
Install and configure Horizon and Tuskar-UI using the instructions at: http://tuskar-ui.readthedocs.org/en/latest/install.html. Make sure to clone the horizon and tuskar-ui repositories in /opt/stack.

.Afterwards, copy /root/stackrc from the undercloud to the lab machine, change the OS_AUTH_URL to point to the undercloud, and source it:
ssh heat-admin@192.0.2.3 "sudo -i cat /root/stackrc" > /root/stackrc
sed -i 's/localhost:5000/192.0.2.3:5000/' /root/stackrc
source /root/stackrc

Configure OPENSTACK_HOST in horizon settings to point to the undercloud machine:
sed -i 's/OPENSTACK_HOST = "127.0.0.1"/OPENSTACK_HOST = "192.0.2.3"/' /opt/stack/horizon/openstack_dashboard/local/local_settings.py

Start the server:
tools/with_venv.sh ./manage.py runserver 0.0.0.0:8111

Now you can see the UI from your browser (you can find credentials for dashboard in undercloud machine in /root/stackrc):
lab-machine:8111

If you want to develop locally, you need only to mount the remote folder containing the code, in your local file system:
yum install -y sshfs && mkdir -p ~/devtest
sshfs root@lab-machine:/opt/stack/ ~/devtest

=== Script for installing Tuskar and Tuskar UI ===
There is a script to automatically do previous two steps: https://gist.github.com/Divius/9525147. It clones to the current directory, uses sudo to do privileged operations, so it can be used by non-root. The only requirement is that stackrc file is in the current directory. Also note that the script is mostly unsupported and can become outdated.

Tuskar/Devtest

2014-08-26T14:41:09Z

Lennart Regebro: /* Install Tuskar */

== Overview ==
Devtest is a script that creates a development environment for TripleO / Tuskar / TuskarUI development. Below you can find step-by-step instructions for setting up the environment using devtest. If you would like more information about how devtest works as well as the script itself, see the following links
* http://docs.openstack.org/developer/tripleo-incubator/index.html
* https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest.sh
However, following the steps below should lead to a complete functioning setup.

== Instructions ==

=== Setup devtest ===
All commands (unless otherwise specified) should be run as root, in a dedicated machine you will use for TripleO development. We will call this machine "the lab machine" or "the lab server". The setup has been tested with lab machines running Fedora 19.

Note: You might want to move the location of the default libvirt storage pool. The default location is /var/lib/libvirt/images. Depending on your file system / partitions setup, there might not be enough space for images at this default location. To be on the safe side, you can move it to another location which has plenty of space. Below is an example how to move the default storage pool location to /home/images. If you are sure that there is enough space at the default location, you can skip this step.

mkdir /home/images
yum install -y libvirt
service libvirtd start
virsh pool-destroy default
virsh pool-undefine default
virsh pool-define-as default dir --target /home/images
virsh pool-start default
virsh pool-autostart default
service libvirtd restart

Install git and kvm. Installing kvm at this point sets the ownership of the /dev/kvm correctly (it should be owned by the root user and kvm group). If you don't do this, devtest will fail due to incorrectly set ownership of /dev/kvm.
yum -y install git kvm

Check access rights on /dev/kvm afterwards. If group is not kvm or group doesn't have rw on this file, execute the following commands and reboot:
chgrp kvm /dev/kvm
chmod g+rw /dev/kvm

Create devtest configuration file:
cat <<'EOF' > ~/.devtestrc
export NODE_CPU=1 NODE_MEM=4096 NODE_DISK=60 NODE_ARCH=amd64
export NODE_DIST="fedora selinux-permissive"
export DIB_RELEASE=20
export UNDERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_COMPUTESCALE=1
export TRIPLEO_CLEANUP=1
EOF

Clone the tripleo-incubator repo and run the devtest script:
export TRIPLEO_ROOT=~/tripleo
mkdir -p $TRIPLEO_ROOT
cd $TRIPLEO_ROOT
git clone https://git.openstack.org/openstack/tripleo-incubator
$TRIPLEO_ROOT/tripleo-incubator/scripts/devtest.sh --trash-my-machine
# wait around 2+ hours
# mawagner: if openvswitch fails to start, see https://bugzilla.redhat.com/show_bug.cgi?id=1006412 (or just mkdir /var/lock/subsys)

If you want to e.g. redeploy Overcloud for testing purposes, you will need to source following
source tripleo-incubator/scripts/devtest_variables.sh
source ./tripleo-undercloud-passwords
source ./tripleo-incubator/undercloudrc
source $TRIPLEO_ROOT/tripleo-incubator/cloudprompt
source ~/.devtestrc

Build the block storage image
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-cinder-volume hosts \
baremetal cinder-volume neutron-openvswitch-agent os-collect-config heat-cfntools pip-cache \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-cinder-volume.log

load-image overcloud-cinder-volume.qcow2

To be able to deploy object storage node, you need this
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-swift-storage hosts \
baremetal swift-storage neutron-openvswitch-agent os-collect-config \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-swift-storage.log

load-image overcloud-swift-storage.qcow2

To list MAC's for virsh listed machines run
virsh list --all
virsh dumpxml baremetal_0 | grep 'mac address' | cut -d\' -f2

At this point, your devtest environment is configured. To proceed, you need to install Tuskar, Horizon and Tuskar-UI, since they are not installed by devtest.

=== Install Tuskar ===

mkdir -p /opt/stack && cd /opt/stack
git clone https://git.openstack.org/openstack/tuskar

Follow the instructions here to complete the install: https://github.com/openstack/tuskar/blob/master/doc/source/INSTALL.rst. Make sure that you create the initial data.

To be able to deploy Overcloud via Tuskar, you need to delete the Overcloud created by Devtest first.
Go to the Undercloud machine and delete the Overcloud:
ssh heat-admin@192.0.2.3
sudo -i
source stackrc
heat stack-delete overcloud

Devtest will also set up Tuskar endpoints, but pointing to the undercloud machine. You need them point to your machine. This can be done by deleting and readding endpoints, but it's quicker to just modify them in mysql:
ssh heat-admin@192.0.2.3
sudo -i
mysql -u keystone -p keystone
Enter password: unset
update endpoint, service set endpoint.url="http://<yourlocalip>:8585/" where service.type="management" and endpoint.service_id=service.id;

=== Add Horizon and Tuskar-UI ===
Install and configure Horizon and Tuskar-UI using the instructions at: http://tuskar-ui.readthedocs.org/en/latest/install.html. Make sure to clone the horizon and tuskar-ui repositories in /opt/stack.

.Afterwards, copy /root/stackrc from the undercloud to the lab machine, change the OS_AUTH_URL to point to the undercloud, and source it:
ssh heat-admin@192.0.2.3 "sudo -i cat /root/stackrc" > /root/stackrc
sed -i 's/localhost:5000/192.0.2.3:5000/' /root/stackrc
source /root/stackrc

Configure OPENSTACK_HOST in horizon settings to point to the undercloud machine:
sed -i 's/OPENSTACK_HOST = "127.0.0.1"/OPENSTACK_HOST = "192.0.2.3"/' /opt/stack/horizon/openstack_dashboard/local/local_settings.py

Start the server:
tools/with_venv.sh ./manage.py runserver 0.0.0.0:8111

Now you can see the UI from your browser (you can find credentials for dashboard in undercloud machine in /root/stackrc):
lab-machine:8111

If you want to develop locally, you need only to mount the remote folder containing the code, in your local file system:
yum install -y sshfs && mkdir -p ~/devtest
sshfs root@lab-machine:/opt/stack/ ~/devtest

=== Script for installing Tuskar and Tuskar UI ===
There is a script to automatically do previous two steps: https://gist.github.com/Divius/9525147. It clones to the current directory, uses sudo to do privileged operations, so it can be used by non-root. The only requirement is that stackrc file is in the current directory. Also note that the script is mostly unsupported and can become outdated.

Tuskar/Devtest

2014-08-26T14:40:54Z

Lennart Regebro: /* Install Tuskar */

== Overview ==
Devtest is a script that creates a development environment for TripleO / Tuskar / TuskarUI development. Below you can find step-by-step instructions for setting up the environment using devtest. If you would like more information about how devtest works as well as the script itself, see the following links
* http://docs.openstack.org/developer/tripleo-incubator/index.html
* https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest.sh
However, following the steps below should lead to a complete functioning setup.

== Instructions ==

=== Setup devtest ===
All commands (unless otherwise specified) should be run as root, in a dedicated machine you will use for TripleO development. We will call this machine "the lab machine" or "the lab server". The setup has been tested with lab machines running Fedora 19.

Note: You might want to move the location of the default libvirt storage pool. The default location is /var/lib/libvirt/images. Depending on your file system / partitions setup, there might not be enough space for images at this default location. To be on the safe side, you can move it to another location which has plenty of space. Below is an example how to move the default storage pool location to /home/images. If you are sure that there is enough space at the default location, you can skip this step.

mkdir /home/images
yum install -y libvirt
service libvirtd start
virsh pool-destroy default
virsh pool-undefine default
virsh pool-define-as default dir --target /home/images
virsh pool-start default
virsh pool-autostart default
service libvirtd restart

Install git and kvm. Installing kvm at this point sets the ownership of the /dev/kvm correctly (it should be owned by the root user and kvm group). If you don't do this, devtest will fail due to incorrectly set ownership of /dev/kvm.
yum -y install git kvm

Check access rights on /dev/kvm afterwards. If group is not kvm or group doesn't have rw on this file, execute the following commands and reboot:
chgrp kvm /dev/kvm
chmod g+rw /dev/kvm

Create devtest configuration file:
cat <<'EOF' > ~/.devtestrc
export NODE_CPU=1 NODE_MEM=4096 NODE_DISK=60 NODE_ARCH=amd64
export NODE_DIST="fedora selinux-permissive"
export DIB_RELEASE=20
export UNDERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_DIB_EXTRA_ARGS=""
export OVERCLOUD_COMPUTESCALE=1
export TRIPLEO_CLEANUP=1
EOF

Clone the tripleo-incubator repo and run the devtest script:
export TRIPLEO_ROOT=~/tripleo
mkdir -p $TRIPLEO_ROOT
cd $TRIPLEO_ROOT
git clone https://git.openstack.org/openstack/tripleo-incubator
$TRIPLEO_ROOT/tripleo-incubator/scripts/devtest.sh --trash-my-machine
# wait around 2+ hours
# mawagner: if openvswitch fails to start, see https://bugzilla.redhat.com/show_bug.cgi?id=1006412 (or just mkdir /var/lock/subsys)

If you want to e.g. redeploy Overcloud for testing purposes, you will need to source following
source tripleo-incubator/scripts/devtest_variables.sh
source ./tripleo-undercloud-passwords
source ./tripleo-incubator/undercloudrc
source $TRIPLEO_ROOT/tripleo-incubator/cloudprompt
source ~/.devtestrc

Build the block storage image
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-cinder-volume hosts \
baremetal cinder-volume neutron-openvswitch-agent os-collect-config heat-cfntools pip-cache \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-cinder-volume.log

load-image overcloud-cinder-volume.qcow2

To be able to deploy object storage node, you need this
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST \
-a $NODE_ARCH -o $TRIPLEO_ROOT/overcloud-swift-storage hosts \
baremetal swift-storage neutron-openvswitch-agent os-collect-config \
dhcp-all-interfaces $DIB_COMMON_ELEMENTS $OVERCLOUD_COMPUTE_DIB_EXTRA_ARGS 2>&1 | \
tee $TRIPLEO_ROOT/dib-overcloud-swift-storage.log

load-image overcloud-swift-storage.qcow2

To list MAC's for virsh listed machines run
virsh list --all
virsh dumpxml baremetal_0 | grep 'mac address' | cut -d\' -f2

At this point, your devtest environment is configured. To proceed, you need to install Tuskar, Horizon and Tuskar-UI, since they are not installed by devtest.

=== Install Tuskar ===

mkdir -p /opt/stack && cd /opt/stack
git clone https://git.openstack.org/openstack/tuskar

Follow the instructions here to complete the install: https://github.com/openstack/tuskar/blob/master/doc/source/INSTALL.rst. Make sure that you create the initial data.

To be able to deploy Overcloud via Tuskar, you need to delete the Overcloud created by Devtest first.
Go to the Undercloud machine and delete the Overcloud:
ssh heat-admin@192.0.2.3
sudo -i
source stackrc
heat stack-delete overcloud

Devtest will also set up Tuskar endpoints, but pointing to the undercloud machine. You need them point to your machine. This can be done by deleting and readding endpoints, but it's quicker to just modify them in mysql:
ssh heat-admin@192.0.2.3
sudo -i
mysql -u keystone -p keystone
Enter password: unset
update endpoint, service set endpoint.url="http://<yourlocalid>:8585/" where service.type="management" and endpoint.service_id=service.id;

=== Add Horizon and Tuskar-UI ===
Install and configure Horizon and Tuskar-UI using the instructions at: http://tuskar-ui.readthedocs.org/en/latest/install.html. Make sure to clone the horizon and tuskar-ui repositories in /opt/stack.

.Afterwards, copy /root/stackrc from the undercloud to the lab machine, change the OS_AUTH_URL to point to the undercloud, and source it:
ssh heat-admin@192.0.2.3 "sudo -i cat /root/stackrc" > /root/stackrc
sed -i 's/localhost:5000/192.0.2.3:5000/' /root/stackrc
source /root/stackrc

Configure OPENSTACK_HOST in horizon settings to point to the undercloud machine:
sed -i 's/OPENSTACK_HOST = "127.0.0.1"/OPENSTACK_HOST = "192.0.2.3"/' /opt/stack/horizon/openstack_dashboard/local/local_settings.py

Start the server:
tools/with_venv.sh ./manage.py runserver 0.0.0.0:8111

Now you can see the UI from your browser (you can find credentials for dashboard in undercloud machine in /root/stackrc):
lab-machine:8111

If you want to develop locally, you need only to mount the remote folder containing the code, in your local file system:
yum install -y sshfs && mkdir -p ~/devtest
sshfs root@lab-machine:/opt/stack/ ~/devtest

=== Script for installing Tuskar and Tuskar UI ===
There is a script to automatically do previous two steps: https://gist.github.com/Divius/9525147. It clones to the current directory, uses sudo to do privileged operations, so it can be used by non-root. The only requirement is that stackrc file is in the current directory. Also note that the script is mostly unsupported and can become outdated.