Difference between revisions of "Python3"
(→Python 3 Status of OpenStack projects) |
(→Dependencies) |
||
Line 460: | Line 460: | ||
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI (1.1.0) || || | | [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI (1.1.0) || || | ||
|- | |- | ||
− | | [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color: | + | | [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color:lightgreen;" | No || style="background-color: lightgreen;" | No || style="background-color: lightgreen;" | No || || https://review.openstack.org/#/c/79311/ |
|- | |- | ||
| [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Must be replaced by oslosphinx (without the dot) | | [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Must be replaced by oslosphinx (without the dot) |
Revision as of 21:22, 14 April 2014
This page tracks the progress of Python 3 effort porting for OpenStack.
Contents
Python 3
Why should OpenStack move to Python 3 right now?
- Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.
Pycon Montreal 2014: Sprint Port OpenStack to Python 3
Enovance organizes a sprint to Port OpenStack to Python 3 during 4 days: between April, 14 (Monday) and April, 17 (Thursday). See the page Python3/SprintPycon2014.
Port Python 2 code to Python 3
OpenStack project chose to use the same code base for Python 2 and Python 3. The Six: Python 2 and 3 Compatibility Library helps to write code working on both versions. OpenStack must still support Python 2.6 for RHEL, but not Python 2.5 and older. Debian Stable provides Python 3 but only Python 3.2, so u'unicode' syntax should be avoided (use six.u('unicode') instead).
Common patterns
- Replace dict.iteritems() with six.iteritems(dict)
- Replace iterator.next() with next(iterator)
- Replace basestring with six.string_types
- Replace unicode with six.text_type
References to port Python 2 code to Python 3
- Porting to Python 3 Book by Lennart Regebro, especially the Language differences and workarounds.
- HOWTO: Porting Python 2 Code to Python 3 by Brett Cannon
- Porting Python Code to 3.x
- python-incompatibility: Demonstrates incompatibilities between Python versions.
Common pitfalls
What is a string ?
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:
Python 2:
>>> type('foo') <type 'str'> >>> type(u'foo') <type 'unicode'> >>> type(b'foo') <type 'str'> >>> isinstance('foo', six.text_type) False >>> isinstance(u'foo', six.text_type) True >>> bytes is str True >>> b'foo'[0] 'f'
Python 3:
>>> type('foo') <class 'str'> >>> type(u'foo') <class 'str'> >>> type(b'foo') <class 'bytes'> >>> isinstance('foo', six.text_type) True >>> isinstance(b'foo', six.text_type) False >>> bytes is str False >>> b'foo'[0] 102
Python 3 of OpenStack Dependencies
Blocker Pointer: it's not yet possible to specify different list of dependencies for Python 2 and Python 3. For example, mox only works on Python 2, mox3 can be used on Python 3.
- Julien Danjou proposed to add requirements-py3.txt: openstack/requirements patch and openstack-dev/pbr patch
- An alternative is to support markers in requirements (in pip): pip issue: Support markers in setup(install_requires)?; Victor Stinner's pull request: "parse requirements in markers"
OpenStack Dependencies:
- mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock)
- eventlet: not available on Python 3 yet, alternatives: asyncio (Tulip for Python 3.3+/Trollius for Python 2), Tornado
Portage in progress
- Oslo Messaging: Portage in Progress by Victor Stinner (dashboard)
- glanceclient: Portage in Progress by Cyril Roelandt (dashboard)
- heatclient: Portage in Progress by Cyril Roelandt (dashboard)
- neutronclient: Portage in Progress by Cyril Roelandt (dashboard)
- glanceclient: Portage in Progress by Cyril Roelandt (dashboard)
Portage done
- keystoneclient: Portage in Progress by Cyril Roelandt (dashboard)
Python 3 Status of OpenStack projects
See also Python3Deps.
Oslo Incubator
BLOCKER BUG: Tests using testscenarios fail on Python 3 with nosetests because of this bug: https://bugs.launchpad.net/testscenarios/+bug/872887
Recently merged reviews:
Test (full path) | Patches | Comment |
---|---|---|
tests/unit/config/test_generator.py | ||
tests/unit/crypto/test_utils.py | ||
tests/unit/db/sqlalchemy/test_migration_common.py | ||
tests/unit/db/sqlalchemy/test_migrate.py | ||
tests/unit/db/sqlalchemy/test_models.py | https://review.openstack.org/#/c/80307/ | |
tests/unit/db/sqlalchemy/test_options.py | https://review.openstack.org/#/c/80627/ | |
tests/unit/db/sqlalchemy/test_migrate_cli.py | ||
tests/unit/db/sqlalchemy/test_utils.py | ||
tests/unit/db/sqlalchemy/test_sqlalchemy.py | ||
tests/unit/fixture/test_logging.py | ||
tests/unit/middleware/test_request_id.py | https://review.openstack.org/#/c/80336/ | |
tests/unit/middleware/test_sizelimit.py | https://review.openstack.org/#/c/80450/ | |
tests/unit/middleware/test_audit.py | ||
tests/unit/reports/test_guru_meditation_report.py | ||
tests/unit/reports/test_base_report.py | ||
tests/unit/reports/test_openstack_generators.py | ||
tests/unit/reports/test_views.py | https://review.openstack.org/87376 | |
tests/unit/rpc/test_common.py | https://review.openstack.org/#/c/80533/ | The RPC code in the incubator is deprecated in favor of oslo.messaging. --doug-hellmann (talk) 15:40, 14 April 2014 (UTC) |
tests/unit/scheduler/test_base_filter.py | https://review.openstack.org/#/c/80321/ | |
tests/unit/scheduler/test_weights.py | https://review.openstack.org/#/c/87336/ | |
tests/unit/test_cliutils | https://review.openstack.org/#/c/74433/ | |
tests/unit/test_fileutils | https://review.openstack.org/#/c/74728/ | |
tests/unit/test_gettext.py | https://review.openstack.org/#/c/80534/ | (issue with testscenarios) |
tests/unit/test_imageutils.py | nosetests + testscenarios failure | |
tests/unit/test_jsonutils.py | https://review.openstack.org/#/c/80370/ | |
tests/unit/test_log.py | depends on https://review.openstack.org/#/c/80534/ | |
tests/unit/test_quota.py | https://review.openstack.org/#/c/80564/ | |
tests/unit/test_strutils.py |
OpenStack clients
Project | Python 3 compatibility | CI tests running? | Python 3 classifiers ? | Blocked by | Comment |
---|---|---|---|---|---|
python-ceilometerclient | Yes | Voting | On PyPI | ||
python-cinderclient | Yes | Voting | https://review.openstack.org/#/c/73844/ | ||
python-ganttclient | ? | ? | ? | ? | |
python-glanceclient | In Progress | Non-voting | No | ||
python-heatclient | Yes | Voting | In the git repo, not on PyPI | ||
python-ironicclient | Yes | Voting | On PyPI | ||
python-keystoneclient | Yes | Voting | On PyPI | ||
python-marconiclient | Yes | Voting | No | ||
python-melangeclient | ? | ? | ? | ||
python-novaclient | Yes | Voting | In the Git repo, not on PyPI | ||
python-neutronclient | In progress | Non-voting | Differences between Python 2 and 3 | ||
python-openstackclient | In Progress | Non-Voting | No | Works with glanceclient HEAD | |
python-savannaclient | In progress | Non-voting | https://review.openstack.org/#/c/73128/ | ||
python-swiftclient | In progress | Non-voting | `No | Differences between Python 2 and 3 | |
python-tuskarclient | Yes | Voting | On PyPI | ||
python-troveclient | Yes | Voting | On PyPI |
Core OpenStack projects
Project | Python 3 compatibility | CI tests running? | Trove classifiers | Blocked by | Comment |
---|---|---|---|---|---|
ceilometer | No | No | No |
|
|
cinder | No | No | No |
|
|
glance | No | No | No |
|
|
heat | No | No | No |
|
|
horizon | No | No | No |
|
|
keystone | No | No | No |
|
|
neutron | No | No | No |
|
|
nova | No | No | No |
|
|
swift | No | No | No |
|
Dependencies
Project | Python 3 compatibility | CI tests running? | Python 3 classifiers ? | Blocked by | Comment |
---|---|---|---|---|---|
boto | No | N/A | No | See https://github.com/boto/boto3 (experimental) | |
django-compressor | No | No | No | https://github.com/django-compressor/django-compressor/issues/484 | |
django-openstack-auth | No | No | No | Ported: https://review.openstack.org/#/dashboard/8122 | |
dnspython | Yes | N/A | Yes | Must use the Python 3 version, see https://github.com/rthalley/dnspython/issues/60 | |
ecdsa | Yes | N/A | In the Git repo | Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master) | |
eventlet | No | No | No | Victor Stinner is working on Trollius (asyncio for Python 2) which may replace eventlet: Use the new asyncio module and Trollius in OpenStack | |
hacking | No | No | No | Cyril Roelandt patch: Make hacking Python 3 compatible | |
jsonrpclib | No | N/A | No | The project seems dead :( | |
mysql-python | No | No | No | 2 pull requests for Python 3 (https://github.com/farcepest/MySQLdb1/pulls). The projects is being renamed to moist (https://github.com/farcepest/moist), Python 3 support might happen there. | |
netifaces | No | N/A | No | Patch sent by Victor Stinner (in private): netifaces_python3.patch, Debian has patches too | |
nose-exclude | No | No | No | https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3 | |
nosehtmloutput | No | No | No |
|
|
nosexcover | No | N/A | On PyPI | Python 3 support since 1.0.9 | |
openstack.nose-plugin | No | No | No | ||
oslo.vmware | No | Voting | Yes | suds | |
oslo.config | Yes | Voting | In the git repo, not on PyPI | ||
oslo.messaging | No | No | No | ||
oslo.rootwrap | Yes | Yes | In the Git repo, not on PyPI (1.1.0) | ||
oslosphinx | No | No | No | https://review.openstack.org/#/c/79311/ | |
oslo.sphinx | No | No | No | Must be replaced by oslosphinx (without the dot) | |
pam | No | No | No | The fork simplepam works on Python 2 and 3 | |
paramiko | Yes | N/A | On PyPI | Get https://review.openstack.org/#/c/81132/ merged | |
paste | No | No | No | https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff | |
pycadf | No | No | No | ||
python-ldap | No | No | No | The project seems dead. | |
qpid-python | No | No | No | ||
rtslib-fb | No | No | No | ||
sphinxcontrib-docbookrestapi | No | No | No | ||
sphinxcontrib-httpdomain | No | No | No | ||
sphinxcontrib-pecanwsme | No | No | No | ||
sqlalchemy-migrate | No | No | No |
|
|
suds | No | No | No | Dead project | |
taskflow | Yes | Yes | In the Git repo, not on PyPI | ||
thrift | No | No | No | ||
websockify | No | No | No |
Reports at OpenStack Summits
- Havana summit notes: https://etherpad.openstack.org/p/havana-python3
- Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3
Dependencies
This is an attempt to document which OpenStack dependencies work under Python 3. We're starting with oslo, since it's the base of everything else, then gradually including other projects.
The caniusepython3 tool can be used to do quick yes/no Python 3 support checks.
oslo.config
pip-requires
- argparse - included with Python 2.7+. The separate package is only needed for 2.6.
test-requires
- mox - mox3 supports Python, or replace mox with mock (which is now included in python 3.3)
- nose - supports Python 3
- nose-exclude - 2.6-2.7, 3.1-3.3
- testtools - 2.6-2.7, 3.2-3.3
- coverage - supports 2.3-3.3
- sphinx - supports Python 3
oslo-incubator
pip-requires
- PasteDeploy: supports 2.5-3.3
- WebOb: support Python 3
- eventlet: NO (MAJOR PAIN POINT) (gevent doesn't either, though there are some old forks that tried)
- greenlet: supports 2.4-3.2 (lack of 3.3 may be false negative)
- lxml: supports 2.4-3.3
- routes: supports 2.6-3.3
- iso8601: NO (perhaps try python-dateutil, specifically the parser module?)
- anyjson: 2.4-3.1 (lack of 3.2-3.3 may be false negatives)
- kombu: supports Python 3 (exact version not given)
- argparse: included in Python 2.7+
- stevedore: supports 2.7, 3.2, and 3.3
- SQLAlchemy: supports Python 3 (exact version not given)
- qpid-python: NO
test-requires
- distribute: 2.4-3.3
- coverage: 2.3-3.3
- fixtures: supports Python 3 (exact version not given)
- mock: 2.5-3.3
- mox: use mox3, or replace mox with mock (which is now included in python 3.3)
- mysql-python: NO (maybe try pymysql instead?)
- nose: Supports Python 3
- nose-exclude: 2.6-2.7, 3.1-3.3
- nosehtmloutput: ?
- pep8: Supports Python 3
- pyflakes: Supports Python 3
- pylint: Supports Python 3 (tested with Python 3.2)
- pyzmq: Supports 2.6-2.7, 3.2+
- redis: Supports 2.5-2.7, 3.2+
- setuptools-git: 2.4-2.7, 3.1-3.3
- sphinx: Supports Python 3
- testtools: 2.6-2.7, 3.2-3.3
- webtest: 2.6-2.7, 3.2-3.3