Difference between revisions of "Cinder/3rdParty-drivers-py3-update"

Revision as of 22:14, 13 August 2019

Cinder Third Party CI update to Python 3.7

The key point of this page is to share tips about updating Third Party CI to run Python 3.7. (Feel free to skip ahead to that part!)

Background

Python 2 (that is, Python 2 itself, not simply OpenStack support for Python 2) is end-of-life on 1 January 2020. This means that the Train release of OpenStack is the last release that will support Python 2. The OpenStack Technical Committee has declared that Python 2 support will be dropped completely during the beginning of the U development cycle (that is, late in 2019).

Reference: 2018-05-29 Python2 Deprecation Timeline

What this means for Cinder Third Party CI

Walt explained on the mailing list:

Hello Cinder folks,

The train release is going to be the last release of OpenStack with python 2 support. Train also is going to require supporting python 3.6 and 3.7. This means that we should be enabling and or switching over all of our 3rd party CI runs to python 3 to ensure that our drivers and all of their required libraries run properly in a python 3.6/3.7 environment. This will help driver maintainers discover any python3 incompatibilities with their driver as well as any required libraries. At the PTG in Denver, the cinder team agreed that we wanted driver CI systems to start using python3 by milestone 2 for Train. This would be the July 22-26th time frame [1].

We are also working on adding driver library requirements to the OpenStack global requirements project [2] [3]. This effort will provide native install primitives for driver libraries in cinder. This process also requires the driver libraries to run in python3.6/3.7.

The Cinder team wants to maintain it's high quality of driver support in the train release. By enabling python 3.6 and python 3.7 in CI tests, this will help everyone ship Cinder with the required support in Train and the following releases.

Walt

[1] https://releases.openstack.org/train/schedule.html

[2] https://review.opendev.org/#/c/656724/

[3] https://review.opendev.org/#/c/657395/

Jay followed up with an additional note that the Cinder team wanted to see drivers running Python 3 in their CI environments by Train Milestone 2 (week of July 22).

Because the Train release of OpenStack is supposed to support both Python 3.6 and Python 3.7, it's acceptable to have your CI running 3.7 only on the theory that anything that runs in 3.7 will also run in 3.6.

Tips for Converting Your CI to Python 3

Please share your experiences to help other CI maintainers. If you can't edit this wiki page (permissions were restricted due to a really bad spam storm in 2017), please slap your comments into a paste and contact jungleboyj, smcginnis, or rosmaita in the #openstack-cinder IRC channel and we'll add them for you.

The latest runtime may not be available as a distribution package for the distro you are using, but that is acceptable. Newer versions can be installed from third party package repos (PPAs or other RPM repos) or by directly installing the Python runtime source release. Another option that has been reported to work well is to use the pyenv Python version manager.

Ubuntu-based CI

Chris M reports that if you are running Ubuntu 16.04, you can convert very easily. (Though he also reported that he only found out how easy it was after trying a bunch of other stuff that didn't work out. In particular, he said "My first thought was to update my CI to Ubuntu 18.04 (Bionic Beaver) to get native support for Python 3.7, but the kernels in the Ubuntu cloud images crashed when I launched nested VMs, so I went back to 16.04 and used the "deadsnakes" PPA to load Python 3.7 on Xenial.")

So you can profit from what he learned, Chris had success converting his Ubuntu 16.04 CI to run Python 3.7 by doing this:

Key steps prior to running stack.sh on Ubuntu 16.04:

   add-apt-repository ppa:deadsnakes/ppa
   apt-get -q update
   apt-get install -y python3.7

Lines to add to local.conf:

   [[local|localrc]]
   USE_PYTHON3=True
   PYTHON3_VERSION=3.7
   :

Using pyenv

Another possibility is to use pyenv to build whatever version of python you want.

Unlike virtualenv, pyenv doesn't use the system python as a base--it builds python from scratch. It gives you plenty of choices for installable pythons, it has a CLI that supports tab completion, allows a user to set their global python version, can set a local python on a per-project basis (you add a .python-version in the root dir of the project), and you can set multiple local pythons (in preference order) for tox to work with.

To install it, you do need some prerequisites, because you are going to compile your own Python interpreter. For Ubuntu 16.04, for example:

 sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
   libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
   xz-utils tk-dev libffi-dev liblzma-dev python-openssl git

For Fedora:

 sudo dnf install zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel \
   openssl-devel xz xz-devel libffi-devel python-devel python3-devel
 sudo dnf group install 'C Development Tools and Libraries'

Those are roughly the packages you'll need. You don't have to search for the other distro equivalents yourself: there's a handy wiki page you can consult:

https://github.com/pyenv/pyenv/wiki/Common-build-problems

You install pyenv as a regular user.

Easy way:

 curl -L \
   https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer \
   | bash

(It will output some instructions to add some stuff to your .bash_profile, or it may add them automatically for you.)

Slightly less easy but more transparent way:

 mkdir -p ~/.pyenv
 git clone git://github.com/pyenv/pyenv.git ~/.pyenv
 git clone git://github.com/pyenv/pyenv-which-ext.git ~/.pyenv/plugins/pyenv-which-ext
 git clone git://github.com/pyenv/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv
 git clone git://github.com/pyenv/pyenv-doctor.git ~/.pyenv/plugins/pyenv-doctor

(There are other plugins, you can poke around on github. And you don't need all these, they're just the ones I use a lot.)

Then add some stuff to your .bashrc :

 cat << 'EOF' >> ~/.bashrc
 # set up pyenv
 export PYENV_ROOT="$HOME/.pyenv"
 export PATH="$PYENV_ROOT/bin:$PATH"
 # enable shims and completions for pyenv
 if command -v pyenv 1>/dev/null 2>&1; then
   eval "$(pyenv init -)"
   eval "$(pyenv virtualenv-init -)"
 fi
 EOF

Open a new shell. The pyenv-doctor plugin checks to make sure you've got all the dependencies installed. Invoke it with

 $ pyenv doctor

Some useful pyenv commands:

 # list all available python versions
 $ pyenv install -l

 # install a version
 $ pyenv install <version>

 # list installed versions (virtualenvs show up here, too)
 $ pyenv versions

 # set global python for current user
 $ pyenv global <version>

 # create a virtualenv (using currently active python version)
 $ pyenv virtualenv <name>

 # create a virtualenv with a specific version of python
 $ pyenv virtualenv <version> <name>

 # activate/deactivate virtualenvs
 $ pyenv activate <name>
 $ pyenv deactivate

 # list virtualenvs
 $ pyenv virtualenvs

 # specify virtualenv or python version for a project
 # (do in root directory of the project)
 $ pyenv local <virtualenv or version>

 # view active python environment
 $ pyenv version

 # remove a python version or virtualenv
 $ pyenv uninstall <virtualenv or version>

And of course there's 'pyenv help'.

One other thing: if you're using pyenv to do openstack development, tox needs to be able to find multiple python versions. A good way to do this is to use pyenv to install the particular versions you want of py27, py35, py36, py37. Then pick the directory where your working openstack repos live (e.g., ~/repos/openstack). In that directory, you can make all the pythons you just built local:

 $ pyenv local 2.7.15 3.5.6 3.6.7 3.7.1

This creates a .python-version file in that directory with these pythons listed (vertically, if you prefer to create the file by hand). The ordering matters: 'python' in that dir or its subs will invoke the first version listed, 'python3' will invoke the first 3.x listed. Tox uses 'python3.5', 'python3.6', 'python3.7' to get the specific py3x it wants; it will get the specific version listed in the .python-version file for each of these.

Once you know that you can build pythons with pyenv, you can install a system-wide python into /usr/local (though you probably don't want to overwrite any exisiting system python that other parts of the system may be depending on).