StarlingX/Containers/K8sUpgradesTesting

There are two options for testing the kubernetes upgrade framework (when there is no new kubernetes version to test with):
 * Perform a no-delta upgrade to a custom upgrade version using re-tagged images. The advantage of this option is that it allows you to test with an existing StarlingX load.
 * Build a load for a previous kubernetes version and upgrade to the current kubernetes version. The advantage of this option is that it tests upgrading to new kubernetes images.

Testing Kubernetes Upgrades (no-delta upgrade)
The steps required:
 * Retag current kubernetes images
 * Build a set of patches for the new kubernetes version
 * Install the load and perform the upgrade

The examples below assume the release of StarlingX you are testing is using kubernetes v1.18.1.

Retag current kubernetes images
To use this option, you must install your system using a private registry, that contains all the required kubernetes images (instead of using k8s.gcr.io). You can get the list of required kuberentes images from a running StarlingX system with: kubadm config images list

Note that only the images tagged with the kubernetes version (e.g. "v1.18.1") will be retagged (e.g. to add a "-upgrade" suffix). To do this, you run the following commands for each image: docker pull :v1.18.1 docker tag :v1.18.1 :v1.18.1-upgrade docker push :v1.18.1-upgrade

Project: config
File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py def get_kube_versions: """Provides a list of supported kubernetes versions.""" return [ {'version': 'v1.18.1', 'upgrade_from': [], 'downgrade_to': [], 'applied_patches': [], 'available_patches': [], },         {'version': 'v1.18.1-upgrade', 'upgrade_from': ['v1.18.1'], 'downgrade_to': [], 'applied_patches': ['KUBE.1'], 'available_patches': ['KUBE.2'], },     ]
 * Change get_kube_versions function to add the new kubernetes version and required patches:

File: sysinv/sysinv/centos/build_srpm.data
 * Increment the TIS_PATCH_VER

Project: ansible-playbooks
Make copies of image list and templates for new version: cp -R playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.18.1 playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.18.1-upgrade cp -R playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.18.1 playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.18.1-upgrade

Otherwise 'adding' a new subcloud after an upgrade will bring up the subcloud in a mis-matched state.

File: playbookconfig/centos/build_srpm.data
 * Increment the TIS_PATCH_VER

Project: integ
File: kubernetes/kubernetes/centos/build_srpm.data
 * Change VERSION to 1.18.1_upgrade.
 * Increment the TIS_PATCH_VER

File: kubernetes/kubernetes/centos/kubernetes.spec
 * Change %global commit to 1.18.1_upgrade.
 * Change %global kube_version to 1.18.1_upgrade.

Repackage tarball for kubernetes
The tarball for the current kubernetes version needs to be repackaged: cd  cp $MY_REPO/stx/downloads/kubernetes-v1.18.1.tar.gz. tar xzvf kubernetes-v1.18.1.tar.gz mv kubernetes-1.18.1 kubernetes-1.18.1_upgrade vim kubernetes-1.18.1_upgrade/hack/lib/version.sh # Add this to the file right before KUBE_GIT_VERSION splits into major/minor KUBE_GIT_VERSION="v1.18.1-upgrade"

It should look something like:

if "${KUBE_GIT_VERSION}" =~ ^v([0-9]+)\.([0-9]+)(\.[0-9]+)?([-].*)?([+].*)?$ ; then KUBE_GIT_VERSION="v1.18.1-upgrade" KUBE_GIT_MAJOR=${BASH_REMATCH[1]} KUBE_GIT_MINOR=${BASH_REMATCH[2]} if -n "${BASH_REMATCH[4]}" ; then KUBE_GIT_MINOR+="+" fi     fi

tar czvf kubernetes-v1.18.1_upgrade.tar.gz kubernetes-1.18.1_upgrade cp kubernetes-v1.18.1_upgrade.tar.gz $MY_REPO/stx/downloads

Build the patches
The patches are built from the same workspace used to build the load with the previous kubernetes version.

First build the updated RPMS: # Build packages cd $MY_WORKSPACE build-pkgs --no-build-info --no-descendants sysinv build-pkgs --no-build-info --no-descendants playbookconfig build-pkgs --no-build-info --no-descendants kubernetes

Several scripts have been provided to build the patches - they can be found in the update project at patch-scripts/kubernetes-upgrades. Copy these files to a local directory, make any necessary edits and then invoke the make_kube_patch.sh script.

Install the load and perform the upgrade
Follow the steps described in the spec for Kubernetes Upgrades.

Testing Kubernetes Upgrades (from previous version)
The steps required:
 * Build a StarlingX load with a previous kubernetes version
 * Build a set of patches for the new kubernetes version
 * Install the load and perform the upgrade

For example, if the release of StarlingX you are testing is currently using kubernetes v1.16.2, you could build a load using kubernetes v1.16.1.

Build a StarlingX load with a previous kubernetes version
The following changes must be made and then a load must be built.

Project: config
File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py def get_kube_versions: """Provides a list of supported kubernetes versions.""" return [ {'version': 'v1.16.1', 'upgrade_from': [], 'downgrade_to': [], 'applied_patches': [], 'available_patches': [], },     ]
 * Change get_kube_versions function to specify previous version. If the previous version was v1.16.1 it would look like this:

Project: ansible-playbooks
Make copies of image list and templates for previous version. If the previous version was v1.16.1 the commands would be: cp -R playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.16.2 playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.16.1 cp -R playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.16.2 playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.16.1

Project: integ
File: kubernetes/kubernetes/centos/build_srpm.data
 * Change VERSION to previous version.

File: kubernetes/kubernetes/centos/kubernetes.spec
 * Change %global commit to previous version.
 * Change %global kube_version to previous version.

Project: stx-puppet
File: puppet-manifests/src/modules/platform/templates/kubeadm.yaml.erb
 * Change kubernetesVersion to previous version.

Retrieve tarball for kubernetes
The tarball for the previous kubernetes version must be available in the downloads directory for your workspace ($MY_REPO/stx/downloads).

Download the tarball for the previous kubernetes version: https://github.com/kubernetes/kubernetes/archive/v1.16.1.tar.gz

Copy this to your workspace and rename it as follows: mv $MY_REPO/stx/downloads/kubernetes-v1.16.1.tar.gz

Build a set of patches for the new kubernetes version
Update get_kube_versions in sysinv/common/kubernetes.py:

Project: config
File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py def get_kube_versions: return [ {'version': 'v1.16.1', 'upgrade_from': [], 'downgrade_to': [], 'applied_patches': [], 'available_patches': [], },       {'version': 'v1.16.2', 'upgrade_from': ['v1.16.1'], 'downgrade_to': [], 'applied_patches': ['KUBE.1'], 'available_patches': ['KUBE.2'], },   ]
 * Change get_kube_versions function to specify both the old and new version and the patches that will be used for the upgrade. If the previous version was v1.16.1 it would look like this:

File: sysinv/sysinv/centos/build_srpm.data
 * Increment the TIS_PATCH_VER

Project: integ
File: kubernetes/kubernetes/centos/build_srpm.data
 * Undo change to VERSION
 * Increment the TIS_PATCH_VER

File: kubernetes/kubernetes/centos/kubernetes.spec
 * Undo all changes

Build the patches
The patches are built from the same workspace used to build the load with the previous kubernetes version.

First build the updated RPMS: # Build packages cd $MY_WORKSPACE build-pkgs --no-build-info --no-descendants sysinv build-pkgs --no-build-info --no-descendants kubernetes

Several scripts have been provided to build the patches - they can be found in the update project at patch-scripts/kubernetes-upgrades. Copy these files to a local directory, make any necessary edits and then invoke the make_kube_patch.sh script.

Install the load and perform the upgrade
Follow the steps described in the spec for Kubernetes Upgrades.