StarlingX/Containers/K8sUpgradesTesting
There are two options for testing the kubernetes upgrade framework (when there is no new kubernetes version to test with):
- Perform a no-delta upgrade to a custom upgrade version using re-tagged images. The advantage of this option is that it allows you to test with an existing StarlingX load.
- Build a load for a previous kubernetes version and upgrade to the current kubernetes version. The advantage of this option is that it tests upgrading to new kubernetes images.
Contents
Testing Kubernetes Upgrades (no-delta upgrade)
The steps required:
- Retag current kubernetes images
- Build a set of patches for the new kubernetes version
- Install the load and perform the upgrade
The examples below assume the release of StarlingX you are testing is using kubernetes v1.18.1.
Retag current kubernetes images
To use this option, you must install your system using a private registry, that contains all the required kubernetes images (instead of using k8s.gcr.io). You can get the list of required kuberentes images from a running StarlingX system with:
kubadm config images list
Note that only the images tagged with the kubernetes version (e.g. "v1.18.1") will be retagged (e.g. to add a "-upgrade" suffix). To do this, you run the following commands for each image:
docker pull <image name>:v1.18.1 docker tag <image name>:v1.18.1 <image name>:v1.18.1-upgrade docker push <image name>:v1.18.1-upgrade
Build a set of patches for the new kubernetes version
Project: config
File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py
- Change get_kube_versions function to add the new kubernetes version and required patches:
def get_kube_versions(): """Provides a list of supported kubernetes versions.""" return [ {'version': 'v1.18.1', 'upgrade_from': [], 'downgrade_to': [], 'applied_patches': [], 'available_patches': [], }, {'version': 'v1.18.1-upgrade', 'upgrade_from': ['v1.18.1'], 'downgrade_to': [], 'applied_patches': ['KUBE.1'], 'available_patches': ['KUBE.2'], }, ]
File: sysinv/sysinv/centos/build_srpm.data
- Increment the TIS_PATCH_VER
Project: ansible-playbooks
Make copies of image list and templates for new version:
cp -R playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.18.1 playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.18.1-upgrade cp -R playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.18.1 playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.18.1-upgrade
(Update May 17 2021) The version of kubernetes in playbookconfig/src/playbooks/roles/common/files/kubeadm.yaml.erb must also be updated.
Otherwise 'adding' a new subcloud after an upgrade will bring up the subcloud in a mis-matched state.
File: playbookconfig/centos/build_srpm.data
- Increment the TIS_PATCH_VER
Project: integ
File: kubernetes/kubernetes/centos/build_srpm.data
- Change VERSION to 1.18.1_upgrade.
- Increment the TIS_PATCH_VER
File: kubernetes/kubernetes/centos/kubernetes.spec
- Change %global commit to 1.18.1_upgrade.
- Change %global kube_version to 1.18.1_upgrade.
Repackage tarball for kubernetes
The tarball for the current kubernetes version needs to be repackaged:
cd <work directory> cp $MY_REPO/stx/downloads/kubernetes-v1.18.1.tar.gz . tar xzvf kubernetes-v1.18.1.tar.gz mv kubernetes-1.18.1 kubernetes-1.18.1_upgrade vim kubernetes-1.18.1_upgrade/hack/lib/version.sh # Add this to the file right before KUBE_GIT_VERSION splits into major/minor KUBE_GIT_VERSION="v1.18.1-upgrade"
It should look something like:
if [[ "${KUBE_GIT_VERSION}" =~ ^v([0-9]+)\.([0-9]+)(\.[0-9]+)?([-].*)?([+].*)?$ ]]; then KUBE_GIT_VERSION="v1.18.1-upgrade" KUBE_GIT_MAJOR=${BASH_REMATCH[1]} KUBE_GIT_MINOR=${BASH_REMATCH[2]} if [[ -n "${BASH_REMATCH[4]}" ]]; then KUBE_GIT_MINOR+="+" fi fi
tar czvf kubernetes-v1.18.1_upgrade.tar.gz kubernetes-1.18.1_upgrade cp kubernetes-v1.18.1_upgrade.tar.gz $MY_REPO/stx/downloads
Build the patches
The patches are built from the same workspace used to build the load with the previous kubernetes version.
First build the updated RPMS:
# Build packages cd $MY_WORKSPACE build-pkgs --no-build-info --no-descendants sysinv build-pkgs --no-build-info --no-descendants playbookconfig build-pkgs --no-build-info --no-descendants kubernetes
Several scripts have been provided to build the patches - they can be found in the update project at patch-scripts/kubernetes-upgrades. Copy these files to a local directory, make any necessary edits and then invoke the make_kube_patch.sh script.
Install the load and perform the upgrade
Follow the steps described in the spec for Kubernetes Upgrades.
Testing Kubernetes Upgrades (from previous version)
The steps required:
- Build a StarlingX load with a previous kubernetes version
- Build a set of patches for the new kubernetes version
- Install the load and perform the upgrade
For example, if the release of StarlingX you are testing is currently using kubernetes v1.16.2, you could build a load using kubernetes v1.16.1.
Build a StarlingX load with a previous kubernetes version
The following changes must be made and then a load must be built.
Project: config
File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py
- Change get_kube_versions function to specify previous version. If the previous version was v1.16.1 it would look like this:
def get_kube_versions(): """Provides a list of supported kubernetes versions.""" return [ {'version': 'v1.16.1', 'upgrade_from': [], 'downgrade_to': [], 'applied_patches': [], 'available_patches': [], }, ]
Project: ansible-playbooks
Make copies of image list and templates for previous version. If the previous version was v1.16.1 the commands would be:
cp -R playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.16.2 playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.16.1 cp -R playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.16.2 playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.16.1
Project: integ
File: kubernetes/kubernetes/centos/build_srpm.data
- Change VERSION to previous version.
File: kubernetes/kubernetes/centos/kubernetes.spec
- Change %global commit to previous version.
- Change %global kube_version to previous version.
Project: stx-puppet
File: puppet-manifests/src/modules/platform/templates/kubeadm.yaml.erb
- Change kubernetesVersion to previous version.
Retrieve tarball for kubernetes
The tarball for the previous kubernetes version must be available in the downloads directory for your workspace ($MY_REPO/stx/downloads).
Download the tarball for the previous kubernetes version: https://github.com/kubernetes/kubernetes/archive/v1.16.1.tar.gz
Copy this to your workspace and rename it as follows:
mv <source file> $MY_REPO/stx/downloads/kubernetes-v1.16.1.tar.gz
Build a set of patches for the new kubernetes version
Update get_kube_versions in sysinv/common/kubernetes.py:
Project: config
File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py
- Change get_kube_versions function to specify both the old and new version and the patches that will be used for the upgrade. If the previous version was v1.16.1 it would look like this:
def get_kube_versions(): return [ {'version': 'v1.16.1', 'upgrade_from': [], 'downgrade_to': [], 'applied_patches': [], 'available_patches': [], }, {'version': 'v1.16.2', 'upgrade_from': ['v1.16.1'], 'downgrade_to': [], 'applied_patches': ['KUBE.1'], 'available_patches': ['KUBE.2'], }, ]
File: sysinv/sysinv/centos/build_srpm.data
- Increment the TIS_PATCH_VER
Project: integ
File: kubernetes/kubernetes/centos/build_srpm.data
- Undo change to VERSION
- Increment the TIS_PATCH_VER
File: kubernetes/kubernetes/centos/kubernetes.spec
- Undo all changes
Build the patches
The patches are built from the same workspace used to build the load with the previous kubernetes version.
First build the updated RPMS:
# Build packages cd $MY_WORKSPACE build-pkgs --no-build-info --no-descendants sysinv build-pkgs --no-build-info --no-descendants kubernetes
Several scripts have been provided to build the patches - they can be found in the update project at patch-scripts/kubernetes-upgrades. Copy these files to a local directory, make any necessary edits and then invoke the make_kube_patch.sh script.
Install the load and perform the upgrade
Follow the steps described in the spec for Kubernetes Upgrades.