Jump to: navigation, search

Difference between revisions of "StarlingX/Containers/K8sUpgradesTesting"

(Repackage tarball for kubernetes)
(5 intermediate revisions by one other user not shown)
Line 1: Line 1:
== Testing Kubernetes Upgrades ==
+
There are two options for testing the kubernetes upgrade framework (when there is no new kubernetes version to test with):
 +
* Perform a no-delta upgrade to a custom upgrade version using re-tagged images. The advantage of this option is that it allows you to test with an existing StarlingX load.
 +
* Build a load for a previous kubernetes version and upgrade to the current kubernetes version. The advantage of this option is that it tests upgrading to new kubernetes images.
  
In order to test a kubernetes upgrade, the recommended approach is to:
+
== Testing Kubernetes Upgrades (no-delta upgrade) ==
 +
 
 +
The steps required:
 +
* Retag current kubernetes images
 +
* Build a set of patches for the new kubernetes version
 +
* Install the load and perform the upgrade
 +
 
 +
The examples below assume the release of StarlingX you are testing is using kubernetes v1.18.1.
 +
 
 +
=== Retag current kubernetes images ===
 +
 
 +
To use this option, you must install your system using a private registry, that contains all the required kubernetes images (instead of using k8s.gcr.io). You can get the list of required kuberentes images from a running StarlingX system with:
 +
  kubadm config images list
 +
 
 +
Note that only the images tagged with the kubernetes version (e.g. "v1.18.1") will be retagged (e.g. to add a "-upgrade" suffix). To do this, you run the following commands for each image:
 +
  docker pull <image name>:v1.18.1
 +
  docker tag <image name>:v1.18.1 <image name>:v1.18.1-upgrade
 +
  docker push <image name>:v1.18.1-upgrade
 +
 
 +
=== Build a set of patches for the new kubernetes version ===
 +
 
 +
==== Project: config ====
 +
File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py
 +
* Change get_kube_versions function to add the new kubernetes version and required patches:
 +
  def get_kube_versions():
 +
      """Provides a list of supported kubernetes versions."""
 +
      return [
 +
          {'version': 'v1.18.1',
 +
          'upgrade_from': [],
 +
          'downgrade_to': [],
 +
          'applied_patches': [],
 +
          'available_patches': [],
 +
          },
 +
          {'version': 'v1.18.1-upgrade',
 +
          'upgrade_from': ['v1.18.1'],
 +
          'downgrade_to': [],
 +
          'applied_patches': ['KUBE.1'],
 +
          'available_patches': ['KUBE.2'],
 +
          },
 +
      ]
 +
 
 +
File: sysinv/sysinv/centos/build_srpm.data
 +
* Increment the '''TIS_PATCH_VER'''
 +
 
 +
==== Project: ansible-playbooks ====
 +
Make copies of image list and templates for new version:
 +
  cp -R playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.18.1 playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.18.1-upgrade
 +
  cp -R playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.18.1 playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.18.1-upgrade
 +
 
 +
File: playbookconfig/centos/build_srpm.data
 +
* Increment the '''TIS_PATCH_VER'''
 +
 
 +
==== Project: integ ====
 +
File: kubernetes/kubernetes/centos/build_srpm.data
 +
* Change '''VERSION''' to 1.18.1_upgrade.
 +
* Increment the '''TIS_PATCH_VER'''
 +
 
 +
File: kubernetes/kubernetes/centos/kubernetes.spec
 +
* Change '''%global commit''' to 1.18.1_upgrade.
 +
* Change '''%global kube_version''' to 1.18.1_upgrade.
 +
 
 +
==== Repackage tarball for kubernetes ====
 +
The tarball for the current kubernetes version needs to be repackaged:
 +
  cd <work directory>
 +
  cp $MY_REPO/stx/downloads/kubernetes-v1.18.1.tar.gz .
 +
  tar xzvf kubernetes-v1.18.1.tar.gz
 +
  mv kubernetes-1.18.1 kubernetes-1.18.1_upgrade
 +
  vim kubernetes-1.18.1_upgrade/hack/lib/version.sh
 +
  # Add this to the file right before KUBE_GIT_VERSION splits into major/minor
 +
  KUBE_GIT_VERSION="v1.18.1-upgrade"
 +
 
 +
It should look something like:
 +
 
 +
    if [[ "${KUBE_GIT_VERSION}" =~ ^v([0-9]+)\.([0-9]+)(\.[0-9]+)?([-].*)?([+].*)?$ ]]; then
 +
        KUBE_GIT_VERSION="v1.18.1-upgrade"
 +
        KUBE_GIT_MAJOR=${BASH_REMATCH[1]}
 +
        KUBE_GIT_MINOR=${BASH_REMATCH[2]}
 +
        if [[ -n "${BASH_REMATCH[4]}" ]]; then
 +
          KUBE_GIT_MINOR+="+"
 +
        fi
 +
      fi
 +
 
 +
 
 +
  tar czvf kubernetes-v1.18.1_upgrade.tar.gz kubernetes-1.18.1_upgrade
 +
  cp kubernetes-v1.18.1_upgrade.tar.gz $MY_REPO/stx/downloads
 +
 
 +
==== Build the patches ====
 +
 
 +
The patches are built from the same workspace used to build the load with the previous kubernetes version.
 +
 
 +
First build the updated RPMS:
 +
  # Build packages
 +
  cd $MY_WORKSPACE
 +
  build-pkgs --no-build-info --no-descendants sysinv
 +
  build-pkgs --no-build-info --no-descendants playbookconfig
 +
  build-pkgs --no-build-info --no-descendants kubernetes
 +
 
 +
Several scripts have been provided to build the patches - they can be found in the update project at patch-scripts/kubernetes-upgrades. Copy these files to a local directory, make any necessary edits and then invoke the make_kube_patch.sh script.
 +
 
 +
=== Install the load and perform the upgrade ===
 +
 
 +
Follow the steps described in the spec for Kubernetes Upgrades.
 +
 
 +
== Testing Kubernetes Upgrades (from previous version) ==
 +
 
 +
The steps required:
 
* Build a StarlingX load with a previous kubernetes version
 
* Build a StarlingX load with a previous kubernetes version
 
* Build a set of patches for the new kubernetes version
 
* Build a set of patches for the new kubernetes version
Line 25: Line 132:
 
           },
 
           },
 
       ]
 
       ]
 +
 +
==== Project: ansible-playbooks ====
 +
Make copies of image list and templates for previous version. If the previous version was v1.16.1 the commands would be:
 +
  cp -R playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.16.2 playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.16.1
 +
  cp -R playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.16.2 playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.16.1
  
 
==== Project: integ ====
 
==== Project: integ ====

Revision as of 20:33, 9 November 2020

There are two options for testing the kubernetes upgrade framework (when there is no new kubernetes version to test with):

  • Perform a no-delta upgrade to a custom upgrade version using re-tagged images. The advantage of this option is that it allows you to test with an existing StarlingX load.
  • Build a load for a previous kubernetes version and upgrade to the current kubernetes version. The advantage of this option is that it tests upgrading to new kubernetes images.

Testing Kubernetes Upgrades (no-delta upgrade)

The steps required:

  • Retag current kubernetes images
  • Build a set of patches for the new kubernetes version
  • Install the load and perform the upgrade

The examples below assume the release of StarlingX you are testing is using kubernetes v1.18.1.

Retag current kubernetes images

To use this option, you must install your system using a private registry, that contains all the required kubernetes images (instead of using k8s.gcr.io). You can get the list of required kuberentes images from a running StarlingX system with:

 kubadm config images list

Note that only the images tagged with the kubernetes version (e.g. "v1.18.1") will be retagged (e.g. to add a "-upgrade" suffix). To do this, you run the following commands for each image:

 docker pull <image name>:v1.18.1
 docker tag <image name>:v1.18.1 <image name>:v1.18.1-upgrade
 docker push <image name>:v1.18.1-upgrade

Build a set of patches for the new kubernetes version

Project: config

File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py

  • Change get_kube_versions function to add the new kubernetes version and required patches:
 def get_kube_versions():
     """Provides a list of supported kubernetes versions."""
     return [
         {'version': 'v1.18.1',
          'upgrade_from': [],
          'downgrade_to': [],
          'applied_patches': [],
          'available_patches': [],
          },
         {'version': 'v1.18.1-upgrade',
          'upgrade_from': ['v1.18.1'],
          'downgrade_to': [],
          'applied_patches': ['KUBE.1'],
          'available_patches': ['KUBE.2'],
          },
     ]

File: sysinv/sysinv/centos/build_srpm.data

  • Increment the TIS_PATCH_VER

Project: ansible-playbooks

Make copies of image list and templates for new version:

 cp -R playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.18.1 playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.18.1-upgrade
 cp -R playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.18.1 playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.18.1-upgrade

File: playbookconfig/centos/build_srpm.data

  • Increment the TIS_PATCH_VER

Project: integ

File: kubernetes/kubernetes/centos/build_srpm.data

  • Change VERSION to 1.18.1_upgrade.
  • Increment the TIS_PATCH_VER

File: kubernetes/kubernetes/centos/kubernetes.spec

  • Change %global commit to 1.18.1_upgrade.
  • Change %global kube_version to 1.18.1_upgrade.

Repackage tarball for kubernetes

The tarball for the current kubernetes version needs to be repackaged:

 cd <work directory>
 cp $MY_REPO/stx/downloads/kubernetes-v1.18.1.tar.gz .
 tar xzvf kubernetes-v1.18.1.tar.gz
 mv kubernetes-1.18.1 kubernetes-1.18.1_upgrade
 vim kubernetes-1.18.1_upgrade/hack/lib/version.sh
 # Add this to the file right before KUBE_GIT_VERSION splits into major/minor
 KUBE_GIT_VERSION="v1.18.1-upgrade"

It should look something like:

   if [[ "${KUBE_GIT_VERSION}" =~ ^v([0-9]+)\.([0-9]+)(\.[0-9]+)?([-].*)?([+].*)?$ ]]; then
       KUBE_GIT_VERSION="v1.18.1-upgrade"
       KUBE_GIT_MAJOR=${BASH_REMATCH[1]}
       KUBE_GIT_MINOR=${BASH_REMATCH[2]}
       if [[ -n "${BASH_REMATCH[4]}" ]]; then
         KUBE_GIT_MINOR+="+"
       fi
     fi


 tar czvf kubernetes-v1.18.1_upgrade.tar.gz kubernetes-1.18.1_upgrade
 cp kubernetes-v1.18.1_upgrade.tar.gz $MY_REPO/stx/downloads

Build the patches

The patches are built from the same workspace used to build the load with the previous kubernetes version.

First build the updated RPMS:

 # Build packages
 cd $MY_WORKSPACE
 build-pkgs --no-build-info --no-descendants sysinv
 build-pkgs --no-build-info --no-descendants playbookconfig
 build-pkgs --no-build-info --no-descendants kubernetes

Several scripts have been provided to build the patches - they can be found in the update project at patch-scripts/kubernetes-upgrades. Copy these files to a local directory, make any necessary edits and then invoke the make_kube_patch.sh script.

Install the load and perform the upgrade

Follow the steps described in the spec for Kubernetes Upgrades.

Testing Kubernetes Upgrades (from previous version)

The steps required:

  • Build a StarlingX load with a previous kubernetes version
  • Build a set of patches for the new kubernetes version
  • Install the load and perform the upgrade

For example, if the release of StarlingX you are testing is currently using kubernetes v1.16.2, you could build a load using kubernetes v1.16.1.

Build a StarlingX load with a previous kubernetes version

The following changes must be made and then a load must be built.

Project: config

File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py

  • Change get_kube_versions function to specify previous version. If the previous version was v1.16.1 it would look like this:
 def get_kube_versions():
     """Provides a list of supported kubernetes versions."""
     return [
         {'version': 'v1.16.1',
          'upgrade_from': [],
          'downgrade_to': [],
          'applied_patches': [],
          'available_patches': [],
          },
     ]

Project: ansible-playbooks

Make copies of image list and templates for previous version. If the previous version was v1.16.1 the commands would be:

 cp -R playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.16.2 playbookconfig/src/playbooks/roles/common/push-docker-images/vars/k8s-v1.16.1
 cp -R playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.16.2 playbookconfig/src/playbooks/roles/bootstrap/bringup-essential-services/templates/k8s-v1.16.1

Project: integ

File: kubernetes/kubernetes/centos/build_srpm.data

  • Change VERSION to previous version.

File: kubernetes/kubernetes/centos/kubernetes.spec

  • Change %global commit to previous version.
  • Change %global kube_version to previous version.

Project: stx-puppet

File: puppet-manifests/src/modules/platform/templates/kubeadm.yaml.erb

  • Change kubernetesVersion to previous version.

Retrieve tarball for kubernetes

The tarball for the previous kubernetes version must be available in the downloads directory for your workspace ($MY_REPO/stx/downloads).

Download the tarball for the previous kubernetes version: https://github.com/kubernetes/kubernetes/archive/v1.16.1.tar.gz

Copy this to your workspace and rename it as follows:

 mv <source file> $MY_REPO/stx/downloads/kubernetes-v1.16.1.tar.gz

Build a set of patches for the new kubernetes version

Update get_kube_versions in sysinv/common/kubernetes.py:

Project: config

File: sysinv/sysinv/sysinv/sysinv/common/kubernetes.py

  • Change get_kube_versions function to specify both the old and new version and the patches that will be used for the upgrade. If the previous version was v1.16.1 it would look like this:
 def get_kube_versions():
   return [
       {'version': 'v1.16.1',
        'upgrade_from': [],
        'downgrade_to': [],
        'applied_patches': [],
        'available_patches': [],
        },
       {'version': 'v1.16.2',
        'upgrade_from': ['v1.16.1'],
        'downgrade_to': [],
        'applied_patches': ['KUBE.1'],
        'available_patches': ['KUBE.2'],
        },
   ]

File: sysinv/sysinv/centos/build_srpm.data

  • Increment the TIS_PATCH_VER

Project: integ

File: kubernetes/kubernetes/centos/build_srpm.data

  • Undo change to VERSION
  • Increment the TIS_PATCH_VER

File: kubernetes/kubernetes/centos/kubernetes.spec

  • Undo all changes

Build the patches

The patches are built from the same workspace used to build the load with the previous kubernetes version.

First build the updated RPMS:

 # Build packages
 cd $MY_WORKSPACE
 build-pkgs --no-build-info --no-descendants sysinv
 build-pkgs --no-build-info --no-descendants kubernetes

Several scripts have been provided to build the patches - they can be found in the update project at patch-scripts/kubernetes-upgrades. Copy these files to a local directory, make any necessary edits and then invoke the make_kube_patch.sh script.

Install the load and perform the upgrade

Follow the steps described in the spec for Kubernetes Upgrades.