StarlingX/Containers/Applications/HowToAddNewFluxCDAppInSTX
This page details steps required to add a new FluxCD Helm chart for a system-managed application to StarlingX.
In order to build STX packages and applications, the developer will need to set up the StarlingX Debian Build Environment.
More information is available about controlling application behavior and integration with the platform.
The recommended naming convention is now: app-<your-app-name>
Contents
- 1 Create FluxCD app
- 2 Introduction
- 3 Create top level repo in openstack/project-config for new Application
- 4 Basic setup of Application Repo
- 5 Update Starlingx/manifest & StarlingX/root to include new Application Repo
- 6 Setup a "StarlingX Master" DEV environment for developing your application
- 7 Develop your application FluxCD packaging
- 8 Develop your Container Images
- 9 Develop your System Application Plugins
- 10 Add the build of your Container Image(s) into main Container-Build
- 11 Finalize Container Image Tag for Starlingx-built Container Images
- 12 Update the image static tag in system app helm charts
- 13 Add the application to the ISO
- 14 Upload, Apply and Test System Application
- 15 Test STX build
- 16 Updating charts from upstream apps in debian
- 17 Commit, Review and Merge your Application Repo Change
Create FluxCD app
Create a top-level folder for the new app under cgcs-root/stx folder.
It is recommended to start off copying another app code structure as starting point and then make necessary changes to various configuration build & configuration files.
As an overview, this is the directory structure for app-rook-ceph:
├── bindep.txt ├── debian_build_layer.cfg ├── debian_bullseye_iso_image_std.inc ├── debian_bullseye_pkg_dirs_std ├── debian_trixie_iso_image_std.inc ├── debian_trixie_pkg_dirs_std ├── helm-charts │ ├── custom │ │ ├── rook-ceph-floating-monitor-helm │ │ │ ├── debian │ │ │ │ └── all │ │ │ │ ├── deb_folder │ │ │ │ │ ├── changelog │ │ │ │ │ ├── control │ │ │ │ │ ├── copyright │ │ │ │ │ ├── rook-ceph-floating-monitor-helm.install │ │ │ │ │ ├── rules │ │ │ │ │ └── source │ │ │ │ │ └── format │ │ │ │ └── meta_data.yaml │ │ │ └── rook-ceph-floating-monitor-helm │ │ │ ├── Makefile │ │ │ └── rook-ceph-floating-monitor │ │ │ ├── Chart.yaml │ │ │ ├── templates │ │ │ │ ├── deployment.yaml │ │ │ │ ├── post-delete-job.yaml │ │ │ │ ├── post-install-job.yaml │ │ │ │ ├── pre-delete-job.yaml │ │ │ │ ├── pre-install-job.yaml │ │ │ │ └── service.yaml │ │ │ └── values.yaml │ │ └── rook-ceph-provisioner-helm │ │ ├── debian │ │ │ └── all │ │ │ ├── deb_folder │ │ │ │ ├── changelog │ │ │ │ ├── control │ │ │ │ ├── copyright │ │ │ │ ├── rook-ceph-provisioner-helm.install │ │ │ │ ├── rules │ │ │ │ └── source │ │ │ │ └── format │ │ │ └── meta_data.yaml │ │ └── rook-ceph-provisioner-helm │ │ ├── Makefile │ │ └── rook-ceph-provisioner │ │ ├── Chart.yaml │ │ ├── files │ │ │ └── sync_osds.py │ │ ├── templates │ │ │ ├── clusterrolebinding.yaml │ │ │ ├── clusterrole.yaml │ │ │ ├── configmap.yaml │ │ │ ├── deployment-stx-ceph-manager.yaml │ │ │ ├── _helpers.tpl │ │ │ ├── job-ceph-mon-audit.yaml │ │ │ ├── job-ceph-osd-audit.yaml │ │ │ ├── job-stx-ceph-mgr-provision.yaml │ │ │ ├── job-sync-osds.yaml │ │ │ ├── post-install-host-provision.yaml │ │ │ ├── post-install-mgr-restful.yaml │ │ │ ├── post-install-rook-provision.yaml │ │ │ ├── pre-delete-rook-provisioner-cleanup.yaml │ │ │ ├── rolebinding.yaml │ │ │ ├── role.yaml │ │ │ └── serviceaccount.yaml │ │ └── values.yaml │ └── upstream │ └── rook-ceph-helm │ ├── debian │ │ └── all │ │ ├── deb_folder │ │ │ ├── changelog │ │ │ ├── control │ │ │ ├── copyright │ │ │ ├── patches │ │ │ │ ├── 0001-Add-chart-for-duplex-preparation.patch │ │ │ │ ├── 0002-Add-starlingx-label-to-operator-and-toolbox-pod.patch │ │ │ │ ├── 0003-Add-support-ceph-dashboard.patch │ │ │ │ ├── 0004-Adjust-the-image-format-as-expected-by-kube-app.patch │ │ │ │ └── series │ │ │ ├── rook-ceph-helm.install │ │ │ ├── rules │ │ │ └── source │ │ │ └── format │ │ └── meta_data.yaml │ └── rook-ceph-helm │ └── files │ └── Makefile ├── python3-k8sapp-rook-ceph │ ├── debian │ │ └── all │ │ ├── deb_folder │ │ │ ├── changelog │ │ │ ├── control │ │ │ ├── copyright │ │ │ ├── python3-k8sapp-rook-ceph.install │ │ │ ├── python3-k8sapp-rook-ceph-wheels.install │ │ │ ├── rules │ │ │ └── source │ │ │ └── format │ │ └── meta_data.yaml │ └── k8sapp_rook_ceph │ ├── k8sapp_rook_ceph │ │ ├── common │ │ │ ├── ceph_utils.py │ │ │ ├── constants.py │ │ │ ├── __init__.py │ │ │ ├── kube_utils.py │ │ │ └── utils.py │ │ ├── helm │ │ │ ├── __init__.py │ │ │ ├── rook_ceph_cluster.py │ │ │ ├── rook_ceph_floating_monitor.py │ │ │ ├── rook_ceph_provisioner.py │ │ │ ├── rook_ceph.py │ │ │ └── storage.py │ │ ├── __init__.py │ │ ├── kustomize │ │ │ ├── __init__.py │ │ │ └── kustomize_rook_ceph.py │ │ ├── lifecycle │ │ │ ├── __init__.py │ │ │ ├── lifecycle_rook_ceph.py │ │ │ └── templates │ │ │ ├── cm-remove-osds-utils.yaml │ │ │ ├── job-remove-osds.yaml │ │ │ ├── job-remove-pluginholder-cephfs.yaml │ │ │ ├── job-remove-pluginholder-rbd.yaml │ │ │ ├── job-wipe-disks.yaml │ │ │ ├── post-fixed-mons-update-in-dx.yaml │ │ │ └── sa-wipe-disks.yaml │ │ └── tests │ │ ├── __init__.py │ │ ├── test_plugins.py │ │ └── test_rook.py │ ├── LICENSE │ ├── pylint.rc │ ├── README.rst │ ├── requirements.txt │ ├── setup.cfg │ ├── setup.py │ ├── test-requirements.txt │ ├── tox.ini │ └── upper-constraints.txt ├── README.md ├── requirements.txt ├── stx-rook-ceph-helm │ ├── debian │ │ └── all │ │ ├── deb_folder │ │ │ ├── changelog │ │ │ ├── control │ │ │ ├── copyright │ │ │ ├── rules │ │ │ ├── source │ │ │ │ └── format │ │ │ └── stx-rook-ceph-helm.install │ │ └── meta_data.yaml │ ├── files │ │ ├── bash_completion │ │ ├── ceph-wrapper.sh │ │ └── rook-mon-exit.sh │ └── stx-rook-ceph-helm │ ├── files │ │ └── metadata.yaml │ └── fluxcd-manifests │ ├── base │ │ ├── helmrepository.yaml │ │ ├── kustomization.yaml │ │ └── namespace.yaml │ ├── kustomization.yaml │ ├── rook-ceph │ │ ├── helmrelease.yaml │ │ ├── kustomization.yaml │ │ ├── rook-ceph-static-overrides.yaml │ │ ├── rook-ceph-system-overrides.yaml │ │ └── service-account-default.yaml │ ├── rook-ceph-cluster │ │ ├── helmrelease.yaml │ │ ├── kustomization.yaml │ │ ├── rook-ceph-cluster-static-block-overrides.yaml -> rook-ceph-cluster-static-blockpool-overrides.yaml │ │ ├── rook-ceph-cluster-static-blockpool-overrides.yaml │ │ ├── rook-ceph-cluster-static-ecblockpool-overrides.yaml │ │ ├── rook-ceph-cluster-static-file-overrides.yaml -> rook-ceph-cluster-static-filesystem-overrides.yaml │ │ ├── rook-ceph-cluster-static-filesystem-overrides.yaml │ │ ├── rook-ceph-cluster-static-no-overrides.yaml │ │ ├── rook-ceph-cluster-static-objectstore-overrides.yaml │ │ ├── rook-ceph-cluster-static-obj-overrides.yaml -> rook-ceph-cluster-static-objectstore-overrides.yaml │ │ ├── rook-ceph-cluster-static-overrides.yaml │ │ └── rook-ceph-cluster-system-overrides.yaml │ ├── rook-ceph-floating-monitor │ │ ├── helmrelease.yaml │ │ ├── kustomization.yaml │ │ ├── rook-ceph-floating-monitor-static-overrides.yaml │ │ └── rook-ceph-floating-monitor-system-overrides.yaml │ └── rook-ceph-provisioner │ ├── helmrelease.yaml │ ├── kustomization.yaml │ ├── rook-ceph-provisioner-static-overrides.yaml │ └── rook-ceph-provisioner-system-overrides.yaml ├── test-requirements.txt └── tox.ini
The file structure does receive minor updates over time, so for the current version as well as the contents of each file in this application example, please check the following link
The debian folder contents are set up according to the following guide StarlingX/DebianBuildStructure. For more in-depth information on the configuration options available for each file, the developer may wish to consult the Debian source package documentation. In general, it is recommended to use an already available app as a template since most of the configuration options available aren't required.
Introduction
There are typically two use cases when building a new system application:
An external opensource application that is being packaged as a system application for easy integration/use in StarlingX where: the container images come from a public docker registry (e.g. docker.io, gcr.io, quay.io, ...); i.e. as released from an opensource project, the helm chart(s) comes from a specific commit or release of an opensource project, you are writing the FluxCD manifest, provide "recommended deployment" by automatically setting tested default helm overrides for application-specific parameters, you are writing the system application plugins to: dynamically set helm chart(s) overrrides based on the current StarlingX infrastructure configuration (e.g. replicas=2 for duplex systems), provide custom behaviour on 'system application-<operation> ...' StarlingX management of the FluxCD application packaging. etc. OR
An internally developed (i.e. StarlingX-developed) containerized application, where: you are developing and building the container image(s), you are writing the helm chart(s) ... and all its yaml manifest files, its default values, etc. you are writing the FluxCD manifest, you are writing the system application plugins to: dynamically set helm chart(s) overrrides based on the current StarlingX infrastructure configuration (e.g. replicas=2 for duplex systems), provide custom behaviour on 'system application-<operation> ...' StarlingX management of the FluxCD application packaging. etc.
NOTE: Some steps below are specific to one usecase versus the other.
NOTE: Some steps can be skipped if you are just working / experimenting locally in a local developer environment.
Create top level repo in openstack/project-config for new Application
This step has an external dependency (openstack infra team) and may take a few days to resolve (i.e. for openstack infra team to review and create new repo).
This step can be skipped if you are just working/experimenting locally in a local developer environment.
Process details are covered here (Skip PyPi part): https://docs.openstack.org/infra/manual/creators.html
In short, the following steps will be needed:
git clone https://git.openstack.org/openstack-infra/project-config cd project-config
Edit the following files to add entries for your new application (e.g. 'starlingx/app-NAME'). Note that the project names have to follow lexicographical order in the list.
gerrit/projects.yaml ← existing file
zuul/main.yaml ← existing file
gerrit/acls/starlingx/<your-project-here>.config ← create new file for this
See this commit for an example of the specific changes required to the files above: https://review.opendev.org/c/openstack/project-config/+/834896 (Ignore the 'armada' in the app name)
AFTER THE ABOVE COMMIT IS MERGED by the external openstack infra team, you must contact the build team (Build (DevOps) Team) to create a new core reviewers group for your new application repo, in opendev gerrit, and populate it with at least two starlingx members.
Basic setup of Application Repo
Once the first step has been completely approved and the repository has been created, the following standard repo files need to be added and committed to your new application repo.
.gitreview .zuul.yaml requirements.txt test-requirements.txt tox.ini
Git clone your new application repo, setup branch in your application repo, create files, commit for review, and merge into git: ( See example commit here: https://review.opendev.org/#/c/716429/ )
git clone https://opendev.org/starlingx/app-NAME cd app-NAME git review -s ← // For Debian, this command might need to be used from outside the build containers git checkout -b setupAPPNAMEApp
// Create the following files in app-NAME/
.gitreview # copy from an existing system app repo and change git name
requirements.txt # copy from an existing system app repo
test-requirements.txt # copy from an existing system app repo
tox.ini # copy from an existing system app repo
.zuul.yaml # copy from an existing system app repo change git name, and update keys in this file for mirroring this new repo from
# OpenDev to GitHub
Wind River's CNCF/Kubernetes certification requires all repos to be mirrored on GitHub. To achieve this, the file needs a number of keys to authorize the mirroring.
- The 'host_key' to GitHub remains the same.
- The key from other projects can be used as-is.
- Generate 'ssh_key' entry. Details of how to generate are captured in this doc: https://docs.starlingx.io/developer_resources/mirror_repo.html. In short, the steps are:
- Run the zuul/tools/encrypt_secret.py script script (details in the doc linked above). Add these keys to the .zuul.yaml file.
- Login to GitHub with username starlingx.github@gmail.com (contact: bin.qian@windriver.com)
- Create repo with preferably the same name
// Commit this to starlingx master
git add .gitreview git add requirements.txt git add test-requirements.txt git add tox.ini git add .zuul.yaml git commit -s
// Upload for review
git review
Update Starlingx/manifest & StarlingX/root to include new Application Repo
This step can be skipped if you are just working/experimenting locally in a local developer environment.
starlingx/manifest repo update
Once the previous step has been completed (wait until commit merged), update the https://opendev.org/starlingx/manifest repo to include the new project.
The files default.xml and flock.xml (or may be other file depending on type of layered build) will need to be updated with new application git details.
See example commit here: https://review.opendev.org/#/c/716117/
starlingx/root repo update
Update the https://opendev.org/starlingx/root repo with new application repo details; i.e. specifically updating ./stx/.gitignore .
See example commit here: https://review.opendev.org/#/c/720193/
After these commits merge, when running 'repo-sync' (as part of the setup of your general DEV environment, e.g. see StarlingX Development Environment ), your new application repo will be cloned into that environment.
Setup a "StarlingX Master" DEV environment for developing your application
Setup a debian build environment by following https://wiki.openstack.org/wiki/StarlingX/DebianBuildEnvironment
Create branch and review under your new application repo in order to start creating files for the development of your new application repo e.g.
cd ${REPO_ROOT}/cgcs-root/stx/app-NAME
git review -s
repo start mybranch
// If you are working/experimenting locally in a local developer environment, // INSTEAD of doing the above commands, // you will have to create your top-level application directory at this point
cd ${REPO_ROOT}/cgcs-root/stx/
mkdir my-app/
cd my-app
git init // some of spec files expect to be inside git repo
... the following steps will assume you are working in this dev environment.
Develop your application FluxCD packaging
Basic application directory structure
├── bindep.txt ├── debian_build_layer.cfg ├── debian_iso_image.inc ├── debian_pkg_dirs ├── helm-charts │ ├── custom │ │ ├── rook-ceph-floating-monitor-helm │ │ │ ├── debian │ │ │ │ ├── deb_folder │ │ │ │ │ ├── changelog │ │ │ │ │ ├── control │ │ │ │ │ ├── copyright │ │ │ │ │ ├── rook-ceph-floating-monitor-helm.install │ │ │ │ │ ├── rules │ │ │ │ │ └── source │ │ │ │ │ └── format │ │ │ │ └── meta_data.yaml │ │ │ └── rook-ceph-floating-monitor-helm │ │ │ ├── Makefile │ │ │ └── rook-ceph-floating-monitor │ │ │ ├── Chart.yaml │ │ │ ├── templates │ │ │ │ ├── deployment.yaml │ │ │ │ ├── post-delete-job.yaml │ │ │ │ ├── post-install-job.yaml │ │ │ │ ├── pre-delete-job.yaml │ │ │ │ ├── pre-install-job.yaml │ │ │ │ └── service.yaml │ │ │ └── values.yaml │ │ └── rook-ceph-provisioner-helm │ │ ├── debian │ │ │ ├── deb_folder │ │ │ │ ├── changelog │ │ │ │ ├── control │ │ │ │ ├── copyright │ │ │ │ ├── rook-ceph-provisioner-helm.install │ │ │ │ ├── rules │ │ │ │ └── source │ │ │ │ └── format │ │ │ └── meta_data.yaml │ │ └── rook-ceph-provisioner-helm │ │ ├── Makefile │ │ └── rook-ceph-provisioner │ │ ├── Chart.yaml │ │ ├── files │ │ │ └── sync_osds.py │ │ ├── templates │ │ │ ├── clusterrolebinding.yaml │ │ │ ├── clusterrole.yaml │ │ │ ├── configmap.yaml │ │ │ ├── deployment-stx-ceph-manager.yaml │ │ │ ├── _helpers.tpl │ │ │ ├── job-ceph-mon-audit.yaml │ │ │ ├── job-ceph-osd-audit.yaml │ │ │ ├── job-stx-ceph-mgr-provision.yaml │ │ │ ├── job-sync-osds.yaml │ │ │ ├── post-install-host-provision.yaml │ │ │ ├── post-install-mgr-restful.yaml │ │ │ ├── post-install-rook-provision.yaml │ │ │ ├── pre-delete-rook-provisioner-cleanup.yaml │ │ │ ├── rolebinding.yaml │ │ │ ├── role.yaml │ │ │ ├── rook-teardown-script.yaml │ │ │ └── serviceaccount.yaml │ │ └── values.yaml │ └── upstream │ └── rook-ceph-helm │ ├── debian │ │ ├── deb_folder │ │ │ ├── changelog │ │ │ ├── control │ │ │ ├── copyright │ │ │ ├── patches │ │ │ │ ├── 0001-Add-chart-for-duplex-preparation.patch │ │ │ │ ├── 0002-Add-starlingx-label-to-operator-and-toolbox-pod.patch │ │ │ │ ├── 0003-Add-support-ceph-dashboard.patch │ │ │ │ └── series │ │ │ ├── rook-ceph-helm.install │ │ │ ├── rules │ │ │ └── source │ │ │ └── format │ │ └── meta_data.yaml │ └── rook-ceph-helm │ └── files │ └── Makefile ├── python3-k8sapp-rook-ceph ← // Application Framework Plugins │ ├── debian │ │ ├── deb_folder │ │ │ ├── changelog │ │ │ ├── control │ │ │ ├── copyright │ │ │ ├── python3-k8sapp-rook-ceph.install │ │ │ ├── python3-k8sapp-rook-ceph-wheels.install │ │ │ ├── rules │ │ │ └── source │ │ │ └── format │ │ └── meta_data.yaml │ └── k8sapp_rook_ceph │ ├── k8sapp_rook_ceph │ │ ├── common │ │ │ ├── ceph_utils.py │ │ │ ├── constants.py │ │ │ ├── __init__.py │ │ │ └── utils.py │ │ ├── helm │ │ │ ├── __init__.py │ │ │ ├── rook_ceph_cluster.py │ │ │ ├── rook_ceph_floating_monitor.py │ │ │ ├── rook_ceph_provisioner.py │ │ │ ├── rook_ceph.py │ │ │ └── storage.py │ │ ├── __init__.py │ │ ├── kustomize │ │ │ ├── __init__.py │ │ │ └── kustomize_rook_ceph.py │ │ ├── lifecycle │ │ │ ├── __init__.py │ │ │ ├── lifecycle_rook_ceph.py │ │ │ └── templates │ │ │ ├── cm-remove-osds-utils.yaml │ │ │ ├── job-remove-osds.yaml │ │ │ ├── job-remove-pluginholder-cephfs.yaml │ │ │ ├── job-remove-pluginholder-rbd.yaml │ │ │ ├── job-wipe-disks.yaml │ │ │ ├── post-fixed-mons-update-in-dx.yaml │ │ │ └── sa-wipe-disks.yaml │ │ └── tests │ │ ├── __init__.py │ │ ├── test_plugins.py │ │ └── test_rook.py │ ├── LICENSE │ ├── pylint.rc │ ├── README.rst │ ├── requirements.txt │ ├── setup.cfg │ ├── setup.py │ ├── test-requirements.txt │ ├── tox.ini │ └── upper-constraints.txt ├── README.md ├── requirements.txt ├── stx-rook-ceph-helm │ ├── debian ← // For specific information on the StarlingX debian folder structure: https://wiki.openstack.org/wiki/StarlingX/DebianBuildStructure │ │ ├── deb_folder │ │ │ ├── changelog │ │ │ ├── control │ │ │ ├── copyright │ │ │ ├── rules │ │ │ ├── source │ │ │ │ └── format │ │ │ └── stx-rook-ceph-helm.install │ │ └── meta_data.yaml │ ├── files │ │ ├── bash_completion │ │ ├── ceph-wrapper.sh │ │ └── rook-mon-exit.sh │ └── stx-rook-ceph-helm │ ├── files │ │ └── metadata.yaml │ └── fluxcd-manifests │ ├── base │ │ ├── helmrepository.yaml │ │ ├── kustomization.yaml │ │ └── namespace.yaml │ ├── kustomization.yaml │ ├── rook-ceph │ │ ├── helmrelease.yaml │ │ ├── kustomization.yaml │ │ ├── rook-ceph-static-overrides.yaml │ │ ├── rook-ceph-system-overrides.yaml │ │ └── service-account-default.yaml │ ├── rook-ceph-cluster │ │ ├── helmrelease.yaml │ │ ├── kustomization.yaml │ │ ├── rook-ceph-cluster-static-block-overrides.yaml -> rook-ceph-cluster-static-blockpool-overrides.yaml │ │ ├── rook-ceph-cluster-static-blockpool-overrides.yaml │ │ ├── rook-ceph-cluster-static-ecblockpool-overrides.yaml │ │ ├── rook-ceph-cluster-static-file-overrides.yaml -> rook-ceph-cluster-static-filesystem-overrides.yaml │ │ ├── rook-ceph-cluster-static-filesystem-overrides.yaml │ │ ├── rook-ceph-cluster-static-no-overrides.yaml │ │ ├── rook-ceph-cluster-static-objectstore-overrides.yaml │ │ ├── rook-ceph-cluster-static-obj-overrides.yaml -> rook-ceph-cluster-static-objectstore-overrides.yaml │ │ ├── rook-ceph-cluster-static-overrides.yaml │ │ └── rook-ceph-cluster-system-overrides.yaml │ ├── rook-ceph-floating-monitor │ │ ├── helmrelease.yaml │ │ ├── kustomization.yaml │ │ ├── rook-ceph-floating-monitor-static-overrides.yaml │ │ └── rook-ceph-floating-monitor-system-overrides.yaml │ └── rook-ceph-provisioner │ ├── helmrelease.yaml │ ├── kustomization.yaml │ ├── rook-ceph-provisioner-static-overrides.yaml │ └── rook-ceph-provisioner-system-overrides.yaml ├── test-requirements.txt └── tox.ini
This is an example of the repo structure of a functional app: https://opendev.org/starlingx/app-rook-ceph
Create the above application repo directory structure under ${REPO_ROOT}/cgcs-root/stx/app-NAME in your DEV environment, using the integration doc as a reference.
For example, for an application that simply uses a helm chart from an opensource application source tarball in downloads/ folder, copy the nginx-ingress-controller-armada-app directory structure and make updates accordingly.
Alternatively, if you want to write a proof of concept before diving into packaging, the Application Generator Tool can be used to create a packaged application given an existing Helm chart. The resulting application tarball can be uploaded to a StarlingX load for preliminary testing.
Suggested changes to an existing app structure
- Copy app directory structure from nginx-ingress-controller-armada-app, app-rook-ceph or any other maintained app.
- Remove patch files
- Rename directories, replacing 'nginx-ingress-controller' with 'APPNAME'
- Update contents of debian_iso_image.inc, debian_pkg_dirs
- Update files under ./stx-APPNAME-helm/stx-app-NAME-helm/fluxcd-manifests/
- Update contents of your kustomization.yaml, helmrelease.yaml, and namespace.yaml files
- Change namespace from kube-system to something specific if you want
- Update contents of your APPNAME-static-overrides.yaml file
- Update your ./stx-harbor-helm/stx-harbor-helm/files/metadata.yaml file. See the app integration docs.
- Update the contents of the debian folder according to the structure described in https://wiki.openstack.org/wiki/StarlingX/DebianBuildStructure. The 'vault' app serves as a good reference here: https://opendev.org/starlingx/vault-armada-app/src/branch/master/stx-vault-helm/debian
- Make sure the src_path matches the source folder (the one on the same level as the debian folder). If you are using a dl_hook, then the source folder is defined there.
- The entries in the deb_folder/changelog should match the debname and debver defined in the meta_data.yaml
- Add a commit to git ... since spec file looks at git commit count
cd ${REPO_ROOT}/cgcs-root/stx/app-NAME
git add -A .
git commit -m "Baseline commit"
Identify the Helm charts for your application
For example, in the case of cert-manager, the external opensource application tarballs (containing helm charts) are here: https://github.com/cert-manager/cert-manager/releases .
You can use the 'source code (tar.gz)' tarball under the appropriate release, e.g. https://github.com/cert-manager/cert-manager/archive/refs/tags/v1.9.1.tar.gz, however sometimes we need to use commits in-between formal releases.
To find the SHA code for a particular commit, for a particular release, go to (say) https://github.com/cert-manager/cert-manager/commits/v1.9.1, and the SHA code for each commit is shown on right hand side ... COPY the SHA for (say) the last commit of the version you want.
Then construct the github URL to pull the source tarball based on commit SHA as follows: https://github.com/<org>/<repo>/archive/<SHA>.tar.gz e.g. for cert-manager, for last commit of v1.91, https://github.com/cert-manager/cert-manager/archive/4486c01f726f17d2790a8a563ae6bc6e98465505.tar.gz
Create the meta_data.yaml file
Create the meta_data.yaml containing the url and the sha to your tarball. Example:
-- debname: cert-manager-helm debver: 1.17-2 dl_path: name: cert-manager-helm-1.17.2.tar.gz url: https://github.com/cert-manager/cert-manager/archive/refs/tags/v1.17.2.tar.gz sha256sum: 77996ac7550534739e4ceeecfb650218900b26da9f9d31e12609946c8f20eeb6 src_files: - cert-manager-helm/files/Makefile revision: dist: $STX_DIST stx_patch: 0 GITREVCOUNT: BASE_SRCREV: 1008d7e9815526b72ad2587422764b826a3d0c5b SRC_DIR: ${MY_REPO}/stx/cert-manager-armada-app/helm-charts/upstream/cert-manager-helm
Build 'stx-app-NAME-helm' package
Try building the 'stx-app-NAME-helm' package.
Specifically this will extract the helm chart from your opensource application source tarball in downloads/ folder, build the fluxcd manifest, build the system-application tarball containing the fluxcd structure (containing the helm chart) and build a source containing the system-application tarball and install instructions for installing it in /usr/local/share/applications/helm/.
"build-pkgs -p stx-APPNAME-helm" inside the debian build container (For more details: https://wiki.openstack.org/wiki/StarlingX/DebianBuildEnvironment#Build_packages) The command might refuse to rebuild the app in some conditions. If you want to force a rebuild use "build-pkgs -c -p stx-APPNAME-helm" The output package and build logs are saved in /localdisk/loadbuild/<user>/<project>/std/<stx-app-NAME-helm> One possible failure reason is that the app failed some tests at build time. If that's the case, try adding the following command to deb_folder/rules: "override_dh_auto_test:"
Select CPU Cores for pods.
Most platform applications should be affined to the platform cores but changing this behavior should be supported via Helm overrides. Check here how to select CPU cores for pods.
Try out initial application packaging
Check the contents of the deb file with:
dpkg -x <deb_file> <output_folder>
You could take that to a StarlingX deployment and upload & apply. NOTE: If you didn't put anything in ./stx-APPNAME-helm/stx-app-NAME-helm/fluxcd-manifests/app-NAME/APPNAME-static-overrides.yaml, then applying your application on StarlingX will only work if the application's helm chart works with no overrides, i.e. default values.
Develop your Container Images
If you are using Container Images released by opensource project:
Reference the released opensource container image in ${REPO_ROOT}/cgcs-root/stx/app-NAME/stx-APPNAME-helm/stx-APPNAME-helm/fluxcd-manifests/APPNAME/APPNAME-static-overrides.yaml, with something like:
image:
repository: quay.io/jetstack/cert-manager-controller tag: v1.7.1
If you are building your own Container Images:
Referenced your soon-to-be-built container image in ${REPO_ROOT}/cgcs-root/stx/app-NAME/stx-APPNAME-helm/stx-APPNAME-helm/fluxcd-manifests/APPNAME/APPNAME-static-overrides.yaml with something like:
image:
repository: docker.io/starlingx/stx-APPNAME-container1 tag: testv1
All starlingx-built container images eventually get pushed to docker hub under starlingx organization.
Create a docker/ folder in your application repo for developing and building your Container Images
You need add a ${REPO_ROOT}/cgcs-root/stx/app-NAME/stx-APPNAME-helm/images/ folder in your application repo for building your container with a Dockerfile and any other make and source files for building your container.
For example, if you had to build a couple of container images for your application, your directory structure might look like this:
Use existing system application repos for examples of specific build specs, Dockerfile details, etc.
Push to the image to the local registry for testing
Once the image is built, pull the image to your testing VM, retag to your registry-local (as registry.local:9001/docker.io/starlingx/stx-APPNAME-container1:testv1) and then push the registry-local tag.
Develop your System Application Plugins
This step covers orientation on how to write application plugins to:
- Dynamically set helm chart(s) overrides based on the current StarlingX infrastructure configuration (e.g. replicas=2 for duplex systems)
- Provide "simple deployment" by automatically setting tested default helm overrides for application-specific parameters
- Provide custom behavior on 'system application-<operation> ...' StarlingX management of the FluxCD application packaging.
App plugins are defined in the "python-k8sapp-APPNAME" directory in the app repo.
Suggested changes to an existing app plugins
For this step, we can use the vault app plugins as a template.
- cd ${REPO_ROOT}/cgcs-root/stx/app-NAME/
- Copy app directory structure from https://opendev.org/starlingx/vault-armada-app/src/branch/master/python-k8sapp-vault
- Rename directories/files, replacing 'vault' with 'APPNAME'
- Update the mentions for the APPNAME in the files in the debian folder. Also update the app description in deb_folder/control in ./python-k8sapp-APPNAME/k8sapp_APPNAME/
- Updating README.rst, setup.cfg, .stestr.conf and tox.init ... generally replacing 'vault' with 'APPNAME'
- Note .stestr.conf is a hidden file
- If zuul fails to set up pbr, disable the sdist step in [tox]: “skipsdist = True”, and use “usedevelop=True” in each testenv that needs to install the local package in ./python-k8sapp-APPNAME/k8sapp_APPNAME/ k8sapp_APPNAME/
- Update * / *.py generally replacing 'vault' with 'APPNAME', 'Vault' with 'APPNAME' and 'VAULT' with 'APPNAME'
- Update ./python-k8sapp-APPNAME/k8sapp_APPNAME/ k8sapp_APPNAME/helm/harbor.py
- Add 'python-k8sapp-APPNAME' package into debian_pkg_dirs
- Add 'python-k8sapp-APPNAME' and 'python-k8sapp-APPNAME-wheels' in stx-APPNAME-helm/debian/deb_folder/control in the "Build-Depends" section
Build 'stx-app-NAME-helm' package
Try building the 'stx-app-NAME-helm' package with the plugins.
Specifically this will build your application plugins, extract the helm chart from your opensource application source tarball in downloads/ folder, build the fluxcd manifest, build the system-application tarball containing both the plugins and the fluxcd structure and build a debian package containing the system-application tarball and install instructions for installing it under /usr/local/share/applications/helm/
Run:
build-pkgs -p stx-app-NAME-helm
inside the debian build container (For more details: https://wiki.openstack.org/wiki/StarlingX/DebianBuildEnvironment#Build_packages)
The command might refuse to rebuild the app dependencies in some conditions. If you want to force a rebuild use "build-pkgs -c -p stx-app-NAME-helm, python3-k8sapp-APPNAME" The output package and build logs are saved in /localdisk/loadbuild/<user>/<project>/std/python3-k8sapp-APPNAME One possible failure reason is that the app failed some tests at build time. If that's the case, try adding the following command to deb_folder/rules: "override_dh_auto_test:"
Try out initial application packaging
You can check the contents of the deb file with:
dpkg -x <deb_file> <output_folder>
The tarball can be found at <output_folder>/usr/local/share/applications/helm/
Add the build of your Container Image(s) into main Container-Build
NOTE: Skip this step if only using opensource container images NOTE: wait for previous Step's commit to merge before doing this step.
Add your system app in "containers.xml" in the https://opendev.org/starlingx/manifest repo to ensure a container image build for your application as part of the build team's containers-build. An example can be found in commit https://review.opendev.org/c/starlingx/manifest/+/795353/.
Finalize Container Image Tag for Starlingx-built Container Images
NOTE: Skip this step if only using opensource container images
Obtain a timestamped formal build and test it
The build team can trigger a containers build and the timestamped docker-hub image can be referenced in your local app testing environment and used to test with it. Once all the tests passed, a static tag can be set in image-tags.yaml, see next step.
Add container image tag to image-tags.yaml
Update "build-tools/build-docker-images/tag-management/image-tags.yaml" with the container image static tag (see example https://review.opendev.org/c/starlingx/root/+/795560).
Update the image static tag in system app helm charts
The charts need to get updated with the new image tag in values.yaml (see example https://review.opendev.org/c/starlingx/audit-armada-app/+/797319)
Add the application to the ISO
For System Applications, the application tarball is put into a package (deb file for debian, rpm file for centos). When the package is installed, the tarball is copied to /usr/local/share/applications/helm/ and made available for the user to install.
To bundle the package with the ISO, just make sure the app names are listed in debian_iso_image.inc in your app repo.
Upload, Apply and Test System Application
After installing the ISO, the tarball can be found in /usr/local/share/applications/helm/. You can also build the package in you build environment, unpack the package and copy the tarball into a running STX system.
Authentication
Acquire the credentials before running the application commands:
source /etc/platform/openrc
Uploading and Applying Your Application
Uploading the new app:
system application-upload <app-tarball>
Installing the new app:
system application-apply <app-name>
Apps can also be reapplied by running the same command above.
Checking the app status:
system application-show <app-name>
App status should be "applied". Then run application specific testing.
Managing Your Application
Removing the new app:
system application-remove <app-name>
That will remove your application. It will be available to be applied again if needed.
Certain essential applications such as cert-manager requires the removal to be forced:
system application-remove --force <app-name>
Deleting the new app:
system application-delete <app-name>
That will fully delete your app. It will not be available to be immediately applied again. If you want to apply it afterwards you need to upload it again.
Updating to the new app:
system application-update <updated-app-tarball>
The update process will automatically upload and apply the new version to the system.
App updates can also be triggered during platform version upgrades. This process happen when the following command is issued during the upgrade process:
software deploy activate
Visit the StarlingX documentation for the full platform upgrade path.
Aborting an operation:
system application-abort <app-name>
That will cancel the current operation if it was not completed or failed.
In case you want to see the status of all apps:
system application-list
Managing Helm Overrides
Listing chart overrides:
system helm-override-list <app-name>
Showing overrides for a particular chart:
system helm-override-show <app_name> <chart_name> <namespace>
Modifying service configuration parameters using user-specified overrides:
system helm-override-update <app_name> <chart_name> <namespace> --reuse-values --reset-values --values <file_name> --set <commandline_overrides>
Enabling/Disabling the installation of a particular Helm chart within an application manifest:
system helm-chart-attribute-modify [--enabled <true/false>] <app_name> <chart_name> <namespace>
Deleting user overrides:
system helm-override-delete <app_name> <chart_name> <namespace>
Test STX build
As a reference, see the cert-manager app Storyboard for all the repositories that are touched during the process and confirm that all work items have been completed. See: https://storyboard.openstack.org/#!/story/2007360
In addition, test builds to confirm that addition of new repo/app doesn't break anything:
- Build app only
- Build-iso (and install from scratch)
- Test layered (flock) build
Updating charts from upstream apps in debian
In case there is a need to update the helm charts provided by an upstream app, the procedure is similar to what is done to patch debian packages, as explained in StarlingX/DebianBuildStructure.
In short, the helm-charts/upstream/chart-name-helm/debian/meta_data.yaml is responsible for defining the upstream tarball to be downloaded by the build system. Then, patches to the files in the tarball can be added to the folder helm-charts/upstream/chart-name-helm/debian/deb_folder/patches The order in which they are applied is defined in the file helm-charts/upstream/chart-name-helm/debian/deb_folder/patches/series
Please view this commit as an example: https://review.opendev.org/c/starlingx/app-rook-ceph/+/923178
Commit, Review and Merge your Application Repo Change
From within your git branch in your application repo:
cd ${MY_REPO_ROOT_DIR}/cgcs-root/stx/app-NAME
git status
git add <any files you've created/changed>
git commit -s
git review
