StarlingX/Containers/ConvertingArmadaAppsToFluxCD

This page provides guidelines for converting Armada Applications to FluxCD

Armada Application archive format
The layout of an Armada application tar file is as follows: Application Tar File Contents application.tgz ├──charts ├── checksum.md5 ├── files ├── metadata.yaml ├── plugins ├── templates ├── ArmadaManifest.yaml

The tar file components are as follows:
 * charts - A mandatory directory which contains the helm charts for the application.
 * checksum.md5 - An MD5 checksum for the tar file contents.
 * files - An optional directory for resource files.
 * metadata.yaml - Application metadata.
 * plugins - An optional directory containing python app plugins.
 * templates - An optional directory for template files.
 * ArmadaManifest.yaml - The Armada manifest file for the application.

FluxCD Application archive format
The layout of an FluxCD application tar file is as follows: application.tgz ├──charts ├── checksum.md5 ├── files ├── metadata.yaml ├── plugins ├── templates ├── fluxcd-manifests │   ├── base │   │   ├── helmrepository.yaml │   │   ├── kustomization.yaml │   │   └── namespace.yaml │   ├── kustomization.yaml │   └── chart-0 │       ├── helmrelease.yaml │       ├── kustomization.yaml │       ├── chart-0-static-overrides.yaml │       └── chart-0-system-overrides.yaml │............................................................ │............................................................ │   └── chart-n │       ├── helmrelease.yaml │       ├── kustomization.yaml │       ├── chart-n-static-overrides.yaml │       └── chart-n-system-overrides.yam

The tar file components are almost the same as the armada components, only the armada manifest file (ArmadaManifest.yaml) is replaced with the fluxcd-manifests folder that will contain the necessary files to deploy the app using FluxCD.

In the fluxcd-manifests, there is a base directory that contains the basic resources required to be created for this particular application (i.e… helm repository). Each chart directory has a helmrelease YAML that defines the HelmRelease object for each helm chart and 2 overrides YAML files that contains the static and the system overrides for the chart. The static overrides and system overrides will be stored in Secret and referenced in its HelmRelease object.

Converting nginx-ingress-controller app to FluxCD
The armada manifest file of the nginx app: --- schema: armada/Chart/v1 metadata: schema: metadata/Document/v1 name: nginx-ingress data: chart_name: ingress-nginx release: nginx-ingress namespace: kube-system wait: timeout: 1800 labels: app.kubernetes.io/name: ingress-nginx install: no_hooks: false upgrade: no_hooks: false pre: delete: - type: job labels: app.kubernetes.io/name: ingress-nginx values: imagePullSecrets: [{"name": "default-registry-key"}] controller: kind: DaemonSet image: # cleans the default digest value since sysinv changes the digest when pushing the image to the local registry digest: "" daemonset: useHostPort: false nodeSelector: node-role.kubernetes.io/master: "" config: # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ nginx-status-ipv4-whitelist: 0.0.0.0/0 # See https://bugs.launchpad.net/starlingx/+bug/1823803 # Note quotes are necessary. worker-processes: '1' scope: enabled: false service: type: "" hostNetwork: true livenessProbe: initialDelaySeconds: 30 readinessProbe: initialDelaySeconds: 30 admissionWebhooks: # default port 8443 conflicts with lighttpd using https port: 5443 patch: tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" defaultBackend: image: repository: k8s.gcr.io/defaultbackend tag: "1.4" nodeSelector: node-role.kubernetes.io/master: "" service: type: "" livenessProbe: initialDelaySeconds: 30 readinessProbe: initialDelaySeconds: 30 source: type: tar location: http://172.17.0.1/helm_charts/stx-platform/ingress-nginx-3.10.1.tgz subpath: ingress-nginx reference: master dependencies: [] --- schema: armada/ChartGroup/v1 metadata: schema: metadata/Document/v1 name: nginx-ingress data: description: "Deploy Nginx Ingress Controller" sequenced: false chart_group: - nginx-ingress --- schema: armada/Manifest/v1 metadata: schema: metadata/Document/v1 name: nginx-ingress-controller-manifest data: release_prefix: ic  chart_groups: - nginx-ingress

The FluxCD structure of the nginx app will be the following: fluxcd-manifests/ ├── base │  ├── helmrepository.yaml │  ├── kustomization.yaml │  └── namespace.yaml ├── kustomization.yaml └── nginx-ingress ├── helmrelease.yaml ├── kustomization.yaml ├── nginx-ingress-static-overrides.yaml └── nginx-ingress-system-overrides.yaml

Each layer has a kustomization.yaml file that contains the resources to apply. The top level one looks like: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: kube-system resources: # the chart directories that need to be applied - base - nginx-ingress

The base chart directory contains the following:

base/helmrepository.yaml apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: HelmRepository metadata: name: stx-platform spec: url: http://192.168.206.1:8080/helm_charts/stx-platform interval: 60m # interval to check the repository for updates base/kustomization.yaml resources: - helmrepository.yaml
 * 1) HelmRepository YAML for Starlingx helm repository.

base/namespace.yaml apiVersion: v1 kind: Namespace metadata: name: kube-system # the same as data.namespace in the armada manifest

The nginx-ingress chart directory contains the following:

nginx-ingress/helmrelease.yaml apiVersion: "helm.toolkit.fluxcd.io/v2beta1" kind: HelmRelease metadata: name: nginx-ingress # the same as metadata.name in the armada manifest labels: chart_group: nginx-ingress # the same as chart_group in the armada manifest spec: releaseName: ic-nginx-ingress # the release_prefix + data.release from the armada manifest chart: # A HelmChart CR of a specific chart is auto created in the cluster with this definition spec: chart: ingress-nginx # the chart name and version usually is        version: 3.10.1 sourceRef: kind: HelmRepository name: stx-platform interval: 5m # Interval to reconcile Helm release timeout: 30m # the same as data.wait.timeout in armada manifest file but in minutes test: enable: false install: disableHooks: false # the same as install.no_hooks in the armada manifest upgrade: disableHooks: false # the same as upgrade.no_hooks in the armada manifest valuesFrom: # We store the static overrides and the system overrides in k8s Secrets, the 2 yaml files are present in the same directory with this file - kind: Secret name: nginx-ingress-static-overrides valuesKey: nginx-ingress-static-overrides.yaml - kind: Secret name: nginx-ingress-system-overrides valuesKey: nginx-ingress-system-overrides.yaml nginx-ingress/kustomization.yaml namespace: kube-system   # the same as data.namespace in the armada manifest resources: - helmrelease.yaml secretGenerator: # this will create the Secrets (that hold the overrides) as part of application install - name: nginx-ingress-static-overrides files: - nginx-ingress-static-overrides.yaml - name: nginx-ingress-system-overrides files: - nginx-ingress-system-overrides.yaml generatorOptions: disableNameSuffixHash: true

nginx-ingress/nginx-ingress-static-overrides.yaml imagePullSecrets: [{"name": "default-registry-key"}] controller: kind: DaemonSet image: # cleans the default digest value since sysinv changes the digest when pushing the image to the local registry digest: "" daemonset: useHostPort: false nodeSelector: node-role.kubernetes.io/master: "" config: # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ nginx-status-ipv4-whitelist: 0.0.0.0/0 # See https://bugs.launchpad.net/starlingx/+bug/1823803 # Note quotes are necessary. worker-processes: '1' scope: enabled: false service: type: "" hostNetwork: true livenessProbe: initialDelaySeconds: 30 readinessProbe: initialDelaySeconds: 30 admissionWebhooks: # default port 8443 conflicts with lighttpd using https port: 5443 patch: tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" defaultBackend: image: repository: k8s.gcr.io/defaultbackend tag: "1.4" nodeSelector: node-role.kubernetes.io/master: "" service: type: "" livenessProbe: initialDelaySeconds: 30 readinessProbe: initialDelaySeconds: 30
 * 1) the static overrides, basically all the values from the armada manifest file

The nginx-ingress/nginx-ingress-system-overrides.yaml is empty and will contain any system overrides or user overrides ( generated by helm plugins or system helm-override-update)

Prerequisites
Before we attempt to install a FluxCD application we need the following: wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv4.2.0/kustomize_v4.2.0_linux_amd64.tar.gz sudo tar -C /usr/local/bin/ -xf kustomize_v4.2.0_linux_amd64.tar.gz
 * This https://review.opendev.org/c/starlingx/ansible-playbooks/+/828592 commit to be added to the ISO
 * Before running ansible add enable_fluxcd_controllers: true into localhost.yml
 * Install the kustomize binary

Installing nginx fluxcd app
For this we need https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/827880 commit into the ISO. Currently, the nginx armada app is installed by default, we need to delete it in order to install the fluxcd version by running system application-remove nginx-ingress-controller --force

After the nginx-ingress-controller armada app is in uploaded state, we can install the fluxcd version by running the following commands:

mkdir -p nginx_fluxcd tar -C nginx_fluxcd/ -xf /usr/local/share/applications/helm/nginx-ingress-controller-fluxcd-1.1-22.tgz sudo helm-upload stx-platform nginx_fluxcd/charts/nginx-ingress-controller-fluxcd-1.1-22.tgz kustomize build nginx_fluxcd/fluxcd-manifests/ | kubectl apply -f -
 * 1) the chart needs to be uploaded to helm repo first
 * 1) kustomize command to install the fluxcd app

To see the status of the installation, the following command can be used:

kubectl get helmreleases.helm.toolkit.fluxcd.io -n kube-system NAME           READY   STATUS                             AGE nginx-ingress  True    Release reconciliation succeeded   3m31

And the Ready column = True means that the app was successfully installed.

For debugging purposes if the install failed or if it's hanging, we can also see if the chart was pulled correctly by flux:

kubectl get helmcharts.source.toolkit.fluxcd.io -n kube-system NAME                       CHART           VERSION   SOURCE KIND      SOURCE NAME    READY   STATUS                                                AGE kube-system-nginx-ingress  ingress-nginx   3.10.1    HelmRepository   stx-platform   True    Pulled 'ingress-nginx' chart with version '3.10.1'. 8m55

To delete the nginx fluxcd app we can do: kustomize build nginx_fluxcd/fluxcd-manifests/ | kubectl delete -f -