Difference between revisions of "StarlingX/Containers/Application/Archive/ConvertingArmadaAppsToFluxCD"
Ghada.khalil (talk | contribs) (Created page with "This page provides guidelines for converting Armada Applications to FluxCD == Armada Application archive format == The layout of an Armada application tar file is as follows:...") |
m (Robert.church moved page StarlingX/Containers/ConvertingArmadaAppsToFluxCD to StarlingX/Containers/Application/Archive/ConvertingArmadaAppsToFluxCD: move to Applications Archive as this is no longer relevant information) |
(No difference)
|
Revision as of 11:22, 10 January 2024
This page provides guidelines for converting Armada Applications to FluxCD
Contents
Armada Application archive format
The layout of an Armada application tar file is as follows: Application Tar File Contents
application.tgz ├──charts ├── checksum.md5 ├── files ├── metadata.yaml ├── plugins ├── templates ├── ArmadaManifest.yaml
The tar file components are as follows:
- charts - A mandatory directory which contains the helm charts for the application.
- checksum.md5 - An MD5 checksum for the tar file contents.
- files - An optional directory for resource files.
- metadata.yaml - Application metadata.
- plugins - An optional directory containing python app plugins.
- templates - An optional directory for template files.
- ArmadaManifest.yaml - The Armada manifest file for the application.
FluxCD Application archive format
The layout of an FluxCD application tar file is as follows:
application.tgz ├──charts ├── checksum.md5 ├── files ├── metadata.yaml ├── plugins ├── templates ├── fluxcd-manifests │ ├── base │ │ ├── helmrepository.yaml │ │ ├── kustomization.yaml │ │ └── namespace.yaml │ ├── kustomization.yaml │ └── chart-0 │ ├── helmrelease.yaml │ ├── kustomization.yaml │ ├── chart-0-static-overrides.yaml │ └── chart-0-system-overrides.yaml │............................................................ │............................................................ │ └── chart-n │ ├── helmrelease.yaml │ ├── kustomization.yaml │ ├── chart-n-static-overrides.yaml │ └── chart-n-system-overrides.yam
The tar file components are almost the same as the armada components, only the armada manifest file (ArmadaManifest.yaml) is replaced with the fluxcd-manifests folder that will contain the necessary files to deploy the app using FluxCD.
In the fluxcd-manifests, there is a base directory that contains the basic resources required to be created for this particular application (i.e… helm repository). Each chart directory has a helmrelease YAML that defines the HelmRelease object for each helm chart and 2 overrides YAML files that contains the static and the system overrides for the chart. The static overrides and system overrides will be stored in Secret and referenced in its HelmRelease object.
Converting nginx-ingress-controller app to FluxCD
The armada manifest file of the nginx app:
--- schema: armada/Chart/v1 metadata: schema: metadata/Document/v1 name: nginx-ingress data: chart_name: ingress-nginx release: nginx-ingress namespace: kube-system wait: timeout: 1800 labels: app.kubernetes.io/name: ingress-nginx install: no_hooks: false upgrade: no_hooks: false pre: delete: - type: job labels: app.kubernetes.io/name: ingress-nginx values: imagePullSecrets: [{"name": "default-registry-key"}] controller: kind: DaemonSet image: # cleans the default digest value since sysinv changes the digest when pushing the image to the local registry digest: "" daemonset: useHostPort: false nodeSelector: node-role.kubernetes.io/master: "" config: # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ nginx-status-ipv4-whitelist: 0.0.0.0/0 # See https://bugs.launchpad.net/starlingx/+bug/1823803 # Note quotes are necessary. worker-processes: '1' scope: enabled: false service: type: "" hostNetwork: true livenessProbe: initialDelaySeconds: 30 readinessProbe: initialDelaySeconds: 30 admissionWebhooks: # default port 8443 conflicts with lighttpd using https port: 5443 patch: tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" defaultBackend: image: repository: k8s.gcr.io/defaultbackend tag: "1.4" nodeSelector: node-role.kubernetes.io/master: "" service: type: "" livenessProbe: initialDelaySeconds: 30 readinessProbe: initialDelaySeconds: 30 source: type: tar location: http://172.17.0.1/helm_charts/stx-platform/ingress-nginx-3.10.1.tgz subpath: ingress-nginx reference: master dependencies: [] --- schema: armada/ChartGroup/v1 metadata: schema: metadata/Document/v1 name: nginx-ingress data: description: "Deploy Nginx Ingress Controller" sequenced: false chart_group: - nginx-ingress --- schema: armada/Manifest/v1 metadata: schema: metadata/Document/v1 name: nginx-ingress-controller-manifest data: release_prefix: ic chart_groups: - nginx-ingress
The FluxCD structure of the nginx app will be the following:
fluxcd-manifests/ ├── base │ ├── helmrepository.yaml │ ├── kustomization.yaml │ └── namespace.yaml ├── kustomization.yaml └── nginx-ingress ├── helmrelease.yaml ├── kustomization.yaml ├── nginx-ingress-static-overrides.yaml └── nginx-ingress-system-overrides.yaml
Each layer has a kustomization.yaml file that contains the resources to apply. The top level one looks like:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: kube-system resources: # the chart directories that need to be applied - base - nginx-ingress
The base chart directory contains the following:
base/helmrepository.yaml
# HelmRepository YAML for Starlingx helm repository. apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: HelmRepository metadata: name: stx-platform spec: url: http://192.168.206.1:8080/helm_charts/stx-platform interval: 60m # interval to check the repository for updates
base/kustomization.yaml
resources: - helmrepository.yaml
base/namespace.yaml
apiVersion: v1 kind: Namespace metadata: name: kube-system # the same as data.namespace in the armada manifest
The nginx-ingress chart directory contains the following:
nginx-ingress/helmrelease.yaml
apiVersion: "helm.toolkit.fluxcd.io/v2beta1" kind: HelmRelease metadata: name: nginx-ingress # the same as metadata.name in the armada manifest labels: chart_group: nginx-ingress # the same as chart_group in the armada manifest spec: releaseName: ic-nginx-ingress # the release_prefix + data.release from the armada manifest chart: # A HelmChart CR of a specific chart is auto created in the cluster with this definition spec: chart: ingress-nginx # the chart name and version usually is version: 3.10.1 sourceRef: kind: HelmRepository name: stx-platform interval: 5m # Interval to reconcile Helm release timeout: 30m # the same as data.wait.timeout in armada manifest file but in minutes test: enable: false install: disableHooks: false # the same as install.no_hooks in the armada manifest upgrade: disableHooks: false # the same as upgrade.no_hooks in the armada manifest valuesFrom: # We store the static overrides and the system overrides in k8s Secrets, the 2 yaml files are present in the same directory with this file - kind: Secret name: nginx-ingress-static-overrides valuesKey: nginx-ingress-static-overrides.yaml - kind: Secret name: nginx-ingress-system-overrides valuesKey: nginx-ingress-system-overrides.yaml
nginx-ingress/kustomization.yaml
namespace: kube-system # the same as data.namespace in the armada manifest resources: - helmrelease.yaml secretGenerator: # this will create the Secrets (that hold the overrides) as part of application install - name: nginx-ingress-static-overrides files: - nginx-ingress-static-overrides.yaml - name: nginx-ingress-system-overrides files: - nginx-ingress-system-overrides.yaml generatorOptions: disableNameSuffixHash: true
nginx-ingress/nginx-ingress-static-overrides.yaml
# the static overrides, basically all the values from the armada manifest file imagePullSecrets: [{"name": "default-registry-key"}] controller: kind: DaemonSet image: # cleans the default digest value since sysinv changes the digest when pushing the image to the local registry digest: "" daemonset: useHostPort: false nodeSelector: node-role.kubernetes.io/master: "" config: # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/ nginx-status-ipv4-whitelist: 0.0.0.0/0 # See https://bugs.launchpad.net/starlingx/+bug/1823803 # Note quotes are necessary. worker-processes: '1' scope: enabled: false service: type: "" hostNetwork: true livenessProbe: initialDelaySeconds: 30 readinessProbe: initialDelaySeconds: 30 admissionWebhooks: # default port 8443 conflicts with lighttpd using https port: 5443 patch: tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" defaultBackend: image: repository: k8s.gcr.io/defaultbackend tag: "1.4" nodeSelector: node-role.kubernetes.io/master: "" service: type: "" livenessProbe: initialDelaySeconds: 30 readinessProbe: initialDelaySeconds: 30
The nginx-ingress/nginx-ingress-system-overrides.yaml is empty and will contain any system overrides or user overrides ( generated by helm plugins or system helm-override-update)
Installing a FluxCD application
Prerequisites
Before we attempt to install a FluxCD application we need the following:
- This https://review.opendev.org/c/starlingx/ansible-playbooks/+/828592 commit to be added to the ISO
- Before running ansible add enable_fluxcd_controllers: true into localhost.yml
- Install the kustomize binary
wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv4.2.0/kustomize_v4.2.0_linux_amd64.tar.gz sudo tar -C /usr/local/bin/ -xf kustomize_v4.2.0_linux_amd64.tar.gz
Installing nginx fluxcd app
For this we need https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/827880 commit into the ISO. Currently, the nginx armada app is installed by default, we need to delete it in order to install the fluxcd version by running
system application-remove nginx-ingress-controller --force
After the nginx-ingress-controller armada app is in uploaded state, we can install the fluxcd version by running the following commands:
mkdir -p nginx_fluxcd tar -C nginx_fluxcd/ -xf /usr/local/share/applications/helm/nginx-ingress-controller-fluxcd-1.1-22.tgz # the chart needs to be uploaded to helm repo first sudo helm-upload stx-platform nginx_fluxcd/charts/nginx-ingress-controller-fluxcd-1.1-22.tgz # kustomize command to install the fluxcd app kustomize build nginx_fluxcd/fluxcd-manifests/ | kubectl apply -f -
To see the status of the installation, the following command can be used:
kubectl get helmreleases.helm.toolkit.fluxcd.io -n kube-system NAME READY STATUS AGE nginx-ingress True Release reconciliation succeeded 3m31
And the Ready column = True means that the app was successfully installed.
For debugging purposes if the install failed or if it's hanging, we can also see if the chart was pulled correctly by flux:
kubectl get helmcharts.source.toolkit.fluxcd.io -n kube-system NAME CHART VERSION SOURCE KIND SOURCE NAME READY STATUS AGE kube-system-nginx-ingress ingress-nginx 3.10.1 HelmRepository stx-platform True Pulled 'ingress-nginx' chart with version '3.10.1'. 8m55
To delete the nginx fluxcd app we can do:
kustomize build nginx_fluxcd/fluxcd-manifests/ | kubectl delete -f -
Notes
- The nginx example above has certain implied conditions.
- application-upload: The application framework for armada populates /var/www/pages/helm_charts/stx-platform/ (or etc) with the helm tarballs. Refer to HelmRelease spec, chart name, version, and sourceRef HelmRepository name ('stx-platform').
- application-apply and application-remove: The application framework for nginx armada application created certain resources that may persist after application-remove, and are necessary for the fluxcd version of the application to work.
- Helm v2 versus Helm v3: The helm chart may have certain incompatibilities with helm v3, which is used for the fluxcd version of the application. These can be resolved with fixes or an uprev of the chart to a new version.