Jump to: navigation, search

StarlingX/Containers/Application/Archive/ConvertingArmadaAppsToFluxCD

THIS PAGE IS ARCHIVED. IT IS NOT ACTIVELY MAINTAINED

This page provides guidelines for converting Armada Applications to FluxCD

Armada Application archive format

The layout of an Armada application tar file is as follows: Application Tar File Contents

application.tgz
├──charts
├── checksum.md5
├── files
├── metadata.yaml
├── plugins
├── templates
├── ArmadaManifest.yaml

The tar file components are as follows:

  • charts - A mandatory directory which contains the helm charts for the application.
  • checksum.md5 - An MD5 checksum for the tar file contents.
  • files - An optional directory for resource files.
  • metadata.yaml - Application metadata.
  • plugins - An optional directory containing python app plugins.
  • templates - An optional directory for template files.
  • ArmadaManifest.yaml - The Armada manifest file for the application.


FluxCD Application archive format

The layout of an FluxCD application tar file is as follows:

application.tgz
├──charts
├── checksum.md5
├── files
├── metadata.yaml
├── plugins
├── templates

├── fluxcd-manifests
│   ├── base
│   │   ├── helmrepository.yaml
│   │   ├── kustomization.yaml
│   │   └── namespace.yaml
│   ├── kustomization.yaml
│   └── chart-0
│       ├── helmrelease.yaml
│       ├── kustomization.yaml
│       ├── chart-0-static-overrides.yaml
│       └── chart-0-system-overrides.yaml

│............................................................
│............................................................

│   └── chart-n
│       ├── helmrelease.yaml
│       ├── kustomization.yaml
│       ├── chart-n-static-overrides.yaml
│       └── chart-n-system-overrides.yam

The tar file components are almost the same as the armada components, only the armada manifest file (ArmadaManifest.yaml) is replaced with the fluxcd-manifests folder that will contain the necessary files to deploy the app using FluxCD.

In the fluxcd-manifests, there is a base directory that contains the basic resources required to be created for this particular application (i.e… helm repository). Each chart directory has a helmrelease YAML that defines the HelmRelease object for each helm chart and 2 overrides YAML files that contains the static and the system overrides for the chart. The static overrides and system overrides will be stored in Secret and referenced in its HelmRelease object.

Converting nginx-ingress-controller app to FluxCD

The armada manifest file of the nginx app:

---
schema: armada/Chart/v1
metadata:
  schema: metadata/Document/v1
  name: nginx-ingress
data:
  chart_name: ingress-nginx
  release: nginx-ingress
  namespace: kube-system
  wait:
    timeout: 1800
    labels:
      app.kubernetes.io/name: ingress-nginx
  install:
    no_hooks: false
  upgrade:
    no_hooks: false
    pre:
      delete:
        - type: job
          labels:
             app.kubernetes.io/name: ingress-nginx
  values:
    imagePullSecrets: [{"name": "default-registry-key"}]
    controller:
      kind: DaemonSet
      image:
        # cleans the default digest value since sysinv changes the digest when pushing the image to the local registry
        digest: ""
      daemonset:
        useHostPort: false
      nodeSelector:
        node-role.kubernetes.io/master: ""
      config:
        # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
        nginx-status-ipv4-whitelist: 0.0.0.0/0
        # See https://bugs.launchpad.net/starlingx/+bug/1823803
        # Note quotes are necessary.
        worker-processes: '1'
      scope:
        enabled: false
      service:
        type: ""
      hostNetwork: true
      livenessProbe:
        initialDelaySeconds: 30
      readinessProbe:
        initialDelaySeconds: 30
      admissionWebhooks:
        # default port 8443 conflicts with lighttpd using https
        port: 5443
        patch:
          tolerations:
            - key: "node-role.kubernetes.io/master"
              operator: "Exists"
              effect: "NoSchedule"
    defaultBackend:
      image:
        repository: k8s.gcr.io/defaultbackend
        tag: "1.4"
      nodeSelector:
        node-role.kubernetes.io/master: ""
      service:
        type: ""
      livenessProbe:
        initialDelaySeconds: 30
      readinessProbe:
       initialDelaySeconds: 30
  source:
    type: tar
    location: http://172.17.0.1/helm_charts/stx-platform/ingress-nginx-3.10.1.tgz
    subpath: ingress-nginx
    reference: master
  dependencies: []
---
schema: armada/ChartGroup/v1
metadata:
  schema: metadata/Document/v1
  name: nginx-ingress
data:
  description: "Deploy Nginx Ingress Controller"
  sequenced: false
  chart_group:
    - nginx-ingress
---
schema: armada/Manifest/v1
metadata:
  schema: metadata/Document/v1
  name: nginx-ingress-controller-manifest
data:
  release_prefix: ic
  chart_groups:
    - nginx-ingress

The FluxCD structure of the nginx app will be the following:

fluxcd-manifests/
├── base
│   ├── helmrepository.yaml
│   ├── kustomization.yaml
│   └── namespace.yaml
├── kustomization.yaml
└── nginx-ingress
    ├── helmrelease.yaml
    ├── kustomization.yaml
    ├── nginx-ingress-static-overrides.yaml
    └── nginx-ingress-system-overrides.yaml

Each layer has a kustomization.yaml file that contains the resources to apply. The top level one looks like:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kube-system
resources: # the chart directories that need to be applied
  - base
  - nginx-ingress

The base chart directory contains the following:

base/helmrepository.yaml

# HelmRepository YAML for Starlingx helm repository.
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: stx-platform
spec:
  url: http://192.168.206.1:8080/helm_charts/stx-platform
  interval: 60m  # interval to check the repository for updates

base/kustomization.yaml

resources:
  - helmrepository.yaml

base/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: kube-system  # the same as data.namespace in the armada manifest

The nginx-ingress chart directory contains the following:

nginx-ingress/helmrelease.yaml

apiVersion: "helm.toolkit.fluxcd.io/v2beta1"
kind: HelmRelease
metadata:
  name: nginx-ingress  # the same as metadata.name in the armada manifest
  labels:
    chart_group: nginx-ingress  # the same as chart_group in the armada manifest
spec:
  releaseName: ic-nginx-ingress  # the release_prefix + data.release from the armada manifest
  chart: # A HelmChart CR of a specific chart is auto created in the cluster with this definition
    spec:
      chart: ingress-nginx  # the chart name and version usually is 
      version: 3.10.1
      sourceRef:
        kind: HelmRepository
        name: stx-platform
  interval: 5m  # Interval to reconcile Helm release
  timeout: 30m  # the same as data.wait.timeout in armada manifest file but in minutes
  test:
    enable: false
  install:
    disableHooks: false  # the same as install.no_hooks in the armada manifest
  upgrade:
    disableHooks: false  # the same as upgrade.no_hooks in the armada manifest
  valuesFrom:  # We store the static overrides and the system overrides in k8s Secrets, the 2 yaml files are present in the same directory with this file
    - kind: Secret
      name: nginx-ingress-static-overrides
      valuesKey: nginx-ingress-static-overrides.yaml
    - kind: Secret
      name: nginx-ingress-system-overrides
      valuesKey: nginx-ingress-system-overrides.yaml  

nginx-ingress/kustomization.yaml

namespace: kube-system   # the same as data.namespace in the armada manifest
resources:
  - helmrelease.yaml
secretGenerator:  # this will create the Secrets (that hold the overrides) as part of application install
  - name: nginx-ingress-static-overrides
    files:
      - nginx-ingress-static-overrides.yaml
  - name: nginx-ingress-system-overrides
    files:
      - nginx-ingress-system-overrides.yaml
generatorOptions:
  disableNameSuffixHash: true

nginx-ingress/nginx-ingress-static-overrides.yaml

# the static overrides, basically all the values from the armada manifest file
imagePullSecrets: [{"name": "default-registry-key"}]
controller:
  kind: DaemonSet
  image:
    # cleans the default digest value since sysinv changes the digest when pushing the image to the local registry
    digest: ""
  daemonset:
    useHostPort: false
  nodeSelector:
    node-role.kubernetes.io/master: ""
  config:
    # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
    nginx-status-ipv4-whitelist: 0.0.0.0/0
    # See https://bugs.launchpad.net/starlingx/+bug/1823803
    # Note quotes are necessary.
    worker-processes: '1'
  scope:
    enabled: false
  service:
    type: ""
   hostNetwork: true
  livenessProbe:
    initialDelaySeconds: 30
  readinessProbe:
    initialDelaySeconds: 30
  admissionWebhooks:
    # default port 8443 conflicts with lighttpd using https
    port: 5443
    patch:
      tolerations:
        - key: "node-role.kubernetes.io/master"
          operator: "Exists"
          effect: "NoSchedule"
defaultBackend:
  image:
    repository: k8s.gcr.io/defaultbackend
    tag: "1.4"
  nodeSelector:
    node-role.kubernetes.io/master: ""
  service:
    type: ""
  livenessProbe:
    initialDelaySeconds: 30
  readinessProbe:
    initialDelaySeconds: 30

The nginx-ingress/nginx-ingress-system-overrides.yaml is empty and will contain any system overrides or user overrides ( generated by helm plugins or system helm-override-update)


Installing a FluxCD application

Prerequisites

Before we attempt to install a FluxCD application we need the following:

wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv4.2.0/kustomize_v4.2.0_linux_amd64.tar.gz
sudo tar -C /usr/local/bin/ -xf kustomize_v4.2.0_linux_amd64.tar.gz

Installing nginx fluxcd app

For this we need https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/827880 commit into the ISO. Currently, the nginx armada app is installed by default, we need to delete it in order to install the fluxcd version by running

system application-remove nginx-ingress-controller --force

After the nginx-ingress-controller armada app is in uploaded state, we can install the fluxcd version by running the following commands:

mkdir -p nginx_fluxcd
tar -C nginx_fluxcd/ -xf /usr/local/share/applications/helm/nginx-ingress-controller-fluxcd-1.1-22.tgz

# the chart needs to be uploaded to helm repo first
sudo helm-upload stx-platform nginx_fluxcd/charts/nginx-ingress-controller-fluxcd-1.1-22.tgz

# kustomize command to install the fluxcd app
kustomize build nginx_fluxcd/fluxcd-manifests/ | kubectl apply -f -

To see the status of the installation, the following command can be used:

kubectl get helmreleases.helm.toolkit.fluxcd.io -n kube-system
NAME            READY   STATUS                             AGE
nginx-ingress   True    Release reconciliation succeeded   3m31

And the Ready column = True means that the app was successfully installed.

For debugging purposes if the install failed or if it's hanging, we can also see if the chart was pulled correctly by flux:

kubectl get helmcharts.source.toolkit.fluxcd.io -n kube-system
NAME                        CHART           VERSION   SOURCE KIND      SOURCE NAME    READY   STATUS                                                AGE
kube-system-nginx-ingress   ingress-nginx   3.10.1    HelmRepository   stx-platform   True    Pulled 'ingress-nginx' chart with version '3.10.1'.   8m55


To delete the nginx fluxcd app we can do:

kustomize build nginx_fluxcd/fluxcd-manifests/ | kubectl delete -f -

Notes

  • The nginx example above has certain implied conditions.
  • application-upload: The application framework for armada populates /var/www/pages/helm_charts/stx-platform/ (or etc) with the helm tarballs.  Refer to HelmRelease spec, chart name, version, and sourceRef HelmRepository name ('stx-platform').
  • application-apply and application-remove: The application framework for nginx armada application created certain resources that may persist after application-remove, and are necessary for the fluxcd version of the application to work.
  • Helm v2 versus Helm v3: The helm chart may have certain incompatibilities with helm v3, which is used for the fluxcd version of the application.  These can be resolved with fixes or an uprev of the chart to a new version.