StarlingX/Containers/ArmadaAppCodeStructure

Overview
It is recommended to start off copying another app code structure as starting point and then make necessary changes to various configuration build & configuration files.

Following screenshot shows the directory structure for cert-manager. Note that most of the files are under 'cert-manager-helm' and 'stx-cert-manager-helm'



The 'cert-manager-helm' structure takes the tarball in stx/downloads (which originated from 3rd party public upstream repository), unpacks, applies patches (more on this below) and compiles to build the cert-manager helm charts using an included Makefile. The files of particular interest here are in the 'centos' folder. The .spec file directs the details of the build.

The 'stx-cert-manager-helm' builds (stx branded) Armada Helm Charts (output of this exercise). Again, files that direct the build steps are in 'centos' folder and the .spec file contains the details. The manifest folder contains the main manifest file for the application that contains the Armada ChartGroups and one or more Helm charts & any particular overrides. Details on how to write this manifest can be found here: https://readthedocs.org/projects/airship-armada/downloads/pdf/latest/

Patches

 * Apply patches if needed.
 * Copy generated patch file over to 'files' folder and make appropriate changes in spec file to apply the patch.

Image Pull Secrets
Our applications use images from registry.local:9001 for deployment. This is an authenticated registry and require image pull secrets to deploy pods successfully. Upstream helm charts will almost certainly not have image pull secrets set up. This means pods will fail to deploy with an "ImagePullBackOff" error. We need to add an "imagePullSecrets" section to each "containers" entry from the helm charts. This needs to be done through a patch if using upstream helm charts, or directly if we are making the helm charts ourselves. Refer to the following commit: https://review.opendev.org/c/starlingx/vault-armada-app/+/772616

To test, deploy your application and ensure the pods come up as opposed to failing due to the "ImagePullBackOff" error.

Pod Security Policies
PodSecurityPolicy is an optional plugin to control pod deployments. If this plugin is enabled, by default no pods can be deployed. We need to add policies to allow pods to be deployed in each namespace. Applications normally create their own namespace when being deployed. If your application deploys in an existing namespace, you will not need to do this. Refer to the following commit to add pod security policies for your application: https://review.opendev.org/c/starlingx/openstack-armada-app/+/736002

This commit deploys the pod security policy helm chart under the common "helm-charts" repo. You will need to add this helm chart to your application's armada manifest and provide the correct "rolebindingNamespace" and "serviceAccount". The spec and build_srpm.data change is to build the common helm-charts chart for pod security policies.

There are currently some issues enabling the pod security policies plugin. I will update this section when that gets resolved. To test, simply follow the instructions above, and ensure that when you apply your application, the rolebinding is deployed along with it:

kubectl get rolebinding -n  | grep psp
 * your_namespace-privileged-psp-users

Defining CRDs via Helm Charts
If your application requires CRDs, there may be multiple ways to declare and define the CRDs. For e.g., the Deployment Manager has a section where CRDs are defined and can be a viable option.

If the CRDs have to be defined as part of the application deployment at runtime (dynamically performed during the 'system application-upload' and 'system application-apply'), a separate sequenced Helm Chart may be an option. This is the option chosen for cert-manager. There are two charts - first chart defines the CRDs and the second chart runs the various container images necessary for the app.

To get the Helm Charts for CRDs, the following steps were performed (Note that you need a system with helm installed).
 * Get the CRD yaml files from 3rd party upstream codebase
 * Google the commands for Helm chart creation (such as this page). In short, the steps required include:
 * helm create
 * Remove unneeded files
 * In 'templates' folder, remove the existing *.yaml files
 * There is a 'tests' folder created, (which was noted to cause issues during deploying non-existent "certmgr-crds-test-connection" pod. Fix was to simply delete the 'tests' folder).
 * Copy the app's CRD yaml files to the 'templates' folder
 * helm install --dry-run --debug
 * helm lint
 * (Optional) helm package
 * Manually run 'helm install' command on a machine to test if the chart was created ok and the CRDs get defined as expected (use 'kubectl get crd' to check if it worked after 'helm install')
 * Copy the helm charts created above in the stx-cert-manager-helm folder under "helm-charts"

Note: During the deployment (during 'system application-apply'), Armada checks if resources came up successfully and by default checks for a 'pod' or 'job' with the same name as the chartname (or a label). In case of CRDs, since we are not going to start any pods, this will result in a failure of Armada checkpoint. To fix this issue, set 'resources: []' under 'wait' of the CRD chart. This directs Armada to nullify any resource checks after running this Helm chart. See the screenshot with highlight below.



Lifecycle hooks
Apps can package their own code to be run around sysinv events. (Hint: search the codebase for APP_LIFECYCLE_, LifecycleHook, LifecycleHookInfo. Also open the overview)

Code is located under: ${MY_REPO}/stx/openstack-armada-app/python-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/lifecycle

Creating the lifecycle operator: https://review.opendev.org/c/starlingx/openstack-armada-app/+/763059

Files will be extracted on the running controller under: /opt/platform/helm/20.12/stx-openstack/1.0-78-centos-stable-latest/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py

Control behavior through metadata
The metadata file is located at stx- -helm/stx- -helm/files/metadata.yaml in the source code.

Validation of metadata is done in ${MY_REPO}/stx/config/sysinv/sysinv/sysinv/sysinv/common/utils.py::find_metadata_file.

The bridge between metadata and code is APP_EVALUATE_REAPPLY_TRIGGER_TO_METADATA_MAP.

metadata.yaml behavior: platform_managed_app: yes evaluate_reapply: after: - platform-integ-apps
 * Apps can specify which apps should be evaluated before self

metadata.yaml behavior: platform_managed_app: yes evaluate_reapply: triggers: - type: unlock - type: force-unlock - type: runtime-apply-puppet - type: host-availability-updated filters: - availability: services-enabled - type: host-delete filters: - openstack_worker: True
 * Apps can subscribe to events that trigger a reapply evaluation

metadata.yaml behavior: platform_managed_app: true
 * Apps can specify if they want to be audited. Retries for failed operations (apply, update, recover)

metadata.yaml behavior: platform_managed_app: true desired_state: uploaded
 * Apps can specify which states they want to achieve (uploaded, applied)

metadata.yaml behavior: forbidden_manual_operations: - delete - remove
 * Apps can specify which actions (remove, delete) are forbidden from being performed; for application-delete: and for application-remove:

An example of how to apply a reapply trigger filter (both using code, and equivalent behavior achieved by using metadata): https://review.opendev.org/c/starlingx/openstack-armada-app/+/774065/4/python-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/lifecycle/lifecycle_openstack.py
 * Apps can now run a semantic check on the reapply triggers.

Debugging

 * During 'system application-upload' and 'system application-apply', logs are captured in /var/log/sysinv.log at a high level.
 * App specific logs will be captured in /var/log/armada/xxx.log
 * Other directories of interest during app-upload and app-apply are:
 * /opt/platform/armada/20.01/
 * /scratch/apps/ → where the app tarball gets extracted and hosted