Jump to: navigation, search

Difference between revisions of "StarlingX/Containers/Application/Archive/ArmadaAppCodeStructure"

(Created page with "It is recommended to start off copying another app code structure as starting point and then make necessary changes to various configuration build & configuration files. Foll...")
 
Line 1: Line 1:
 +
 +
== Overview ==
 
It is recommended to start off copying another app code structure as starting point and then make necessary changes to various configuration build & configuration files.
 
It is recommended to start off copying another app code structure as starting point and then make necessary changes to various configuration build & configuration files.
  
Line 5: Line 7:
 
<gallery>
 
<gallery>
 
Example.jpg|Caption1
 
Example.jpg|Caption1
 +
</gallery>
 +
 +
The 'cert-manager-helm' structure takes the tarball in stx/downloads (which originated from 3rd party public upstream repository), unpacks, applies patches (more on this below) and compiles to build the cert-manager helm charts using an included Makefile. The files of particular interest here are in the 'centos' folder. The .spec file directs the details of the build.
 +
 +
The 'stx-cert-manager-helm' builds (Stx/WRCP branded) Armada Helm Charts (output of this exercise). Again, files that direct the build steps are in 'centos' folder and the .spec file contains the details. The manifest folder contains the main manifest file for the application that contains the Armada ChartGroups and one or more Helm charts & any particular overrides. Details on how to write this manifest can be found here: https://readthedocs.org/projects/airship-armada/downloads/pdf/latest/
 +
 +
== Patches ==
 +
* Apply patches if needed.
 +
* Copy generated patch file over to 'files' folder and make appropriate changes in spec file to apply the patch.
 +
 +
== Image Pull Secrets ==
 +
Our applications use images from registry.local:9001 for deployment. This is an authenticated registry and require image pull secrets to deploy pods successfully. Upstream helm charts will almost certainly not have image pull secrets set up. This means pods will fail to deploy with an "ImagePullBackOff" error. We need to add an "imagePullSecrets" section to each "containers" entry from the helm charts. This needs to be done through a patch if using upstream helm charts, or directly if we are making the helm charts ourselves. Refer to the following commit:
 +
https://review.opendev.org/c/starlingx/vault-armada-app/+/772616
 +
 +
To test, deploy your application and ensure the pods come up as opposed to failing due to the "ImagePullBackOff" error.
 +
 +
== Pod Security Policies ==
 +
PodSecurityPolicy is an optional plugin to control pod deployments. If this plugin is enabled, by default no pods can be deployed. We need to add policies to allow pods to be deployed in each namespace. Applications normally create their own namespace when being deployed. If your application deploys in an existing namespace, you will not need to do this. Refer to the following commit to add pod security policies for your application:
 +
https://review.opendev.org/c/starlingx/openstack-armada-app/+/736002
 +
 +
This commit deploys the pod security policy helm chart under the common "helm-charts" repo. You will need to add this helm chart to your application's armada manifest and provide the correct "rolebindingNamespace" and "serviceAccount". The spec and build_srpm.data change is to build the common helm-charts chart for pod security policies.
 +
 +
There are currently some issues enabling the pod security policies plugin. I will update this section when that gets resolved. To test, simply follow the instructions above, and ensure that when you apply your application, the rolebinding is deployed along with it:
 +
 +
kubectl get rolebinding -n <your_namespace> | grep psp
 +
* your_namespace-privileged-psp-users
 +
 +
== Defining CRDs via Helm Charts ==
 +
If your application requires CRDs, there may be multiple ways to declare and define the CRDs. For e.g., the Deployment Manager has a section where CRDs are defined and can be a viable option.
 +
 +
If the CRDs have to be defined as part of the application deployment at runtime (dynamically performed during the 'system application-upload' and 'system application-apply'), a separate sequenced Helm Chart may be an option. This is the option chosen for cert-manager. There are two charts - first chart defines the CRDs and the second chart runs the various container images necessary for the app.
 +
 +
To get the Helm Charts for CRDs, the following steps were performed (Note that you need a system with helm installed).
 +
* Get the CRD yaml files from 3rd party upstream codebase
 +
* Google the commands for Helm chart creation (such as this page). In short, the steps required include:
 +
** helm create <chartname>
 +
** Remove unneeded files
 +
*** In 'templates' folder, remove the existing *.yaml files
 +
*** There is a 'tests' folder created, (which was noted to cause issues during deploying non-existent "certmgr-crds-test-connection" pod. Fix was to simply delete the 'tests' folder).
 +
** Copy the app's CRD yaml files to the 'templates' folder
 +
** helm install --dry-run --debug <chartname>
 +
** helm lint <chartname>
 +
** (Optional) helm package <chartname>
 +
** Manually run 'helm install' command on a machine to test if the chart was created ok and the CRDs get defined as expected (use 'kubectl get crd' to check if it worked after 'helm install')
 +
* Copy the helm charts created above in the stx-cert-manager-helm folder under "helm-charts"
 +
 +
 +
''Note:'' During the deployment (during 'system application-apply'), Armada checks if resources came up successfully and by default checks for a 'pod' or 'job' with the same name as the chartname (or a label). In case of CRDs, since we are not going to start any pods, this will result in a failure of Armada checkpoint. To fix this issue, set 'resources: []' under 'wait' of the CRD chart. This directs Armada to nullify any resource checks after running this Helm chart. See the screenshot with highlight below.
 +
 +
<gallery>
 
Example.jpg|Caption2
 
Example.jpg|Caption2
 
</gallery>
 
</gallery>
 +
 +
== Lifecycle hooks ==
 +
Apps can package their own code to be run around sysinv events.
 +
(Hint: search the codebase for APP_LIFECYCLE_, LifecycleHook, LifecycleHookInfo. Also open the overview)
 +
 +
Code is located under: ${MY_REPO}/stx/openstack-armada-app/python-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/lifecycle
 +
 +
Creating the lifecycle operator: https://review.opendev.org/c/starlingx/openstack-armada-app/+/763059
 +
 +
Files will be extracted on the running controller under: /opt/platform/helm/20.12/stx-openstack/1.0-78-centos-stable-latest/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py
 +
 +
== Control behavior through metadata ==
 +
The metadata file is located at stx-<app>-helm/stx-<app>-helm/files/metadata.yaml in the source code.
 +
 +
Validation of metadata is done in ${MY_REPO}/stx/config/sysinv/sysinv/sysinv/sysinv/common/utils.py::find_metadata_file.
 +
 +
The bridge between metadata and code is APP_EVALUATE_REAPPLY_TRIGGER_TO_METADATA_MAP.
 +
 +
* Apps can specify which apps should be evaluated before self [https://review.opendev.org/c/starlingx/config/+/775611]
 +
metadata.yaml
 +
behavior:
 +
  platform_managed_app: yes
 +
  evaluate_reapply:
 +
    after:
 +
      - platform-integ-apps
 +
 +
* Apps can subscribe to events that trigger a reapply evaluation [https://review.opendev.org/c/starlingx/config/+/773451]
 +
metadata.yaml
 +
behavior:
 +
  platform_managed_app: yes
 +
  evaluate_reapply:
 +
    triggers:
 +
      - type: unlock
 +
      - type: force-unlock
 +
      - type: runtime-apply-puppet
 +
      - type: host-availability-updated
 +
        filters:
 +
          - availability: services-enabled
 +
      - type: host-delete
 +
        filters:
 +
          - openstack_worker: True
 +
 +
* Apps can specify if they want to be audited. Retries for failed operations (apply, update, recover) [https://review.opendev.org/c/starlingx/config/+/774292]
 +
metadata.yaml
 +
behavior:
 +
  platform_managed_app: true
 +
 +
* Apps can specify which states they want to achieve (uploaded, applied) [https://review.opendev.org/c/starlingx/config/+/774292]
 +
metadata.yaml
 +
behavior:
 +
  platform_managed_app: true
 +
  desired_state: uploaded
 +
 +
* Apps can specify which actions (remove, delete) are forbidden from being performed; for application-delete: [https://review.opendev.org/c/starlingx/config/+/776518] and for application-remove: [https://review.opendev.org/c/starlingx/config/+/779023]
 +
metadata.yaml
 +
behavior:
 +
  forbidden_manual_operations:
 +
    - delete
 +
    - remove
 +
 +
* Apps can now run a semantic check on the reapply triggers.
 +
An example of how to apply a reapply trigger filter (both using code, and equivalent behavior achieved by using metadata):
 +
https://review.opendev.org/c/starlingx/openstack-armada-app/+/774065/4/python-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/lifecycle/lifecycle_openstack.py
 +
 +
 +
== Debugging ==
 +
* During 'system application-upload' and 'system application-apply', logs are captured in /var/log/sysinv.log at a high level.
 +
* App specific logs will be captured in /var/log/armada/xxx.log
 +
* Other directories of interest during app-upload and app-apply are:
 +
** /opt/platform/armada/20.01/<app>
 +
** /scratch/apps/<app> → where the app tarball gets extracted and hosted

Revision as of 00:55, 19 February 2022

Overview

It is recommended to start off copying another app code structure as starting point and then make necessary changes to various configuration build & configuration files.

Following screenshot shows the directory structure for cert-manager. Note that most of the files are under 'cert-manager-helm' and 'stx-cert-manager-helm'

The 'cert-manager-helm' structure takes the tarball in stx/downloads (which originated from 3rd party public upstream repository), unpacks, applies patches (more on this below) and compiles to build the cert-manager helm charts using an included Makefile. The files of particular interest here are in the 'centos' folder. The .spec file directs the details of the build.

The 'stx-cert-manager-helm' builds (Stx/WRCP branded) Armada Helm Charts (output of this exercise). Again, files that direct the build steps are in 'centos' folder and the .spec file contains the details. The manifest folder contains the main manifest file for the application that contains the Armada ChartGroups and one or more Helm charts & any particular overrides. Details on how to write this manifest can be found here: https://readthedocs.org/projects/airship-armada/downloads/pdf/latest/

Patches

  • Apply patches if needed.
  • Copy generated patch file over to 'files' folder and make appropriate changes in spec file to apply the patch.

Image Pull Secrets

Our applications use images from registry.local:9001 for deployment. This is an authenticated registry and require image pull secrets to deploy pods successfully. Upstream helm charts will almost certainly not have image pull secrets set up. This means pods will fail to deploy with an "ImagePullBackOff" error. We need to add an "imagePullSecrets" section to each "containers" entry from the helm charts. This needs to be done through a patch if using upstream helm charts, or directly if we are making the helm charts ourselves. Refer to the following commit: https://review.opendev.org/c/starlingx/vault-armada-app/+/772616

To test, deploy your application and ensure the pods come up as opposed to failing due to the "ImagePullBackOff" error.

Pod Security Policies

PodSecurityPolicy is an optional plugin to control pod deployments. If this plugin is enabled, by default no pods can be deployed. We need to add policies to allow pods to be deployed in each namespace. Applications normally create their own namespace when being deployed. If your application deploys in an existing namespace, you will not need to do this. Refer to the following commit to add pod security policies for your application: https://review.opendev.org/c/starlingx/openstack-armada-app/+/736002

This commit deploys the pod security policy helm chart under the common "helm-charts" repo. You will need to add this helm chart to your application's armada manifest and provide the correct "rolebindingNamespace" and "serviceAccount". The spec and build_srpm.data change is to build the common helm-charts chart for pod security policies.

There are currently some issues enabling the pod security policies plugin. I will update this section when that gets resolved. To test, simply follow the instructions above, and ensure that when you apply your application, the rolebinding is deployed along with it:

kubectl get rolebinding -n <your_namespace> | grep psp

  • your_namespace-privileged-psp-users

Defining CRDs via Helm Charts

If your application requires CRDs, there may be multiple ways to declare and define the CRDs. For e.g., the Deployment Manager has a section where CRDs are defined and can be a viable option.

If the CRDs have to be defined as part of the application deployment at runtime (dynamically performed during the 'system application-upload' and 'system application-apply'), a separate sequenced Helm Chart may be an option. This is the option chosen for cert-manager. There are two charts - first chart defines the CRDs and the second chart runs the various container images necessary for the app.

To get the Helm Charts for CRDs, the following steps were performed (Note that you need a system with helm installed).

  • Get the CRD yaml files from 3rd party upstream codebase
  • Google the commands for Helm chart creation (such as this page). In short, the steps required include:
    • helm create <chartname>
    • Remove unneeded files
      • In 'templates' folder, remove the existing *.yaml files
      • There is a 'tests' folder created, (which was noted to cause issues during deploying non-existent "certmgr-crds-test-connection" pod. Fix was to simply delete the 'tests' folder).
    • Copy the app's CRD yaml files to the 'templates' folder
    • helm install --dry-run --debug <chartname>
    • helm lint <chartname>
    • (Optional) helm package <chartname>
    • Manually run 'helm install' command on a machine to test if the chart was created ok and the CRDs get defined as expected (use 'kubectl get crd' to check if it worked after 'helm install')
  • Copy the helm charts created above in the stx-cert-manager-helm folder under "helm-charts"


Note: During the deployment (during 'system application-apply'), Armada checks if resources came up successfully and by default checks for a 'pod' or 'job' with the same name as the chartname (or a label). In case of CRDs, since we are not going to start any pods, this will result in a failure of Armada checkpoint. To fix this issue, set 'resources: []' under 'wait' of the CRD chart. This directs Armada to nullify any resource checks after running this Helm chart. See the screenshot with highlight below.

Lifecycle hooks

Apps can package their own code to be run around sysinv events. (Hint: search the codebase for APP_LIFECYCLE_, LifecycleHook, LifecycleHookInfo. Also open the overview)

Code is located under: ${MY_REPO}/stx/openstack-armada-app/python-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/lifecycle

Creating the lifecycle operator: https://review.opendev.org/c/starlingx/openstack-armada-app/+/763059

Files will be extracted on the running controller under: /opt/platform/helm/20.12/stx-openstack/1.0-78-centos-stable-latest/plugins/k8sapp_openstack/lifecycle/lifecycle_openstack.py

Control behavior through metadata

The metadata file is located at stx-<app>-helm/stx-<app>-helm/files/metadata.yaml in the source code.

Validation of metadata is done in ${MY_REPO}/stx/config/sysinv/sysinv/sysinv/sysinv/common/utils.py::find_metadata_file.

The bridge between metadata and code is APP_EVALUATE_REAPPLY_TRIGGER_TO_METADATA_MAP.

  • Apps can specify which apps should be evaluated before self [1]
metadata.yaml
behavior:
 platform_managed_app: yes
 evaluate_reapply:
   after:
     - platform-integ-apps
  • Apps can subscribe to events that trigger a reapply evaluation [2]
metadata.yaml
behavior:
 platform_managed_app: yes
 evaluate_reapply:
   triggers:
     - type: unlock
     - type: force-unlock
     - type: runtime-apply-puppet
     - type: host-availability-updated
       filters:
         - availability: services-enabled
     - type: host-delete
       filters:
         - openstack_worker: True
  • Apps can specify if they want to be audited. Retries for failed operations (apply, update, recover) [3]
metadata.yaml
behavior:
 platform_managed_app: true
  • Apps can specify which states they want to achieve (uploaded, applied) [4]
metadata.yaml
behavior:
 platform_managed_app: true
 desired_state: uploaded
  • Apps can specify which actions (remove, delete) are forbidden from being performed; for application-delete: [5] and for application-remove: [6]
metadata.yaml
behavior:
 forbidden_manual_operations:
   - delete
   - remove
  • Apps can now run a semantic check on the reapply triggers.

An example of how to apply a reapply trigger filter (both using code, and equivalent behavior achieved by using metadata): https://review.opendev.org/c/starlingx/openstack-armada-app/+/774065/4/python-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/lifecycle/lifecycle_openstack.py


Debugging

  • During 'system application-upload' and 'system application-apply', logs are captured in /var/log/sysinv.log at a high level.
  • App specific logs will be captured in /var/log/armada/xxx.log
  • Other directories of interest during app-upload and app-apply are:
    • /opt/platform/armada/20.01/<app>
    • /scratch/apps/<app> → where the app tarball gets extracted and hosted