StarlingX/Containers/StarlingXAppFrameworkInternals

This page describes details, use cases and checks that Application Framework Team developers should be aware of when testing code changes to AppFwk.

= Metadata Correctness Tests = The data provided on the metadata.yaml file is loaded into the sysinv database. The data from the file can be checked against the database for correctness. For instance:


 * Check if "maintain_user_overrides" database value matches the value from metadata.yaml
 * Check if "maintain_attributes" database value matches the value from metadata.yaml
 * Check if "plataform_managed_apps" database value matches the value from metadata.yaml
 * Check if "desired_state" database value matches the value from metadata.yaml
 * Check if evaluate reapply trigger values match the ones from metadata.yaml
 * Check if "auto_update" value matches the one from metadata.yaml

Example 1: snmp
Extract the metadata.yaml from the application tarball:

sysadmin@controller-0:~$ mkdir /home/sysadmin/snmp && tar -xvf /usr/local/share/applications/helm/snmp-1.0-1.tgz -C /home/sysadmin/snmp

Check the metadata.yaml content:

sysadmin@controller-0:~$ cat /home/sysadmin/snmp/metadata.yaml maintain_user_overrides: true

app_name: snmp app_version: 1.0-1 helm_repo: stx-platform

Compare against the database (data is json formatted):

sysadmin@controller-0:~$ sudo -u postgres psql -d sysinv -c "select name,app_metadata from kube_app where name='snmp';" snmp | {"maintain_user_overrides": true, "app_name": "snmp", "app_version": "1.0-1", "helm_repo": "stx-platform"}

Example 2: platform-integ-apps
Extract the metadata.yaml from the application tarball:

sysadmin@controller-0:~$ mkdir /home/sysadmin/platform-integ-apps && tar -xvf /usr/local/share/applications/helm/platform-integ-apps-1.0-56.tgz -C /home/sysadmin/platform-integ-apps

Check the metadata.yaml content:

sysadmin@controller-0:~$ cat /home/sysadmin/platform-integ-apps/metadata.yaml app_name: platform-integ-apps app_version: 1.0-56 helm_repo: stx-platform maintain_user_overrides: true behavior: platform_managed_app: yes desired_state: applied evaluate_reapply: triggers: - type: runtime-apply-puppet - type: host-availability-updated - type: kube-upgrade-complete filters: - availability: services-enabled - type: host-delete filters: - personality: controller

Compare against the database (data is json formatted):

sysadmin@controller-0:~$ sudo -u postgres psql -d sysinv -c "select name,app_metadata from kube_app where name='platform-integ-apps'" platform-integ-apps | {"app_name": "platform-integ-apps", "app_version": "1.0-56", "helm_repo": "stx-platform", "maintain_user_overrides": true, "behavior": {"platform_managed_app": "yes", "desired_state": "applied", "evaluate_reapply": {"triggers": [{"type": "runtime-apply-puppet"}, {"type": "host-availability-updated"}, {"type": "kube-upgrade-complete", "filters": [{"availability": "services-enabled"}]}, {"type": "host-delete", "filters": [{"personality": "controller"}]}]}}}

= Metadata Functional Tests =


 * Update app and check whether overrides are preserved when "maintain_user_overrides" is true
 * Update app and check whether attributes are preserved when "maintain_attributes" is true
 * Check if app was automatically updated when "auto_update" is set to true
 * Check if application apply is triggered by automatic re-evaluation

= Legacy Code Related Tests = Armada was replaced by Flux since stx-8 but Armada-based apps may still be around when updating from previous versions


 * Update Armada app version to Flux app version. For instance during upgrades from stx-7 to stx-8.
 * Make sure the upgrade-activate step is successful

= Helm Release Tests =


 * Check if helm release version is being correctly updated on sysinv database
 * Raise error when helm chart is invalid

= Robustness Tests =


 * Flux pods:
 * Check helm-controller and source-controller pods are running and ready
 * Check heml-controller and source controller pods memory and cpu usage
 * Test application upload/apply/update/remove/delete across helm-controller and source-controller Kubernetes deployments restarts
 * Test application upload/apply/update/remove/delete during and after sysinv restarts

Example 1
Check if pods are running and ready

sysadmin@controller-0:~$ kubectl get pods -n flux-helm -o wide NAME                                READY   STATUS    RESTARTS   AGE     IP              NODE           NOMINATED NODE   READINESS GATES helm-controller-7d89fdbc47-2z7j2    1/1     Running   0          5h14m   172.16.192.95   controller-0 source-controller-67f6d76b4b-jj8bb  1/1     Running   0          5h14m   172.16.192.96   controller-0

Check pods CPU time. This is given in nanoseconds:

sysadmin@controller-0:~$ kubectl exec -it -n flux-helm helm-controller-7d89fdbc47-2z7j2 -- cat /sys/fs/cgroup/cpu/cpuacct.usage 96434038481

sysadmin@controller-0:~$ kubectl exec -it -n flux-helm source-controller-67f6d76b4b-jj8bb -- cat /sys/fs/cgroup/cpu/cpuacct.usage 60297801826

Check pods memory. Output converted to Megabytes:

sysadmin@controller-0:~$ kubectl exec -it -n flux-helm helm-controller-7d89fdbc47-2z7j2 -- cat /sys/fs/cgroup/memory/memory.usage_in_bytes | awk '{ foo = $1 / 1024 / 1024 ; print foo "MB" }' 34.8594MB

kubectl exec -it -n flux-helm source-controller-67f6d76b4b-jj8bb -- cat /sys/fs/cgroup/memory/memory.usage_in_bytes | awk '{ foo = $1 / 1024 / 1024 ; print foo "MB" }' 29.5742MB

Those values can be assessed in comparison to the pods requests and limits:

kubectl describe node controller-0

Check the "Non-terminated Pods" section:

Non-terminated Pods:          (21 in total) Namespace                   Name                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age -                                                                    --  ---  -  ---  [...]  flux-helm                    helm-controller-68775d8db6-9s5nh                   0 (0%)        1 (50%)     64Mi (0%)        1Gi (5%)       83m flux-helm                   source-controller-64d9d88f74-lf9js                 0 (0%)        1 (50%)     64Mi (0%)        1Gi (5%)       83m [..]

Example 2
Restart all Flux pods:

sysadmin@controller-0:~$ kubectl rollout restart deployment -n flux-helm deployment.apps/helm-controller restarted deployment.apps/source-controller restarted

Test application operations:

Upload application: system application-upload system application-upload /usr/local/share/applications/helm/snmp-1.0-1.tgz

Apply application: system application-apply snmp

Remove application:

system application-remove snmp

Delete application:

system application-delete snmp

Example 3
Restart all sysinv related services:

sysadmin@controller-0:~$ sudo systemctl restart sysinv-agent.service sysinv-api.service sysinv-conductor.service

Test application operations:

Upload application:

system application-upload /usr/local/share/applications/helm/platform-integ-apps-1.0-56.tgz

Apply/reapply application: system application-apply platform-integ-apps

Remove application:

system application-remove platform-integ-apps

Delete application:

system application-delete platform-integ-apps