This page describes details, use cases and checks that Application Framework Team developers should be aware of when testing code changes to AppFwk.

Metadata Correctness Tests

The data provided on the metadata.yaml file is loaded into the sysinv database. The data from the file can be checked against the database for correctness. For instance:

Check if "maintain_user_overrides" database value matches the value from metadata.yaml
Check if "maintain_attributes" database value matches the value from metadata.yaml
Check if "plataform_managed_apps" database value matches the value from metadata.yaml
Check if "desired_state" database value matches the value from metadata.yaml
Check if evaluate reapply trigger values match the ones from metadata.yaml
Check if "auto_update" value matches the one from metadata.yaml

Example 1: snmp

Extract the metadata.yaml from the application tarball:

sysadmin@controller-0:~$ mkdir /home/sysadmin/snmp && tar -xvf /usr/local/share/applications/helm/snmp-1.0-1.tgz -C /home/sysadmin/snmp

Check the metadata.yaml content:

 sysadmin@controller-0:~$ cat /home/sysadmin/snmp/metadata.yaml 
 maintain_user_overrides: true

 app_name: snmp
 app_version: 1.0-1
 helm_repo: stx-platform

Compare against the database (data is json formatted):

sysadmin@controller-0:~$ sudo -u postgres psql -d sysinv -c "select name,app_metadata from kube_app where name='snmp';"
snmp | {"maintain_user_overrides": true, "app_name": "snmp", "app_version": "1.0-1", "helm_repo": "stx-platform"}

Example 2: platform-integ-apps

Extract the metadata.yaml from the application tarball:

sysadmin@controller-0:~$ mkdir /home/sysadmin/platform-integ-apps && tar -xvf /usr/local/share/applications/helm/platform-integ-apps-1.0-56.tgz -C /home/sysadmin/platform-integ-apps

Check the metadata.yaml content:

sysadmin@controller-0:~$ cat /home/sysadmin/platform-integ-apps/metadata.yaml 
app_name: platform-integ-apps
app_version: 1.0-56
helm_repo: stx-platform
maintain_user_overrides: true
behavior:
  platform_managed_app: yes
  desired_state: applied
  evaluate_reapply:
    triggers:
      - type: runtime-apply-puppet
      - type: host-availability-updated
      - type: kube-upgrade-complete
        filters:
          - availability: services-enabled
      - type: host-delete
        filters:
          - personality: controller

Compare against the database (data is json formatted):

sysadmin@controller-0:~$ sudo -u postgres psql -d sysinv -c "select name,app_metadata from kube_app where name='platform-integ-apps'"
platform-integ-apps | {"app_name": "platform-integ-apps", "app_version": "1.0-56", "helm_repo": "stx-platform", "maintain_user_overrides": true, "behavior": {"platform_managed_app": "yes", "desired_state": "applied", "evaluate_reapply": {"triggers": [{"type": "runtime-apply-puppet"}, {"type": "host-availability-updated"}, {"type": "kube-upgrade-complete", "filters": [{"availability": "services-enabled"}]}, {"type": "host-delete", "filters": [{"personality": "controller"}]}]}}}

Metadata Functional Tests

Update app and check whether overrides are preserved when "maintain_user_overrides" is true
Update app and check whether attributes are preserved when "maintain_attributes" is true
Check if app was automatically updated when "auto_update" is set to true
Check if application apply is triggered by automatic re-evaluation

Legacy Code Related Tests

Armada was replaced by Flux since stx-8 but Armada-based apps may still be around when updating from previous versions

Update Armada app version to Flux app version. For instance during upgrades from stx-7 to stx-8.
Make sure the upgrade-activate step is successful

Helm Release Tests

Check if helm release version is being correctly updated on sysinv database
Raise error when helm chart is invalid

Robustness Tests

Flux pods:
- Check helm-controller and source-controller pods are running and ready
- Check heml-controller and source controller pods memory and cpu usage
- Test application upload/apply/update/remove/delete across helm-controller and source-controller Kubernetes deployments restarts
Test application upload/apply/update/remove/delete during and after sysinv restarts

Example 1

Check if pods are running and ready

sysadmin@controller-0:~$ kubectl get pods -n flux-helm -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE           NOMINATED NODE   READINESS GATES
helm-controller-7d89fdbc47-2z7j2     1/1     Running   0          5h14m   172.16.192.95   controller-0   <none>           <none>
source-controller-67f6d76b4b-jj8bb   1/1     Running   0          5h14m   172.16.192.96   controller-0   <none>           <none>

Check pods CPU time. This is given in nanoseconds:

sysadmin@controller-0:~$ kubectl exec -it -n flux-helm helm-controller-7d89fdbc47-2z7j2 -- cat /sys/fs/cgroup/cpu/cpuacct.usage
96434038481

sysadmin@controller-0:~$ kubectl exec -it -n flux-helm source-controller-67f6d76b4b-jj8bb -- cat /sys/fs/cgroup/cpu/cpuacct.usage
60297801826

Check pods memory. Output converted to Megabytes:

sysadmin@controller-0:~$ kubectl exec -it -n flux-helm helm-controller-7d89fdbc47-2z7j2 -- cat /sys/fs/cgroup/memory/memory.usage_in_bytes | awk '{ foo = $1 / 1024 / 1024 ; print foo "MB" }'
34.8594MB

kubectl exec -it -n flux-helm source-controller-67f6d76b4b-jj8bb -- cat /sys/fs/cgroup/memory/memory.usage_in_bytes | awk '{ foo = $1 / 1024 / 1024 ; print foo "MB" }'
29.5742MB

Those values can be assessed in comparison to the pods requests and limits:

kubectl describe node controller-0

Check the "Non-terminated Pods" section:

Non-terminated Pods:           (21 in total)
 Namespace                    Name                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
 ---------                    ----                                               ------------  ----------  ---------------  -------------  ---
 [...]
 flux-helm                    helm-controller-68775d8db6-9s5nh                   0 (0%)        1 (50%)     64Mi (0%)        1Gi (5%)       83m
 flux-helm                    source-controller-64d9d88f74-lf9js                 0 (0%)        1 (50%)     64Mi (0%)        1Gi (5%)       83m
 [..]

Example 2

Restart all Flux pods:

sysadmin@controller-0:~$ kubectl rollout restart deployment -n flux-helm 
deployment.apps/helm-controller restarted
deployment.apps/source-controller restarted

Test application operations:

Upload application:

system application-upload system application-upload /usr/local/share/applications/helm/snmp-1.0-1.tgz

Apply application:

system application-apply snmp

Remove application:

system application-remove snmp

Delete application:

system application-delete snmp

Example 3

Restart all sysinv related services:

sysadmin@controller-0:~$ sudo systemctl restart sysinv-agent.service sysinv-api.service sysinv-conductor.service