StarlingX/Containers/Applications/AppIntegration
This page should provide an insight about configuration, features and general guidelines of StarlingX Apps and interaction with the App Framework.
For a build perspective of StarlingX Apps this tutorial covers application integration.
StarlingX management commands: https://docs.starlingx.io/cli_ref/system.html#application-management
General directory structure of a StarlingX App at build time.
This page receives constant improvements. Plan is to improve the app_lifecycle_actions section and possibly add a diagram for app state transitions.
Contents
- 1 Core of a StarlingX App
- 1.1 metadata.yaml
- 1.1.1 app_name
- 1.1.2 app_version
- 1.1.3 downgrades
- 1.1.4 upgrades
- 1.1.5 supported_k8s_version
- 1.1.6 k8s_upgrades
- 1.1.7 supported_releases
- 1.1.8 repo
- 1.1.9 disabled_charts
- 1.1.10 maintain_attributes
- 1.1.11 maintain_user_overrides
- 1.1.12 behavior
- 1.1.13 class
- 1.1.14 dependent_apps
- 1.1.15 dependent_parent_exceptions
- 1.2 app_lifecycle_actions
- 1.2.1 LifecycleHookInfo
- 1.2.1.1 Lists of lifecycle constants
- 1.2.1.2 Utility file
- 1.2.1.3 Exceptions
- 1.2.1.4 LifecycleHookInfo/mode
- 1.2.1.5 LifecycleHookInfo/lifecycle_type
- 1.2.1.6 LifecycleHookInfo/relative_timing
- 1.2.1.7 LifecycleHookInfo/operation
- 1.2.1.7.1 APP_UPLOAD_OP
- 1.2.1.7.2 APP_APPLY_OP
- 1.2.1.7.3 APP_REMOVE_OP
- 1.2.1.7.4 APP_DELETE_OP
- 1.2.1.7.5 APP_UPDATE_OP
- 1.2.1.7.6 APP_DOWNGRADE_OP
- 1.2.1.7.7 APP_ABORT_OP
- 1.2.1.7.8 APP_EVALUATE_REAPPLY_OP
- 1.2.1.7.9 APP_BACKUP
- 1.2.1.7.10 APP_ETCD_BACKUP
- 1.2.1.7.11 APP_RESTORE
- 1.2.1.7.12 APP_LIFECYCLE_OPERATION_MTC_ACTION
- 1.2.1.8 LifecycleHookInfo/extra
- 1.2.2 Lifecycle exceptions
- 1.2.1 LifecycleHookInfo
- 1.3 Helm Chart Overrides
- 1.1 metadata.yaml
- 2 Guidelines
- 3 Considerations About Classes and Direct Dependencies
- 4 Considerations About Application Updates
- 5 Considerations About Applications on Subclouds
- 6 Related work
Core of a StarlingX App
metadata.yaml
Referencing the general directory structure (https://wiki.openstack.org/wiki/StarlingX/Containers/HowToAddNewFluxCDAppInSTX#Step_7:_Develop_your_application_FluxCD_packaging), enabling features can be done by modifying a yaml file located at: stx-APPNAME-helm/stx-APPNAME-helm/files/metadata.yaml
There is a function definition called 'def find_metadata_file' in https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/common/utils.py, that lists entries used by the framework.
Note: Currently there are a few entries missing from utils.py, this will be updated when code there will be updated. The guide in later steps may still explain and show examples of entries not listed here. For example this snapshot of https://opendev.org/starlingx/config/src/commit/e9705f5bc61f29618dd34b408a4608797422a7ad/sysinv/sysinv/sysinv/sysinv/common/utils.py#L2235, doesn't have `maintain_attributes` entry:
app_name: <name>
app_version: <version>
upgrades:
auto_update: <true/false/yes/no>
update_failure_no_rollback: <true/false/yes/no>
from_versions:
- <version.1>
- <version.2>
supported_k8s_version:
minimum: <version>
maximum: <version>
k8s_upgrades:
auto_update: <true/false/yes/no>
timing: <pre/post>
supported_releases:
<release>:
- <patch.1>
- <patch.2>
...
repo: <helm repo> - optional: defaults to HELM_REPO_FOR_APPS
disabled_charts: - optional: charts default to enabled
- <chart name>
- <chart name>
...
maintain_attributes: <true|false>
- optional: defaults to false. Over an app update any system overrides
are preserved for the new version of the application. This can be
renamed to 'maintain_system_overrides', but will require more effort
to keep the naming of 'helm-chart-attribute-modify' command in sync
with this.
maintain_user_overrides: <true|false>
- optional: defaults to false. Over an app update any user overrides are
preserved for the new version of the application
behavior: - optional: describes the app behavior
platform_managed_app: <true/false/yes/no> - optional: when absent
behaves as false
desired_state: <uploaded/applied> - optional: state the app should
reach
evaluate_reapply: - optional: describe the reapply evaluation behaviour
triggers: - optional: list of what triggers the reapply evaluation
- type: <key in APP_EVALUATE_REAPPLY_TRIGGER_TO_METADATA_MAP>
filters: - optional: list of field:value, that aid filtering
of the trigger events. All pairs in this list must be
present in trigger dictionary that is passed in
the calls (eg. trigger[field_name1]==value_name1 and
trigger[field_name2]==value_name2).
Function evaluate_apps_reapply takes a dictionary called
'trigger' as parameter. Depending on trigger type this
may contain custom information used by apps, for example
a field 'personality' corresponding to node personality.
It is the duty of the app developer to enhance existing
triggers with the required information.
Hard to obtain information should be passed in the trigger.
To use existing information it is as simple as defining
the metadata.
- <field_name.1>: <value_name.1>
- <field_name.2>: <value_name.2>
filter_field: <field_name> - optional: field name in trigger
dictionary. If specified the filters are applied
to trigger[filter_field] sub-dictionary instead
of the root trigger dictionary.
class: <critical/storage/discovery/optional/reporting> - optional
dependent_apps: - optional: list of dependent apps.
- name: <app_name>
version: <regular expression (full match)> e.g: 25\.09-\d+
action: <ignore|warn|error|apply>
dependent_parent_exceptions: exceptions for parent apps dependencies.
- name: <app_name>
version: <regular expression (full match)> e.g: 25\.09-1
app_name
This is a string and it represents the name of an application. This field is required.
app_version
This is a string and it represents the version of an application. This field is required. Example: <major>.<minor>-<patch> or 1.0-1
downgrades
This section details downgrade behavior of the application.
Automatic application downgrades should be triggered when platform patches are removed, taking into account the active Kubernetes version and the latest application version available under /usr/local/share/applications/helm/.
downgrades/auto_downgrade
This is a mechanism to allow apps to be automatically downgraded. This field is not required but the default behavior is to do downgrade an app upon patch removal unless this field is set, and a specific app does not allow it. To override this behavior, a specific app should set this to 'false' or 'no'. Some applications may not be able to downgrade due to data migration or other reason, so implement this change as needed.
| auto_downgrade flag / update procedure | N+1->N(down-version) |
|---|---|
| enabled on N version | automatic down-version |
| enabled on N+1 version | not affected |
| disabled on N version | needs manual intervention to update to older(N) version |
| disabled on N+1 version | not affected |
upgrades
This section details upgrade behavior of the application.
upgrades/auto_update
This is a mechanism to allow apps to be automatically updated. This can be used both when 1) delivering an updated app part of a platform patch and 2) after platform upgrades `system upgrade-complete` step. Probably a bad naming because it is present under a key named 'upgrades', but it was designed under an upgrades context, I suspect no one realized there is a `patching` context at that moment. For explaining the platform upgrades see Upgrade consideration.
The auto update will be triggered only when the app is in applied state.
If you want to enable the auto update feature you need to update the metadata.yaml ([1] example location for one app), adding at root level the following
upgrades: auto_update: true
In case of patching a live system, after the patch is applied, when a new version of an app is delivered via the patch and the new version has this mechanism enabled, it will get automatically updated(up-versioned) to the new version. In case of patching a live system, after the patch is applied, when a new version of an app is delivered via the patch and the new version does not have this mechanism enabled, it will not get automatically updated(up-versioned) to the new version.
In case of patching a live system, after the patch is removed, when an old version of an app is delivered via the patch removal and the old version has this mechanism enabled, it will get automatically updated(down-versioned) to the old version. In case of patching a live system, after the patch is removed, when an old version of an app is delivered via the patch removal and the old version does not have this mechanism enabled, it will not get automatically updated(down-versioned) to the old version.
| auto_update flag / update procedure | N->N+1(up-version) |
|---|---|
| enabled on N version | not affected |
| enabled on N+1 version | automatic up-version |
| disabled on N version | not affected |
| disabled on N+1 version | needs manual intervention to update to newer(N+1) version |
Please note: The N+1->N(down-version) previously addressed here is not defined in the downgrades/auto_downgrade section.
This was introduced in stx.6.0 by https://review.opendev.org/c/starlingx/config/+/800821
Refer to this section for more information about automatic application updates.
upgrades/update_failure_no_rollback
This field controls whether the framework should automatically rollback to the previous application version when an update fails. It is not required and defaults to "false".
When set to "false" (default), a failed update triggers automatic recovery to the previous version. When set to "true", recovery is skipped and the application remains in "apply-failed" state with the new version, requiring manual intervention.
Format:
upgrades: update_failure_no_rollback: <true/false/yes/no>
Example:
upgrades: auto_update: true update_failure_no_rollback: true
When to use:
Enable this flag when rolling back to the previous version is undesirable or potentially harmful.
Important notes:
- During an activate-rollback scenario, automatic recovery is always disabled regardless of this flag's value.
upgrades/from_versions
This field defines a list of source version patterns from which the application is allowed to be updated. It acts as a safeguard to prevent unsupported update paths, ensuring that only known and tested version transitions are performed.
Each entry in the list is interpreted as a Python regular expression (matched using re.fullmatch). The current version of the installed application is checked against every entry: if it matches any one of them, the update is allowed to proceed.
Key behaviors:
- If "from_versions" is not defined in metadata, the update is always allowed (backward compatible behavior — no restriction).
- If "from_versions" is defined but the list is empty, all updates are rejected (no valid source version exists).
- If "from_versions" is defined and the currently installed version does not match any entry, the update is rejected with a "KubeAppUpdateFailure" error. The application reverts to its previous applied state.
- The check applies to both manual updates (
system application-update) and automatic updates (triggered by patching or platform upgrades).
Format:
upgrades:
from_versions:
- <regex_pattern_1>
- <regex_pattern_2>
Entries support both exact version strings and regular expressions. Since the matching uses "re.fullmatch", the entire version string must match the pattern (partial matches are not accepted).
Examples:
Allow update only from version 1.0-1 (exact match):
upgrades:
from_versions:
- 1\.0-1
Allow update from any 1.2 patch version and from version 2.0-1 specifically:
upgrades:
from_versions:
- 1\.2-\d+
- 2\.0-1
Allow update from any patch of version 1.5:
upgrades:
from_versions:
- 1\.5-\d+
When to use:
Use "from_versions" when an application requires specific data migration steps or has known incompatibilities between certain versions. For instance, if version 3.0-1 only supports direct migration from versions 2.5 or 2.6, you can restrict the update path:
# metadata.yaml for version 3.0-1
upgrades:
from_versions:
- 2\.5-\d+
- 2\.6-\d+
With this configuration, only systems running version 2.5-x or 2.6-x can update directly to 3.0-1. Systems on older versions (e.g. 2.4-1 or 1.0-1) will have their update rejected, requiring them to first update to an intermediate version (2.5 or 2.6) before reaching 3.0-1.
supported_k8s_version
Applications are a collection of one or more helm charts, and those helm charts may or may not have a limitation placed upon which version of Kubernetes it works with.
minimum: This is the minimum kubernetes version the application will run with. This field is required. If a given helm chart in the app is compatible with any Kubernetes version, then recommend using the lowest K8S version that STX starts with. Use the lowest version of Kubernetes across all helm charts within a single application.
maximum: This field is not required, but if provided, would be the maximum Kubernetes version the app is compatible with. This field is typically set only in the case where a newer and incompatible Kubernetes version is introduced, and the application is being updated to a higher version.
These are the formats accepted for both minimum and maximum fields:
- major.minor.patch (e.g. 1.29.2)
- vmajor.minor.patch (e.g. v1.29.2)
- vmajor.minor (e.g. v1.29 - this is interpreted as 1.29.0)
Please note, if the STX platform is not either above the minimum Kubernetes version and at or under the maximum, if supplied, then the application will be rejected by the platform.
k8s_upgrades
The following describe behavior of the application during a platform upgrade of Kubernetes from N to N+1 or N to > N+1, or multi version jump.
k8s_upgrades/auto_update: Set this to true to have the application framework update the application upon the event where K8S is upgraded. The default is to do so: True. To force manual intervention, set this field to false. This field is required.
k8s_upgrades/timing: In the event of a Kubernetes upgrade, if the auto_update is set to True, then the application framework will need to know whether to perform the application upgrade prior to (pre) or after the upgrade occurs (post). The default behavior is "post", or after the K8S upgrade. This field is required.
supported_releases
A specific release and patch can be specified as required for the applciation to install properly. If at time of upload the release and patch combo are not present the there will be an error. This field is not required and there is no default.
<Release>:
- patch 1
- patch 2
repo
The repo field actually refers to Helm Repository, and it defaults to the value of HELM_REPO_FOR_APP which is a constant coded for "stx-platform" which is the STX platform local helm repository. This field is not required, but specify it in instances where the helm chart is to be stored somewhere other than the local stx-platform helm repository.
disabled_charts
There is the ability to disable a chart in an application by naming them in this section. This field is not required nor is a default specified. However, if a chart is specified then it must be a valid helm chart in the application. If a chart in the application is specified then upon upload/apply that chart will not be installed to start with. The user would have to manually enable the chart after application install.
disabled_charts: - chart name 1 - chart name 2
maintain_attributes
This field defaults to false and is not required.
Currently if you disable a helm chart when you update an app it will be re-enabled by default on the newer version.
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-chart-attribute-modify usage: system helm-chart-attribute-modify [--enabled <true/false>] <app name> <chart name> <namespace>
system helm-chart-attribute-modify --enabled false MY_APP MY_CHART MY_NAMESPACE
The chart attribute(enabled/disabled) itself is stored in sysinv database table helm_overrides in a column called 'system_overrides' (bad naming, will be aligned later).enabled
If you want to keep the disabled status during app update you can update the metadata.yaml ([1] example location for one app), adding at root level the following `maintain_attributes: true`
There is more. You can override the behavior present by adding a special flag during 'system application-update'. You can force the information either way: reuse(will keep disabled the charts that were disabled) or not reuse(reset all the charts to be enabled).
system application[sysadmin@controller-0 ~(keystone_admin)]$ system application-update usage: system application-update [-n <app name>] [-v <app version>] [--reuse-user-overrides <true/false>] [--reuse-attributes <true/false>] <tar file>
system application[sysadmin@controller-0 ~(keystone_admin)]$ system application-update -n MY_APP -v MY_VERSION --reuse-attributes true /path/to/tar.gz
This was introduced in stx.8.0 by: https://review.opendev.org/c/starlingx/config/+/865327
maintain_user_overrides
This field defaults to False and is not required.
Currently if you create overrides for a helm chart, when you update an app the overrides will be lost.
system helm-override-update <app name> <chart name> <namespace>
The overrides themselves are stored in sysinv database table helm_overrides in a column called 'user_overrides'.
If you want to keep the overrides during app update you can update the metadata.yaml ([1] example location for one app), adding at root level the following `maintain_user_overrides: true`
There is more. You can override the behavior present by adding a special flag during 'system application-update'. You can force the information either way: reuse(will keep the overrides) or not reuse(reset overrides).
system application[sysadmin@controller-0 ~(keystone_admin)]$ system application-update usage: system application-update [-n <app name>] [-v <app version>] [--reuse-user-overrides <true/false>] [--reuse-attributes <true/false>] <tar file>
system application[sysadmin@controller-0 ~(keystone_admin)]$ system application-update -n MY_APP -v MY_VERSION --reuse-user-overrides true /path/to/tar.gz
behavior
The following describes additional behavior that can be prescribed to an application. The entire behavior section itself is not required nor is there any specific defaults.
behavior/platform_managed_app
There is a mechanism to tell the Framework the app should be managed by the Framework. We call this a platform managed app. The advantages of a platform managed app are:
- the Framework can perform some automated tasks such as auto reapply of the app based on specific triggers.
- the Framework can achieve a desired state after unlocking the first controller.
The functionalities are described later in their specific sections.
If you want to enable the platform managed feature you need to update the metadata.yaml ([1] example location for one app), adding at root level the following
behavior: platform_managed_app: yes
This was introduced in stx.5.0 by https://review.opendev.org/c/starlingx/config/+/773451
behavior/desired_state In case an app is declared a platform managed app the desired state to be achieve by the Framework can controlled using a variable called desired_state. Only uploaded and applied states are supported now.
Currently the Framework can achieve uploaded state by setting this metadata:
behavior: platform_managed_app: yes desired_state: uploaded
Currently the Framework can achieve applied state by setting this metadata:
behavior: platform_managed_app: yes desired_state: applied
This was introduced in stx.5.0 by https://review.opendev.org/c/starlingx/config/+/773451
behavior/evaluate_reapply
In case an app is declared a platform managed app doing automatic app re-applies by the Framework, based on specific conditions, can be controlled using a variable called evaluate_reapply. The Framework will determine(evaluate) if there is a change that require an app apply.
This was introduced in stx.5.0 by https://review.opendev.org/c/starlingx/config/+/773451
behavior/evaluate_reapply/triggers
Application developers can control what triggers the Framework to evaluate whether a given app should be re-applied. The app can be instructed to subscribe to some events by creating the necessary configuration in metadata.yaml for it. That can be done using a list variable called triggers:
behavior:
platform_managed_app: yes
evaluate_reapply:
triggers:
- trigger1
- trigger2
The definition is explained in [1]. A snapshot of trigger types can be found at [2].
Triggers are created and passed to evaluate_apps_reapply function inside sysinv-conductor [3]. A real life example of such call is
self.evaluate_apps_reapply(context, trigger={'type': constants.APP_EVALUATE_REAPPLY_HOST_AVAILABILITY,
'availability': availability})
The most basic trigger is
evaluate_apps_reapply(context, trigger={'type': 'some-type'})
which can be subscribed to by this metadata:
behavior:
platform_managed_app: yes
evaluate_reapply:
triggers:
- type: some-type
A list of filters can be applied to a trigger (all key:value pairs must match)
evaluate_apps_reapply(context, trigger={'type': 'some-type', 'key1': 'value1', 'key2': 'value2'})
which can be subscribed to by this metadata:
behavior:
platform_managed_app: yes
evaluate_reapply:
triggers:
- type: some-type
filter:
- key1: value1
- key2: value2
A list of filters can be applied to a trigger (all key:value pairs must match) on a subdictionary
evaluate_apps_reapply(context, trigger={'type': 'some-type', 'subdict1': {'key1': 'value1', 'key2': 'value2'}})
which can be subscribed to by this metadata:
behavior:
platform_managed_app: yes
evaluate_reapply:
triggers:
- type: some-type
filter_field: subdict1
filter:
- key1: value1
- key2: value2
This was introduced in stx.5.0 by https://review.opendev.org/c/starlingx/config/+/773451
Available trigger types:
The following table lists all trigger types available in the platform. Apps can subscribe to any of these by declaring them in the behavior.evaluate_reapply.triggers section of their metadata.
| Type (metadata value) | When it is fired | Additional fields (filter) |
|---|---|---|
| unlock | Host unlock action is performed. | configure_required |
| force-unlock | Host force-unlock action is performed. | configure_required |
| lock | Host lock action is performed. | configure_required |
| force-lock | Host force-lock action is performed. | configure_required |
| force-unsafe-lock | Host force-unsafe-lock action is performed. | configure_required |
| swact | Host swact action is performed. | configure_required |
| force-swact | Host force-swact action is performed. | configure_required |
| detected-swact | Platform detects that the active controller has changed. | — |
| host-add | A new host is added to the system. | — |
| host-delete | A host is deleted from the system. | openstack_worker, personality |
| reinstall | Host reinstall action is performed. | configure_required |
| host-availability-updated | Host availability status changes. | availability |
| host-availability-saved | Host availability status is saved to the database (different from previous value). | availability |
| host-modify | Host configuration is modified (e.g. LVM provisioning, CPU frequency update). | — |
| host-label-assign | Host labels are assigned. | — |
| runtime-apply-puppet | Runtime puppet manifests are applied to a host. | — |
| system-modify | System parameters are modified via system modify. | delta_fields |
| storage-backend-modify | A storage backend (Ceph) is modified. | delta_fields |
| kube-upgrade-complete | Kubernetes upgrade is completed. | — |
| usm-upgrade-complete | USM platform upgrade is completed. | — |
| on-demand-reapply | Triggered via helm-override-update with --reapply flag, or via the evaluate_apps_reapply REST API. | — |
Notes:
- The additional fields listed in the table can be used as filter criteria in the metadata. For example, filtering "host-availability-updated" by "availability: available" ensures the reapply evaluation only runs when a host becomes available.
- Triggers with no additional fields can only be subscribed to by type, without filters.
- The "on-demand-reapply" trigger allows to explicitly request a reapply evaluation via CLI or REST API, independent of platform events.
- The full list of trigger constants is defined in sysinv/common/constants.py under "APP_EVALUATE_REAPPLY_TRIGGER_TO_METADATA_MAP".
class
Applications can make use of application classes. Classes allow the Application Framework to figure out in which order automatic operations, such as reapplies and updates, should be performed. Here are the classes available listed in priority order:
- Critical
- Storage
- Discovery
- Optional
- Reporting
One of those classes can be added to the application metadata as the following:
behavior: platform_managed_app: yes class: storage
For instance, app-rook-ceph belongs to the storage class: https://opendev.org/starlingx/app-rook-ceph/src/commit/63daf776fdfb4dca748fa6a205e7cc3947bb0b76/stx-rook-ceph-helm/stx-rook-ceph-helm/files/metadata.yaml#L33
Refer to this section for more details about application classes.
dependent_apps
Alternatively to application classes, developers can define direct application dependencies. When a new application is installed, all other required applications for it to work will be automatically installed as well.
This behavior is managed in the application's metadata.yaml through the dependent_apps key, where you can define a list of required applications and actions to be taken for each of these dependencies:
dependent_apps:
- name: <app_name>
version: <regex>
action: <apply|error|warn|ignore>
- name: <app_name>
version: <regex>
action: <apply|error|warn|ignore>
Details of each key:
- name: Application name
- version: The version is parsed as a regular expression. It is possible to do an exact math (25\.09-1) as well as allowing all versions of the application of a release to be valid (25\.09-\d+).
- action:
- apply
- Upload process: Once complete, the application status shows any missing dependent applications if they have not already been installed.
- Apply process: It will automatically upload and apply dependent apps. If all dependencies are installed successfully, the target app will proceed with its installation. If any dependency fails to install, the target app will also fail.
- error:
- Upload process: Once complete, the application status shows any missing dependent applications if they have not already been installed.
- Apply process: The target app's apply will fail if the dependent apps have not been installed first.
- warn:
- Upload process: Once complete, the application status shows any missing dependent applications if they have not already been installed.
- Apply process: The application will continue with the installation normally, but some services will not work correctly.
- ignore:
- Upload process: nothing happens.
- Apply process: nothing happens.
- apply
Example:
dependent_apps:
- name: node-feature-discovery
version: 25\.09-\d+
action: apply
Meaning that the app declaring this depends on any 25.09 versions of node-feature-discovery, which is set to be automatically applied.
Refer to this section for more details about application dependencies.
dependent_parent_exceptions
This option allows defining exceptions for scenarios where dependencies of a parent app need to be updated:
dependent_parent_exceptions
- name: <app_name>
version: <regular expression (full match)>
For instance, suppose that app A version 25.09-0 has an exact match dependency on app B version 25.09-0. If B version 25.09-1 is provided as an update that would cause the dependency chain to break, given that A depends on B version 25.09-0, instead of version 25.09-1. In order to allow the update to proceed in such a scenario, the following can be defined in the metadata for app B version 25.09-1:
dependent_parent_exceptions:
- name: A
version: 25\.09-0
app_lifecycle_actions
The intent is to allow apps to run custom code to do specific operation. One of such mechanisms are the lifecycle plugins. The interface between App Framework and apps is the app_lifecycle_actions function [1][2].
This code in [2] is where the transition from AppFramework to App lifecycle plugin happens.
lifecycle_op = self._helm.get_app_lifecycle_operator(app.name) lifecycle_op.app_lifecycle_actions(context, conductor_obj, self, app, hook_info)
For an app to implement its own lifecycle actions it needs to:
- create an entry point in setup.cfg with this format
systemconfig.app_lifecycle =
<app name> = <python module path>:<new class that extends AppLifecycleOperator>
- extend the abstract interface AppLifecycleOperator[3]
For example this will create an entry point https://opendev.org/starlingx/platform-armada-app/src/commit/33d690327f6d3d199e143d280b5e9c3b21f8834c/python3-k8sapp-platform/k8sapp_platform/setup.cfg#L41-L42 and this will create the lifecycle https://opendev.org/starlingx/platform-armada-app/src/commit/33d690327f6d3d199e143d280b5e9c3b21f8834c/python3-k8sapp-platform/k8sapp_platform/k8sapp_platform/lifecycle/lifecycle_platform.py#L25
If an app doesn't provide its own lifecycle actions it will inherit the default class[4].
As we can see the default extends the abstract interface[3] without changing anything.
Ideally each app concrete implementation of AppLifecycleOperator.app_lifecycle_actions takes care of all the resources, but in some cases we see that the apps override default functionality then calling super.app_lifecycle_actions to leverage default behavior implemented in [3]. See for example [5].
This has pros and cons. The pro is we can have common code that apps would do in place, the con is if some behavior needs to change it will affect all apps. The current common code between apps is placed in [6].
Going back to the transition from AppFramework to App lifecycle plugin happens, the intent is to assemble and pass necessary information between in an object called LifecycleHookInfo [7]. This is used like a bridge, so when populating/reading information in/to the object it is desired to use constans defined in LifecycleConstants [8].
The intent is for the app_lifecycle_hook to raise specific exceptions to inform the framework of some event, this will be described here: Lifecycle exceptions
LifecycleHookInfo
Searching for all occurrences of app_lifecycle_actions will reveal how the LifecycleHookInfo object is used. The gist would be something like this real example
hook_info = LifecycleHookInfo()
hook_info.init(constants.APP_LIFECYCLE_MODE_AUTO,
constants.APP_LIFECYCLE_TYPE_SEMANTIC_CHECK,
constants.APP_LIFECYCLE_TIMING_PRE,
constants.APP_UPDATE_OP)
hook_info[LifecycleConstants.EXTRA][LifecycleConstants.FROM_APP] = True
self.app_lifecycle_actions(context, app, hook_info)
Lists of lifecycle constants
Utility file
Exceptions
LifecycleHookInfo/mode
Possible values:
- APP_LIFECYCLE_MODE_MANUAL = 'manual'
- APP_LIFECYCLE_MODE_AUTO = 'auto'
APP_LIFECYCLE_MODE_MANUAL
Manual means an action triggered through CLI/REST API (application upload/apply/remove/delete/update + lock/unlock actions).
APP_LIFECYCLE_MODE_AUTO
Auto means an action triggered by the framework internally.
Example
LifecycleHookInfo/lifecycle_type
Possible values:
- APP_LIFECYCLE_TYPE_SEMANTIC_CHECK = 'check'
- APP_LIFECYCLE_TYPE_OPERATION = 'operation'
- APP_LIFECYCLE_TYPE_RBD = 'rbd'
- APP_LIFECYCLE_TYPE_RESOURCE = 'resource'
- APP_LIFECYCLE_TYPE_MANIFEST = 'manifest'
- APP_LIFECYCLE_TYPE_FLUXCD_REQUEST = 'fluxcd-request'
APP_LIFECYCLE_TYPE_SEMANTIC_CHECK
Before application operations + lock/unlock allow a semantic check to happen. An app can check specific conditions and reject the operation by raising Lifecycle exceptions
The purpose of this lifecycle is checking whether operations should proceed or not. Do not use it for long and costly resource changing operations.The APP_LIFECYCLE_TYPE_OPERATION or APP_LIFECYCLE_TYPE_RESOURCE can be used for those purposes.
APP_LIFECYCLE_TYPE_OPERATION
Should be used to execute mandatory custom actions before or after application operations such as apply or remove operations.
APP_LIFECYCLE_TYPE_RBD
Should be used to execute custom actions related to rados block devices.
APP_LIFECYCLE_TYPE_RESOURCE
Should be used to execute custom actions related to application resources.
APP_LIFECYCLE_TYPE_MANIFEST
Should be used to execute custom actions related to application manifests.
APP_LIFECYCLE_TYPE_FLUXCD_REQUEST
Should be used to respond to actions triggered by FluxCD when release operations are in progress.
LifecycleHookInfo/relative_timing
Possible values:
- APP_LIFECYCLE_TIMING_PRE = 'pre'
- APP_LIFECYCLE_TIMING_POST = 'post'
- APP_LIFECYCLE_TIMING_STATUS = 'status'
With a few exceptions (semantic check, and what else?) hooks are called around specific portions, meaning they are called before(pre) and after(post). Think of a hook before doing an app apply, and a hook after doing the app apply. In addition, there is a "status" hook that is called during application apply, which can be used to check if application resources are ready.
Examples
LifecycleHookInfo/operation
Possible values:
- APP_UPLOAD_OP = 'upload'
- APP_APPLY_OP = 'apply'
- APP_REMOVE_OP = 'remove'
- APP_DELETE_OP = 'delete'
- APP_UPDATE_OP = 'update'
- APP_DOWNGRADE_OP = 'downgrade'
- APP_ABORT_OP = 'abort'
- APP_EVALUATE_REAPPLY_OP = 'evaluate-reapply'
- APP_BACKUP = 'backup'
- APP_ETCD_BACKUP = 'etcd-backup'
- APP_RESTORE = 'restore'
- APP_LIFECYCLE_OPERATION_MTC_ACTION = 'mtc-action'
APP_UPLOAD_OP
Represents the application upload operation. This operation involves storing application information into the database. It makes applications available to be applied.
APP_APPLY_OP
Represents the application apply operation. This operation downloads the container images and performs the actual deployment of the application.
APP_REMOVE_OP
Represents the application removal operation. This operation uninstalls the application returning it to the uploaded state.
APP_DELETE_OP
Represents the application delete operation. This operation purges the application from the system deleting all its files.
APP_UPDATE_OP
Represents the application update operation. This operation updates an applied application.
APP_DOWNGRADE_OP
Represents the application downgrade operation. This operation downgrades an applied application.
APP_ABORT_OP
Represents the abort operation. This operation aborts any of the aforementioned operations.
APP_EVALUATE_REAPPLY_OP
Represents the abort reapply evaluation operation. This operation verifies if any application overrides changes. If so, an apply operation is scheduled for the next audit task.
APP_BACKUP
Represents the backup operation. This operation signals the execution of backup and restore procedures.
APP_ETCD_BACKUP
Represents the ETCD backup operation. This operation signals the execution of an ETCD backup.
APP_RESTORE
Represents the restore operation. This operation restores an app to its previous state upon another operational failure.
APP_LIFECYCLE_OPERATION_MTC_ACTION
Represents the maintenance operation. This operation signals maintenance actions. For instance, the ones performed during lock/unlock host operations.
LifecycleHookInfo/extra
This field was intended to pass information to and from the lifecycle hook itself.
Lifecycle exceptions
Searching for all occurrences of class Lifecycle.*Exception regex will reveal the existing. An in-depth explanation of why we have 3 types of LifecycleSemanticCheck exceptions is needed. At first glance I see that the intent is to surpass some logging or add special logging.
The intent is to throw LifecycleSemanticCheckException if you want to reject the operation (deny the operation to happen). Searching for all occurrences of 'APP_LIFECYCLE_TYPE_SEMANTIC_CHECK' will show what semantic checks(for what operations) will be called by the App Framework.
Helm Chart Overrides
Apps have overrides per namespace per chart. There are sysinv commands that can be used to query the overrides.
Here is an example query for platform-integ-apps.
[sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-list platform-integ-apps +--------------------+----------------------+ | chart name | overrides namespaces | +--------------------+----------------------+ | ceph-pools-audit | ['kube-system'] | | cephfs-provisioner | ['kube-system'] | | rbd-provisioner | ['kube-system'] | +--------------------+----------------------+ [sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-show platform-integ-apps rbd-provisioner kube-system +--------------------+------------------------------------------------------+ | Property | Value | +--------------------+------------------------------------------------------+ | attributes | enabled: true | | | | | combined_overrides | classdefaults: | EDITED | name | rbd-provisioner | | namespace | kube-system | | system_overrides | classdefaults: | EDITED | user_overrides | None | +--------------------+------------------------------------------------------+
You can define behaviors by implementing BaseHelm class ([4]). An example of such implementation is RbdProvisionerHelm for platform-integ-armada app ([6] + [7]).
While inspecting the code we observe [1] called in [2], [2] called in [3] and [3] is further used throughout some apps and sysinv.
Guidelines
We strongly encourage you to enable the auto update feature unless there are special extra steps needed for an application update to happen. See `updates/auto_update` above.
We strongly encourage you to enable maintaining user overrides feature unless there are special extra steps needed for an application update to happen. See `maintain_user_overrides` above.
Rationale for updates and maintaining user overrides would be:
Are there any steps required before `system application-update`when applying a patch on a live system? Are there any steps required before `system application-update` at `system upgrade-activate` time? Was there an override format change that requires a transformation between version N and N+1 of the format? Can I update helm-charts to allow both formats so that a transformation can be skipped?
We strongly encourage you to enable maintaining helm-charts disabled over application updates. See `maintain_attributes` above.
If your application is bundled with the StarlingX ISO, we also recommend configuring it to be platform managed. See `behavior` above.
Resource Accounting Caveat
Most platform applications should be affined to the platform cores, but all other containerized workloads would use the application or application-isolated cores. For legacy applications this is controlled by namespace, and is managed by a customization to kubelet. Going forward (as of June 2023) we are modifying the system to use the "app.starlingx.io/component=platform" label on the application pod or namespace to signify that it should be run on the platform cores.
We also need to ensure the resources consumed by platform pods (cpu/memory requests) are not counted against the application node resources since they should be accounted against the platform resources. This includes both the Pod requests and the platform accounting [2]. In particular, containers running on platform cores must not request cpu resources from Kubernetes. Memory resources are less of a concern as we usually are not memory-constrained the way that we are CPU-constrained.
If your application should run on platform cores, it is important to minimize the amount of CPU time it uses. We have a limited amount of platform CPU and it needs to be carefully managed.
[2] Resource accounting: monitoring/collectd-extensions/src/plugin_common.py (K8S_NAMESPACE_SYSTEM)
Parameterizing the Component Label
All platform services that are affined to the platform cores use the component label. To allow for the application code to optionally run on application cores, the identification of infrastructure services must be updated to use a Kubernetes resource label to properly identify which namespaces (or specific pods) should be classified as a platform service.
Note: If the application uses the "kube-system" namespace, then the application doesn't need to do anything. That namespace will be handled as a special case.
Adding the Label to an Application
For an example of how to configure the app.starlingx.io/component label, see: https://review.opendev.org/c/starlingx/app-node-feature-discovery/+/946957
# values.yaml
component: <platform|application>
# Helm chart label logic to ensure a valid value for resource labels
app.starlingx.io/component: {{ ternary "application" "platform" (eq .Values.component "application")}}
Considerations About Classes and Direct Dependencies
- Automatic operations such as application updates are performed for each class in the order stated here.
- Apps within each class are updated in parallel.
- Applications that declare direct dependencies are updated after all critical apps.
- Direct dependencies have priority over application classes in case both are present in the metadata.
- Only platform managed apps are classified.
- If the dependency version rule matches more than one tarball, then the most updated one will be chosen to be automatically uploaded and applied.
Considerations About Application Updates
Manual Updates
Apps can be manually updated by running the following command:
system application-update path_to_tarball.tgz
Automatic Updates
Automatic updates may take place in two different scenarios:
- During the activation phase of a platform upgrade.
- After applying a patch containing new application versions.
The following will happen during automatic updates:
- If the app is in uploaded state, an application delete will happen for the N version, followed by a application upload of the N+1 version.
- If the app is in applied state, an application update will be performed applying the N+1 version.
Testing Automatic Application Updates
An auto update can be tested by following these steps:
- Build an application tarball.
- Copy the tarball to "/usr/local/share/applications/helm/".
- Add a record to the kube_app_bundle table in the sysinv database corresponding to the new application version.
- This can be done by simply running "sudo touch /ostree/lock && sudo rm /ostree/lock". That command simulates an ostree update, which in turn, triggers a database update.
- Wait for the application audit task to run and the framework will automatically start updating the app.
Considerations About Applications on Subclouds
Subclouds are commonly configured to pull application images from the central cloud registry using the registry.central address. In that scenario, application images will not be pulled directly from a public/external registry. Therefore, when deploying subclouds make sure that the central cloud registry is populated with all required images. That can be achieved by following one of these strategies:
- Prestaging application images into the central cloud registry before deploying the subclouds.
- Applying in the central cloud all the applications that are required for the subclouds.
Please refer to the Distributed Cloud documentation for detailed steps.
Related work
We continue with a wiki targeting people involved in App Framework area. This next page is not currently designed to be facing App Developers themselves, and probably never will: App Framework page
