StarlingX/Containers/Applications/app-audit
Contents
Application: audit-armada-app
Source
Building
From the Debian Build environment:
build-pkgs -c -p audit-helm
The command above produces the package "stx-audit-helm_<version>_<arch>.deb" that contains the tarball "auditd-<version>.tgz"
Testing
Enable Auditd in the Kernel
The Linux Auditing System is disabled in the StarlingX kernel by default.
To enable auditd in the kernel of all hosts in the system, set the system service parameter audit to ‘1’ and apply the service-parameter change, using the following commands, executed on the active controller.
~(keystone_admin)]$ system service-parameter-modify platform kernel audit=1 ~(keystone_admin)]$ system service-parameter-apply platform
To persist the service parameter change, all hosts need to be locked and unlocked, using the following commands for each host depending on the deployed configuration:
For AIO-SX deployments:
~(keystone_admin)$ system host-lock controller-0 ~(keystone_admin)$ system host-unlock controller-0
For AIO-DX and Standards deployments, after controller-1 is locked/unlocked swact controller-0 to make controller-1 the active node. The next set of commands are executed on controller-0 node:
~(keystone_admin)$ system host-lock controller-1 ~(keystone_admin)$ system host-unlock controller-1 ~(keystone_admin)$ system host-swact controller-0
On controller-1, after controller-0 is locked/unlocked swact controller-1 to go back to controller-0 as the active node. The next set of commands are executed on controller-1 node:
~(keystone_admin)$ system host-lock controller-0 ~(keystone_admin)$ system host-unlock controller-0 ~(keystone_admin)$ system host-swact controller-1
For each worker node in the configuration execute the commands from controller-0:
~(keystone_admin)$ system host-lock worker-0 ~(keystone_admin)$ system host-unlock worker-0
To verify if the grub kernel parameter audit was updated to ‘1’, for a particular host, ssh to the host, and check the cmdline parameters, for example:
~(keystone_admin)]$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.10.57-200.185.tis.el7.x86_64 root=UUID=e11d78a2-7e1c-4613-84c7-002647b1cf8d ro security_profile=standard module_blacklist=integrity,ima tboot=false crashkernel=512M biosdevname=0 console=ttyS0,115200 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 softdog.soft_panic=1 intel_iommu=on user_namespace.enable=1 nopti nospectre_v2 nospectre_v1 hugepagesz=2M hugepages=0 default_hugepagesz=2M irqaffinity=2-3 rcu_nocbs=2-3 kthread_cpus=0-1 audit=1 audit_backlog_limit=8192
Start Auditd System Application
Prerequisites
Set the audit grub kernel parameter to ‘1’.
The auditd container that runs the auditd daemon must be started by uploading and applying the audit-armada-app.
The auditd system application is installed as part of the software install or upgrade.
The auditd system application tarball can be found after installation in the /usr/local/share/applications/helm directory. The name of the tarball is auditd-<version>.tgz, for example, auditd-1.0-2.tgz.
Use the following commands to upload and apply the auditd system application:
~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/auditd-1.0-2.tgz # check the app was uploaded
~(keystone_admin)]$ system application-show auditd # if status is "uploaded" proceed with app apply
~(keystone_admin)]$ system application-apply auditd # check the app was applied
~(keystone_admin)]$ system application-show auditd # if successful, status will be "applied"
To check that auditd container/pod is created and running on each master and worker node, use the following command:
~(keystone_admin)]$ kubectl get pods -n kube-system -o wide | grep auditd ns-auditd-9hgq5 1/1 Running 0 2m46s face::e95d:7b0:368d:55f8 compute-0 <none> <none> ns-auditd-btww5 1/1 Running 1 2m46s face::2d8f:b75d:d511:81ef compute-1 <none> <none> ns-auditd-czsdf 1/1 Running 1 2m46s face::977:4894:111d:5bf0 compute-2 <none> <none> ns-auditd-hs62t 1/1 Running 0 2m46s face::3 controller-1 <none> <none> ns-auditd-nn8jw 1/1 Running 0 2m46s face::2 controller-0 <none> <none>
Auditd Configuration Overrides
The Auditd daemon specific configuration is available in the /etc/audit/auditd.conf file. For more information, see, https://man7.org/linux/man-pages/man5/auditd.conf.5.html.
Besides the auditd main configuration file auditd.conf, auditd uses audit rules configuration that is available in the /etc/audit/audit.rules file which defines what audit events are logged. For more information on how audit rules are configured, see https://linux.die.net/man/7/audit.rules.
In the StarlingX Platform containerized auditd solution, both configuration files have default settings that can be overwritten using Helm chart overrides.
The Helm chart overrides are applied using the following command:
~(keystone_admin)]$ system helm-override-update auditd auditd kube-system --reuse-values --values /home/sysadmin/<user_specific_config>.yaml
The <user_specific_config>.yaml defines the overrides that will apply either to the auditd.conf and/or to the audit.rules files.
* Note The default values for auditd.conf should be sufficient and you do not need to update them.
In rare cases, the following example describes how to update the default value with the desired value.
Example of user defined overrides file for auditd.conf:
auditdconf: |- ########################################################################## # # auditd.conf # ########################################################################## local_events = yes write_logs = yes log_file = /var/log/audit/audit.log log_group = root log_format = RAW flush = INCREMENTAL_ASYNC freq = 50 max_log_file = 8 num_logs = **8** priority_boost = 4 disp_qos = lossy dispatcher = /sbin/audispd name_format = NONE ##name = mydomain max_log_file_action = IGNORE space_left = 75 space_left_action = SYSLOG ##verify_email = yes ##action_mail_acct = root admin_space_left = 50 admin_space_left_action = SYSLOG disk_full_action = SYSLOG disk_error_action = SYSLOG use_libwrap = yes ##tcp_listen_port = 60 ##tcp_listen_queue = 5 ##tcp_max_per_addr = 1 ##tcp_client_ports = 1024-65535 ##tcp_client_max_idle = 0 enable_krb5 = no krb5_principal = auditd distribute_network = no
Example of user defined overrides file for audit.rules:
auditdrules: |- ## First rule - delete all -D ## Increase the buffers to survive stress events. ## Make this bigger for busy systems -b 8192 ## Set failure mode to syslog -f 1 -a always,exit -F arch=b32 -S init_module,finit_module,delete_module -F key=modules -a always,exit -F arch=b64 -S init_module,finit_module,delete_module -F key=modules
* Note The log rotation configuration in auditd.conf file must not be updated, and must use the default value, max_log_file_action = IGNORE, since the logrotate linux utility is used to manage auditd log rotation.
Apply the audit rules overrides using the following command:
~(keystone_admin)]$ system application-apply auditd Check that application apply has completed successfully:
~(keystone_admin)]$ system application-show auditd The Helm chart overrides system helm-override-update command, automatically applies the additional rules from the user provided yaml file to the audit.rules in the auditd container.
Similarly, configuration overrides can be applied to update the default configuration of auditd.conf using the system helm-override-update command.
Auditd logs
auditd logs can be viewed on the host in the /var/log/audit directory. Logs are generated by the auditd daemon running in the container and the logs record auditable events configured using the audit.rules file. Log rotation is automatically configured by the system.
To verify log rotate is working properly, the user should wait at least 24 hours and verify if the a new audit log file was created and if there is a old audit log file in the same directory, it should look like bellow:
root@controller-0:/var/log/audit# ls audit.log audit.log-<date>.gz
Disable Auditd
You may decide to disable auditd for performance reasons. First, you must remove the auditd application. Then, you must set the kernel service parameter audit to ‘0’. These steps removes the auditd containers on all hosts and the auditd application.
Use the following system commands to disable auditd:
To remove the auditd application:
~(keystone_admin)]$ system application-remove auditd ~(keystone_admin)]$ system application-delete auditd
To verify that the application does not exist in the system:
~(keystone_admin)]$ system application-list |grep auditd
To set the kernel service parameter audit to ‘0’:
~(keystone_admin)]$ system service-parameter-modify platform kernel audit=0 ~(keystone_admin)]$ system service-parameter-apply platform
To persist the kernel parameter change, all hosts need to be locked and unlocked:
~(keystone_admin)]$ system host-lock controller-0 ~(keystone_admin)]$ system host-unlock controller-0
