Audit Logging in Loft

Loft auditing provides a security-relevant, chronological set of records documenting the sequence of actions in loft. Loft audits the activities generated by users and by applications that use the Loft API.

Loft is able to log activities related to:

• Management Instance changes, such as creation of new virtual clusters, spaces etc.
• Changes within a virtual cluster or space
• Changes within a connected cluster

Auditing in Loft is very similar to auditing Kubernetes clusters in general.

Enable Auditing

Loft auditing is configured through the Loft config in the Loft UI (Admin -> Config).

An example configuration could look like:

audit:
  enabled: true
  level: 1

Loft Restart Required

Changing the Loft auditing configuration requires a restart to take effect. You can restart Loft either through the Loft UI or via kubectl: kubectl rollout restart deploy/loft -n loft

Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records.

Each request can be recorded with an associated stage. The defined stages are:

• RequestReceived - The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.
• ResponseComplete - The response body has been completed and no more bytes will be sent.
• Panic - Events generated when a panic occurred.

info

The audit logging feature increases the memory consumption of Loft because some context required for auditing is stored for each request. Memory consumption depends on the audit logging configuration.

Audit Levels

For easier configuration, Loft provides audit levels, that are preconfigured audit policies for the most common use cases. These levels range from 1 to 4 where 1 logs the fewest requests, while 4 logs the most. A more detailed description can be found below:

• Level 1: will log modifying requests such as creation / modification or deletion of any objects
• Level 2: like Level 1 but will also log the metadata of reading requests, such as listing pods inside a virtual cluster or space. It won't log the response or request payload and instead only the metadata such as request origin, target etc.
• Level 3: like Level 2 but instead of only logging the request metadata will also log the complete request payload sent to Loft
• Level 4: like Level 3 but instead of only logging metadata and request payload, will also log the response Loft has sent to the requester

info

For all of these levels certain internal read-only apis are not logged since those might pollute the log and drastically increase log size. If you also want to log these, please create a custom audit policy as described below.

You can configure the audit level through the Loft config, that can be modified either through the Loft UI or helm:

UI

1. Go to the Admin > Config view using the menu on the left.
2. In the input field that appears, enter the following config:

   audit:
     enabled: true
     level: 1
3. Click on the button and wait until Loft has restarted.

Helm

Create a new file values.yaml:

config:
  audit:
    enabled: true
    level: 1

Then apply these values with helm (make sure to specify the correct Loft version):

helm upgrade loft loft --namespace loft \
                       --repo https://charts.loft.sh \
                       --version $LOFT_VERSION \
                       --reuse-values \
                       --values values.yaml

Optional: Audit Policy

danger

It is recommended to use audit levels instead of audit policy directly, because a policy is much more complex to define.

As an alternative to Audit levels, policy allows you to define exact rules about what events should be recorded and what data they should include. When an event is processed, it's compared against the list of rules in order. The first matching rule sets the audit granularity of the event. The defined audit granularity options are:

• None - don't log events that match this rule.
• Metadata - log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
• Request - log event metadata and request body but not response body. This does not apply for non-resource requests.
• RequestResponse - log event metadata, request and response bodies. This does not apply for non-resource requests.

An example policy that catches all requests would look like this:

audit:
  enabled: true
  policy:
    rules:
    - level: Metadata
      omitStages:
      - RequestReceived

Below you can find a complete policy reference:

policy required object

The audit policy to use and log requests. By default loft will not log anything

rules required object[]

Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered.

level required string

The Level that requests matching this rule are recorded at.

users required string[]

The users (by authenticated user name) this rule applies to. An empty list implies every user.

userGroups required string[]

The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group.

verbs required string[]

The verbs that match this rule. An empty list implies every verb.

resources required object[]

Resources that this rule matches. An empty list implies all kinds in all API groups.

####### group required string {#status-audit-policy-rules-resources-group}

Group is the name of the API group that contains the resources. The empty string represents the core API group.

####### resources required string[] {#status-audit-policy-rules-resources-resources}

Resources is a list of resources this rule applies to.

For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '' matches all resources and their subresources. 'pods/' matches all subresources of pods. '*/scale' matches all scale subresources.

If wildcard is present, the validation rule will ensure resources do not overlap with each other.

An empty list implies all resources and subresources in this API groups apply.

####### resourceNames required string[] {#status-audit-policy-rules-resources-resourceNames}

ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched.

namespaces required string[]

Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace.

nonResourceURLs required string[]

NonResourceURLs is a set of URL paths that should be audited. s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz" - Log all health checks

omitStages required string[]

OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply.

requestTargets required string[]

RequestTargets is a list of request targets for which events are created. An empty list implies every request.

clusters required string[]

Clusters that this rule matches. Only applies to cluster requests. If this is set, no events for non cluster requests will be created. An empty list means no restrictions will apply.

omitStages required string[]

OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted.

Persisting Audit Logs

There are 2 ways how to persist Loft audit logs. Either you can deploy Loft with a persistent volume claim or let Loft connect to a persistent database. The PVC approach does not work for HA mode for Loft.

Deploy Loft with a PVC to save Audit Logs

Create a new values.yaml with the following values:

audit:
  persistence:
    enabled: true
    # size: 30Gi

Then apply the values via helm:

helm upgrade loft -n loft --version $VERSION \
                          --repo https://charts.loft.sh \
                          --reuse-values \
                          -f values.yaml

Use a persistent database as Loft audit backend

Go to Admin > Config and specify the following Loft config setting:

audit:
  dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Then press Apply and wait until Loft is restarted.

Viewing and Exporting Audit Logs

By default, Loft will log audit events to the following locations:

• To a log file in json format located at /var/log/loft/audit.log inside the Loft container. Each line inside the log represents a single audit event.
• To an internal sqlite storage located at /var/log/loft/audit.db inside the Loft container. This sqlite database is used to display audit log events in the Loft UI. By default audit events in the sqlite are not persisted, so restarting Loft will clear the database. Instead of using a sqlite database, Loft is also able to write those events to a persistent mysql database that can be configured through the Loft config. E.g.:

audit:
  enabled: true
  dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Enable Audit SideCar

To easily export the audit events to third party systems, we recommend to enable the audit log sidecar that will print all the audit events onto stdout in a separate container which then can be easily watched and exported. Enabling the sidecar is only possible through helm values.

Create a values.yaml with the following contents:

audit:
  enableSideCar: true

danger

You cannot configure this under Admin > Config, since this requires a change in the Loft deployment itself, which is why this is a helm option only

Then update the helm release via:

helm upgrade loft loft --namespace loft \
                       --repo https://charts.loft.sh \
                       --version $LOFT_VERSION \
                       --reuse-values \
                       --values values.yaml

Wait until Loft has restarted, then you can view the audit logs via:

kubectl logs -n loft -l app=loft -c audit -f

Audit Config Reference

audit required object

Audit holds audit configuration

enabled required boolean false

If audit is enabled and incoming api requests will be logged based on the supplied policy.

disableAgentSyncBack required boolean false

If true, the agent will not send back any audit logs to Loft itself.

level required integer

Level is an optional log level for audit logs. Cannot be used together with policy

policy required object

The audit policy to use and log requests. By default loft will not log anything

rules required object[]

Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered.

level required string

The Level that requests matching this rule are recorded at.

users required string[]

The users (by authenticated user name) this rule applies to. An empty list implies every user.

userGroups required string[]

The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group.

verbs required string[]

The verbs that match this rule. An empty list implies every verb.

resources required object[]

Resources that this rule matches. An empty list implies all kinds in all API groups.

####### group required string {#status-audit-policy-rules-resources-group}

Group is the name of the API group that contains the resources. The empty string represents the core API group.

####### resources required string[] {#status-audit-policy-rules-resources-resources}

Resources is a list of resources this rule applies to.

For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '' matches all resources and their subresources. 'pods/' matches all subresources of pods. '*/scale' matches all scale subresources.

If wildcard is present, the validation rule will ensure resources do not overlap with each other.

An empty list implies all resources and subresources in this API groups apply.

####### resourceNames required string[] {#status-audit-policy-rules-resources-resourceNames}

ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched.

namespaces required string[]

Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace.

nonResourceURLs required string[]

NonResourceURLs is a set of URL paths that should be audited. s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz" - Log all health checks

omitStages required string[]

OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply.

requestTargets required string[]

RequestTargets is a list of request targets for which events are created. An empty list implies every request.

clusters required string[]

Clusters that this rule matches. Only applies to cluster requests. If this is set, no events for non cluster requests will be created. An empty list means no restrictions will apply.

omitStages required string[]

OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted.

dataStoreEndpoint required string

DataStoreEndpoint is an endpoint to store events in.

dataStoreTTL required integer

DataStoreMaxAge is the maximum number of hours to retain old log events in the datastore

path required string

The path where to save the audit log files. This is required if audit is enabled. Backup log files will be retained in the same directory.

maxAge required integer

MaxAge is the maximum number of days to retain old log files based on the timestamp encoded in their filename. Note that a day is defined as 24 hours and may not exactly correspond to calendar days due to daylight savings, leap seconds, etc. The default is not to remove old log files based on age.

maxBackups required integer

MaxBackups is the maximum number of old log files to retain. The default is to retain all old log files (though MaxAge may still cause them to get deleted.)

maxSize required integer

MaxSize is the maximum size in megabytes of the log file before it gets rotated. It defaults to 100 megabytes.

compress required boolean false

Compress determines if the rotated log files should be compressed using gzip. The default is not to perform compression.

How does Audit Logging work for Direct Cluster Endpoints?

If the direct cluster endpoint feature is enabled, Loft audit configuration is synced to each agent and each Loft agent will propagate audit events that it receives back to the central Loft instance, which then logs it as a regular audit event. Such "propagated" events can be identified through the annotations.audit.loft.sh/sent-by-agent identifier in an audit event.

Disable Agent Sync Back

You can disable event sync back from the agent to the central loft instance via the audit config option disableAgentSyncBack.