# Kubernetes Audit Policy: What to Log, at What Level

> A Kubernetes audit policy decides which API requests kube-apiserver records and how much of each it keeps. The first matching rule wins, so one misplaced rule can write Secret values into your logs. Here is a six-rule policy we ran against kube-apiserver v1.34.1, what each rule captured, and the jq queries that turn the log into answers.

- URL: https://firewallsync.com/posts/kubernetes-audit-policy-what-to-log-and-at-what-level/
- Site: FirewallSync (https://firewallsync.com/)
- Topic: Cloud Security
- Tags: kubernetes, audit logging, incident response, detection
- Published: 2026-10-25
- Author: Craig Leslie

## Key takeaways

- kube-apiserver logs nothing unless it is started with --audit-policy-file, and a policy with zero rules is rejected. Each request is matched against the rules in order, and the first match sets its level: None, Metadata, Request or RequestResponse.
- Log Secrets, ConfigMaps, TokenRequests and TokenReviews at Metadata only. In our test, putting a broad Request rule for writes above that rule wrote a new Secret's base64 value straight into the audit log.
- Metadata is enough to answer most incident questions: who (user, impersonated user, source IPs), what (verb, resource, name, namespace), and whether it was allowed (response code and the authorization.k8s.io/decision annotation).
- Log RBAC and admission-control changes at RequestResponse so you can see exactly which subjects and roles a binding granted. For exec, filter on the subresource, not the verb: one kubectl exec produced both get and create events in our test, and the command is in requestURI.
- On managed clusters the provider controls the policy and audit export is often off by default. On EKS the audit log type is off until you enable it; on AKS, kube-audit-admin leaves out get and list events.



A Kubernetes audit policy is the file that tells kube-apiserver which API requests to record and how much of each to keep. Without one, upstream kube-apiserver records nothing. A good policy logs everything at `Metadata` at minimum, keeps request bodies for changes that alter access, and never logs the contents of Secrets. Rule order is what makes that work: the first matching rule wins, and a broad rule placed too high will quietly write Secret values into your logs. We demonstrate that below.

This guide gives a six-rule policy we ran against kube-apiserver v1.34.1, shows what each rule captured, and ends with queries you can run against the log during an incident.

## How kube-apiserver evaluates a policy

Every request produces audit events at one or more **stages**:

| Stage | When |
|---|---|
| `RequestReceived` | As soon as the audit handler receives the request |
| `ResponseStarted` | After response headers are sent, for long-running requests such as watch |
| `ResponseComplete` | When the response body is finished |
| `Panic` | When a panic occurred |

The policy is a list of rules. kube-apiserver compares each event with the rules **in order**, and the first rule that matches sets the event's **level**:

| Level | What gets recorded |
|---|---|
| `None` | Nothing |
| `Metadata` | User, groups, source IPs, user agent, verb, resource, namespace, name, response status, timestamps, annotations |
| `Request` | Metadata plus the request body |
| `RequestResponse` | Metadata plus request and response bodies |

`Request` and `RequestResponse` don't apply to non-resource URLs such as `/healthz`. A rule can match on `users`, `userGroups`, `verbs`, `resources` (group, resource, subresource and `resourceNames`), `namespaces` and `nonResourceURLs`. A rule with no match fields matches everything, which is what you want for the last rule and exactly what you don't want near the top.

The policy file is passed with `--audit-policy-file`. If the flag is omitted, no events are logged, and a policy with no rules is rejected.

## A policy that answers incident questions

The policy below is built around the questions you get asked during an incident: who read this Secret, who exec'd into that Pod, who granted this binding, and was it allowed. It is a starting point, not a complete production policy; see the limits at the end.

```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # 1. Drop health and version probes.
  - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
  # 2. Sensitive payloads: who touched them, never the contents.
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps", "serviceaccounts/token"]
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
  # 3. Interactive access into running workloads.
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward", "pods/proxy", "nodes/proxy"]
  # 4. Changes to who can do what, and to admission control.
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
    resources:
      - group: "rbac.authorization.k8s.io"
      - group: "admissionregistration.k8s.io"
  # 5. Other writes: keep the submitted object.
  - level: Request
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
  # 6. Everything else (reads, watches, non-resource URLs).
  - level: Metadata
```

Why each rule is where it is:

| Rule | Level | Reason |
|---|---|---|
| 1. Probes | None | High volume, no investigative value. Leaving out `RequestReceived` globally also halves events for most requests, since `ResponseComplete` carries the same metadata plus the result. |
| 2. Secrets, ConfigMaps, TokenRequest, TokenReview | Metadata | Their bodies hold credentials and tokens. The upstream GCE policy treats them the same way, noting they "can contain sensitive & binary data". It must come before any rule that logs bodies. |
| 3. exec, attach, port-forward, proxy | Metadata | These are how people get inside workloads. The body adds nothing, but you always want the event. |
| 4. RBAC and admission configuration | RequestResponse | You need the exact subjects, `roleRef`, rules and webhook targets that were written, and the response confirms what was stored. Listing a group with no `resources` matches every resource in it. |
| 5. Other writes | Request | Keeps the object that was submitted, such as a Pod spec with a privileged container or a hostPath mount. |
| 6. Catch-all | Metadata | Every read and watch is still attributable to someone. |

Pod Security Admission and our [RBAC permission audit](/posts/kubernetes-rbac-audit-11-permissions-that-escalate-access/) tell you what *could* happen. The audit log is how you find out what did.

## What it recorded

We ran the policy on kube-apiserver v1.34.1 (the envtest build from the kubernetes-sigs/controller-tools releases) with etcd and RBAC authorization, then made requests as a `system:masters` user and as an unprivileged user `alice`. There were no nodes, so no Pod actually ran. Results:

| Action | Level logged | What the event held |
|---|---|---|
| `kubectl create secret generic db-creds --from-literal=password=...` | Metadata | User, namespace, name, `201`. No request body; the value appeared nowhere in the log. |
| `kubectl create rolebinding alice-view --clusterrole=view --user=alice` | RequestResponse | Full `requestObject` and `responseObject`, including the subject `alice` and `roleRef` `view`. |
| `alice` lists Secrets in `kube-system` | Metadata | `403`, annotation `authorization.k8s.io/decision: forbid` |
| Admin runs `kubectl --as alice get pods` | Metadata | `user.username: admin` and `impersonatedUser.username: alice` in the same event |
| `kubectl create deployment web --image=nginx:1.27` | Request | `requestObject` with the Deployment spec, no response body |
| `kubectl create token ci-bot` | Metadata | Resource `serviceaccounts`, subresource `token`, name `ci-bot`, no token in the log |
| `kubectl exec shell -- cat /etc/shadow` | Metadata | `requestURI` of `/api/v1/namespaces/default/pods/shell/exec?command=cat&command=%2Fetc%2Fshadow&...` |

Two details from the test are worth building detections around:

- **Exec shows up under more than one verb.** That single `kubectl exec` produced events with verb `get` and verb `create` for the `exec` subresource. Filter on `objectRef.subresource == "exec"`, not on the verb.
- **Impersonation keeps both identities.** The authenticated user is in `user` and the identity they acted as is in `impersonatedUser`. If you only alert on `user.username`, impersonated actions look like they came from the admin's own account.

Also keep in mind what the API reference warns about two fields: `userAgent` is reported by the client "and must not be trusted", and in `sourceIPs` all entries except the last "can be arbitrarily set by the client".

## The rule-order mistake that leaks Secrets

We then moved rule 5 (Request for all writes) above rule 2, restarted kube-apiserver with that policy, and created a Secret with the value `S3cretValue-456`. The audit log now contained:

```text
"data":{"key":"UzNjcmV0VmFsdWUtNDU2"}
```

That is the base64 encoding of `S3cretValue-456`. The broad write rule matched first, logged the request body at `Request` level, and the Metadata rule for Secrets was never reached. With the correct order, the equivalent test (a Secret holding `S3cretValue-123`) left no trace of either the plain or the encoded value in the log.

This is the most common way an audit log becomes a credential store. Your audit pipeline usually has wider read access than the cluster's Secrets do, so treat a body-logging rule above your Secrets rule as a bug. Check it in code review with a simple test: create a dummy Secret in a test cluster and search the log for its encoded value.

If you already store [Secrets encrypted at rest](/posts/kubernetes-secrets-encryption-at-rest-what-it-protects/), remember that encryption protects etcd, not the audit log. And a [Secret-based service account token](/posts/kubernetes-service-account-tokens-bound-vs-legacy-secrets/) leaked this way never expires.

## Backends and volume

There are two backends: a log file and a webhook.

- **Log backend:** `--audit-log-path` sets the file (`-` means stdout). Rotation is controlled by `--audit-log-maxage` (days), `--audit-log-maxbackup` (files) and `--audit-log-maxsize` (megabytes). Batching is off by default for the log backend.
- **Webhook backend:** `--audit-webhook-config-file` points at a kubeconfig-format file for the remote endpoint. Batching is on by default, with a maximum of 400 events per batch and a 30-second maximum wait.
- **Truncation** of oversized events is off by default for both backends and has to be enabled explicitly.

The Kubernetes docs also note that auditing increases API server memory use, because some context is kept for each request, and that the increase depends on the policy. `RequestResponse` on a busy resource is where cost grows: the GCE reference policy, for instance, logs `get`, `list` and `watch` at `Request` level rather than `RequestResponse` because "Get responses can be large".

Ship the log off the control plane. An attacker with control-plane access can edit a local file, and an [incident log is only useful](/posts/logging-for-incidents-not-for-dashboards/) if it survives the incident.

## Managed Kubernetes

On EKS, AKS and similar services the provider runs kube-apiserver, so you generally choose whether audit events are exported, not what the policy contains. Check your provider's documentation for what its policy logs.

- **Amazon EKS:** control plane logs are not sent to CloudWatch Logs by default. Enable the `audit` log type per cluster, for example with `aws eks update-cluster-config --logging`. The audit stream is named `kube-apiserver-audit-<id>` in the `/aws/eks/<cluster>/cluster` log group. AWS notes delivery is best effort and standard CloudWatch Logs charges apply.
- **Azure AKS:** resource logs are not collected until you create a diagnostic setting. Microsoft's guidance warns that `kube-audit` can be costly and suggests `kube-audit-admin`, which leaves out `get` and `list` events. That saves money but drops exactly the read events, such as Secret reads, that matter in a credential-theft investigation. Decide that trade-off deliberately.

## Queries to keep ready

These run against a JSON-lines audit log with `jq`. We ran each one against the log from the test above.

```bash
# Who read or changed Secrets (excluding the API server itself)
jq -c 'select(.objectRef.resource=="secrets" and .user.username!="system:apiserver")
  | {ts:.requestReceivedTimestamp, user:.user.username, verb, ns:.objectRef.namespace,
     name:.objectRef.name, code:.responseStatus.code}' audit.log

# Every exec into a Pod, with the command line
jq -c 'select(.objectRef.subresource=="exec")
  | {ts:.requestReceivedTimestamp, user:.user.username, ns:.objectRef.namespace,
     pod:.objectRef.name, uri:.requestURI}' audit.log

# Denied requests, including impersonated ones
jq -c 'select(.annotations["authorization.k8s.io/decision"]=="forbid")
  | {user:.user.username, as:.impersonatedUser.username, verb,
     res:.objectRef.resource, ns:.objectRef.namespace}' audit.log

# RBAC writes by people rather than the control plane, with what they granted
jq -c 'select(.objectRef.apiGroup=="rbac.authorization.k8s.io"
       and (.verb|test("create|update|patch|delete"))
       and (.user.username|startswith("system:")|not))
  | {user:.user.username, verb, res:.objectRef.resource, name:.objectRef.name,
     subjects:.requestObject.subjects, role:.requestObject.roleRef.name}' audit.log
```

A burst of `forbid` events from one identity across many resources can be a sign that someone is enumerating what a stolen credential can do.

## Limits of this article

We tested on a single kube-apiserver v1.34.1 with no nodes, no controller manager and static token authentication, so no workload actually ran and no kubelet or controller traffic was present. A real cluster produces far more events, and you will want extra `None` or `Metadata` rules for high-volume system traffic (the GCE policy is a good reference). The policy does not cover custom resources that hold credentials; add those to rule 2. `Request` level on Pod writes will log any credentials placed directly in environment variables. We did not test the webhook backend or any managed service.

## Sources and verification

| Important claim | Source | Status |
|---|---|---|
| Stages, levels, first-match evaluation, no logging without `--audit-policy-file`, zero-rule policy rejected, log and webhook flags, batching defaults, truncation off by default, memory note | [Kubernetes docs: Auditing](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/) ([source](https://github.com/kubernetes/website/blob/main/content/en/docs/tasks/debug/debug-cluster/audit.md)) | Verified |
| Policy fields, `sourceIPs` and `userAgent` trust warnings, empty `resources` matches whole group | [audit.k8s.io/v1 types](https://github.com/kubernetes/apiserver/blob/master/pkg/apis/audit/v1/types.go) | Verified |
| Secrets, ConfigMaps, TokenRequest and TokenReviews logged at Metadata; "Get responses can be large" | [GCE configure-helper.sh audit policy](https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh) | Verified |
| `authorization.k8s.io/decision` annotation | [kube-apiserver authorization filter](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/filters/authorization.go) | Verified |
| Test results: Metadata Secret events without values, RequestResponse RBAC bodies, forbid annotation, impersonatedUser, exec URI and verbs, Secret value leaked with wrong rule order | Our test on kube-apiserver v1.34.1 (envtest build) | Verified (single test environment) |
| EKS audit log off by default, stream naming, best-effort delivery, CloudWatch charges | [Amazon EKS: Send control plane logs to CloudWatch Logs](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html) | Verified |
| AKS `kube-audit-admin` excludes get and list; logs need a diagnostic setting | [Microsoft: Monitor the AKS API server](https://docs.azure.cn/en-us/aks/monitor-apiserver) (Azure China documentation mirror) | Qualified |
| Managed providers control the audit policy | General provider architecture; not stated in one primary source | Qualified |

## Frequently asked questions

### What is the default Kubernetes audit policy?

Upstream kube-apiserver has none. If --audit-policy-file is not set, no audit events are logged at all. Managed services and installers ship their own policies; the Kubernetes docs point to the policy generated by the GCE configure-helper.sh script as a starting point.

### Which audit level should I use for Kubernetes Secrets?

Metadata. It records who read or changed which Secret, when, from where, and whether it was allowed, without the request or response body. At Request or RequestResponse the body is logged, which for a Secret includes its base64-encoded data.

### How do audit policy rules get evaluated?

In order, top to bottom. The first rule whose users, groups, verbs, resources, namespaces or non-resource URLs all match the request sets its audit level, and later rules are ignored for that request. Put narrow, sensitive rules first and the broad catch-all last.

### Does the audit log show the command run with kubectl exec?

Yes, at Metadata level. In our v1.34.1 test the exec event's requestURI held each argument as a command= query parameter, for example command=cat&command=%2Fetc%2Fshadow. It does not record what the command printed.


Image credit: Photo by [camera obscura](https://unsplash.com/photos/rvVhr2LngP4) on Unsplash