Safety

The safety system ensures that resource changes do not degrade application health. It combines graduated rollout modes, automatic revert triggers, and cooldown enforcement.

Graduated rollout modes

The five modes provide increasing levels of automation:

flowchart LR
    O[Observe] --> R[Recommend]
    R --> S[OneShot]
    S --> C[Canary]
    C --> A[Auto]

Mode	Risk	What happens
Recommend	None	Metrics collected, recommendations computed and written to status
OneShot	Low	OneShot applies at most one needing pod per cycle. Replicas that are already at the applied target, or that are blocked by QoS, node pressure, quota, or Infeasible plus InPlaceOnly, are skipped so another replica can still resize.
Canary	Medium	Percentage-based rollout with observation
Auto	Higher	All eligible pods resized

Observe collects metrics and tracks data-point progress but does not surface recommendations or savings. It serves as a zero-footprint warm-up before Recommend.

The recommended production path is Recommend, then Canary, then Auto.

Auto-revert triggers

Tracking annotations on the pod (attune.io/resized-at, attune.io/original-*, attune.io/tracked) are the source of truth for this lifecycle. Each reconcile classifies them into Observing (period not elapsed), Evaluating (period elapsed), RestorePending (live already at the original snapshot and the template still needs restore), or Incomplete (keys present but unreadable). The policy SafetyObservation condition reports that classification. There is no separate in-memory state machine.

When autoRevert: true (the default), the safety monitor checks each resized pod for the following conditions. A match during the observation period reverts the resize via UpdateResize. Right after apply, only OOMKill and restart+2 revert immediately. NotReady, throttle, and SLO wait for the observation period (they are often transient while the container adjusts):

OOMKill

if cs.LastTerminationState.Terminated.Reason == "OOMKilled" &&
   cs.LastTerminationState.Terminated.FinishedAt.After(record.ResizedAt)

The container was killed by the OOM killer after the resize. This indicates the new memory allocation is too low.

Mitigation: increase memory.overhead or raise memory.minAllowed.

Restart spike

if cs.RestartCount >= record.RestartCount + 2

The container has restarted 2 or more times since the resize. This catches crash loops that may be caused by insufficient resources.

Mitigation: check application logs for the root cause. The crash may not be resource-related.

CPU throttle

When a ThrottleChecker is configured (backed by Prometheus), the monitor queries the CPU throttle ratio:

rate(container_cpu_cfs_throttled_periods_total[5m])
/ rate(container_cpu_cfs_periods_total[5m])

If the ratio exceeds 50%, the resize is reverted. That threshold is the Go constant DefaultThrottleThreshold in internal/safety/monitor.go. It is not a policy field, a Helm value, or a flag. A high throttle ratio after a CPU reduction means the new allocation is too low.

Throttle grace period

The throttle check is skipped for the first 5 minutes after a resize because the Prometheus rate(...[5m]) window still contains pre-resize data. If the configured safetyObservationPeriod is shorter than 5 minutes, the operator automatically extends observation until the throttle check can execute. This prevents false-positive reverts on containers that were heavily throttled before upscaling.

Mitigation: increase cpu.overhead or raise cpu.minAllowed.

Pod NotReady

if condition.Type == PodReady && condition.Status != ConditionTrue

The pod's Ready condition is False, meaning readiness probes are failing. This check runs only after the observation period. A brief Ready=False right after UpdateResize does not revert.

Mitigation: verify that readiness probes are not sensitive to resource allocation changes. Some applications expose health endpoints that degrade under CPU throttling.

SLO guardrails

After a resize, the safety monitor can evaluate application-level PromQL queries to detect degradation. Each guardrail specifies a query, a threshold, and a comparison direction (above or below).

The check runs only after the guardrail's evaluationWindow (default: 5m, minimum: 1m) elapses post-resize. This delay gives the application time to stabilize before comparing against SLO thresholds.

Window longer than the observation period

If evaluationWindow is still open when the observation period elapses, Attune keeps the tracking annotations and evaluates that guardrail on a later pass. Elapsed guardrails are queried in the meantime. A breach reverts with slo:<name> and does not keep waiting on another open window. Query errors, NaN, and Inf still fail open and do not extend observation by themselves.

If a guardrail query breaches its threshold, the resize is reverted with reason slo:<guardrail-name>. The event names the comparison and does not include the numeric value. The monitor fails open: if a query returns an error, NaN, or Inf, the guardrail is skipped with a log message rather than triggering a false revert. Datadog and CloudWatch do not register a guardrail querier, so they do not hold observation for a guardrail window. A tenant guardrail that would use operator Prometheus credentials is limited to the policy namespace first. A query that cannot be limited is skipped and does not hold observation.

Mitigation: review the guardrail's PromQL query and threshold in updateStrategy.sloGuardrails. Adjust the threshold, widen the evaluationWindow, or remove the guardrail if the metric is unreliable.

For configuration examples, template variables, and operational tips, see the SLO guardrails guide.

Memory OOM bump hold

memory.oomBump is off until the block is set. An empty oomBump: {} turns it on. The memory request is chosen by the estimator, including its maxAllowed cap, then replaced by the OOM floor when that floor is higher. When the published value is that in-hold floor and it is above maxAllowed, the request is clamped to the cap. Default memory allowDecrease false then keeps the highest in-hold pod request, not the workload template, and does not raise another replica above the cap. A percentile with no in-hold floor stays where the estimator left it. The stored annotation keeps the uncapped floor. While the pod annotation holdUntil is still in the future, the bump floor blocks a memory revert below that floor. A newer OOM during that hold still steps above the live request. Auto, OneShot, and Canary record that oomAt and restart on the pod. The hold stops the percentile from falling below the floor. It does not ignore a new OOM. When maxAllowed is already at or below the live request, Auto, OneShot, and Canary consume the signal once and the request stays. Recommend and Observe do not write that stamp, so the same OOM still counts as skipped on every reconcile. An OOMKill verdict does not undo the bump. A non-OOM termination still reverts, including memory. A held OOM does not end observation. Throttle, NotReady, and SLO still run on that pass, still revert CPU, and keep memory at or above the floor for the rest of the hold. A Ready pod with nothing left to check ends observation on that pass. On that revert, a positive memory limit below the floor is raised only when the container's effective controlledValues is RequestsAndLimits. An empty containerPolicies list uses the policy memory block. A zero or missing limit is left alone. oomBump stays on the policy, not on a container entry. Hold expiry does not clear the original request stored on the pod. Attune does not evict a pod to change QoS. A Guaranteed pod with RequestsOnly is skipped.

Observation period

After a resize, the operator observes the pod for a configurable period before concluding the resize is safe. The observation period is configured via updateStrategy.safetyObservationPeriod (default: 5m, minimum: 1m).

Precedence: safetyObservationPeriod > canary.observationPeriod > 5m default. The safetyObservationPeriod field applies to all modes (Auto, OneShot, Canary) while canary.observationPeriod is canary-specific.

The period does not finish while the kubelet reports PodResizeInProgress, or PodResizePending with a reason other than Infeasible. If the period would already have elapsed, Attune sets attune.io/resize-apply-pending once and keeps the other tracking annotations. When that condition clears, attune.io/resized-at moves to that time and the period starts over. A resize that finishes during the original period does not start a second one. Infeasible does not extend the period. An in-progress condition that has been true for an hour is stale, same as resize eligibility. It does not hold the period, and it does not start a new one when attune.io/resize-apply-pending is already set.

Early critical detection

During the observation period (before it elapses), the operator checks for critical events that warrant an immediate revert without waiting for the full period. Currently, two conditions trigger early detection:

• OOMKill: The container was killed by the OOM killer after the resize.
• Excessive restarts: The container restarted 2+ times since the resize.

Non-critical checks (CPU throttle, Pod NotReady) still wait for the full observation period because they may be transient during resource adjustment.

Exponential backoff

When consecutive resizes are reverted, the cooldown period is doubled per consecutive revert (capped at 16x the base cooldown). This prevents the operator from repeatedly resizing a workload that is failing:

Consecutive reverts	Cooldown multiplier
0	1x (base)
1	2x
2	4x
3	8x
4+	16x (cap)

A successful resize resets the backoff to the base cooldown.

LimitRange and ResourceQuota compatibility

Before executing a resize, the controller performs two checks:

1. LimitRange: Checks per-container min/max constraints. If the target CPU or memory request would fall below the minimum or exceed the maximum defined in any LimitRange in the namespace, the resize is skipped.

2. ResourceQuota: Checks aggregate namespace headroom. If the resize would increase CPU or memory requests beyond the remaining headroom (hard - used) in any ResourceQuota, the resize is skipped. Headroom uses requests.cpu / requests.memory when those names are set, or the cpu / memory aliases when a quota uses the short names.

If listing ResourceQuotas or LimitRanges fails (for example missing list/watch RBAC), request increases fail closed and are skipped. Decreases still proceed.

Both checks prevent resize failures that would produce confusing API errors. Skipped resizes are logged but do not set error conditions.

Kubernetes Events

The controller emits Kubernetes Events for visibility:

Event type	Reason	When
Normal	Resized	A container was successfully resized
Normal	ResizeUnchanged	Applied target already matches live after filtering, dest clamp, or usage floor
Normal	ResizeDeferred	Previous-release Attune no-op Event (same meaning as ResizeUnchanged). Reserved now for kubelet node-capacity deferral; do not treat this reason as an Attune no-op
Normal	WorkloadIdle	Apply skipped because HPA ScaledToZero is True, or spec.replicas is 0
Warning	ResizeFailed	The resize API call returned an error
Warning	ResizeSkipped	A resize was skipped (QoS change, node capacity, quota, pod-level envelope)
Warning	Reverted	A resize was reverted due to a safety violation

Events are visible via kubectl describe attunepolicy and kubectl get events.

Degraded condition

When 3 or more of the last 5 resize operations are reverted, the controller sets a Degraded condition with reason HighRevertRate. This signals that the policy's parameters need adjustment before further resizes should be attempted.

Revert mechanics

When a safety violation is detected:

1. The monitor calls RevertPod(), which deep-copies the pod and sets the container resources back to the original values from the ResizeRecord.
2. The revert uses UpdateResize (the same in-place mechanism), so no pod restart occurs.
3. When templatePersistence.when=AfterSuccessfulResize is enabled, the controller also restores the Deployment/StatefulSet template for that container from the pre-resize snapshot (OriginalResources). Restore replaces the container resources, including clearing limits the persist step added. Persist already patched the template on Success or Evicted, before the observation window; without this restore, rollouts keep the unsafe size. A failed template restore keeps tracking annotations so the next reconcile retries.
4. The resize history entry is updated to result: Reverted.
5. The attune_reverts_total counter is incremented with the violation reason as a label.

sequenceDiagram
    participant E as Resize Engine
    participant S as Safety Monitor
    participant K as K8s API

    E->>K: UpdateResize (new resources)
    E->>K: Get pod status
    E->>S: CheckCriticalStatuses(pod)
    alt OOMKill / Restart+2
        S-->>E: Verdict{Safe: false}
        E->>S: RevertPod(record)
        S->>K: UpdateResize (original resources)
    else NotReady / throttle / SLO / healthy
        Note over E,S: Defer non-critical checks to observation
        S-->>E: Verdict{Safe: true}
    end

Cooldown enforcement

After any resize operation (successful or reverted), the operator records attune.io/last-resize-time (policy-wide) and attune.io/last-resize-time.<workload> (per app). A later resize of app A does not lock app B.

On the next reconciliation, each workload is skipped while now - lastResizeTime.<app> < effectiveCooldown (base cooldown times 2^N consecutive reverts for that app). When every matched app is still cooling, RequeueAfter is the soonest remaining window, not a fresh full cooldown. A watch event mid-window must not restart the timer.

The default cooldown is 1 hour. This prevents rapid-fire resizes that could destabilize workloads.

Node capacity pre-check

Before submitting a resize, the controller sums all container requests in the pod (applying the new target for the container being resized) and compares the total against the node's allocatable resources:

if totalCPU > allocCPU || totalMem > allocMem {
    // skip resize for this container
}

This prevents resizes that would exceed the node's capacity and cause eviction. Request increases also subtract other pods' requests on the same node (neighbor free request budget). A failed neighbor list skips the increase. If the node lookup fails (for example node not found), request increases are skipped (fail-closed). Decreases still proceed. See Node capacity.

Last-replica eviction

When resizeMethod is InPlaceOrRecreate and in-place resize cannot proceed, the operator may evict the pod. Before calling the Eviction API it lists matching pods through the typed Clientset (live API, not the informer cache) and counts only pods with status.phase=Running and no deletionTimestamp.

spec.replicas and NotReady or Pending pods do not count. A Deployment with replicas: 3 and only one Running pod is treated as a last replica. History reason is eviction_last_replica and the operator emits EvictionBlocked. A missing selector or a list failure emits the same event with reason eviction_no_selector or eviction_list_failed. A PodDisruptionBudget denial uses EvictionDenied / eviction_denied.

Memory usage floor

When decreasing a memory limit, the controller compares the target against the live container limit on the pod, not the workload template. The template (and rec.Current in status) can lag after an in-place resize. The target is then raised if it would fall at or below recent usage (recommendation raw percentile) times (1 + decreaseUsageMarginPercent/100). See resize API.

Template persist uses the same usage floor. After an in-place resize, rec.Current can still be the old template. Persist raises a recommended memory limit that is below recent usage even when that is not a decrease versus rec.Current. RequestsOnly persist does not write limits.

Container exclusion

Well-known mesh and sidecar names (for example istio-proxy, linkerd-proxy) are auto-excluded by default when excludeKnownSidecars is true. The excludedContainers field adds more names to skip (union with the known list), for custom agents beyond the built-in set. Excluded containers are not used for recommendations or resizes.

Conflict detection

Before resizing, the controller checks for potential conflicts:

• Active rollout: Auto, OneShot, and Canary skip resize (RolloutInProgress) during a RollingUpdate replacement. Recommendations are still computed. Recreate and OnDelete are not a skip: those pods are resized. A RollingUpdate Deployment is not skipped because availableReplicas is behind, or because a scale-out has not finished, when generation is observed and no old pods remain. A ReplicaSet is not skipped because readyReplicas is behind when generation is observed, so unready pods can be resized. A Rollout waits on phase Paused, Progressing, or Degraded, and on status.abort. A Healthy phase, or an empty phase, does not wait because updatedReplicas is behind spec.replicas.
• Opt-out annotation: workloads with attune.io/skip: "true" are skipped entirely.
• Namespace freeze: attune.io/freeze=true on the namespace skips new in-place resize, eviction, startup boost, template persist, and CREATE initial sizing. Recommendations still compute. Pending safety observation still reverts unsafe pods and restores AfterSuccessfulResize templates. If the namespace cannot be read, apply is skipped (fail closed).
• QoS preservation: a resize that would change QoS class is skipped before UpdateResize (Burstable to Guaranteed, BestEffort to Burstable, or Guaranteed to Burstable). Init containers count. For Guaranteed pods, requests must stay equal to limits. A memory limitMultiplier raises the request to the multiplied limit, so the applied request can exceed memory.maxAllowed and the pod stays Guaranteed. maxAllowed capped the engine request. It does not cap that raise. A CPU multiplier that would leave Guaranteed is skipped and is not evicted. RequestsOnly does not apply the multiplier.
• HPA coexistence: an informational notice is logged but resizing proceeds. See HPA Coexistence.
• HPA memory targets: when attune.io/auto-tune is "true", a successful in-place memory resize retunes memory utilization targets on that HPA in the same update as CPU. The original target and request are stored on the HPA. hpaTargetBounds can clamp the published percent. It stays unset unless the policy or AttuneDefaults sets it.

High revert rate

When multiple consecutive resizes are reverted (visible in .status.resizeHistory), the policy's parameters likely need adjustment before further resizes should be attempted. Check the revert reasons (oomkill, restart, notready, throttle, slo:<name>) and adjust overheads or cooldown accordingly.