Troubleshooting

Common conditions

Check the policy's conditions for a quick diagnosis:

kubectl get attunepolicy <name> -o jsonpath='{.status.conditions}' | jq .

PrometheusSeriesCapped

Symptom: Ready is True with reason PrometheusSeriesCapped, or logs show "Prometheus range query series capped" or a CloudWatch series/page cap.

Cause: A range query returned more series than --max-prometheus-series (default 5000). CloudWatch GetMetricData uses the same series default and also stops after 20 result pages. Attune keeps partial data (at least one series per container when the cap allows) and continues. With cpu.startupBoost.excludeFromHistory: true, the CPU cap also spreads across pods. See One container missing under a capped startup query.

Fix:

1. Keep the default metricsSource.podAggregation: Max (or set it explicitly) so series count tracks containers, not pods.
2. Raise maxPrometheusSeries / --max-prometheus-series if you need more series under None or Avg.
3. Use recording rules (cpuRecordingMetric / memoryRecordingMetric) for pre-aggregated metrics.

See Scaling: PrometheusSeriesCapped.

MetricsUnavailable

Symptom: Ready condition is False with reason MetricsUnavailable. Policies last written before this reason existed may still show the older PrometheusUnavailable alias until the next reconcile.

Cause: MetricsUnavailable means the controller could not use the metrics backend (Prometheus, Datadog, or CloudWatch) for this reconcile. The condition message tells you which step failed. During this state the operator requeues at min(cooldown, queryStep) and does not add requeueJitter, the same bootstrap interval as InsufficientData.

• Cannot resolve Prometheus config means address resolution failed. The operator checks (in order): policy spec, one defaults source (AttuneNamespaceDefaults if present, otherwise AttuneDefaults), Prometheus Operator CRD, then well-known service names.
• Cannot create metrics collector, reading secret, or transport errors like TLS handshake timeout mean the address was found but auth, headers, bearer token secret, CA bundle, or TLS setup failed.
• Metrics query timeout exceeded means the reconcile-level timeout expired before all backend queries completed.
• Metrics query errors ( means the backend answered, but one or more metric queries failed. This can still happen when the backend is reachable.
• cannot read Datadog API key, datadog API returned 403, or datadog API returned 429 means the Datadog Secret is missing, the API key is rejected, or Datadog rate-limited the collector.
• loading AWS config, AccessDeniedException, or AssumeRole means CloudWatch IAM credentials, role assumption, or GetMetricData failed.

If the condition message includes Cannot resolve Prometheus config: SSRF blocked, the configured address points at localhost, 127.0.0.1, ::1, or a cloud metadata endpoint. Replace it with the in-cluster Prometheus Service DNS name or ClusterIP. A local kubectl port-forward URL on your workstation will not work.

Fix address resolution failures:

1. Set the address explicitly in a AttuneDefaults resource:

   apiVersion: attune.io/v1alpha1
   kind: AttuneDefaults
   metadata:
     name: default
   spec:
     metricsSource:
       prometheus:
         address: http://prometheus-server.monitoring:80
   

2. Verify the Prometheus Service exists and note its port:

   kubectl get svc -n monitoring
   # Check the PORT(S) column: "80/TCP" means use :80, not :9090
   

3. Test connectivity from inside the cluster:

   kubectl run prom-test --image=curlimages/curl --restart=Never --rm --attach --command -- \
     curl -sf http://prometheus-server.monitoring:80/-/healthy
   

If the condition message includes Cannot create metrics collector, reading secret, or a transport error like TLS handshake timeout, verify the credentials and connection details before changing timeouts:

1. Check the referenced Secret exists in the policy namespace and contains the expected bearer token.
2. Re-check custom headers, CA bundle, and insecureSkipVerify settings.
3. Test the exact Prometheus URL from inside the cluster with the same auth mechanism the operator uses.

If the condition message includes Metrics query timeout exceeded, the operator's reconcile-level timeout expired before all workload queries completed. This typically happens when the metrics backend is slow to respond (not down, just overloaded) or when a policy targets many workloads.

Fix query timeouts:

1. Increase the timeout: set Helm prometheusTimeout: "10m" (or --prometheus-timeout=10m).
2. Reduce per-query cost: decrease historyWindow or increase queryStep on the AttunePolicy or AttuneDefaults.
3. Check Prometheus health: high query latency often indicates Prometheus itself needs more resources or recording rules.

If the condition message includes Metrics query errors (, the backend was reachable but one or more metric queries still failed.

Fix query errors:

1. Check the operator logs for the exact failing query and backend error.
2. Replay the failing query directly against Prometheus to confirm whether the backend rejects it or returns partial data.
3. If the backend is overloaded, reduce query cost with a shorter historyWindow or a larger queryStep.

See the Prometheus Setup guide for full details on address resolution and common installations.

Prometheus reachable but queries return no data

Symptom: Ready condition is InsufficientData even after days of running. Operator logs show "cpuPoints":0,"memPoints":0.

Cause: Prometheus is reachable but cadvisor metrics are not being scraped, or label names have been relabeled.

Fix:

1. Verify cadvisor metrics exist in Prometheus:

   kubectl run prom-check --image=curlimages/curl --restart=Never --rm --attach --command -- \
     curl -s 'http://prometheus-server.monitoring:80/api/v1/query?query=container_cpu_usage_seconds_total' \
     | head -c 200
   

2. If the result is empty ("result":[]), cadvisor scraping is not configured. Check your Prometheus scrape configuration for a kubernetes-nodes-cadvisor or equivalent job.

3. If the result has data but the operator still reports 0 data points, check that the namespace, pod, and container label names match. Some Prometheus configurations relabel these.

CronJob policy stays InsufficientData

Symptom: A policy with targetRef.kind: CronJob stays InsufficientData although its Jobs have run.

Cause: Attune finds CronJob pods by name. The pod name does not have the shape Attune queries.

Fix:

1. List the pods of the CronJob's Jobs:

   kubectl get pods -n <namespace> -l batch.kubernetes.io/job-name
   

2. Compare the names with the two shapes Attune matches. The minute is the scheduled time in Unix minutes (8 digits today, 9 later):

   • <cronjob>-<minute>-<5 characters>, for example nightly-29453760-fghij
   • <cronjob>-<minute>-<index>-<5 characters> when the job template sets completionMode: Indexed, for example nightly-29453760-3-fghij

3. Check that the backend has samples for those names. Batch pods are short-lived, so use a range query. A non-indexed template uses pod=~"<cronjob>-[0-9]{8,9}-[a-z0-9]{5}":

   kubectl run prom-check --image=curlimages/curl --restart=Never --rm --attach --command -- \
     curl -s -G 'http://prometheus-server.monitoring:80/api/v1/query' \
     --data-urlencode 'query=count by (pod) (last_over_time(container_memory_working_set_bytes{namespace="<namespace>",pod=~"<cronjob>-[0-9]{8,9}-[a-z0-9]{5}"}[1d]))'
   

An Indexed template uses pod=~"<cronjob>-[0-9]{8,9}-[0-9]+-[a-z0-9]{5}":

```bash
kubectl run prom-check --image=curlimages/curl --restart=Never --rm --attach --command -- \
  curl -s -G 'http://prometheus-server.monitoring:80/api/v1/query' \
  --data-urlencode 'query=count by (pod) (last_over_time(container_memory_working_set_bytes{namespace="<namespace>",pod=~"<cronjob>-[0-9]{8,9}-[0-9]+-[a-z0-9]{5}"}[1d]))'
```

1. A CronJob name longer than 48 characters does not match. Kubernetes cuts the pod name prefix to 58 characters, which removes the separator before the random suffix or part of the minute stamp. An Indexed template reaches that limit with a shorter name.

NoWorkloadsFound

Symptom: Ready condition is False with reason NoWorkloadsFound.

Cause: The policy's targetRef does not match any workloads in the namespace. This is usually a typo in the workload name or an incorrect kind (e.g., targeting a Deployment when the workload is a StatefulSet).

Fix:

1. Verify the workload exists:

   kubectl get deploy,sts,ds -n <namespace>
   

2. Check the targetRef.name spelling in your policy. If using a label selector, verify the labels exist on the target workload:

   kubectl get deploy <name> -n <namespace> --show-labels
   

3. Ensure the targetRef.kind matches the workload type (Deployment, StatefulSet, DaemonSet, ReplicaSet, Job, CronJob, or Rollout).

ConflictCheckFailed

Symptom: Ready condition is False with reason ConflictCheckFailed.

Cause: The operator could not list AttunePolicy objects in the namespace while checking for overlapping targets. That cycle does not compute new recommendations. Last-known recommendations stay on status so this does not look like bootstrap InsufficientData.

Fix:

1. Confirm the operator ServiceAccount can list and watch attunepolicies.attune.io.
2. Check API server health and operator logs for the list error.
3. Watch attune_reconcile_errors_total{error_type="list_policies"}.

401 after a trailing newline in a Secret

Symptom: Prometheus or Datadog returns 401 or 403 for a key that works once the newline is removed.

Cause: The Secret was copied from a file or a YAML block scalar and kept whitespace at either end. Attune trims both ends and does not strip spaces in the middle of the value. A whitespace-only bearer token or Datadog API key fails at read time with an empty-key error. A whitespace-only Datadog app key is omitted and does not fail the API key. Operator logs do not print the secret.

Fix: No policy change is required for whitespace at either end. If the key still fails, confirm the Secret data has no extra characters in the middle.

OpenShift Thanos Querier 401 or secret not found

Symptom: Ready MetricsUnavailable with reading secret <policy-ns>/<token-name>: secrets "..." not found, or HTTP 401/403 against thanos-querier.openshift-monitoring.svc:9091.

Cause: Cluster AttuneDefaults.bearerTokenSecret copies the Secret name onto each policy and looks it up in the policy namespace. OpenShift Thanos Querier on port 9091 wants the operator ServiceAccount with cluster-monitoring-view, not a copied token in every app namespace.

Fix: Remove bearerTokenSecret from AttuneDefaults. Set Helm openshift.bindClusterMonitoringView: true and prometheusAuth.queryServiceAccount.create: true. OperatorHub and the kustomize install already pass --prometheus-use-service-account-token and --prometheus-query-service-account=attune-prometheus-query. Bind cluster-monitoring-view to attune-prometheus-query, not the manager ServiceAccount. Keep the Thanos address and TLS on AttuneDefaults. See OpenShift: Thanos Querier.

Amazon Managed Prometheus returns 403

Symptom: Ready MetricsUnavailable and a Prometheus query to an aps-workspaces host returns HTTP 403.

Cause: The policy has no sigv4 block, sigv4.region does not match the workspace, or the pod identity / assumed role cannot call aps:QueryMetrics.

Fix: Set metricsSource.prometheus.sigv4.region to the workspace region. Use the workspace root as address (no /api/v1/query suffix). Confirm the IAM role allows aps:QueryMetrics. Operator logs record the workspace host and the status code. They do not record Authorization or X-Amz-Security-Token.

Datadog API key secret not found

Symptom: Ready MetricsUnavailable with reading secret <policy-ns>/<key-name>: secrets "..." not found while cluster AttuneDefaults sets metricsSource.datadog.apiKeySecretRef.

Cause: That name is copied onto each policy and read in the policy namespace, unless the manager is given an operator-namespace Secret.

Fix: Create the Secret in the operator namespace (keys api-key and optional app-key). Set Helm datadogAuth.existingSecret.name and, if the key is not api-key, datadogAuth.existingSecret.key (or --datadog-api-key-secret and --datadog-api-key-secret-key). Leave apiKeySecretRef on cluster AttuneDefaults: the schema requires it, and with the operator Secret set that name is not read. A policy or AttuneNamespaceDefaults ref stays in that namespace. See Datadog setup.

InsufficientData

Symptom: Ready condition is False with reason InsufficientData.

Check first: kubectl attune doctor confirms Kubernetes 1.32+ and pods/resize. A failed Prometheus ping is optional (printed as WARN) and does not prove the operator cannot reach an in-cluster address. A 401 or 403 on an address that sets bearerTokenSecret or custom headers is skipped the same way: doctor does not send those credentials. Doctor also skips 401/403 for operator auth only when that address comes from the selected cluster AttuneDefaults and the manager is configured to send it. A policy or namespace-defaults address still warns. Doctor does not send the token itself.

Cause: Not enough Prometheus data points to generate recommendations. The default minimum is 48 Prometheus range-query samples. With the default queryStep: 5m, that is about 4 hours of data.

During this state the operator requeues at min(cooldown, queryStep) and does not add requeueJitter. A policy with cooldown: 1m and the default 5m step therefore retries every minute, not every 1–3 minutes.

If memory.memoryFromCpuRatio is set to a valid ratio, Attune waits for a CPU recommendation and does not use memory usage gauges or a VPA memory target. Ready stays InsufficientData and retries at min(cooldown, queryStep). A CPU query error is MetricsUnavailable and uses the same short requeue (no requeueJitter). A prior rec whose memory explanation contains memoryFromCpuRatio is kept as Stale until CPU returns. The Collecting data: X/Y message counts CPU samples only while waiting. See memory.memoryFromCpuRatio.

Fix: Wait for more data to accumulate, or adjust these settings:

• minimumDataPoints: Lower for faster (but less confident) recommendations.
• historyWindow: If too short (e.g. 1h), Prometheus may not have enough samples within the window. The default is 168h (7 days). Ensure the window is long enough for your scrape interval to produce at least minimumDataPoints data points.

spec:
  metricsSource:
    minimumDataPoints: 48   # ~4 hours of data at the default queryStep: 5m
    historyWindow: 168h     # query the last 7 days of metrics

CloudWatch policy has no samples

Symptom: A CloudWatch policy stays InsufficientData while Container Insights is scraping the workload.

Cause: With prefer_full_pod_name left false, PodName is the Deployment name or the CronJob name, not the pod name. A policy for api does not read a series named api-v2 or api-worker. Those siblings share a prefix and are a different workload. DaemonSet and StatefulSet series are unchanged.

Fix:

1. In CloudWatch, open one container_cpu_usage_total point for the namespace and read PodName.
2. A Deployment named api shows api. A CronJob named nightly shows nightly. A cluster that sets prefer_full_pod_name: true shows the full pod name instead.
3. If PodName is a sibling, point the policy at that workload. Attune has no field that selects the dimension.

CronJob pod name does not match

Symptom: A CronJob policy stays InsufficientData while its pods run.

Cause: The query matches the pod name. A non-indexed pod is <cronjob>-<minute>-<hash>, for example nightly-29453760-abc12. The minute stamp is unix time divided by 60 (8 digits until 2160-02-18, 9 digits after that). An indexed pod inserts the completion index: nightly-29453760-0-abc12. A 10-digit unix-seconds suffix does not match. The Job name (nightly-29453760) is not a pod name.

Fix: Compare a live pod name from kubectl get pods with the query. When the name matches, wait for minimumDataPoints.

Recommendations missing while a pod is crash looping

Symptom: Recommendations are missing for a Deployment or ReplicaSet while one of its pods is crash looping.

Cause: The workload gate used to treat a Deployment with availableReplicas behind spec.replicas, or a ReplicaSet with readyReplicas behind spec.replicas, as a rollout. That skipped recommendations. A crash loop is not a rollout when generation is observed and no old pods remain.

Fix: Recommendations are still computed in that state. If a resize does not happen, read the pod-level skip on the policy events. ResizeSkipped covers a pod that is not ready, terminating, Deferred, Infeasible, or a QoS change. RolloutInProgress is only a resize skip for Auto, OneShot, and Canary during a real replacement.

StatefulSet pods below a partition are not resized

Symptom: Some pods of a RollingUpdate StatefulSet stay at their current requests. Pods whose controller-revision-hash matches status.updateRevision change. Pods on status.currentRevision do not. The policy emits RolloutInProgress for the skipped pods.

Cause: A partition that has not reached every replica leaves currentRevision different from updateRevision on purpose. Pods whose hash is not updateRevision stay skipped for as long as those two revisions differ. The hold does not end when the StatefulSet stops making progress. Attune does not read the pod ordinal.

A stale generation is a different skip. metadata.generation ahead of status.observedGeneration skips every pod, including pods already on updateRevision. When the generation is observed and the two revision strings match, this check skips no pod.

OnDelete is not this skip. Those pods are resized, including pods still on the previous template.

Fix: Lower partition, or wait until currentRevision equals updateRevision, when the older pods should be resized. Leaving the partition in place keeps the older pods unchanged.

DaemonSet pods are not resized

Symptom: Pods of a RollingUpdate DaemonSet stay at their current requests. The policy emits DaemonSetRevisionUnavailable or RolloutInProgress.

Cause: Attune resizes a pod when its controller-revision-hash label matches that label on the current ControllerRevision. The revision name is <daemonset>-<hash>. The pod label is only <hash>. Pods still on the previous hash are skipped with RolloutInProgress. A pod with no hash label is skipped while that current hash is known. The annotation pod-template-generation is not a revision.

DaemonSetRevisionUnavailable means the current revision could not be read. The list is limited to the DaemonSet pod selector, then to revisions owned by that DaemonSet. The event means the list failed, the current revision has no hash label, or an owned revision does not match the selector. Attune skips every pod of that DaemonSet until the read succeeds. OnDelete DaemonSets do not read ControllerRevisions. The pod label compared with the revision is still the hash, not the revision name.

Fix: Apply the ClusterRole with the new image so it can get and list controllerrevisions. The lookup uses the API reader, not a cached watch. A Helm upgrade adds the rule.

InvalidConfig

Symptom: Ready condition is False with reason InvalidConfig.

Cause: The controller could not fetch or apply defaults cleanly before continuing. The condition message includes the failing step, such as Failed to fetch defaults: listing AttuneNamespaceDefaults ....

Fix:

1. Check whether the operator can list AttuneDefaults and AttuneNamespaceDefaults.
2. Verify the defaults objects themselves are valid and that only the expected objects exist in the namespace.
3. Check operator logs for the exact failing API call or validation error.

InvalidConfig from limitMultiplier

Symptom: Ready is False with reason InvalidConfig, and the message says cpu.limitMultiplier cannot be set when cpu.controlledValues is RequestsOnly (or the same sentence for memory).

Cause: The policy omits controlledValues. Built-in defaults fill RequestsOnly, and a limitMultiplier inherited from AttuneDefaults or AttuneNamespaceDefaults is then illegal. A policy that already sets controlledValues: RequestsOnly does not inherit that multiplier, so it does not hit this reason.

Fix:

1. Set controlledValues: RequestsAndLimits on the policy when the multiplier should apply.
2. Or remove limitMultiplier from the defaults object.
3. An explicit RequestsOnly policy needs no change. It keeps the live limit.

An object already stored with a multiplier and no RequestsAndLimits is not rewritten. It stays on this reason until an update sets RequestsAndLimits or removes the multiplier. A new apply of that shape fails in admission instead, with cpu.limitMultiplier requires cpu.controlledValues RequestsAndLimits, or remove the multiplier.

Policy admission denied: listing AttuneNamespaceDefaults or AttuneDefaults

Symptom: kubectl apply of a policy fails with:

admission webhook "validation.attune.io" denied the request:
listing AttuneNamespaceDefaults in <namespace>: ...

or, when the namespace list succeeds and the cluster list fails, listing AttuneDefaults: ....

Cause: Admission lists defaults in two cases. A list error rejects the policy instead of continuing.

1. The policy sets cpu.surge.window or memory.surge.window and omits metricsSource.historyWindow. The surge window must not be longer than the inherited history window, so admission lists the defaults to find it. A list error rejects the policy instead of falling back to 168h.
2. The policy, or a named containerPolicies entry, omits minAllowed while maxAllowed is set. Admission lists defaults to see whether an inherited minimum sits above that maximum. See Apply rejected: defaults min above policy max when the list succeeds and the values conflict.

Policies that set historyWindow, set no surge window, and do not omit a min beside a known max on the policy or a named container do not read defaults at admission.

Fix:

1. Restore list access to AttuneNamespaceDefaults and AttuneDefaults for the operator, as for InvalidConfig.
2. When the failure is the surge window, set metricsSource.historyWindow on the policy. The surge window is then checked against that value. A policy or named container that still omits minAllowed beside a known maxAllowed lists defaults for that check.

WorkloadDiscoveryFailed

Symptom: Ready condition is False with reason WorkloadDiscoveryFailed.

Cause: The operator could not resolve the policy's targetRef into the workloads it should inspect. The condition message includes the failing step, for example an unsupported kind, an invalid selector, or a client/list error.

Fix:

1. Verify spec.targetRef.kind is one of Deployment, StatefulSet, DaemonSet, CronJob, Job, ReplicaSet, or Rollout.
2. If you use targetRef.name, confirm the workload exists in the same namespace as the policy.
3. If you use targetRef.selector, confirm it matches at least one workload and includes real matchLabels or matchExpressions entries.
4. Check operator logs for the exact discovery error if the target still looks correct.

WorkloadCRDMissing

Symptom: Ready condition is False with reason WorkloadCRDMissing and message argoproj.io/v1alpha1 Rollout CRD is not installed.

Cause: spec.targetRef.kind is Rollout, and the Argo Rollouts CRD is not installed in this cluster. This is cluster state, not a bad policy. Attune does not watch Rollout and does not install the CRD. Reconcile succeeds and retries every couple of minutes. A missing Rollout object uses NoWorkloadsFound instead. A mapper error that is not "kind not found" uses WorkloadDiscoveryFailed.

Fix:

1. Install Argo Rollouts, or point the policy at a kind that exists.

2. Confirm the CRD:

   kubectl get crd rollouts.argoproj.io
   

3. Policies that do not set kind: Rollout do not take this path.

Rollout template was not patched

Symptom: Template persistence is on, but a Rollout's spec.template keeps its requests. TemplatePersistence is False.

Cause: The Rollout sets spec.workloadRef. The reason tells you why:

• TemplateWorkloadRef: this reconcile's recorded errors did not include a workloadRef read failure, and Attune left the Rollout template alone on purpose. When the Rollout template has no containers, recommendations use the referenced pod template. Nothing to fix.
• WorkloadRefUnread: spec.workloadRef has an empty name or an unsupported kind, or the referenced Deployment, StatefulSet, or ReplicaSet is missing, could not be read (for example forbidden), or has no containers. No recommendation is stored for that Rollout. The condition message and status.workloadErrors carry the error. This reason is kept while a recorded workload error is a workloadRef read failure, even when another Rollout's reference was read.

status.workloadErrors keeps at most 10 entries. With more failing workloads, a reference error can be dropped from it, and the reason may show TemplateWorkloadRef or not appear even though a read failed.

Fix for WorkloadRefUnread:

1. Read the error:

   kubectl get attunepolicy <name> -o jsonpath='{.status.workloadErrors}' | jq .
   

2. Check that spec.workloadRef names an object of that kind in the Rollout's namespace, and that the operator can get it.

3. On the next reconcile whose workload errors no longer include a workloadRef read failure, that reason is removed. Template persistence then records TemplateWorkloadRef when a Rollout still sets spec.workloadRef, or removes the condition when none do.

New pods still start at template size

Symptom: updateStrategy.initialSizing is true, but new pods keep the Deployment template requests. The pod has no attune.io/initial-sizing=applied annotation.

Cause: The mutating webhook only patches CREATE when every gate passes. A selector policy also has to fetch the owning Deployment, StatefulSet, DaemonSet, ReplicaSet, Job, or CronJob and match targetRef.selector. A Deployment-owned ReplicaSet is resolved to the Deployment and is not a ReplicaSet target. CronJob pods are owned by a Job; CREATE walks that Job to the CronJob. Get or parse errors skip the pod (the CREATE is still allowed). A stale recommendation also skips initial sizing (last-known values stay in status, but CREATE is not patched).

Fix:

1. Confirm the Helm/operator value initialSizing.enabled is true and the namespace has label attune.io/initial-sizing=enabled.
2. Confirm updateStrategy.initialSizing: true. Auto, OneShot, and Canary apply CREATE for any owner. Recommend applies CREATE only for Job and CronJob. Observe skips CREATE for every owner.
3. If targetRef.selector is set, confirm the owner object exists and its labels match. An empty selector matches nothing.
4. On Canary, CREATE sizing waits until that app is promoted, or the assigned pod name is already in status.canary.workloads[].pods. ReplicaSet CREATE often has an empty metadata.name and only generateName. An empty name is not treated as a canary-slice identity, so those pods stay at template size until the app is promoted.
5. The webhook applies a rec when every container has confidence at least 0.5, or this workload already has a successful in-place resize. A 1h historyWindow never reaches 0.5 on its own.
6. Check operator logs for fetching workload for initial-sizing selector or initial sizing applied. When CREATE has no assigned name, that Info line uses generateName (for example my-app-abc-), not the name kubelet later assigns.
7. Check the namespace is not frozen. attune.io/freeze=true skips CREATE initial sizing. Confirm ResizeBlocked=NamespaceFrozen on the policy. Grep webhook logs for the fail-closed admission message cannot read namespace for attune.io/freeze (Allowed, not Denied) if the operator cannot read the namespace. The same skip applies when ResizeBlocked is HPAListUnavailable or VPAListUnavailable (leftover recs stay in status; new pods keep the template).
8. CREATE uses the same merged controlledValues as reconcile (AttuneDefaults, then AttuneNamespaceDefaults, then the policy). A list error for those CRs skips the mutation (Allowed, not Denied). Grep webhook logs for listing AttuneDefaults for initial sizing. If the policy leaves controlledValues unset and cluster/namespace defaults set RequestsAndLimits, new pods should receive recommended limits too; missing limits after a successful initial-sizing=applied annotation means the defaults were not readable or the rec has no limit.

Paused

Symptom: Ready condition is False with reason Paused.

Cause: spec.paused is set to true on the policy. The operator skips all reconciliation: no metrics collection, no recommendations, no resizes. CREATE initial sizing is also skipped (leftover recs stay in status). Existing resizes are not reverted.

Fix: Set spec.paused: false or remove the field entirely. The operator will resume reconciliation on the next cycle.

NamespaceFrozen

Symptom: ResizeBlocked is True with reason NamespaceFrozen. Events say namespace has attune.io/freeze=true; new apply skipped (pending safety revert still runs). Recommendations still appear in status.

Cause: The policy namespace has attune.io/freeze=true, or the operator could not read the namespace (fail closed). New in-place resizes, evictions, startup boosts, template persistence, and CREATE initial sizing are skipped. Freeze is not a rollback of already-applied successful recommendations. Pending safety observation still reverts unsafe pods and restores AfterSuccessfulResize templates. Metrics collection and recommendations continue so kubectl attune recommendations still works.

Fix: Remove the annotation or set it to any value other than true:

kubectl annotate namespace <ns> attune.io/freeze-

If the condition message says the namespace could not be read, check RBAC for namespaces get/list/watch on the operator ServiceAccount.

HPAListUnavailable

Symptom: ResizeBlocked is True with reason HPAListUnavailable. Ready stays Monitoring. Events say cannot list HorizontalPodAutoscalers; new apply skipped (check horizontalpodautoscalers list/watch RBAC). Recommendations still appear in status. No in-place resizes, template persistence, startup boosts, or CREATE initial sizing run that cycle.

Cause: The operator could not list HorizontalPodAutoscaler objects in the policy namespace (missing RBAC list/watch, or an apiserver error). Apply is skipped so a ScaledToZero HPA is not treated as "no HPA" (active). Recommendations still compute.

Fix:

1. Confirm the operator ServiceAccount can list and watch horizontalpodautoscalers in the policy namespace (or cluster-wide).
2. Check API server health and operator logs for the list error.
3. Watch attune_reconcile_errors_total{error_type="list_hpas"}.

VPAListUnavailable

Symptom: ResizeBlocked is True with reason VPAListUnavailable. Ready stays Monitoring. Events say cannot list VerticalPodAutoscalers; new apply skipped (check verticalpodautoscalers list/watch RBAC). Recommendations still appear in status. No in-place resizes, template persistence, startup boosts, or CREATE initial sizing run that cycle.

Cause: The operator could not list VerticalPodAutoscaler objects in the policy namespace (missing RBAC list/watch, or an apiserver error). A missing VPA CRD is not this reason. Apply is skipped so an applying VPA is not treated as absent.

Fix:

1. Confirm the operator ServiceAccount can list and watch verticalpodautoscalers in the policy namespace (or cluster-wide).
2. Check API server health and operator logs for the list error.
3. Watch attune_reconcile_errors_total{error_type="list_vpas"}.

Template persist skipped after a pod list error

Symptom: Operator logs Failed to list pods for live envelope; skipping template persistence. Container recs stay off the Deployment/StatefulSet template. Resize history has no TemplatePatched row for that cycle.

Cause: Persist must read live spec.resources before it writes container recs. A pod list error used to look like "no envelope", so persist could write requests and drop a live envelope on the next rollout. It now skips, same as an HPA list error.

Fix:

1. Confirm the operator can list/watch pods in the policy namespace.
2. Check API server health and operator logs for the list error.
3. The next successful list retries persist (including AfterSuccessfulResize lagging retry outside cooldown).

Resizing stays InProgress

Symptom: Resizing is True with reason InProgress for longer than a few minutes, including across cooldown. The message says workloads were resized this cycle, and the policy requeues at the safety observation period instead of the cooldown.

Cause: Older builds copied every successful in-place row still in status.resizeHistory (a 50-entry cap, not a time window) into status.workloads.resized. Later idle reconciles looked like a fresh resize. With autoRevert left on (the default), that shortened the requeue for as long as any successful in-place row remained.

Fix: Upgrade to a release that counts only resizes from the latest reconcile. After that, sustained InProgress means resizes are still happening. Check status.resizeHistory timestamps and live pod requests. CooldownActive or Idle with workloads.resized: 0 is the idle state, even when history still lists older successes.

CooldownActive

Symptom: The operator logs "Cooldown active, skipping resize" and no pods are resized.

Cause: A resize was performed recently and the cooldown period has not elapsed.

Fix: Wait for the cooldown to expire, or shorten it:

kubectl patch attunepolicy <name> --type merge \
  -p '{"spec":{"updateStrategy":{"cooldown":"30m"}}}'

Stored cooldown of 0s

Symptom: kubectl describe shows updateStrategy.cooldown: 0s on an AttunePolicy, AttuneDefaults, or AttuneNamespaceDefaults, and the policy reconciles about once an hour. Changing an unrelated field, or clearing a finalizer, on a defaults object that already stores 0s fails on an older operator with cooldown must be at least 1m, or omit the field for the default. A create that sets 0s, or an update that changes a positive duration to 0s, still fails with that message on all three kinds. The same shape of message applies to safetyObservationPeriod and sloGuardrails[0].evaluationWindow. An unchanged historyWindow: 0s still fails with historyWindow must be at least 1h.

Cause: Zero is not a wait. Admission rejects a new 0s. An object that already stores 0s can be edited and deleted while that value stays. That includes AttuneDefaults and AttuneNamespaceDefaults. Attune adds no finalizer to defaults objects. Defaults have no delete-time skip, so another invalid field still blocks the edit. The controller still maps a cooldown of 0s to 1h after defaults merge. A stored safetyObservationPeriod of 0s is unset (5m, or a positive canary period). A stored SLO evaluationWindow of 0s uses 5m, and is accepted only when the stored entry at the same list index is also 0s. Canary observationPeriod: 0s is different: omitted and 0s both mean the built-in observation period.

Fix: Leave the stored 0s in place when the edit is unrelated. Omit the field, or set at least 1m, when the stored zero should go:

kubectl patch attunepolicy <name> --type merge \
  -p '{"spec":{"updateStrategy":{"cooldown":"1h"}}}'

The same merge patch works on attunedefaults and attunenamespacedefaults.

Schedule window never opens

Symptom: ScheduleBlocked is True with reason OutsideWindow at every hour, including the minute you set. The message says the window never opens. attune_schedule_skipped_total increases on each resize reconcile.

Cause: The window start and end are the same minute, such as 02:00 to 02:00. The end minute is exclusive, so that pair matches no time.

Fix: Set the end to a later minute, or to an earlier minute when the window should run overnight (22:00 to 06:00). Creating or updating a policy or defaults object with an equal pair is rejected.

Webhook / cert-manager issues

Webhook connection refused

Symptom: kubectl apply -f policy.yaml returns:

Error from server (InternalError): Internal error occurred: failed calling
webhook "vattunepolicy.kb.io": Post "https://...": dial tcp ...: connection refused

Cause: The webhook server is not running or the TLS certificate is not ready. This typically means cert-manager is missing or broken.

Fix:

1. Verify cert-manager is installed and running:

   kubectl get pods -n cert-manager
   # All 3 pods (cert-manager, cainjector, webhook) should be Running
   

2. If cert-manager is not installed, install it:

   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.1/cert-manager.yaml
   kubectl wait --for=condition=Available deployment/cert-manager-webhook -n cert-manager --timeout=120s
   

3. Check the Certificate status:

   kubectl get certificate -n attune-system
   # Status should be True (Ready)
   

4. If the Certificate is not ready, check the cert-manager logs:

   kubectl logs -n cert-manager deploy/cert-manager --tail=20
   

Webhook timeout

Symptom: Policy creation takes 30 seconds then fails with timeout.

Cause: The webhook pod is running but the cainjector has not patched the CA bundle into the webhook configuration yet.

Fix: Wait for cainjector to inject the CA bundle (usually resolves within 1-2 minutes after cert-manager is ready):

kubectl get validatingwebhookconfiguration -o yaml | grep caBundle | head -1
# If empty, cainjector has not run yet. Wait and retry.

Resize failures

Resize subresource not found (K8s 1.32)

Symptom: Operator logs contain the server does not allow this method on the requested resource or pod resize subresource is not enabled when attempting a resize.

Cause: On Kubernetes 1.32, the In-Place Pod Resize feature is alpha and disabled by default. The /resize subresource is only available when the InPlacePodVerticalScaling feature gate is enabled on all control plane components and kubelets.

Check first: kubectl attune doctor reports whether the cluster is 1.32+ and whether discovery lists pods/resize. The optional cgroup v2 row is WARN and inconclusive on k3s, kind, and most managed clusters. It does not change the exit code.

Fix: Enable the feature gate on all components. For managed clusters, check your provider's documentation. For self-managed clusters:

# API server, controller-manager, and scheduler flags:
--feature-gates=InPlacePodVerticalScaling=true

# Kubelet config (on every node):
featureGates:
  InPlacePodVerticalScaling: true

On Kubernetes 1.33+, this feature gate is enabled by default and no action is needed.

Doctor cgroup v2 is WARN

Symptom: kubectl attune doctor prints cgroup v2 WARN [optional] with could not determine.

Cause: Kubernetes has no Node field for the runtime cgroup version. Doctor does not exec onto nodes. The default on k3s, kind, and most managed clusters is inconclusive.

What it is not: A required FAIL. Optional WARN does not change the exit code. An NFD label feature.node.kubernetes.io/kernel.config.CGROUP_V2=true is kernel compile-time only; doctor never treats that as Pass.

On 1.37+: The same WARN mentions kubelet failCgroupV1, which defaults to rejecting cgroup v1.

Recommendation looks wrong or never resizes

Symptom: kubectl attune recommendations shows values you do not trust, or Auto/Canary never applies them.

Diagnose:

kubectl attune explain -n <ns> <policy>
kubectl get attunepolicy <policy> -n <ns> -o jsonpath='{.status.conditions}' | jq .
kubectl attune history -n <ns>

Use the explanation chain (percentile → overhead → confidence → bounds → change filter) to see which stage moved the number. See Recommend mode: reading recommendations for the field table and skip reasons (cooldown, budget, canary, Deferred/Infeasible, stale, schedule).

Common fixes:

• Too aggressive: raise overhead or tighten maxAllowed
• Too conservative / sparse data: wait for more data points or lower minimumDataPoints only for evaluation
• Never resizes with tiny delta: change filter; expected when already near target
• Stuck on node capacity: Deferred/Infeasible section below

CPU percentile stayed high after a boost

Symptom: cpu.startupBoost.excludeFromHistory is true, the boost has ended, and the CPU recommendation is still high.

Cause: Compare attune.io/startup-boost-at with the pod's creationTimestamp. The window starts at that stamp when it is present and not before creation. Samples between creation and the stamp stay, so a spike in that gap still trains the percentile. Samples from the stamp until stamp plus startupBoost.duration plus the rate window are dropped. A missing or unreadable stamp starts at creation, so a spike after that creation window stays. A sample exactly at the cutoff stays.

What it is not: Omitting excludeFromHistory. Nil and false keep every CPU sample. Memory samples are not part of this window.

High CPU after startup samples are excluded

Symptom: cpu.startupBoost.excludeFromHistory is true and the CPU recommendation is still high.

Cause: Deleted pods stay in the series until historyWindow, because there is no creation time or stamp to cut on. A recreated pod keeps samples older than its new creation time. The cutoff is the boost stamp, or creation when that stamp is missing, plus startupBoost.duration plus the rate window, so points at the cutoff still count. A series with no pod label is not filtered (explanation note startupExcluded=skipped).

What it is not: Setting excludeFromHistory to false. False keeps today's percentile and is not the fix for a stuck high recommendation.

Throttling while the long window still looks idle

Symptom: The pod is CPU throttled now, but the long-window percentile still looks idle, so the recommendation does not rise.

Cause: The parent percentile uses the full historyWindow (default 168h) and the busiest hour. A short hot stretch is diluted. Burst detection is a multiplier on that same profile (max > 3 * p95), not a shorter window.

Fix: Set cpu.surge: {} (or memory.surge: {} for memory). That turns the short window on and fills trigger ratio 1.5, percentile 99, and window 30m. The feature stays off until the block is set. A policy that omits surge still inherits surge from AttuneDefaults. To keep a workload off, omit surge on both.

When the short window is in use, explanation.cpu.finalAdjustment (or explanation.memory.finalAdjustment) contains surge. One spike does not switch the window. If the note is absent, the short percentile did not clear the trigger ratio, the short window did not contain enough finite samples, or the long window is still below minimumDataPoints.

SafetyObservation stuck True

Symptom: SafetyObservation stays True after resizes should have finished.

kubectl get attunepolicy <name> -o jsonpath='{range .status.conditions[?(@.type=="SafetyObservation")]}{.reason} {.message}{"\n"}{end}'
kubectl get pods -n <ns> -l attune.io/tracked=true \
  -o custom-columns=NAME:.metadata.name,RESIZED:.metadata.annotations.attune\\.io/resized-at

Reason	Meaning
Observing	safetyObservationPeriod has not elapsed. Wait, or check for OOM/restarts (those revert early).
Evaluating	Period elapsed; CheckPod, revert, or cleanup is still in progress. Also means a throttle grace period or an SLO evaluation window is still open.
RestorePending	Live pod already matches the original snapshot; template restore after AfterSuccessfulResize is retrying.
Incomplete	Tracking keys exist but attune.io/resized-at is missing or not RFC3339.

The condition is computed from those annotations each reconcile. It is removed when no matching tracked pods remain.

Deferred or Infeasible resize (stuck pods)

Symptom: Policy status shows ResizeBlocked=True, or:

kubectl get attunepolicy <name> -o jsonpath='{.status.workloads}' | jq .
# deferred > 0 and/or infeasible > 0

kubectl get attunepolicy <name> -o jsonpath='{range .status.conditions[?(@.type=="ResizeBlocked")]}{.reason} {.message}{"\n"}{end}'

kubectl attune history -n <ns>
# Failed rows with reason "infeasible"

Signal	Meaning	Operator behavior
Deferred	Kubelet accepted the request but cannot apply it yet (often free request capacity on the node). Pod condition PodResizePending reason Deferred.	Pod is not eligible for a new resize until the condition clears. Retry: every reconcile after eligibility returns (no extra config). Safety observation does not finish while the resize is Deferred or PodResizeInProgress.
Infeasible	Kubelet cannot complete the resize in-place on this node.	With default resizeMethod: InPlaceOnly, skip + history Failed/infeasible + event InfeasibleBlocked. With InPlaceOrRecreate, attempt eviction fallback (PDB / last live Running replica). Infeasible does not hold the safety observation clock.

attune.io/resize-apply-pending=true means the original observation period already elapsed while the resize was still Deferred or in progress. Tracking stays. When the kubelet clears that condition, the observation period starts again. An in-progress condition that has been true for an hour does not start it again. A resize that finishes during the original period does not write this annotation.

Metrics (see metrics reference):

attune_pods_deferred{namespace="...", policy="..."}
attune_pods_infeasible{namespace="...", policy="..."}
histogram_quantile(0.95, sum by (le) (rate(attune_deferred_age_seconds_bucket[15m])))
rate(attune_infeasible_skipped_total[15m])
sum by (result) (rate(attune_eviction_total[15m]))

Fix:

1. Deferred: free capacity on the node, wait for other pods to scale down, or reduce the recommended increase (maxAllowed / change caps). No restart is required for Attune; it retries automatically when the condition clears.
2. Infeasible: free node capacity, lower bounds, or enable eviction fallback:

spec:
  updateStrategy:
    resizeMethod: InPlaceOrRecreate  # opt-in eviction when in-place is impossible
  cpu:
    maxAllowed: "2000m"  # keep increases within typical node headroom

1. Confirm the live pod condition:

kubectl get pod <pod> -o jsonpath='{range .status.conditions[?(@.type=="PodResizePending")]}{.reason} {.message}{"\n"}{end}'

Retry policy (current defaults):

• Deferred: skip until kubelet clears PodResizePending; next reconcile retries. No max deferred age cut-off (watch attune_deferred_age_seconds and ResizeBlocked message for escalation).
• Infeasible + InPlaceOnly: skip every cycle until the condition clears or you change resizeMethod / capacity.
• Infeasible + InPlaceOrRecreate: one eviction attempt per container resize path. History reason is eviction_last_replica when only one live Running replica remains, eviction_denied when the Eviction API rejects (PDB), or eviction_list_failed / eviction_no_selector when the live count cannot be taken. The next cycle may try again. If templatePersistence.when=AfterSuccessfulResize is enabled, a successful Eviction also patches the template so the replacement is not supposed to boot at the old request.

InPlaceOrRecreate but no eviction

Symptom: resizeMethod is InPlaceOrRecreate, the pod is Infeasible, but it is not evicted. History shows Failed / eviction_last_replica. Event EvictionBlocked.

Cause: Only one live Running replica. The last-replica guard lists pods through the typed Clientset and counts status.phase=Running with no deletion timestamp. spec.replicas and NotReady or Pending pods do not count.

HPA ScaledToZero=True is not a manual scale-down. Attune skips apply for that workload (including eviction) so leftover Running pods are not evicted to finish scale-to-zero. A Deployment with spec.replicas greater than 0 and status.replicas 0 is still active.

Fix: Scale until at least two pods are Running, or wait for another replica to become Running.

HPA ScaledToZero left leftover pods at old requests

Symptom: HPA reports ScaledToZero, one or more pods are still Running, and Attune does not resize or evict them.

Cause: Idle skip. HPA ScaledToZero (or spec.replicas==0) skips executeResizes for that workload: no in-place resize, template persist, startup boost, or eviction.

Fix: This is expected. When HPA scales the workload back up, new pods get CREATE initial sizing, and running pods are eligible again once the ScaledToZero condition is no longer True.

OneShot skipped the first replica but others still need a resize

Symptom: OneShot did not resize the first listed replica, but other replicas still need a resize. Events show ResizeSkipped for QoS, node pressure, quota, or Infeasible plus InPlaceOnly.

Cause: OneShot applies at most one needing pod per cycle. Replicas that are already at the applied target, or that are blocked by QoS, node pressure, quota, or Infeasible plus InPlaceOnly, are skipped so another replica can still resize.

Fix: Check events on the skipped replica. The next needing replica in the same cycle should still resize. Remaining needing replicas wait for later cycles and cooldown.

Pod-level resource envelope blocked an increase

Symptom: Events show ResizeSkipped with "pod-level resource envelope would be exceeded". History reason is envelope_constraint.

Cause: The pod already has spec.resources (an aggregate CPU and memory envelope). Raising a container request past that envelope is rejected by the API. Attune does not invent an envelope. When in-place pod-level resize is on, one /resize writes the container target and raises envelope requests to cover the container sum. Existing envelope limits are raised only as far as needed (max of the current limit, the needed request, and the max single container limit, never the sum of limits) and stay at least the request. When the field exists but in-place raise is off, an increase that would exceed the current envelope is skipped. Decreases that stay under the envelope still apply. Under RequestsOnly plus Burstable, Attune skips rather than lift an existing envelope limit (that limit is a user cap).

Fix:

1. Confirm the pod has spec.resources (kubectl get pod <name> -o yaml).
2. On clusters with in-place pod-level resize, the next eligible increase should raise the envelope in the same resize.
3. If in-place raise is off, lower minAllowed / maxAllowed so the recommendation stays under the envelope, or raise the envelope in the workload template.
4. For RequestsOnly plus Burstable, switch that resource to controlledValues: RequestsAndLimits if the envelope limit should move with the recommendation.

QoS class change blocked

Symptom: Operator logs Skipping resize: would change QoS class from Burstable to Guaranteed, from BestEffort to Burstable, or from Guaranteed. The event is ResizeSkipped. The pod is not evicted.

Cause: The resources that would be sent leave the pod's current class. RequestsOnly keeps existing limits, so raising a request up to a limit that already matches can move Burstable to Guaranteed. Adding any positive request or limit moves BestEffort to Burstable. Init containers, including completed ones, count. A Guaranteed pod is blocked when the sent requests and limits would no longer match.

A Guaranteed pod will not accept a CPU multiplier above 1 in place. The CPU request and limit would differ, so the resize is skipped and the pod is not evicted. resizeMethod: InPlaceOrRecreate does not evict that pod either. RequestsOnly does not apply the multiplier.

A memory multiplier on an already Guaranteed pod sets the limit to the multiplier times the engine request, then raises the memory request to that limit. The resize proceeds and the pod stays Guaranteed. That applied request can exceed memory.maxAllowed. The excess is not a failed cap. maxAllowed capped the engine request. It does not cap this raise. Capping only the request would make the pod Burstable, and the QoS skip would then block the resize.

Fix: For Burstable, raise the limit that already equals the new request, or choose a request that stays below that limit. Set controlledValues: RequestsAndLimits only when the result stays Burstable. For BestEffort, leave requests and limits unset. For Guaranteed, use controlledValues: RequestsAndLimits so requests stay equal to limits. On Kubernetes 1.33, a memory limit decrease may also need resizePolicy: RestartContainer.

Fractional memory limit warning

Symptom: The operator log shows an API server warning that a memory limit has a fractional byte value, such as 1301300m, when Attune patches a Deployment or StatefulSet template (template persistence) with memory.controlledValues: RequestsAndLimits. The operator logs these warnings. They do not appear in kubectl output or as events. In-place pod resizes return no warning. A pod created through initial sizing warns the controller that creates it, so that warning appears in the kube-controller-manager log, not the operator log. kubectl attune explain can also show a memory request with a fractional byte value, such as 1395864371200m, when a decimal memory bound and the change cap apply in the same cycle.

Cause: Earlier releases scaled the memory limit by a non-integer ratio and kept millibyte precision. Both an explicit memory.limitMultiplier and the live request-to-limit ratio could do this. The change filter did the same for a capped memory request when the recommended quantity was decimal, for example a 2G or 500M bound.

Fix: Upgrade. Scaled memory limits are now rounded up to a whole byte. A pod or template that already carries a fractional limit is not changed for this alone; the next resize that changes the limit for another reason writes a whole-byte value. A capped memory request is rounded up to a whole byte as well. A request that does not hit the cap is not rounded, and a pod or template that already carries a fractional request is not resized for this rounding alone. See Upgrading and whole-byte capped memory requests.

ResourceQuota exceeded

Symptom: Operator logs Skipping resize: quota/limitrange violation with a message mentioning exceed ResourceQuota.

Cause: The resize would increase CPU or memory requests beyond the remaining headroom in the namespace's ResourceQuota. Attune uses requests.cpu / requests.memory when those names are present, or the cpu / memory aliases when a quota uses the short names.

Fix:

1. Check current quota usage:

   kubectl get resourcequota -n <namespace>
   

2. Either increase the quota limits, or tighten the policy's resource bounds so recommendations stay within quota.

Resize skipped: quota list unavailable

Symptom: Operator logs quota list unavailable; skipping request increase or ResourceQuota list unavailable / LimitRange list unavailable. Request increases are skipped.

Cause: The controller could not list ResourceQuotas or LimitRanges in the pod namespace, usually because the operator ServiceAccount is missing list/watch on those resources. Request increases fail closed so a quota cannot be exceeded while the check is unavailable. Decreases still proceed.

Fix:

1. Confirm the operator ServiceAccount can list quotas and limit ranges in the workload namespace:

   # Helm release "attune" → ServiceAccount "attune" in the operator
   # namespace (typically attune-system). Raw manifests use
   # attune-controller-manager instead.
   kubectl auth can-i list resourcequotas -n <ns> \
     --as system:serviceaccount:<operator-ns>:attune
   kubectl auth can-i list limitranges -n <ns> \
     --as system:serviceaccount:<operator-ns>:attune
   

2. Grant list and watch on resourcequotas and limitranges if either command returns no.

Revert issues

High revert rate

Symptom: Degraded condition is True with reason HighRevertRate, or multiple entries in .status.resizeHistory show result: Reverted.

Cause: 3+ of the last 5 resize operations were reverted due to safety violations. The controller applies exponential backoff (2x cooldown per consecutive revert, capped at 16x).

Check the current backoff state:

kubectl get attunepolicy <name> -o jsonpath='{.status.cooldown}'
# Example: {"backoffMultiplier":8,"consecutiveReverts":3,"effectiveCooldown":"8h0m0s"}

Fix: Investigate the revert reasons:

kubectl get attunepolicy <name> -o jsonpath='{.status.resizeHistory}' | \
  jq '[.[] | select(.result=="Reverted")]'

Common causes:

• oomkill: overhead is too low for memory. Increase memory.overhead.
• throttle: CPU throttle ratio exceeded 50% post-resize. Increase cpu.overhead.
• restart: the application crashes at the new resource level. Check application logs.
• notready: readiness probe fails post-resize. Verify probe configuration.
• slo:<name>: an SLO guardrail query breached its threshold after resize. Review the guardrail's PromQL query and threshold in updateStrategy.sloGuardrails.
• annotation-persist-conflict: the tracking merge patch itself conflicted three times. A kubelet status update does not cause this. Check operator logs for Annotation update conflict on that pod.

Fleet report export failures or empty fleet dashboard

Symptom: attune_fleet_report_export_total{result="failed"} increases, or the fleet Grafana dashboard shows no series when filtering by cluster.

Cause:

1. Fleet report is disabled (default) or the ConfigMap write failed (RBAC / wrong namespace).
2. Federated dashboards require Prometheus external_labels.cluster on each cluster scrape. Without that label, cluster=~"$cluster" panels stay empty.
3. Operator watchNamespaces is set, so the report only includes a subset of policies.
4. estimatedMonthlySavingsUSD is 0 even though some policies show a savings string. The rollup only adds parseable dollar amounts. Empty, non-numeric, NaN, and Inf values count as 0. Check unparseableSavings in report.json (non-zero means values were dropped).

Fix:

1. Enable export: Helm fleetReport.enabled=true (or --fleet-report-enabled).
2. Check the ConfigMap: kubectl -n <release-ns> get cm attune-fleet-report -o yaml
3. Set global.external_labels.cluster on each Prometheus; reload federation.
4. Use leader election with HA when fleet report is enabled.
5. If the USD total is 0, inspect status.savings.estimatedMonthlySavings on each policy. Fix costPricing so the string is a finite number such as $12.50.

sum(rate(attune_fleet_report_export_total{result="failed"}[5m]))

Resize skipped for node capacity or pressure

Symptom: Events show ResizeSkipped with "exceed node allocatable", "node free request budget exceeded by neighbors", "node has MemoryPressure/DiskPressure/PIDPressure", or "node status unavailable", and attune_capacity_skip_total increments (reason label: allocatable, neighbors, pressure, or unavailable).

Cause: Always-on safety gates refuse request increases that would make this pod's total requests exceed the node's allocatable, that would not fit after other pods on the node have reserved requests, that would raise requests while the node is under pressure, or when the Node object cannot be loaded (API/RBAC failure). Decreases still proceed.

Fix:

1. Free capacity on the node (evict low-priority pods) or move the workload.
2. Lower maxAllowed / change caps so recommendations fit typical node shapes.
3. For DaemonSets, size against the smallest node pool that runs them.
4. For unavailable: check operator RBAC for nodes get/list/watch and apiserver health; Attune fails closed on increases until the node is readable.
5. Inspect formulas in Node capacity.

sum by (namespace, policy, reason) (rate(attune_capacity_skip_total[1h]))

Utilization drifted after a second CPU resize

Symptom: The HPA CPU utilization target moved again after a later resize, and attune.io/original-cpu-request is larger than the pod CPU total from before the first Attune resize.

Cause: A stored full-pod base was replaced with a grown pod sum, or the no-history containers were added onto a base that already included them. A base that records its containers (attune.io/original-cpu-request-containers or attune.io/original-memory-request-containers) adds only a container missing from that list. A base with no list is replaced when no counted history row grew and the quantity is below the pre-resize pod sum and equals that sum with native sidecars removed, or the history-old sum. When a counted row grew until its old request equals either of those sums, the quantity stays and the container list is recorded. A quantity an older operator already doubled is left as stored.

Fix:

1. Read attune.io/original-cpu-request or attune.io/original-memory-request on the HPA, and the matching containers annotation when it is set.
2. If the stored quantity is already the doubled base, delete the target and request annotations for that resource: attune.io/original-target-cpu and attune.io/original-cpu-request, or attune.io/original-target-memory and attune.io/original-memory-request. Delete the containers annotation too if it is set. Leave attune.io/auto-tune: "true". The next successful resize of that resource stores a fresh base, the container list, and moves the live target. A retune already in progress reads the HPA again when its update conflicts. The base it stores is the one on that read, so a reset that landed before the successful update is kept. Read the annotations again after that update finishes.
3. Do not expect this version to restore the number from before that rewrite.

HPA target did not change after a resize

Symptom: Pods were resized and the HorizontalPodAutoscaler utilization target is still the old percent.

Cause: The HPA update failed, or the computed integer percent is already the current target. A conflict reads the HPA again and decides from that object. A status-only conflict still publishes the resize target. A target or Attune annotation edited on the re-read object is the input to that decision. Any other error is logged as Failed to update HPA target and is not retried until that workload is resized again. The first time a resize leaves the percent unchanged, Attune still stores the pre-resize request (attune.io/original-cpu-request, attune.io/original-memory-request, or attune.io/original-container-memory). That write does not log HPA utilization target unchanged after resize. The next resize scales from that stored request, not from the request after the resize that left the percent unchanged. Verbosity 1 logs HPA utilization target unchanged after resize when the percent stays put and there is nothing new to store: the base is already stored, the request did not change, or the stored base did not parse. A recognized metric with no usable request base logs HPA auto-tune skipped a metric with no usable request base. A stored base that is set but does not parse logs HPA auto-tune ignored a stored base that did not parse, and this cycle uses the live percent. Those lines are off at the default log level. A base that was already stored is left in place.

Fix:

1. Check the operator log for Failed to update HPA target.
2. Confirm the HPA has attune.io/auto-tune: "true".
3. If this was the first retune and the percent did not move, read the original-request annotation. A value there is the pre-resize request. Later resizes use it.
4. The next successful in-place resize of that workload tries the retune again.
5. To see why a later percent stayed put, run the operator with -v=1 and search for the messages above. Do not delete a stored base only because the percent did not move.

HPABaseRepaired and the stored request disagree

Symptom: The policy shows Warning HPABaseRepaired, but attune.io/original-cpu-request or attune.io/original-memory-request on the HPA is not the value you expected from that event.

Cause: This version emits the Warning only after the HPA update succeeds. The event names the annotation it rewrote, and it describes the attempt that succeeded. A reset already present on the HPA when that update runs is not reported as a repair of the previous base. A Warning whose annotation does not match is from an older operator, or a later writer changed the annotation. It is not a failed write from this version.

Fix: Read the annotation named in the event. That is attune.io/original-cpu-request or attune.io/original-memory-request. Do not treat the Warning alone as proof of the stored value.

Memory HPA scaled out after a memory decrease

Symptom: Attune lowered a memory request and the HorizontalPodAutoscaler added replicas. The HPA memory utilization target is still the old percent.

Cause: Auto-tune is off for that HPA. Attune retunes memory utilization targets only when the annotation attune.io/auto-tune is "true". Without it, usage divided by the smaller request crosses the old target and the HPA scales out.

Fix:

1. Annotate the HPA with attune.io/auto-tune: "true". Attune writes the new percent on the next successful in-place resize. If replicas already grew, set the current utilization percent on the HPA now.
2. Confirm the metric is Utilization, not AverageValue. Object and External metrics are left unchanged.
3. Optional: set updateStrategy.hpaTargetBounds.memory.max when the retuned percent should stay under a ceiling. This does not turn auto-tune on. 50 and 90 are a common choice, not a default.

OOM after memory limit decrease

Symptom: After enabling memory decreases (memory.allowDecrease: true and controlledValues: RequestsAndLimits) on Kubernetes 1.34+, pods OOMKill when limits shrink, Events show MemoryLimitUsageFloor, metrics show attune_memory_limit_decrease_total{result="clamped_usage"} or skipped_unsafe, or the opt-in AttuneMemoryLimitUnsafe alert fires.

Cause: Live memory limit decreases race with usage spikes. Attune floors the target limit above recent usage (recommendation raw percentile) plus memory.decreaseUsageMarginPercent (default 10%). If OOM still occurs, the margin or overhead is too low, or usage is spikier than the percentile window.

Fix:

1. Raise margin: memory.decreaseUsageMarginPercent: 20 (or higher).
2. Raise memory.overhead so recommendations (and limits) stay further above steady usage.
3. Keep memory.maxDecreasePercent modest so large drops step down over multiple cycles.
4. Confirm cluster version is 1.34 or newer; on 1.33 and earlier, limit decreases are platform-clamped (result="clamped_platform").

# Floored or blocked memory limit decreases
sum by (namespace, policy, result) (
  rate(attune_memory_limit_decrease_total[1h])
)

Memory request rose after OOMKilled

Symptom: A container memory request increased after OOMKilled. This happens only when memory.oomBump is set. An empty oomBump: {} is enough to turn it on. Omitting the block leaves memory on the percentile path.

Read the annotation:

kubectl get pod <name> -n <ns> -o jsonpath='{.metadata.annotations}' | jq .

The key is attune.io/oom-bump.<container>. The container name must fit so the name segment oom-bump.<container> is at most 63 characters. The value is count=<n>,origin=<qty>,floor=<qty>,oomAt=<RFC3339>,restart=<n>,holdUntil=<RFC3339>. origin is the live memory request before the first bump of the streak. count increments only after a successful resize. When holdUntil passes, the percentile path resumes, but origin stays.

Which event:

Event	Meaning
OOMBumpCapped	count is already maxBumps. No further step. If maxAllowed is omitted, maxBumps is the only cap.
OOMBumpClamped	The step was above maxAllowed, so the request was clamped to maxAllowed.
IncreaseExceedsBudget	One container's increase is larger than maxMemoryIncreasePerMinute or maxTotalMemoryIncrease. Waiting does not help. The bump annotation is not written. Raise the cap or lower the step.

Info alert: AttuneOOMBumpCapped does not mean the bump failed to apply. It means the result was capped or clamped. One such sample in the last hour is enough. The alert is info and waits 5m unless oomBumpCapped.for is set. The PrometheusRule object stays off until metrics.prometheusRule.enabled is true. The alert does not turn memory.oomBump on.

A Guaranteed pod with controlledValues: RequestsOnly is skipped (ResizeSkipped, metric result="skipped") instead of evicted. Attune does not evict to change QoS.

Limit left below the request, or a request-only limit changed: During hold, a throttle, NotReady, or SLO revert keeps the memory request at the floor. The limit moves up to that floor only when the container's effective controlledValues is RequestsAndLimits. A container set to RequestsOnly keeps its limit, even when policy memory is RequestsAndLimits. A RequestsAndLimits container is raised when policy memory is RequestsOnly. A missing or zero limit is not created. oomBump is not a per-container field.

sum by (namespace, policy, result) (rate(attune_oom_bump_total[1h]))

Request did not move after a second OOM

Symptom: memory.oomBump is set, the container was OOMKilled again while holdUntil is still in the future, and the memory request stayed put.

Cause: The live request was already above the next step from the original origin. A current operator takes one step above that live request. The original origin stays. maxBumps and maxAllowed still cap the step. Auto, OneShot, and Canary record this oomAt and restart on the pod, so the same OOM is not counted again. Recommend and Observe do not write that stamp.

When maxAllowed is already at or below the live request, the request cannot rise. Auto, OneShot, and Canary store that signal on the pod, so it counts once as skipped and count does not increase. Recommend and Observe do not write the stamp, so the same OOM still increments skipped on every reconcile. A climb of this in-hold signal in Auto, OneShot, or Canary means an older operator counted that same OOM on every reconcile. A budget skip, or a Guaranteed pod with RequestsOnly, still increments skipped on every reconcile on a current operator.

Auto, OneShot, and Canary resize to the new step. Recommend stores the recommendation and does not resize. Observe does not resize.

sum by (namespace, policy, result) (increase(attune_oom_bump_total[1h]))

Skipped samples climbed while the request was already at maxAllowed

Symptom: memory.oomBump is set, holdUntil is not in the future, the live memory request is at or above memory.maxAllowed, and attune_oom_bump_total{result="skipped"} climbed for hours.

Cause: Older builds counted that OOM on every reconcile and did not store it. A current build stores the signal and counts one clamped sample. The request does not rise. With the chart PrometheusRule enabled, AttuneOOMBumpCapped can fire for that sample. One clamped sample in the last hour is enough for the alert.

Fix: Raise memory.maxAllowed if the workload needs a higher request.

sum by (namespace, policy) (increase(attune_oom_bump_total{result="clamped"}[1h]))

Request stayed above maxAllowed during an OOM hold

Symptom: memory.oomBump is set, holdUntil is still in the future, memory.maxAllowed is below the stored floor, and the live memory request is still above that cap.

Cause: Default memory allowDecrease is false. The operator publishes the cap, then the decrease gate keeps the highest in-hold pod request. That request is not the workload template. Template persistence defaults to off, so the pod can stay above both the template and the cap. explanation.memory.finalAdjustment on the recommendation contains allowDecrease=false. The stored floor is not rewritten, and the hold is not extended. attune_oom_bump_total{result="clamped"} increments once for that floor and cap per operator process, not on every reconcile. Recommend keeps the live request in status and does not resize. Observe does not resize. When memory.allowDecrease is true, the recommendation shows the cap, and Auto, OneShot, and Canary can resize down to it.

Omitting maxAllowed does not cap the held floor.

Fix: Set memory.allowDecrease to true if the request should fall to the new cap during the hold.

sum by (namespace, policy) (increase(attune_oom_bump_total{result="clamped"}[1h]))

Revert failures

Symptom: Entries in .status.resizeHistory show result: Failed, or attune_revert_failures_total is incrementing.

Cause: The operator detected a safety issue (OOMKill, throttle, etc.) and tried to revert the pod to its original resources, but the /resize subresource call failed. The pod remains at the post-resize resource level.

Fix: Check operator logs for the revert error:

kubectl logs -l app.kubernetes.io/name=attune --tail=100 | grep "Failed to revert"

Common causes:

• Conflict: another controller (HPA, VPA) is modifying the same pod. Use attune_revert_failures_total to track frequency.
• Pod evicted: the pod was evicted between the safety check and revert.
• RBAC: the operator ServiceAccount lacks update on the pods/resize subresource.

# Alert when reverts are failing
sum by (namespace, workload) (rate(attune_revert_failures_total[5m])) > 0

Template restore after safety revert

Symptom: The live pod is back at the original resources after a safety revert, but the workload template still has the persisted post-resize requests. .status.resizeHistory still shows result: Success for that resize. There is no TemplatePatchFailed event. attune_revert_failures_total stays 0.

Cause: The in-place /resize revert succeeded, then restoring the AfterSuccessfulResize template failed. The helper logs Failed to restore template after safety revert and returns before attune_reverts_total and the Reverted event. Tracking annotations stay on the pod so the next reconcile retries.

Fix: Grep operator logs for the restore-fail line (not Failed to revert):

kubectl logs -l app.kubernetes.io/name=attune --tail=200 | grep \
  "Failed to restore template after safety revert"

A zero attune_revert_failures_total does not mean restore finished. That counter counts failed /resize revert calls only, including observation-window revert. Check attune_reconcile_errors_total{error_type="safety_observation"} if list, confirm, cleanup, or template restore also failed. Tracking stays; the next reconcile retries the restore even when the pod is Ready again, as long as live resources still match the applied revert target (clamped original, not always the raw snapshot).

Safety observation stuck

Symptom: Pods keep resize tracking annotations after the observation period, or attune_reconcile_errors_total{error_type="safety_observation"} is incrementing.

Cause: Listing tracked pods, confirming an unsafe verdict, removing tracking annotations, or restoring the template after a safety revert failed. The operator keeps observationsPending so the next reconcile retries. Tracking stays until cleanup succeeds.

Fix: Grep operator logs:

kubectl logs -l app.kubernetes.io/name=attune --tail=200 | grep -E \
  "Failed to list pods for safety observation|Failed to remove resize tracking annotations|Safety confirm Get failed|Failed to restore template after safety revert"

sum(rate(attune_reconcile_errors_total{error_type="safety_observation"}[5m])) > 0

Resizes not happening during expected window

Symptom: Operator logs "Outside resize window, skipping resize" even though you expect the window to be open.

Cause: The schedule.timezone does not match your local time. Windows are evaluated in the configured timezone (default: UTC).

Fix: Verify your timezone is correct:

schedule:
  windows:
    - start: "02:00"
      end: "06:00"
  timezone: "America/New_York"  # not UTC

Check the current time in the configured timezone:

TZ="America/New_York" date "+%H:%M %A"

Budget exhausted

Symptom: Operator logs "Budget exhausted, deferring resize to next cycle" and some pods are not resized.

Cause: The total CPU or memory increase across all pods exceeds the budget left this cycle (maxTotalCpuIncrease / maxTotalMemoryIncrease) or the budget available this minute (maxCpuIncreasePerMinute / maxMemoryIncreasePerMinute). The policy event reason is BudgetExhausted.

Fix: Either raise the cap your policy sets (both families apply when both are set) or accept that resizes are spread across multiple reconcile cycles (this is the intended behavior for gradual rollout).

If the event reason is IncreaseExceedsBudget, one container's increase is larger than the configured cap. The bucket only holds one minute of maxCpuIncreasePerMinute / maxMemoryIncreasePerMinute, and maxTotalCpuIncrease / maxTotalMemoryIncrease reset to the same number every cycle, so that resize will not run until the cap is raised or the recommendation shrinks. It is not waiting for the next cycle.

updateStrategy:
  maxCpuIncreasePerMinute: "4000m"    # 4 cores per minute
  maxMemoryIncreasePerMinute: "8Gi"   # 8 GiB per minute
  # maxTotalCpuIncrease / maxTotalMemoryIncrease are the older per-cycle
  # cap (deprecated); raise them too if your policy still sets them.

Apply rejected: defaults min above policy max

Symptom: kubectl apply fails with:

memory minAllowed (2Gi) from AttuneDefaults "global" is above maxAllowed (1Gi) on the policy

A container ceiling uses the same sentence with containerPolicies "app".

Cause: The policy or container sets maxAllowed and omits minAllowed. AttuneNamespaceDefaults or AttuneDefaults supplies a higher min. Admission checks that pair on create and on update.

Effect: The apply does not change the object. An object already stored keeps its spec. Recommendations for that stored pair are clamped to the non-zero max. boundsApplied is max.

Fix: Lower the defaults min, raise the policy or container max, or set an explicit min that is at or below the max. kubectl attune explain can show the merged pair. Explain does not reject the apply.

One container missing under a capped startup query

Symptom: cpu.startupBoost.excludeFromHistory is true, Ready is PrometheusSeriesCapped, and one container has no CPU samples.

Cause: That CPU query is one series per pod and container. The cap keeps one series per pod before any pod gets a second series, and chooses the container with fewer series already kept. A cap smaller than the number of containers still leaves some containers out. samplesForContainer does not borrow another container's series.

Fix: Raise maxPrometheusSeries / --max-prometheus-series above the number of pods times the number of containers you need, or shorten historyWindow so replaced pods leave the matrix. Omitting excludeFromHistory uses the default per-container cap instead.

Policy rejected: unknown namespace for the cache

Symptom: kubectl apply of an AttunePolicy fails with:

listing AttuneNamespaceDefaults in team-b: unable to list: team-b because of unknown namespace for the cache

Cause: watchNamespaces is set, and the running controller read AttuneNamespaceDefaults from the informer cache. The validating webhook has failurePolicy: Fail and no namespace selector, so a policy outside the list was denied when it set maxAllowed without minAllowed, or a surge window without metricsSource.historyWindow.

Fix: Upgrade the controller. Current admission reads AttuneDefaults and AttuneNamespaceDefaults from the API server. The controller still does not reconcile namespaces outside the list. A defaults min above the policy max is still rejected.

Policy rejected: invalid schedule timezone

Symptom: kubectl apply fails with:

admission webhook "validation.attune.io" denied the request:
updateStrategy.schedule.timezone "PST" is not a valid IANA timezone

Cause: The timezone must be a valid IANA timezone name from the tz database. Common mistakes include using abbreviations that Go's time.LoadLocation does not recognize.

Fix: Use the canonical IANA region/city name:

Invalid	Valid alternative
PST	America/Los_Angeles
IST	Asia/Kolkata

Note: US/Eastern, EST, and CET are valid IANA timezone links and will be accepted, but the canonical forms (America/New_York, Europe/Berlin) are recommended for clarity.

# List all valid timezones on your system:
timedatectl list-timezones

Policy rejected: invalid day of week

Symptom: kubectl apply fails with:

admission webhook "validation.attune.io" denied the request:
updateStrategy.schedule.daysOfWeek contains invalid day "Wed"

Cause: Day names must be the full English name. Abbreviations and non-English names are not accepted.

Fix: Use the full name (case-insensitive):

schedule:
  daysOfWeek: ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"]

Valid values: Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday.

Policy rejected: invalid CloudWatch clusterName

Symptom: kubectl apply fails with:

admission webhook "validation.attune.io" denied the request:
metricsSource.cloudwatch.clusterName: clusterName must be an EKS-style name

Cause: clusterName is interpolated into a CloudWatch SEARCH expression. Only EKS-style names are accepted: 1-100 characters, start with alphanumeric, then alphanumeric, hyphen, or underscore. Quotes, spaces, and other SEARCH metacharacters are rejected.

Fix: Use the EKS cluster name as shown in the AWS console (for example my-eks-cluster).

Deleting a policy

When you delete a AttunePolicy, the operator uses a attune.io/cleanup finalizer to clean up before the resource is garbage-collected:

1. Annotations removed: all tracking annotations (attune.io/resized-at, attune.io/policy, etc.) and the attune.io/tracked label are removed from pods managed by that policy.
2. Resources retained: pods keep their current (resized) CPU and memory values. The operator does not revert resources to pre-resize values.
3. Gauges cleaned: Prometheus gauge metrics for the policy are removed.
4. Finalizer removed: only after cleanup succeeds. If a pod update fails, the finalizer remains and the controller retries on the next reconcile cycle.

If the policy appears stuck in Terminating, check the operator logs for pod update errors during cleanup:

kubectl logs -n attune-system deploy/attune | grep "deletion cleanup"

Large cluster performance

Start with the Scaling Guide ops checklist (clusterSize, watchNamespaces, CRD window/step/cooldown). The notes below match common symptoms to those knobs.

Stale recommendations (slow reconciliation)

If workqueue_depth is consistently > 0 and workqueue_longest_running_processor_seconds climbs, the operator cannot keep up with the reconcile queue. Solutions (in order of impact):

1. Increase maxConcurrentReconciles (or use a clusterSize preset).
2. Scope with --watch-namespaces to reduce informer cache size.
3. Policies targeting many workloads via label selector process up to 10 workloads in parallel per reconcile cycle.
4. High-replica Deployments: reduce historyWindow and increase queryStep (query payload scales with pods today). See Large Deployments.

See the Scaling Guide for tuning details and preset values.

High memory usage

If the operator pod is OOMKilled or uses unexpectedly high memory, the informer cache may be caching too many objects. Use --watch-namespaces to limit the cache to the namespaces where your policies exist. Also raise operator memory via a clusterSize preset if you intentionally watch a large pod count.

Apply paths skipped due to stale recommendations

When Prometheus does not return fresh data during a reconcile cycle, the operator marks the recommendation as stale and skips apply paths that would act on outdated metrics. A rec is stale when the newest finite sample is older than 3 * queryStep (default 15m), or when the last known rec is reused after an empty query. Reuse expires after that same bound; then the rec drops and Ready can become InsufficientData. Stale recs do not increment workloads.withRecommendations and do not enter savings gauges. Resize is one skipped apply path; startup boost, initial sizing (CREATE webhook), and template persistence are also skipped, the recommendation ConfigMap is not rewritten, and a GitOps apply PR is not opened. You will see this in the operator logs (resize example):

Skipping resize for workload with stale recommendation  workload=my-app

The attune_stale_recommendations_total counter increments each time this happens. Common causes:

1. Prometheus is temporarily unavailable or responding slowly.
2. The historyWindow is too short for the workload's scrape interval, so range queries return no data.
3. Pod label changes caused the PromQL regex to stop matching.

To diagnose, enable debug logging and check the Prometheus query results:

kubectl logs -n attune-system deploy/attune \
  | grep -E "stale|Prometheus query returned no data"

Apply paths resume automatically once fresh data is available.

Deployment-owned ReplicaSet targeting

If a AttunePolicy targets a ReplicaSet that is owned by a Deployment, the operator rejects it with an error:

ReplicaSet my-ns/my-rs is owned by a Deployment; target the Deployment instead

Deployment-owned ReplicaSets are also automatically filtered from selector-based discovery to prevent double-resizing (the Deployment and its child ReplicaSet would both match). To right-size the workload, target the parent Deployment instead.

Known limitations

Maximum Prometheus addresses

The operator caches at most 64 unique Prometheus collector connections. Clusters with more than 64 distinct Prometheus addresses across all policies will see errors on additional addresses. In practice this is rarely hit since most clusters use 1-2 Prometheus instances.

Minimum cooldown floor

A positive cooldown shorter than 1 minute, such as 10s, is raised to 1m. This prevents accidental resource churn. 0s is not raised to 1m. Admission rejects a new 0s, and an object already stored with 0s waits the 1h default, including a policy that inherits 0s from defaults. See Stored cooldown of 0s.

Enabling debug logs

The operator supports multiple log verbosity levels. By default it runs at info level. To enable debug logging:

# Enable debug logs (V(1): queries, pod selection, cache, recommendations)
helm upgrade attune attune/attune \
  --set logging.level=debug

# Enable verbose trace logs (V(2): per-sample data, full recommendation chain)
helm upgrade attune attune/attune \
  --set logging.level=2

You can also switch to human-readable text format for local debugging:

helm upgrade attune attune/attune \
  --set logging.level=debug --set logging.format=text

Revert to normal after debugging:

helm upgrade attune attune/attune \
  --set logging.level=info

NaN or Inf values in Prometheus data

Symptom: Debug logs (V(1)) show messages like All CPU samples were NaN/Inf or All memory samples were NaN/Inf, and the policy remains in InsufficientData state despite Prometheus being reachable.

Cause: Prometheus queries can return NaN (e.g., 0/0 division in rate queries when no samples exist yet) or Inf when scrape data is missing or contains malformed values. The operator filters out non-finite values before computing recommendations to prevent corrupted percentile calculations.

Fix:

1. Check if Prometheus has cAdvisor metrics for your namespace:

   kubectl exec -n monitoring prometheus-0 -- \
     wget -qO- 'http://localhost:9090/api/v1/query?query=container_cpu_usage_seconds_total{namespace="YOUR_NS"}' \
     | head -c 200
   

2. If the query returns data but values are NaN, check for recording rules or relabeling that might divide by zero.

3. Wait for more scrape cycles. NaN values are common during the first few minutes after pod creation when Prometheus has only one data point (rate computation needs at least two).

The attune_nan_inf_samples_total counter increments each time this happens, broken down by container and metric type (cpu or memory). A Datadog gap encoded as JSON null, or a point with no value, is dropped the same way. A numeric zero is kept, because a container can be idle. A Datadog series whose points are all null or missing increments this counter once. Use it to alert on persistent data quality issues:

rate(attune_nan_inf_samples_total[1h]) > 0

Requests clamped to limits

Symptom: Debug logs (V(1)) show Requests clamped to limits with a list of affected resources (e.g., cpu, memory).

Cause: The recommended CPU or memory request exceeds the container's current limit. This happens when controlledValues is set to RequestsOnly (limits stay at their current values) and the recommendation grows beyond those limits. The same leftover dest clamp runs on CREATE initial sizing and persist merge when controlledValues is RequestsOnly (leftover pod or LimitRange dest). Live resize also dest-caps leftover pod dest, not only rec-blob dest. When controlledValues is RequestsAndLimits, CREATE and live startup boost dest-cap the rec dest so leftover dest does not skip the boost. The operator caps the request at the limit to prevent the API server from rejecting the resize or the Deployment/StatefulSet patch.

Fix: Either increase the container's limits, or switch to controlledValues: RequestsAndLimits so the operator can scale limits proportionally with requests.

The attune_request_clamped_total counter increments each time a request is capped on a live apply, on CREATE, or on a persist write that still changes the template. A dest clamp that is already at the applied target does not increment. Use it to detect policies where limits are consistently too tight:

rate(attune_request_clamped_total[1h]) > 0

Sidecar not resized (or proxy resized unexpectedly)

Known sidecar auto-exclude is on (default)

Attune skips well-known mesh and sidecar container names by default (excludeKnownSidecars: true), including istio-proxy, linkerd-proxy, consul-dataplane, kuma-dp, vault-agent, and common Cloud SQL proxy names. Operator logs show reason=known sidecar auto-exclude when this path applies.

If you want to right-size those containers again (previous behavior):

spec:
  excludeKnownSidecars: false

Or set the same field on AttuneDefaults / AttuneNamespaceDefaults for cluster- or namespace-wide opt-out when policies leave the field unset.

Custom sidecars still resized

Names not on the built-in list are still right-sized unless listed in excludedContainers. Add the container name explicitly:

spec:
  excludedContainers:
    - my-company-agent

kubectl attune explain <policy> prints Exclude known sidecars and the effective excluded set (known list union user list).

Sidecar and app share one max

When containerPolicies is omitted, the sidecar and the app use the same spec.cpu.maxAllowed and spec.memory.maxAllowed. That is unchanged. Set containerPolicies when those containers need different ceilings. excludedContainers and excludeKnownSidecars still win: a named entry does not manage an excluded container. istio-proxy stays excluded while excludeKnownSidecars is unset. It is managed only when that field is false and the name is not listed in excludedContainers. A normal init container is not managed. A native sidecar (restartPolicy: Always) is. startupBoost, limitMultiplier, oomBump, and surge stay policy-wide. A container maxAllowed caps the boosted CPU.

spec:
  cpu:
    maxAllowed: "4000m"
  containerPolicies:
    - containerName: sidecar
      cpu:
        maxAllowed: "200m"

Template persistence not updating the workload template

Opt-in updateStrategy.templatePersistence patches Deployment/StatefulSet pod templates so replacement pods start correctly sized. If the live template never changes, check these first.

Feature disabled or wrong when

Default is off. Enable explicitly:

spec:
  updateStrategy:
    type: Auto # or Recommend with when: OnRecommendation
    templatePersistence:
      enabled: true
      when: AfterSuccessfulResize # default when enabled

• AfterSuccessfulResize: after a successful in-place resize, and also after a successful Eviction (InPlaceOrRecreate) so the owner recreates from the updated template. In Recommend/Observe modes this never fires (webhook warns). Use when: OnRecommendation for Recommend. A successful memory-only resize still patches the template (CPU can stay at the original request). Do not require both resources to change. A later safety revert also restores the template to the pre-resize snapshot so rollouts do not keep the unsafe size.
• OnRecommendation: still skipped in Observe mode.
• Canary: template patches wait until canary reaches FullRollout.
• Stale recommendation: recommendations[].stale is true; the template is not patched until fresh Prometheus data.
• Memory usage floor: persist will not write a memory limit below recent usage (raw percentile plus decreaseUsageMarginPercent), even if rec.Current still shows the old template. RequestsOnly persist never writes limits.
• Namespace freeze: attune.io/freeze=true skips new persist. Pending safety restore still runs.
• Safety restore fail: after a successful in-place revert, a failed template restore leaves the pod at original resources, the template at the persisted size, history at Success, and no TemplatePatchFailed event. Operator logs Failed to restore template after safety revert. Tracking stays; the next reconcile retries when live resources match the applied revert target. attune_revert_failures_total does not increment.

Rollout spec.workloadRef

Attune does not patch the Rollout template when spec.workloadRef is set. The pod template lives on the referenced Deployment, StatefulSet, or ReplicaSet.

Tell the two skips apart on status.conditions:

• Reason WorkloadRefUnread means spec.workloadRef has an empty name or an unsupported kind, or the referenced object could not be read or has no containers. The same text is on status.workloadErrors. No recommendation is stored for that Rollout. Fix the name, kind, or RBAC, then wait for the next reconcile. The reason is removed when recorded workload errors no longer include a workloadRef read failure. status.workloadErrors keeps at most 10 entries, so a dropped error can clear it while the read is still failing.
• Reason TemplateWorkloadRef means this reconcile's recorded errors did not include a workloadRef read failure. Leaving the Rollout template alone is intentional. Recommendations use the referenced pod template. Resize of the Rollout's pods still proceeds from that template.

Recommend mode reports the same reasons. Observe mode does not run template persistence. A recorded workloadRef read failure still sets WorkloadRefUnread.

Mid-rollout or no-op

The operator skips patches while a Deployment or StatefulSet is mid-replacement. An Argo Rollout is skipped while it is aborted, Paused, Progressing, or Degraded. Degraded is an aborted rollout, a progress deadline, or an invalid spec, so the skip stays when abort is false and replica counts match. A Healthy phase, or an empty phase, is patched even when updatedReplicas is behind spec.replicas. That lag is a scale-out. A no-op also happens when the template already matches. Events:

• TemplatePatched (Normal) on success
• TemplatePatchFailed (Warning) on API errors

History entries use method=TemplatePersistence and result=TemplatePatched (or Failed). Metric:

rate(attune_template_patch_total{result="failed"}[15m]) > 0

GitOps thrash

If Argo CD / Flux reverts the template every sync, do not enable template persistence under unmanaged sync. Prefer export.configMap or initialSizing (see GitOps integration).

GitOps PR opens empty PRs every cooldown

Symptom: A new GitHub PR appears on each pullRequest.cooldown (default 24h) with only an empty bootstrap commit. The description table matches the previous PR.

Cause: GitOps PR automation writes the drift table in the PR body. It does not patch Deployment YAML. Merging that PR (and deleting the head branch) leaves the live template unchanged, so drift stays true.

Fix:

1. Do not merge empty notification PRs as the apply step.
2. Apply a real patch: kubectl attune diff -n <ns> -o yaml, commit it to git, let Flux/Argo sync. Wait for reason NoDrift.
3. To stop new PRs immediately, set export.pullRequest.enabled: false or dryRun: true.
4. On Attune versions that include the unchanged-drift skip, the condition is PullRequestUnchanged instead of another empty PR. A dry-run of the same table does not block the first live PR (no gitops-pr-url yet). If a 0.1.24 dry-run already wrote attune.io/gitops-pr-last-attempt, delete that annotation or wait out cooldown before the first live open.
5. v0.1.24 wrote attune.io/gitops-pr-drift only after opening a PR. Upgrading from 0.1.22/0.1.23 therefore still opened one more empty PR when cooldown expired (last-attempt and URL were present, the fingerprint was not). Later versions record the live table when a prior PR URL exists and the fingerprint annotation is missing, then skip. Confirm the annotation is present:

kubectl get attunepolicy -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" drift="}{.metadata.annotations.attune\.io/gitops-pr-drift}{" url="}{.metadata.annotations.attune\.io/gitops-pr-url}{"\n"}{end}'

If url is set and drift is empty after upgrade, Flux/Argo may be replacing operator annotations. Current Attune also stores the fingerprint on status.gitopsPR, which GitOps apply does not replace. Helm does not update CRDs on helm upgrade; apply crds.yaml or the API server prunes status.gitopsPR (see Upgrading). Leave the last PR open or set dryRun: true only if status.gitopsPR.driftFingerprint is also empty. Also print:

kubectl get attunepolicy -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" fp="}{.status.gitopsPR.driftFingerprint}{" url="}{.status.gitopsPR.url}{"\n"}{end}'

GitOps PR failing

Symptom: Status condition GitOpsPullRequest is False with reason PullRequestFailed or GitOpsEndpointBlocked, Events or metrics show attune_gitops_pr_total{result="failed"}, or the AttuneGitOpsPRFailures alert fires.

Cause: Opt-in updateStrategy.export.pullRequest could not create or update a PR. Common cases:

1. Token Secret missing, wrong key, or RBAC cannot read Secrets.
2. Invalid provider / repository / optional apiUrl. Preflight rejects a non-HTTPS URL, userinfo, loopback, link-local (including IMDS), or a private RFC1918/ULA host when allowPrivateEndpoints is false. Dial-time SSRF still blocks loopback, link-local, and IMDS even when allowPrivateEndpoints: true. Set that flag only for self-hosted forges on RFC1918/ULA. GitOps apiUrl is stricter than Prometheus: in-cluster loopback and metadata addresses stay blocked.
3. Forge API error (auth, branch protection, missing head branch before bootstrap, rate limits). Status is the static message PR API request failed (no API body).

Match the condition message to the check that failed:

Message	When
apiUrl is not an allowed HTTPS host	Preflight on apiUrl (reason GitOpsEndpointBlocked)
GitOps API URL resolved to a disallowed address	Dial-time SSRF (reason GitOpsEndpointBlocked)
PR API request failed	Forge API error after the URL was allowed (reason PullRequestFailed)

Fix:

1. Inspect the condition message (never paste tokens into tickets):

kubectl get attunepolicy <name> -n <ns> \
  -o jsonpath='{range .status.conditions[?(@.type=="GitOpsPullRequest")]}{.reason}{" "}{.message}{"\n"}{end}'

1. Confirm Secret name/key and that the operator ServiceAccount can get that Secret.
2. Use dryRun: true first to validate drift detection without forge calls.
3. Check PromQL:

sum by (namespace, policy, result) (rate(attune_gitops_pr_total[1h]))

See GitOps integration: pull request automation.

Helm install ImagePullBackOff

Symptom: After helm install from oci://ghcr.io/attune-io/charts/attune, the operator pod stays 0/1 with ErrImagePull or ImagePullBackOff for ghcr.io/attune-io/attune:0.1.x (no v prefix).

Cause: Chart appVersion is SemVer without v. Releases through 0.1.23 published only vX.Y.Z. Chart 0.1.23 used bare appVersion as the image tag when image.tag was empty.

Fix: Pin a published tag, then upgrade:

helm upgrade attune oci://ghcr.io/attune-io/charts/attune \
  --namespace attune-system \
  --reuse-values \
  --set image.tag=v0.1.23

From 0.1.24, releases publish both vX.Y.Z and X.Y.Z. You can still override image.tag for local or E2E images.

Debug commands

Operator logs:

kubectl -n attune-system logs -l app.kubernetes.io/name=attune --tail=100

List all policies with status:

kubectl get attunepolicy --all-namespaces -o wide

Inspect a specific policy in detail:

kubectl describe attunepolicy <name>

Check operator metrics:

kubectl -n attune-system port-forward svc/attune-metrics 8080:8080 >/tmp/attune-metrics-pf.log 2>&1 &
PF_PID=$!
trap 'kill "$PF_PID" 2>/dev/null || true' EXIT
curl -s localhost:8080/metrics | grep attune