Upgrading

This page covers breaking changes and notable behavior shifts between versions. If you are upgrading from an earlier pre-release, apply every section below your current version (newest sections first).

Maintainers: before publishing a release after multi-version product changes, run the full E2E Nightly matrix on tip of main (see Releasing: full E2E matrix).

Unreleased

HPA auto-tune names the containers in a stored base

The first Resource retune for CPU or memory now stores which containers that base includes:

• attune.io/original-cpu-request-containers
• attune.io/original-memory-request-containers

The value is a sorted comma-separated list. It includes spec.containers and init containers with restartPolicy: Always, including a container whose pre-resize request is zero. A one-shot init stays out. A later resize of a container that was already named does not add that request on top of the stored quantity.

A later resize adds a container that is in the pod sum and missing from the list, at that container's pre-resize request. A container that joins with a zero pre-resize request is added to the list without changing the stored quantity and without HPABaseRepaired, including when the other containers change by the same amount and the pod total stays put. A full base is not replaced when the app's history old later equals the stored quantity. A name that stays on the list after the container leaves the pod stays in the stored quantity.

A base written by v0.1.33 or earlier has no list. Attune still replaces that quantity when it is below this cycle's pre-resize pod sum and equals the history old sum, or that sum with native sidecars removed. That rule can still count a sidecar twice when a full base equals history old. Delete attune.io/original-target-cpu and attune.io/original-cpu-request, or the memory pair, and the next resize stores a fresh base and the list. Delete the containers annotation too if it is set. Leave attune.io/auto-tune set. A quantity already rewritten to a larger value is left as stored.

HPABaseRepaired is still emitted only after the HPA update succeeds, and only when the stored quantity changes. No policy YAML change and no CRD change are needed.

See Utilization drifted after a second CPU resize.

v0.1.32 to v0.1.33

v0.1.33 adds per-container CPU and memory settings, memory HPA retune, a usage surge window, an OOM bump, limit multipliers, Amazon Managed Prometheus signing, and Argo Rollout targets. Those stay off until you set them.

v0.1.33 also limits what a namespace author can make the operator do. Cluster AttuneDefaults is unchanged. Set the allowlists in the checklist below before the new controller reconciles if a policy or AttuneNamespaceDefaults sets sigv4 or cloudwatch.roleArn.

Three changes apply on the next reconcile with no policy edit and no install change. An omitted maxAllowed is no longer a hidden ceiling of 4000m CPU and 8Gi memory. CloudWatch policies that omit cpuUnit treat container_cpu_usage_total as millicores. A stored HPA Resource base that left out a native sidecar is rewritten to the pre-resize pod sum.

These also apply with no policy edit. A cross-namespace metricsSource.vpa.namespace on a policy or namespace defaults object becomes InvalidConfig. A sigv4.roleArn or cloudwatch.roleArn that is not on the operator allowlist becomes InvalidConfig, and sigv4 with no role is rejected until the workspace host is allowlisted. Tenant SLO guardrails that would use operator Prometheus credentials are limited to the policy namespace. Datadog and CloudWatch stop evaluating guardrails. A paused policy stays Paused. A policy that is not paused still reverts an OOM or crash from a resize it already applied, and it still expires a startup boost, but it does not query the rejected metrics source or raise a new boost.

Apply the v0.1.33 CRDs before the controller. Helm does not update CRDs on helm upgrade.

VPA namespace

metricsSource.vpa.namespace on an AttunePolicy or AttuneNamespaceDefaults must be empty or that object's namespace. A different namespace is InvalidConfig and is not read. A namespace inherited only from cluster AttuneDefaults is still read.

SigV4 and CloudWatch roles

--sigv4-allowed-role-arns and --sigv4-allowed-workspace-hosts are empty by default. Helm values are sigv4.allowedRoleArns and sigv4.allowedWorkspaceHosts.

A sigv4.roleArn or cloudwatch.roleArn on a policy or namespace defaults object must match the role list. sigv4 with no roleArn must match the host list. Cluster AttuneDefaults is not filtered.

Add the role and host before upgrade if a policy sets them today:

sigv4:
  allowedRoleArns:
    - "arn:aws:iam::123456789012:role/attune-*"
  allowedWorkspaceHosts:
    - "aps-workspaces.us-east-1.amazonaws.com"

* matches any characters, including /.

A namespaced AssumeRole now sends ExternalId attune:<namespace>. A trust policy that does not check ExternalId still works. One that requires a different value must allow attune:<namespace>.

SLO guardrails

Datadog and CloudWatch do not evaluate guardrails, and they do not extend canary promotion or safety observation for a guardrail window.

A guardrail written on the policy or on AttuneNamespaceDefaults is scoped to the policy namespace when the query would use operator Prometheus credentials. sum(foo) gains namespace="<policy namespace>". A selector that already matches another namespace is not sent and does not hold observation or canary promotion. Guardrails inherited from cluster AttuneDefaults are not rewritten.

--slo-guardrail-enforce-namespace defaults to true. Set it to false to skip those tenant guardrails instead. Ready is not changed. The SLOGuardrails condition reason is SLOGuardrailNoTenantCredentials.

A scoped query that returns no samples does not revert. The condition reason is SLOGuardrailNoSamples. That happens when the series use a different namespace label than the policy namespace, for example an ingress controller's own namespace. After a later query returns a sample, the reason becomes Scoped.

Move a cluster-wide guardrail to AttuneDefaults, or give the policy its own Prometheus credentials, if it must keep reading outside the policy namespace.

Breach events no longer include the query value. The operator log at V(1) still has it.

See Tenancy.

Startup series cap keeps each container

With cpu.startupBoost.excludeFromHistory: true, a capped CPU query keeps one series per pod before any pod gets a second series. The series chosen for a pod is the container with fewer series already kept. A cap smaller than the number of containers still leaves some containers without CPU samples.

Policies that omit excludeFromHistory are unchanged. No policy YAML change and no CRD change are needed.

See One container missing under a capped startup query.

Policies outside watchNamespaces are admitted

Admission reads AttuneDefaults and AttuneNamespaceDefaults from the API server. A policy outside watchNamespaces can be created and updated. The controller still does not reconcile it. A defaults min above the policy max is still rejected. An API error while reading defaults still rejects the write.

No policy YAML change and no CRD change are needed.

See Policy rejected: unknown namespace for the cache.

Stored HPA base includes a native sidecar

A pod Resource utilization target counts spec.containers plus init containers with restartPolicy: Always. A one-shot init stays out of that sum.

On the next successful retune, Attune rewrites a stored attune.io/original-cpu-request or attune.io/original-memory-request that left the native sidecar out. The stored value has to be below this cycle's pre-resize pod sum, and it has to equal the history old sum or that sum with native sidecars removed. A stored base that already includes the sidecar stays, including after later growth. The HPA update emits HPABaseRepaired, and the event names the annotation that changed. A failed update does not emit the warning. No policy edit and no CRD change are needed.

See HPA coexistence.

CloudWatch matches the Deployment and CronJob names

Container Insights writes the Deployment name in PodName for a Deployment, and the CronJob name for a CronJob, when prefer_full_pod_name is left false. A Deployment or CronJob policy that had no samples for that reason now reads those series. No policy YAML change and no CRD change are needed.

The ReplicaSet hash and the Job stamp still match when the parent name is absent. A sibling such as api-v2 or api-worker does not match a policy for api. DaemonSet, StatefulSet, and bare Job matching stay as they are. prefer_full_pod_name: true still matches the full pod name.

Auto mode can resize once the new samples pass the existing gates. Recommend and Observe show recommendations and do not resize.

See CloudWatch Container Insights.

kubectl attune doctor names a missing policy type

When the AttunePolicy type is not installed, kubectl attune doctor prints Attune CRDs are not installed and the Helm install command. The row stays optional WARN and the command still exits 0. A Ready=False policy is named with its reason even when another policy has no Ready condition. An incomplete list stays WARN when the policies that came back are Ready. No policy edit and no CRD change are needed.

See CLI: doctor.

Startup boost follows a resize in the same reconcile

A startup boost or boost expiry reads the pod from the API server when that pod is inside the boost window or already stamped. A resize earlier in the same reconcile no longer leaves a young pod at the steady CPU, and the boost write keeps the memory request that resize just stored. A failed read still uses the earlier pod list. No policy edit and no CRD change are needed.

A committed resize still runs safety after a write timeout

An in-place resize or revert that returns a timeout or a decode error after the API server stored the new requests is treated as success when a follow-up read shows that container already has the target. Immediate safety observation still runs, and the workload budget stays spent. A rejected write is unchanged. A canceled context cannot confirm the write and still reports failure. No policy edit and no CRD change are needed.

Startup boost stamps survive status updates

A startup boost timestamp is written with a metadata merge patch. A kubelet status update no longer drops the stamp by conflicting with a full pod update. The boosted request can still expire when the window ends. A patch conflict is retried with backoff. No policy edit and no CRD change are needed.

ClusterRole drops two unused verbs

Helm, Kustomize, and OperatorHub upgrades drop list on config.openshift.io apiservers and watch on argoproj.io rollouts. OpenShift TLS profile detection still reads the cluster API server. Rollout targets still resolve, list by selector, and persist templates. No policy edit and no CRD change are needed.

OperatorHub installs grant the manager ClusterRole

An OperatorHub install grants the same cluster permissions as the Helm chart and the Kustomize install. The bundle lists namespaces, creates SubjectAccessReviews, reads controller revisions, patches Deployments, StatefulSets, and Argo Rollouts, patches ConfigMaps, and watches Prometheus objects. A policy held as NamespaceFrozen only because the install could not read namespaces starts applying on the next reconcile. A policy rejected because a Secret reference could not be checked can be applied again. No policy edit and no CRD change are needed. A namespace frozen with attune.io/freeze: "true" stays frozen. Helm and Kustomize installs are unchanged.

A held OOM no longer ends observation

When memory.oomBump is set and the hold suppresses an OOM or restart revert, Attune still checks throttle, SLO guardrails, and Ready. A breach reverts and keeps the memory floor. An open throttle grace period or SLO window keeps the pod tracked. A Ready pod with nothing pending still ends observation on that pass. Policies that omit memory.oomBump are unchanged. A policy that already set memory.oomBump can see a notready, throttle, or slo:<name> revert after a held OOM, where observation used to end. That revert keeps the OOM floor. No policy edit and no CRD change are needed.

Canary promotion waits for a longer SLO window

With autoPromote: true, a canary used to promote when observationPeriod ended, even if a guardrail evaluationWindow was still open. Promotion now waits for the longer of the two when the metrics source can run the guardrail query. A missing window counts as 5m. VPA does not extend the wait. For example, canary.observationPeriod: 10m and a p99-latency guardrail with evaluationWindow: 30m promotes at the end of 30 minutes. A revert during that wait still resets the watch. autoPromote: false is unchanged. No policy edit and no CRD change are needed.

An OOM at maxAllowed is counted once

A container already at memory.maxAllowed that OOMs outside memory.oomBump.hold used to add a skipped sample on every reconcile. Attune now stores that signal and counts one clamped sample. The request does not move. With the chart PrometheusRule enabled, AttuneOOMBumpCapped can fire for that sample. An OOM during an active hold is still skipped in Auto, OneShot, and Canary. No policy edit and no CRD change are needed.

SLO windows longer than the observation period

An SLO evaluationWindow longer than the observation period is now evaluated. Attune keeps the tracking annotations until that window elapses, then runs the query. A breach reverts with slo:<name>. SafetyObservation stays True while the window is open. Requeue stays at min(cooldown, observationPeriod), so a long window and a short period reconcile more often. No policy edit and no CRD change are needed.

HPA retune retries a conflicting write

A conflicting update of an auto-tuned HorizontalPodAutoscaler is retried. The next attempt reads the HPA again and writes the same pending target. Annotations set by another controller stay. A non-conflict error is still logged and is not retried until that workload is resized again. An HPA that already lost a retune keeps the stale target until the next resize. This does not scan HPAs that were missed before the upgrade. Recommend and Observe still do not retune. No policy edit and no CRD change are needed.

Tracking annotations no longer conflict with status writes

Attune writes resize tracking labels and annotations with a merge patch. A kubelet status update no longer conflicts with that write, so a resize that already applied is not reverted just because the status resourceVersion changed. No policy edit and no CRD change are needed.

Safety observation waits until the resize is applied

The safety observation period now waits while the kubelet still reports the resize as in progress or deferred. If that period would already have elapsed, Attune keeps the tracking annotations and writes attune.io/resize-apply-pending once. When the kubelet finishes, the observation period starts again from that time. A resize that applies during the original period is unchanged and does not gain a second window. An infeasible resize does not wait. An in-progress condition that has already been true for an hour is treated as stale, same as resize eligibility. That does not start a new observation window, even if attune.io/resize-apply-pending was already set. No policy edit and no CRD change are needed.

Schedule window with the same start and end

A schedule window whose start and end are the same minute never opened. The end minute is exclusive, so 02:00 to 02:00 matches no time. Admission now rejects that window on AttunePolicy, AttuneDefaults, and AttuneNamespaceDefaults. An object that already stores an equal pair still reconciles. ScheduleBlocked stays OutsideWindow, and the message says the window never opens. Change the end time to a later minute, or to an earlier minute for an overnight window. No CRD change is needed.

Memory limit decreases on Kubernetes 1.34

On Kubernetes 1.34 and newer, a policy that allows memory limit decreases now applies them in place. Earlier builds clamped the limit back to the current value on 1.34, the same as on 1.33. Kubernetes 1.34 allows that decrease when the resize policy is NotRequired. Clusters older than 1.34 still clamp. The usage floor (memory.decreaseUsageMarginPercent, default 10) is unchanged. No policy edit and no CRD change are needed.

Namespace defaults check Secret get access

Creating or updating AttuneNamespaceDefaults now requires the admission user to get each Secret named by bearerTokenSecret, apiKeySecretRef, or tokenSecretRef, in that object's namespace. An object already stored keeps using those Secret refs. No edit and no CRD change are needed. The next update of that object runs the check. A missing SubjectAccessReview still allows the object, same as a policy. Cluster AttuneDefaults is unchanged and still warns.

Whole-byte capped memory requests

A memory request that hits maxChangePercent, maxIncreasePercent, or maxDecreasePercent is now rounded up to a whole byte. Earlier builds kept millibyte precision when that capped value was decimal, for example a 2G or 500M bound, and could publish a request such as 1395864371200m. Binary bounds such as 2Gi were already whole bytes. When the cap is not already a whole number of bytes, the published request is higher by less than one byte. A request that does not hit the cap is unchanged. CPU caps stay in millicores. A pod or template that already carries a fractional memory request is not resized for this rounding alone. No policy edit and no CRD change are needed.

This is the request counterpart of the whole-byte memory limit rounding below.

Whole-byte memory limits

With memory.controlledValues: RequestsAndLimits, a scaled memory limit is now rounded up to a whole byte. This covers an explicit memory.limitMultiplier and the live request-to-limit ratio used when the multiplier is omitted. Earlier builds kept millibyte precision when the product was not whole, and could write a limit such as 1301300m. The new limit is then less than one byte higher than before. A whole product is unchanged. CPU limits keep millicore precision. API server warnings about fractional byte values on this path stop. No policy edit and no CRD change are needed.

Memory limits are compared at byte precision, with a fractional value counted as the next whole byte. A pod or workload template that already carries a fractional memory limit written by an earlier build is not resized or patched for this alone. The next resize or template patch that changes the limit, or the request, for another reason writes a whole-byte value.

Auto, Canary, and OneShot apply the whole-byte limit to pods. Recommend shows it but does not resize pods. With template persistence when: OnRecommendation, Recommend writes it into the template when the template changes for another reason. Observe neither resizes pods nor patches templates.

Healthy Rollout scale-out is resized

A Rollout whose phase is Healthy, or whose phase is empty, used to skip resize and template persistence while status.updatedReplicas was behind spec.replicas. A nil spec.replicas counted as 1, so a new Rollout with no updated replicas also waited.

That lag is a scale-out. Argo reports a replica lag as Progressing, not Healthy. Attune now resizes and can patch the template in that window. Paused, Progressing, and status.abort still wait. Degraded also waits, including when replica counts already match and status.abort is false. Argo uses Degraded for an aborted rollout, a progress deadline, or an invalid spec.

Deployments, StatefulSets, and DaemonSets are unchanged.

See Mid-rollout or no-op.

Explain names the container-policy source

kubectl attune explain used to print source: policy and configured: <unset> for a container percentile the policy had set, and for a max nobody set. A percentile set on the policy now prints that number as the configured value. An unset max prints source: built-in. A max copied from defaults prints defaults, namespace defaults, or cluster defaults. Named entries and * are unchanged. Recommendations, admission, and resize do not read these labels.

One capped OOM bump can raise AttuneOOMBumpCapped

Chart installs that set metrics.prometheusRule.enabled and left oomBumpCapped at the defaults used to miss a single capped or clamped bump. The expression was a one-hour rate(), and the alert also waited 1h, so that one sample usually aged out before the alert fired.

The chart expression is now:

sum by (namespace, policy) (increase(attune_oom_bump_total{result=~"capped|clamped"}[1h])) > 0

The previous expression was:

sum by (namespace, policy) (rate(attune_oom_bump_total{result=~"capped|clamped"}[1h])) > 0

The default pending window is 5m. Severity stays info. A value set on oomBumpCapped.for is kept. metrics.prometheusRule.enabled stays false, so an install that does not turn the object on still has no PrometheusRule. The alert does not turn memory.oomBump on. Resizes do not change.

A PrometheusRule copied outside this chart is not rewritten. Replace that copy with the increase expression above.

AttuneMemoryLimitUnsafe is unchanged.

Defaults min above a policy max is rejected

A policy that omits minAllowed and sets maxAllowed used to be admitted when AttuneDefaults or AttuneNamespaceDefaults carried a higher min. That apply now fails. The error names both values and both objects. A container max is included when * and the policy omit max.

A policy that sets its own min is unchanged, even when defaults carry a higher min. A max with no defaults min anywhere is unchanged. Omitted max stays uncapped.

Objects already stored keep reconciling until the next update. A stored min above a non-zero max is clamped to the max. The recommendation explanation says max. That runtime result does not change on upgrade.

See Apply rejected: defaults min above policy max.

Prometheus and Datadog secrets ignore surrounding whitespace

A Prometheus bearer token or a Datadog API key with a leading or trailing newline used to be sent unchanged. The provider then returned 401 or 403. Attune now trims both ends before the value is used. Spaces in the middle of the value stay. A whitespace-only bearer token or API key is rejected as empty when the Secret is read. A whitespace-only Datadog app key is treated as unset and is not sent.

A secret that already authenticated does not change. A secret that failed only because of surrounding whitespace starts working on the next reconcile. No policy YAML change. Auto can resize once metrics flow. Recommend and Observe start showing recommendations.

A GitOps pull-request token read from a Secret is trimmed the same way.

See 401 after a trailing newline in a Secret.

Opt-in startup exclusion follows the boost stamp

cpu.startupBoost.excludeFromHistory: true used to drop CPU samples from pod creation. The window now starts at attune.io/startup-boost-at when that stamp is set and not before creation. A later stamp drops the boosted samples. The CPU recommendation can fall on the next reconcile. Auto, OneShot, and Canary can lower the request, still subject to allowDecrease, maxDecreasePercent, and minAllowed.

Policies that omit startupBoost, or omit excludeFromHistory, do not change. A boost stamped at creation does not change. Memory samples are not filtered.

See CPU percentile stayed high after a boost.

Lowering maxAllowed during an OOM hold takes effect on the next reconcile

A held memory.oomBump floor used to be published again with no clamp after memory.maxAllowed was lowered or added. The next reconcile now publishes that cap. The stored floor and holdUntil stay. The clamp is not a new OOM.

Omitted maxAllowed is still uncapped. Default memory allowDecrease is false, so a pod already above the new cap is not resized down. The request that stays is the highest in-hold pod request, not the workload template. Template persistence defaults to off, so that floor can sit above the template. A replica still under the cap is raised only to the cap. The recommendation keeps that live request and records the allowDecrease skip. Set memory.allowDecrease to true to let Auto, OneShot, and Canary resize down to the cap. Recommend shows that recommendation and does not resize. Observe does not resize.

attune_oom_bump_total{result="clamped"} increments once for that floor and cap per operator process, including when the request does not move. It does not increment on every reconcile. A restart counts that pair once more. A new OOM step that is clamped still counts only after the resize succeeds.

Policies that omit memory.oomBump do not change. A Guaranteed pod whose limit multiplier raises the request to the multiplied limit still does that, including when the multiplied limit is above maxAllowed.

See Request stayed above maxAllowed during an OOM hold.

A second OOM during hold steps above live

During memory.oomBump.hold, a newer OOM publishes one memory request above the live request when the next step from the original origin is not already above live. The origin stays the start of the streak. maxBumps and maxAllowed still cap the step. Auto, OneShot, and Canary resize to that step. Recommend stores the recommendation and does not resize. Observe does not resize.

If maxAllowed is already at or below the live request, the request does not move. Auto, OneShot, and Canary store that signal on the pod, so that OOM counts once as attune_oom_bump_total{result="skipped"}. Recommend and Observe do not write the stamp, so the same OOM still counts on every reconcile. A chart of the whole skipped series does not become a per-event count: a budget skip, or a Guaranteed pod with RequestsOnly, still increments on every reconcile. AttuneOOMBumpCapped still keys off capped and clamped.

Policies that omit memory.oomBump do not change. The default hold stays 24h.

OOM revert follows container controlledValues

An OOM-bump revert raises a positive memory limit to the bump floor only when that container's effective controlledValues is RequestsAndLimits. An empty containerPolicies list still uses spec.memory.controlledValues, so policies with no per-container mode do not change.

A container set to RequestsOnly keeps its limit. A container set to RequestsAndLimits gets the limit raised, including when the policy block says RequestsOnly. * and an exact name follow the usual overlay. A zero or missing limit is not created. oomBump stays policy-wide. The request floor is unchanged. New resizes run only in Auto, OneShot, and Canary. A pod already under safety observation still reverts in Recommend and Observe when autoRevert is on (the default). autoRevert: false skips the revert in every mode.

CronJob pods match the minute stamp

A CronJob pod name uses the minute stamp from the CronJob controller (unix time divided by 60). That stamp is 8 digits until 2160-02-18 and 9 digits after that. Kubernetes 1.32 and later do not write the pre-1.21 10-digit unix-seconds suffix.

Prometheus queries and CloudWatch owner-name matching accept that 8- or 9-digit stamp. A non-indexed pod is <cronjob>-<stamp>-<hash>. An indexed pod is <cronjob>-<stamp>-<index>-<hash>. A 10-digit suffix does not match. No policy YAML change. CloudWatch still matches the owner name in PodName. See CronJob pod name does not match.

CronJobs on a supported cluster that the old 10-digit pattern missed can now produce samples. Status recommendations, export ConfigMaps (when export is on), and kubectl attune diff gain CronJob values.

Running CronJob pods are not resized in place in any mode. New CronJob pods can be sized at creation in Recommend (or with no type set), Auto, and OneShot. That needs the initial sizing webhook enabled, updateStrategy.initialSizing: true (on the policy or inherited from AttuneDefaults or AttuneNamespaceDefaults), the namespace label attune.io/initial-sizing=enabled, and confidence at least 0.5 on every container of the recommendation. See Initial Sizing. Observe never sizes pods. Policies for other workload kinds do not change.

Policy limit multiplier requires RequestsAndLimits

A policy that sets cpu.limitMultiplier or memory.limitMultiplier must set controlledValues: RequestsAndLimits on that same block. Admission rejects an omitted or empty mode. RequestsOnly with a multiplier was already rejected.

An object already stored without that mode is not rewritten. Reconcile still reports InvalidConfig until an update sets RequestsAndLimits or removes the multiplier. AttuneDefaults and AttuneNamespaceDefaults may still store a multiplier without the mode. A policy that omits the multiplier still inherits it.

RequestsOnly does not inherit a limit multiplier

A policy that sets cpu.controlledValues or memory.controlledValues to RequestsOnly no longer copies limitMultiplier from AttuneDefaults or AttuneNamespaceDefaults. The live limit stays, and that policy is no longer InvalidConfig because of a defaults multiplier.

A policy that omits controlledValues still copies the multiplier. Built-in defaults then fill RequestsOnly, and reconcile still reports InvalidConfig. Set RequestsAndLimits on that policy, or remove the multiplier from defaults. A defaults object may still store a multiplier without RequestsAndLimits.

Rollout workloadRef read failures stay visible

Template persistence still does not patch a Rollout whose spec.workloadRef is set. When that object cannot be read, the condition reason stays WorkloadRefUnread for the rest of the reconcile. It is not replaced by TemplateWorkloadRef, and it is not removed while the read error is still present.

TemplateWorkloadRef means the reference was read and the template was left alone. A later reconcile with no workloadRef read error removes a stale WorkloadRefUnread. Resize behavior does not change. Observe mode already kept the unread reason. Recommend mode now does too.

Amazon Managed Prometheus signing

Policies that omit metricsSource.prometheus.sigv4 stay unsigned. There is no Helm default and no environment variable that turns signing on. Set sigv4.region on a policy, or on AttuneDefaults, when that Prometheus address is an Amazon Managed Prometheus workspace.

Argo Rollout workloads

targetRef.kind accepts Rollout (argoproj.io/v1alpha1). Helm upgrades add a ClusterRole rule for argoproj.io rollouts: get, list, watch, patch, and update. Raw manifests need that rule on the operator ClusterRole. Policies that do not set kind: Rollout do not change. Attune does not install the Rollout CRD. A Rollout policy stays Ready False with reason WorkloadCRDMissing until Argo Rollouts is installed.

With spec.workloadRef set, TemplatePersistence can be False with one of two reasons. WorkloadRefUnread is set whether or not template persistence is on. It means spec.workloadRef has an empty name or an unsupported kind, or the referenced object could not be read or has no containers; the error is also in status.workloadErrors. TemplateWorkloadRef is set only with template persistence on, and only on a reconcile that reaches the template persistence step (Observe mode and other early exits skip it). It means Attune left the Rollout template alone and this reconcile's recorded errors did not include a workloadRef read failure. WorkloadRefUnread wins while a recorded workload error is a workloadRef read failure. Template persistence does not replace that reason in the same reconcile, including a safety restore. A later reconcile whose workload errors no longer include that failure removes it.

status.workloadErrors keeps at most 10 entries, so with more failing workloads a reference error can be dropped from it and the reason may not show.

What changes on upgrade: a WorkloadRefUnread reason that template persistence used to overwrite with TemplateWorkloadRef, or remove, now stays. Resize behavior, defaults, and the CRD schema do not change.

Resize during a rollout

Recommendations are still computed while pods are being replaced. Which pods are resized depends on the workload kind and its strategy.

A RollingUpdate Deployment, including the default strategy, skips resize only while its generation is ahead of the observed generation, or while status.replicas is still above status.updatedReplicas. It does not freeze because availableReplicas is behind spec.replicas, or because a scale-out has not finished. Pods still on the old template are skipped. A paused Deployment skips pods that are not on the current pod-template-hash when that hash is known.

A Deployment whose strategy is Recreate is not a resize skip. Pods are resized while updatedReplicas is still behind spec.replicas. Release v0.1.32 skipped the whole Deployment in that case.

A RollingUpdate StatefulSet resizes pods whose revision matches status.updateRevision and skips the others, including while a partition has not reached every replica. A stale generation skips every pod. An OnDelete StatefulSet is not a resize skip. Pods still on the previous template are resized. Release v0.1.32 skipped the whole StatefulSet while updatedReplicas was behind spec.replicas.

A RollingUpdate DaemonSet resizes pods on the current ControllerRevision and skips older pods. An OnDelete DaemonSet is not a resize skip. Its pods are resized.

A standalone ReplicaSet is skipped only while its generation is ahead of the observed generation. readyReplicas behind spec.replicas does not freeze it, so ready and unready pods can be resized. Release v0.1.32 skipped the ReplicaSet while readyReplicas was behind.

DaemonSet controllerrevisions

A RollingUpdate DaemonSet needs apps/controllerrevisions get, list, and watch. A Helm upgrade adds that ClusterRole rule. Raw manifests need the same rule on the operator ClusterRole.

If that list fails, Attune skips every pod of that DaemonSet and emits DaemonSetRevisionUnavailable. It does not resize them. OnDelete does not read ControllerRevisions.

The list uses the DaemonSet pod selector, then the owner UID. Resize decisions stay the same when that selector matches the current revision. The call returns fewer objects. No new RBAC rule and no user action. An owned revision the selector does not match still skips the DaemonSet.

Attune reads only the pod label controller-revision-hash. It does not read pod-template-generation. Pods that have the hash label resize the same way. A pod with no hash label is still skipped when the current hash is known.

Stored cooldown of 0s

A stored cooldown: 0s is not a wait. The controller treats it as the 1h default, including when a policy omits cooldown and inherits 0s from AttuneDefaults or AttuneNamespaceDefaults. An update that leaves that stored 0s unchanged is accepted on AttunePolicy, AttuneDefaults, and AttuneNamespaceDefaults, including the update that removes the finalizer, so the object can be deleted. Attune adds no finalizer to defaults objects. On a defaults object, an unrelated edit is accepted, including removing a finalizer that another tool added. Defaults have no delete-time skip, so another invalid field still blocks the edit. A create that sets 0s, or an update that changes a positive duration to 0s, is still rejected. Omit the field for the 1h default, or set at least 1m. The same admission rule applies to safetyObservationPeriod (a stored 0s is unset, so the controller uses 5m or a positive canary period) and an SLO evaluationWindow (a stored 0s uses 5m, and is accepted only when the stored entry at the same list index is also 0s). Canary observationPeriod: 0s still means the built-in observation period. historyWindow below 1h stays rejected on every write, including an unchanged 0s.

Datadog null points

A Datadog point with a JSON null, or no value, is dropped. A numeric zero is kept. Dropping a gap can raise the CPU percentile. CPU allowDecrease still defaults to true, so the next reconcile can raise the CPU request. Memory allowDecrease still defaults to false.

QoS class changes are not evicted

An in-place resize that would change the pod QoS class is skipped, including when resizeMethod is InPlaceOrRecreate. The pod is not evicted. InPlaceOnly, the default, already could not apply that resize, because the API rejects a QoS class change.

When the pod has spec.resources, both sides of that check use one rule. If in-place pod-level resize is on, both sides use the pod-level envelope. If it is off, both sides use the containers. Attune does not compare status.qosClass with a container class. A pod with no spec.resources still uses status.qosClass for the current class. See Resize API.

Standalone ReplicaSet initial sizing

initialSizing still defaults to false. A policy that already sets initialSizing: true now sizes new pods of a standalone ReplicaSet whose selector matches. A ReplicaSet owned by a Deployment still follows the Deployment.

Memory HPA targets move when auto-tune is on

A memory utilization HPA used to keep its percent when Attune changed the memory request, so a smaller request could scale the workload out. An HPA annotated attune.io/auto-tune: "true" now gets the same retune for memory as for CPU. The original target and the original request are stored on the HPA. A later resize multiplies that stored target by the stored request divided by the new request.

updateStrategy.hpaTargetBounds is optional. Leave it unset to keep today's limit cap, including CPU targets above 90. Set cpu or memory min and max only when you want a tighter band. 50 and 90 are a common choice, not a default. The Helm chart does not turn auto-tune on.

Usage surge is off until cpu.surge or memory.surge is set

CPU and memory stay on the long history window unless that resource's surge block is set. An empty surge: {} turns the feature on for that resource and fills trigger ratio 1.5, percentile 99, and window 30m. There is no cluster default unless an AttuneDefaults object sets surge. A policy that omits surge inherits a defaults surge. To keep a workload off, omit surge on both the policy and AttuneDefaults.

When recent usage is hot, Attune shortens the history window for that resource. The parent percentile still describes the long window. The short window uses its own percentile and only the samples inside surge.window. One spike does not switch the window. minimumDataPoints still gates the long window.

A policy surge.window is checked against the inherited history window. Admission reads AttuneNamespaceDefaults and AttuneDefaults for that check when the policy sets surge.window and omits metricsSource.historyWindow. A policy or named container policy that omits minAllowed while maxAllowed is set lists the same objects for the minimum-above-maximum check, including when it sets historyWindow or sets no surge window.

OOM bump is off until memory.oomBump is set

Memory requests stay on the percentile path unless memory.oomBump is set. An empty oomBump: {} turns the feature on and fills ratio 1.2, minBump 100Mi, maxBumps 3, and hold 24h. The step is max(ceil(origin * ratio^count), origin + minBump * count), then maxAllowed. Origin is the live memory request before the first bump of the streak, not the latest live request and not the pod template. When maxAllowed is omitted, maxBumps is the only cap. After hold expires, recommendations follow the normal percentile, allowDecrease, and template rules. Hold expiry does not clear the original request stored on the pod. A later OOM can step again from that same origin until maxBumps.

Startup history exclusion is opt-in

cpu.startupBoost.excludeFromHistory drops CPU samples from the percentile while a pod is inside startupBoost.duration plus the rate window. The cutoff is the pod CreationTimestamp plus duration plus rateWindow. Sample timestamps are the end of rate().

Existing startup boost policies keep today's percentile until you set excludeFromHistory: true. Nil and false do not change the percentile. Memory samples are unchanged. Deleted pods stay in the history window, because there is no CreationTimestamp to cut on. A recreated pod keeps samples older than its new CreationTimestamp. A series with no pod label is left unfiltered.

RequestsAndLimits limit multiplier

cpu.limitMultiplier and memory.limitMultiplier stay off until you set them. Omitted keeps the live request-to-limit ratio. Explicit "1" forces the limit equal to the new request. Values below 1 are rejected, because a limit cannot be smaller than its request. maxAllowed caps the request, not the limit. A limit above that cap is the HPA CPU ceiling when controlledValues is RequestsAndLimits.

During a startup-boost window the CPU limit is the greater of the boosted request and the steady multiplied limit. The startup-boost multiplier does not multiply that limit again. Expiry restores the steady multiplied limit. A multiplied memory limit stays in place through that window and through expiry. The boost does not put the pre-resize memory limit back.

A Guaranteed pod skips an in-place CPU multiplier above 1 because the request and limit would differ. A memory multiplier on an already Guaranteed pod raises the memory request to the new limit and the resize proceeds. That applied request can exceed maxAllowed.

RequestsOnly plus a multiplier on the same object is rejected. AttuneDefaults may carry a multiplier without RequestsAndLimits. A policy that sets RequestsOnly does not inherit that multiplier. A policy that omits controlledValues still inherits it, then resolves to RequestsOnly, so that pair is rejected at reconcile.

HPA auto-tune keeps the stored base

An HPA annotated attune.io/auto-tune: "true" stores the pod CPU total and the pod memory total on the first Resource retune for that resource. The sum is spec.containers plus init containers with restartPolicy: Always. A one-shot init stays out.

Later resizes reuse that stored request. Attune replaces it when it is below this cycle's pre-resize pod sum and equals either the history old sum or that sum with native sidecars removed. The update writes the pre-resize sum and emits HPABaseRepaired only after the HPA update succeeds. The event names attune.io/original-cpu-request or attune.io/original-memory-request. A failed write does not emit the Warning.

A stored base that matches neither sum stays. That includes a full base after the same containers grow, and a smaller base that already includes the sidecar. ContainerResource bases are not repaired this way.

An annotation that an older operator already replaced with a grown sum is left as stored. This release does not write the earlier number back.

Deleting the keys still stores a fresh base on the next resize. Leave attune.io/auto-tune in place:

• attune.io/original-target-cpu
• attune.io/original-cpu-request
• attune.io/original-target-memory
• attune.io/original-memory-request
• any attune.io/hpa-cpu-target.* key
• any attune.io/hpa-cpu-base.* key

The next successful resize of that resource stores the full base when the keys are gone.

Omitted maxAllowed is not capped

Policies and AttuneDefaults objects that omitted maxAllowed were held at 4000m CPU and 8Gi memory. CPU allowDecrease defaults to true, so a CPU request already above 4000m could drop to that ceiling in one cycle. Memory allowDecrease defaults to false, so the 8Gi ceiling blocked further growth. A memory request already above 8Gi stayed at the current request until decrease was enabled.

To restore the old ceiling, set it on the policy or on AttuneDefaults:

cpu:
  maxAllowed: "4000m"
memory:
  maxAllowed: "8Gi"

An omitted minimum still floors at 1m CPU and 4Mi memory. An explicit maxAllowed of "0" remains a cap. The engine applies an explicit max again after the percent cap. A memory request above that max is published downward only when allowDecrease is true. Explanation status omits bounds.max when the policy has no maximum, so it is not reported as 0.

Apply CRDs before the controller upgrade. Helm does not update CRDs on helm upgrade. The previous CRD requires explanation.cpu.bounds.max and explanation.memory.bounds.max. The next status write for a policy that omits maxAllowed fails that required field until the new CRDs are applied:

kubectl apply --server-side --force-conflicts -f \
  https://github.com/attune-io/attune/releases/latest/download/crds.yaml

CloudWatch CPU unit

CloudWatch policies that omit cpuUnit now treat container_cpu_usage_total as millicores (divide by 1000). Older releases divided that metric by 1e9. The next reconcile changes CPU recommendations.

To keep the old scale, set cpuUnit: Nanocores on each policy that has its own cloudwatch block. Setting it only on AttuneDefaults does not reach those policies. Defaults apply when the policy omits the whole metrics provider.

If the new scale is too small, CPU requests sit at 1m or minAllowed. If it is too large, they move toward maxAllowed. CPU allowDecrease still defaults to true, and CloudWatch does not run throttle revert, so a bad step is not undone by the throttle check. Confirm one raw GetMetricData sample against the container limit before rollout.

The OpenTelemetry awscontainerinsightreceiver multiplies the core rate by 1000, and its README lists container_cpu_usage_total as Millicore (cpu extractor).

Upgrade the chart and image

1. Apply the v0.1.33 CRDs before the controller. The previous CRD requires explanation.cpu.bounds.max and explanation.memory.bounds.max. The next status write for a policy that omits maxAllowed fails until these CRDs are applied. Helm does not update CRDs on helm upgrade.

kubectl apply --server-side --force-conflicts -f \
  https://github.com/attune-io/attune/releases/latest/download/crds.yaml

releases/latest is v0.1.33 only after that tag is published. Do not apply an older crds.yaml.

1. Upgrade the chart to 0.1.33, or set image.tag to 0.1.33 or v0.1.33. A Helm upgrade updates the ClusterRole. Raw manifests need the same rules: apps/controllerrevisions get, list, and watch, and argoproj.io rollouts get, list, watch, patch, and update. list on config.openshift.io apiservers and watch on rollouts are no longer required.

2. Pull ghcr.io/attune-io/attune:v0.1.33 or ghcr.io/attune-io/attune:0.1.33. Both tags point at the same digest.

3. To keep the old ceiling, set cpu.maxAllowed: "4000m" and memory.maxAllowed: "8Gi" on the policy or on AttuneDefaults before the new controller reconciles.

4. To keep the v0.1.32 CloudWatch scale, set cpuUnit: Nanocores on each policy that has its own cloudwatch block.

5. Before the new controller starts, set sigv4.allowedRoleArns when a policy or AttuneNamespaceDefaults sets prometheus.sigv4.roleArn or cloudwatch.roleArn. Set sigv4.allowedWorkspaceHosts when sigv4 has no roleArn. Both lists are empty until you set them. Cluster AttuneDefaults is not filtered. A namespaced assume sends ExternalId attune:<namespace>.

6. sloGuardrails.enforceNamespace defaults to true. A tenant guardrail that uses operator Prometheus credentials gains namespace="<policy namespace>" on every selector. A selector that already names another namespace is not sent (SLOGuardrailQueryRejected). A scoped query that returns no samples does not revert (SLOGuardrailNoSamples). Set the Helm value to false only to skip those guardrails instead. Move a cluster-wide guardrail to AttuneDefaults, or give the policy its own Prometheus credentials, when the series use a different namespace label (for example an ingress controller).

v0.1.31 to v0.1.32

v0.1.32 restores CloudWatch samples for Deployments and CronJobs, and it stops a long history from pushing a configured percentile upward. Existing policy YAML keeps working. No CRD change is required.

CloudWatch matches the owner name

Container Insights stores the owner in PodName, not the full pod name. v0.1.31 matched only the full pod name, so Deployments and CronJobs got no samples.

v0.1.32 accepts that owner name:

• A Deployment matches the ReplicaSet hash. The hash uses Kubernetes SafeEncodeString (consonants plus the digits 2, 4, 5, 6, 7, 8, and 9, length 5 to 10). A sibling such as api-v2 or api-worker is not included.
• A CronJob matches the Job name, including the timestamp.
• A DaemonSet, StatefulSet, or Job matches the workload name.
• Clusters that set prefer_full_pod_name still match the full pod name.

A DaemonSet, StatefulSet, and Job that share one name still share one CloudWatch series. That metric has no kind.

See CloudWatch Container Insights.

Long histories keep the configured percentile

After 10,000 samples, v0.1.31 kept the highest value in each time window and computed the percentile from those highs. A configured P95 could land near the real P98. v0.1.32 keeps the midpoint of each window. The highest sample in the series is still kept, so burst detection still sees a short spike.

A recommendation can be lower than the one v0.1.31 published for the same history.

An empty usage percentile no longer grows the request

When the usage percentile was missing, v0.1.31 still applied the confidence buffer and could raise the request. v0.1.32 leaves that request unchanged until a real percentile exists.

Other corrections

• GitOps drift after a merged branch stays on the workload that owns the branch.
• A clock step backward, or a revert that fails, no longer drops the increase budget. A failed revert stays recorded as in-place.
• Configured bounds and the increase budget still apply after the change filter. Label selectors and metric pod filters keep matching the intended pods.

Upgrade the chart and image

1. Upgrade the chart to 0.1.32, or set image.tag to 0.1.32 or v0.1.32.
2. Pull ghcr.io/attune-io/attune:v0.1.32 or ghcr.io/attune-io/attune:0.1.32. Both tags point at the same digest.
3. CRDs are unchanged for this release. Helm still does not upgrade CRDs on helm upgrade.

v0.1.30 to v0.1.31

v0.1.31 adds operator-level Prometheus authentication and deprecates bearerTokenSecret on cluster AttuneDefaults only. Policy YAML that already copies a token Secret into each namespace keeps working.

Operator Prometheus identity

The manager can authenticate to Prometheus without a Secret in every policy namespace:

• Helm openshift.bindClusterMonitoringView: true binds OpenShift cluster-monitoring-view (to the query ServiceAccount when prometheusAuth.queryServiceAccount.create is true, otherwise the manager SA) and sends that token (--prometheus-use-service-account-token).
• Helm prometheusAuth.queryServiceAccount.create: true TokenRequests a dedicated query SA instead of the manager token. The OLM bundle and config/rbac also create attune-prometheus-query, a Role for serviceaccounts/token on that account, and pass --prometheus-query-service-account=attune-prometheus-query plus --prometheus-use-service-account-token. Bind OpenShift cluster-monitoring-view to attune-prometheus-query.
• Helm prometheusAuth.useServiceAccountToken: true sends the SA token without that binding (vanilla clusters or a binding you created).
• Helm prometheusAuth.existingSecret reads one Secret in the operator namespace (--prometheus-bearer-token-secret).
• If cluster AttuneDefaults still names bearerTokenSecret and that Secret is missing in the policy namespace, the operator falls back to the operator identity and logs it.

Operator auth applies only to addresses from cluster AttuneDefaults. It is not sent to a policy address, a namespace-defaults address, or an auto-discovered Prometheus.

A policy (or AttuneNamespaceDefaults) bearerTokenSecret still wins and is still read in that namespace.

See OpenShift: Thanos Querier.

Operator Datadog API key

Helm datadogAuth.existingSecret (--datadog-api-key-secret) reads one Secret in the operator namespace when cluster AttuneDefaults chose Datadog. A policy or AttuneNamespaceDefaults apiKeySecretRef stays in that namespace. Without the flag, an apiKeySecretRef name on cluster AttuneDefaults is still read in each policy namespace. Admission warns on that field. See Datadog setup.

AttuneDefaults bearerTokenSecret is deprecated

Setting metricsSource.prometheus.bearerTokenSecret on cluster AttuneDefaults emits an admission warning. The Secret name is still copied onto each policy and looked up in the policy namespace. A later 0.1.x will reject that field on AttuneDefaults after two tagged minors have carried the warning. Move cluster auth to the operator identity above. AttunePolicy and AttuneNamespaceDefaults are unchanged.

v0.1.29 to v0.1.30

v0.1.30 changes how memory.memoryFromCpuRatio waits for CPU and how bootstrap requeues work. Existing policy YAML keeps working. Read this section if you set memoryFromCpuRatio, use a cooldown longer than queryStep, or watch Ready MetricsUnavailable.

memoryFromCpuRatio waits for CPU

A valid ratio no longer falls back to Prometheus memory gauges or the VPA memory target while CPU samples (or the VPA CPU target) are missing. Ready stays InsufficientData and retries at min(cooldown, queryStep) with no requeueJitter. Progress (Collecting data: X/Y) counts CPU samples only. A prior rec whose memory explanation contains memoryFromCpuRatio is kept as Stale across a CPU-only gap so kubectl attune and the hold baseline stay. A leftover usage rec from v0.1.29 is dropped. An invalid ratio (abc, 0, NaN, or above 1000) is ignored and Attune uses the memory signal; the webhook still rejects those when admission is enabled.

See memory.memoryFromCpuRatio and Troubleshooting: InsufficientData.

MetricsUnavailable uses the bootstrap requeue

Ready MetricsUnavailable (query error or timeout, including a CPU-only blip under memoryFromCpuRatio) now requeues at min(cooldown, queryStep) without jitter. A policy with cooldown: 2h and the default 5m step retries every 5 minutes instead of waiting the full cooldown. Resize spacing is unchanged once Ready is Monitoring.

See Troubleshooting: MetricsUnavailable.

Upgrade the chart and image

1. Upgrade the chart to 0.1.30, or set image.tag to 0.1.30 or v0.1.30.
2. Pull ghcr.io/attune-io/attune:v0.1.30 or ghcr.io/attune-io/attune:0.1.30. Both tags point at the same digest.
3. CRDs are unchanged for this release. Helm still does not upgrade CRDs on helm upgrade; apply them only if you skipped a previous release that added CEL rules.

v0.1.28 to v0.1.29

v0.1.29 keeps GPU, hugepages, and other extended resources on live /resize, and it fails closed when HPA cannot be listed or when CREATE would re-admit a just-reverted size. OneShot now records envelope skips the same way Auto already did. Existing policy YAML keeps working. Bare integer minAllowed / maxAllowed values that v0.1.26 accepted are accepted again. Read this section if you use GPU or hugepages, OneShot, startup boost, CREATE initial sizing, HPA auto-tune, or scripts that watch kubectl attune doctor.

Live /resize keeps GPU, hugepages, and ephemeral-storage

A successful in-place resize or safety revert used to drop extended resources from the live request map. Apply now overlays CPU and memory only. Template persist still writes only CPU and memory, so those extra keys stay on the pod.

AfterSuccessfulResize restore-retry needs a Reverted row

A Safe pod could have its workload template rolled back to the pre-resize snapshot when live requests happened to match the clamped revert target. Restore-retry now requires a real Reverted history row.

See Troubleshooting: Template restore after safety revert.

Persist skips only the failed container

One Failed history row on a single replica used to block template persist for the whole workload. Persist now omits only the pod and container that failed or reverted. Envelope math counts running containers only, not run-to-completion init containers.

See Troubleshooting: Pod-level resource envelope blocked an increase.

OneShot records envelope skips

OneShot still walks past a replica that cannot apply so a sibling can move. If every needing replica is blocked, it keeps the first blocked pod and emits ResizeSkipped plus a Failed envelope_constraint history row, matching Auto. Boost-window CPU decreases are walked past the same way QoS and node-pressure skips already were. Envelope and boost skips do not consume maxTotalCpuIncrease, so a later replica is not deferred with a misleading BudgetExhausted event.

See Troubleshooting: OneShot skipped the first replica but others still need a resize and Troubleshooting: Budget exhausted.

Startup boost stamps only after a successful boost

attune.io/startup-boost-at is written only after a successful boost /resize. Expiry of a never-boosted stamp no longer shrinks the pod. RequestsAndLimits boost raises CPU dest only and does not add a memory limit. CREATE no longer boosts Job or CronJob pods (expiry skips batch workloads). A leftover Running pod under HPA ScaledToZero no longer receives a startup boost. A malformed attune.io/startup-boost-at blocks CPU decrease the same way a failed expiry already did.

See Startup boost and Troubleshooting: HPA ScaledToZero left leftover pods at old requests.

CREATE fails closed on HPA list and after a safety revert

CREATE no longer sizes a new pod while ResizeBlocked=HPAListUnavailable, or from a recommendation that was just safety-reverted. Scripts that assumed CREATE would still size while HPA could not be listed will see those pods admitted at template size until the list succeeds. A CronJob pod is no longer admitted unsized when its Job is not yet in the informer cache; admission reads the Job from the API.

See Troubleshooting: HPAListUnavailable.

HPA retune is per-pod

HPA auto-tune used to sum CPU across every resized pod, so the target depended on how many replicas had already resized. Retune is per-pod From/To. RequestsAndLimits still caps that dest at 100% of the recommendation dest. ContainerResource HPA metrics on sidecars are retuned with the resized container. VPA missing-resource hold ignores temporary boost stamps so the hold does not ratchet up.

See HPA coexistence.

Immediate safety revert stamps cooldown

An immediate safety revert now stamps cooldown so the same recommendation is not re-applied on the next reconcile.

minAllowed / maxAllowed accept bare integers again

CEL rejected bare integers that v0.1.26 accepted. The rule type-guards quantity() so integer-or-string values work again. Apply CRDs before helm upgrade so the restored rule is on the cluster.

kubectl attune doctor Ready

kubectl attune doctor reports Ready only when every required check passed. A failed list or webhook check no longer prints Ready.

See CLI: doctor.

Upgrade the chart and image

1. Upgrade the chart to 0.1.29, or set image.tag to 0.1.29 or v0.1.29.
2. Pull ghcr.io/attune-io/attune:v0.1.29 or ghcr.io/attune-io/attune:0.1.29. Both tags point at the same digest.
3. Helm does not upgrade CRDs on helm upgrade. Apply CRDs before the chart upgrade so the restored integer-or-string CEL rule is on the cluster:

kubectl apply --server-side --force-conflicts -f \
  https://github.com/attune-io/attune/releases/latest/download/crds.yaml

v0.1.27 to v0.1.28

v0.1.28 keeps existing pod-level resource envelopes valid when containers resize, and it treats HPA scale-to-zero as idle instead of a human shutdown. VPA and HPA list failures now skip apply. A post-resize OOMKill reverts even when kubelet omits FinishedAt. Existing policy YAML keeps working. Read this section if you use pod-level spec.resources, HPA minReplicas: 0, recommend-only VPA, VPA targets that omit CPU or memory, or scripts that watch Attune ResizeDeferred Events.

Existing spec.resources envelopes are raised, not invented

When a pod already has spec.resources, live /resize, CREATE, and AfterSuccessfulResize persist raise that envelope so the new container requests stay valid. Attune does not invent an envelope when the field is missing, and it does not shrink one.

On clusters that report in-place pod-level resources, one /resize writes the container target and raises envelope requests to cover the container sum. If that capability is off, an increase that would exceed the envelope is skipped. Decreases still apply. Persist copies a live envelope onto the template so the next rollout keeps it. If listing live envelopes fails, persist is skipped instead of writing a template without the envelope.

See Troubleshooting: Pod-level resource envelope blocked an increase.

HPA ScaledToZero is idle; CREATE still sizes at zero replicas

Workloads with HPA ScaledToZero=True skip in-place resize, persist, startup boost, and eviction. Leftover Running pods stay at their current requests. That is idle, not a human setting spec.replicas to 0. CREATE still applies initial sizing when the owner is at zero replicas, so scale-up pods are sized.

See HPA coexistence: Scale to zero and Troubleshooting: HPA ScaledToZero left leftover pods at old requests.

Recommend-only VPA no longer emits VPAConflict

A VPA with updateMode: Off is the documented VPA-as-source path. VPAConflict is skipped for Off. Applying modes (Initial, Recreate, InPlaceOrRecreate, Auto) still warn.

Omitted VPA target CPU or memory is unset

A VPA target that omits cpu or memory is unset, not zero. Attune holds the omitted resource from live pods (or the template) instead of recommending from zero. Unlimited metrics sampling (--max-pods-in-metrics-query=-1) still lists pods so that hold can run.

VPA or HPA list errors skip apply

Any VPA or HPA list error other than a missing VPA CRD now skips apply, persist, boost, and CREATE. ResizeBlocked is VPAListUnavailable or HPAListUnavailable. kubectl attune status and explain show those reasons. A missing VPA CRD still skips the conflict check.

See Troubleshooting: VPAListUnavailable and Troubleshooting: HPAListUnavailable.

Post-resize OOMKill classifies without a later FinishedAt

Attune classifies OOMKilled on the current Terminated state or LastTerminationState, including a 2s kubelet skew. It no longer waits for FinishedAt to be strictly after resized-at. After a successful safety revert, cooldown is stamped so the same reconcile does not re-apply the shrink.

Attune no-op Events are ResizeUnchanged

The Attune no-op Event reason is ResizeUnchanged (applied target already matches live after filtering, dest clamp, or usage floor). Kubernetes 1.36 kubelet still uses ResizeDeferred when the node is out of room. Scripts that watched Attune ResizeDeferred for no-ops should watch ResizeUnchanged.

kubectl attune doctor optional cgroup v2 WARN

kubectl attune doctor prints an optional cgroup v2 row. Default is WARN with could not determine (expected on k3s, kind, and most managed clusters). Missing pods/resize remains a required FAIL. The optional row does not change the exit code.

See Troubleshooting: Doctor cgroup v2 is WARN.

Upgrade the chart and image

1. Upgrade the chart to 0.1.28, or set image.tag to 0.1.28 or v0.1.28.
2. Pull ghcr.io/attune-io/attune:v0.1.28 or ghcr.io/attune-io/attune:0.1.28. Both tags point at the same digest.
3. Helm does not upgrade CRDs on helm upgrade. Apply CRDs before the chart upgrade if you need new schema fields:

kubectl apply --server-side --force-conflicts -f \
  https://github.com/attune-io/attune/releases/latest/download/crds.yaml

v0.1.26 to v0.1.27

v0.1.27 tightens last-replica eviction, limit-only resizes, the memory usage floor, and kubectl --sort-by validation. It also adds a fail-closed namespace freeze kill-switch. Existing policy YAML keeps working. Read this section if you use InPlaceOrRecreate, controlledValues: RequestsAndLimits, memory limit decreases, eviction metrics, kubectl attune --sort-by, hand-managed RBAC, OneShot, or maxTotalCpuIncrease / maxTotalMemoryIncrease.

Namespace reads are now required

Apply now reads the policy namespace for attune.io/freeze=true and fails closed if that Get fails. A missing namespaces get, list, and watch grant looks like a cluster-wide freeze:

• no in-place resize, eviction, startup boost, or template persist
• CREATE initial sizing is skipped
• ResizeBlocked=True with reason=NamespaceFrozen
• webhook allow message: cannot read namespace for attune.io/freeze; skipping initial sizing (check namespaces get/list/watch RBAC)

Chart installs with rbac.create=true already have the verbs. If you maintain your own ClusterRole, add:

- apiGroups: [""]
  resources: [namespaces]
  verbs: [get, list, watch]

Then confirm the operator ServiceAccount can get the policy namespace. See Troubleshooting: NamespaceFrozen. The same annotation (attune.io/freeze=true) is the intentional incident kill-switch: recommendations and pending safety revert still run.

Total increase budgets now warn at admission

Setting maxTotalCpuIncrease or maxTotalMemoryIncrease emits an admission warning that those fields are deprecated. Prefer maxCpuIncreasePerMinute / maxMemoryIncreasePerMinute so the cap does not depend on reconcileInterval. The old fields still apply until you migrate.

OneShot selection walks past a blocked first replica

OneShot no longer sticks on the first listed replica when that pod is already at the applied target, or when MemoryPressure, Infeasible plus InPlaceOnly, quota, or QoS would skip it. Later replicas in the same workload can now move in the same cycle.

RequestsAndLimits can apply a limit-only resize

When requests already match the recommendation, RequestsAndLimits may still apply a resize that only changes limits (for example a missing or drifted memory limit). Previously a request match could skip the whole container. Check resize history if you see limit-only updates.

Memory usage floor uses the live container limit

The usage floor compares the target against the live container limit on the pod, not the workload template. After an in-place resize the template (and rec.Current) can lag. Alerts that assume the template limit is authoritative should use the live pod instead. Persist applies the same floor when writing the template, so a lagged rec.Current cannot persist a limit below recent usage.

Last-replica eviction uses a live Running count

Eviction fallback lists pods through the typed Clientset and counts only status.phase=Running with no deletion timestamp. spec.replicas and NotReady or Pending pods do not count. The list is paginated and stops once two Running pods are visible. Concurrent resizes on the same workload serialize that list-plus-evict so two replicas cannot both be evicted in one pass. History reasons are eviction_last_replica, eviction_denied, eviction_list_failed, and eviction_no_selector. A failed eviction attempt is not retried for remaining containers on the same pod in that cycle.

kubectl attune --sort-by rejects unknown values

kubectl attune status and kubectl attune savings now exit 1 when --sort-by is not name, namespace, savings, or age. Scripts that passed a typo and got unsorted output now fail closed.

attune_eviction_total result labels

attune_eviction_total now increments for skipped attempts as well as successes. The result label is success, denied, last_replica, list_failed, or no_selector. Alerts that treated every increment as an eviction should filter result="success".

v0.1.25 to v0.1.26

v0.1.26 tightens stale-recommendation handling, CloudWatch collection, ResourceQuota list fail-closed, VPA memoryFromCpuRatio, and GitOps labels. Ready reason MetricsUnavailable replaces PrometheusUnavailable for all metrics backends. Most existing policy YAML keeps working. Read this section if you inherit maxConcurrentResizes from AttuneDefaults, use CloudWatch, ResourceQuota, VPA with a memory ratio, GitOps labels, Prometheus URL userinfo, or run a self-hosted Git forge.

Ready reason MetricsUnavailable

When the metrics backend cannot be resolved or queried, Ready is now MetricsUnavailable instead of PrometheusUnavailable. Datadog Secret failures and CloudWatch IAM failures use the same reason. Alerts that match reason="PrometheusUnavailable" should also match reason="MetricsUnavailable". The operator still treats the old reason as bootstrap (no requeue jitter) until the next reconcile overwrites it.

Inherit maxConcurrentResizes from AttuneDefaults

The built-in default moved from the CRD OpenAPI schema into the controller so AttuneDefaults can set a cluster-wide value. Policies created before that change already have maxConcurrentResizes: 1 stored in etcd. That explicit 1 wins over AttuneDefaults.

To inherit a cluster-wide value, remove the field from those policies:

kubectl patch attunepolicy POLICY -n NS --type=json \
  -p='[{"op":"remove","path":"/spec/updateStrategy/maxConcurrentResizes"}]'

Helm does not upgrade CRDs on helm upgrade. Apply CRDs before the chart upgrade or new policies on a stale CRD still materialize 1:

kubectl apply --server-side --force-conflicts -f \
  https://github.com/attune-io/attune/releases/latest/download/crds.yaml

Stale recommendations expire

A recommendation is fresh only when the newest finite sample is newer than 3 * queryStep (default 15m). Older in-window samples are marked stale. Empty Prometheus results reuse the last rec only inside that same window. After it expires, the rec drops and Ready can become InsufficientData. Stale recs no longer count toward workloads.withRecommendations or savings gauges.

One-sided Prometheus sample gap

If Prometheus has fresh samples for only one resource (CPU or memory) and the other series is empty or all NaN, Attune no longer recommends the pod-template request for the missing arm. After an in-place resize the template can be stale (256Mi while the live pod is already 1Gi). The missing arm now holds the live request or the last rec instead.

CloudWatch pod prefix

CloudWatch SEARCH no longer emits PodName="prefix*". Quoted CloudWatch tokens are exact matches, so that term matched nothing. Filtering is client-side only.

CloudWatch series and page cap

GetMetricData now stops after 5000 kept series (same default as Prometheus maxSeries) or 20 result pages. The operator keeps partial data and may set Ready PrometheusSeriesCapped. Large Container Insights namespaces no longer paginate without bound. This is fail-soft, not a hard query error.

Prometheus address must not include userinfo

metricsSource.prometheus.address no longer accepts http://user:password@host. The webhook rejects create and update. The reconciler also rejects the address, so existing objects fail immediately after the operator rolls (MetricsUnavailable, SSRF blocked). Move credentials to bearerTokenSecret or headers. Apply the same change on AttuneDefaults and AttuneNamespaceDefaults before you upgrade.

AttuneDefaults GitOps pullRequest validation

AttuneDefaults and AttuneNamespaceDefaults now use the same export.pullRequest rules as AttunePolicy: when enabled is true, repository and tokenSecretRef are required, and apiUrl must be https without userinfo. Defaults CRs that used to persist an incomplete pullRequest block fail on the next UPDATE. Complete the fields or set enabled: false before you upgrade.

Self-hosted GitOps forges

export.pullRequest.apiUrl still rejects private IP literals by default. Set allowPrivateEndpoints: true for RFC1918/ULA self-hosted GitLab, Gitea, or Bitbucket. Loopback and link-local (including IMDS) stay blocked. Corporate HTTPS_PROXY on a private address is no longer treated as the SSRF target.

ResourceQuota List fail-closed

If listing ResourceQuotas or LimitRanges fails (missing list/watch RBAC or an API error), Attune skips request increases. Decreases still apply. This is the same class of fail-closed behavior as node-status unavailability. See troubleshooting: quota list unavailable.

Conflict-check policy list fail-closed

If listing AttunePolicy objects for conflict detection fails, that reconcile no longer computes recommendations. Ready is ConflictCheckFailed instead of looking like bootstrap InsufficientData. Last-known recommendations stay on status. Check RBAC list/watch on attunepolicies and API server health. See troubleshooting: ConflictCheckFailed.

VPA honors memoryFromCpuRatio

When metricsSource.vpa is set and memory.memoryFromCpuRatio is also set, Attune now derives memory from the CPU recommendation instead of using the VPA memory target. Unset memory.memoryFromCpuRatio to keep VPA memory targets.

A VPA target that omits cpu or memory is unset, not zero. Attune holds the live pod request when pods are listed, otherwise the last rec or template request. A cpu-only target no longer feeds 0 into the memory engine, and does not reset live memory back to the template.

A VPA list error (RBAC or apiserver) no longer looks like "no VPA CRD". Apply is skipped (ResizeBlocked=VPAListUnavailable) so an applying VPA is not treated as absent. A missing CRD still skips the conflict check.

GitOps labels fail-closed

If export.pullRequest.labels is set and the forge rejects the labels API, PR create and update fail instead of succeeding unlabeled. Grant the forge label permission or drop labels.

GitLab merge request matching

The GitLab list call now filters by source branch, target branch (baseBranch, default main), and per_page=100. Attune updates an existing open MR only when that MR's target is baseBranch. An MR whose target is not baseBranch is not kept in sync. Attune may open a new MR against baseBranch and leave the old one open.

GitLab update no longer clears labels

A GitLab MR update used to send an empty labels field when export.pullRequest.labels was unset. GitLab treats that as "remove every label." Updates now omit labels unless the policy sets them.

GitLab re-bootstrap after merge

After the first Attune MR merges, the next drift cycle writes a new marker commit (timestamped) so GitLab can open another MR. A reused empty marker file no longer blocks the next cycle.

v0.1.24 to v0.1.25

v0.1.25 is a GitOps reliability patch. Existing policy YAML keeps working.

Apply CRDs on Helm upgrade

Helm does not update CRDs on helm upgrade. This release adds status.gitopsPR (driftFingerprint, lastAttempt, url) so skip state survives a Flux or Argo apply that replaces metadata.annotations. Without the new CRD, the API server prunes that status object and only annotations remain.

Before helm upgrade, apply:

kubectl apply --server-side --force-conflicts -f \
  https://github.com/attune-io/attune/releases/latest/download/crds.yaml

Raw dist/install.yaml / dist/crds.yaml installs already include the field.

Empty GitOps PRs after upgrade

0.1.22 stored last-attempt and PR URL but not a drift fingerprint. After cooldown, 0.1.24 could open another empty PR for the same table. 0.1.25 adopts the live table when a PR URL exists and no fingerprint is stored, and it does not write last-attempt on dry-run (so the first live cycle can still open). See GitOps integration.

v0.1.22 to v0.1.23

v0.1.23 is a reliability patch. Existing policies keep working without YAML edits.

Requeue jitter is skipped during data collection

While Ready is InsufficientData or PrometheusUnavailable, the operator no longer adds --requeue-jitter (default 2m) on top of the cooldown. First recommendations and ConfigMap export can land sooner. Jitter still applies to steady-state cooldown requeues.

Fleet report savings stay finite

estimatedMonthlySavingsUSD adds only finite numbers. Unparseable values (NaN, Inf, or garbage) increment unparseableSavings and count as 0 in the USD total. Consumers of report.json should treat that field as additive.

v0.1.21 to v0.1.22

v0.1.22 ships scale defaults and capacity safety that were previously only on main. Review before upgrading from v0.1.21.

Default PromQL pod aggregation is Max

When metricsSource.podAggregation is unset, Attune now defaults to Max (max by (container) over the selected series) instead of leaving series unaggregated. Recommendations then follow the hottest pod for each container name, and Prometheus query cost stays proportional to containers rather than replicas.

If you relied on multi-pod sample pools (legacy unaggregated behavior), set explicitly:

spec:
  metricsSource:
    podAggregation: None   # or Avg

See Scaling: PromQL aggregation and Scaling: large fleets for operator flags related to large fleets (maxPodsInMetricsQuery, maxProfileSamples, informer field strip).

Default max concurrent reconciles is 2

The operator default for --max-concurrent-reconciles is 2 (was higher in some earlier builds). Helm clusterSize presets still override this when set. Large clusters that previously relied on more concurrent reconcilers should set the flag or a preset explicitly.

Request increases fail closed when node status is unavailable

If the operator cannot read node status for a pod, it skips request increases (decreases still allowed) and increments attune_capacity_skip_total{reason="unavailable"}. This is protective if node API access or informer lag fails.

Batch throttle chunking (no config change)

Safety observation batches CPU throttle PromQL queries and splits large pod/container sets into chunks of 64. No CRD field changes; Prometheus load for high-replica policies should drop further under rate limiting.

v0.1.20 to v0.1.21

v0.1.21 is a feature release. Existing policies keep working without YAML edits. Most new capabilities are opt-in. Read this section if you run Kubernetes 1.35+, GitOps export, multi-cluster rollups, or memory limit control.

Safe by default (no action required)

Area	Behavior
GitOps pull request automation	Off unless export.pullRequest.enabled: true
Fleet report ConfigMap export	Off unless fleetReport (or Helm equivalent) is enabled
Runtime profiles	Only apply when runtimeProfile is set
Export schema versioning	Additive fields on recommendation ConfigMaps; consumers can ignore new keys

Behavior that can change without new fields

Memory limit decreases on Kubernetes 1.35+. On 1.35+, Attune no longer clamps memory limits the way it did on 1.33/1.34 when the platform allows live decreases and the policy uses controlledValues: RequestsAndLimits with decrease allowed. A usage floor still keeps the target limit above recent usage (default memory.decreaseUsageMarginPercent: 10). On 1.33–1.34, limit decreases remain clamped as before.

If you rely on “limits never go down in place,” pin an older cluster version, set memory.allowDecrease: false, use a restrictive runtime profile, or keep controlledValues: RequestsOnly (the default).

Capacity and node pressure. Resizes may be skipped more often when nodes are under pressure; metrics and status explain the skip. This is protective, not a CRD break.

Opt-in features worth enabling deliberately

Feature	Where to start
GitOps PR automation	GitOps integration
Multi-cluster fleet report	Multi-cluster
Language runtime profiles	Runtime profiles
Deferred / Infeasible UX	Status conditions + troubleshooting
SLO PromQL guardrails (unchanged API; guide improved)	SLO guardrails

Operator / install notes

• Refresh CRDs with the release install path (helm upgrade or dist/crds.yaml / dist/install.yaml from the tag).
• Grafana dashboard and PrometheusRule assets gain panels/alerts for GitOps PR outcomes, memory limit decrease safety, capacity skips, and related signals. Re-apply chart or dashboard ConfigMaps if you manage them out of band.
• kubectl attune explain surfaces GitOps PR and runtime profile effective values; upgrade the plugin with the release for matching CLI help.

After upgrade, confirm policies with:

kubectl attune status -A
kubectl attune explain -n <namespace> <policy>

v1alpha1 Field Renames (v0.1.0)

Five CRD fields were renamed to align with ecosystem conventions. Existing AttunePolicy, AttuneDefaults, and AttuneNamespaceDefaults resources must be updated before applying the new CRDs.

Field mapping

Old field	New field	Conversion
safetyMargin: "1.2"	overhead: "20"	(old - 1) * 100
updateStrategy.mode	updateStrategy.type	rename only
bounds.min / bounds.max	minAllowed / maxAllowed	rename only
InPlaceOrEvict	InPlaceOrRecreate	rename only
excludeContainers	excludedContainers	rename only
updateStrategy.maxCpuChangePercent	cpu.maxChangePercent	move to cpu section
updateStrategy.maxMemoryChangePercent	memory.maxChangePercent	move to memory section

Overhead conversion examples

Old safetyMargin	New overhead	Meaning
"1.1"	"10"	10% headroom
"1.15"	"15"	15% headroom
"1.2"	"20"	20% headroom (CPU default)
"1.3"	"30"	30% headroom (memory default)
"1.5"	"50"	50% headroom

Automated migration

Using sed (covers all five renames):

# All five renames in one pass
sed -i \
  -e 's/safetyMargin:/overhead:/g' \
  -e 's/overhead: "1.1"/overhead: "10"/g' \
  -e 's/overhead: "1.15"/overhead: "15"/g' \
  -e 's/overhead: "1.2"/overhead: "20"/g' \
  -e 's/overhead: "1.25"/overhead: "25"/g' \
  -e 's/overhead: "1.3"/overhead: "30"/g' \
  -e 's/overhead: "1.5"/overhead: "50"/g' \
  -e 's/InPlaceOrEvict/InPlaceOrRecreate/g' \
  -e 's/excludeContainers:/excludedContainers:/g' \
  manifests/*.yaml

# mode -> type (only in updateStrategy context to avoid false positives)
sed -i '/updateStrategy/,/^[^ ]/{s/mode:/type:/g}' manifests/*.yaml

# bounds.min/max -> minAllowed/maxAllowed (remove nesting manually if used)

Using yq (handles overhead conversion and bounds restructuring):

# Export current policies
kubectl get attunepolicies -n production -o yaml > policies.yaml

# Rename safetyMargin to overhead and convert values
yq -i '
  .items[].spec.cpu |= (
    .overhead = ((.safetyMargin | tonumber - 1) * 100 | tostring) |
    del(.safetyMargin)
  ) |
  .items[].spec.memory |= (
    .overhead = ((.safetyMargin | tonumber - 1) * 100 | tostring) |
    del(.safetyMargin)
  )
' policies.yaml

# Apply the new CRDs first, then re-apply policies
kubectl apply -f config/crd/bases/
kubectl apply -f policies.yaml

Helm values migration

If you use the Helm chart with custom defaults.cpu.overhead or defaults.memory.overhead in your values.yaml, update the values:

# Before
defaults:
  cpu:
    safetyMargin: "1.2"
  memory:
    safetyMargin: "1.3"

# After
defaults:
  cpu:
    overhead: "20"
  memory:
    overhead: "30"