Prometheus Setup¶

Attune relies on Prometheus for historical CPU and memory usage data. This guide covers which metrics are required, how to configure the Prometheus address, and how to verify the integration is working.

The operator process queries Prometheus. A token is not stored on each AttunePolicy unless that policy, or its namespace, brings its own.

Choose how Attune authenticates¶

	Prometheus auth	Where the credential lives	Operator token
1	None	Nowhere	Not sent
2	Per namespace	Secret in that namespace	Not sent
3	One identity for the cluster	Operator namespace	Sent for the cluster address
4	Cluster identity, plus a namespace exception	Operator namespace, and a Secret in the exception namespace	Sent only where the address is still inherited from cluster `AttuneDefaults`

1. No authentication¶

Set the address and leave credentials unset. This is the in-cluster Prometheus that the chart examples use, usually http://prometheus-server.monitoring:80.

apiVersion: attune.io/v1alpha1
kind: AttuneDefaults
metadata:
  name: cluster-defaults
spec:
  metricsSource:
    prometheus:
      address: http://prometheus-server.monitoring:80

Policies that omit metricsSource inherit that address. See examples/05-cluster-defaults.yaml.

Helm does not send a token unless you set prometheusAuth or openshift.bindClusterMonitoringView. A default helm install is this case.

2. A token in each namespace¶

Put bearerTokenSecret on the AttunePolicy or on AttuneNamespaceDefaults, and create that Secret in the same namespace. Also set prometheus.address on that same object. A Secret name by itself is not attached to an address inherited from cluster AttuneDefaults.

One AttuneNamespaceDefaults can serve every policy in the namespace. Those policies omit metricsSource.

apiVersion: attune.io/v1alpha1
kind: AttuneNamespaceDefaults
metadata:
  name: production-prometheus
  namespace: production
spec:
  metricsSource:
    prometheus:
      address: https://prometheus.example.com
      bearerTokenSecret:
        name: prom-token
        key: token

kubectl -n production create secret generic prom-token \
  --from-file=token=./prom-token

The operator reads production/prom-token. It does not send its own ServiceAccount token to this address. Cross-namespace names such as attune-system/prom-token are rejected.

The same shape on a single AttunePolicy is in examples/13-multi-datasource.yaml.

3. One identity for the whole cluster¶

Put only the address on cluster AttuneDefaults. Do not set bearerTokenSecret there. That field still copies the Secret name into each policy namespace (deprecated; admission warns).

apiVersion: attune.io/v1alpha1
kind: AttuneDefaults
metadata:
  name: cluster-defaults
spec:
  metricsSource:
    prometheus:
      address: https://thanos-querier.openshift-monitoring.svc:9091
      tls:
        insecureSkipVerify: true

Prefer the cluster CA over insecureSkipVerify when you have the bundle. Then turn on operator auth once.

Helm on OpenShift:

openshift:
  bindClusterMonitoringView: true
prometheusAuth:
  queryServiceAccount:
    create: true

That binds cluster-monitoring-view to a query ServiceAccount and sends that token for every policy whose address came from cluster AttuneDefaults.

Helm elsewhere, when you already bound the query account yourself:

prometheusAuth:
  useServiceAccountToken: true
  queryServiceAccount:
    create: true

A long-lived token in the operator namespace, for Mimir or Grafana Cloud, is prometheusAuth.existingSecret instead of the ServiceAccount token. See OpenShift: Thanos Querier.

OperatorHub and the kustomize install already pass --prometheus-use-service-account-token and --prometheus-query-service-account=attune-prometheus-query. Bind cluster-monitoring-view to attune-prometheus-query once:

oc adm policy add-cluster-role-to-user cluster-monitoring-view \
  -z attune-prometheus-query -n <operator-namespace>

The operator token is not sent to an address set on a policy, an address set on AttuneNamespaceDefaults, or an auto-discovered Prometheus. It is also skipped when the resolved config already has an Authorization header.

4. Cluster identity, with a namespace exception¶

Use case 3 for the cluster. In the namespace that needs a different token, or a different Prometheus, set both the address and bearerTokenSecret on AttuneNamespaceDefaults (or on that AttunePolicy).

apiVersion: attune.io/v1alpha1
kind: AttuneNamespaceDefaults
metadata:
  name: payments-prometheus
  namespace: payments
spec:
  metricsSource:
    prometheus:
      address: https://prometheus.example.com
      bearerTokenSecret:
        name: prom-token
        key: token

payments uses payments/prom-token. Every other namespace that still inherits the cluster address keeps the operator token.

Repeating prometheus.address on AttuneNamespaceDefaults selects this exception, even when the URL string matches the cluster address. Omit the prometheus block on AttuneNamespaceDefaults when that namespace should keep the cluster address and the operator token. Setting only bearerTokenSecret, with no address, does not attach that Secret to the cluster address.

Worked YAML is in examples/prometheus-auth/.

Amazon Managed Prometheus¶

Amazon Managed Prometheus does not accept a bearer token. Set sigv4 on the Prometheus address. Attune signs each query with AWS SigV4 for service aps. Omitted sigv4 does not sign. There is no Helm value and no environment variable that turns signing on.

The address is the workspace root. Attune appends /api/v1/query. If you copy the console query URL, remove the /api/v1/query suffix. Leaving it on sends the query to /api/v1/query/api/v1/query.

apiVersion: attune.io/v1alpha1
kind: AttunePolicy
metadata:
  name: payments
  namespace: payments
spec:
  targetRef:
    kind: Deployment
    name: payments
  metricsSource:
    prometheus:
      address: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-example
      sigv4:
        region: us-east-1
        roleArn: arn:aws:iam::123456789012:role/attune-amp

region is required when sigv4 is set. It does not default from the AWS SDK environment. roleArn is optional. Leave it unset to use the pod identity chain (IRSA or Pod Identity). A set roleArn is assumed through STS. The role needs aps:QueryMetrics. See the AMP QueryMetrics API.

On AttunePolicy and AttuneNamespaceDefaults, a set roleArn must match --sigv4-allowed-role-arns (sigv4.allowedRoleArns in Helm). sigv4 with no roleArn signs with the operator identity, and the workspace host must match --sigv4-allowed-workspace-hosts (sigv4.allowedWorkspaceHosts). Both lists are empty until you set them. Cluster AttuneDefaults is not filtered. A namespaced assume sends ExternalId attune:<namespace>, so the role trust policy can require that value. A trust policy that does not check ExternalId still works.

{
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::123456789012:role/attune-operator"},
  "Action": "sts:AssumeRole",
  "Condition": {"StringEquals": {"sts:ExternalId": "attune:payments"}}
}

Do not set bearerTokenSecret, an Authorization header, or an X-Amz-* header together with sigv4. The webhook rejects that combination. When sigv4 is set, Attune does not attach prometheusAuth, the operator ServiceAccount token, or the operator bearer Secret.

Required Prometheus metrics¶

The operator queries these metrics, all scraped automatically by cadvisor (built into the kubelet):

Metric	Query	What it measures
`container_cpu_usage_seconds_total`	`rate(...[5m])`	CPU cores consumed per container
`container_memory_working_set_bytes`	instant	Memory actively used per container
`container_cpu_cfs_throttled_periods_total`	`rate(...[5m]) / rate(cfs_periods[5m])`	CPU throttle ratio (safety monitor)
`container_cpu_cfs_periods_total`	used in throttle ratio denominator	Total CPU scheduling periods

The first two metrics are required for recommendations. The CFS throttle metrics are used by the safety monitor when autoRevert: true (default) to detect CPU under-provisioning after a resize. If these metrics are missing, throttle detection is silently skipped.

These metrics are available out of the box in any Prometheus installation that scrapes the kubelet's /metrics/cadvisor endpoint. No additional exporters or recording rules are needed.

Note

The queries filter by namespace, pod (regex prefix match), and container name. If your Prometheus relabels these labels, the queries will return empty results.

Prometheus address resolution¶

The operator resolves the Prometheus address in this order:

flowchart TD
    A[Policy spec<br/>metricsSource.prometheus.address] -->|set?| Z[Use it]
    A -->|not set| B{Namespace defaults exist?}
    B -->|yes| C[AttuneNamespaceDefaults<br/>metricsSource.prometheus.address]
    C --> D{Address set?}
    D -->|yes| Z
    D -->|no| G[Auto-discovery:<br/>Prometheus Operator CRD]
    B -->|no| E[AttuneDefaults<br/>metricsSource.prometheus.address]
    E --> F{Address set?}
    F -->|yes| Z
    F -->|no| G
    G -->|found?| Z
    G -->|not found| H[Auto-discovery:<br/>well-known service names]
    H -->|found?| Z
    H -->|not found| I[MetricsUnavailable<br/>condition set]

1. Policy-level address (highest priority)¶

spec:
  metricsSource:
    prometheus:
      address: http://prometheus-server.monitoring:80

Use this when different namespaces use different Prometheus instances.

If you configure metricsSource.prometheus.bearerTokenSecret on an AttunePolicy or AttuneNamespaceDefaults, the Secret must live in that namespace. Cross-namespace Secret names (ns/name) are rejected. See Choose how Attune authenticates.

On cluster AttuneDefaults, metricsSource.datadog.apiKeySecretRef and updateStrategy.export.pullRequest.tokenSecretRef have the same inherit-name behavior. Put those Secrets on the policy or AttuneNamespaceDefaults, or use the operator-namespace Datadog Secret described in Datadog setup.

OpenShift Thanos steps are in Thanos Querier.

Use an in-cluster address

The operator validates metricsSource.prometheus.address to block loopback and cloud metadata endpoints. http://127.0.0.1:9090, http://[::1]:9090, http://169.254.169.254/..., and metadata hostnames are rejected, including a trailing dot on those names and other spellings of the same addresses. Addresses with URL userinfo (http://user:password@host) are also rejected; use bearerTokenSecret, headers, or sigv4. Do not point a policy at a local port-forward or a workstation URL. Use a Service DNS name or ClusterIP that the operator can reach from inside the cluster, such as http://prometheus-server.monitoring:80. Private cluster IPs are allowed.

2. Namespace defaults¶

apiVersion: attune.io/v1alpha1
kind: AttuneNamespaceDefaults
metadata:
  name: team-defaults
  namespace: production
spec:
  metricsSource:
    prometheus:
      address: http://prometheus-server.monitoring:80

Policies that omit metricsSource.prometheus.address inherit from this first. Use namespace defaults when different teams or environments need different Prometheus backends.

An AttuneNamespaceDefaults that leaves metricsSource.prometheus unset, and does not select Datadog, CloudWatch, or VPA, still inherits the cluster address and, when operator auth is on, the operator token. An AttuneNamespaceDefaults that sets metricsSource.prometheus replaces the cluster Prometheus block. Put the address on that object. That address does not receive the operator token. A Prometheus block with no address does not fall through to the cluster address; the controller then tries auto-discovery.

3. Cluster-wide defaults¶

apiVersion: attune.io/v1alpha1
kind: AttuneDefaults
metadata:
  name: default
spec:
  metricsSource:
    prometheus:
      address: http://prometheus-server.monitoring:80

Policies that omit metricsSource.prometheus inherit this address. Put address, headers, query parameters, and TLS here. For a shared token, use case 3 above. Do not put bearerTokenSecret on this object.

4. Auto-discovery (Prometheus Operator)¶

If the Prometheus Operator is installed, Attune lists monitoring.coreos.com/v1 Prometheus resources and constructs the address from the first one found:

http://prometheus-<name>.<namespace>:<port>

No configuration needed. This works with kube-prometheus-stack and any Prometheus Operator deployment.

5. Auto-discovery (well-known services)¶

As a last resort, the operator checks for services with well-known names in common namespaces:

Namespace	Service name
`monitoring`	`prometheus-server`
`monitoring`	`prometheus-kube-prometheus-prometheus`
`prometheus`	`prometheus-server`
`kube-prometheus-stack`	`prometheus-kube-prometheus-prometheus`

If found, the actual port from the Service spec is used (falls back to 9090 if no ports are defined).

Service port vs process port

The Prometheus process usually listens on port 9090, but the Kubernetes Service may expose a different port (e.g., port 80 in the prometheus-community chart). Well-known service auto-discovery uses the Service port from the Service spec and only falls back to 9090 when the Service declares no ports. Set the address explicitly only if you use a non-standard Service name or namespace, or if you want to bypass auto-discovery.

Common Prometheus installations¶

prometheus-community/prometheus (Helm)¶

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus \
  --namespace monitoring --create-namespace \
  --set server.persistentVolume.enabled=true

The Service is prometheus-server.monitoring on port 80 (not 9090):

# AttuneDefaults. No token: this is case 1 above.
spec:
  metricsSource:
    prometheus:
      address: http://prometheus-server.monitoring:80

kube-prometheus-stack (Helm)¶

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prom prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

The Service is prometheus-kube-prometheus-prometheus.monitoring on port 9090:

spec:
  metricsSource:
    prometheus:
      address: http://prometheus-kube-prometheus-prometheus.monitoring:9090

Auto-discovery (both Prometheus Operator CRD and well-known service name) works out of the box with this stack.

Prometheus Operator (standalone)¶

If you deploy Prometheus via the Prometheus Operator's Prometheus CRD, auto-discovery finds it automatically. No address configuration needed.

Verifying the integration¶

Step 1: Check the Prometheus Service port¶

kubectl get svc -n monitoring prometheus-server
# NAME                TYPE        CLUSTER-IP     PORT(S)
# prometheus-server   ClusterIP   10.96.x.x      80/TCP

Use the PORT(S) column value, not 9090.

Step 2: Verify cadvisor metrics exist¶

kubectl run prom-check --image=curlimages/curl --restart=Never --rm --attach --command -- \
  curl -s 'http://prometheus-server.monitoring:80/api/v1/query?query=container_cpu_usage_seconds_total' \
  | head -c 200

You should see "status":"success" with result data. If you see "resultType":"vector","result":[], cadvisor scraping is not configured.

Step 3: Test a namespace-scoped query¶

Replace <namespace> and <pod-prefix> with a real workload:

kubectl run prom-check --image=curlimages/curl --restart=Never --rm --attach --command -- \
  curl -s 'http://prometheus-server.monitoring:80/api/v1/query?query=rate(container_cpu_usage_seconds_total{namespace="<namespace>",pod=~"<pod-prefix>.*"}[5m])'

Non-empty results confirm Attune can query metrics for that workload.

Step 4: Check policy conditions¶

kubectl get attunepolicy -A

Condition	Meaning
`Ready: True, Reason: Monitoring`	Prometheus reachable, recommendations computed
`Ready: False, Reason: InsufficientData`	Prometheus reachable but not enough history yet
`Ready: False, Reason: MetricsUnavailable`	Prometheus could not be used for this reconcile. Check the condition message and Troubleshooting for address, auth/TLS, timeout, or query failures.

If the condition is InsufficientData, wait for enough samples to accumulate. By default, recommendations need minimumDataPoints: 48 Prometheus range-query samples. With the default queryStep: 5m, that is about 4 hours of data within the default historyWindow: 168h.

Operator metrics (what Attune exposes)¶

Attune itself exposes Prometheus metrics on its :8080/metrics endpoint. To scrape these, either:

Enable the Helm chart's ServiceMonitor (metrics.serviceMonitor.enabled: true), or
Add a scrape annotation to the operator pod

See Metrics Reference for the full list of attune_* metrics.

Grafana dashboard¶

Option A: Auto-provision via sidecar (kube-prometheus-stack)¶

If you use the Grafana sidecar from kube-prometheus-stack, enable the dashboard ConfigMap:

helm upgrade attune oci://ghcr.io/attune-io/charts/attune \
  --set grafanaDashboard.enabled=true

This creates a ConfigMap with the grafana_dashboard: "1" label. The Grafana sidecar automatically detects it and imports the dashboard.

Option B: Manual import (standalone Grafana)¶

If your Grafana is not using the sidecar pattern:

Download the dashboard JSON from deploy/grafana/dashboard.json
In Grafana, go to Dashboards > New > Import
Paste the JSON or upload the file
Select your Prometheus data source when prompted

The dashboard covers resizes, reverts, savings, recommendations, confidence scores, reconcile latency, and Prometheus query health.

Alerting with PrometheusRule¶

Enable the Helm chart's PrometheusRule to get out-of-the-box alerts:

helm upgrade attune oci://ghcr.io/attune-io/charts/attune \
  --set metrics.prometheusRule.enabled=true

This creates 15 alerts covering the key operational signals:

Alert	Fires when	Default severity
`AttuneReconcileErrors`	Reconcile error rate > 0 sustained for 10m	warning
`AttunePrometheusUnreachable`	Prometheus query errors sustained for 10m	warning
`AttuneDegraded`	A workload is reverted more than 3 times in 15 minutes, for 5m	critical
`AttuneHighRevertRate`	More than 50% of resizes in the last hour were reverted, for 15m	critical
`AttuneReconcileStale`	No successful reconcile in the last 30m, for 5m	warning
`AttuneBudgetExhausted`	Increase budget refused a resize in the last hour, for 30m: `BudgetExhausted` (retried later) or `IncreaseExceedsBudget` (larger than a per-minute or per-cycle cap)	warning
`AttunePodsInfeasible`	Pods stuck Infeasible for in-place resize for 30m	warning
`AttunePodsDeferred`	Pods stuck Deferred for in-place resize for 1h	warning
`AttuneDataQuality`	NaN/Inf values from Prometheus for 30m	warning
`AttuneRequestsClamped`	Recommendations capped at limits for 1h	info
`AttuneStaleRecommendations`	Recommendations stale due to data gaps for 1h	warning
`AttuneRevertFailures`	Resize revert failed for 5m	critical
`AttuneGitOpsPRFailures`	GitOps pull request automation failed for 5m	warning
`AttuneMemoryLimitUnsafe`	Memory limit decrease floored or skipped as unsafe for 1h	info
`AttuneOOMBumpCapped`	One or more OOM bumps capped at maxBumps or clamped to maxAllowed in the last hour, pending 5m	info

Individual alerts can be disabled or tuned:

metrics:
  prometheusRule:
    enabled: true
    rules:
      reconcileErrors:
        severity: critical   # escalate to critical
      degraded:
        enabled: false       # disable this alert
      reconcileStale:
        staleDuration: 1h    # fire after 1 hour instead of 30m

See the Helm chart README for the full list of configurable parameters.

Full observability stack¶

To enable metrics scraping, alerts, and the Grafana dashboard in one command:

helm upgrade attune oci://ghcr.io/attune-io/charts/attune \
  --set metrics.enabled=true \
  --set metrics.serviceMonitor.enabled=true \
  --set metrics.prometheusRule.enabled=true \
  --set grafanaDashboard.enabled=true

Or in a values file:

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
  prometheusRule:
    enabled: true
grafanaDashboard:
  enabled: true