Introduction

Every Helm upgrade creates a new numbered revision. If the upgrade breaks something, you roll back to a previous revision with a single command. This revision history is stored as Kubernetes secrets in the release namespace and survives across cluster restarts, making rollback a reliable recovery mechanism when deployments go wrong.

Rollback matters in production because not every failed deployment is caught by pre-deployment testing. A config change that seemed safe in staging may interact unexpectedly with production data. A new image tag may have a subtle bug that only manifests under production load. When this happens, you need to restore the previous working state quickly, and Helm rollback lets you do that without manually reverting changes in Git or applying old manifests.

This guide covers Helm release revision history, manual rollback procedures, automated rollback in CI/CD pipelines, and hooks for pre and post-upgrade tasks. You will learn when rollback is the right tool and when database migrations or forward fixes are required instead. By the end, you will be able to inspect release history, execute rollbacks safely, implement automated rollback with the atomic flag, and design upgrade processes that survive real-world failures.

When to Use / When Not to Use

When Helm rollback makes sense

Helm rollback is the right tool when you have an intact release history and the previous state is known and safe. Database schema changes that went wrong, config changes that broke behavior, and canary deployments that revealed problems are all good rollback candidates. The --atomic flag makes automated rollback in CI/CD practical for these cases.

Helm rollback also works well when your application is stateless or when storage resources are not affected by the change. A bad Deployment image tag or a misconfigured ConfigMap rolls back cleanly because the underlying Kubernetes resources are reverted to their previous definitions.

The clearest rollback wins are ConfigMap or Secret changes that break application behavior — rolling back restores the working values immediately. Image tag rollbacks where a new tag has a subtle bug are also clean: Helm reapplies the old image reference. Replica count changes that cause OOMkills, and annotation or label drift where production tooling depends on specific metadata, round out the list.

Rollback is also the fastest path to recovery when you are in the middle of an incident and the previous release was working acceptably. Running helm rollback takes seconds; debugging a misconfigured Deployment under pressure takes far longer.

When rollback is not enough

If your upgrade included database migrations that modified schema, rollback does not undo those changes. PostgreSQL migrations that added columns with non-null constraints, MongoDB schema changes, and any irreversible data transformation cannot be fixed by rolling back the Kubernetes resources.

In these cases, you need forward-fix migrations, not rollback. Design your upgrade process to handle this before deploying.

Beyond databases, rollback struggles with any change that has effects outside the cluster. A third-party API contract change — a payment processor API version, a webhook payload format — cannot be undone by rolling back your Kubernetes resources. Data written to S3 or GCS under a new prefix scheme is already there. An Ingress rule change that allowed traffic which wrote to your database leaves that data regardless of whether you roll back the Ingress resource.

The practical test: if the blast radius of the bad change extends beyond the Kubernetes cluster state that Helm controls, rollback cannot fully undo it. In these scenarios, design your upgrade process to handle forward-fix recovery and test the failure path in staging before deploying.

Rollback Decision Flow

Keep this diagram handy when things break at 3am. It walks you through a deployment failure by asking two questions: was this from a Helm upgrade or from a config change? And if it was a config change, does rolling back actually fix it?

The branches matter because rollback does not undo everything. PersistentVolumeClaims and storage provisioners have their own constraints — some rollback operations fail silently, leaving your storage in an unexpected state. If you have PVCs that were deleted in the new version, a rollback will not bring them back.

Three failure patterns show up most often in practice. Config changes that broke something roll back cleanly — Helm just reapplies the old ConfigMap or Secret. Database migrations never roll back safely — the schema change is already done, and you need a forward-fix migration to undo it. Stateful workload changes fall in between: back up your data first, then roll back.

Pin this somewhere accessible during incidents. When production is on fire and you are trying to remember the right recovery procedure, the flowchart gives you a straight path through the decision tree instead of improvising under pressure.

flowchart TD
    A[Deployment fails<br/>or degraded] --> B{Upgrade or<br/>Config change?}
    B -->|Config| C{Rollback<br/>fixes it?}
    B -->|Migration| D[Forward-fix<br/>required]
    C -->|Yes| E[helm rollback<br/>to previous revision]
    C -->|No| D
    E --> F{Stateful<br/>resources affected?}
    F -->|Yes| G[Backup data<br/>then rollback]
    F -->|No| H[Rollback safe<br/>proceed]

Release Revision History

Helm stores release history in Kubernetes secrets within the release namespace. Each time you install, upgrade, or roll back, Helm creates a new revision.

# List releases
helm list

# List releases with history
helm history myapp

REVISION  UPDATED                  STATUS     CHART          DESCRIPTION
1         2026-03-20 10:30:00      superseded myapp-1.0.0    Install complete
2         2026-03-21 14:15:00      superseded myapp-1.1.0    Upgrade complete
3         2026-03-22 09:00:00      deployed   myapp-1.2.0     Upgrade complete

# Get detailed release status
helm status myapp

# Show release values at any revision
helm get values myapp --revision 2

# Show all hooks and values
helm get all myapp --revision 3

History is stored as Kubernetes secrets:

kubectl get secrets -n mynamespace -l "owner=helm"

NAME                     TYPE       DATA  AGE
sh.helm.release.v1.myapp.v1   helm.sh/release.v1   1   5d
sh.helm.release.v1.myapp.v2   helm.sh/release.v1   1   4d
sh.helm.release.v1.myapp.v3   helm.sh/release.v1   1   3d

Manual Rollback Procedures

Rolling back to a previous revision takes a single command:

# Rollback to previous revision
helm rollback myapp

# Rollback to specific revision
helm rollback myapp 2

# Rollback with timeout
helm rollback myapp 1 --timeout 5m

# Dry-run rollback
helm rollback myapp 1 --dry-run

Helm performs the rollback by applying the stored manifests from that revision. This is not a reverse diff but a full recreation of the state.

# Watch rollback progress
helm rollback myapp 1 --wait

# Rollback with cleanup on failure
helm rollback myapp 1 --cleanup-on-fail

The --cleanup-on-fail flag causes Helm to delete new resources that were created by the failed upgrade but are not present in the rollback target revision.

Automated Rollback in CI/CD

In automated pipelines, you need criteria for when to trigger a rollback:

# GitHub Actions workflow excerpt
- name: Deploy to production
  run: |
    helm upgrade --install myapp ./charts/myapp \
      --namespace production \
      --values ./config/production.yaml \
      --wait --timeout 10m \
      --atomic

  env:
    KUBECONFIG: /tmp/kubeconfig

- name: Verify deployment
  run: |
    # Check rollout status
    kubectl rollout status deployment/myapp -n production

    # Run smoke tests
    curl -f https://myapp.example.com/health || exit 1

    # Check metrics
    prometheus query "app:http_requests_total{app='myapp'}" > /dev/null

The --atomic flag automatically rolls back on failure:

helm upgrade --install myapp ./charts/myapp \
  --atomic \
  --timeout 5m

If the upgrade fails or pods do not become ready within the timeout, Helm automatically rolls back to the previous successful revision.

Prometheus-based rollback:

# deploy-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: deploy-myapp
  namespace: cicd
spec:
  template:
    spec:
      containers:
        - name: deploy
          image: helm/helm:3.14
          command:
            - /bin/bash
            - -c
            - |
              # Deploy
              helm upgrade --install myapp ./charts/myapp \
                --wait --timeout 10m

              # Wait for metrics
              sleep 60

              # Check error rate
              ERROR_RATE=$(prometheus query "rate(http_errors_total{app='myapp'}[5m])")
              if [ "$ERROR_RATE" > "0.01" ]; then
                echo "Error rate too high: $ERROR_RATE"
                helm rollback myapp
                exit 1
              fi

Hooks for Pre/Post Upgrade Tasks

Helm hooks run arbitrary Kubernetes jobs at specific points in the release lifecycle:

# templates/backup-hook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: "{{ .Release.Name }}-pre-upgrade-backup"
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": hook-succeeded,hook-failed
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: backup
          image: postgres-client:15
          command:
            - /bin/bash
            - -c
            - |
              pg_dump -h postgres -U app $DATABASE > /backups/pre-upgrade.sql

# templates/migration-hook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: "{{ .Release.Name }}-db-migration"
  annotations:
    "helm.sh/hook": post-upgrade
    "helm.sh/hook-weight": "5"
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myapp/migrator:1.5
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: myapp-db-credentials
                  key: connection-string
      restartPolicy: OnFailure

Hook weights control execution order. Lower weights run first for pre-upgrade hooks, higher weights run first for post-upgrade hooks.

Delete policies:

• hook-succeeded: Delete job after successful completion
• hook-failed: Delete job after failed execution
• before-hook-creation: Delete existing job before creating new one

Rollback Failure Scenarios

Some situations prevent straightforward rollbacks:

Resource no longer exists in old revision:

If the previous version created resources that were deleted in a later version, rollback will recreate them.

Storage resource changes:

Storage classes and persistent volumes may not be rollback-able depending on their provisioner.

API version deprecations:

If you upgraded to a resource using a newer API version that is no longer available in the old version, rollback cannot proceed.

Manual edits:

If someone manually edited Kubernetes resources managed by Helm, rollback may conflict with those changes.

# Force rollback (may leave resources in unexpected state)
helm rollback myapp 1 --force

# Check what will happen before rolling back
helm template myapp --revision 1 ./charts/myapp > /tmp/rollback.yaml
kubectl diff -f /tmp/rollback.yaml

Best Practices for Production

Always use --wait and --timeout:

helm upgrade myapp ./charts/myapp \
  --wait \
  --timeout 10m \
  --cleanup-on-fail

Keep reasonable history:

# Limit secrets stored by Helm
helm.sh/resource-policy: keep

# Or use postrenderer to exclude hook resources

Test rollbacks in staging:

# In your staging pipeline
- name: Rollback test
  run: |
    # Deploy current version
    helm upgrade --install myapp ./charts/myapp --namespace staging

    # Rollback immediately
    helm rollback myapp --namespace staging

    # Verify rollback completed
    helm history myapp --namespace staging

Monitor rollback events:

# Alerting rule
- alert: HelmRollback
  expr: |
    increase(helm_release_rollback_total[5m]) > 0
  labels:
    severity: warning
  annotations:
    summary: "Helm rollback detected"
    description: "Release {{ $labels.name }} rolled back in namespace {{ $labels.namespace }}"

Rollback Trade-offs

Helm rollback is not always the right tool. Here is how it compares to other recovery strategies.

Scenario	Helm Rollback	Forward Fix	Blue-Green Deploy
Config change broke things	Fast (single command)	Takes longer	Fast (switch traffic)
Bad deployment image tag	Fast	Rebuild and redeploy	Fast switch back
Database migration failure	Dangerous (schema already changed)	Correct approach	Roll traffic, fix migration
CRD changes	May not work	Often required	Depends on change type
Data loss risk	High if storage affected	Lower with proper backup	Low

The key question: does the previous revision actually represent a safe state? If your upgrade modified persistent data, rolling back may not undo those changes.

Observability Hooks

Track rollback health and deployment reliability with these monitoring practices.

Key metrics to track:

# Rollback frequency by release
sum(rate(helm_release_rollback_total[5m])) by (name, namespace)

# Failed upgrades that triggered atomic rollback
sum(rate(helm_upgrade_failed_total[5m])) by (name, namespace)

# Release revision count (high count = unstable releases)
helm_release_info{owner=helm}

# Time since last successful deployment
time() - helm_release_last_deployed_timestamp

Alert rules for Helm releases:

# Alert when a rollback occurs
- alert: HelmRollbackExecuted
  expr: increase(helm_release_rollback_total[5m]) > 0
  labels:
    severity: warning
  annotations:
    summary: "Helm rollback executed for {{ $labels.name }}"
    description: "Release {{ $labels.name }} in {{ $labels.namespace }} was rolled back. Investigate the cause."

# Alert when release is in failed state
- alert: HelmReleaseFailed
  expr: helm_release_info{status="failed"} == 1
  labels:
    severity: critical
  annotations:
    summary: "Helm release {{ $labels.name }} is in failed state"
    description: "Release has been failing. Manual intervention may be required."

# Alert on high revision count (unstable release)
- alert: HelmReleaseUnstable
  expr: count(helm_release_info) by (name, namespace) > 10
  labels:
    severity: warning
  annotations:
    summary: "Release {{ $labels.name }} has {{ $value }} revisions"
    description: "High revision count indicates frequent changes or rollbacks. Review release stability."

Debugging commands:

# Get full release history with details
helm history myapp --all

# See what changed between revisions
helm diff revision myapp 2 3

# Check current release status
helm status myapp

# Get all values at a specific revision
helm get values myapp --revision 3

# View the manifest that would be applied at revision 1
helm template myapp --revision 1 ./charts/myapp > /tmp/rev1.yaml

Common Pitfalls / Anti-Patterns

Relying on rollback for database migrations

Rollback does not undo database migrations. If your upgrade runs SQL migrations and then the application fails to start, rolling back the Kubernetes Deployment does not roll back the database schema. This leaves your app in a broken state against a migrated schema.

Always design database migrations to be reversible, or design your upgrade process so that a failed migration aborts the Helm upgrade before the new application code is deployed.

The concrete failure looks like this: your Helm upgrade runs a pre-upgrade hook that executes ALTER TABLE users ADD COLUMN last_seen_at TIMESTAMP NOT NULL DEFAULT now(). The migration succeeds. The new application pods start, but a subtle bug only appears under production load. You run helm rollback. The Kubernetes resources revert, but the database still has the new column with a NOT NULL constraint. The old application code did not expect it — queries fail, pods crash loop.

For PostgreSQL, make migrations additive by default: add columns with defaults, add tables rather than modifying existing ones. For MongoDB, use schema versioning so old code can still read new documents. The rule: a Helm rollback should never be your recovery plan for a failed database migration. Forward-fix migrations are the only reliable path.

Not using —atomic in automated deployments

Without --atomic, a failed upgrade leaves the release locked in pending-upgrade or failed state. The next helm upgrade rejects with “another operation is in progress.” Your CI pipeline is then stuck: it can wait forever, force the upgrade (which causes conflicts), or run helm rollback manually before retrying.

--atomic handles this for you. When the upgrade fails or pods do not become ready before --timeout, Helm rolls back to the previous successful revision before exiting non-zero. No explicit rollback logic needed in your pipeline:

# Without --atomic: you handle rollback yourself
helm upgrade --install myapp ./charts/myapp --wait --timeout 10m
if [ $? -ne 0 ]; then
  helm rollback myapp
  exit 1
fi

# With --atomic: Helm rolls back automatically on any failure
helm upgrade --install myapp ./charts/myapp --wait --timeout 10m --atomic

In automated pipelines, --atomic is not optional — it is the difference between a deployment that self-heals and one that leaves your cluster in a broken state.

Manual edits between deploy and rollback

If someone uses kubectl edit to modify a resource managed by Helm after an upgrade, that change survives the next helm upgrade because Helm overwrites whatever is in the cluster with what is in the chart. It also complicates rollback since the cluster state no longer matches the stored revision.

Use helm.sh/resource-policy: keep annotations to exclude specific resources from Helm management when manual edits are necessary.

The problem manifests in two distinct ways. First, manual edits are overwritten on the next helm upgrade — Helm reads from the chart and applies it to the cluster, so a kubectl edit to a Deployment’s replica count or environment variable gets overwritten without warning. Second, if you roll back after a manual edit, Helm applies the old revision’s manifest which does not include the manual change — the cluster state diverges from what anyone expected.

Consider a real scenario: an SRE manually scales a production Deployment to 20 replicas during peak traffic. The next day a developer runs helm upgrade with an updated image tag. Helm overwrites the replica count back to 3, the chart default. Peak traffic hits, pods OOMkill. The manual scale was never documented, so debugging takes longer than it should.

When manual intervention is unavoidable, use the keep policy:

# templates/deployment.yaml
metadata:
  annotations:
    "helm.sh/resource-policy": keep
spec:
  replicas: { { .Values.replicas } }

This tells Helm to skip updates to this resource during upgrade and rollback. It still owns the initial creation but stops managing subsequent changes. keep on a Deployment is usually safe; keep on a ServiceAccount can cause permission drift.

Not limiting release history

Helm stores every revision as a Kubernetes secret named sh.helm.release.v1.<release>.v<N>. Run the math:50 microservices, 3 deploys a day, 7 days = 450 secrets per release per week. Over a few months, your namespace is full of old revisions, helm list slows down, and you are putting unnecessary pressure on etcd.

The helm.sh/resource-policy: keep annotation on a resource tells Helm to skip deleting it during uninstall:

# In your deployment.yaml template
metadata:
  annotations:
    "helm.sh/resource-policy": keep

This is useful for persistent volumes you want to survive accidental helm uninstall. It does not limit how many revisions Helm stores.

To cap revision history, use --history-max on your install or upgrade:

helm upgrade --install myapp ./charts/myapp \
  --history-max 10 \
  --wait --timeout 10m

When a new revision pushes past the limit, Helm deletes the oldest revision secrets. Keep between 5 and 20 revisions for production — enough to rollback a couple steps, not enough to accumulate months of secrets.

If you already have old revisions cluttering your namespace, prune them manually:

# Check how many revisions you have
helm history myapp

# Delete oldest revisions, keep last 5
REVISIONS=$(helm history myapp --output json | jq -r '.[0:-5].revision')
for rev in $REVISIONS; do
  kubectl delete secrets -n mynamespace "sh.helm.release.v1.myapp.v${rev}"
done

Watch your secret count with kubectl get secrets -n mynamespace -l "owner=helm" | wc -l. Spikes mean a failing pipeline looping on upgrade attempts.

Forgetting to test rollback in staging

A rollback procedure that has never been tested is unreliable. If your first rollback attempt happens during a production incident at 3am, you do not want to discover that your rollback hook has a bug.

Test rollback in staging on every significant release.

What a proper rollback test actually looks like in practice:

# staging-pipeline.yaml
- name: Deploy candidate to staging
  run: |
    helm upgrade --install myapp ./charts/myapp \
      --namespace staging \
      --values ./config/staging.yaml \
      --wait --timeout 10m

- name: Wait for stabilization
  run: sleep 30

- name: Execute rollback
  run: |
    helm rollback myapp --namespace staging --wait --timeout 5m

- name: Verify rollback completed
  run: |
    helm history myapp --namespace staging
    helm status myapp --namespace staging

This validates three things: the rollback command completes without error, the release enters deployed state, and the revision counter decrements correctly. If your chart uses pre-rollback or post-rollback hooks, the test also catches hook failures — permission issues, incorrect image references. Hook failures are the most common rollback problem in my experience.

Run this on every release candidate that touches stateful components or modifies hook resources. Skipping it for “minor config changes” is how teams end up with rollback procedures that work in theory but fail when pressure is highest.

Interview Questions

Further Reading

• Helm Charts - Comprehensive Helm chart development guide
• Helm Repository Management - ChartMuseum, Harbor, and repository strategies
• OCI Artifacts Distribution - Chart distribution via OCI registries
• CI/CD Pipelines - Deployment pipeline patterns
• GitOps with ArgoCD and Flux - GitOps deployment strategies
• Helm Documentation - Official Helm documentation

Conclusion

Key Takeaways

• Helm stores every release as a numbered revision; rollback replays the stored state of that revision
• Use --atomic for automated pipelines to guarantee clean state on failure
• Design migrations to be reversible or build forward-fix procedures instead of relying on rollback
• Test rollback in staging before relying on it in production
• Monitor rollback events with Prometheus alerts to catch problems early

Rollback Checklist

# Check release history
helm history myapp

# See what values were deployed at each revision
helm get values myapp --revision 2

# Rollback to previous working revision
helm rollback myapp --wait --timeout 5m

# Rollback to specific revision
helm rollback myapp 1 --wait

# Check rollback succeeded
helm status myapp
helm history myapp

Helm revision history makes rollback straightforward in most cases. Use --atomic for automatic rollback in CI/CD, implement proper hooks for data migration, and test rollback procedures regularly in non-production environments. For more on Helm, see our Helm Charts overview, and for deployment strategies that reduce rollback need, see our CI/CD Pipelines guide. For GitOps patterns with ArgoCD, see our GitOps with ArgoCD and Flux post.