OCI Artifacts: Distributing Container Images and Helm Charts
Package and distribute container images, Helm charts, and other artifacts using the OCI (Open Container Initiative) specification for portable artifact management.
The OCI distribution specification has evolved beyond container images to handle Helm charts, SBOMs, policy bundles, and arbitrary binary artifacts through the same container registry infrastructure used for Docker images. This consolidation simplifies authentication, access control, and retention policies by leveraging a single platform. ORAS extends this by letting you push multiple related artifacts, like an image with its SBOM and cryptographic signature, in a single authenticated operation.
Introduction
The Open Container Initiative (OCI) distribution specification has moved well beyond its original job of just shipping container images. These days it provides a vendor-neutral API for distributing Helm charts, SBOMs, policy bundles, and arbitrary binary artifacts through the same container registry infrastructure your organization already uses for Docker images. This convergence matters because it means you can consolidate artifact management onto a single platform with unified authentication, access control, and retention policies.
OCI artifacts become particularly valuable when you need to ship related pieces as a unit. ORAS (OCI Registry As Storage) lets you push a container image, its software bill of materials, and its cryptographic signature in a single authenticated operation. Helm 3.8+ added native OCI support, so helm chart push works directly against Harbor, Azure Container Registry, Google Container Registry, and Amazon ECR without separate chart repository infrastructure.
This guide covers the OCI distribution specification mechanics, practical workflows for pushing and promoting artifacts across environments, Helm chart distribution via OCI, cryptographic signing with Cosign, and the failure modes you will encounter in production. By the end, you will be able to distribute any artifact type through OCI-compatible registries, sign it cryptographically, and set up monitoring to catch distribution failures before they reach production clusters.
When to Use / When Not to Use
When OCI artifacts make sense
OCI artifacts are the right choice when you want to distribute non-image artifacts through the same registry infrastructure you already use for container images. If your organization has standardized on a container registry (Harbor, ACR, GCR, ECR), OCI artifacts let you store Helm charts, SBOMs, policy bundles, and other artifacts in the same place with the same access controls and authentication.
Use OCI artifacts when you need to bundle multiple related artifacts together. ORAS lets you push a container image, its SBOM, and its signature as a single authenticated operation with one oras push command.
OCI distribution is also useful when you want vendor-neutral artifact distribution. The OCI spec is an open standard, so artifacts stored in any OCI-compatible registry are portable.
The most common concrete use case is Helm charts alongside container images. Your CI pipeline builds a microservice as a Docker image and packages its Kubernetes manifests as a Helm chart. With OCI artifacts, both live in the same ACR or Harbor instance, authenticated with the same service principal, and promoted through environments with the same tools. There is no separate Helm repository to host and no index.yaml to manage.
Beyond Helm, specific artifact types that map well to OCI distribution include SBOMs in SPDX or CycloneDX format, OPA and Gatekeeper policy bundles, language-specific packages (Wolfi, APK, Deb), and architecture diagrams stored as signed artifacts. ORAS handles all of these without custom tooling: oras push myregistry.azurecr.io/myteam/mypolicy:v1.0.0 ./policy.yaml:application/x-yaml works without any registry-side configuration beyond standard OCI support.
The bundling scenario deserves emphasis because it is where OCI artifacts genuinely outperform alternatives. A single oras push can attach the container image tarball, its SPDX SBOM, and a Cosign signature as separate layers in one authenticated operation. The alternative without OCI is three separate storage systems, three separate authentication flows, and three separate promotion pipelines. For regulated environments where you need to prove the SBOM and signature travel with the exact image bits, the OCI approach is auditable in a way that scattered artifact stores are not.
OCI distribution is also useful when you want vendor-neutral artifact distribution. The OCI spec is an open standard, so artifacts stored in any OCI-compatible registry are portable. In practice this means you can push a Helm chart to Harbor, then later pull it into GCR for a Google Cloud deployment, or into ECR for an AWS environment, without re-tagging or converting formats. The chart bytes are identical; only the registry hostname changes. This portability matters for multi-cloud or hybrid-cloud setups where you need to move artifacts between environments without rebuilding them.
When to stick with traditional approaches
OCI artifact support requires a compatible registry. Older self-hosted registries or registries that have not updated their OCI support cannot store arbitrary artifact types. In these cases, chart repositories (for Helm) or separate artifact stores work better.
If you only distribute container images and do not need to bundle additional artifacts, standard Docker image push and pull is simpler and more widely understood.
If your artifacts are large binary files that change frequently, the registry storage costs can add up. Some artifact types are better served by object storage with a CDN.
Registry compatibility is the first concrete constraint. Harbor 2.x supports OCI artifacts but requires the feature to be explicitly enabled at the project level; Harbor 1.x does not support them at all. ACR, GCR (now Artifact Registry), and ECR have native OCI artifact support with no additional configuration. Self-hosted registries based on Docker Distribution 2.7 or earlier lack OCI artifact support entirely. If your infrastructure includes a self-hosted ChartMuseum or an older Harbor instance, OCI push operations may succeed but pulls will return malformed content or HTTP 404.
The second constraint is tooling integration. Helm 3.8+ ships with native OCI support, but if your CI systems run older Helm versions you cannot use helm chart push without upgrading. Similarly, helm search repo has no equivalent in OCI registries because there is no index.yaml file to query. Teams that rely on helm search repo mychart to discover available versions need to either migrate to helm search repo against a traditional chart repository, or adopt a third-party tool like ORAS to list tags in an OCI registry. This is a genuine workflow gap that matters for teams with established Helm operational patterns.
Large binary artifacts that change frequently are better served by object storage with a CDN. Registry storage is priced at container image storage rates, which are higher than S3 or GCS blob storage for large files. More importantly, registries are not optimized for high-frequency overwrite patterns. If you push a 500 MB model file ten times per day as part of a training pipeline, you accumulate dangling blobs quickly, and registry garbage collection becomes a recurring operational burden. A workflow that uploads to S3, serves through CloudFront, and tracks the CDN URL in a config map is cheaper and simpler for this scenario. Use the registry for immutable versioned artifacts, not for mutable large binaries.
OCI Artifact Distribution Flow
flowchart TD
A[Developer] --> B[oras push<br/>or helm chart push]
B --> C{Registry Type}
C -->|OCI Compatible| D[Harbor / ACR / GCR / ECR]
C -->|Traditional| E[ChartMuseum<br/>HTTP API]
D --> F[Artifacts stored in<br/>OCI Registry]
E --> G[index.yaml +<br/>chart tarballs]
F --> H[Clusters pull via<br/>standard docker/helm pull]
D --> I[Cosign signs<br/>manifests]
I --> H
What is OCI?
OCI defines two specifications: the Image Specification (defines container image format) and the Distribution Specification (defines how images are distributed via APIs compatible with container registries).
The distribution specification works over HTTPS with the same registry APIs you use for Docker images. This means any OCI-compatible registry can store and serve any OCI artifact type, not just container images.
OCI artifact types include:
- Container images (the original use case)
- Helm charts
- Architecture diagrams
- Policy bundles
- SBOMs (Software Bills of Materials)
- Binary blobs
OCI Distribution Specification
The OCI distribution spec describes a REST API for distributing artifacts. Registries implementing this spec expose endpoints like:
GET /v2/<name>/blobs/<digest>
POST /v2/<name>/blobs/uploads/
PUT /v2/<name>/blobs/uploads/<uuid>?digest=<digest>
GET /v2/<name>/manifests/<ref>
PUT /v2/<name>/manifests/<ref>
DELETE /v2/<name>/manifests/<ref>
Each artifact is identified by:
- Name: Repository path (e.g.,
myregistry.azurecr.io/myteam/myartifact) - Digest: SHA256 hash of the manifest content
- Tag: Human-readable reference (e.g.,
v1.0.0,latest)
Pushing Artifacts to Registries
Modern tooling supports OCI artifact push and pull directly.
Using ORAS (OCI Registry As Storage):
# Install ORAS
brew install oras
# Login to your registry
oras login myregistry.azurecr.io
# Push a generic artifact (any file)
oras push myregistry.azurecr.io/myteam/config:v1.0.0 \
./config.json:application/json
# Push with custom media type
oras push myregistry.azurecr.io/myteam/mydata:v1.0.0 \
./data.tar.gz:application/x-tar+gzip
Push Helm chart as OCI artifact:
# Login
helm registry login myregistry.azurecr.io
# Push (Helm 3.8+ supports OCI natively)
helm chart push myregistry.azurecr.io/myteam/mychart:1.0.0
# Pull
helm chart pull myregistry.azurecr.io/myteam/mychart:1.0.0
helm chart export myregistry.azurecr.io/myteam/mychart:1.0.0 ./extracted/
Push multiple files as a single artifact:
# Create artifact manifest with annotations
oras push myregistry.azurecr.io/myteam/bundle:v1.0.0 \
./image.tar:image/docker \
./sbom.spdx:application/spdx+json \
./signature:text/plain
Helm Charts as OCI Artifacts
Helm charts work naturally as OCI artifacts since Helm 3.8. The chart tarball becomes the artifact content, and the registry stores and serves it.
Workflow example:
# Package chart locally
helm package ./mychart
# Login to OCI registry
helm registry login -u myuser myregistry.azurecr.io
# Push with full registry path
helm chart push myregistry.azurecr.io/mycorp/charts/mychart:1.0.0
# Or pull and inspect
helm chart pull myregistry.azurecr.io/mycorp/charts/mychart:1.0.0
helm show all myregistry.azurecr.io/mycorp/charts/mychart:1.0.0
# List tags in repository
oras repo tags myregistry.azurecr.io/mycorp/charts/mychart
CI/CD integration:
# GitHub Actions example
- name: Push Helm chart to ACR
run: |
helm chart push ${{ env.REGISTRY }}/${{ env.CHART_NAME }}:${{ github.sha }} \
--registry-config ${{ env.REGISTRY_CONFIG }} \
--username ${{ env.REGISTRY_USER }} \
--password ${{ env.REGISTRY_PASS }}
- name: Create release tag
run: |
helm chart push ${{ env.REGISTRY }}/${{ env.CHART_NAME }}:${{ env.VERSION }} \
--registry-config ${{ env.REGISTRY_CONFIG }}
Cosign for Artifact Signing
Cosign, part of the Sigstore project, signs OCI artifacts and stores signatures alongside them in registries.
Installation:
brew install cosign
# or
go install github.com/sigstore/cosign/cmd/cosign@latest
Sign an image:
# Sign without key (for testing)
cosign sign myregistry.azurecr.io/myteam/myimage:v1.0.0
# Sign with key (production)
cosign sign --key mykey.pem myregistry.azurecr.io/myteam/myimage:v1.0.0
Sign Helm charts:
# Push chart first
helm chart push myregistry.azurecr.io/myteam/mychart:1.0.0
# Sign the chart
cosign sign --key mykey.pem myregistry.azurecr.io/myteam/mychart:1.0.0
Verify signatures:
# Verify image
cosign verify --key mykey.pem myregistry.azurecr.io/myteam/myimage:v1.0.0
# Verify with attestation (e.g., SLSA provenance)
cosign verify-attestation --type slsaprovenance \
--key mykey.pem myregistry.azurecr.io/myteam/myimage:v1.0.0
GitHub Actions for signing:
- name: Sign image with Cosign
uses: sigstore/cosign-installer@v3
with:
cosign-version: "v2"
- name: Sign container image
run: |
cosign sign --yes --key-env-file GOOGLE_APPLICATION_CREDENTIALS \
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ env.DIGEST }}
- name: Verify signature
run: |
cosign verify --key https://pkg.go.dev/security/augurs/cosign \
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ env.DIGEST }}
Practical Workflow Examples
Multi-stage artifact promotion:
#!/bin/bash
# promote-artifact.sh
ARTIFACT="$1"
FROM_TAG="$2"
TO_TAG="$3"
REGISTRY="myregistry.azurecr.io"
# Pull from source environment
oras pull $REGISTRY/$ARTIFACT:$FROM_TAG -o /tmp/artifact
# Push to target environment
oras push $REGISTRY/$ARTIFACT:$TO_TAG /tmp/artifact
# Sign the promoted artifact
cosign sign --yes $REGISTRY/$ARTIFACT:$TO_TAG
Bundling SBOM with image:
# Push image
docker push myregistry.azurecr.io/myteam/myapp:v1.0.0
# Create and push SBOM as separate artifact
cosign attest --predicate-type spdxjson \
--certificate identity \
--oidc-issuer https://v.stateful.so \
myregistry.azurecr.io/myteam/myapp:sbom
# Or use ORAS for SBOM
oras push myregistry.azurecr.io/myteam/myapp:sbom \
./sbom.spdx:application/spdx+json
Verify before deployment:
# admission-controller policy
apiVersion: v1
kind: ConfigMap
metadata:
name: cosign-policy
namespace: gatekeeper-system
data:
policy: |
package sigstore
deny[msg] {
not cosign.verify(image)
msg := "Image signature verification failed"
}
Production Failure Scenarios
OCI artifacts can fail in ways that are specific to the distribution model.
Registry does not support OCI artifact types
If you push to a registry that has OCI artifact support disabled or using an older version, pushes may silently succeed but pulls fail or return incorrect content. Harbor requires OCI artifact support to be enabled at the project level.
Always test pull after push, especially with new registries or new artifact types.
The silent success / failed pull problem has specific manifestations per registry. Harbor versions before 2.8 did not enable OCI artifact support by default on new projects, and the push appears to succeed even when the artifact is stored in an incompatible format. The Harbor UI shows the artifact, oras manifest fetch returns the manifest, but helm chart pull returns 404 Not Found because Helm cannot interpret the internal artifact type. The workaround is to enable OCI artifact support in the Harbor project settings before pushing. In Harbor 2.8+, new projects have it enabled by default, but upgraded instances retain their original settings.
GCR (now Artifact Registry) and ECR have native OCI artifact support with no project-level toggles. ACR also has native support but enforces strict media type validation: if the manifest’s mediaType field does not match the declared content type, the push is rejected with HTTP 415. This is actually preferable to the Harbor behavior because the error surfaces immediately rather than silently. The verification checklist after any push is: helm chart pull immediately after helm chart push to confirm the artifact is retrievable, and oras manifest fetch to inspect the manifest structure if the pull fails.
For ORAS-pushed artifacts, the equivalent check is oras manifest fetch myregistry.azurecr.io/myteam/myartifact:v1.0.0 | jq to confirm the manifest was stored correctly. If the manifest fetch returns valid JSON with the expected layers and media types, the artifact is intact. If it returns empty or a 404, the push wrote to a path the registry cannot serve. This is distinct from the Harbor project-level toggle issue, which produces a successful push but an unreachable artifact. Registries that fully support OCI artifact distribution expose this failure immediately rather than hiding it.
Media type mismatches
ORAS and Helm use specific media types for artifact content. If you push with an incorrect media type, clients may not recognize the artifact when pulling. Use standard media types (application/vnd.oci.image.manifest.v1+json, application/helm.chart.v1.tar+gzip) unless you have a specific reason to deviate.
What a media type mismatch looks like in practice:
A registry that rejects an unrecognized media type returns HTTP 415 (Unsupported Media Type). ORAS clients surface this as Error: failed to resolve the manifest from the registry: 415 Unsupported Media Type. Helm says Error: unexpected 415 response.
Registries vary in how strict they are. GCR takes most OCI media types but blocks custom vendor types from third-party tools. ECR is more forgiving but still rejects malformed or non-standard types. ACR sits in the middle: it handles standard OCI types fine but will reject pushes where the manifest’s mediaType field does not match what was declared.
The usual culprit is tooling that predates the OCI spec. Helm 3.7 and earlier pushed chart tarballs with application/tar+gzip instead of the now-standard application/helm.chart.v1.tar+gzip. This means Helm 3.8+ clients reject charts pushed by older versions, and the reverse is also true. Match your Helm client versions across the entire pipeline.
Detecting and resolving media type issues:
# Inspect the manifest media type of an artifact
oras manifest fetch myregistry.azurecr.io/myteam/myapp:v1.0.0 | jq '.mediaType'
# Check what media type was used during push
oras manifest fetch myregistry.azurecr.io/myteam/mychart:1.0.0 | jq '.config.mediaType'
# Push with explicit media type override (rarely needed)
oras push myregistry.azurecr.io/myteam/mychart:1.0.0 \
./mychart-1.0.0.tgz:application/helm.chart.v1.tar+gzip
Inheriting a registry full of mixed media types happens more often than it should. Re-pushing each artifact with the correct type works, but do it in a maintenance window with no active deployments touching those tags.
Digest mismatch after promotion
When promoting artifacts between registries (dev to staging to prod), always pull by digest, not by tag. Tags can be overwritten, causing the same tag to point to different content in different registries. Using digests ensures bit-for-bit identity across environments.
# Wrong: pull by tag (may change)
oras pull myregistry.azurecr.io/myteam/myapp:v1.0.0
# Right: pull by digest (immutable)
oras pull myregistry.azurecr.io/myteam/myapp@sha256:abc123...
Cosign signature verification failures
Cosign signatures can fail verification if the image was modified after signing (e.g., re-tagged to a different registry without re-signing). If you sign in one environment and promote to another, the signature stays valid but the target registry must be in the Cosign verification policy.
Three things cause most verification failures. Cross-registry re-tagging without re-signing tops the list: you sign dev.azurecr.io/myteam/myapp:v1.0.0, someone copies it to prod.azurecr.io/myteam/myapp:v1.0.0 with oras copy, and the signature never makes the trip. Clock skew comes second: if the signing host’s clock runs fast or slow, Fulcio certificates look expired to the verifier. Misconfigured verification policy comes third: the admission controller does not have the right public key or OIDC configuration for the target registry.
Run the verification command to see what went wrong:
# Check if any signature exists for the image
cosign verify --insecure-ignore-tlog=true myregistry.azurecr.io/myteam/myapp:v1.0.0
# Verify with explicit key (shows exact error if key is wrong)
cosign verify --key mykey.pem myregistry.azurecr.io/myteam/myapp:v1.0.0
# For keyless signing, verify against Fulcio root CA
cosign verify --oidc-issuer https://v.stateful.so myregistry.azurecr.io/myteam/myapp:v1.0.0
The Error: verification failed: no matching signatures message means either the signature does not exist in the registry or the wrong key is being used. Error: x509: certificate has expired or is not yet valid points to clock skew. Error: could not get ca cert means the admission controller cannot reach the trust anchor.
Prevention beats debugging in production every time. Re-sign right after cross-registry promotion instead of copying signatures separately. If you run multiple registries, use a centralized signing service that covers all endpoints at once. Store the public key or OIDC config in a Kubernetes secret that the admission controller reads at startup, not at pod scheduling time, so key rotation does not force pod restarts.
OCI vs Alternatives
| Aspect | OCI Artifacts | Chart Repositories | Object Storage + CDN |
|---|---|---|---|
| Storage | Container registry | HTTP server with index.yaml | S3 / GCS / Blob |
| Access control | Integrated with registry auth | Separate per-repo auth | Separate IAM |
| Artifact bundling | Multiple artifacts in one push | Single artifact per package | Manual bundling |
| Helm support | Native (Helm 3.8+) | Native | Requires download + install |
| Cosign integration | Native | Not supported | Not supported |
| Cross-registry promotion | Pull by digest | Pull by URL | Copy via CLI |
| Cost | Registry storage rates | Cheap static hosting | Pay per GB |
OCI artifacts win when you already use a container registry and want unified artifact management. Chart repositories remain practical for pure Helm distributions without artifact bundling.
Observability Hooks
Track artifact distribution health with these monitoring practices.
Key metrics:
# Artifact push success rate
sum(rate(oras_push_success_total[5m])) by (repository, artifact_type)
# Artifact pull latency (p95)
histogram_quantile(0.95,
sum(rate(oras_pull_duration_seconds_bucket[5m])) by (le, repository)
)
# Signature verification failures
sum(rate(cosign_verify_failure_total[5m])) by (repository, error_code)
# Registry storage used by artifact type
sum(registry_artifact_storage_bytes) by (artifact_type, repository)
Alert rules:
# Alert if artifact pushes are failing
- alert: OCIArtifactPushFailures
expr: sum(rate(oras_push_failure_total[5m])) > 0.01
labels:
severity: critical
annotations:
summary: "OCI artifact push failures detected"
description: "{{ $value }} push failures per second. Check registry connectivity and authentication."
# Alert if Cosign verification failures spike
- alert: CosignVerificationFailures
expr: sum(rate(cosign_verify_failure_total[5m])) / sum(rate(cosign_verify_total[5m])) > 0.05
labels:
severity: critical
annotations:
summary: "Cosign verification failure rate above 5%"
description: "Image signature verification is failing for {{ $labels.repository }}. Unverified images may be running in production."
# Alert if artifact storage is running low
- alert: RegistryStorageHigh
expr: registry_storage_used_bytes / registry_storage_limit_bytes > 0.85
labels:
severity: warning
annotations:
summary: "Registry storage above 85%"
description: "Artifact storage is running low. Plan cleanup or expansion."
Debugging commands:
# Check ORAS login status
oras login localhost:5000 -u admin -p admin
# Inspect artifact manifest
oras manifest fetch myregistry.azurecr.io/myteam/myapp:v1.0.0
# Verify artifact signature
cosign verify --key mykey.pem myregistry.azurecr.io/myteam/myapp:v1.0.0
# List all artifacts in a repository
oras repo tags myregistry.azurecr.io/myteam
# Check Helm chart in OCI registry
helm chart pull myregistry.azurecr.io/myteam/mychart:v1.0.0
helm chart export myregistry.azurecr.io/myteam/mychart:v1.0.0 /tmp/chart
# Inspect SBOM attached to image
cosign attest --type spdxjson \
--predicate /tmp/sbom.json \
myregistry.azurecr.io/myteam/myapp:v1.0.0
Common Pitfalls / Anti-Patterns
Pulling by mutable tag instead of immutable digest
Pushing myapp:latest and then pulling myapp:latest later may give you a different image. Tags can be overwritten. Always pull by digest for reproducible builds and deployments.
Consider a CI pipeline that builds and pushes myapp:build-123 and myapp:latest on every commit to main. Between your helm install --set image=myapp:latest and the actual pod scheduling on the cluster, another pipeline run may have overwritten the latest tag with a newer build. The pod that eventually starts runs code you never tested with that commit.
Digest-based pulling eliminates this race entirely:
# Capture the digest at push time
DIGEST=$(oras manifest fetch --platform linux/amd64 \
myregistry.azurecr.io/myteam/myapp:v1.0.0 \
-o json | jq -r '.digest')
# Use the digest in all downstream pulls
helm install mychart oci://myregistry.azurecr.io/myteam/mychart \
--version $DIGEST
Store the digest as an environment variable or in a config file checked into source control alongside your deployment manifests. GitOps tools like Flux and Argo CD support digest-based image references natively, so the manifest stays immutable once you commit it.
Forgetting to re-sign after cross-registry promotion
If you sign an image in your CI registry and then copy it to production with oras copy, the signature does not follow. You must sign after promoting, or use a signing service that covers both registries.
This catches teams that set up a solid signing workflow in CI but then break the chain at the promotion step. You sign myregistry.azurecr.io/myteam/myapp:v1.0.0 in the dev registry, everything passes, then someone uses a script to copy the image to prodregistry.azurecr.io/myteam/myapp:v1.0.0 without re-signing. Now your admission controller rejects the production deployment because it cannot find a valid signature for the artifact in the prod registry, even though the image bytes are identical.
The fix is to treat signing as a promotion step rather than a CI-only step:
# Pull by digest from source (signature is a separate artifact)
oras pull myregistry.azurecr.io/myteam/myapp@sha256:abc123 -o /tmp/artifact
oras push prodregistry.azurecr.io/myteam/myapp@sha256:abc123 /tmp/artifact
# Re-sign in target environment immediately after promotion
cosign sign --yes --key prod-key.pem prodregistry.azurecr.io/myteam/myapp@sha256:abc123
If you manage multiple registries through a central promotion pipeline, consider using a dedicated signing service (like Sigstore’s Rekor-backed keyless signing) that issues signatures covering all registry endpoints simultaneously.
Not enabling OCI artifact support in Harbor projects
Harbor has OCI artifact support disabled by default in new projects. You must explicitly enable it in the Harbor project settings before pushing Helm charts as OCI artifacts.
What happens when OCI artifact support is disabled:
The push will appear to succeed. The Harbor UI will even show the artifact in the repository. But helm chart pull against it returns Error: failed to fetch chart: 404 Not Found. Helm cannot make sense of the artifact because it was stored as a different internal type.
In Harbor 2.x, you toggle this at the project level:
- Log in to Harbor as an administrator
- Go to Projects, pick your project
- Click Policy, then Artifact: under OCI Artifact, flip it to Enabled
- For Helm charts specifically, also check Helm Chart: Helm Chart Repository must be Enabled
You can also do this over the API:
# Get project ID
curl -u admin:password https://harbor.mycorp.io/api/v2.0/projects/myproject
# Enable OCI artifact support
curl -u admin:password -X PUT \
https://harbor.mycorp.io/api/v2.0/projects/myproject \
-H "Content-Type: application/json" \
-d '{"oci_version": "v1", "artifact_annotation": true}'
Harbor 2.8+ creates new projects with OCI artifact support enabled by default. Upgraded instances keep whatever settings they had, so older projects often have it disabled. Check all projects during registry onboarding, not just the ones you create fresh.
One more thing: if Harbor sits behind a proxy in an air-gapped setup, make sure the proxy passes OCI artifact manifest requests through unchanged. Some proxies rewrite the Content-Type header, which turns apparently successful pushes into artifacts that cannot be pulled.
Using non-standard media types without client configuration
Some registries and clients only recognize standard OCI media types. Custom media types may work locally but fail in production environments with stricter validation. Stick to standard types unless you control both the push and pull tooling.
The problem with deviating from standard media types:
The OCI spec defines a set of canonical media types. Push an artifact with something non-standard and it will be technically valid inside the spec, but every client and registry along the pull path may reject or mishandle it. This shows up most in mixed tooling environments where ORAS, Helm, and Docker all hit the same registry.
Here is a scenario that played out at a company I heard about: a team pushed a policy bundle with ORAS using application/mycompany.policy.v1+json. Their internal ORAS client could pull it back fine. Then a CI pipeline running a different ORAS version tried to fetch the same artifact and got blocked, because that version’s media type allowlist did not include the custom type. The artifact sat valid in the registry and completely unreachable from CI.
Standard media types to use:
| Artifact Type | Standard Media Type |
|---|---|
| OCI Image Manifest | application/vnd.oci.image.manifest.v1+json |
| OCI Config Blob | application/vnd.oci.image.config.v1+json |
| Helm Chart Tarball | application/helm.chart.v1.tar+gzip |
| Generic Tarball | application/x-tar+gzip |
| SPDX SBOM JSON | application/spdx+json |
| Generic JSON | application/json |
| Plain Text | text/plain |
When you genuinely need a custom media type:
Vendor-prefixed types like application/vnd.mycompany.artifact.v1+json are structurally valid and acceptable if your tooling requires them. The tradeoff is that every client pulling that artifact must also be configured to recognize the custom type. This only works when you own both ends of the pipeline. The moment you distribute to third parties or across team boundaries where you do not control the tooling, stick with the standard types.
Not cleaning up dangling manifests
When you push a manifest and then push again with the same tag, the old manifest becomes a dangling reference. Some registries do not automatically garbage collect these, leading to storage bloat. Monitor dangling manifest counts and run registry garbage collection periodically.
Dangling manifests pile up through normal CI/CD. Every pipeline push to myapp:v1.0.0 leaves the previous manifest untagged but still stored as a blob. Harbor, ACR, and GCR keep blobs and manifests in separate storage, so removing a tag does not remove the layers underneath. Months of frequent releases without GC means a registry full of unreachable blobs you are still paying for.
Finding dangling manifests depends on your registry. Harbor shows them in the UI under the artifact list with an “untagged” filter, or you can hit the Harbor API:
# List all untagged artifacts in a Harbor project
curl -u admin:password https://harbor.mycorp.io/api/v2.0/projects/myproject/artifacts \
?tags= \ 2>/dev/null | jq '.[] | select(.tags == null) | .digest'
# Count dangling blobs in GCR/Artifact Registry (using gcloud)
gcloud artifacts docker images list myproject/myrepo \
--filter="tags=MISSING" --format="value(digest)" | wc -l
GC is also registry-specific, and most registries block writes while it runs to avoid race conditions. Harbor runs GC via a cron job in the Harbor core pod, or you can trigger it manually through the UI or API. ACR has no user-facing GC command; it runs cleanup automatically on a schedule and when storage quotas get tight. GCR and Artifact Registry offload cleanup to Cloud Storage lifecycle policies, so the blob storage backend handles it independently of the registry layer.
The practical approach is prevention plus occasional cleanup. Use immutable tags based on digests or semantic versions you never overwrite. If you must retag, do it in a maintenance window and run registry GC right after. Watch registry_storage_used_bytes so you catch bloat before the bill does. Some teams schedule GC quarterly or after major version bumps.
Interview Questions
Expected answer points:
- The OCI Image Specification defines the format and structure of a container image (manifest, config, filesystem layers)
- The OCI Distribution Specification defines the HTTP API for distributing artifacts through compatible registries
- They are independent specifications: images are the content, distribution is the transport
- The distribution spec works over HTTPS using REST endpoints compatible with Docker Registry v2
Expected answer points:
- ORAS (OCI Registry As Storage) treats an OCI registry as a general-purpose artifact store
- It allows pushing any file type as an artifact with custom media types
- Supports artifact bundling: push multiple related files (image, SBOM, signature) in one operation
- Uses the OCI distribution spec like a key-value store, leveraging existing auth and storage infrastructure
Expected answer points:
- Tags are mutable: `myapp:latest` can point to different content at different times
- Digests are content-addressable: `myapp@sha256:abc123...` always returns the same bits
- Pulling by digest ensures bit-for-bit reproducibility across environments
- Critical for reproducible builds, audit trails, and security verification
- Tags are useful for promotion pipelines but should resolve to digests for production pulls
Expected answer points:
- Helm 3.8 introduced `helm chart push` and `helm chart pull` commands for OCI registries
- The chart tarball (`.tgz`) is pushed as an OCI artifact with media type `application/helm.chart.v1.tar+gzip`
- Authentication uses `helm registry login` which configures Docker credential store
- The chart manifest is stored as an OCI image manifest in the registry
- No local `index.yaml` needed; the registry acts as the chart index
Expected answer points:
- Cosign is part of the Sigstore project, used for cryptographic signing of OCI artifacts
- It generates a separate signature artifact stored in the same registry alongside the image
- Signature is associated with the image digest, not the tag
- Supports key-based signing (private key) and keyless signing (OIDC identity via Sigstore)
- Can sign Helm charts, container images, and any OCI artifact type
- Signature attestation supports predicates like SLSA provenance and SPDX SBOMs
Expected answer points:
- Keyless signing uses OIDC identities (GitHub Actions, Google, Microsoft) to derive signing keys
- The private key is short-lived and managed by Fulcio, Sigstore's certificate authority
- Fulcio issues a short-lived certificate bound to the identity and the artifact digest
- Rekor (the transparency log) records the signature entry for auditability
- Supports GitHub Actions OIDC, Google OIDC, and other OIDC providers
- No long-lived private keys to manage, reducing key rotation burden
Expected answer points:
- The standard media type is `application/helm.chart.v1.tar+gzip` for chart tarballs
- ORAS uses `application/vnd.oci.image.manifest.v1+json` for the artifact manifest
- Incorrect media types cause pull failures: clients do not recognize the artifact format
- Some registries enforce strict media type validation
- Custom media types work in controlled environments but break in heterogeneous setups
Expected answer points:
- Pull by digest from the source registry: `oras pull dev.azurecr.io/myapp@sha256:abc123`
- Push to the target registry by digest: `oras push prod.azurecr.io/myapp:sha256:abc123`
- Sign the artifact again in the target environment: `cosign sign --key prod-key.pem prod.azurecr.io/myapp@sha256:abc123`
- Signatures are not automatically transferred across registries; they must be re-issued
- Using a centralized signing service that covers both registries is an alternative approach
Expected answer points:
- OCI artifact support is disabled by default in new Harbor projects and must be explicitly enabled
- Authentication failures if `helm registry login` was not run against the correct registry URL
- Media type mismatches if the chart was packaged with older Helm or a custom tool
- Network timeouts during large chart pushes over high-latency connections
- Harbor project quota exhaustion preventing new artifact pushes
- Pulling a chart that was pushed but with an unsupported artifact type returns HTTP 415
Expected answer points:
- `docker push` only supports container image media types defined in the Docker specification
- `oras push` supports arbitrary media types, making it suitable for non-image artifacts
- ORAS can push multiple files as separate layers in a single artifact manifest
- ORAS works without Docker daemon; it talks directly to the registry HTTP API
- ORAS can also push images, acting as a lightweight alternative to the Docker client
Expected answer points:
- With ORAS: `oras push myregistry.azurecr.io/myapp:v1.0.0 ./image.tar:image/docker ./sbom.spdx:application/spdx+json`
- ORAS pushes the SBOM as a separate artifact layer associated with the image digest
- With Cosign attestations: `cosign attest --predicate-type spdxjson --certificate identity myregistry.azurecr.io/myapp:v1.0.0`
- Cosign creates a separate attestation artifact linked to the image digest
- Verification requires pulling both artifacts and checking the attestation predicate
Expected answer points:
- `oras_push_success_total` and `oras_push_failure_total` by repository and artifact type
- `oras_pull_duration_seconds` histogram to detect latency spikes
- `cosign_verify_failure_total` to track signature verification failures
- `registry_storage_used_bytes` by artifact type to monitor storage consumption
- `cosign_verify_success_total` as a ratio to total verification attempts
- Alert thresholds: push failure rate > 1%, verification failure rate > 5%, storage > 85%
Expected answer points:
- A Cosign signature is a cryptographic proof that the artifact content has not changed since signing
- A Cosign attestation is a signed JSON document (predicate) about the artifact, such as SLSA provenance or an SBOM
- Attestations use `cosign attest`; signatures use `cosign sign`
- Use attestations when you need to attach metadata: build provenance, scan results, policy compliance
- Use signatures when you just need integrity verification: "this artifact has not been tampered with"
- Both are stored as separate artifacts linked to the image digest via Rekor entries
Expected answer points:
- A dangling manifest occurs when a new manifest is pushed with the same tag as an existing manifest
- The old manifest is no longer reachable by tag but still exists as ablob in storage
- The registry does not automatically delete it; it remains until garbage collection runs
- Over time with frequent re-tags, this accumulates unused blobs consuming storage
- Registry garbage collection must be run periodically to reclaim space
- Some registries (like Harbor) do not enable auto-gc by default
Expected answer points:
- `GET /v2/
/blobs/ ` — download a layer/blob by its SHA256 digest - `POST /v2/
/blobs/uploads/` — initiate a resumable blob upload session - `PUT /v2/
/blobs/uploads/ ?digest= ` — complete a blob upload - `GET /v2/
/manifests/` — fetch a manifest by tag or digest - `PUT /v2/
/manifests/` — push a manifest (creates or updates a tag) - `DELETE /v2/
/blobs/ ` — delete a blob (not always supported) - The registry tracks blobs separately from manifests; manifests reference blobs by digest
Expected answer points:
- SLSA (Supply Chain Levels for Software Artifacts) defines security levels for software supply chains
- Cosign can generate and verify SLSA provenance attestations using `cosign attest --type slsaprovenance`
- SLSA provenance describes how an artifact was built: source repo, build system, materials, entry point
- Policy engines like Gatekeeper and OPA can enforce SLSA level requirements before deployment
- Rekor stores provenance attestations for auditability and traceability
Expected answer points:
- Verify login: `oras login localhost:5000 -u admin -p admin` — check credentials are stored
- Check registry accessibility: `curl -I https://myregistry.azurecr.io/v2/`
- Inspect the manifest: `oras manifest fetch myregistry.azurecr.io/myteam/myapp:v1.0.0`
- Check network: `oras push` may fail due to MTU issues, proxy configuration, or TLS certificate problems
- Verify artifact does not already exist with different content at that tag: use digest-based pulls
- Use `--debug` flag for verbose output from ORAS
Expected answer points:
- Repositories follow a DNS-like path structure: `myregistry.azurecr.io/myteam/myartifact`
- The registry hostname is the first path component; subsequent segments are namespace tiers
- Artifact name combines namespace + artifact name: `mycorp/charts/mychart`
- Tags are mutable references within a repository: `v1.0.0`, `latest`, `sha256-abcd...`
- Digests are the canonical immutable identifier: `sha256:
` of the manifest content - Name-to-digest resolution is registry-internal; clients cache the mapping
Expected answer points:
- OCI artifacts use existing registry infrastructure; chart repos require separate HTTP hosting with `index.yaml`
- OCI artifacts support Cosign signing; chart repos have no native signing story
- OCI artifacts allow bundling multiple charts or SBOMs in a single push operation
- Chart repos are simpler to serve from static hosting (S3 + CloudFront) without registry auth complexity
- OCI artifact support requires Helm 3.8+; chart repos work with all Helm 3 versions
- OCI artifacts win in Kubernetes-native environments where registry auth is already configured
Expected answer points:
- Use a Kubernetes admission controller: OPA Gatekeeper, Kyverno, or Open Policy Agent
- Configure the admission controller to query Cosign verification results before allowing pod creation
- Example Gatekeeper policy: deny pods where `cosign.verify(image)` returns false
- Requires the cluster to have the Cosign verification public key configured in the admission controller
- For keyless verification, the admission controller must trust the Sigstorefulcio certificate authority
- Monitor `cosign_verify_failure_total` to detect attempts to deploy unsigned images
- Combine with image scanning (Trivy) to enforce both signature verification and vulnerability thresholds
Further Reading
- OCI Distribution Specification — The official OCI distribution specification repository with full REST API reference
- ORAS Project — Documentation for OCI Registry As Storage, including CLI reference and artifact manifest format
- Helm OCI Support — Official Helm documentation on using OCI registries for chart distribution
- Sigstore Cosign — Complete reference for Cosign signing, verification, and attestation workflows
- Harbor OCI Artifact Support — Harbor-specific configuration for OCI artifact projects
- SLSA Supply Chain Security — Framework for securing software supply chains, with guidance on provenance attestation
Conclusion
Key Takeaways
- OCI artifacts let you store any artifact type in standard container registries alongside your images
- ORAS provides flexible artifact bundling; Helm 3.8+ has native OCI support for charts
- Cosign adds cryptographic signatures that travel with the artifact through promotion pipelines
- Always pull by digest, not by tag, for reproducible artifact distribution
- Monitor push/pull success rates, verification failures, and storage usage
Artifact Distribution Checklist
# Push Helm chart as OCI artifact
helm registry login myregistry.azurecr.io
helm chart push myregistry.azurecr.io/myteam/mychart:1.0.0
# Sign the pushed artifact
cosign sign --key mykey.pem myregistry.azurecr.io/myteam/mychart:1.0.0
# Pull and verify
helm chart pull myregistry.azurecr.io/myteam/mychart:1.0.0
helm chart export myregistry.azurecr.io/myteam/mychart:1.0.0 /tmp/chart
cosign verify --key mykey.pem myregistry.azurecr.io/myteam/mychart:1.0.0
# Bundle SBOM with ORAS
oras push myregistry.azurecr.io/myteam/myapp:v1.0.0 \
./image.tar:image/docker \
./sbom.spdx:application/spdx+json
Promoting Artifacts Across Environments
# Pull by digest from source
oras pull myregistry.azurecr.io/myteam/myapp@sha256:abc123 -o /tmp/artifact
# Push to target registry (signature NOT included)
oras push target.azurecr.io/prod/myapp:v1.0.0 /tmp/artifact
# Re-sign in target environment
cosign sign --key prod-key.pem target.azurecr.io/prod/myapp:v1.0.0 Category
Related Posts
Container Images: Building, Optimizing, and Distributing
Learn how Docker container images work, layer caching strategies, image optimization techniques, and how to publish your own images to container registries.
Container Registry: Image Storage, Scanning, and Distribution
Set up and secure container registries for storing, scanning, and distributing container images across your CI/CD pipeline and clusters.
Developing Helm Charts: Templates, Values, and Testing
Create production-ready Helm charts with Go templates, custom value schemas, and testing using Helm unittest and ct.