Object Storage: S3, Blob Storage, and the Scale of Unstructured Data

Object storage changes how you think about files. There are no folders, no hierarchy, no filesystem. Just buckets filled with objects identified by keys. This simplicity scales to exabytes and beyond.

I used to think of object storage as a place to dump files that did not fit anywhere else. That was wrong. Object storage is a first-class storage platform with capabilities that file systems cannot match at scale.

The Object Storage Model

Object storage organizes data as objects in buckets. Each object has a key, data, and metadata. The key is a string, like a file path but without directory semantics. The data is arbitrary bytes. Metadata is key-value pairs describing the object.

import boto3

s3 = boto3.client('s3')

# Upload an object
s3.put_object(
    Bucket='my-bucket',
    Key='images/product/photo123.jpg',
    Body=image_data,
    ContentType='image/jpeg',
    Metadata={'product-id': '12345', 'uploaded-by': 'jane'}
)

# Retrieve an object
response = s3.get_object(
    Bucket='my-bucket',
    Key='images/product/photo123.jpg'
)
image_data = response['Body'].read()

The key looks like a path but the storage is flat. There are no directories, though tools often display keys as if they had folders. The slash in the key is just another character.

S3 and Compatible Services

Amazon S3 created the object storage API pattern. Other services implement S3-compatible APIs. MinIO runs on-premises with S3 compatibility. Cloud providers offer blob storage with similar semantics: Google Cloud Storage, Azure Blob Storage.

S3 has become the de facto standard for object storage. Learning S3 concepts transfers to other services. The API semantics differ slightly but the mental model is the same.

MinIO is popular for Kubernetes environments where you want object storage without cloud provider dependencies. It speaks the S3 protocol and works with existing S3 tools.

Buckets and Keys

Buckets are containers for objects. They have global names across regions. You choose the bucket name, and it must be unique across all S3 users worldwide.

# Create a bucket
s3.create_bucket(Bucket='my-unique-bucket-name')

# List buckets
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(bucket['Name'])

Keys identify objects within a bucket. The combination of bucket name and key uniquely identifies every object. No two objects in the same bucket can have the same key.

Keys can contain slashes, which tools display as folder hierarchies. But the storage treats them as flat. You can list objects with a prefix filter to simulate directory browsing.

Object Metadata and Tags

Object metadata travels with the object. System metadata includes things like content type, size, and last-modified time. User metadata is custom key-value pairs you define.

# Retrieve metadata without downloading
head = s3.head_object(
    Bucket='my-bucket',
    Key='images/product/photo123.jpg'
)
print(head['ContentType'])  # image/jpeg
print(head['ContentLength'])  # 245632
print(head['Metadata'])  # {'product-id': '12345', 'uploaded-by': 'jane'}

Tags are separate from metadata. They are key-value pairs used for filtering and billing. You can have up to 10 tags per object. Tags are indexed separately for efficient querying.

# Add tags to an existing object
s3.put_object_tagging(
    Bucket='my-bucket',
    Key='images/product/photo123.jpg',
    Tagging={
        'TagSet': [
            {'Key': 'department', 'Value': 'marketing'},
            {'Key': 'project', 'Value': 'spring-campaign'}
        ]
    }
)

Versioning

S3 versioning keeps multiple versions of an object. When you overwrite an object, the previous version is preserved. You can retrieve any historical version.

# Enable versioning on a bucket
s3.put_bucket_versioning(
    Bucket='my-bucket',
    VersioningConfiguration={'Status': 'Enabled'}
)

# List object versions
versions = s3.list_object_versions(Bucket='my-bucket')
for version in versions['Versions']:
    print(f"{version['Key']} - {version['VersionId']}")

Versioning costs more storage since every version is kept. But it protects against accidental overwrites and deletions. You can retrieve any previous state.

Versioning also enables point-in-time recovery. Combined with lifecycle policies, you can keep historical versions for compliance without manual intervention.

Storage Classes and Lifecycle Policies

S3 offers multiple storage classes with different cost and availability characteristics.

Standard is the default, most expensive tier. Infrequent Access stores data cheaper but charges more for access. Glacier is for archiving, with retrieval times of minutes to hours. Intelligent Tiering moves data automatically based on access patterns.

from datetime import datetime

# Define lifecycle rule to transition objects to IA after 30 days
s3.put_bucket_lifecycle_configuration(
    Bucket='my-bucket',
    LifecycleConfiguration={
        'Rules': [
            {
                'ID': 'MoveToIA',
                'Filter': {'Prefix': 'logs/'},
                'Status': 'Enabled',
                'Transitions': [
                    {'Days': 30, 'StorageClass': 'STANDARD_IA'},
                    {'Days': 90, 'StorageClass': 'GLACIER'}
                ]
            }
        ]
    }
)

Lifecycle policies automate data movement between tiers. Old logs might move to Infrequent Access after 30 days, then to Glacier after 90 days. You set it and forget it.

flowchart LR
    Upload[("Object<br/>Uploaded")] --> Standard[("S3 Standard<br/>Frequently accessed")]

    Standard -->|30 days<br/>no access| IA[("S3 Standard-IA<br/>Infrequent access")]

    IA -->|90 days<br/>no access| Glacier[("S3 Glacier<br/>Archived")]

    Glacier -->|180 days<br/>no access| DeepArchive[("S3 Deep Archive<br/>Long-term archive")]

    DeepArchive -->|365 days<br/>no access| Delete[("Lifecycle<br/>Expiration")]

    Upload -.->|versioning<br/>enabled| Versions[("Version<br/>Preserved")]

    Versions -.->|MFA delete<br/>required| Restore[("Restore<br/>before delete")]

Without versioning, overwrites and deletes are permanent. With versioning, previous versions persist until you explicitly delete them or the lifecycle policy purges old versions.

Storage classes across AWS S3:

Class	Durability	Availability	Access Latency	Min Storage Duration	Best For
Standard	11 nines	99.99%	Milliseconds	None	Frequently accessed data
Standard-IA	11 nines	99.9%	Milliseconds	30 days	Infrequently accessed data
Intelligent Tiering	11 nines	99.9%	Milliseconds	None	Unknown or changing access patterns
One Zone-IA	99.5%	99.5%	Milliseconds	30 days	Re-creatable non-critical data in one AZ
Glacier Instant Retrieval	11 nines	99.9%	Milliseconds	90 days	Rarely accessed but needs instant retrieval
Glacier Flexible Retrieval	11 nines	99.99%	1-12 hours	90 days	Long-term archives with occasional access
Glacier Deep Archive	11 nines	99.99%	12-48 hours	180 days	Longest-term retention, regulatory compliance

Most teams keep everything in Standard. They should not. Standard-IA is about half the price, and Glacier Deep Archive is roughly 95% cheaper. The tradeoff is retrieval cost and latency — match the class to how you actually access the data.

Durability and Availability

Object storage is designed for massive scale and high durability. S3 Standard offers 11 nines of durability. That means losing an object is extraordinarily unlikely.

The durability comes from storing multiple copies across availability zones. S3 automatically replicates your data. You do not need to configure RAID or backup software.

Availability guarantees differ from durability. S3 Standard guarantees 99.99% availability. That is about 52 minutes of downtime per year. Different storage classes offer different availability SLAs.

# Check if an object exists
try:
    s3.head_object(Bucket='my-bucket', Key='important-file.pdf')
    print("Object exists")
except ClientError as e:
    error_code = e.response['Error']['Code']
    if error_code == '404':
        print("Object not found")

Using Object Storage in Applications

Object storage replaces both file storage and some database use cases. Static assets like images and videos live in object storage. User uploads go there. Backup files. Data lakes.

The pattern is straightforward. Generate a unique key, upload with metadata, store the key in your database if needed. The object storage handles the rest.

import uuid

def handle_upload(file_data, content_type):
    # Generate unique key
    key = f"uploads/{uuid.uuid4()}/{file_data.filename}"

    # Upload to S3
    s3.put_object(
        Bucket='my-bucket',
        Key=key,
        Body=file_data.read(),
        ContentType=content_type
    )

    # Store key in database, not the file itself
    db.files.insert({'s3_key': key, 'original_name': file_data.filename})

    return key

CDN integration is common. CloudFront, Cloudflare, and others cache objects from S3. Users download from edge locations rather than origin servers. This reduces latency and origin load.

When to Use and When Not to Use Object Storage

When to Use Object Storage:

• Storing unstructured files: images, videos, documents, audio files
• Backing up databases, logs, or system files
• Hosting static website assets (HTML, CSS, JS, images)
• Building data lakes for analytics workloads
• Distributing large files via CDN integration
• Storing user uploads that exceed database BLOB limits
• Archiving data for compliance with infrequent access patterns

When Not to Use Object Storage:

• You need filesystem semantics (directories, symlinks, permissions)
• Your workload requires sub-millisecond latency (use block storage or memory)
• You need to modify parts of files frequently (object storage is write-all-or-nothing)
• Your application requires strong consistency for concurrent modifications
• You need database-like queries across object metadata (use a database index)
• Real-time file system operations are required (use NFS, EBS, or similar)

Production Failure Scenarios

Failure	Impact	Mitigation
Accidental deletion	Permanent data loss without versioning	Enable versioning, implement soft-delete policies, use MFA delete
Bucket policy misconfiguration	Public exposure or access denial	Review policies with least privilege, use policy validation tools
Cost overrun from unconstrained uploads	Unexpectedly high storage costs	Set budget alerts, implement upload size limits, lifecycle policies
Cross-region replication delay	Stale data in DR region	Set realistic RPO, test replication lag, use synchronous replication if needed
Throttling from request rate limits	Upload/download failures under load	Implement retry with exponential backoff, request rate smoothing
Object key namespace collision	Data overwrites	Use UUIDs or timestamp-based keys, implement key validation
Bucket name conflicts	Deployment failures	Use globally unique naming conventions, automate naming
CDN cache invalidation delays	Stale content served after updates	Plan invalidation strategy, use versioned object keys

Capacity Estimation: Cost Calculation for S3 Tiers

Object storage pricing varies significantly by storage class. Understanding tier costs enables cost-effective lifecycle management.

Monthly storage cost formula:

storage_cost_per_month = storage_gb × price_per_gb_per_month
total_storage_cost = sum(storage_tier_gb × tier_price_per_gb_per_month)

For a media platform with the following storage distribution:

Tier	Storage	Price/GB/mo	Monthly Cost
S3 Standard	10TB	$0.023	$230
S3 IA	50TB	$0.0125	$625
S3 Glacier	200TB	$0.004	$800
S3 Glacier Deep Archive	500TB	$0.00099	$495
Total	760TB		$2,150/mo

Request costs matter as much as storage:

request_cost_per_month = get_requests × price_per_1k_gets + put_requests × price_per_1k_puts

S3 Standard pricing: $0.023/GB storage, $0.0004 per 1,000 GET requests, $0.005 per 1,000 PUT requests. For a high-traffic application with 100M GET/day and 1M PUT/day: storage might be $500/mo but requests add $40 + $5 = $45/mo. For a low-traffic archive with 1M GET/day and 10K PUT/day: storage $500/mo, requests $0.40 + $0.05 = $0.45/mo. Storage dominates for cold data; requests dominate for hot data.

Early deletion fees: S3 Glacier and Glacier Deep Archive charge early deletion fees if data is deleted within 90 or 180 days respectively. Plan lifecycle transitions accordingly — do not move data to Glacier if you might need to delete it within 90 days.

Real-World Case Study: S3 Outage Impact Analysis

On February 28, 2017, S3 in the US-East-1 region experienced a significant outage lasting approximately 4 hours. The cause was a routine command intended to remove a small number of servers from one of the S3 subsystems. Due to a typo, a larger set of servers was removed than intended. These servers supported two S3 subsystems: the billing service and the index subsystem.

The billing service failure prevented new S3 buckets from being created or deleted. The index subsystem failure meant S3 could not serve GET, LIST, or DELETE requests for a large number of objects.

The impact: thousands of companies using S3 as their primary storage backend experienced application failures. Canva, Slack, Quartz, and many other companies reported outages. The Software-as-a-service ecosystem was broadly affected because many SaaS providers had centralized their storage in US-East-1.

The lesson for architecture: centralized storage in a single region is a single point of failure. The mitigation: multi-region replication for critical data. The 2017 outage demonstrated that even a 4-hour S3 disruption could take down a significant portion of the internet-facing SaaS ecosystem. Organizations that had replicated to us-east-2 or eu-west-1 recovered quickly. Those relying solely on us-east-1 were down for the full 4 hours.

The follow-up: AWS improved S3’s resilience by reducing the blast radius of similar future failures — they made the billing subsystem independent of the core S3 subsystem, so a billing issue would not take down object operations.

---

Observability Checklist

Metrics to Monitor:

• Request count by type (GET, PUT, DELETE)
• Request latency (time to first byte, total request time)
• HTTP error rates by status code (4xx, 5xx)
• Storage utilization per bucket and total
• Network bandwidth consumption
• Upload/download throughput
• Replication lag (if using cross-region replication)
• Lifecycle transition counts and storage class distribution

Logs to Capture:

• All API calls via server access logging
• Data access patterns for compliance
• Authentication failures and access denied events
• Lifecycle transition events
• Replication status and failures
• Cross-region transfer usage

Alerts to Set:

• Error rate spike above baseline
• Storage growth exceeding forecast
• Request throttling events
• Replication lag exceeding RPO
• Cost approaching budget threshold
• Unusual access patterns (security concern)
• Failed operations exceeding threshold

# S3: Enable and query access logs
import boto3

s3 = boto3.client('s3')
# Enable logging
s3.put_bucket_logging(
    Bucket='my-bucket',
    BucketLoggingStatus={
        'LoggingEnabled': {
            'TargetBucket': 'my-logs-bucket',
            'TargetPrefix': 'access-logs/'
        }
    }
)

# CloudWatch metrics for monitoring
# s3.namespace = 'AWS/S3'
# Key metrics: BucketSizeBytes, NumberOfObjects, AllRequests, GetRequests, PutRequests

Security Checklist

• Enable server-side encryption (SSE-S3 or SSE-KMS)
• Use IAM policies with least privilege (not bucket policies for everything)
• Block public access at bucket and account level
• Enable versioning with MFA delete for critical data
• Implement access logging and regular audit reviews
• Use pre-signed URLs or signed cookies for temporary access (not long-term keys)
• Enable bucket policies to enforce encryption in transit
• Use VPC endpoints for internal access (no internet route)
• Regularly rotate access keys and review IAM roles
• Implement bucket lifecycle policies to auto-delete old versions
• Test bucket policies with policy simulation before deployment
• Use resource-based policies alongside IAM for defense in depth

Common Pitfalls and Anti-Patterns

1. Using object storage as a database: Object storage has no query language. Storing millions of objects with no index makes retrieval extremely slow. Use a database for structured data with query needs.

2. Not planning for request rate limits: S3 has per-bucket and per-prefix limits. Popular content from a single prefix throttles. Spread across prefixes and use CloudFront.

3. Storing too many small objects: Each object has overhead. Millions of tiny objects waste money and slow listings. Consider archiving small files together.

4. Ignoring storage class optimization: Storing everything in Standard costs more than necessary. Use lifecycle policies to move rarely-accessed data to cheaper tiers.

5. Not using versioning for mutable objects: Overwriting without versioning destroys the previous version. Enable versioning for any object that changes.

6. Assuming strong consistency immediately: New objects and updates take time to propagate. Do not assume immediate consistency for read-after-write.

7. Leaking pre-signed URLs: Pre-signed URLs grant access to anyone with the URL. Do not log or expose them. Use short expiration times.

8. Not implementing cleanup policies: Upload failures, test runs, and temporary files accumulate. Implement lifecycle rules to auto-delete incomplete multipart uploads and old temp files.

Quick Recap

Key Bullets:

• Object storage scales massively but offers no query language or filesystem semantics
• S3 API is the de facto standard; MinIO, GCS, and Azure Blob support compatible APIs
• Use pre-signed URLs for temporary access, not long-term credentials
• Lifecycle policies automate storage tier transitions and cleanup
• Versioning protects against accidental overwrites and deletions
• CDN integration (CloudFront, Cloudflare) dramatically reduces latency

Copy/Paste Checklist:

import boto3
import uuid

s3 = boto3.client('s3')

# Generate unique key for uploads
def upload_file(bucket, file_data, content_type):
    key = f"uploads/{uuid.uuid4()}/{file_data.filename}"
    s3.put_object(
        Bucket=bucket,
        Key=key,
        Body=file_data.read(),
        ContentType=content_type,
        ServerSideEncryption='AES256'
    )
    return key

# Enable lifecycle policy via CLI
# aws s3api put-bucket-lifecycle-configuration \
#   --bucket my-bucket \
#   --lifecycle-configuration file://lifecycle.json

# Lifecycle policy JSON:
# {"Rules": [{"ID": "MoveToGlacier", "Status": "Enabled",
#   "Transitions": [{"Days": 30, "StorageClass": "GLACIER"}]}]}

Conclusion

Object storage scales in ways traditional file systems cannot. The flat namespace with bucket and key naming handles exabytes of data. Built-in versioning, lifecycle policies, and storage classes manage data automatically.

S3 set the standard that others follow. Learning its patterns transfers to MinIO, Google Cloud Storage, and Azure Blob. The API is consistent across providers.

For most applications, object storage is the right choice for unstructured files. Images, videos, documents, backups. Keep the metadata in your database and the files in object storage.

For related reading, see Database Scaling to learn about scaling database storage, and NoSQL Databases to understand other data storage patterns.