Key-Value Stores: Redis and DynamoDB Patterns

The key-value store is the simplest kind of database. You have a key, you get a value. No queries, no joins, no complex schema. This simplicity is the feature.

Redis and DynamoDB represent two ends of the key-value spectrum. Redis is an in-memory store with optional persistence. DynamoDB is a fully managed, persistent, distributed key-value store with configurable consistency. Understanding both helps you choose the right tool.

---

The Key-Value Model

At its core, the model is:

# Basic operations
store.set(key, value)
value = store.get(key)
store.delete(key)
exists = store.exists(key)

No WHERE clauses. No aggregations. You know exactly where your data is.

This simplicity enables two things: extremely fast operations and horizontal scalability. When every lookup goes directly to a specific location, there is no query planner overhead, no index traversal, no join computation.

flowchart LR
    Client["Client"]
    Redis["Redis Cluster<br/>(3 Nodes)"]
    A["Node A<br/>hash_tag: {user}"]
    B["Node B<br/>hash_tag: {order}"]
    C["Node C<br/>hash_tag: {product}"]
    Client --> Redis
    Redis --> A & B & C

Redis Cluster shards data by hash slot (16384 slots total). The hash tag in curly braces determines which slot a key maps to — {user}:123 and {user}:456 go to the same node. DynamoDB works similarly under the hood: partition keys determine which storage node holds your data.

---

Redis: In-Memory Speed

Redis stores data primarily in memory, with optional durability to disk. This makes it fast for read-heavy workloads.

Basic Operations

import redis

r = redis.Redis(host='localhost', port=6379, db=0)

# String operations
r.set('user:123:session', json.dumps(session_data))
r.set('rate:limit:192.168.1.1', 100, ex=60)  # With 60-second TTL
value = r.get('user:123:session')

# Multiple operations (pipelines reduce round trips)
pipe = r.pipeline()
pipe.hset('user:123', mapping={'name': 'Alice', 'email': 'alice@example.com'})
pipe.expire('user:123', 3600)
pipe.execute()

# Atomic counters
r.incr('api:request:count')
r.incrby('user:123:balance', 100)

Data Structures Beyond Strings

Redis is actually a data structure server. Values can be more than strings.

# Lists (queues, activity feeds)
r.lpush('queue:jobs', 'job1', 'job2', 'job3')
next_job = r.rpop('queue:jobs')

# Sets (unique items, tags)
r.sadd('user:123:likes', 'item1', 'item2', 'item3')
is_member = r.sismember('user:123:likes', 'item1')
all_items = r.smembers('user:123:likes')

# Sorted sets (leaderboards, priorities)
r.zadd('leaderboard', {'alice': 100, 'bob': 200, 'charlie': 150})
top_players = r.zrevrange('leaderboard', 0, 9, withscores=True)
rank = r.zrank('leaderboard', 'alice')

# Hashes (objects)
r.hset('product:SKU-001', mapping={
    'name': 'Gaming Laptop',
    'price': 1299.99,
    'stock': 50
})
product = r.hgetall('product:SKU-001')

TTL and Expiration

Redis handles time-limited data well.

# Session with 30-minute expiry
r.setex('session:abc123', 1800, json.dumps(session_data))

# Rate limiting: allow 100 requests per minute
def rate_limit(identifier, limit=100, window=60):
    key = f'ratelimit:{identifier}'
    current = r.get(key)

    if current and int(current) >= limit:
        return False  # Rate limit exceeded

    pipe = r.pipeline()
    pipe.incr(key)
    pipe.expire(key, window)
    pipe.execute()
    return True

# Distributed locks
def acquire_lock(lock_name, timeout=10):
    lock_key = f'lock:{lock_name}'
    acquired = r.set(lock_key, '1', nx=True, ex=timeout)
    return acquired

def release_lock(lock_name):
    lock_key = f'lock:{lock_name}'
    r.delete(lock_key)

---

DynamoDB: Managed Persistent Storage

DynamoDB is a fully managed NoSQL database by AWS. It offers persistent storage with automatic sharding, eventual or strong consistency options, and pay-per-request pricing.

Table Structure

DynamoDB has a simple primary key: either a partition key alone, or a partition key plus sort key.

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('UserSessions')

# Put and get items
table.put_item(Item={
    'userId': 'user-123',
    'sessionId': 'session-abc',
    'data': {'preferences': {'theme': 'dark'}},
    'expiresAt': int(time.time()) + 3600
})

response = table.get_item(
    Key={
        'userId': 'user-123',
        'sessionId': 'session-abc'
    }
)

Access Patterns Drive Key Design

In DynamoDB, you design keys based on how you query. Unlike relational databases where you can freely query any column, DynamoDB queries are limited to key attributes.

# Single table design: multiple entity types in one table
# Key: PK (partition key), SK (sort key)

# User entity
{'PK': 'USER#alice', 'SK': 'PROFILE', 'name': 'Alice', 'email': 'alice@example.com'}

# Orders for user
{'PK': 'USER#alice', 'SK': 'ORDER#2024-01-01', 'total': 129.99, 'items': [...]}
{'PK': 'USER#alice', 'SK': 'ORDER#2024-01-15', 'total': 49.99, 'items': [...]}

# Products
{'PK': 'PRODUCT#LAPTOP-001', 'SK': 'METADATA', 'name': 'Gaming Laptop', 'price': 1299.99}

# Query all orders for a user
response = table.query(
    KeyConditionExpression=Key('PK').eq('USER#alice') & Key('SK').begins_with('ORDER#')
)

# Query single user profile
response = table.get_item(
    Key={'PK': 'USER#alice', 'SK': 'PROFILE'}
)

Global Secondary Indexes

When you need alternative access patterns, use GSIs.

# Create table with GSI for email lookup
table = dynamodb.create_table(
    TableName='Users',
    KeySchema=[
        {'AttributeName': 'userId', 'KeyType': 'HASH'}
    ],
    AttributeDefinitions=[
        {'AttributeName': 'userId', 'AttributeType': 'S'},
        {'AttributeName': 'email', 'AttributeType': 'S'}
    ],
    GlobalSecondaryIndexes=[
        {
            'IndexName': 'EmailIndex',
            'KeySchema': [{'AttributeName': 'email', 'KeyType': 'HASH'}],
            'Projection': {'ProjectionType': 'ALL'}
        }
    ],
    BillingMode='PAY_PER_REQUEST'
)

# Later, query by email
response = table.query(
    IndexName='EmailIndex',
    KeyConditionExpression=Key('email').eq('alice@example.com')
)

---

Use Cases: Where Key-Value Stores Excel

Caching

The classic use case. Store frequently accessed data in Redis to reduce database load.

def get_user_profile(user_id):
    cache_key = f'user:profile:{user_id}'

    # Try cache first
    cached = redis.get(cache_key)
    if cached:
        return json.loads(cached)

    # Cache miss - fetch from database
    profile = database.fetch_user(user_id)

    # Store in cache for next time
    redis.setex(cache_key, 300, json.dumps(profile))  # 5 minute TTL

    return profile

Session Storage

Sessions are naturally key-value: session ID maps to session data.

# Web session
session_id = cookies.get('session_id')
if not session_id:
    session_id = generate_session_id()
    redis.setex(f'session:{session_id}', 86400, json.dumps({}))

session_data = json.loads(redis.get(f'session:{session_id}'))
session_data['page_views'] = session_data.get('page_views', 0) + 1
redis.setex(f'session:{session_id}', 86400, json.dumps(session_data))

Leaderboards

Sorted sets make ranked lists straightforward.

def update_leaderboard(player_id, score):
    redis.zadd('leaderboard', {player_id: score})

def get_top_players(n=10):
    return redis.zrevrange('leaderboard', 0, n-1, withscores=True)

def get_player_rank(player_id):
    rank = redis.zrank('leaderboard', player_id)
    return rank + 1 if rank is not None else None

Rate Limiting

Atomic operations enable distributed rate limiting.

def is_allowed(client_id, limit=100, window=60):
    key = f'ratelimit:{client_id}'

    # Lua script for atomic check-and-increment
    script = """
    local current = redis.call('GET', KEYS[1])
    if current and tonumber(current) >= tonumber(ARGV[1]) then
        return 0
    end
    current = redis.call('INCR', KEYS[1])
    if tonumber(current) == 1 then
        redis.call('EXPIRE', KEYS[1], ARGV[2])
    end
    return 1
    """

    result = redis.eval(script, 1, key, limit, window)
    return bool(result)

Distributed Locks

Redis implements coordination primitives.

import uuid
import time

class RedisLock:
    def __init__(self, redis_client, lock_name, timeout=10):
        self.redis = redis_client
        self.lock_key = f'lock:{lock_name}'
        self.timeout = timeout
        self.token = str(uuid.uuid4())

    def acquire(self, blocking=True, blocking_timeout=10):
        start = time.time()
        while True:
            if self.redis.set(self.lock_key, self.token, nx=True, ex=self.timeout):
                return True
            if not blocking:
                return False
            if time.time() - start >= blocking_timeout:
                return False
            time.sleep(0.01)

    def release(self):
        # Only release if we own the lock
        script = """
        if redis.call('GET', KEYS[1]) == ARGV[1] then
            return redis.call('DEL', KEYS[1])
        else
            return 0
        end
        """
        self.redis.eval(script, 1, self.lock_key, self.token)

---

Eviction Policies

Redis offers multiple eviction policies when memory limits are reached.

# maxmemory set to 2gb
# maxmemory-policy defined how eviction works

# noeviction: reject writes, reads allowed (default)
# allkeys-lru: evict least recently used keys across all keys
# allkeys-random: evict random keys
# volatile-lru: evict LRU keys only in keys with TTL
# volatile-random: evict random keys with TTL
# volatile-ttl: evict keys with shortest TTL
# allkeys-lfu: evict least frequently used keys
# volatile-lfu: evict LFU keys with TTL

For caching, allkeys-lru is typically the best choice. For session storage with TTL, volatile-lru ensures only expiring keys get evicted.

---

In-Memory vs Persistent: Tradeoffs

Characteristic	Redis (In-Memory)	DynamoDB (Persistent)
Latency	Sub-millisecond	Single-digit milliseconds
Durability	Optional (RDB/AOF)	Always durable
Scalability	Vertical plus read replicas	Automatic sharding
Cost	Pay for memory	Pay for throughput
Complexity	Single instance relatively simple	Requires understanding capacity
Data size	Limited by memory	Virtually unlimited

Redis with persistence is not equivalent to DynamoDB. Redis persistence (RDB snapshots, AOF logs) guards against crashes but not against disk failures. DynamoDB’s architecture distributes data across multiple AZs by default.

For true durability in Redis, you need replication to follower instances that can take over if the primary fails.

---

Common Production Failures

Redis OOM kills crashing the database: Redis runs out of memory and the OS kernel kills it via OOM killer. Your application starts returning errors and recovery takes time to reload data. Set maxmemory and maxmemory-policy, configure vm.overcommit_memory=1 at the OS level, and have a reload plan from your primary database ready.

Keys without TTL filling up Redis: Someone writes SET mykey value instead of SET mykey value EX 3600 in the rate limiter. The key never expires. Redis fills up, allkeys-lru starts evicting keys you actually need. Default to TTL on every key — a missing expiration is a bug, not a feature.

DynamoDB hot partition throttling: Your session table uses userId as the partition key. Turns out user-0000 is the test account that every load test hits. That partition maxes out at 300 RCU while the rest of the table sits idle. Spread write volume across partition keys, and implement exponential backoff with jitter when you hit ProvisionedThroughputExceededException.

AOF always mode killing Redis write throughput: You enable AOF with appendfsync always for “maximum durability”. Under write load, Redis blocks on every sync to disk and throughput collapses. Switch to appendfsync everysec or appendfsync no, and match your disk I/O to the write rate if you use everysec.

Distributed lock released by wrong owner: The lock code calls DEL on expiry. But the lock expired while your work was still running, another client grabbed it, and now two processes hold the same lock. The token-based release (compare-and-delete via Lua script) shown in the code example prevents this — only the lock owner can release it.

DynamoDB single-table design gone wrong: You pack 15 entity types into one table to avoid “wasting” tables. The GSI situation becomes a mess and access patterns start colliding. Single-table design shines with 3-5 entity types that share access patterns. Beyond that, the operational complexity wins.

---

Capacity Estimation: Memory-per-Key and DynamoDB Partition Math

Redis memory is the primary constraint. Each key-value pair has overhead on top of the value size. A string key with a 100-byte value uses roughly 100 bytes plus 40-60 bytes of key metadata (keyspace overhead, pointer to value, expiry metadata if set). A hash with 10 fields uses more — each field name is stored alongside the value.

Rough formula for Redis memory: total_keys * (avg_key_size + avg_value_size + 40 bytes_overhead). For a cache with 1 million keys averaging 500 bytes each: roughly 540 MB plus Redis’s own internal fragmentation. Set maxmemory with headroom — at 80% used, Redis starts evicting and you lose predictability.

For TTLs, the expiration queue has a cost: Redis scans the keyspace periodically to expire keys. With millions of keys with TTLs, ACTIVE_EXPIRE_CYCLE_SLOW runs every 100ms. This is generally fast but if you set TTLs on millions of keys at once (a batch import, for example), Redis can briefly block. The latency-sensitive work — expiring the keys you care about — happens incrementally.

DynamoDB partition sizing: DynamoDB creates partitions based on partition key values and throughput. Each partition supports up to 1,000 RCU, 1,000 WCU, and 10 GB of data. For a table with 50 GB and 5,000 RCU, you need at minimum 5 partitions (10 GB each), and the RCU requirement means you actually need 5 partitions just for throughput even if the data were smaller.

If you exceed 10 GB per partition, DynamoDB splits automatically. If you exceed 1,000 WCU on a single partition before splitting, you get throttled. DynamoDB adaptive capacity can temporarily burst above 1,000 WCU per partition, but sustained traffic above that requires either better key distribution or requesting a provisioning increase from AWS support.

Observability Hooks: Redis INFO and DynamoDB CloudWatch

For Redis, INFO is the primary observability interface. The output has sections: Server (Redis build info, version, uptime), Clients (connected clients, blocked clients), Memory (used_memory, used_memory_peak, mem_fragmentation_ratio), Persistence (RDB/AOF状态), Stats (total commands processed, keyspace hits/misses), Replication (role, master link status), CPU, Latency stats.

The metrics to watch in production: mem_fragmentation_ratio above 1.5 means Redis is using 50% more memory than it needs — restart Redis or adjust activedefrag settings. keyspace_hits / (keyspace_hits + keyspace_misses) is your cache hit ratio. Below 80% and your cache is not earning its memory. instantaneous_ops_per_sec shows current throughput. connected_clients approaching maxclients (default 10,000) is a connection exhaustion warning.

For DynamoDB, CloudWatch metrics matter: ConsumedReadCapacityUnits and ConsumedWriteCapacityUnits against ProvisionedThroughput. If you consistently consume above 80% of provisioned, DynamoDB autoscales or you need to increase. ThrottledRequests is the key alarm — non-zero throttling means your application is failing requests. UserErrors with ProvisionedThroughputExceededException means your keys are unevenly distributed. ReplicationLatency matters for global tables — cross-region replication lag should stay under 1 second.

Real-World Case Study: Twitter’s Redis Timeline Storage

Twitter’s early architecture used Redis to store user timelines — the list of tweets a user sees when they open their home timeline. The problem was timeline size: celebrities with millions of followers had timelines that could not be computed on read (pulling from all followed users’ tweets in real time was too slow).

Their solution was write-time fanout: when a user posts a tweet, Twitter pushes that tweet into the timelines of all their followers via a Redis pipeline. Reading a timeline became a simple Redis range read — fast, predictable. The tradeoff was write amplification: one tweet from a celebrity required writes to millions of Redis instances.

The operational consequence: the Redis cluster storing timelines was among the largest in production. Memory pressure was constant. When a Redis node failed, millions of users noticed missing timeline entries until repair ran. Twitter’s fix was a hybrid model — push timelines for active users, pull from graph for inactive ones. The lesson: Redis makes reads cheap at the cost of write amplification and memory pressure. Design for the read-to-write ratio of your actual workload.

Interview Questions

Q: Your Redis instance is using 8 GB of memory out of 10 GB max. You restart the process and memory drops to 5 GB. What happened?

Redis does not immediately return freed memory to the OS — used_memory reflects the allocator’s view, not the OS’s. After a restart, the allocator gets a fresh heap and fragmentation memory is recovered. The 3 GB drop was likely memory fragmentation (the allocator holding freed chunks that have not been coalesced) and dirty pages waiting to write to AOF or RDB snapshots. Check mem_fragmentation_ratio — if it was above 1.5, fragmentation was eating your memory. The fix is either restarting Redis periodically or enabling activedefrag (Redis 4.0+).

Q: You are using DynamoDB with a composite primary key (userId, timestamp). Your application is reading a user’s data range within a time window but latency has spiked. What do you check?

First check CloudWatch ThrottledRequests and ConsumedReadCapacityUnits. If the table is in provisioned mode and the partition key userId has hot spots (one user receiving more reads than others), that partition maxes out at 1,000 RCU and throttles. Check PartitionKey distribution in CloudWatch — if one key is consuming disproportionate capacity, switch to a sparse attribute as the partition key or add a random suffix to userId to spread the load. If the table is on-demand, check whether DynamoDB adaptive capacity is catching up — it splits hot partitions but takes a few minutes under sustained load.

Q: When would you choose Redis over DynamoDB for storing session data?

Redis when session data is ephemeral enough that losing it is acceptable (users can re-authenticate) and when sub-millisecond latency matters. Redis sessions work well in single-region deployments where recovery from a Redis failure is fast (users re-login quickly). DynamoDB when sessions must survive regional failures (cross-region replication with global tables), when you need auditability (DynamoDB Streams captures session changes), or when your session data has complex querying needs (you need to query sessions by arbitrary attributes). For most web applications, the deciding factor is whether a session lookup failure should require re-authentication or not.

Security Checklist

• Use TLS for all Redis client connections to prevent credential interception on the wire
• Set bind to localhost or an internal interface only; never expose Redis to the internet
• Enable Redis AUTH or ACLs to require passwords; use named ACL users with least-privilege command permissions in production
• For DynamoDB: apply IAM policies with least-privilege access; use bucket policies and VPC endpoints to restrict access to DynamoDB within your VPC
• Encrypt DynamoDB tables at rest using AWS-managed or customer-managed KMS keys; for Redis, enable Protectedredis mode and use encrypt: true in transit
• Implement key expiration policies (TTL) to automatically purge sensitive session data rather than relying on manual deletion

Common Pitfalls and Anti-Patterns

Using a single key for everything: Storing serialized objects (e.g., JSON blobs) under one key prevents efficient partial updates and makes TTL management impossible. Fix: use hierarchical key naming (user:{id}:session:{session_id}) for granular control.

Ignoring memory eviction under memory pressure: Redis maxmemory eviction policies can silently drop data. noeviction will reject writes instead. Fix: choose allkeys-lru or volatile-lru based on whether all keys have TTL, and monitor evicted_keys metric.

Storing unbounded growth data in Redis: Using Redis lists or sorted sets without size limits leads to unbounded memory growth. Fix: enforce maximum list/set sizes using LTRIM or ZREMRANGEBYRANK on every write, or switch to a bounded data store for bulk data.

DynamoDB hot partitions: A partition key with low cardinality (e.g., a fixed attribute like region = "US") concentrates all read/write throughput on a single partition. Fix: add a high-cardinality suffix to partition keys or use a composite partition key.

Forgetting TTL for ephemeral data: Session data and caches that do not expire consume memory indefinitely. Fix: always set TTL on ephemeral data; use Redis EXPIRE or DynamoDB’s TTL attribute.

Quick Recap Checklist

• Redis suits sub-millisecond reads, caching, and ephemeral data; DynamoDB suits managed scalable persistent storage
• Design partition keys with high cardinality to avoid hot partitions on DynamoDB
• Use TTL on all ephemeral data to prevent unbounded memory growth
• Monitor eviction rates and memory fragmentation; set maxmemory-policy appropriate to your access patterns
• Secure Redis with TLS, ACLs, and bind restrictions; secure DynamoDB with IAM policies and VPC endpoints

Conclusion

Key-value stores trade query flexibility for raw speed and simplicity. Redis works well as a cache, session store, and real-time data structure server where sub-millisecond latency matters. DynamoDB works well as a managed, scalable, durable store for application data with predictable access patterns.

The key insight is that key-value stores require knowing your access patterns upfront. You cannot query on arbitrary fields. Design your keys around how you actually read data, not around the structure of the data itself.

For more on NoSQL databases, see the NoSQL overview. To learn about caching strategies, see Caching Strategies. For comparison with Redis and Memcached, see Redis vs Memcached.