Key-Value Stores: Redis and DynamoDB Patterns

The key-value store is the simplest kind of database. You have a key, you get a value. No queries, no joins, no complex schema. This simplicity is the feature.

Redis and DynamoDB represent two ends of the key-value spectrum. Redis is an in-memory store with optional persistence. DynamoDB is a fully managed, persistent, distributed key-value store with configurable consistency. Understanding both helps you choose the right tool.

---

The Key-Value Model

At its core, the model is:

# Basic operations
store.set(key, value)
value = store.get(key)
store.delete(key)
exists = store.exists(key)

No WHERE clauses. No aggregations. You know exactly where your data is.

This simplicity enables two things: extremely fast operations and horizontal scalability. When every lookup goes directly to a specific location, there is no query planner overhead, no index traversal, no join computation.

flowchart LR
    Client["Client"]
    Redis["Redis Cluster<br/>(3 Nodes)"]
    A["Node A<br/>hash_tag: {user}"]
    B["Node B<br/>hash_tag: {order}"]
    C["Node C<br/>hash_tag: {product}"]
    Client --> Redis
    Redis --> A & B & C

Redis Cluster shards data by hash slot (16384 slots total). The hash tag in curly braces determines which slot a key maps to — {user}:123 and {user}:456 go to the same node. DynamoDB works similarly under the hood: partition keys determine which storage node holds your data.

---

Redis: In-Memory Speed

Redis stores data primarily in memory, with optional durability to disk. This makes it fast for read-heavy workloads.

Basic Operations

import redis

r = redis.Redis(host='localhost', port=6379, db=0)

# String operations
r.set('user:123:session', json.dumps(session_data))
r.set('rate:limit:192.168.1.1', 100, ex=60)  # With 60-second TTL
value = r.get('user:123:session')

# Multiple operations (pipelines reduce round trips)
pipe = r.pipeline()
pipe.hset('user:123', mapping={'name': 'Alice', 'email': 'alice@example.com'})
pipe.expire('user:123', 3600)
pipe.execute()

# Atomic counters
r.incr('api:request:count')
r.incrby('user:123:balance', 100)

Data Structures Beyond Strings

Redis is actually a data structure server. Values can be more than strings.

# Lists (queues, activity feeds)
r.lpush('queue:jobs', 'job1', 'job2', 'job3')
next_job = r.rpop('queue:jobs')

# Sets (unique items, tags)
r.sadd('user:123:likes', 'item1', 'item2', 'item3')
is_member = r.sismember('user:123:likes', 'item1')
all_items = r.smembers('user:123:likes')

# Sorted sets (leaderboards, priorities)
r.zadd('leaderboard', {'alice': 100, 'bob': 200, 'charlie': 150})
top_players = r.zrevrange('leaderboard', 0, 9, withscores=True)
rank = r.zrank('leaderboard', 'alice')

# Hashes (objects)
r.hset('product:SKU-001', mapping={
    'name': 'Gaming Laptop',
    'price': 1299.99,
    'stock': 50
})
product = r.hgetall('product:SKU-001')

TTL and Expiration

Redis handles time-limited data well.

# Session with 30-minute expiry
r.setex('session:abc123', 1800, json.dumps(session_data))

# Rate limiting: allow 100 requests per minute
def rate_limit(identifier, limit=100, window=60):
    key = f'ratelimit:{identifier}'
    current = r.get(key)

    if current and int(current) >= limit:
        return False  # Rate limit exceeded

    pipe = r.pipeline()
    pipe.incr(key)
    pipe.expire(key, window)
    pipe.execute()
    return True

# Distributed locks
def acquire_lock(lock_name, timeout=10):
    lock_key = f'lock:{lock_name}'
    acquired = r.set(lock_key, '1', nx=True, ex=timeout)
    return acquired

def release_lock(lock_name):
    lock_key = f'lock:{lock_name}'
    r.delete(lock_key)

---

DynamoDB: Managed Persistent Storage

DynamoDB is a fully managed NoSQL database by AWS. It offers persistent storage with automatic sharding, eventual or strong consistency options, and pay-per-request pricing.

Table Structure

DynamoDB has a simple primary key: either a partition key alone, or a partition key plus sort key.

import boto3

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('UserSessions')

# Put and get items
table.put_item(Item={
    'userId': 'user-123',
    'sessionId': 'session-abc',
    'data': {'preferences': {'theme': 'dark'}},
    'expiresAt': int(time.time()) + 3600
})

response = table.get_item(
    Key={
        'userId': 'user-123',
        'sessionId': 'session-abc'
    }
)

Access Patterns Drive Key Design

In DynamoDB, you design keys based on how you query. Unlike relational databases where you can freely query any column, DynamoDB queries are limited to key attributes.

# Single table design: multiple entity types in one table
# Key: PK (partition key), SK (sort key)

# User entity
{'PK': 'USER#alice', 'SK': 'PROFILE', 'name': 'Alice', 'email': 'alice@example.com'}

# Orders for user
{'PK': 'USER#alice', 'SK': 'ORDER#2024-01-01', 'total': 129.99, 'items': [...]}
{'PK': 'USER#alice', 'SK': 'ORDER#2024-01-15', 'total': 49.99, 'items': [...]}

# Products
{'PK': 'PRODUCT#LAPTOP-001', 'SK': 'METADATA', 'name': 'Gaming Laptop', 'price': 1299.99}

# Query all orders for a user
response = table.query(
    KeyConditionExpression=Key('PK').eq('USER#alice') & Key('SK').begins_with('ORDER#')
)

# Query single user profile
response = table.get_item(
    Key={'PK': 'USER#alice', 'SK': 'PROFILE'}
)

Global Secondary Indexes

When you need alternative access patterns, use GSIs.

# Create table with GSI for email lookup
table = dynamodb.create_table(
    TableName='Users',
    KeySchema=[
        {'AttributeName': 'userId', 'KeyType': 'HASH'}
    ],
    AttributeDefinitions=[
        {'AttributeName': 'userId', 'AttributeType': 'S'},
        {'AttributeName': 'email', 'AttributeType': 'S'}
    ],
    GlobalSecondaryIndexes=[
        {
            'IndexName': 'EmailIndex',
            'KeySchema': [{'AttributeName': 'email', 'KeyType': 'HASH'}],
            'Projection': {'ProjectionType': 'ALL'}
        }
    ],
    BillingMode='PAY_PER_REQUEST'
)

# Later, query by email
response = table.query(
    IndexName='EmailIndex',
    KeyConditionExpression=Key('email').eq('alice@example.com')
)

---

Use Cases: Where Key-Value Stores Excel

Caching

The classic use case. Store frequently accessed data in Redis to reduce database load.

def get_user_profile(user_id):
    cache_key = f'user:profile:{user_id}'

    # Try cache first
    cached = redis.get(cache_key)
    if cached:
        return json.loads(cached)

    # Cache miss - fetch from database
    profile = database.fetch_user(user_id)

    # Store in cache for next time
    redis.setex(cache_key, 300, json.dumps(profile))  # 5 minute TTL

    return profile

Session Storage

Sessions are naturally key-value: session ID maps to session data.

# Web session
session_id = cookies.get('session_id')
if not session_id:
    session_id = generate_session_id()
    redis.setex(f'session:{session_id}', 86400, json.dumps({}))

session_data = json.loads(redis.get(f'session:{session_id}'))
session_data['page_views'] = session_data.get('page_views', 0) + 1
redis.setex(f'session:{session_id}', 86400, json.dumps(session_data))

Leaderboards

Sorted sets make ranked lists straightforward.

def update_leaderboard(player_id, score):
    redis.zadd('leaderboard', {player_id: score})

def get_top_players(n=10):
    return redis.zrevrange('leaderboard', 0, n-1, withscores=True)

def get_player_rank(player_id):
    rank = redis.zrank('leaderboard', player_id)
    return rank + 1 if rank is not None else None

Rate Limiting

Atomic operations enable distributed rate limiting.

def is_allowed(client_id, limit=100, window=60):
    key = f'ratelimit:{client_id}'

    # Lua script for atomic check-and-increment
    script = """
    local current = redis.call('GET', KEYS[1])
    if current and tonumber(current) >= tonumber(ARGV[1]) then
        return 0
    end
    current = redis.call('INCR', KEYS[1])
    if tonumber(current) == 1 then
        redis.call('EXPIRE', KEYS[1], ARGV[2])
    end
    return 1
    """

    result = redis.eval(script, 1, key, limit, window)
    return bool(result)

Distributed Locks

Redis implements coordination primitives.

import uuid
import time

class RedisLock:
    def __init__(self, redis_client, lock_name, timeout=10):
        self.redis = redis_client
        self.lock_key = f'lock:{lock_name}'
        self.timeout = timeout
        self.token = str(uuid.uuid4())

    def acquire(self, blocking=True, blocking_timeout=10):
        start = time.time()
        while True:
            if self.redis.set(self.lock_key, self.token, nx=True, ex=self.timeout):
                return True
            if not blocking:
                return False
            if time.time() - start >= blocking_timeout:
                return False
            time.sleep(0.01)

    def release(self):
        # Only release if we own the lock
        script = """
        if redis.call('GET', KEYS[1]) == ARGV[1] then
            return redis.call('DEL', KEYS[1])
        else
            return 0
        end
        """
        self.redis.eval(script, 1, self.lock_key, self.token)

---

Eviction Policies

Redis offers multiple eviction policies when memory limits are reached.

# maxmemory set to 2gb
# maxmemory-policy defined how eviction works

# noeviction: reject writes, reads allowed (default)
# allkeys-lru: evict least recently used keys across all keys
# allkeys-random: evict random keys
# volatile-lru: evict LRU keys only in keys with TTL
# volatile-random: evict random keys with TTL
# volatile-ttl: evict keys with shortest TTL
# allkeys-lfu: evict least frequently used keys
# volatile-lfu: evict LFU keys with TTL

For caching, allkeys-lru is typically the best choice. For session storage with TTL, volatile-lru ensures only expiring keys get evicted.

---

In-Memory vs Persistent: Tradeoffs

Characteristic	Redis (In-Memory)	DynamoDB (Persistent)
Latency	Sub-millisecond	Single-digit milliseconds
Durability	Optional (RDB/AOF)	Always durable
Scalability	Vertical plus read replicas	Automatic sharding
Cost	Pay for memory	Pay for throughput
Complexity	Single instance relatively simple	Requires understanding capacity
Data size	Limited by memory	Virtually unlimited

Redis with persistence is not equivalent to DynamoDB. Redis persistence (RDB snapshots, AOF logs) guards against crashes but not against disk failures. DynamoDB’s architecture distributes data across multiple AZs by default.

For true durability in Redis, you need replication to follower instances that can take over if the primary fails.

---

Common Production Failures

Redis OOM kills crashing the database: Redis runs out of memory and the OS kernel kills it via OOM killer. Your application starts returning errors and recovery takes time to reload data. Set maxmemory and maxmemory-policy, configure vm.overcommit_memory=1 at the OS level, and have a reload plan from your primary database ready.

Keys without TTL filling up Redis: Someone writes SET mykey value instead of SET mykey value EX 3600 in the rate limiter. The key never expires. Redis fills up, allkeys-lru starts evicting keys you actually need. Default to TTL on every key—a missing expiration is a bug, not a feature.

DynamoDB hot partition throttling: Your session table uses userId as the partition key. Turns out user-0000 is the test account that every load test hits. That partition maxes out at 300 RCU while the rest of the table sits idle. Spread write volume across partition keys, and implement exponential backoff with jitter when you hit ProvisionedThroughputExceededException.

AOF always mode killing Redis write throughput: You enable AOF with appendfsync always for “maximum durability”. Under write load, Redis blocks on every sync to disk and throughput collapses. Switch to appendfsync everysec or appendfsync no, and match your disk I/O to the write rate if you use everysec.

Distributed lock released by wrong owner: The lock code calls DEL on expiry. But the lock expired while your work was still running, another client grabbed it, and now two processes hold the same lock. The token-based release (compare-and-delete via Lua script) shown in the code example prevents this—only the lock owner can release it.

DynamoDB single-table design gone wrong: You pack 15 entity types into one table to avoid “wasting” tables. The GSI situation becomes a mess and access patterns start colliding. Single-table design shines with 3-5 entity types that share access patterns. Beyond that, the operational complexity wins.

---

Capacity Estimation: Memory-per-Key and DynamoDB Partition Math

Redis memory is the primary constraint. Each key-value pair has overhead on top of the value size. A string key with a 100-byte value uses roughly 100 bytes plus 40-60 bytes of key metadata (keyspace overhead, pointer to value, expiry metadata if set). A hash with 10 fields uses more — each field name is stored alongside the value.

Rough formula for Redis memory: total_keys * (avg_key_size + avg_value_size + 40 bytes_overhead). For a cache with 1 million keys averaging 500 bytes each: roughly 540 MB plus Redis’s own internal fragmentation. Set maxmemory with headroom — at 80% used, Redis starts evicting and you lose predictability.

For TTLs, the expiration queue has a cost: Redis scans the keyspace periodically to expire keys. With millions of keys with TTLs, ACTIVE_EXPIRE_CYCLE_SLOW runs every 100ms. This is generally fast but if you set TTLs on millions of keys at once (a batch import, for example), Redis can briefly block. The latency-sensitive work — expiring the keys you care about — happens incrementally.

DynamoDB partition sizing: DynamoDB creates partitions based on partition key values and throughput. Each partition supports up to 1,000 RCU, 1,000 WCU, and 10 GB of data. For a table with 50 GB and 5,000 RCU, you need at minimum 5 partitions (10 GB each), and the RCU requirement means you actually need 5 partitions just for throughput even if the data were smaller.

If you exceed 10 GB per partition, DynamoDB splits automatically. If you exceed 1,000 WCU on a single partition before splitting, you get throttled. DynamoDB adaptive capacity can temporarily burst above 1,000 WCU per partition, but sustained traffic above that requires either better key distribution or requesting a provisioning increase from AWS support.

---

Trade-off Analysis: Redis vs DynamoDB vs Memcached

Use Case	Redis	DynamoDB	Memcached
Sub-millisecond caching	Native, in-memory	Single-digit ms, NVMe-backed	Native, in-memory
Distributed locks	Native (SET NX EX)	Conditional writes	Not natively supported
Session storage	Fast, ephemeral	Durable, managed	Fast, ephemeral
Write-heavy workloads	Good with persistence	Excellent auto-scaling	Good, but no persistence
Managed service	Self-hosted or Redis Enterprise	Fully managed by AWS	Self-hosted or ElastiCache
Data size limits	Bounded by memory	400KB per item, PB total	1MB per item
Replication	Master-replica, Cluster	Multi-AZ by default	Replica support via consistent hashing

---

Observability Hooks: Redis INFO and DynamoDB CloudWatch

For Redis, INFO is the primary observability interface. The output has sections: Server (Redis build info, version, uptime), Clients (connected clients, blocked clients), Memory (used_memory, used_memory_peak, mem_fragmentation_ratio), Persistence (RDB/AOF status), Stats (total commands processed, keyspace hits/misses), Replication (role, master link status), CPU, Latency stats.

The metrics to watch in production: mem_fragmentation_ratio above 1.5 means Redis is using 50% more memory than it needs — restart Redis or adjust activedefrag settings. keyspace_hits / (keyspace_hits + keyspace_misses) is your cache hit ratio. Below 80% and your cache is not earning its memory. instantaneous_ops_per_sec shows current throughput. connected_clients approaching maxclients (default 10,000) is a connection exhaustion warning.

For DynamoDB, CloudWatch metrics matter: ConsumedReadCapacityUnits and ConsumedWriteCapacityUnits against ProvisionedThroughput. If you consistently consume above 80% of provisioned, DynamoDB autoscales or you need to increase. ThrottledRequests is the key alarm — non-zero throttling means your application is failing requests. UserErrors with ProvisionedThroughputExceededException means your keys are unevenly distributed. ReplicationLatency matters for global tables — cross-region replication lag should stay under 1 second.

Real-World Case Study: Twitter’s Redis Timeline Storage

Twitter’s early architecture used Redis to store user timelines — the list of tweets a user sees when they open their home timeline. The problem was timeline size: celebrities with millions of followers had timelines that could not be computed on read (pulling from all followed users’ tweets in real time was too slow).

Their solution was write-time fanout: when a user posts a tweet, Twitter pushes that tweet into the timelines of all their followers via a Redis pipeline. Reading a timeline became a simple Redis range read — fast, predictable. The tradeoff was write amplification: one tweet from a celebrity required writes to millions of Redis instances.

The operational consequence: the Redis cluster storing timelines was among the largest in production. Memory pressure was constant. When a Redis node failed, millions of users noticed missing timeline entries until repair ran. Twitter’s fix was a hybrid model — push timelines for active users, pull from graph for inactive ones. The lesson: Redis makes reads cheap at the cost of write amplification and memory pressure. Design for the read-to-write ratio of your actual workload.

Interview Questions

Further Reading

For more on NoSQL databases, see the NoSQL overview. To learn about caching strategies, see Caching Strategies. For comparison with Redis and Memcached, see Redis vs Memcached.

Conclusion

Key-value stores trade query flexibility for raw speed and simplicity. Redis works well as a cache, session store, and real-time data structure server where sub-millisecond latency matters. DynamoDB works well as a managed, scalable, durable store for application data with predictable access patterns.

The key insight is that key-value stores require knowing your access patterns upfront. You cannot query on arbitrary fields. Design your keys around how you actually read data, not around the structure of the data itself.