NoSQL Databases: Document, Key-Value, Column-Family, and Graph Databases

NoSQL covers a lot of ground. The name itself is a negative definition: databases that are not SQL. That grouping includes wildly different systems with different strengths and trade-offs. Understanding what each type actually offers helps you pick the right tool, rather than defaulting to whichever database has the loudest marketing.

Relational databases dominated for decades, and they still handle most workloads well. But when your access patterns do not fit the tabular model, NoSQL databases can simplify your architecture significantly.

What NoSQL Actually Means

NoSQL databases emerged to solve problems relational systems handle poorly. Some data is naturally document-structured, not tabular. Some access patterns require single-key lookups at massive scale. Some workloads need graph traversal rather than filtered aggregations.

The common thread is flexibility. NoSQL systems typically trade away features to get it. You might lose ACID transactions across documents, or find yourself deciding between consistency and availability. Understanding these trade-offs matters more than the marketing around any particular database.

CAP Theorem Revisited

The CAP theorem describes a fundamental limitation in distributed data systems. You can only guarantee two of three properties: Consistency, Availability, and Partition tolerance. Since network partitions will happen in any distributed system, the real choice is between consistency and availability.

graph TD
    A[CAP Theorem] --> B[Consistency]
    A --> C[Availability]
    A --> D[Partition Tolerance]
    B --> E[Every read gets<br/>most recent write]
    C --> F[Every request gets<br/>a response]
    D --> G[Must handle<br/>network failures]

Relational databases typically choose consistency. Many NoSQL databases choose availability, trading consistency for the ability to stay available during network partitions. This is why some NoSQL systems are called “eventually consistent.”

For more on CAP theorem implications, see my post on the CAP Theorem.

Document Databases

Document databases store data as documents, typically JSON or similar formats. Each document is self-contained with all its related data. You query by document fields rather than joining across tables.

MongoDB is the workhorse of document databases. Documents look like this:

{
  "_id": ObjectId("..."),
  "email": "user@example.com",
  "name": "Jane Smith",
  "orders": [
    {
      "id": "order123",
      "total": 99.99,
      "items": ["widget", "gadget"]
    }
  ]
}

The document model fits naturally when data comes from APIs or when objects have variable structure. A user document can include orders, preferences, and metadata without normalizing into separate tables.

MongoDB offers secondary indexes, aggregation pipelines, and transaction support (within a single document or across multiple documents in a replica set). It handles scale through horizontal sharding.

DynamoDB, from AWS, also speaks document style but with a different philosophy. It is key-value at core, with document support bolted on. DynamoDB requires planning your access patterns upfront, which can feel restrictive but pays off at massive scale.

Key-Value Stores

Key-value stores are the simplest NoSQL model. Every value is accessed by a unique key. There is no query language, no joins, no aggregation. Just get by key, put by key.

Redis is the most versatile key-value store. It supports strings, lists, sets, sorted sets, hashes, and more. Redis runs in memory, making it extremely fast. It persists to disk for durability but the primary use case is caching and session storage.

# Redis key-value operations
redis.set("user:123:session", session_data)
session = redis.get("user:123:session")
redis.expire("user:123:session", 3600)  # 1 hour TTL

Redis shines when you need simple lookups at high speed. Session storage, caching, rate limiting, leaderboards. The data structures give you more power than pure key-value but the access pattern stays simple.

Amazon DynamoDB works as a key-value store when you only use the primary key. The partition key gives you direct access to an item.

Column-Family Databases

Column-family databases store data in column families rather than rows. Think of it as a two-level map: partition key, then column names with associated values. Cassandra is the most prominent column-family database.

Cassandra was built for write-heavy workloads across commodity servers. It offers tunable consistency, letting you choose how many replicas must acknowledge a write before it succeeds.

CREATE TABLE users_by_region (
    region text,
    user_id uuid,
    email text,
    name text,
    PRIMARY KEY (region, user_id)
);

This schema groups users by region. All users in Europe are stored together, making queries by region efficient. Queries across regions require scatter-gather across all partitions.

Cassandra uses a query-first approach to schema design. You design tables for your specific query patterns, not to represent entity relationships.

Column families handle time-series data well. You can model sensor readings where each partition covers a time window. Range queries within a partition are efficient.

Graph Databases

Graph databases store data as nodes and edges. Nodes represent entities, edges represent relationships between them. This model shines when relationships matter as much as the data itself.

Neo4j is the most established graph database. Its Cypher query language expresses patterns in the graph:

MATCH (person:Person)-[:KNOWS]->(friend:Person)
WHERE person.name = 'Alice'
RETURN friend.name

This query finds everyone Alice knows. In a relational database, this requires multiple joins. In a graph database, following edges is native.

Graph databases excel for social networks, recommendation engines, fraud detection, and knowledge graphs. When the relationship traversal is the primary access pattern, graphs blow relational systems out of the water.

Choosing Between NoSQL Options

Picking a NoSQL database depends on your access patterns and scale requirements.

Use document databases when your data is document-structured, varies in schema, or benefits from embedding related data. MongoDB handles most document use cases well. DynamoDB is the choice if you need massive scale and can plan your access patterns upfront.

Use key-value stores for caching, sessions, and simple lookups. Redis is incredibly fast and versatile for these use cases. If you need durability at massive scale with simple operations, DynamoDB covers this too.

Use column-family databases for write-heavy workloads, time-series data, and when your query patterns fit a two-level key structure. Cassandra handles multi-datacenter replication well.

Use graph databases when relationship traversal is the core of your workload. Social graphs, recommendation systems, and fraud detection are natural fits.

NoSQL Type Comparison

When evaluating NoSQL databases, different types offer different trade-offs across key dimensions:

Dimension	Document (MongoDB, DynamoDB)	Key-Value (Redis, DynamoDB)	Column-Family (Cassandra)	Graph (Neo4j)
Consistency Model	Tunable (eventual to strong)	Eventual or strong	Tunable quorum	ACID or eventually consistent
Query Flexibility	High (ad-hoc queries, secondary indexes)	Low (key-only access)	Low (CQL requires known column family)	High (traversal queries)
Write Throughput	Medium-High	Very High	Very High	Medium
Read Performance	Medium (indexed)	Extremely Fast (O(1))	Medium-High (range queries)	Fast for traversals, slow for scans
Scaling Model	Horizontal sharding	Horizontal partitioning	Horizontal (multi-datacenter native)	Vertical + read replicas
Operational Complexity	Medium	Low	Medium-High	Medium
Cost (Managed)	$$	$-$$	$-$$	$$$
Transaction Scope	Single document or multi-doc replica set	Single key	Single row (LWT adds latency)	Single or multi-node
Schema Flexibility	Dynamic schema	None (value opaque)	Defined columns per family	Property graph schema

Interpretation Guide:

• Consistency: Systems on the left favor consistency; those on the right favor availability during partitions
• Query Flexibility: Ability to perform ad-hoc queries without knowing keys upfront
• Cost: $- = low cost, $$$ = enterprise pricing (varies by provider and scale)

For most teams starting out, document databases offer the best balance of flexibility and operational simplicity. Key-value stores shine when your access patterns are simple and throughput is critical. Column-family databases work well for write-heavy time-series workloads. Graph databases are specialized tools for relationship-heavy problems.

Decision Flowchart: Choosing a NoSQL Type

Use this flowchart to identify which NoSQL type fits your workload:

graph TD
    Start[What is your primary<br/>access pattern?] --> Q1{Do you need to<br/>traverse relationships?}

    Q1 -->|Yes| Graph[Graph Database<br/>Neo4j, Amazon Neptune]

    Q1 -->|No| Q2{Is your data<br/>document-structured?}

    Q2 -->|Yes| Q3{Can you plan all<br/>access patterns upfront?}

    Q3 -->|Yes| Q4{Do you need massive scale<br/>with simple operations?}

    Q4 -->|Yes| KeyValue[Key-Value Store<br/>DynamoDB, Redis]

    Q4 -->|No| Document[Document Database<br/>MongoDB, CouchDB]

    Q3 -->|No| Document

    Q2 -->|No| Q5{Are writes your<br/>bottleneck?}

    Q5 -->|Yes| Q6{Do you need<br/>multi-datacenter replication?}

    Q6 -->|Yes| ColumnFamily[Column-Family<br/>Cassandra, HBase]

    Q6 -->|No| Q7{Can you accept<br/>eventual consistency?}

    Q7 -->|Yes| KeyValue

    Q7 -->|No| Q8{Are relationships<br/>simple?}

    Q8 -->|Yes| ColumnFamily

    Q8 -->|No| Document

    Q5 -->|No| Q9{Are reads your<br/>bottleneck?}

    Q9 -->|Yes| Q10{Is data relatively<br/>static or read-heavy?}

    Q10 -->|Yes| KeyValue[Caching Layer<br/>Redis, Memcached]

    Q10 -->|No| Document

    Q9 -->|No

Quick Decision Guide:

If your answer is yes to…	Choose…
Traversing relationships is core to my workload	Graph database
I need massive scale with simple key lookups	Key-value store
My data is document-structured with variable schema	Document database
I have write-heavy workloads across multiple datacenters	Column-family database
None of the above fit well	Start with document database

When Relational Still Wins

NoSQL is not always the answer — and honestly, most of the time it probably is not.

If your data fits a tabular model with clear relationships, relational databases offer more features and better tooling. ACID transactions across multiple tables are hard to match. SQL provides powerful querying for analytical workloads.

Operational simplicity matters. Most developers know SQL. Most teams can debug a MySQL or PostgreSQL issue quickly. The ecosystem around relational databases is mature.

NoSQL databases often require more operational expertise. You need to understand the CAP trade-offs, figure out capacity planning for sharding, and learn operational procedures specific to whichever system you chose.

Start with relational unless you have specific reasons not to. The flexibility argument for NoSQL is often overstated.

When to Use and When Not to Use NoSQL Databases

When to Use NoSQL Databases:

• Document databases: Your data is document-structured, varies in schema, or benefits from embedding related data
• Key-value stores: You need simple lookups at extreme speed for caching or sessions
• Column-family databases: You have write-heavy workloads with predictable query patterns
• Graph databases: Relationship traversal is the core of your workload
• You need horizontal write scaling beyond what relational databases offer
• Your access patterns are simple and predictable at scale

When Not to Use NoSQL Databases:

• You need ACID transactions across multiple documents or entities
• Your data has complex relationships requiring multi-table joins
• You need ad-hoc querying capabilities or powerful aggregation
• Your team lacks operational expertise for the specific NoSQL system
• You are using the “flexibility” argument without specific requirements

Production Failure Scenarios

Failure	Impact	Mitigation
Partition key hotspots	Some nodes receive disproportionate traffic	Choose high-cardinality partition keys, implement virtual nodes
Replication factor too low	Data loss on node failure	Set replication factor >= 3 for production, use quorum reads/writes
Network partition during writes	Lost or duplicated writes depending on consistency level	Use tunable consistency appropriate for each operation
Cascading failures from network split	Cluster becomes unavailable or split-brain	Implement failure detection, use odd node counts for quorum
Compaction storms	I/O spikes causing latency spikes	Schedule compactions during off-peak, use tiered compaction
Unpredictable query patterns	Full cluster scans under load	Design query patterns around partition key access, denormalize for common queries
Schema evolution issues	Application incompatibility with stored documents	Use versioning in documents, run migration scripts before deployment
Backup failures	No recoverable point in case of data corruption	Test backups regularly, implement incremental backup strategies

Observability Checklist

Metrics to Monitor:

Track request latency (p50, p95, p99) by operation type and request throughput (reads/writes per second). Watch node availability and cluster health, disk usage per node and cluster-wide, and network I/O between nodes. Replication lag and pending writes matter for consistency. Compaction progress and queue depth affect performance unpredictably. If you use caching, track cache hit ratios. Connection pool utilization catches exhaustion before it becomes an outage.

Logs to Capture:

Log node failures and restarts, compaction events with I/O patterns, authentication failures and denied permissions. Slow query logs (set a threshold that makes sense for your workload) are essential. For JVM-based systems like Cassandra, GC pauses and heap pressure show up at the worst times. Network partition events are worth capturing for post-mortems.

Alerts to Set:

Alert on node down or cluster minority partition immediately. Set thresholds for latency spikes that exceed your SLA, disk usage above 80% on any node, and replication lag beyond your consistency requirements. Watch for failed request rate increases and compaction queue depth spiking. Connection pool exhaustion will take down your app before you notice anything else.

# Example: Cassandra nodetool commands for monitoring
# nodetool status           # Cluster health and node status
# nodetool tpstats          # Thread pool statistics
# nodetool cfstats          # Keyspace and table statistics
# nodetool proxyhistograms  # Client request latency

Security Checklist

• Enable authentication (internal authentication or external like Kerberos)
• Implement authorization with role-based access control
• Encrypt client-to-node and node-to-node traffic
• Encrypt data at rest (filesystem or application-level encryption)
• Use TLS for all inter-node communication
• Restrict network access to database nodes (firewall rules)
• Implement keyspace-level or table-level permissions
• Audit log sensitive operations and access patterns
• Sanitize all user inputs to prevent injection attacks
• Use secure secret management for database credentials
• Regularly rotate credentials and encryption keys
• Test security configuration with penetration testing

Common Pitfalls and Anti-Patterns

1. Using NoSQL without clear access patterns: NoSQL requires understanding your query patterns upfront. Without this, you either over-engineer or end up with hot spots and inefficient access.

2. Ignoring eventual consistency implications: Reading from replicas before writes propagate means you might get stale data. Consider your consistency requirements per operation.

3. Unbounded partition key growth: Time-based or counter-based partition keys create unbounded partitions that will haunt you. Use natural keys with fixed cardinality.

4. Over-reliance on secondary indexes: Secondary indexes often cause full cluster scans. Design your primary access patterns around partition keys instead.

5. Ignoring data modeling for access patterns: In relational DBs you model data. In NoSQL you model queries. Design tables for your query patterns, not your entity structure.

6. Skipping backup and recovery testing: Many NoSQL systems have complex backup mechanisms. Test recovery procedures regularly or you will lose data eventually.

7. Underestimating operational complexity: Each NoSQL system has its own operational quirks. Make sure your team has training and runbooks for your specific system.

8. Using wide partitions for “flexibility”: Stuffing too much data into single partitions kills performance. Keep partition size bounded.

9. Assuming linear scalability: Adding nodes does not instantly double your capacity. Rebalancing takes time and causes temporary load spikes you did not plan for.

Quick Recap

Key takeaways:

• NoSQL databases are purpose-built for specific access patterns and scale requirements
• CAP theorem trade-offs force you to choose between consistency and availability during partitions
• Document databases work well for flexible schemas and embedded data patterns
• Key-value stores give you extreme speed for simple lookups
• Column-family databases handle write-heavy workloads with predictable queries
• Graph databases are built for relationship traversal workloads

Copy/Paste Checklist:

# MongoDB connection with monitoring
from pymongo import MongoClient
client = MongoClient("mongodb://user:pass@host:27017/?replicaSet=rs0")
db = client.admin
# Check replica set status
print(db.command('replSetGetStatus'))

# Cassandra connection with monitoring
from cassandra.cluster import Cluster
cluster = Cluster(['host1', 'host2', 'host3'])
session = cluster.connect('mykeyspace')
# Check cluster health
session.execute("SELECT * FROM system.local")
session.execute("SELECT * FROM system.peers")

# Redis security and monitoring
redis-cli INFO clients  # Client connections
redis-cli INFO stats     # Command statistics
redis-cli CONFIG GET *   # Configuration audit

Performance Benchmarks

NoSQL databases sit at different points on the latency-throughput curve depending on their architecture. These numbers are representative from published benchmarks and vary significantly based on workload and configuration.

Throughput Comparison

Database	Write Throughput (ops/sec)	Read Throughput (ops/sec)	Notes
Cassandra	400,000+	1,000,000+	Tunable consistency affects speed
DynamoDB	4,000 (provisioned)	4,000 (provisioned)	On-demand mode: ~1000/second per shard
MongoDB	50,000-100,000	100,000-300,000	Depends on sharding and replica set configuration
Redis	1,000,000+	1,000,000+	In-memory, no persistence overhead in benchmark mode
Neo4j	10,000-50,000	100,000+	Traversal speed depends heavily on graph topology
DynamoDB (DAX)	Same as DynamoDB	10x DynamoDB reads	DAX adds microsecond latency for cache hits

Latency Profiles

Database	p50 Latency	p99 Latency	p99.9 Latency	Consistency Model
Redis	0.1-0.5ms	0.5-1ms	1-2ms	Strong (single node)
Cassandra	1-3ms	5-10ms	15-30ms	Tunable (quorum lower)
MongoDB	1-5ms	10-30ms	50-100ms	Tunable (majority)
DynamoDB	1-5ms	10-20ms	30-50ms	Eventually consistent
DynamoDB (DAX)	0.1-0.5ms	0.5-1ms	2-5ms	Eventually consistent
Neo4j	1-10ms	20-50ms	100ms+	Causal (per operation)

Numbers assume local datacenter. Cross-region latency adds 50-200ms depending on geography.

Factors That Break Benchmarks

Hot partitions wreck Cassandra throughput. Secondary indexes force full scans. Connection pool exhaustion hits during traffic spikes. GC pauses plague JVM-based databases. Network jitter. Compaction storms. Benchmarks assume ideal conditions. Production throws all of these at you at once, usually at 3am.

# Estimate actual p99 for your workload
def estimated_p99(base_p99_ms, replication_factor, consistency_level):
    factor = {
        'ONE': 1.0,
        'LOCAL_QUORUM': 1.3,
        'QUORUM': 1.5,
        'ALL': 2.0
    }.get(consistency_level, 1.3)

    # Compaction, GC, and network add ~20% at high load
    load_factor = 1.2
    return base_p99_ms * factor * load_factor

NewSQL: SQL at Distributed Scale

NewSQL databases promise the consistency of SQL with the horizontal scaling of NoSQL. They use distributed SQL architectures — typically a coordinator node routing queries across multiple data nodes — while presenting a single logical database to applications.

CockroachDB

CockroachDB implements distributed SQL with PostgreSQL wire compatibility. It uses the Raft consensus algorithm for data replication and supports distributed transactions across nodes. Tables are split into ranges (64MB by default) and distributed across the cluster.

-- CockroachDB: multi-region deployment
CREATE TABLE orders (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    customer_id UUID NOT NULL,
    total DECIMAL(10,2) NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT now()
) WITH (zone_configs = {
    'datacenter': 'us-east-1',
    'num_replicas': 5,
    'num_voters': 3
});

-- Follow-the-workload replication: moves data to where it's accessed
ALTER TABLE orders SET locality = 'follow-the-workload';

CockroachDB is a good fit for teams that need PostgreSQL compatibility with multi-region write distribution. It handles node failures without downtime and provides serializable isolation. The trade-off is higher latency (2-4x vs single-node PostgreSQL) and significant operational complexity.

TiDB

TiDB is a distributed SQL database compatible with MySQL. It separates compute and storage via TiKV (distributed key-value storage) and TiDB (SQL compute layer). It supports horizontal scaling of both compute and storage independently.

-- TiDB: distributed table
CREATE TABLE users (
    id BIGINT NOT NULL AUTO_INCREMENT,
    email VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (id),
    KEY idx_email (email)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

-- Enable horizontal scaling via TiKV
-- TiDB automatically splits tables into regions across TiKV stores
-- Usepd (Placement Driver) manages data placement and replica scheduling

TiDB fits MySQL migration scenarios where you need to scale write throughput beyond single-node MySQL without rearchitecting. The MySQL compatibility is high, making it a practical migration target.

NewSQL vs NoSQL Decision

Factor	NewSQL (CockroachDB, TiDB)	NoSQL (Cassandra, DynamoDB)
SQL support	Full ACID SQL	Key-value or document only
Transaction scope	Distributed ACID	Single partition or tunable
Consistency model	Strong (serializable)	Eventual or tunable
Write scaling	Moderate (coordinator bottleneck)	Very high (no coordinator)
Operational complexity	High (distributed SQL tuning)	Medium
Use when	Need SQL + multi-region writes	Need extreme scale, simple ops

NewSQL is overkill for most applications. If you do not need distributed transactions across multiple tables, a well-configured PostgreSQL or MySQL replica set handles millions of queries per day. NewSQL earns its complexity when you genuinely need both SQL semantics and geographic write distribution — and not a moment before.

Cost Analysis for Managed Services

Managed NoSQL services abstract away operational complexity but come with pricing that compounds at scale. Here is how the major managed options stack up against each other.

DynamoDB Pricing Model

Component	Cost
On-demand writes	$1.25 per million WCUs
On-demand reads	$0.25 per million RCUs
Provisioned (10 RCU + 10 WCU)	$0.00 (free tier) + $0.0009/RCU + $0.00045/WCU
Storage	$0.25 per GB-month
DAX	$0.12 per hour per node (t3.medium)
Global Tables	2x base replication cost

DynamoDB on-demand mode charges per operation. At high throughput, provisioned capacity with auto-scaling is significantly cheaper. DAX adds cache costs but reduces read costs by 10x for cache-friendly workloads.

MongoDB Atlas Pricing Model

Component	Cost
M10 (single node)	~$0.08/hour ($60/month)
M30 (3-node replica)	~$0.50/hour ($360/month)
M50 (sharded)	~$1.50/hour ($1,080/month)
Storage (500 GB)	~$125/month
Backup	$0.20/GB-month
Data Transfer	$0.01-0.12/GB (tiered)

Atlas pricing scales with RAM per node. Sharded clusters cost 3-5x replica sets for equivalent total storage because each shard runs a full replica set.

Cassandra Pricing (DataStax Astra vs Self-Hosted)

Component	DataStax Astra (Serverless)	Self-Hosted (3 nodes)
Base cost	$0.10/vCPU-hour	EC2 instances + EBS
Storage (1TB)	~$230/month	~$100/month (gp3)
Multi-region	+20% per region	+inter-region transfer costs
Estimated 3-node HA	~$600/month	~$400/month (r6i.2xlarge)

Self-hosting is cheaper at scale but requires operational expertise. DataStax Astra reduces operational burden at a 30-50% premium.

Cost Break-Even Calculator

def dynamodb_vs_cassandra(reads_per_month, writes_per_month, storage_gb):
    # DynamoDB on-demand
    dynamodb_read_cost = (reads_per_month / 1_000_000) * 0.25
    dynamodb_write_cost = (writes_per_month / 1_000_000) * 1.25
    dynamodb_storage_cost = storage_gb * 0.25
    dynamodb_total = dynamodb_read_cost + dynamodb_write_cost + dynamodb_storage_cost

    # Self-hosted Cassandra (3x r6i.2xlarge + EBS)
    ec2_cost = 3 * 0.38 * 730  # $0.38/hr * 730 hrs/month
    ebs_cost = storage_gb * 0.08 * 3  # gp3 with provisioned IOPS
    cassandra_total = ec2_cost + ebs_cost

    return {
        'dynamodb_monthly': round(dynamodb_total, 2),
        'cassandra_monthly': round(cassandra_total, 2),
        'recommendation': 'DynamoDB' if dynamodb_total < cassandra_total else 'Cassandra'
    }

# Example: 100M reads, 10M writes, 100GB storage
# {'dynamodb_monthly': 50.00, 'cassandra_monthly': 921.60, 'recommendation': 'DynamoDB'}

At low-to-medium scale, managed DynamoDB is cheaper. At very high scale (TB+ data, billions of ops/month), self-hosted Cassandra or provisioned DynamoDB becomes more economical.

Vendor-Specific Configuration

Each managed service has configuration knobs that matter significantly for performance.

DynamoDB On-Demand vs Provisioned

import boto3

dynamodb = boto3.client('dynamodb')

# Switch to on-demand after estimating capacity
dynamodb.update_table(
    TableName='orders',
    BillingMode='PAY_PER_REQUEST'
)

# Or set provisioned capacity with auto-scaling
dynamodb.update_table(
    TableName='orders',
    BillingMode='PROVISIONED',
    ProvisionedThroughput={
        'ReadCapacityUnits': 100,
        'WriteCapacityUnits': 50
    }
)

# CloudWatch metric to monitor
# AWS/DynamoDB > ConsumedReadCapacityUnits
# AWS/DynamoDB > ConsumedWriteCapacityUnits
# Alert at > 80% utilization for 5 minutes

DynamoDB adaptive capacity automatically increases throughput for tables with uneven access patterns. Hot partitions get more capacity dynamically. Do not rely on this as a substitute for good partition key design — it has limits.

MongoDB Atlas Performance Configuration

// Atlas cluster settings via Atlas API or UI
// Indexes are critical — design before going to production

// Create compound index matching your query patterns
db.orders.createIndex({ customer_id: 1, created_at: -1 });

// Monitoring: Atlas Proactive Bot alerts when:
// - Query targeting > 1000 documents per second
// - Slow queries (> 100ms)
// - Connection pool > 80% utilization

// Atlas Auto-Scaling
// Scale up: triggered when CPU > 75% for 5 minutes
// Scale down: triggered when CPU < 25% for 30 minutes (once per hour)
// Atlas Data Explorer for schema visualization and index recommendations

The most common MongoDB performance mistake is not creating indexes for your query patterns. Atlas Performance Advisor suggests indexes based on slow query logs.

Cassandra Configuration for Write-Heavy Workloads

# cassandra.yaml — tuning for write throughput

# Concurrent writes: match number of CPU cores
concurrent_writes: 32

# Concurrent reads: 4x number of CPU cores for SSD
concurrent_reads: 128

# Memtable flush writer threads
memtable_flush_writers: 8  # For SSD-backed nodes

# Batch statements for batching writes
# cqlsh example:
BEGIN BATCH
    INSERT INTO orders (id, customer, total) VALUES ('1', 'a', 10.00) USING TTL 86400;
    INSERT INTO orders (id, customer, total) VALUES ('2', 'b', 20.00) USING TTL 86400;
APPLY BATCH;

Cassandra write path is memory-resident until flush. Sizing memtable_flush_writers to match I/O capacity prevents write stalls during compaction.

Real-World Case Studies

Netflix runs one of the largest Cassandra deployments in the world. At peak they handle over 14 million concurrent streams, each backed by metadata queries to their Cassandra clusters. Their architecture uses Cassandra for its linear scalability — adding nodes directly increases throughput without redesigning queries or rebalancing clusters.

Their specific pattern: they partition data by user ID, so each user’s viewing history, preferences, and recommendations live on a small set of partitions. This gives them predictable latency per user and avoids the scatter-gather problem that kills performance when one query touches half the cluster. The lesson is that Cassandra rewards thoughtful partition design upfront. A hot partition — say, a popular show — can saturate a single node even in a 100-node cluster. Netflix mitigates this with virtual nodes and by spreading popular content across many partition keys.

Amazon DynamoDB was designed with a different constraint: partition the problem or lose availability. Their 2012 paper described how they partition by key range, with each partition responsible for a slice of the key space. If a partition receives more traffic than it can handle, DynamoDB splits it automatically. This adaptive splitting is what makes DynamoDB feel infinite — you never hit a ceiling, the service redistributes before you notice.

The tradeoff is that DynamoDB requires upfront access pattern planning. In DynamoDB you define your primary key structure and secondary indexes, and queries are efficient only within those structures. Ad-hoc queries that would be trivial in PostgreSQL require either scanning the entire table (expensive) or redesigning your indexes. This is the right tradeoff for Amazon’s use case — their services have well-defined access patterns. It is the wrong tradeoff for a startup that does not yet know how users will query their data.

Interview Questions

Q: What are the main differences between MongoDB and DynamoDB when deciding which to use at scale?

MongoDB gives you flexible querying — ad-hoc filters, secondary indexes, aggregation pipelines — and you can change your schema without rewriting queries. DynamoDB requires you to plan access patterns upfront, and changing your key structure means migrating data or maintaining multiple indexes. At very small scale, MongoDB is simpler. At massive scale, DynamoDB’s predictable performance and automatic partitioning win. The deciding factor is usually whether your access patterns are known and stable. If they are, DynamoDB avoids operational complexity. If they are not, MongoDB’s flexibility saves you from constant schema migrations.

Q: You are designing a Cassandra keyspace for a messaging application. What partition key would you use and why?

A user-centric partition key works well: (recipient_id, created_at) or (conversation_id, message_id). The goal is keeping related messages co-located on the same partition so reads for a conversation are a single partition query, not a scatter-gather across hundreds of nodes. The risk is unbounded partitions — a prolific user’s conversation can grow without limit and eventually saturate a single node. The mitigation is bucketing: using (conversation_id, bucket_id, message_id) where bucket is a time window (e.g., monthly), so any single partition stays bounded. The partition key design is the most consequential decision in Cassandra — get it wrong and you either have hot spots or inefficient queries.

Q: When would you choose Redis over a disk-based database for a given workload?

Redis when your working set fits in memory and you need sub-millisecond latency. Redis is single-threaded (with some parallelization for persistence) and O(1) for key operations, so latency is extremely consistent. It is the right choice for session storage, rate limiting, caching with TTLs, and leaderboards. It is the wrong choice when your data does not fit in memory (it will swap and performance collapses), when you need durability guarantees that survive crashes without careful appendfsync configuration, or when your access patterns require complex queries that Redis cannot serve.

Q: Your team is considering a NewSQL database (CockroachDB or TiDB) over PostgreSQL with read replicas. What questions would you ask before making that decision?

The key question is whether you need distributed writes. If reads are the bottleneck, PostgreSQL replicas solve that problem cheaply. If writes need to scale horizontally across regions, NewSQL earns its complexity. Ask specifically: do you need multi-region write latency below what eventual-consistency NoSQL offers? Can you tolerate 2-4x write latency compared to single-node PostgreSQL? Is your team prepared for distributed SQL operational quirks — rebalancing, zone failures, distributed transaction contention? For most teams, PostgreSQL with proper indexing and connection pooling handles far more than they expect. NewSQL is for when you have exhausted PostgreSQL’s vertical scalability and need geographic distribution with strong consistency.

Conclusion

NoSQL databases solve real problems but come with trade-offs. Document databases trade consistency for flexible schemas. Key-value stores trade querying for speed. Column-family databases trade ad-hoc queries for write scalability. Graph databases trade simplicity for relationship traversal power.

Understanding CAP theorem implications helps you reason about consistency behavior. Most NoSQL systems offer tunable consistency, letting you choose the right balance for your workload.

NoSQL is not a replacement for relational databases. It is a set of tools for specific problems. The key is knowing which tool fits which problem — and having the humility to admit that for many teams, PostgreSQL is still the right answer.

For related reading, see my posts on Relational Databases to understand when relational systems are still the right choice, and Database Scaling to learn how to scale databases horizontally.