System Design: Twitter Feed Architecture and Scalability

Twitter handles millions of users posting billions of tweets. The core challenge is not just storing tweets but delivering them to followers in real-time. This case study examines how to design a Twitter-like social media platform.

This is an advanced system design topic. If you are new to distributed systems, start with our Database Scaling and Caching Strategies guides.

Requirements Analysis

Functional Requirements

Users should be able to:

• Post tweets (text, images, videos)
• Follow/unfollow other users
• View a timeline of tweets from followed users
• Search for tweets and users
• Like, retweet, and reply to tweets
• Receive notifications for interactions

Non-Functional Requirements

The system needs:

• Timeline latency under 200ms for 99th percentile
• Support for 500 million users
• Handle 200 million daily active users
• Process 500 million tweets per day (6,000 tweets per second peak)
• 99.99% availability

Data Models

Core Entities

-- Users table
CREATE TABLE users (
    id BIGSERIAL PRIMARY KEY,
    username VARCHAR(15) NOT NULL UNIQUE,
    email VARCHAR(255) NOT NULL UNIQUE,
    password_hash VARCHAR(255) NOT NULL,
    display_name VARCHAR(50),
    bio TEXT,
    avatar_url VARCHAR(500),
    follower_count BIGINT DEFAULT 0,
    following_count BIGINT DEFAULT 0,
    verified BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Tweets table
CREATE TABLE tweets (
    id BIGSERIAL PRIMARY KEY,
    author_id BIGINT NOT NULL REFERENCES users(id),
    parent_id BIGINT REFERENCES tweets(id),  -- For replies
    content TEXT NOT NULL,
    media_urls JSONB DEFAULT '[]',
    retweet_of BIGINT REFERENCES tweets(id),
    like_count BIGINT DEFAULT 0,
    retweet_count BIGINT DEFAULT 0,
    reply_count BIGINT DEFAULT 0,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    deleted BOOLEAN DEFAULT FALSE,

    INDEX idx_tweets_author (author_id),
    INDEX idx_tweets_created (created_at DESC)
);

-- Follows table
CREATE TABLE follows (
    follower_id BIGINT NOT NULL REFERENCES users(id),
    following_id BIGINT NOT NULL REFERENCES users(id),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    PRIMARY KEY (follower_id, following_id)
);

-- Likes table
CREATE TABLE likes (
    user_id BIGINT NOT NULL REFERENCES users(id),
    tweet_id BIGINT NOT NULL REFERENCES tweets(id),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    PRIMARY KEY (user_id, tweet_id)
);

NoSQL Alternative

For extreme write throughput, use a wide-column store:

{
  "table": "tweets",
  "key": "tweet_id",
  "columns": {
    "author_id": "bigint",
    "content": "text",
    "created_at": "timestamp",
    "media": ["list of urls"]
  },
  "clustering": ["created_at DESC"]
}

Timeline Architecture

The timeline is where Twitter’s architecture gets interesting. There are two main approaches: push (fan-out on write) and pull (fan-out on read).

Push Model (Fan-out on Write)

When a user posts a tweet, push it to all followers’ timelines immediately:

sequenceDiagram
    participant U as User Posts Tweet
    participant API as API Server
    participant FF as Fan-out Service
    participant Q as Message Queue
    participant TM as Timeline Cache

    U->>API: POST /tweets {content: "Hello"}
    API->>DB: Store tweet
    DB-->>API: Tweet created
    API->>FF: Fan-out tweet
    FF->>DB: Get followers (10K)
    FF->>Q: Enqueue fan-out jobs
    Q->>TM: Push to each timeline cache

This approach provides fast reads but expensive writes. For celebrities with millions of followers, fan-out becomes problematic.

Pull Model (Fan-out on Read)

When a user loads their timeline, aggregate tweets from all followed users:

sequenceDiagram
    participant U as User Views Timeline
    participant API as API Server
    participant TM as Timeline Cache
    participant DB as Tweet DB
    participant MR as Merge & Rank

    U->>API: GET /timeline
    API->>TM: Get cached timeline IDs
    TM-->>API: Timeline IDs (200 tweets)
    API->>MR: Request tweet details
    MR->>DB: Batch fetch tweets
    DB-->>MR: Tweet objects
    MR-->>API: Enriched tweets
    API-->>U: Timeline response

Hybrid approach combines both: pull for celebrities, push for regular users.

Feed Generation

Timeline Service

class TimelineService:
    def __init__(self, cache: RedisCluster, db: Database):
        self.cache = cache
        self.db = db

    async def get_timeline(
        self,
        user_id: int,
        limit: int = 200,
        cursor: str = None
    ) -> TimelineResponse:
        # Try cache first
        cached = await self.cache.get(f"timeline:{user_id}")
        if cached and not cursor:
            return self._parse_cached_timeline(cached)

        # Fetch from multiple sources
        following = await self._get_following(user_id)
        tweets = await self._fetch_tweets(user_id, following, limit, cursor)

        return TimelineResponse(
            tweets=tweets,
            cursor=self._generate_cursor(tweets)
        )

    async def _fetch_tweets(
        self,
        user_id: int,
        following: List[int],
        limit: int,
        cursor: str
    ) -> List[Tweet]:
        # Split into celebrities and regular users
        celebrities, regular = await self._categorize_users(following)

        # Fetch in parallel
        celebrity_tweets = await self._fetch_ celebrity_tweets(celebrities, limit)
        regular_tweets = await self._fetch_regular_tweets(
            user_id, regular, limit - len(celebrity_tweets), cursor
        )

        # Merge and rank
        merged = self._merge_tweets(celebrity_tweets, regular_tweets)
        return self._rank_tweets(merged)[:limit]

Caching Timeline

Cache timelines with user-specific keys:

CACHE_CONFIG = {
    "timeline_ttl": 3600,        # 1 hour
    "max_timeline_size": 800,    # Store 800 tweets, return 200
    "min_followers_for_fanout": 10000  # Celebrity threshold
}

async def cache_timeline(user_id: int, tweets: List[Tweet]):
    key = f"timeline:{user_id}"
    serialized = serialize_tweets(tweets)

    await self.cache.setex(
        key,
        CACHE_CONFIG["timeline_ttl"],
        serialized
    )

Fan-out Service

The fan-out service distributes tweets to follower timelines:

High-Follower Handling

class FanoutService:
    async def fanout_tweet(self, tweet: Tweet):
        author_id = tweet.author_id
        follower_count = await self.db.fetch_val(
            "SELECT follower_count FROM users WHERE id = $1",
            author_id
        )

        if follower_count > CACHE_CONFIG["min_followers_for_fanout"]:
            # Celebrity: skip fan-out, users pull on read
            await self._mark_tweet__celebrity(tweet)
        else:
            # Regular user: fan-out to all followers
            await self._fanout_to_followers(tweet)

    async def _fanout_to_followers(self, tweet: Tweet):
        async with self.db.pool.acquire() as conn:
            async for row in conn.cursor(
                "SELECT follower_id FROM follows WHERE following_id = $1",
                tweet.author_id
            ):
                follower_id = row["follower_id"]
                await self._push_to_timeline(follower_id, tweet)

    async def _push_to_timeline(self, user_id: int, tweet: Tweet):
        key = f"timeline:{user_id}"
        tweet_preview = {
            "id": tweet.id,
            "author_id": tweet.author_id,
            "created_at": tweet.created_at.isoformat()
        }

        # Add to front of list
        await self.cache.lpush(key, serialize_tweet_preview(tweet_preview))
        # Trim to max size
        await self.cache.ltrim(key, 0, CACHE_CONFIG["max_timeline_size"] - 1)

Search Architecture

Twitter search requires full-text search across billions of tweets.

Elasticsearch Mapping

{
  "mappings": {
    "properties": {
      "tweet_id": { "type": "long" },
      "author_id": { "type": "long" },
      "content": {
        "type": "text",
        "analyzer": "twitter_analyzer",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "hashtags": { "type": "keyword" },
      "mentions": { "type": "keyword" },
      "created_at": { "type": "date" },
      "engagement_score": { "type": "integer" }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "twitter_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "snowball"]
        }
      }
    }
  }
}

Search Query

async def search_tweets(
    query: str,
    limit: int = 20,
    since_id: str = None,
    max_id: str = None
) -> SearchResponse:
    must_clauses = [
        {"match": {"content": query}}
    ]

    filter_clauses = [
        {"range": {"created_at": {"gte": "2020-01-01"}}}
    ]

    if since_id:
        filter_clauses.append({"range": {"tweet_id": {"gt": int(since_id)}}})

    result = await es.search(
        index="tweets",
        body={
            "query": {
                "bool": {
                    "must": must_clauses,
                    "filter": filter_clauses
                }
            },
            "sort": [
                {"_score": "desc"},
                {"tweet_id": "desc"}
            ],
            "size": limit
        }
    )

    return SearchResponse(
        tweets=[Tweet(**hit["_source"]) for hit in result["hits"]["hits"]],
        count=result["hits"]["total"]["value"]
    )

Notification System

Notification Types

Type	Trigger	Delivery
Mention	@username in tweet	Immediate
Reply	Reply to user’s tweet	Immediate
Like	Someone likes your tweet	Batch (hourly)
Follow	Someone follows you	Batch (hourly)
Retweet	Someone retweets your tweet	Batch (hourly)

Notification Pipeline

graph LR
    A[Tweet Event] --> B[Event Router]
    B --> C[Real-time Queue]
    B --> D[Batch Queue]
    C --> E[Push Service]
    C --> F[WebSocket]
    D --> G[Batch Processor]
    G --> H[Email Service]
    G --> I[Push Notifications]

Notification Service

class NotificationService:
    def __init__(self, queue: KafkaProducer, db: Database):
        self.queue = queue
        self.db = db

    async def notify_mention(self, tweet: Tweet, mentioned_users: List[int]):
        for user_id in mentioned_users:
            await self.queue.send("notifications", {
                "type": "mention",
                "user_id": user_id,
                "tweet_id": tweet.id,
                "author_id": tweet.author_id,
                "created_at": datetime.utcnow().isoformat()
            })

    async def notify_followers(self, follower_ids: List[int], event: Dict):
        batch = {
            "type": "follow",
            "user_ids": follower_ids,
            "actor_id": event["actor_id"],
            "created_at": datetime.utcnow().isoformat()
        }
        await self.queue.send_batch("notifications-batch", [batch])

Caching Patterns

Multi-Layer Caching

CACHE_LAYERS = [
    {"name": "tweet", "ttl": 86400, "size": "16MB"},
    {"name": "timeline", "ttl": 3600, "size": "100MB"},
    {"name": "user", "ttl": 900, "size": "16MB"},
    {"name": "social-graph", "ttl": 300, "size": "50MB"}
]

Cache-Aside for Tweets

async def get_tweet(tweet_id: int) -> Optional[Tweet]:
    # L1: Memcached
    cached = await mc.get(f"tweet:{tweet_id}")
    if cached:
        return Tweet(**json.loads(cached))

    # L2: Redis cluster
    cached = await redis.get(f"tweet:{tweet_id}")
    if cached:
        tweet = Tweet(**json.loads(cached))
        await mc.set(f"tweet:{tweet_id}", json.dumps(tweet.dict()))
        return tweet

    # Database
    tweet = await db.fetch_one(
        "SELECT * FROM tweets WHERE id = $1", tweet_id
    )

    if tweet:
        await redis.setex(f"tweet:{tweet_id}", 3600, json.dumps(tweet))
        await mc.set(f"tweet:{tweet_id}", json.dumps(tweet.dict()))

    return tweet

Write Path Optimization

Write-Behind Cache

async def post_tweet(user_id: int, content: str) -> Tweet:
    # Write to database
    tweet = await db.create(
        "INSERT INTO tweets (author_id, content) VALUES ($1, $2) RETURNING *",
        user_id, content
    )

    # Update timeline cache asynchronously
    asyncio.create_task(self._update_timeline_caches(user_id, tweet))

    # Index for search asynchronously
    asyncio.create_task(self._index_tweet(tweet))

    return tweet

Tweet ID Generation

Use snowflake IDs for tweet ordering:

class SnowflakeID:
    def __init__(self, datacenter_id: int = 0, worker_id: int = 0):
        self.datacenter_id = datacenter_id
        self.worker_id = worker_id
        self.timestamp = 0
        self.sequence = 0

    def generate(self) -> int:
        timestamp = int(time.time() * 1000) - 1609459200000  # Epoch offset

        if timestamp == self.timestamp:
            self.sequence = (self.sequence + 1) & 4095
        else:
            self.sequence = 0

        self.timestamp = timestamp

        return (
            (timestamp << 22) |
            (self.datacenter_id << 17) |
            (self.worker_id << 12) |
            self.sequence
        )

API Design

Core Endpoints

Method	Endpoint	Description
POST	/api/v2/tweets	Create tweet
DELETE	/api/v2/tweets/:id	Delete tweet
GET	/api/v2/users/:id/timeline	Get user timeline
GET	/api/v2/timelines/home	Get home timeline
GET	/api/v2/tweets/search	Search tweets
POST	/api/v2/users/:id/follow	Follow user
DELETE	/api/v2/users/:id/follow	Unfollow user

Response Format

{
  "data": {
    "id": "1234567890",
    "text": "Hello, Twitter!",
    "author_id": "9876543210",
    "created_at": "2026-03-22T10:30:00Z",
    "like_count": 1523,
    "retweet_count": 342,
    "reply_count": 89
  },
  "meta": {
    "result_count": 1,
    "next_token": "b26v89c19zqg8o3fosdk7n9l"
  }
}

Scalability Challenges

The Celebrity Problem

Users like @elonmusk have millions of followers. Fanning out their tweets to all timelines takes time and resources.

Solutions:

• Skip fan-out for high-follower accounts
• Use CDN for celebrity content
• Implement rate limiting on celebrity tweets

Hot Keys

Popular tweets create hot spots in the database.

Solutions:

• Partition by hash(tweet_id)
• Use eventual consistency for counts
• Implement client-side rate limiting

Write Amplification

Fan-out multiplies write load.

Solutions:

• Async fan-out via queues
• Hybrid push/pull model
• Periodic batch updates

Conclusion

Twitter’s architecture balances read and write optimization through clever hybrid approaches. The key insights are:

• Push for regular users, pull for celebrities
• Multi-layer caching throughout
• Eventual consistency for non-critical data
• Async processing for expensive operations

---

Production Failure Scenarios

Failure Scenario	Impact	Mitigation
Fan-out queue backlog	Timeline updates delayed for hours	Auto-scale fan-out workers; prioritize active users
Celebrity tweet viral	Massive fan-out burst crashes servers	Rate limit celebrity tweets; pre-compute popular timelines
Timeline cache miss	Cold start: timeline loads from DB, high latency	Pre-warm caches for trending users; use background refresh
Search index lag	New tweets not appearing in search	Accept search lag; prioritize by engagement
Hot tweet partition	Single partition overloaded by likes/retweets	Shard by tweet_id; use eventual consistency for counts
Tweet DB primary failure	Writes fail; no new tweets	Use multi-primary replication; promote read replica

---

Capacity Estimation

QPS Calculations

Daily active users: 200 million
Tweets per user per day: 2 (average)
Peak QPS: 200M × 2 / 86400 ≈ 4,600 tweets/second (average)
Peak spike factor: 10x → 46,000 tweets/second

Timeline reads:
- 50% users check timeline 3x/day
- Average timeline: 200 tweets
- Read QPS: 200M × 0.5 × 3 / 86400 ≈ 3,500 reads/second
- Peak: ~35,000 reads/second

Storage Estimation

Tweets stored: 500 million tweets/day × 365 days × 5 years
= 500M × 365 × 5 = 912.5 billion tweets

Per tweet (avg): 500 bytes content + 200 bytes metadata = 700 bytes
Total: 912.5B × 700 bytes ≈ 639 TB

With 3x replication and indices: ~2 PB storage needed

---

Observability Checklist

Metrics to Capture

• tweets_published_total (counter) - By author tier (celebrity vs regular)
• timeline_load_duration_seconds (histogram) - P50, P95, P99
• timeline_cache_hit_ratio (gauge) - Hit vs miss rate
• fanout_queue_depth (gauge) - Messages pending fan-out
• search_indexing_lag_seconds (gauge) - Time from tweet to searchable
• engagement_events_total (counter) - Likes, retweets, replies by tier

Logs to Emit

{
  "timestamp": "2026-03-22T10:30:00Z",
  "event": "tweet_published",
  "author_id": "123456",
  "author_tier": "celebrity",
  "follower_count": 15000000,
  "fanout_queued": true,
  "fanout_queue_depth": 50000
}

Alerts to Configure

Alert	Threshold	Severity
Timeline P99 > 500ms	500ms for 5 min	Warning
Fan-out queue > 1M	1000000 pending	Critical
Cache hit ratio < 80%	80% for 10 min	Warning
Search lag > 5 min	300s	Warning

---

Security Checklist

• Rate limiting on tweet publication (prevent spam)
• Tweet content moderation (pre-scan for abuse)
• Anti-scraping measures (limit bulk reads)
• DM encryption (end-to-end for private messages)
• OAuth 2.0 for all API access
• IP-based access controls for internal services
• Audit logging of admin operations

---

Common Pitfalls / Anti-Patterns

Pitfall 1: Fanning Out to All Followers Synchronously

Problem: Synchronous fan-out on every tweet creates massive latency spikes for users with many followers.

Solution: Use async fan-out via message queues. Return success to user immediately, fan-out in background.

Pitfall 2: Not Handling the Celebrity Problem

Problem: A user with 10M followers causes fan-out storms that overwhelm the system.

Solution: Implement hybrid model. Skip fan-out for users above a follower threshold; their tweets are pulled on read.

Pitfall 3: Storing Full Tweets in Timeline Cache

Problem: Caching 800 tweets × 5KB each = 4MB per user timeline in cache.

Solution: Store only tweet IDs and timestamps in timeline cache; fetch full tweet objects separately with multi-get.

Pitfall 4: Ignoring Engagement Calculation Cost

Problem: Calculating “trending” or “ranked” timeline requires scanning engagement data on every read.

Solution: Pre-compute engagement scores; update asynchronously; use Redis sorted sets for real-time ranking.

---

Interview Q&A

Q: How does Twitter handle celebrity users with millions of followers?

A: Twitter uses a hybrid approach. For regular users (under 10,000 followers), tweets are fanned out to follower timelines on write. For celebrities and high-follower accounts, fan-out is skipped. Their tweets are pulled on read when users load their timeline. This prevents fan-out storms that would overwhelm the system.

Q: Why does Twitter store only tweet IDs in timeline cache, not full tweet objects?

A: Each timeline stores 800 tweet IDs (to serve 200 tweets). Storing full tweet objects at 5KB each would mean 4MB per user timeline cache. With millions of users, memory requirements become prohibitive. Storing only IDs keeps cache compact; full tweet objects are fetched with multi-get on timeline load.

Q: How does Twitter ensure message ordering in timelines?

A: Tweet IDs are Snowflake IDs containing timestamps. They are monotonically increasing within a datacenter. When merging tweets from celebrities (pulled) and regular users (pushed), the client sorts by tweet ID which naturally sorts by time.

Q: What happens when you like a tweet?

A: The like action triggers an async notification pipeline. The like count on the tweet is incremented (with eventual consistency, not synchronous). If the tweet author has batch notifications enabled, they receive a batched hourly notification instead of an immediate push.

---

Scenario Drills

Scenario 1: Surge in Celebrity Tweet Fan-out

Situation: A major celebrity posts a tweet that goes viral. Their follower count is 15 million.

Analysis:

• Synchronous fan-out would require 15 million timeline cache writes
• Each write takes ~1ms, totaling 15,000 seconds of fan-out time
• This blocks the API response and creates massive queue backlog

Solution: Skip fan-out for accounts above the follower threshold. The tweet appears in follower timelines only when they next refresh, pulling the tweet on read.

Scenario 2: Hot Tweet Partition Overload

Situation: A tweet about breaking news receives millions of likes and retweets within minutes.

Analysis:

• Likes and retweets update counters on the tweet record
• All updates route to the same partition by tweet_id
• The partition becomes a hot spot, causing high latency

Solution: Use eventual consistency for counters. Accept that like counts may be slightly stale. Implement client-side rate limiting. For viral tweets, batch counter updates.

Scenario 3: Timeline Cache Miss on User Return

Situation: A user who has not opened Twitter for 3 days returns. Their timeline cache has expired.

Analysis:

• Cache miss triggers full timeline recomputation
• Must fetch all followed users’ recent tweets
• Merging celebrity pulls with regular user pushes takes time

Solution: Pre-warm caches for trending users. Use background refresh for active but not recently active users. Show a “loading timeline” state while recomputing.

---

Failure Flow Diagrams

Timeline Cache Failure

graph TD
    A[User Requests Timeline] --> B{Cache Available?}
    B -->|Yes| C[Return Cached Timeline]
    B -->|No| D[Query Followed Users]
    D --> E{Celebrity in List?}
    E -->|Yes| F[Pull Celebrity Tweets from DB]
    E -->|No| G[Pull Regular Tweets from Cache]
    F --> H[Merge Timeline]
    G --> H
    H --> I[Store in Cache]
    I --> J[Return Timeline]
    C --> J
    J --> K{Request Success?}
    K -->|No| L[Show Error State]
    K -->|Yes| M[Render Timeline UI]

Fan-out Queue Backlog

graph LR
    A[Tweet Published] --> B{Follower Count > 10K?}
    B -->|Yes| C[Skip Fan-out]
    B -->|No| D[Get Follower List]
    d --> E[Batch Followers]
    E --> F[Queue Fan-out Jobs]
    F --> G{Queue Depth < Threshold?}
    G -->|Yes| H[Process in Order]
    G -->|No| I[Prioritize Active Users]
    H --> J[Push to Timeline Cache]
    I --> J
    J --> K{Timeout?}
    K -->|Yes| L[Retry with Backoff]
    K -->|No| M[Confirm Delivery]

---

Quick Recap

• Push (fan-out on write) for regular users, pull (fan-out on read) for celebrities.
• Hybrid timeline: pre-computed for active users, computed on-demand for inactive users.
• Multi-layer caching: tweet cache, timeline cache, user cache, social graph cache.
• Async fan-out via message queues to handle burst traffic.
• Use snowflake IDs for globally sortable tweet ordering.

Copy/Paste Checklist

- [ ] Implement hybrid push/pull for celebrity accounts
- [ ] Cache timeline IDs, not full tweet objects
- [ ] Use async fan-out via message queue
- [ ] Partition by tweet_id for hot tweet handling
- [ ] Monitor fan-out queue depth and auto-scale workers
- [ ] Pre-warm caches for trending users
- [ ] Implement rate limiting on tweet publication
- [ ] Use snowflake IDs for ordering

For more on caching patterns, see our Caching Strategies guide. For database sharding, see Horizontal Sharding. For real-time features like Twitter uses, see the Design Chat System case study.