System Design: URL Shortener from Scratch

URL shorteners are deceptively simple systems. The core functionality just converts long URLs to short ones and redirects users when they visit the short URL. But building one that handles millions of users with low latency and high availability reveals interesting challenges.

This case study walks through designing a URL shortener like bit.ly or TinyURL.

Requirements Analysis

Functional Requirements

Users need to:

• Shorten a long URL into a compact link
• Access the short URL and get redirected to the original
• Optionally set expiration dates
• Optionally customize the short code
• View statistics on link usage

Non-Functional Requirements

The system must be:

• Fast: Redirects under 100ms
• Available: Handle service disruptions gracefully
• Scalable: Millions of links, billions of redirects
• Durable: No lost links

Capacity Estimation

Daily active users: 100 million URL creations Redirect ratio: 100:1 (one creation, one hundred redirects)

Storage needed over 5 years:

• 100M links/day 365 days 5 years = 182.5 billion links
• At 500 bytes per link: ~91 TB

Redirect QPS: 100M * 100 / 86400 = ~115,000 QPS

Overall Architecture

sequenceDiagram
    participant U as User
    participant LB as Load Balancer
    participant API as API Server
    participant Cache as Cache
    participant DB as Database
    participant Short as Shortener Service

    U->>LB: POST /shorten {url: "long..."}
    LB->>API: Forward request
    API->>Short: Generate short code
    Short->>Cache: Check if code exists
    Short->>DB: Store mapping
    Cache->>API: Confirm store
    API->>U: Return short URL

    U->>LB: GET /{shortCode}
    LB->>API: Forward request
    API->>Cache: Lookup original URL
    Cache-->>API: Return URL
    API->>U: 302 Redirect to URL

Core Components

Short Code Generation

The short code is the heart of the system. It needs to be:

• Unique
• Random enough to be unpredictable
• Short (6-8 characters typical)
• URL-safe (alphanumeric)

Hash Function Approaches

Approach 1: MD5/SHA hash of long URL + salt

import hashlib
import base62

def generate_short_code(url: str, salt: str = "mysalt") -> str:
    hash_input = f"{url}:{salt}"
    md5_hash = hashlib.md5(hash_input.encode()).hexdigest()
    # Take first 8 characters and encode in base62
    return base62.encode(int(md5_hash[:8], 16))[:8]

# Example
short = generate_short_code("https://example.com/very/long/url/path")
# Result: "xV2bP9qK"

Problem: Same URL always produces same hash, enabling URL enumeration attacks.

Approach 2: Hash + counter for uniqueness

import hashlib
import base62
import time

def generate_short_code(url: str, salt: str = "mysalt") -> str:
    # Combine URL with timestamp and random to ensure uniqueness
    combined = f"{url}:{time.time_ns()}:{random.randint(0, 999999)}"
    md5_hash = hashlib.md5(combined.encode()).hexdigest()
    return base62.encode(int(md5_hash[:8], 16))[:8]

Approach 3: Counter-based (KGS approach)

Use a Key Generation Service that pre-generates short codes:

class KeyGenerationService:
    def __init__(self, batch_size=1000):
        self.batch_size = batch_size
        self.available_keys = []

    def get_next_key(self) -> str:
        if not self.available_keys:
            self._refill_batch()
        return self.available_keys.pop()

    def _refill_batch(self):
        # Generate batch from counter
        start = self._get_current_counter()
        for i in range(start, start + self.batch_size):
            self.available_keys.append(base62.encode(i))
        self._increment_counter(start + self.batch_size)

The counter approach guarantees uniqueness and allows easy key management.

Base62 Encoding

Base62 uses characters 0-9, A-Z, a-z giving 62 characters per position:

Length	Possible Combinations	Equivalent URLs
6	62^6 = 56.8 billion	Enough for all links
7	62^7 = 3.5 trillion	Generous headroom
8	62^8 = 218 trillion	Future-proof

Data Model

Relational Schema

CREATE TABLE urls (
    id BIGSERIAL PRIMARY KEY,
    short_code VARCHAR(12) NOT NULL UNIQUE,
    original_url TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    expires_at TIMESTAMP WITH TIME ZONE,
    is_custom BOOLEAN DEFAULT FALSE,
    creator_id BIGINT,
    click_count BIGINT DEFAULT 0,
    is_active BOOLEAN DEFAULT TRUE,

    CONSTRAINT urls_short_code_idx UNIQUE (short_code)
);

CREATE INDEX idx_urls_short_code ON urls(short_code);
CREATE INDEX idx_urls_creator ON urls(creator_id);
CREATE INDEX idx_urls_expires ON urls(expires_at) WHERE expires_at IS NOT NULL;

NoSQL Alternative (DynamoDB)

{
  "TableName": "urls",
  "KeySchema": [{ "AttributeName": "short_code", "KeyType": "HASH" }],
  "AttributeDefinitions": [
    { "AttributeName": "short_code", "AttributeType": "S" },
    { "AttributeName": "creator_id", "AttributeType": "N" }
  ],
  "GlobalSecondaryIndexes": [
    {
      "IndexName": "creator-index",
      "KeySchema": [{ "AttributeName": "creator_id", "KeyType": "HASH" }],
      "Projection": { "ProjectionType": "ALL" },
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 100,
        "WriteCapacityUnits": 50
      }
    }
  ],
  "ProvisionedThroughput": {
    "ReadCapacityUnits": 1000,
    "WriteCapacityUnits": 500
  }
}

Caching Strategy

Redirect latency is critical. Cache aggressively.

Cache Structure

# Redis cache key pattern
cache_key = f"url:{short_code}"

# Cache value
cache_value = {
    "original_url": "https://example.com/very/long/path",
    "expires_at": "2027-01-01T00:00:00Z",
    "is_active": True
}

Cache TTL Strategy

CACHE_TTL = {
    "frequently_accessed": 3600,    # 1 hour for popular links
    "recently_created": 300,        # 5 minutes for new links
    "custom": 86400,                # 24 hours for custom links
    "expired": 60                   # 1 minute for recently expired
}

Write-Through Cache

async def create_short_url(url: str, custom_code: str = None) -> str:
    short_code = custom_code or generate_short_code(url)

    # Write to database
    await db.urls.create({
        "short_code": short_code,
        "original_url": url,
        "is_custom": custom_code is not None
    })

    # Write to cache
    await cache.set(f"url:{short_code}", {
        "original_url": url,
        "expires_at": None,
        "is_active": True
    }, ttl=CACHE_TTL["recently_created"])

    return short_code

Cache Miss Handling

async def get_original_url(short_code: str) -> Optional[str]:
    # Check cache first
    cached = await cache.get(f"url:{short_code}")
    if cached:
        return cached["original_url"]

    # Cache miss - fetch from database
    url_record = await db.urls.get(short_code=short_code)

    if not url_record:
        return None

    # Populate cache
    await cache.set(f"url:{short_code}", {
        "original_url": url_record.original_url,
        "expires_at": url_record.expires_at,
        "is_active": url_record.is_active
    }, ttl=CACHE_TTL["recently_created"])

    return url_record.original_url

Redirect Logic

HTTP Redirect Types

Status	Use Case	Browser Behavior
301	Permanent move	Caches redirect
302	Temporary redirect	No cache
303	Post -> Get	Converts to GET
307	Temporary	Preserves method
308	Permanent	Preserves method

For URL shorteners, typically use 301 (permanent) for SEO or 302 (temporary) for analytics tracking.

Redirect Handler

from fastapi import FastAPI, HTTPException, status
from fastapi.responses import RedirectResponse

app = FastAPI()

@app.get("/{short_code}")
async def redirect_to_original(short_code: str):
    # Check for special paths
    if short_code in ["health", "metrics", "docs"]:
        raise HTTPException(status_code=404)

    # Validate short code format
    if not is_valid_short_code(short_code):
        raise HTTPException(status_code=400, detail="Invalid short code")

    # Get original URL
    original_url = await get_original_url(short_code)

    if not original_url:
        raise HTTPException(status_code=404, detail="URL not found")

    # Track click asynchronously
    asyncio.create_task(track_click(short_code))

    return RedirectResponse(
        url=original_url,
        status_code=status.HTTP_302_FOUND
    )

Rate Limiting

Prevent abuse with rate limits per IP:

from slowapi import Limiter
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)

@app.post("/shorten")
@limiter.limit("10/minute")
async def create_short_url(request: Request, url: str = Body(...)):
    # Validate URL
    if not is_valid_url(url):
        raise HTTPException(status_code=400, detail="Invalid URL")

    # Check if already shortened by this user
    existing = await find_existing_mapping(url, request.client.host)
    if existing:
        return {"short_url": f"https://short.ly/{existing.short_code}"}

    short_code = await create_mapping(url)
    return {"short_url": f"https://short.ly/{short_code}"}

High Availability Design

Database High Availability

-- PostgreSQL synchronous replication
ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM SET synchronous_standby_names = '*';

-- Read replicas for redirects
CREATE PUBLICATION url_shares FOR TABLE urls;

-- On replica
CREATE SUBSCRIPTION url_sub CONNECTION 'host=primary port=5432 dbname=urlshort' PUBLICATION url_shares;

Multiple Redis Instances

from rediscluster import RedisCluster

# Redis Cluster configuration
rc = RedisCluster(
    startup_nodes=[
        {"host": "redis-1", "port": 6379},
        {"host": "redis-2", "port": 6379},
        {"host": "redis-3", "port": 6379}
    ],
    decode_responses=True
)

async def get_original_url(short_code: str) -> Optional[str]:
    # Consistent hashing handles failover automatically
    cached = await rc.get(f"url:{short_code}")
    return json.loads(cached)["original_url"] if cached else None

Geographic Distribution

Deploy redirector clusters in multiple regions:

# Route 53 latency routing
- Name: short.ly
  Type: A
  SetIdentifier: us-east-1
  Region: us-east-1
  AliasTarget:
    DNSName: dualstack.api-elb-us-east-1.amazonaws.com
    EvaluateTargetHealth: true

# EU redirector
- Name: short.ly
  Type: A
  SetIdentifier: eu-west-1
  Region: eu-west-1
  AliasTarget:
    DNSName: dualstack.api-elb-eu-west-1.amazonaws.com
    EvaluateTargetHealth: true

Users are routed to the nearest cluster based on latency.

Analytics Pipeline

Track clicks without slowing redirects:

async def track_click(short_code: str):
    # Fire and forget - don't await
    asyncio.ensure_future(
        kafka.send("clicks", {
            "short_code": short_code,
            "timestamp": datetime.utcnow().isoformat(),
            "user_agent": request.headers.get("user-agent"),
            "referer": request.headers.get("referer"),
            "ip_hash": hash_ip(request.client.host)
        })
    )

Click Analytics Consumer

async def process_clicks():
    consumer = KafkaConsumer("clicks", bootstrap_servers=["kafka:9092"])

    for message in consumer:
        event = json.loads(message.value)

        # Update click count in background
        await db.query("""
            UPDATE urls
            SET click_count = click_count + 1
            WHERE short_code = $1
        """, event["short_code"])

        # Update analytics warehouse
        await warehouse.insert("click_events", event)

Complete API Specification

Endpoints

Method	Endpoint	Description
POST	/api/v1/shorten	Create short URL
GET	/{short_code}	Redirect to original
GET	/api/v1/links/{short_code}	Get link info
GET	/api/v1/links/{short_code}/stats	Get click statistics
DELETE	/api/v1/links/{short_code}	Delete a link
PUT	/api/v1/links/{short_code}	Update link settings

Request/Response Examples

# Create short URL
curl -X POST https://short.ly/api/v1/shorten \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/very/long/path/that/needs/shortening"}'

# Response
{
  "short_code": "xV2bP9qK",
  "short_url": "https://short.ly/xV2bP9qK",
  "original_url": "https://example.com/very/long/path/that/needs/shortening",
  "created_at": "2026-03-22T10:30:00Z",
  "expires_at": null
}

Conclusion

A URL shortener seems simple but requires careful design for scale. The key decisions are:

• Short code generation: Counter-based with KGS ensures uniqueness and predictability
• Caching: Aggressive caching with Redis handles redirect traffic
• Database: PostgreSQL with read replicas for availability
• Analytics: Asynchronous click tracking via Kafka

---

Abuse Prevention and Security

Malicious URL Detection

URL shorteners are frequently abused for phishing, malware distribution, and spam. Implement safeguards:

class MaliciousURLDetector:
    """Detect potentially malicious URLs before shortening"""

    def __init__(self, threat_intel_client: ThreatIntelClient):
        self.threat_intel = threat_intel_client
        self.suspicious_tlds = {
            '.tk', '.ml', '.ga', '.cf', '.gq',  # Free tier often abused
            '.xyz', '.top', '.club'  # Often used in spam
        }

    async def check_url(self, url: str) -> ThreatAssessment:
        checks = await asyncio.gather(
            self._check_domain_reputation(url),
            self._check_url_pattern(url),
            self._check_content_scan(url),
            self._check_google_safe_browsing(url)
        )

        if any(check.threat for check in checks):
            return ThreatAssessment(
                threat=True,
                reason="URL flagged by security checks",
                severity="high"
            )

        return ThreatAssessment(threat=False)

    async def _check_url_pattern(self, url: str) -> CheckResult:
        parsed = urlparse(url)

        # Check for suspicious TLDs
        if any(parsed.netloc.endswith(tld) for tld in self.suspicious_tlds):
            return CheckResult(threat=True, reason="Suspicious TLD")

        # Check for IP address instead of domain
        if self._is_ip_address(parsed.netloc):
            return CheckResult(threat=True, reason="IP address used")

        # Check for excessive subdomains
        if parsed.netloc.count('.') > 4:
            return CheckResult(threat=True, reason="Excessive subdomains")

        return CheckResult(threat=False)

Rate Limiting Tiers

RATE_LIMITS = {
    "anonymous": {"shorten": "5/hour", "redirect": "100/hour"},
    "authenticated_free": {"shorten": "100/hour", "redirect": "1000/hour"},
    "authenticated_pro": {"shorten": "10000/hour", "redirect": "100000/hour"},
}

@app.middleware
async def rate_limit_middleware(request: Request, call_next):
    user_tier = get_user_tier(request)

    if user_tier == "anonymous":
        # Rate limit by IP
        client_ip = request.client.host
        if not await rate_limiter.check_limit(f"ip:{client_ip}", RATE_LIMITS["anonymous"]["shorten"]):
            raise HTTPException(status_code=429, detail="Rate limit exceeded")

    elif user_tier == "authenticated":
        user_id = get_user_id(request)
        if not await rate_limiter.check_limit(f"user:{user_id}", RATE_LIMITS[user_tier]["shorten"]):
            raise HTTPException(status_code=429, detail="Rate limit exceeded")

    return await call_next(request)

Spam Link Prevention

async def create_short_url(url: str, user_id: int = None) -> ShortUrl:
    # Validate URL format
    if not is_valid_url(url):
        raise HTTPException(status_code=400, detail="Invalid URL format")

    # Check for known spam domains
    if await spam_database.is_spam_domain(extract_domain(url)):
        raise HTTPException(status_code=403, detail="URL blocked")

    # Require authentication for custom codes
    if custom_code and not user_id:
        raise HTTPException(status_code=401, detail="Authentication required for custom codes")

    # Create short URL
    return await url_service.create(url, custom_code, user_id)

---

Production Failure Scenarios

Failure Scenario	Impact	Mitigation
Redis cache failure	All redirects hit DB, high latency	Fallback to direct DB reads; circuit breaker on cache
KGS (key gen) failure	Cannot create new short URLs	Use hash-based codes as fallback; KGS recovery priority
Database primary failure	Cannot create or redirect	Promote read replica; use eventual consistency for analytics
DNS resolution failure	short.ly domain unreachable	Multi-cloud DNS; anycast IP; aggressive caching
CDN failure for stats page	Stats load slowly	Static asset caching; local caching

Cache Failure Handling

async def get_original_url(short_code: str) -> Optional[str]:
    try:
        # Try cache first
        cached = await redis.get(f"url:{short_code}")
        if cached:
            return json.loads(cached)["original_url"]
    except RedisConnectionError:
        # Cache unavailable - fall through to DB
        pass

    # Fallback to database
    url_record = await db.urls.get(short_code=short_code)
    if not url_record:
        return None

    # Don't try to repopulate cache if Redis is down
    return url_record.original_url

---

Observability Checklist

Metrics to Capture

• url_redirects_total (counter) - By short_code prefix, status code
• url_shortens_total (counter) - By user_tier, custom vs auto
• redirect_latency_seconds (histogram) - P50, P95, P99
• cache_hit_ratio (gauge) - Cache efficiency
• malicious_url_attempts_total (counter) - Blocked attempts by type
• kgs_available_keys (gauge) - Key generation health

Logs to Emit

{
  "timestamp": "2026-03-22T10:30:00Z",
  "event": "redirect",
  "short_code": "xV2bP9qK",
  "status": 302,
  "latency_ms": 12,
  "cache_hit": true,
  "user_ip_hash": "abc123"
}

Alerts to Configure

Alert	Threshold	Severity
Redirect latency P99 > 200ms	200ms	Warning
Cache hit ratio < 50%	50%	Warning
KGS keys < 1000	1000	Critical
Malicious attempts spike	> 100/min	Warning
DB connection pool > 80%	80%	Warning

---

Security Checklist

• TLS 1.3 for all connections
• URL validation and sanitization
• Malicious URL scanning (Google Safe Browsing API)
• Rate limiting per IP and per user
• Custom code length and character validation
• Authentication required for custom short codes
• Audit logging of all URL creations
• GDPR compliance for analytics data
• Regular security audits

---

Common Pitfalls / Anti-Patterns

Pitfall 1: Using Sequential IDs as Short Codes

Problem: Sequential IDs (1, 2, 3…) allow URL enumeration - attackers can guess other short URLs.

Solution: Use cryptographically random codes (base62) with minimum 6 characters. Use KGS for guaranteed uniqueness without predictability.

Pitfall 2: Not Handling URL Expiration

Problem: Expired URLs still redirect until cache expires.

Solution: Check expiration on every redirect. Set aggressive cache TTL for URLs with near-term expiration.

Pitfall 3: Storing Only Short Code in Cache

Problem: Cache miss requires DB query for every redirect.

Solution: Cache generously. Use “recently accessed” eviction. Pre-populate cache for trending links.

---

Quick Recap

• Short codes should be random (base62), not sequential, to prevent enumeration.
• Aggressive caching (Redis) handles redirect traffic; database handles writes.
• KGS (Key Generation Service) provides predictable, unique codes without collision.
• Abuse prevention (rate limiting, malicious URL scanning) is essential.
• Monitor cache hit ratio, redirect latency, and KGS key availability.

Copy/Paste Checklist

- [ ] Use base62 encoding with minimum 6 characters
- [ ] Implement KGS for predictable unique codes
- [ ] Redis caching with write-through on create
- [ ] Rate limiting per IP and user tier
- [ ] Malicious URL scanning integration
- [ ] Handle cache failure with DB fallback
- [ ] Monitor cache hit ratio and latency
- [ ] Set appropriate cache TTLs by URL type

For more system design patterns, see our Caching Strategies guide which covers caching patterns used here. The Load Balancing guide covers geographic distribution.