Publish/Subscribe Patterns: Topics, Subscriptions, and Filtering

Publish/subscribe is a messaging pattern where senders (publishers) do not send messages directly to specific receivers. Instead, they broadcast messages to topics. Subscribers receive messages from the topics they care about. This decoupling is useful but introduces challenges around topic design, filtering, and delivery semantics.

This post covers the practical aspects of building pub/sub systems: topic hierarchies, subscription management, message filtering, and fan-out patterns.

Topic Design

Topics are the core abstraction in pub/sub. Designing them well determines how flexible your system becomes.

Flat Topics

The simplest approach: one topic per message type.

user.created
order.placed
payment.processed

Each subscriber chooses which topics to listen to. Simple, but limited. What if you want all user events? You subscribe to multiple topics.

Hierarchical Topics

Hierarchies organize topics into trees, enabling broader subscriptions:

users/
users.created
users.updated
users.deleted

orders/
orders.placed
orders.updated
orders.cancelled

payments/
payments.processed
payments.failed

Subscribing to orders/ gives you all order events. Subscribing to users.* gives you all user lifecycle events.

graph TD
    Publisher -->|publish| Topic[orders.placed]

    subgraph Subscribers
        Analytics[Analytics Service]
        Notifications[Notification Service]
        Audit[Audit Service]
    end

    Topic -->|orders.placed| Analytics
    Topic -->|orders.placed| Notifications
    Topic -->|orders.placed| Audit

The tradeoff is naming conventions. Without discipline, hierarchies become messy.

Subject-Based vs Content-Based

Most pub/sub systems are subject-based: topics are predefined strings that publishers use. Subscribers choose topics to listen to.

Content-based systems route based on message content. A subscriber might say “give me all messages where amount > 1000.” This is more flexible but harder to implement efficiently and raises security concerns (what if a subscriber crafts queries to sniff data they should not see?).

Most practical systems stick with subject-based routing.

Subscription Management

Subscriptions represent a subscriber interest in a topic. Managing them lifecycle-wise is non-trivial.

Durable Subscriptions

A durable subscription persists even when the subscriber is offline. When it reconnects, it receives messages that arrived while it was disconnected.

// Pseudo-code for durable subscription
subscription = client.subscribe("orders.*", durable=true)
client.disconnect()
client.reconnect()
subscription.resume() // missed messages delivered

This matters for mobile clients and services that restart. Without durability, offline periods mean message loss.

Shared Subscriptions

When multiple instances of a service run (say, three notification service instances), you want them to share the work. Shared subscriptions distribute messages across instances.

orders/ -> [notification-1, notification-2, notification-3]
          each instance gets ~1/3 of messages

This is essential for scaling consumers horizontally. Without sharing, all instances receive all messages, defeating the purpose of multiple workers.

Subscription Types by Delivery

• Exclusive: Only one consumer receives messages (no sharing)
• Shared: Messages distributed across multiple consumers
• Failover: One consumer receives all messages, others standby

Message Filtering

Sometimes subscribers need only a subset of messages in a topic. Filtering lets subscribers narrow what they receive.

Topic-Level Filtering

The simplest form: use narrow topics. A subscriber to orders.high-value gets only high-value orders, not all orders.

This requires publishers to classify messages correctly, which couples publisher and subscriber knowledge.

Header-Based Filtering

Messages have headers (key-value pairs) that can be used for filtering:

{
  "topic": "orders",
  "headers": {
    "priority": "high",
    "region": "us-west",
    "source": "web"
  },
  "body": { "order_id": "12345", "amount": 5000 }
}

A subscriber can filter: headers.region = 'us-west' AND headers.priority = 'high'.

This separates classification (publisher job) from filtering (subscriber job).

SQL-Based Filtering

Some systems let subscribers express filters as SQL-like conditions:

SELECT * FROM orders WHERE amount > 1000 AND region = 'us-west'

Amazon SNS supports SQL-based filtering:

{
  "FilterPolicy": {
    "amount": [{ "numeric": [">", 1000] }],
    "region": ["us-west"]
  }
}

Fan-Out Patterns

Fan-out describes how one message reaches multiple subscribers. The pattern varies depending on what you need.

Broadcast Fan-Out

Every subscriber receives every message. The simplest model.

graph LR
    Pub[Publisher] -->|1 message| Topic
    Topic -->|copy 1| Sub1[Subscriber 1]
    Topic -->|copy 2| Sub2[Subscriber 2]
    Topic -->|copy 3| Sub3[Subscriber 3]

The broker makes copies. Cost scales with subscriber count.

Selective Fan-Out

With filtering, only matching subscribers receive the message. More efficient but requires the broker to evaluate filters.

Dead Letter Queues

When a subscriber cannot process a message (crashes, returns error, times out), what happens? Dead letter queues capture failed messages for later inspection.

graph LR
    Pub[Publisher] --> Topic
    Topic -->|normal| Sub[Subscriber]
    Topic -->|failed after retries| DLQ[Dead Letter Queue]

Without DLQs, poison messages block the queue or get dropped silently. DLQs let you debug processing failures.

Implementation Considerations

Message Ordering

Most pub/sub systems do not guarantee ordering across topics. Within a single ordered stream (like Kafka partitions), ordering holds. Across multiple publishers or topics, it does not.

If you need global ordering, you need a totally ordered broadcast protocol, which is expensive. More commonly, you accept per-partition ordering and use correlation IDs to reconstruct order client-side.

Message Deduplication

At-least-once delivery (most pub/sub systems) can deliver the same message twice. Idempotent consumers handle this by tracking processed message IDs.

if message.id in processed_ids:
    skip
else:
    process(message)
    add message.id to processed_ids

Design for at-least-once and make processing idempotent. Exactly-once is usually not worth the cost.

Related Patterns

Pub/sub connects closely to event-driven architecture, where topics carry domain events between services. It also relates to message queue types for understanding where pub/sub fits in the broader messaging landscape.

When to Use / When Not to Use

When to Use Publish-Subscribe

• Multiple independent consumers need the same data: When one event should trigger actions in analytics, notifications, audit, and other services simultaneously
• Microservices decoupling: When services should not call each other directly and should instead react to shared events
• Event broadcasting: When something happened and multiple parts of the system need to know
• Real-time streaming to clients: When pushing updates to web clients, mobile apps, or IoT devices
• Data replication: When the same data needs to appear in multiple systems (databases, caches, search indexes)

When Not to Use Publish-Subscribe

• Request/response patterns: When you need a reply from a specific service, use direct RPC or a queue with a reply-to header
• Sequential processing: When messages must be processed in strict order and only one consumer should handle them
• Load balancing work: When tasks should be distributed evenly across workers, not broadcast to all
• Simple task queues: When you just need work distribution without fan-out
• Strong consistency requirements: When all consumers must process before the operation is considered complete (pub/sub is eventually consistent)

Production Failure Scenarios

Failure	Impact	Mitigation
Broker goes down	All publishing and subscribing stops	Deploy broker clusters with replication; use multiple broker endpoints
Subscriber crash during processing	Message may be lost or reprocessed	Use manual acknowledgment; configure prefetch limits
Message explosion (too many subscribers)	Network saturation, increased latency	Limit subscription count; use content filtering at broker
Topic explosion (too many topics)	Memory pressure on broker	Implement topic naming policies; use topic hierarchies instead of flat topics
Subscription drift	Consumers receiving unintended messages	Use filter policies; regular audit of subscriptions
Ordering violations	Messages arrive out of order across topics	Use correlation IDs to reorder; use single-partition streams for ordering
Poison message blocking	Failed messages block subscriber	Configure dead letter queues; set retry limits with exponential backoff

Observability Checklist

Metrics to Monitor

• Topic message rate: Messages published per second per topic
• Subscription delivery rate: Messages delivered per second per subscription
• Subscriber lag: Time between message publish and delivery to subscribers
• Dead letter queue depth: Failed messages awaiting inspection
• Filter match rate: Percentage of published messages matching subscriber filters
• Connection count: Active publishers and subscribers per topic

Logs to Capture

• Topic creation and deletion events
• Subscription creation, modification, and deletion
• Message publish events with topic, routing keys, and headers
• Message delivery events per subscription
• Dead letter queue arrivals with original topic and failure reason
• Filter policy evaluations (when a message is filtered out)

Alerts to Configure

• Dead letter queue depth exceeds threshold
• Subscriber delivery failures exceed rate threshold
• Topic message rate anomalies (spike or drop)
• Subscription lag exceeds SLA threshold
• Broker memory or connection limits approaching

Security Checklist

• Authentication: Require credentials for publishers and subscribers; use mutual TLS for authentication
• Authorization: Implement topic-level access controls; restrict who can publish to which topics
• Encryption in transit: Enable TLS for all broker connections
• Encryption at rest: Enable disk encryption if broker stores messages
• Message-level security: Validate message schemas; sanitize content to prevent injection attacks
• Subscription security: Prevent unauthorized subscription creation; validate subscriber identity
• Audit trails: Log all topic and subscription administrative actions
• Data classification: Do not publish sensitive data to shared topics without encryption

Common Pitfalls / Anti-Patterns

Pitfall 1: Topic Proliferation

Creating too many topics (one per event type) leads to management overhead and broker resource exhaustion. Use hierarchical topics to group related events and apply naming policies.

Pitfall 2: Treating Pub/Sub as Synchronous Communication

Pub/sub is inherently asynchronous. If you need synchronous request/response, do not use topics. Clients must be designed to handle out-of-order, asynchronous responses.

Pitfall 3: Forgetting That Subscribers Can Go Offline

Without durable subscriptions, offline subscribers miss messages permanently. For critical subscribers, ensure durability is configured.

Pitfall 4: Not Designing for Message Replay

Subscribers that crash and restart need to reprocess messages they missed. Design consumers to handle gaps in the message stream gracefully.

Pitfall 5: Over-Filtering at the Publisher

When publishers classify messages for subscribers, they become coupled to subscriber requirements. Push filtering to subscribers and keep publishers unaware of subscriber specifics.

Pitfall 6: Ignoring Dead Letter Queue Accumulation

If DLQs are accumulating, something is wrong with message processing. Monitor DLQ depth and investigate root causes promptly.

Quick Recap

Key Points

• Pub/sub broadcasts messages to all subscribers; each subscriber receives a copy
• Topic hierarchies organize messages by domain and enable broad subscriptions
• Durable subscriptions persist messages for offline subscribers
• Shared subscriptions distribute load across multiple consumer instances
• Dead letter queues capture poison messages for debugging
• Design for at-least-once delivery with idempotent processing
• Ordering is not guaranteed across topics; use correlation IDs when needed

Pre-Deployment Checklist

- [ ] Topic naming convention documented and enforced
- [ ] Durable subscriptions enabled for critical subscribers
- [ ] Dead letter queues configured with monitoring
- [ ] Filter policies documented per subscription
- [ ] Manual acknowledgment implemented in consumers
- [ ] Idempotent message processing implemented
- [ ] Retry limits set with exponential backoff
- [ ] TLS/encryption enabled for all connections
- [ ] Topic-level access controls configured
- [ ] Alert thresholds set for DLQ depth and delivery failures
- [ ] Consumer group scaling strategy defined
- [ ] Correlation ID propagation implemented for distributed tracing

Conclusion

Pub/sub works well when you need one message to reach many consumers. Topic hierarchies organize messages by domain. Subscriptions track what each consumer wants. Filtering lets subscribers narrow their interest. Dead letter queues handle failures.

The complexity comes from ordering guarantees (or lack thereof), deduplication, and subscription management at scale. Design for at-least-once delivery and make processing idempotent.