Publish/Subscribe Patterns: Topics, Subscriptions, and Filtering

Introduction

An order service often starts by calling every downstream service itself. When inventory, billing, fraud detection, and analytics each need the same order event, those calls couple checkout to services that should be independent:

// Direct calls make checkout own every downstream dependency.
inventory.reserve(order)
billing.sendReceipt(order)
fraud.check(order)
analytics.record(order)

// With pub/sub, checkout publishes the event and subscribers react.
broker.publish("orders.placed", { order_id: order.id })

In publish/subscribe, publishers send messages to named topics and subscribers listen to the topics they need. The publisher does not need to know who is listening, and a durable subscription can retain messages while a subscriber is offline.

This guide covers topic design, subscription management, filtering, fan-out, delivery guarantees, and failure handling. It also explains when a queue or direct RPC is a simpler fit.

Topic-Specific Deep Dives

Topic Design

Topics are the core abstraction in pub/sub. Getting them right makes everything else easier.

Flat Topics

The simplest approach: one topic per message type.

user.created
order.placed
payment.processed

Each subscriber chooses which topics to listen to. Simple, but limited. What if you want all user events? You subscribe to multiple topics.

Hierarchical Topics

Hierarchies organize topics into trees, enabling broader subscriptions:

users/
users.created
users.updated
users.deleted

orders/
orders.placed
orders.updated
orders.cancelled

payments/
payments.processed
payments.failed

Subscribing to orders/ gives you all order events. Subscribing to users.* gives you all user lifecycle events.

graph TD
    Publisher -->|publish| Topic[orders.placed]

    subgraph Subscribers
        Analytics[Analytics Service]
        Notifications[Notification Service]
        Audit[Audit Service]
    end

    Topic -->|orders.placed| Analytics
    Topic -->|orders.placed| Notifications
    Topic -->|orders.placed| Audit

Naming conventions matter. Without some discipline, hierarchies turn into a mess pretty quickly.

Subject-Based vs Content-Based

Most pub/sub systems are subject-based: topics are predefined strings that publishers use. Subscribers choose topics to listen to.

Content-based systems route based on message content. A subscriber might say “give me all messages where amount > 1000.” This is more flexible but harder to implement efficiently. It also raises security concerns—what if a subscriber crafts queries to sniff data they should not see?

Most teams end up using subject-based routing. It is simpler to implement and performs well at scale.

Subscription Management

Subscriptions track what each subscriber wants. Managing their lifecycles is trickier than it sounds.

Durable Subscriptions

A durable subscription persists even when the subscriber is offline. When it reconnects, it gets the messages that arrived while it was disconnected.

// Pseudo-code for durable subscription
subscription = client.subscribe("orders.*", durable=true)
client.disconnect()
client.reconnect()
subscription.resume() // missed messages delivered

This matters for mobile clients and services that restart. Without durability, offline periods mean permanent message loss.

Shared Subscriptions

When multiple instances of a service run (say, three notification service instances), you want them to share the work. Shared subscriptions distribute messages across instances.

orders/ -> [notification-1, notification-2, notification-3]
          each instance gets roughly 1/3 of messages

This is essential for scaling consumers horizontally. Without sharing, all instances receive all messages, which defeats the purpose of running multiple workers.

Subscription Types by Delivery

• Exclusive: Only one consumer receives messages (no sharing)
• Shared: Messages distributed across multiple consumers
• Failover: One consumer receives all messages, others standby

Message Filtering

Sometimes subscribers only need a subset of what a topic publishes. Filtering lets them narrow what they receive.

Topic-Level Filtering

The simplest form: use narrow topics. A subscriber to orders.high-value gets only high-value orders, not everything else.

This requires publishers to classify messages correctly, which couples them to subscriber knowledge.

Header-Based Filtering

Messages have headers (key-value pairs) that you can use for filtering:

{
  "topic": "orders",
  "headers": {
    "priority": "high",
    "region": "us-west",
    "source": "web"
  },
  "body": { "order_id": "12345", "amount": 5000 }
}

A subscriber can filter: headers.region = 'us-west' AND headers.priority = 'high'.

This separates classification (publisher job) from filtering (subscriber job).

SQL-Based Filtering

Some systems let subscribers express filters as SQL-like conditions:

SELECT * FROM orders WHERE amount > 1000 AND region = 'us-west'

Amazon SNS supports SQL-based filtering:

{
  "FilterPolicy": {
    "amount": [{ "numeric": [">", 1000] }],
    "region": ["us-west"]
  }
}

Fan-Out Patterns

Fan-out describes how one message reaches multiple subscribers. The pattern changes depending on what you need.

Broadcast Fan-Out

Every subscriber receives every message. The simplest model.

graph LR
    Pub[Publisher] -->|1 message| Topic
    Topic -->|copy 1| Sub1[Subscriber 1]
    Topic -->|copy 2| Sub2[Subscriber 2]
    Topic -->|copy 3| Sub3[Subscriber 3]

The broker makes copies. Cost scales with subscriber count.

Selective Fan-Out

With filtering, only matching subscribers receive the message. More efficient but requires the broker to evaluate filters.

Dead Letter Queues

When a subscriber cannot process a message (crashes, returns error, times out), what happens? Dead letter queues capture failed messages so you can inspect them later.

graph LR
    Pub[Publisher] --> Topic
    Topic -->|normal| Sub[Subscriber]
    Topic -->|failed after retries| DLQ[Dead Letter Queue]

Without DLQs, poison messages block the queue or get dropped silently. DLQs let you see what failed and why.

Implementation Considerations

Picking the right topic design and subscription model is only half the work. Real pub/sub systems face practical challenges once messages start flowing — messages arrive out of order, consumers fall behind, duplicates sneak through, and delivery guarantees need to match what your processing model can handle. Each consideration below shapes how you configure brokers, write consumers, and plan for failure.

Message Ordering

Most pub/sub systems do not guarantee ordering across topics. Within a single ordered stream (like Kafka partitions), ordering holds. Across multiple publishers or topics, it does not.

If you need global ordering, you need a totally ordered broadcast protocol, which is expensive. More commonly, you accept per-partition ordering and use correlation IDs to reconstruct order client-side.

Backpressure Handling

When subscribers cannot keep up with message throughput, backpressure prevents the system from collapsing. Without it, consumers run out of memory or broker queues grow without bound.

Buffer-based approaches accumulate messages locally up to a limit:

local_buffer = RingBuffer(capacity=1000)
while local_buffer.has_capacity():
    message = broker.fetch()
    local_buffer.push(message)

process_batch(local_buffer.drain())

When the buffer fills, the subscriber must either drop messages, pause consumption, or signal the broker to slow down.

Flow control signals let subscribers push back on publishers:

subscriber --> [Window Full] --> broker --> publisher slows production

RabbitMQ uses basic.qos with prefetch to limit unacknowledged messages. Kafka has consumer group rebalance hooks for similar behavior.

Exponential backoff on consumption failures keeps repeated processing failures from creating message pileup:

retry_delay = 1_000  # ms
for attempt in range(max_retries):
    try:
        process(message)
        ack(message)
        break
    except TransientError:
        sleep(retry_delay)
        retry_delay *= 2
        retry_delay = min(retry_delay, max_delay)

The key takeaway: backpressure should propagate upward, not accumulate silently. If your subscribers cannot keep up, the system should throttle publishers or shed load, not buffer forever.

Message Deduplication

At-least-once delivery can deliver the same message twice. Idempotent consumers handle this by tracking processed message IDs.

if message.id in processed_ids:
    skip
else:
    process(message)
    add message.id to processed_ids

Design for at-least-once and make processing idempotent. Exactly-once is usually not worth the cost.

Delivery Guarantees Deep Dive

Pub/sub systems offer different delivery guarantees. The right choice depends on your processing model.

Guarantee	Description	Idempotent Required	Use Case
At-most-once	Message delivered once or not at all	No	Metrics, fire-and-forget events
At-least-once	Message delivered, may be duplicated	Yes	Processing jobs, state updates
Exactly-once	Message delivered exactly once	Yes (and coord.)	Payments, critical business events

Most broker systems (RabbitMQ, Amazon SNS/SQS, Google Cloud Pub/Sub) provide at-least-once by default. Exactly-once requires distributed consensus and adds significant overhead.

When designing consumers, assume at-least-once arrival. Make every processing step idempotent using message IDs stored in a deduplication table or bloom filter.

Pub/sub connects closely to event-driven architecture, where topics carry domain events between services. It also relates to message queue types for understanding where pub/sub fits in the broader messaging landscape.

Trade-off Analysis

Design Choice	Pros	Cons	When to Use
Flat Topics	Simple, easy to understand	No wildcard subscriptions, subscribers must list each topic	Small number of independent topics
Hierarchical Topics	Wildcard subscriptions, natural organization	Naming discipline required, can become messy	Large topic spaces with natural groupings
Subject-Based Routing	Efficient, simple broker logic	Limited flexibility	Most production systems
Content-Based Routing	Maximum flexibility for subscribers	Complex broker logic, security concerns	Specialized filtering requirements
Durable Subscriptions	No message loss during offline periods	Higher broker memory/storage	Mobile clients, fault-tolerant services
Shared Subscriptions	Horizontal scaling, efficient resource use	Message ordering not guaranteed per instance	Stateless workers, scaled consumers
Exclusive Subscriptions	Strict ordering guarantee	No horizontal scaling	Ordered processing, single consumer
Broadcast Fan-Out	Simple, all subscribers get all messages	Scales with subscriber count	Notifications, event broadcasting
Selective Fan-Out	Efficient, subscribers get only relevant messages	Broker must evaluate filters	Targeted deliveries, filtered streams

Delivery Guarantee	Message Loss Risk	Duplicate Risk	Implementation Complexity
At-most-once	Possible (message may not arrive)	None	Low
At-least-once	None	Possible (duplicates)	Medium (idempotency needed)
Exactly-once	None	None	High (distributed consensus)

When to Use / When Not to Use

When to Use Publish-Subscribe

Pub/sub is for those one-to-many situations. One event, multiple consumers, all independent of each other. Fits naturally when you notice your code calling three different services after a single operation and each one is doing something unrelated.

Where it fits:

Scenario	Why Pub/Sub Fits	Example
Event broadcasting	One message triggers several downstream actions	order.placed → inventory update, receipt generation, fraud check, analytics push
Cross-service data sync	Same data ends up in multiple systems	Profile update → refresh search index, invalidate cache, push to warehouse
Webhook delivery	External services subscribe to internal events	Payment complete → notify partners, update dashboards, fire Slack alerts
IoT ingestion	Many devices, many consumers of that stream	Sensor readings → real-time dashboard, anomaly detection, long-term archive
Audit logging	Every action recorded without coupling to business logic	Admin action → audit trail, suspicious-activity alert

The tell: you have a service that currently calls three downstream services in sequence after one database write, and none of them have anything to do with the original operation. The order service should not know about your notification system or your analytics pipeline. Pub/sub lets you drop that coupling — publish order.placed and move on. What happens downstream is someone else’s problem.

When Not to Use Publish-Subscribe

Pub/sub breaks down when your use case needs something it was never designed to provide. The decoupling that makes it powerful works against you the moment you need to track outcomes or enforce sequence.

The clearest misfit is request-response. When service A calls service B and needs a result before continuing, topics do not help. A publishes and moves on without knowing whether B received anything or what it did. For synchronous calls, use RPC (gRPC, REST) or a queue with a reply-to header.

Global ordering is the second common mismatch. Most brokers do not guarantee ordering across topics or publishers. Getting strict total ordering requires a consensus protocol, which adds real overhead. Most systems can live with per-partition ordering plus correlation IDs to reorder on the client side. If you genuinely need global sequence — sequence number generators, distributed locks — reach for a dedicated log system instead.

Work distribution sounds like a good fit for topics but is simpler with a queue. Topics broadcast to all subscribers by default. Without filtering, every worker gets every message, and you end up building de-duplication and load balancing on top of pub/sub when a queue with competing consumers gives you that for free.

The trickiest case is synchronous choreography. When the order service needs to wait until inventory, fraud check, and receipt generation all finish before it can confirm the order, you are building a synchronous workflow on an asynchronous backbone. A saga pattern or a direct call chain fits that better than topics.

Bottom line: pub/sub assumes producers and consumers are independent and the producer does not care what happens downstream. The moment you need to track outcomes, know who handled something, or enforce sequence, use a different pattern.

Production Failure Scenarios

Failure	Impact	Mitigation
Broker goes down	All publishing and subscribing stops	Deploy broker clusters with replication; use multiple broker endpoints
Subscriber crash during processing	Message may be lost or reprocessed	Use manual acknowledgment; configure prefetch limits
Message explosion (too many subscribers)	Network saturation, increased latency	Limit subscription count; use content filtering at broker
Topic explosion (too many topics)	Memory pressure on broker	Implement topic naming policies; use topic hierarchies instead of flat topics
Subscription drift	Consumers receiving unintended messages	Use filter policies; audit subscriptions regularly
Ordering violations	Messages arrive out of order across topics	Use correlation IDs to reorder; use single-partition streams for ordering
Poison message blocking	Failed messages block subscriber	Configure dead letter queues; set retry limits with exponential backoff

Common Pitfalls / Anti-Patterns

Common Pitfalls

Even experienced teams make the same mistakes when adopting pub/sub. These anti-patterns show up in production repeatedly — topic sprawl that overwhelms the broker, treating asynchronous messaging like synchronous RPC, ignoring what happens when subscribers go offline, and forgetting about poison messages that silently pile up. Watch for these during design reviews and post-incident analysis.

Pitfall 1: Topic Proliferation

Creating too many topics (one per event type) leads to management overhead and broker resource exhaustion. Use hierarchical topics to group related events and apply naming policies.

Pitfall 2: Treating Pub/Sub as Synchronous Communication

Pub/sub is inherently asynchronous. If you need synchronous request/response, do not use topics. Clients must be designed to handle out-of-order, asynchronous responses.

Pitfall 3: Forgetting That Subscribers Can Go Offline

Without durable subscriptions, offline subscribers miss messages permanently. For critical subscribers, make sure durability is configured.

Pitfall 4: Not Designing for Message Replay

Subscribers that crash and restart need to reprocess messages they missed. Design consumers to handle gaps in the message stream gracefully.

Pitfall 5: Over-Filtering at the Publisher

When publishers classify messages for subscribers, they become coupled to subscriber requirements. Push filtering to subscribers and keep publishers unaware of subscriber specifics.

Pitfall 6: Ignoring Dead Letter Queue Accumulation

If DLQs are accumulating, something is wrong with message processing. Monitor DLQ depth and investigate root causes promptly.

Observability Checklist

Track pub/sub health per topic and subscription. A healthy broker can still hide a subscriber that has stopped making progress, so compare published, delivered, and acknowledged messages rather than watching broker uptime alone.

• Measure publish, delivery, and acknowledgment rates for each subscription.
• Track subscriber lag as both message distance and the age of the oldest unprocessed event where the broker supports it.
• Count delivery failures, retries, redeliveries, and dead letter queue arrivals; break them down by subscription.
• Record filter match and discard rates so a changed policy does not silently stop useful deliveries.
• Monitor active connections, consumer rebalances, broker capacity, and publish-to-delivery latency.
• Put a trace or correlation ID on each event and carry it through consumer logs. Log topic and subscription metadata, not message bodies or secrets.
• Alert on lag against each consumer’s service objective, sustained delivery failures, unusual publish-rate changes, and growing dead letter queues.

Avoid using an unbounded tenant ID as a metrics label. Keep per-tenant diagnosis in access-controlled logs or traces, and use bounded dimensions for fleet-wide dashboards.

Security and Compliance Notes

Treat topic access as a data boundary. Authenticate publishers, subscribers, and administrators, then grant each identity only the actions and topics it needs. A service in one tenant should not be able to publish to or subscribe to another tenant’s topics, even when wildcard subscriptions or content filters are available.

• Enforce topic-level publish, subscribe, and administration permissions; review wildcard patterns and filter expressions server-side.
• Separate tenant topics or enforce tenant identity in broker authorization rules. Do not rely on a client-supplied tenant header alone.
• Put only routing metadata in topic names and headers. Keep personal data, credentials, and payment details out of them.
• Minimize event payloads. Use an authorized reference to fetch sensitive data when subscribers do not all need the full record.
• Encrypt broker connections and stored messages. Use field-level encryption when broker operators or unrelated subscribers must not read particular fields.
• Validate event schemas and size limits before accepting messages; treat payloads and filter values as untrusted input.
• Keep payloads out of logs, traces, and dead letter queue dashboards. Redact sensitive fields before recording failure details.
• Restrict and audit topic creation, subscription changes, replay access, and dead letter queue inspection. Set retention and deletion rules for stored messages.

For regulated data, map each topic and subscription to its data classification, retention period, and authorized consumers before enabling broad fan-out.

Quick Recap Checklist

• Use a documented topic naming scheme that groups related events without creating unnecessary topics.
• Choose durable subscriptions for consumers that must receive messages published while they are offline.
• Use shared subscriptions when multiple consumer instances should divide the work.
• Assume at-least-once delivery can produce duplicates, and make message processing idempotent.
• Set retry limits and route messages that keep failing to a monitored dead letter queue.
• Track subscriber lag, delivery failures, and dead letter queue depth.

Interview Questions

Further Reading

• Event-Driven Architecture - Companion pattern for building event-driven systems
• Message Queue Types - Understanding where pub/sub fits in the broader messaging landscape
• Amazon SNS Documentation - AWS’s pub/sub service deep dive
• RabbitMQ Routing Topics - RabbitMQ’s official tutorial on topic routing patterns

Conclusion

Publish/subscribe lets one publisher send an event to several independent subscribers through a topic. Topic structure, subscription durability, and filtering determine who receives each message and how the system behaves when consumers fall behind or go offline. Most systems should plan for at-least-once delivery, make consumers idempotent, and monitor retries and dead letter queues. Pub/sub works best when the publisher can continue without waiting for a result from each subscriber.

Pre-Deployment Checklist

- [ ] Topic naming convention documented and enforced
- [ ] Durable subscriptions enabled for critical subscribers
- [ ] Dead letter queues configured with monitoring
- [ ] Filter policies documented per subscription
- [ ] Manual acknowledgment implemented in consumers
- [ ] Idempotent message processing implemented
- [ ] Retry limits set with exponential backoff
- [ ] TLS/encryption enabled for all connections
- [ ] Topic-level access controls configured
- [ ] Alert thresholds set for DLQ depth and delivery failures
- [ ] Consumer group scaling strategy defined
- [ ] Correlation ID propagation implemented for distributed tracing