Message Queues

Introduction

Pipes work well for streaming data between processes, but both ends need to be open at the same time. If a producer outpaces its consumer or the consumer is offline, a pipe does not provide durable storage for pending messages. Message queues offer asynchronous, message-oriented communication that can hold work until a receiver is ready.

This article covers POSIX and System V message queues, including priorities, kernel behavior, synchronization, cleanup, and failure handling. It also explains when queues are a fit and when sockets, pipes, shared memory, or a broker are more appropriate.

When to Use / When Not to Use

Use message queues when:

• You need asynchronous communication — sender and receiver do not need to run simultaneously
• You want message boundaries preserved (discrete, typed messages)
• You need message priorities (urgent messages jump ahead of normal ones)
• You need messages to wait in a kernel queue while the receiver is temporarily unavailable (until the queue is removed or the system reboots)
• You are implementing a producer-consumer or work queue pattern
• You want to decouple processing stages in a pipeline

Do not use message queues when:

• You need bidirectional communication (use sockets instead)
• You need high-throughput streaming (pipes or shared memory are faster)
• You need total ordering of messages with no priority (use a pipe with proper framing)
• You are communicating between machines (use network sockets)
• You need transactional message delivery with acknowledgments (use a message broker like Kafka or RabbitMQ)

Architecture or Flow Diagram

POSIX Message Queue Flow

graph TD
    A[Process A calls mq_open] --> B[Kernel creates/opens queue at path]
    B --> C[Process A calls mq_send<br/>Kernel copies message to queue buffer]
    C --> D[Message stored in kernel queue<br/>with priority and timestamp]
    E[Process B calls mq_receive<br/>Kernel copies message to Process B's buffer]
    D --> E
    E --> F[Kernel removes message<br/>from queue after successful receive]

System V Message Queue Flow

graph TD
    G[Process A calls msgget with key] --> H[Kernel creates/retrieves<br/>System V queue by key]
    H --> I[Process A calls msgsnd<br/>Kernel enqueues message with type]
    I --> J[Kernel maintains message list<br/>ordered by message type]
    K[Process B calls msgrcv<br/>Kernel dequeues by type matching]
    J --> K
    K --> L[Kernel copies message<br/>and removes from queue]

Core Concepts

POSIX Message Queues

POSIX message queues use a filesystem-like API. Open a queue with mq_open(), send messages with mq_send(), and receive them with mq_receive():

#include <mqueue.h>
#include <stdio.h>
#include <string.h>

// Open or create a message queue with a known maximum message size
struct mq_attr attr = { .mq_maxmsg = 10, .mq_msgsize = 1024 };
mqd_t mq = mq_open("/my_queue", O_CREAT | O_RDWR, 0666, &attr);
if (mq == (mqd_t)-1) {
    perror("mq_open");
}

// Send a message
const char *msg = "Hello, queue!";
mq_send(mq, msg, strlen(msg), 0);  // priority = 0

// Receive a message
char buf[1025];
unsigned int prio;
ssize_t len = mq_receive(mq, buf, sizeof(buf) - 1, &prio);
if (len >= 0) {
    buf[len] = '\0';
    printf("Received (prio %u): %s\n", prio, buf);
}

mq_close(mq);
mq_unlink("/my_queue");  // Clean up when done

Key attributes of POSIX message queues:

• Path-based: Queues are identified by filesystem paths (must start with /)
• Priority: Messages have a priority (0-lower to higher), higher priority messages are delivered first
• Notification: Queues can notify processes when they become non-empty via signals or thread notification
• Size limits: Each queue has a max message size and max number of messages (queryable via mq_getattr())

System V Message Queues

System V message queues use integer keys and queue IDs. IPC_PRIVATE is a special key value that asks msgget() to create a new queue; it does not itself provide a key other processes can look up:

#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/msg.h>

// Create or get a queue by key
int msgid = msgget(IPC_PRIVATE, 0666);
if (msgid == -1) {
    perror("msgget");
}

// Message structure (must start with long mtype)
struct msgbuf {
    long mtype;       // Message type (must be > 0)
    char mtext[256]; // Message payload
};

struct msgbuf msg;
msg.mtype = 1;
strcpy(msg.mtext, "Hello from System V");

// Send message
if (msgsnd(msgid, &msg, sizeof(msg.mtext), 0) == -1) {
    perror("msgsnd");
}

// Receive message (mtype filter - 0 means any)
if (msgrcv(msgid, &msg, sizeof(msg.mtext), 0, 0) == -1) {
    perror("msgrcv");
}

printf("Got: %s\n", msg.mtext);

// Clean up
msgctl(msgid, IPC_RMID, NULL);

Key attributes of System V message queues:

• Key-based: Queues are obtained by integer key; IPC_PRIVATE creates a new queue that must be shared by passing its queue ID
• Message typing: Each message has a type field used for selective receive
• Operations: Full set of control operations via msgctl() including removal, statistics
• Legacy: More portable across different Unix variants, but API feels dated

Kernel Implementation

At a high level, the kernel tracks queue attributes, message payloads, and processes waiting for space or data. The exact data structures differ by operating system and implementation, so a linked-list description should not be treated as a universal layout. Linux System V queues and POSIX queues also use different internal code paths.

Sending copies the payload from user space into kernel-managed storage, and receiving copies it back to the caller. A receive removes one message from the queue; it does not confirm that the application finished processing it. Processes waiting on an empty or full queue sleep until the relevant queue state changes, then become eligible to run again.

Each message of size N generally requires copying its payload into and out of kernel-managed storage, along with system-call overhead. For high-throughput workloads, measure this cost against shared memory, which avoids per-message payload copies but requires explicit synchronization. Shared memory sidesteps the copy entirely by letting processes access the same physical pages directly, at the cost of more involved synchronization logic.

Production Failure Scenarios

Queue Full — Senders Block

When a message queue reaches its maximum capacity (either max bytes or max messages), a sender blocks until a receiver removes messages. In a high-throughput system with slow consumers, producers can accumulate in the kernel wait queue indefinitely.

The blocking behavior can cause cascading latency in producer processes. For a blocking mq_send() call on a full queue, the calling process waits until space becomes available. The process sleeps until the receive side drains enough messages to make room. Under sustained overload, producers can block for seconds or longer.

Mitigation: Use nonblocking sends (O_NONBLOCK for POSIX queues or IPC_NOWAIT for System V), handle failures, and apply backpressure at the application level. POSIX timed sends use mq_timedsend() with an absolute timeout. Queue attribute snapshots can help monitoring, but do not guarantee that a subsequent send will succeed. In multi-threaded producers, consider a bounded thread pool where senders drop work rather than block when the queue is full — this prevents head-of-line blocking for urgent messages.

Queue capacity depends on both message-count and byte limits. POSIX queues use mq_maxmsg and mq_msgsize; Linux also applies system-wide limits such as /proc/sys/fs/mqueue/msg_max, msgsize_max, and queues_max, plus per-user resource limits. System V queues use msg_qbytes for the byte limit on an individual queue, while system settings such as msgmni and msgmax constrain queue count and message size. A full queue blocks a sender unless nonblocking mode is used.

What surprises developers is that a queue with many small messages can become full even when the total byte count is low, and vice versa. A queue with mq_maxmsg = 100 and mq_msgsize = 1024 holds at most 100 messages even if the total byte count is far below the limit. Conversely, a queue with MSGMNB = 16384 can fill up with just a handful of oversized messages. Always check both limits when debugging queue-full behavior.

For POSIX queues, mq_getattr() reports mq_curmsgs and mq_maxmsg. For System V, msgctl(msgid, IPC_STAT, &buf) reports msg_qbytes (the byte limit), msg_cbytes (Linux current-byte count), and msg_qnum (current message count). These snapshots can support monitoring, but they are not reservations: another sender can fill the queue before the next send. Handle the send result directly and treat EAGAIN or blocking as normal overload signals.

Queue Overflow — Messages Lost

System V message queues have limits on message size and bytes per queue. When a queue reaches capacity, the behavior depends on the flags passed to the send operation. With blocking sends (the default), the sender waits until space becomes available instead of dropping the message because of queue capacity. A successful send still does not protect the message from queue removal or system reboot. With IPC_NOWAIT set, msgsnd() returns -1 and sets errno=EAGAIN immediately when the queue is full; the message is not queued and the caller must handle the failure.

A successful send only places the message in a kernel queue; it does not make the message durable across reboot or guarantee that a consumer processed it. An unconsumed message remains available while the queue exists, but is lost if the queue is removed or the system reboots. If a non-blocking sender gets EAGAIN and the application does not retry or record the failure, the message is lost from the application’s perspective.

Mitigation: Monitor queue fill levels, implement retry logic with exponential backoff, use larger queue sizes, or switch to a persistent message broker. Also implement application-level acknowledgment: have the consumer send a confirmation back on a response queue after processing, so the producer knows the message was handled. This lets the producer detect and recover from silent losses.

The distinction between blocking and non-blocking behavior matters for design. Blocking sends are safe — you never lose a message — but they make your producer vulnerable to latency spikes when the consumer is slow. Non-blocking sends are safe from stalling, but they require explicit handling of the EAGAIN case. The common mistake is using non-blocking sends without that handling, which silently drops messages. If you use IPC_NOWAIT, treat EAGAIN as a first-class error: log it, retry with backoff, and consider whether the message should be written to a fallback store.

For applications that need guaranteed delivery even under load, pair a message queue with a disk-backed retry log. When msgsnd() returns EAGAIN, write the message to a local file or a backup queue. When the primary queue becomes available, drain the retry log first before accepting new messages. This gives you non-blocking sends in the normal case and durability when the queue backs up.

Permission Issues on Queue Access

POSIX message queues are accessed via filesystem paths under /dev/mqueue/. The kernel creates virtual files there for each queue, and standard Unix file permissions govern access. Several error codes can surface from permission mismatches: EACCES when the calling process lacks read or write permission on the queue; ENOENT when the queue path does not exist (queue not created yet or already unlinked); EMFILE when the process has exhausted its file descriptor limit; ENFILE when the system has hit its total file descriptor limit.

The EACCES case is especially tricky in multi-process scenarios. Process A creates the queue with 0666 permissions, but if the directory /dev/mqueue/ itself has restrictive permissions (common on hardened systems), Process B cannot traverse the directory to reach the queue file. Process B then gets EACCES even though the queue permissions are wide open.

System V queues can be inspected with tools such as ipcs -q, subject to the system and IPC namespace. Knowing a queue ID does not bypass its permission checks. Changing ownership, permissions, or queue limits with IPC_SET, and removing a queue with IPC_RMID, requires the caller to be the owner or creator, or to have the required privilege. Ordinary send and receive access is governed by the queue permission bits.

Mitigation: Create queues in directories with proper permissions (e.g., /var/run/myapp/), use consistent permission modes (0666 or 0660 with appropriate group), ensure the directory itself is traversable by all processes that need access, and handle EACCES, ENOENT, EMFILE, and ENFILE explicitly in error handling. For System V, restrict access to /proc/sysvipc/ in containerized environments and use msgctl() to set appropriate permission bits immediately after msgget().

The directory traversal problem is the most frequently missed part. POSIX queue permissions have two layers: the queue file itself and the directory containing it. A queue created with mode 0666 in a directory with mode 0700 is inaccessible to anyone other than the directory owner. The queue file permissions are irrelevant because no process can open the directory to reach the queue. This catches people who harden /dev/mqueue/ thinking they are improving security while breaking their own queue access. Always check both the queue path permissions and the directory permissions when debugging EACCES on POSIX queues.

For multi-process access, choose queue permissions deliberately and ensure intended processes can open the queue. POSIX queue names live in the mounted mqueue filesystem when available; queue access is controlled by the queue permissions and the mount and namespace configuration.

Message Queue Key Conflicts (System V)

System V queues are identified by integer keys of type key_t. If two unrelated applications use the same key for different queues, they may accidentally share a queue or encounter EEXIST on creation. IPC_PRIVATE generates a unique key but then requires some external mechanism to share the queue ID with other processes.

The ftok() function can derive a key from a path and project identifier, but it does not guarantee uniqueness:

#include <sys/types.h>
#include <sys/ipc.h>

// Generate a key from a path and project ID character
key_t key = ftok("/var/run/myapp/queue_dir", 'M');
if (key == (key_t)-1) {
    perror("ftok failed — path does not exist");
    exit(1);
}

int msgid = msgget(key, IPC_CREAT | IPC_EXCL | 0666);
if (msgid == -1 && errno == EEXIST) {
    // Queue already exists — try to open it instead
    msgid = msgget(key, 0666);
}

ftok() combines parts of the path’s device and inode numbers with the project ID character. If the path is recreated or its filesystem metadata changes, it may produce a different key. Distinct paths can also collide because the function uses only selected bits of those values. Treat the result as a convenient lookup key, not a globally unique identifier.

Collisions also occur when different applications choose overlapping project IDs on shared systems. The project ID is a single byte, so only 256 distinct values exist. Pick a path specific to your application (a directory that persists) and a unique character from the application name to keep collisions unlikely.

Mitigation: Use a consistent key generation scheme (e.g., ftok() with a known project ID and path), use IPC_CREAT | IPC_EXCL to detect collisions, or use POSIX queues which use path-based discovery. For production systems where queue sharing across processes is needed, store the queue ID in a state file rather than calling ftok() every time the application restarts.

Orphaned Queues After Process Crash

If a process that created a queue terminates without calling mq_unlink() (POSIX) or msgctl(IPC_RMID) (System V), the queue persists in the kernel until the system is rebooted or an administrator removes it manually. For POSIX queues, the queue survives even after all processes close their file descriptors — it remains in /dev/mqueue/ until explicitly unlinked. For System V queues, the queue is marked for deletion when IPC_RMID is called, but lingers until the last process detaches.

The orphaning behavior is a double-edged sword. If a producer crashes mid-execution, queued messages survive and are available to the consumer when it restarts — useful for recovery. But if the crash scrambles the application’s sense of which queues it owns (it forgets, or creates new ones instead of reopening existing ones), the old queues pile up and consume kernel memory.

Orphaned queues are visible via ipcs -q for System V and ls /dev/mqueue/ for POSIX queues. Each orphaned queue consumes a small amount of kernel memory for its queue descriptor and any remaining messages. Under heavy churn — many crashed applications each leaving queues behind — this can add up.

System V queues created with IPC_PRIVATE have no reusable lookup key. The creating process must pass the returned queue ID to other processes; if it crashes before doing that, the queue can be difficult to identify and remove.

Mitigation: Use mq_getattr() to check queue state at application startup, implement a cleanup mechanism that scans for and removes queues created by previous instances, use wrapper scripts or daemons to clean up stale queues, and monitor for orphaned queues in production. For System V, add a startup step that runs ipcrm -q <id> on any stale queues tied to your application, filtering ipcs -q output by creation time or owner. For POSIX queues, track queue names in a state file so a restart handler can unlink old queues before creating new ones.

Trade-off Table

Implementation Snippet(s)

C: POSIX Message Queue with Notification

#include <mqueue.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <unistd.h>

static mqd_t g_mq;

static void drain_queue(union sigval value) {
    mqd_t mq = *(mqd_t *)value.sival_ptr;
    char buf[1025];
    unsigned int prio;
    ssize_t len;

    // Register again before draining so a later empty-to-nonempty transition is noticed.
    struct sigevent sev = {0};
    sev.sigev_notify = SIGEV_THREAD;
    sev.sigev_notify_function = drain_queue;
    sev.sigev_value.sival_ptr = &g_mq;
    if (mq_notify(mq, &sev) == -1) {
        perror("mq_notify");
        return;
    }

    while ((len = mq_receive(mq, buf, sizeof(buf) - 1, &prio)) >= 0) {
        buf[len] = '\0';
        printf("Notification received (prio %u): %s\n", prio, buf);
    }
}

int main(void) {
    struct mq_attr attr = {
        .mq_flags = 0,
        .mq_maxmsg = 10,
        .mq_msgsize = 1024,
        .mq_curmsgs = 0,
    };

    g_mq = mq_open("/demo_queue", O_CREAT | O_RDWR, 0666, &attr);
    if (g_mq == (mqd_t)-1) {
        perror("mq_open");
        return EXIT_FAILURE;
    }

    struct sigevent sev = {0};
    sev.sigev_notify = SIGEV_THREAD;
    sev.sigev_notify_function = drain_queue;
    sev.sigev_value.sival_ptr = &g_mq;
    if (mq_notify(g_mq, &sev) == -1) {
        perror("mq_notify");
        mq_close(g_mq);
        mq_unlink("/demo_queue");
        return EXIT_FAILURE;
    }

    for (;;) {
        pause();
    }
}

Python: POSIX Message Queue (via POSIX module)

import os
import mmap
import struct

# Note: Python's standard library doesn't have native POSIX mq support
# For production use, consider the posix_ipc module or use System V msg queues

# Example using a workaround with pipes for simple cases:
import subprocess
import signal

# Simpler approach: use a pipe with proper framing
pipe_path = "/tmp/demo_pipe"
os.mkfifo(pipe_path)

# For actual message queues in Python, use the 'posix_ipc' module:
# import posix_ipc
# mq = posix_ipc.MessageQueue("/demo_queue", posix_ipc.O_CREAT)
# mq.send("Hello", priority=5)
# msg, prio = mq.receive()

Bash: System V Message Queue Basics

# Create a System V message queue
# Using ipcs to inspect current queues
ipcs -q

# C program needed for actual usage:
# The key is generated with ftok()
# msgget(ftok("/some/path", 'A'), IPC_CREAT | 0666)

# Cleanup orphaned queues
ipcrm -q <msgid>

# Monitor queue statistics
ipcs -q -i <msgid>

Observability Checklist

• Queue existence and attributes: Use ipcs -q to list all System V message queues and their parameters
• Queue fill level: Check current number of messages and total bytes with ipcs -q or mq_getattr()
• Blocking processes: Check which processes are blocked on send/receive with ps aux | grep msgsnd/msgrcv or strace
• Queue limits: Check system-wide limits in /proc/sys/kernel/msg* parameters (System V) or mq_open() will fail with EMFILE if process limit reached
• Notification delivery: Verify signal handlers are properly installed for POSIX queue notifications
• Resource leaks: Monitor that queues are being properly closed/unlinked — orphaned queues accumulate over time
• strace/dtrace: strace -e trace=msgsnd,msgrcv,msgget,msgctl -p <pid> for System V queue operations

Common Pitfalls / Anti-Patterns

Queue Permissions & Security: Message queues respect standard Unix permissions — any user can read/write queues they have access to. Use 0660 or 0664 when creating queues to restrict access. In shared hosting environments, other users can potentially discover queue identifiers if they have read access to /proc/sysvipc/. Messages stored in kernel queues are readable by any process with appropriate permissions, so do not store sensitive data (passwords, tokens, PII) without encryption.

Queue Limits as DoS Vector: An attacker could fill all message queues (system-wide limits on total bytes and number of queues) by sending messages to public queues. Monitor /proc/sys/kernel/msgmnb, /proc/sys/kernel/msgmni, and implement proper input validation.

Audit Requirements: Message queue operations (creation, send, receive) may not generate standard filesystem audit events. Consider application-level logging for compliance requirements.

1. Not handling EAGAIN on non-blocking send — when the queue is full and IPC_NOWAIT/MQ_DONTWAIT is set, msgsnd/mq_send returns -1 with errno=EAGAIN. Ignoring this causes message loss.

2. Confusing message type with priority — System V mtype is a filter for selective receive (receive only messages of type N), not a priority. POSIX message queues have explicit priority levels.

3. Forgetting to unlink queues — POSIX queues persist until explicitly unlinked, even after all processes close them. This causes resource leaks. Always call mq_unlink() during cleanup.

4. Reading with wrong buffer size — mq_receive() returns the message length. If your buffer is smaller than mq_msgsize, the message is truncated and the call fails with EMSGSIZE. Always allocate buffers large enough for mq_msgsize.

5. Not accounting for message ordering under load — with priority queues, a steady stream of high-priority messages can starve low-priority ones. Design your priority scheme carefully.

6. Mixing POSIX and System V for the same queue — they are completely separate implementations and do not interoperate. Pick one and stick with it.

7. Assuming queue persistence across reboots — System V message queues are kernel memory and do not survive system reboots. For persistence, use a disk-backed message broker.

Quick Recap Checklist

• Message queues preserve message boundaries and support priority-based delivery
• POSIX message queues use path-based API (mq_open/send/receive/close/unlink); System V uses key-based API (msgget/msgsnd/msgrcv/msgctl)
• Messages remain in the kernel queue until received, removed, or lost on reboot; the sender does not need to wait for a receiver
• Both implementations have kernel-enforced limits on queue size and message count
• Use non-blocking mode (IPC_NOWAIT/MQ_DONTWAIT) to avoid blocking when queue is full
• Always clean up with mq_unlink() or msgctl(IPC_RMID) to prevent orphaned queues
• Message queue notifications can be delivered via signals (POSIX) or polled manually
• For production message passing with persistence, durability, and clustering, consider dedicated message brokers (Kafka, RabbitMQ, NATS)

Interview Questions

Further Reading

• mq_overview(7) — Linux man page — POSIX message queues overview
• mq_open(3) — Linux man page — POSIX message queue API reference
• msgop(2) — Linux man page — System V message queue operations
• mq_notify(3) — Linux man page — POSIX queue notification mechanism
• Shared Memory — zero-copy IPC mechanism — Memory-mapped IPC with direct physical memory access
• Pipes & Named Pipes (FIFO) — unidirectional IPC — Anonymous and named pipe communication

Conclusion

Message queues fill the gap between pipes (streaming, connection-oriented) and shared memory (high-speed, complex to synchronize). Their asynchronous delivery lets producers and consumers operate at different times while the kernel queue remains available. Both POSIX and System V implementations remain useful for local IPC: POSIX offers notification mechanisms and priority ordering, while System V supports selective receive by message type. Their messages remain in kernel-managed queues only until received, removed, or lost on reboot; neither API provides durable delivery or processing acknowledgments.

At scale, message queues evolve into full message brokers (Kafka, RabbitMQ, NATS) that provide persistence, durability, routing, and clustering. Understanding the kernel-level fundamentals of POSIX and System V message queues makes these higher-level systems comprehensible because you grasp what the middleware is abstracting away.

For continued learning, explore how ZeroMQ or Nanomsg provide socket-like APIs over various transport mechanisms including in-process, inter-process, and network, and study the design of durable message brokers that guarantee delivery across system restarts.