Pipes & Named Pipes (FIFO)

If you have ever typed ls | grep "error" in a terminal, you have used a pipe without even knowing it. Pipes are one of the oldest inter-process communication mechanisms in Unix-like operating systems. They embody the Unix philosophy of building complex behavior from simple, composable parts. Yet beneath that deceptively simple | operator lies a rich set of kernel-level details that every systems programmer should understand.

Introduction

A pipe is a unidirectional data channel that connects two processes. Data written to the write end of a pipe can be read from the read end. The kernel implements pipes as an in-memory circular buffer, which means the data never touches disk unless the system is under extreme memory pressure.

Anonymous pipes are the simpler variant. They are created using the pipe() system call and exist only as kernel objects — there is no filesystem representation. Because they rely on inheritance through fork(), anonymous pipes only work between processes with a common ancestor (typically a parent-child or sibling relationship).

Named pipes, also known as FIFOs (First In, First Out), are filesystem objects. You create them with mkfifo() and they persist in the filesystem just like a regular file. Any process that knows the path to a named pipe can open it for reading or writing, regardless of their ancestry. This makes FIFOs the tool of choice for communication between completely unrelated processes.

The key distinction is provenance. Anonymous pipes are for “processes I already know” — ones I fork or inherit from. Named pipes are for “processes I will eventually meet” — daemons, co-processes, and scripts that need to find each other through filesystem paths.

When to Use / When Not to Use

Use anonymous pipes when:

• You have a parent-child or sibling process relationship
• You want to stream data between a filter command and its input (shell pipelines)
• You need a simple, low-overhead one-directional channel
• The communicating processes share a common ancestor

Use named pipes when:

• You need to communicate between unrelated processes
• You want the communication channel to persist across process lifetimes
• You need multiple writers or multiple readers for the same channel
• You need to coordinate between processes that are started independently
• You want to debug communication between two long-running processes

Do not use pipes when:

• You need bidirectional communication (use a socket pair instead)
• You need to persist data across system reboots (use a regular file)
• You need to seek within the data (pipes are stream-oriented, not random access)
• You want to communicate between machines (use TCP/UDP sockets)
• You need message boundaries preserved with multiple discrete messages (use message queues)

Architecture or Flow Diagram

graph TD
    subgraph Anonymous Pipe Creation
        A[Parent process calls pipe] --> B[Kernel creates pipe buffer<br/>and two file descriptors]
        B --> C[fd[0] = read end, fd[1] = write end]
    end

    subgraph Fork Pattern
        C --> D[Parent forks child process]
        D --> E[Child inherits file descriptors]
        E --> F[Parent closes read end<br/>Child closes write end]
        F --> G[Data flows: Parent writes to fd[1]<br/>Child reads from fd[0]]
    end

    subgraph Named Pipe Creation
        H[Process calls mkfifo] --> I[Kernel creates FIFO file<br/>in filesystem]
        I --> J[Any process with path<br/>can open FIFO for R/W]
    end

    K[Reader process] --> L[Blocks on open FIFO until writer connects]
    M[Writer process] --> L
    L --> N[Both sides now communicate]

Core Concepts

Anonymous Pipe Creation

The pipe() system call creates an anonymous pipe and returns two file descriptors:

#include <unistd.h>

int pipefd[2];
if (pipe(pipefd) == -1) {
    perror("pipe");
    exit(1);
}
// pipefd[0] = read end
// pipefd[1] = write end

When you call pipe(), the kernel allocates a kernel buffer (typically 64KB on modern Linux systems) and creates two file objects pointing to it. The read file descriptor references the buffer’s head, and the write file descriptor references its tail.

The Fork Pattern

Anonymous pipes are almost always used with fork(). When a process forks, its file descriptor table is duplicated, so the child inherits the pipe’s file descriptors. This creates a situation where both parent and child have both ends of the pipe open — which is usually wrong. The standard pattern is for one side to close the end it does not need:

int pipefd[2];
pipe(pipefd);

pid_t pid = fork();
if (pid == 0) {
    // Child process — reads from pipe
    close(pipefd[1]);  // Close write end
    char buf[128];
    read(pipefd[0], buf, sizeof(buf));
    close(pipefd[0]);
} else {
    // Parent process — writes to pipe
    close(pipefd[0]);  // Close read end
    write(pipefd[1], "hello child", 10);
    close(pipefd[1]);
}

Pipe Capacity and Blocking

Every pipe has a finite capacity. When the buffer is full, a write() blocks until a reader drains data. On Linux, you can query and modify pipe capacity with fcntl(F_GETPIPE_SZ) and fcntl(F_SETPIPE_SZ). The default is typically 65536 bytes (one page), but it can be increased up to /proc/sys/fs/pipe-max-size (usually 1MB).

If all file descriptors referring to the write end are closed and a reader attempts to read(), it returns 0 (end-of-file). This is how the reading side detects that the writer has finished.

Named Pipes (FIFO)

A FIFO is created with mkfifo():

#include <sys/stat.h>

if (mkfifo("/tmp/my_fifo", 0666) == -1) {
    perror("mkfifo");
    // Might fail if already exists — handle appropriately
}

Once created, a FIFO appears in the filesystem and can be opened with open() just like a regular file. However, the open() call has special semantics for FIFOs:

• open() for reading blocks until another process opens the same FIFO for writing
• open() for writing blocks until another process opens the same FIFO for reading
• You can pass O_RDWR to open both ends without blocking (but this is rarely the right approach)

This blocking-on-open behavior is a common source of confusion and bugs. Both sides of a FIFO need to be opened carefully to avoid deadlocks.

Pipe Buffer Internals

On Linux, pipe buffers are managed as a circular buffer of struct pipe_buffer entries. Each buffer entry can hold up to one page (4KB) of data. When you write large amounts of data, the kernel may allocate multiple buffer slots. The pipe inode tracks these entries and their consumer/producer positions.

Production Failure Scenarios

Broken Pipe Signal (SIGPIPE)

When you write to a pipe whose read end has been closed, the process receives a SIGPIPE signal. By default, this terminates the process. This happens frequently in shell pipelines like yes | head -n 1 — yes writes to a pipe whose reader (head) exits after producing one line, and yes receives SIGPIPE.

Mitigation: Set SIGPIPE to SIG_IGN in your signal handlers, or use the MSG_NOSIGNAL flag on send():

signal(SIGPIPE, SIG_IGN);
// or
send(fd, data, len, MSG_NOSIGNAL);

FIFO Deadlock Due to Blocking Open

If both a reader and writer process try to open the same FIFO in blocking mode, and the reader opens first and waits for a writer (and the writer does the same), you have a classic deadlock. The open() call blocks indefinitely because each side is waiting for the other.

Mitigation: Open FIFOs in non-blocking mode, or ensure the reader runs first and creates the FIFO if it does not exist:

int fd = open("/tmp/my_fifo", O_RDONLY | O_NONBLOCK);
// Must also open write end to prevent blocking even in non-blocking mode
int wfd = open("/tmp/my_fifo", O_WRONLY | O_NONBLOCK);

Forgetting to Close Unused Pipe Ends

In the fork pattern, if you forget to close the unused end of the pipe, the reading side may never see end-of-file because there is still a writer descriptor open (even if the original writer process does not actually write). This leads to the reader blocking forever on read().

Mitigation: Be disciplined about closing both ends immediately after fork() before any data operations begin. Use a wrapper function or clearly comment which side closes which end.

Pipe Capacity Exhaustion

If a fast producer writes to a pipe faster than a slow consumer reads from it, the buffer fills up and the producer blocks. In a pipeline, this can cause the entire pipeline to stall — a slow consumer causes the producer to block, which causes the pipeline to back up.

Mitigation: Use pipe() with larger capacity via fcntl(F_SETPIPE_SZ), implement backpressure signals in your protocol, or use non-blocking I/O with an event loop (e.g., epoll or select).

FIFO Permissions and Security

FIFOs created with mkfifo() respect the umask of the creating process. If you create a FIFO that is world-readable/writable, any user on the system can read or write to it. This is a security concern in multi-user environments.

Mitigation: Set appropriate permissions (e.g., mkfifo(path, 0660) for a group-owned FIFO), use filesystem ACLs, or place FIFOs in directories with restricted access.

Trade-off Table

Feature	Anonymous Pipe	Named Pipe (FIFO)	Unix Domain Socket
Creation	pipe() system call	mkfifo() system call	socketpair() or socket() + bind()
Filesystem Presence	None (kernel-only)	Yes (persistent filesystem entry)	None (kernel-only) or filesystem path for named sockets
Process Relationship	Must share ancestry (fork inheritance)	Any process (no relationship required)	Any process (no relationship required)
Bidirectional	No (unidirectional only)	No (unidirectional only)	Yes (full-duplex)
Capacity	Configurable (default ~64KB)	Same as anonymous pipe (kernel buffer)	Configurable, typically larger
Message Boundaries	None (byte stream)	None (byte stream)	None (byte stream for SOCK_STREAM) / Yes (for SOCK_DGRAM)
Persistence	Destroyed when all descriptors close	Persists until unlinked	Destroyed when all descriptors close (unnamed) or until unlinked (named)
Multiple Readers/Writers	Supported but complex	Supported but complex	Supported natively
Typical Use Case	Shell pipelines, parent-child streaming	Daemon communication, cross-process coordination	Network IPC, full-duplex local communication

Implementation Snippet(s)

C: Fork-based Anonymous Pipe

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>

int main() {
    int pipefd[2];
    if (pipe(pipefd) == -1) {
        perror("pipe");
        exit(1);
    }

    pid_t pid = fork();
    if (pid == -1) {
        perror("fork");
        exit(1);
    }

    if (pid == 0) {
        // Child: read from pipe
        close(pipefd[1]);  // Close write end
        char buf[256];
        ssize_t n = read(pipefd[0], buf, sizeof(buf) - 1);
        if (n > 0) {
            buf[n] = '\0';
            printf("Child received: %s\n", buf);
        }
        close(pipefd[0]);
        _exit(0);
    } else {
        // Parent: write to pipe
        close(pipefd[0]);  // Close read end
        const char *msg = "Hello from parent!";
        write(pipefd[1], msg, strlen(msg));
        close(pipefd[1]);
        wait(NULL);  // Wait for child to finish
    }

    return 0;
}

Python: Using Named FIFO

import os
import time
import errno

FIFO_PATH = "/tmp/my_python_fifo"

# Create FIFO if it doesn't exist
try:
    os.mkfifo(FIFO_PATH)
except FileExistsError:
    pass

# Writer process
def writer():
    with open(FIFO_PATH, 'w') as fifo:
        fifo.write("Message via named pipe\n")
        fifo.flush()

# Reader process (in practice, run this as a separate process)
def reader():
    with open(FIFO_PATH, 'r') as fifo:
        line = fifo.readline()
        print(f"Reader received: {line.strip()}")

# To test: open two terminal sessions
# Terminal 1: python3 reader.py
# Terminal 2: python3 writer.py

Bash: Anonymous Pipe in Shell

#!/bin/bash
# Classic shell pipeline
cat /var/log/syslog | grep "error" | head -n 20

# Programmatically create and use a pipe
mkfifo /tmp/my_pipe

# In one terminal
# cat /tmp/my_pipe

# In another terminal
# echo "Hello from bash FIFO" > /tmp/my_pipe

# Cleanup
rm /tmp/my_pipe

Observability Checklist

• Process state: Check if processes are blocked on pipe read/write using ps aux or lsof -p <pid>
• File descriptors: Monitor open file descriptors for pipes with ls -la /proc/<pid>/fd/
• Pipe capacity: Query current pipe size with fcntl(fd, F_GETPIPE_SZ) in C or check via /proc/<pid>/fdinfo/
• SIGPIPE handling: Verify signal handler is set (cat /proc/<pid>/status | grep SigPnd)
• FIFO existence: Check if FIFOs exist in expected locations with ls -la /tmp/*.fifo or similar
• strace: Use strace -e trace=pipe,mkfifo,read,write -p <pid> to trace pipe operations
• dtrace/SystemTap: Trace pipe create/open/read/write events across the system

Common Pitfalls / Anti-Patterns

Permissions: FIFOs respect the standard Unix permission model. Ensure the FIFO is created with appropriate access controls — world-readable FIFOs allow any local user to participate in the communication channel. Use mode 0660 or 0664 with a specific group when possible.

Content Exposure: Unlike sockets, FIFOs are filesystem objects and their content can be inspected with ls. If you are communicating sensitive data, ensure the FIFO resides in a protected directory (e.g., /var/run/myapp/) with restricted permissions.

Denial of Service: A malicious process could open a FIFO for reading or writing and never close it, causing the legitimate counterpart to block on open(). Use non-blocking open (O_NONBLOCK) and implement timeouts for all I/O operations.

No Authentication: Pipes and FIFOs provide no built-in authentication. Any process that can reach the pipe (by file descriptor inheritance or by path) can read or write data. Implement your own application-layer authentication if needed.

Audit Trail: Pipe operations do not generate filesystem audit events by default (since FIFOs are not regular files). Monitor process-level audit logs for access patterns.

Common Pitfalls / Anti-patterns

1. Forgetting to close unused pipe ends — causes readers to hang forever waiting for EOF that never comes because a write end is still open somewhere.

2. Assuming read returns the same write unit — write() of 100 bytes does not guarantee read() of 100 bytes. It may return 50, then another 50, or 1 byte at a time. Always handle partial reads.

3. Ignoring SIGPIPE — writing to a closed pipe kills your process by default. Set signal(SIGPIPE, SIG_IGN) or use MSG_NOSIGNAL.

4. Blocking on FIFO open without a counterpart — opening a FIFO in blocking mode hangs if no other process has the other end open. Use non-blocking mode or coordinate startup order.

5. Using pipes for message-oriented data — pipes are byte streams with no message boundaries. If you send discrete messages, you need to frame them yourself (length prefix, delimiter, etc.).

6. Not checking for errors — pipe(), mkfifo(), read(), and write() can all fail. Always check return values.

7. Leaving FIFOs in filesystem after crash — if a process creating a FIFO terminates abruptly, the FIFO may remain. Implement cleanup on startup or use a process ID in the filename.

Quick Recap Checklist

• Pipes are unidirectional byte streams implemented as in-memory circular buffers in the kernel
• Anonymous pipes (pipe()) require a common ancestor (fork inheritance); named pipes (mkfifo()) work between any processes via filesystem path
• Always close the unused end of a pipe in both parent and child after fork() to avoid hangs
• Set SIGPIPE to SIG_IGN or use MSG_NOSIGNAL to prevent unexpected process termination
• FIFOs block on open() until both ends are connected — use O_NONBLOCK or coordinate startup order
• Pipes are for streaming data; they have no message boundaries — implement your own framing for discrete messages
• Pipe capacity is finite (default ~64KB on Linux) — large writes will block when buffer is full
• FIFOs persist in the filesystem and require cleanup (unlink) when no longer needed
• Named pipes enable unrelated processes to communicate but lack built-in authentication

Interview Questions

Further Reading

• Unix Pipes and FIFOs — Linux man pages — Official documentation for named pipes
• pipe(2) — Linux man page — System call reference for pipe creation
• select(2) — Linux man page — Multiplexing I/O across file descriptors
• epoll(7) — Linux man page — Linux-specific scalable I/O event notification

Conclusion

Pipes and FIFOs are foundational IPC primitives in Unix systems — simple in concept but packed with kernel-level details. Anonymous pipes excel at connecting related processes in parent-child or pipeline arrangements, while FIFOs extend that capability to unrelated processes through the filesystem. Their byte-stream nature makes them ideal for streaming data, though the lack of message boundaries means you need careful protocol design when discrete messages are involved.

As systems become more distributed, pipes give way to socket-based IPC for cross-machine communication. But on a single host, pipes remain among the fastest mechanisms available — data moves through kernel-managed buffers without per-byte system call overhead. Understanding pipe semantics — blocking behavior, capacity limits, SIGPIPE handling — sets you up for all forms of stream-oriented communication.

For continued learning, explore Unix domain sockets (which offer bidirectional communication while keeping pipe-like efficiency), and study how select()/poll()/epoll() APIs let you multiplex multiple pipes within a single thread.