Signals

Every process in a Unix-like system lives under the constant awareness of signals — asynchronous notifications from the kernel and other processes that something important has happened. When you press Ctrl+C in a terminal, a signal is sent. When a process crashes and the kernel detects an illegal memory access, a signal is delivered. When a child process exits, its parent receives a signal. Signals are the oldest and most fundamental form of inter-process communication, predating pipes, message queues, and sockets.

Despite their simplicity, signals are often misunderstood and mishandled. The async nature of signals makes them notoriously tricky to reason about — the signal can arrive at any point in your program’s execution, interrupting system calls, modifying global state, and causing subtle bugs if not handled carefully. Getting signal handling right is a hallmark of systems programming competence.

Introduction

A signal is a notification sent to a process (or thread) to indicate that an event has occurred. Signals are asynchronous — they can arrive at any time, interrupting the normal flow of execution. When a signal arrives, the process either:

1. Runs a signal handler — a function you define to handle the specific signal
2. Takes the default action — which varies by signal (terminate, ignore, stop, dump core)
3. Ignores it — if you explicitly request to ignore the signal

There are over 30 standard signals defined in POSIX, each with a name (like SIGTERM, SIGINT, SIGSEGV) and a numeric value. Signals are delivered for various reasons:

• User-initiated: Ctrl+C sends SIGINT, Ctrl+Z sends SIGTSTP
• Process communication: One process sending a signal to another via kill()
• Kernel-generated: Hardware exception (SIGSEGV, SIGFPE), timer (SIGALRM), child exit (SIGCHLD)
• Resource limits: SIGXCPU (CPU time exceeded), SIGXFSZ (file size exceeded)

Signals are delivered to a specific thread. If the process has multiple threads, the signal is delivered to one arbitrarily selected thread that is not blocking the signal.

When to Use / When Not to Use

Use signals when:

• You need to send a simple notification to a process (terminate, pause, resume)
• You need to handle asynchronous events like timer expirations or child process exits
• You need to implement a clean shutdown mechanism (catch SIGTERM and cleanup gracefully)
• You are building a daemon that responds to external control commands
• You need to interrupt a blocking system call

Do not use signals when:

• You need to transfer data (signals carry no payload — use pipes or sockets)
• You need synchronous, ordered communication (use message queues or pipes)
• You need many different message types (use sockets with custom protocols)
• You are building complex IPC that requires confirmation/reply patterns

Architecture or Flow Diagram

sequenceDiagram
    participant P1 as Process A (Sender)
    participant Kernel
    participant P2 as Process B (Receiver)

    Note over P1: kill(PID_B, SIGTERM)
    P1->>Kernel: raise(SIGTERM, target=PID_B)
    Kernel->>Kernel: Validate target process exists<br/>Check sender permissions

    alt Process B has handler
        Kernel->>P2: Deliver signal (may interrupt syscall)
        P2->>P2: Run signal handler sigaction(SIGTERM)
        P2-->>P2: Resume execution after handler
    else Process B ignores
        Kernel->>P2: Signal delivered but ignored
        Note over P2: No action taken
    else Default action
        Kernel->>P2: Terminate process
        Note over P2: Process exits, status = 143 (SIGTERM)
    end

Core Concepts

Signal Dispositions and sigaction

Every signal has a “disposition” — what happens when the signal is delivered. The default disposition varies by signal. You can change the disposition using signal() (simple, legacy) or sigaction() (more control, portable):

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>

void handle_sigterm(int sig) {
    // Signal-safe operations only here
    // No printf, malloc, or non-async-signal-safe functions!
    const char msg[] = "Received SIGTERM, shutting down...\n";
    write(STDOUT_FILENO, msg, sizeof(msg) - 1);
    // Perform cleanup...
    _exit(0);
}

int main() {
    struct sigaction sa;
    sa.sa_handler = handle_sigterm;  // Handler function
    sigemptyset(&sa.sa_mask);        // No signals blocked during handler
    sa.sa_flags = 0;                  // No special flags
    // SA_RESTART: restart syscalls interrupted by this signal
    // SA_NODEFER: don't block this signal during its own handler

    if (sigaction(SIGTERM, &sa, NULL) == -1) {
        perror("sigaction");
        exit(1);
    }

    // Now SIGTERM will call handle_sigterm instead of terminating
    while (1) {
        pause();  // Wait for signals
    }
}

Key sigaction fields:

• sa_handler: Function pointer (SIG_DFL for default, SIG_IGN to ignore)
• sa_mask: Set of signals to block during this handler’s execution
• sa_flags: Modifiers like SA_RESTART (restart syscalls), SA_SIGINFO (use sa_sigaction with extra info)

Signal Masks

Each thread (and by extension, process) has a signal mask — a set of signals that are currently blocked. Blocked signals are not delivered, but they may be pending. You can manipulate the signal mask with pthread_sigmask() (for threads) or sigprocmask() (for single-threaded processes):

#include <signal.h>

void block_sigchld() {
    sigset_t mask;
    sigemptyset(&mask);
    sigaddset(&mask, SIGCHLD);

    sigset_t oldmask;
    pthread_sigmask(SIG_BLOCK, &mask, &oldmask);
    // In this block, SIGCHLD won't interrupt us

    // ... do work ...

    // Restore previous mask
    pthread_sigmask(SIG_SETMASK, &oldmask, NULL);
}

Sending Signals

You send signals with the kill() system call:

#include <signal.h>
#include <unistd.h>

pid_t target_pid = 1234;

// Send SIGTERM to a process
if (kill(target_pid, SIGTERM) == -1) {
    perror("kill");
}

// Send SIGUSR1 with no default action (user-defined)
kill(target_pid, SIGUSR1);

The raise() function sends a signal to the current process:

raise(SIGALRM);  // Equivalent to kill(getpid(), SIGALRM)

Signal Safety and Async Signal Safety

Not all functions can be safely called from a signal handler. A function is async-signal-safe if it can be safely called from a signal handler — meaning it does not use non-reentrant locks, does not allocate memory from the heap, and does not make assumptions about the interrupted execution context. The problem goes beyond just locks: when a signal handler runs, it interrupts whatever system call was in progress. If that call was holding a libc internal lock (say, for malloc’s arena), and your handler then calls malloc, you deadlock. If the interrupted code was in the middle of printf writing to a buffer, and your handler calls printf again, output gets corrupted or interleaved.

The rules that make a function async-signal-safe are:

• No internal synchronization (no mutexes, no semaphores that might already be held)
• No dynamic memory allocation (no heap malloc/free/realloc)
• No legacy buffered I/O (no stdio functions that maintain internal state)
• Reentrant (safe to call from nested signal contexts)

A practical breakdown:

Safe to call from handlers:

• _exit(), write(), sync(), fsync() — direct kernel calls, no libc buffering
• wait(), waitpid(), kill() — process and signal control
• sigprocmask(), sigemptyset(), sigaddset() — signal mask manipulation
• memcpy(), memset() — raw memory operations, no allocation
• open() (some flags), close() — low-level, no stdio buffering

Never safe in a handler:

• printf(), fprintf(), puts() — stdio maintains internal buffers and locks
• malloc(), free(), realloc() — heap allocation uses internal locks
• pthread_mutex_lock(), pthread_mutex_unlock() — libc internal locks conflict with the interrupted code’s locks
• syslog() — internally uses stdio and socket operations
• getpwnam(), getgrnam() — these functions cache in static buffers and use locks

One gotcha that bites frequently: signal handlers clobber errno. If your interrupted code had just set errno to something meaningful and the handler calls any function that modifies errno (including write() on some platforms), the original value is lost. Save and restore errno at the handler boundaries:

void handler(int sig) {
    int saved_errno = errno;
    // ... handler logic that may call write() ...
    errno = saved_errno;
}

The list of async-signal-safe functions is in man 7 signal-safety. Keep signal handlers minimal — set a volatile sig_atomic_t flag, write a byte to a pipe, or call _exit(). Do complex cleanup in the main program after checking the flag.

Waiting for Signals: pause(), sigsuspend(), sigwaitinfo()

// Simple pause (wait for any signal)
pause();

// Block in sigsuspend atomically replacing signal mask
sigset_t mask;
sigemptyset(&mask);
sigaddset(&mask, SIGINT);
sigprocmask(SIG_BLOCK, &mask, NULL);  // Block SIGINT
// Critical section - SIGINT blocked
sigprocmask(SIG_SETMASK, &mask, NULL);  // Unblock, delivering SIGINT

// Synchronous wait for specific signals (real-time signals)
sigset_t waitmask;
sigemptyset(&waitmask);
sigaddset(&waitmask, SIGRTMIN);
sigwaitinfo(-1, &waitmask, NULL);  // Wait for SIGRTMIN

Production Failure Scenarios

Interrupted System Calls (EINTR)

When a signal handler runs during a blocking system call (like read(), write(), connect(), sleep()), the kernel may interrupt the call and return -1 with errno = EINTR. If you do not handle this, your program may interpret the interrupted call as an error.

Mitigation: Always check for EINTR and retry interrupted calls. Use SA_RESTART flag with sigaction() to automatically restart some syscalls. Review all blocking calls for EINTR handling:

ssize_t n;
while ((n = read(fd, buf, sizeof(buf))) == -1 && errno == EINTR) {
    // Retry
}
if (n == -1) {
    perror("read");
}

Signal Handler Race Conditions

If your signal handler modifies a global variable that is also modified by the main program without synchronization, you get a race condition. For example, a handler setting a shutdown_requested flag while the main loop checks it:

volatile sig_atomic_t shutdown_requested = 0;  // Volatile needed!

void handler(int sig) {
    shutdown_requested = 1;
}

int main() {
    // ...
    while (!shutdown_requested) {
        // Do work - but this is still racy on some architectures!
    }
}

Mitigation: Use volatile sig_atomic_t for simple flags (atomic enough for single-flag access). For more complex synchronization, use a self-pipe — the signal handler writes to a pipe, and the main loop monitors the pipe with select()/epoll().

Missing SIGCHLD Handling — Zombie Processes

When a child process terminates, it does not disappear immediately. The kernel keeps the child’s exit status (PID, exit code, resource usage) in the process table entry. The parent must retrieve this with wait() or waitpid() — until then, the child sits in zombie state. A zombie process consumes a slot in the kernel’s process table, and if enough zombies accumulate, you can exhaust process table entries and prevent new processes from being created.

A concrete scenario:

for (int i = 0; i < 10; i++) {
    pid_t pid = fork();
    if (pid == 0) {
        // child: do work and exit
        sleep(1);
        _exit(0);
    }
    // parent: no wait() call here — each child becomes a zombie
}
// After this loop: 10 zombie processes
while (1) sleep(1);  // they stay zombies until this process exits

In this example, each child exits within a second, but the parent never calls wait(), so each remains a zombie until the parent eventually exits. On a busy server that forks many child processes per request and does not reap them, zombies can pile up rapidly.

Three ways to handle this:

Option 1 — Ignore SIGCHLD (simplest):

signal(SIGCHLD, SIG_IGN);

On Linux, this tells the kernel to discard child exit statuses automatically — children are reaped immediately upon termination without becoming zombies. The trade-off: you cannot use wait() or waitpid() to retrieve exit statuses, so you lose that information.

Option 2 — SA_NOCLDWAIT flag:

struct sigaction sa;
sa.sa_handler = SIG_IGN;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_NOCLDWAIT;
sigaction(SIGCHLD, &sa, NULL);

This has the same effect as ignoring SIGCHLD but uses the POSIX sigaction() interface. Same trade-off: no exit status retrieval.

Option 3 — Explicit handler with WNOHANG (most control):

void handle_sigchld(int sig) {
    int saved_errno = errno;
    while (waitpid(-1, NULL, WNOHANG) > 0);
    errno = saved_errno;
}

struct sigaction sa;
sa.sa_handler = handle_sigchld;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_RESTART;
sigaction(SIGCHLD, &sa, NULL);

The loop with WNOHANG reaps all terminated children in one call, so multiple children exiting before the handler runs still get cleaned up. This approach gives you the child’s exit status if you need it, and is the right choice for daemons that need to track child lifecycle.

Detecting zombies: ps aux | grep -i zombie shows processes in state Z. In /proc//status, State: shows Z for zombies. The PPid: field shows which parent should be reaping them.

SIGPIPE Misconfiguration

When writing to a pipe or socket whose reading end has closed, the writer receives SIGPIPE. The default action terminates the process. If you have not handled SIGPIPE, any write() to a broken pipe terminates your program.

When the reader closes its end, the kernel sends SIGPIPE to the writer on the next write(). The signal arrives immediately, before write() returns. If you have not installed a handler, the default action fires before your code can respond. This catches many developers off guard because write() returns -1 with errno = EPIPE in addition to sending the signal. The signal terminates the process before you ever check that return value.

Two mitigations cover most cases:

Option 1 — Ignore globally (simplest):

signal(SIGPIPE, SIG_IGN);

Place this once at program startup, before any network or pipe I/O. On Linux, this works process-wide. You lose the ability to act on SIGPIPE delivery specifically, but write() still returns errno = EPIPE, so you catch the error in your normal error path.

Option 2 — Per-call suppression (more targeted):

send(fd, buf, len, MSG_NOSIGNAL);

The MSG_NOSIGNAL flag suppresses SIGPIPE for that specific call. write() still returns -1 with errno = EPIPE, so handle it in your error path.

For servers managing many connections, ignoring SIGPIPE globally avoids silent terminations on connection drops. For clients that need to tell a clean close from an error, MSG_NOSIGNAL with explicit EPIPE handling gives you that distinction.

To debug unexpected terminations: run the program under strace. Look for --- SIGPIPE in the output, which shows the write() call that triggered it. A quick diagnostic is to replace SIG_IGN with a custom handler that prints a backtrace, so you can see exactly which code path is writing to a closed pipe.

Signal Masking Deadlocks

Signal masking deadlocks bite when a blocked signal arrives while your code holds a lock, and the handler for that signal tries to grab the same lock. The thread waits on a lock the handler cannot release because the handler cannot run while the signal is masked. Recursive mutexes make this worse — the same thread can re-enter a lock it already holds, but only if the handler fires during the critical section. With SA_NODEFER set, the handler can even re-enter itself on the same signal, compounding the problem into a full deadlock.

Here is what this looks like in practice:

pthread_mutex_t guard = PTHREAD_MUTEX_INITIALIZER;
sigset_t mask;

void init() {
    sigemptyset(&mask);
    sigaddset(&mask, SIGUSR1);
    pthread_sigmask(SIG_BLOCK, &mask, NULL);  // Block SIGUSR1
}

void handle_usr1(int sig) {
    pthread_mutex_lock(&guard);   // May deadlock here!
    // ... handler work ...
    pthread_mutex_unlock(&guard);
}

void critical_section() {
    pthread_mutex_lock(&guard);
    // If SIGUSR1 arrives here, handler runs after mutex_unlock
    // If handler runs BEFORE mutex_lock in this path, we deadlock
    pthread_mutex_unlock(&guard);
}

The issue surfaces in the narrow window between entering and exiting the critical section. On Linux, pthread_mutex_lock() is not async-signal-safe. Calling it from a handler that interrupted a lock-holding region produces undefined behavior — usually a deadlock, sometimes memory corruption.

The self-pipe pattern eliminates this entire class of problem. The handler never touches application locks — it writes a single byte to a pipe and returns. The main event loop watches that pipe and does all the real work in a context where no signal can interrupt.

Beyond that: avoid putting locks in signal handlers entirely, use sigsuspend() to atomically unblock signals during critical sections, prefer non-recursive mutexes so re-entrancy fails loudly instead of silently deadlocking, and run thread-safety analysis tools to find any path where a handler touches a lock the main code also uses.

To debug: check /proc/<pid>/status | grep -i sig for blocked and pending signals. If all threads are stuck in pthread_mutex_lock() and no signals are pending, a handler is probably trying to grab a lock the interrupted code holds. strace -f shows signal delivery and lock operations interleaved.

Trade-off Table

Implementation Snippet(s)

C: Clean Shutdown with SIGTERM

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdbool.h>

volatile sig_atomic_t g_shutdown = false;

void sigterm_handler(int sig) {
    g_shutdown = true;
}

int main() {
    struct sigaction sa;
    sa.sa_handler = sigterm_handler;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_RESTART;  // Restart interrupted syscalls

    sigaction(SIGTERM, &sa, NULL);
    sigaction(SIGINT, &sa, NULL);

    printf("Service running (PID %d). Send SIGTERM to stop.\n", getpid());

    while (!g_shutdown) {
        // Do work — simulate with sleep
        sleep(1);
    }

    printf("Shutting down gracefully...\n");
    // Cleanup: close files, release resources, flush buffers
    printf("Done.\n");
    return 0;
}

Python: Signal Handling

import signal
import sys

def sigterm_handler(signum, frame):
    print("Received SIGTERM, shutting down gracefully...")
    # Note: in Python, most functions are safe to call from signal handlers
    # because the GIL provides some protection
    sys.exit(0)

def sigint_handler(signum, frame):
    print("Received Ctrl+C, shutting down...")
    sys.exit(0)

signal.signal(signal.SIGTERM, sigterm_handler)
signal.signal(signal.SIGINT, sigint_handler)

print(f"PID: {sys.executable}")
while True:
    signal.pause()

Bash: Signal Trapping in Scripts

#!/bin/bash
# Trap signals for graceful shutdown
cleanup() {
    echo "Received signal, cleaning up..."
    rm -f /tmp/myapp.lock
    exit 0
}

trap cleanup SIGTERM SIGINT SIGQUIT

# Or ignore signals during critical section
trap '' SIGINT
echo "In critical section - Ctrl+C ignored"
sleep 5
trap SIGINT  # Restore default handling

# Background job with signal control
sleep 100 &
sleep_pid=$!
# Wait for job or signal
wait $sleep_pid 2>/dev/null || echo "Interrupted"

Observability Checklist

• Pending signals: Check pending signals for a process with cat /proc/<pid>/status | grep -i SigPnd
• Blocked signals: Check blocked signals with cat /proc/<pid>/status | grep -i SigBlk
• Signal handler registration: Use strace -e trace=signal to see signal-related system calls
• Process state on signal: ps aux shows processes in interruptible sleep (state S) waiting for signals
• Core dumps: Check ulimit -c and /proc/sys/kernel/core_pattern to ensure core dumps are captured for crashes
• Signal delivery timing: Use perf sched or ftrace to measure signal delivery latency
• SIGCHLD zombies: ps aux | grep -i zombie to find zombie processes indicating improper wait handling

Common Pitfalls / Anti-Patterns

Signal Permissions & DoS: Sending a signal requires appropriate permissions — either the sender must have the same effective user ID as the receiver (kill(pid, sig) succeeds), or the sender must have CAP_KILL capability (root). A process could flood another with signals (e.g., repeatedly sending SIGURG which has no default action but still triggers handler execution), consuming CPU time. Use rate limiting if implementing signal-sending functionality.

Signal-based Attacks & Audit: Some privilege escalation exploits use signals to manipulate process state — ensure signal handlers do not perform unsafe operations and security-sensitive state is not accessible via handler interfaces. Signal delivery may not generate standard audit events; for compliance requiring complete process behavior tracking, implement application-level logging around signal handling.

1. Calling non-async-signal-safe functions in handlers — printf(), malloc(), free(), pthread_mutex_lock() are all unsafe in signal handlers. Keep handlers minimal: set a flag, write to a pipe, call _exit().

2. Not handling EINTR on blocking calls — always check for and retry interrupted system calls, or use SA_RESTART to let the kernel handle it.

3. Using non-volatile global flags — the compiler may optimize away reads of a global variable that the signal handler sets, causing infinite loops. Use volatile sig_atomic_t.

4. Ignoring SIGCHLD — not handling child exits causes zombie processes that accumulate and consume process table entries.

5. Not setting SA_RESTART on time-consuming operations — if a signal arrives during a long read() from a slow device, and you did not set SA_RESTART, the read() returns -1 with EINTR. Handle this or use select()/poll().

6. Signal handler modifying errno — signal handlers clobber errno. Save and restore errno in handlers if needed.

7. Mixing signal() and sigaction() — the behavior of signal() varies across platforms (some restart syscalls, some do not). Always use sigaction() for portability.

8. Sending SIGKILL to processes that need cleanup — SIGKILL cannot be caught, blocked, or ignored. It terminates the process immediately without running cleanup handlers. Use SIGTERM for graceful shutdown.

Quick Recap Checklist

• Signals are async notifications delivered to processes; they can interrupt any execution point
• Use sigaction() for portable, full-featured signal handling (preferred over signal())
• Signal handlers must only call async-signal-safe functions — keep handlers minimal
• volatile sig_atomic_t for simple flags; use self-pipe pattern for complex synchronization
• SIGPIPE defaults to terminating the process — set SIG_IGN or use MSG_NOSIGNAL
• SA_RESTART flag automatically restarts certain interrupted system calls
• SIGCHLD should be handled or ignored to prevent zombie processes
• kill(pid, sig) sends a signal to a specific process; raise(sig) sends to current process
• Real-time signals (SIGRTMIN to SIGRTMAX) support queuing and carry extra payload
• Always handle EINTR on blocking calls or use SA_RESTART

Interview Questions

struct sigaction sa;
sa.sa_handler = handle_sigchld;
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_RESTART | SA_NOCLDWAIT;
sigaction(SIGCHLD, &sa, NULL);

int sigpipefd[2];
pipe(sigpipefd);
signal(SIGTERM, handler);  // handler does write(sigpipefd[1], "!", 1)
while (epoll_wait(efd, …) > 0) {
for each event:
if (event.fd == sigpipefd[0]) {
read(sigpipefd[0], &byte, 1);
// Do cleanup now - we are in safe context
}
}

Further Reading

• signal(7) — Linux man page — Overview of signals on Linux
• sigaction(2) — Linux man page — Signal action API reference
• signal-safety(7) — Linux man page — Async-signal-safe functions list
• kill(2) — Linux man page — Sending signals between processes
• Pipes & Named Pipes (FIFO) — pipe-based IPC — Anonymous and named pipe communication
• Process Concept — process management and scheduling — Process creation, termination, and lifecycle management

Conclusion

Signals are the oldest form of process communication in Unix systems — async notifications that can interrupt execution at any point. While the concept is simple, signal handling exposes the tension between the kernel’s event delivery model and safe user-space execution. The rules around async-signal-safety, EINTR handling, and signal mask management exist because signals cross protection boundaries in ways that normal function calls do not.

As systems evolved, signals gave way to more sophisticated event delivery mechanisms: event loops with select()/poll()/epoll(), kernel event queues in modern kernels, and asynchronous I/O APIs. Yet signals persist because they remain the only kernel-to-process notification mechanism for certain events (child exit, resource limits, crashes) that have no alternative.

For continued learning, explore how eventfd() provides a signal-like notification mechanism compatible with epoll(), and study how modern Linux uses real-time signals (SIGRTMIN to SIGRTMAX) with siginfo delivery for richer signal payloads beyond simple notifications.