System Call Security

Every operation your program performs that requires kernel involvement passes through system calls. Opening files, creating processes, allocating memory, network communication, all of these traverse the boundary between your application and the operating system kernel. This boundary is where security happens. Understanding how to control what system calls a program can make is fundamental to building secure systems.

System call security provides mechanisms to restrict what privileged operations a process can perform. Rather than running with full kernel privileges, processes can be confined to only the specific operations they actually need. This principle of least privilege dramatically reduces the blast radius when something goes wrong. If a vulnerability exists in a web server that has been restricted from making any file operations except reading specific document files, the attacker cannot use it to spawn a shell or read sensitive configuration.

The Linux kernel provides several complementary mechanisms for syscall security: seccomp filtering, capabilities, and namespace isolation. Each addresses different threat models and operational requirements. Together they form the foundation of container security and secure computing baselines.

Overview

System calls are the fundamental interface between user space and kernel space. When a user space program needs to do anything privileged, it invokes a system call instruction that transfers control to a predefined kernel handler. The kernel validates the request, performs the operation, and returns results to user space.

Without restrictions, a compromised process can invoke any system call, potentially escalating privileges or accessing resources it should not touch. System call security mechanisms allow administrators and developers to restrict which system calls a process can make, effectively shrinking its privilege domain.

Seccomp (secure computing mode) was originally introduced in kernel 2.6.12 as a simple mode that could either allow all system calls or none. The BPF extension added in 2.6.25 transformed seccomp into a powerful filtering mechanism. Modern seccomp-bpf allows fine-grained control over system call arguments, return values, and even interposition of system calls for debugging.

Capabilities split traditional root privileges into discrete units. Instead of having all privileges (or none), a process can have specific capabilities like CAP_NET_BIND_SERVICE to bind to privileged ports or CAP_SYS_ADMIN for system administration tasks. This fine-grained model enables precise privilege assignment.

When to Use / When Not to Use

System call security mechanisms are essential when running untrusted code, implementing security-critical services, or building container runtimes. Any application that processes external input from potentially malicious sources should consider syscall filtering.

Container orchestration platforms like Docker and Kubernetes rely heavily on these mechanisms. When you run a container, the runtime configures namespace isolation and capability sets to confine what the containerized process can do. Understanding these mechanisms helps you debug container security issues and design more secure deployments.

However, these mechanisms add complexity. Development and debugging become harder when system calls are blocked. Some applications require capabilities that are difficult to grant without introducing risk. In development environments or systems without external exposure, the overhead may not justify the benefits.

Architecture or Flow Diagram

graph TD
    A[User Space Process] -->|syscall| B[System Call Entry]
    B --> C{Seccomp Filter Active?}
    C -->|No| D[Execute System Call]
    C -->|Yes| E[BPF Program Runs]
    E --> F{BPF Rules Allow?}
    F -->|No| G[Deliver Signal<br/>EPERM]
    F -->|Yes| D
    D --> H[Return to User Space]

    subgraph Capabilities Check
    I{CAP_SYS_ADMIN?}
    I -->|Yes| J[Privileged Operation]
    I -->|No| K[Permission Denied]
    end

    style G fill:#ff6b6b
    style K fill:#ff6b6b

The system call filtering flow begins when a process makes a system call. The kernel intercepts this call and checks whether a seccomp filter is installed for the process. If a filter exists, the BPF program runs and evaluates the system call number and its arguments. Based on the filter rules, the BPF program either returns a verdict allowing the call, blocks it with an error, or triggers a signal to the process.

Capabilities are checked independently for specific privileged operations. The kernel maintains a set of permitted and effective capabilities for each process. When a privileged operation is attempted, the kernel verifies the process has the required capability before proceeding.

Core Concepts

Seccomp Modes

Linux provides three seccomp modes with increasing sophistication.

Seccomp Mode 1 (SECCOMP_MODE_STRICT) allows only read, write, _exit, and sigreturn system calls. Any other system call results in process termination. This mode is rarely used in practice due to its extreme restrictions.

Seccomp Mode 2 (SECCOMP_MODE_FILTER) enables BPF filtering with system call arguments. This mode is what people typically refer to when discussing seccomp. It allows fine-grained control but requires writing BPF programs.

Seccomp User Notifications (added in kernel 5.0) allows a user space supervisor to make seccomp decisions. This enables userspace seccomp implementations and is used by tools like strace and container runtimes.

BPF for Seccomp Filtering

Seccomp BPF programs follow the classic BPF execution model. The kernel runs the BPF program with the system call number and arguments available in BPF context. The program returns a verdict: SECCOMP_RET_ALLOW permits the call, SECCOMP_RET_KILL terminates the process, SECCOMP_RET_ERRNO returns a specified error code, or SECCOMP_RET_TRAP delivers a signal.

#include <linux/seccomp.h>
#include <linux/filter.h>
#include <sys/prctl.h>

void install_seccomp_filter() {
    struct sock_filter filter[] = {
        // Load system call number
        BPF_STMT(BPF_LD, BPF_W+BPF_ABS, offsetof(struct seccomp_data, nr)),
        // Allow read
        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_read, 0, 1),
        BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
        // Allow write
        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_write, 0, 1),
        BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
        // Allow exit
        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_exit, 0, 1),
        BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
        // Kill for everything else
        BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_KILL),
    };

    struct sock_fprog prog = {
        .len = sizeof(filter) / sizeof(filter[0]),
        .filter = filter,
    };

    prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &prog);
}

Linux Capabilities

Traditional Unix distinguished between privileged processes (those running as root with UID 0) and unprivileged processes. Capabilities split these privileges into discrete units.

Permitted capabilities are capabilities the process is allowed to use. Effective capabilities are currently active capabilities. Inheritable capabilities persist across execve calls. Bounding set limits which capabilities can ever be acquired.

# View capabilities of a process
getpcaps $$

# Capabilities explanation
# CAP_NET_BIND_SERVICE - Bind to ports below 1024
# CAP_SYS_ADMIN - Most system administration operations
# CAP_SYS_CHROOT - Use chroot()
# CAP_NET_RAW - Use raw sockets
# CAP_DAC_OVERRIDE - Bypass file permission checks

# Drop all capabilities except specific ones
capsh --drop=CAP_NET_RAW --print

Production Failure Scenarios + Mitigations

Scenario: Seccomp Blocks Required System Call

Problem: Application fails because seccomp filter blocks a system call the application legitimately needs.

Symptoms: Process receives SIGKILL or SIGSYS signals, application crashes with seccomp violations in logs.

Mitigation: Use auditd to identify required system calls before deploying seccomp filters. Generate allowlists by running applications under strace to capture all system calls made during normal operation. Test filters thoroughly in staging before production deployment.

# Find system calls made by application
strace -f -e trace=read,write,open,close,stat -o app_trace.log ./your_application

# Review generated allowlist
grep -o '__NR_[a-z_]*' app_trace.log | sort -u

Scenario: Capability Misconfiguration Locks Out Administrator

Problem: Administrative script loses required capabilities and cannot perform necessary operations.

Symptoms: Scripts fail with “Operation not permitted” errors, cron jobs do not run properly, backup scripts fail.

Mitigation: Document required capabilities for each administrative function. Use capability-aware monitoring tools. When dropping capabilities, drop only the minimum set required. Always preserve CAP_SYS_ADMIN for recovery operations.

# Test capability requirements before deployment
sudo -u username capsh --caps="cap_net_bind_service,cap_dac_read_search+ep" -- -c "./your_script"

# Recover from lockout - reboot to single user mode
# Add capability requirements to script comments

Scenario: Container Escape via Syscall

Problem: Vulnerability in container allows malicious container contents to escape namespace isolation.

Symptoms: Unauthorized access to host system files, processes outside container visible, network traffic bypasses container network.

Mitigation: Keep kernel updated to patch container escape vulnerabilities. Use seccomp profiles that block syscalls known to be exploited for container escapes (like unshare with certain flags). Follow container security best practices like not running containers as root.

# Use Docker's default seccomp profile (blocks ~44 syscalls)
docker run --security-opt seccomp=default.json ...

# Block dangerous syscalls
docker run --security-opt seccomp=<(
  echo '{"defaultAction":"SCMP_ACT_ALLOW",
        "syscalls":[{"name":"unshare","action":"SCMP_ACT_ERRNO"}]}'
) ...

Trade-off Table

Implementation Snippets

Basic Seccomp with libseccomp

#define _GNU_SOURCE
#include <seccomp.h>
#include <scmp/seccomp.h>
#include <stdio.h>

int main() {
    scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_KILL);
    if (!ctx) return 1;

    // Allow read, write, exit
    seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
    seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
    seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit), 0);

    // Allow open (but not openat)
    seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0);

    // Load filter
    if (seccomp_load(ctx) < 0) {
        perror("seccomp_load");
        seccomp_release(ctx);
        return 1;
    }

    seccomp_release(ctx);

    // Now run restricted code
    printf("Running with seccomp filter active\n");
    return 0;
}

Checking and Modifying Capabilities

#!/bin/bash
# Drop capabilities not needed by application

# Start with all capabilities
exec capsh --caps="cap_net_bind_service+ep cap_setuid+ep cap_setgid+ep" \
    --keep=1 --user=appuser \
    -- -c "./my_application"

Using strace with Seccomp

# Trace system calls in a seccomp-filtered process
# Requires kernel 5.8+ or use --seccomp=lower:0 to disable

strace -k -f ./seccomp_restricted_program 2>&1 | less

# -k shows stack traces for each syscall
# -f follows forked processes

Observability Checklist

System Call Monitoring:

• Enable auditd rules to log specific system calls
• Monitor for unexpected system call patterns
• Track seccomp filter violations

Capability Monitoring:

• Regularly audit capability assignments with getpcaps
• Alert on unexpected capability changes
• Document capability requirements for each service

Kernel Parameters:

• kernel.yama.ptrace_scope - Control ptrace visibility
• kernel.dmesg_restrict - Restrict kernel message access
• fs.suid_dumpable - Control core dump behavior for setuid programs

Configuration Validation:

• Test seccomp filters in staging before production
• Verify capability drops after privilege operations
• Validate namespace configurations for containers

Security/Compliance Notes

Defense in Depth: Never rely on a single security mechanism. Layer seccomp filtering, capabilities, and mandatory access controls for comprehensive protection.

Minimal Privilege Principle: Grant only the capabilities absolutely required for a service to function. Additional privileges increase attack surface and potential damage from compromise.

Audit Trail Requirements: Many compliance frameworks require logging of privileged operations. Configure auditd to log security-relevant system calls and capability usage.

# Audit rule to log capability changes
echo '-w /proc/self/status -p wa -k cap_change' >> /etc/audit/rules.d/capability.rules

# Audit rule to log seccomp violations
echo '-a always,exit -F arch=b64 -S seccomp -k seccomp' >> /etc/audit/rules.d/seccomp.rules

Container Security Context: When running containers in production, use security contexts to restrict capabilities and system calls.

apiVersion: v1
kind: Pod
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      securityContext:
        capabilities:
          drop:
            - ALL
          add:
            - NET_BIND_SERVICE

Common Pitfalls / Anti-patterns

Allowing execve Under Seccomp: BPF filters cannot inspect the arguments of execve after the call completes (the new program has already started). If a filter allows execve, a compromised process can execute any program. Block execve or carefully audit the new program’s requirements.

Capability Leaks After Fork: When a privileged process forks, child processes inherit capabilities. If the child drops privileges incorrectly, it may retain capabilities it should not have. Always re-evaluate capabilities after fork() and execve().

Ignoring Ambient Capabilities: Some capabilities persist across execve of programs that are not file-capability aware. The securebits flags control whether capabilities survive execve. Misunderstanding these flags leads to security bypasses.

Seccomp Filter Performance: While BPF filtering has minimal overhead, overly complex filters with many rules can add latency. Optimize filters by grouping similar rules and using jumps efficiently.

TOCTOU in Capability Checks: Time-of-check-time-of-use vulnerabilities can occur when capabilities are checked before an operation but the operation happens later with different privileges. Structure code to perform operations immediately after privilege acquisition.

Quick Recap Checklist

• System calls are the boundary between user space and kernel space
• Seccomp BPF filtering allows fine-grained control over allowed system calls
• Capabilities split root privileges into discrete units
• Least privilege principle means granting minimum necessary access
• Container runtimes use these mechanisms for isolation
• Production failures often involve blocked required system calls
• Comprehensive testing in staging prevents deployment issues
• Multiple security mechanisms should be layered together
• Audit logging supports compliance and incident investigation
• Understanding these mechanisms is essential for container security

Interview Questions

scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_KILL);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit), 0);
seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(openat), 0);
seccomp_load(ctx);

Further Reading

• Concurrency Fundamentals — The problem space and why synchronization is needed
• Mutex Implementation — How mutexes are implemented in userspace and kernel
• Semaphores — Counting semaphores for resource management
• Readers-Writer Locks — Optimizing for read-heavy workloads
• Lock-Free Structures — Advanced techniques for highly concurrent systems

Conclusion

System call security sits at the boundary between user space and kernel space. Seccomp BPF, capabilities, and namespace isolation together form the foundation of container and kernel security. The principle of least privilege should guide every decision you make here: grant only the capabilities actually needed, filter only the syscalls actually used. These mechanisms only work as a layered defense because each has blind spots that others cover.

If you want to dig deeper, the BPF verifier source is surprisingly readable, seccomp user notifications enable some clever supervisor patterns, and studying how containerd implements syscall filtering will show you how the theory translates to production code.