Tracepoints are static, predefined hooks in kernel code that are always compiled in (but can be disabled). kprobes are dynamic instrumentation that attaches to any kernel function entry or any instruction. uprobes are the user-space equivalent, attaching to user-space binary functions. Tracepoints have lowest overhead and stable interfaces; kprobes/uprobes are more flexible but require knowing exact symbol addresses.

Modern CPUs mispredict branches and pipeline instructions speculatively. When a profiler takes a sample during speculative execution, you may see functions in stack traces that weren't actually on the critical path. Use perf record - branches to filter for only taken branches, or rely on last branch record (LBR) when available. Always profile with realistic data that exercises real branch behavior, not synthetic test cases.

Modern CPUs can execute multiple instructions per cycle via pipelining and superscalar execution. "Cycles" counts actual clock cycles elapsed, while "instructions" counts retired instructions. The ratio (cycles per instruction, CPI) reveals pipeline efficiency. If cycles exceed time significantly on a multi-core system, it means the workload is using multiple cores in parallel—each core contributing its own cycles.

eBPF (extended Berkeley Packet Filter) allows sandboxed programs to run in the kernel without modifying kernel source or loading modules. Programs are verified for safety before execution and can be attached to kprobes, tracepoints, or network interfaces. This enables custom metrics collection, networking, and security enforcement—all without kernel patches. BCC provides high-level languages (Python, Lua) that compile to eBPF bytecode.

Statistical sampling at 99Hz or 999Hz cannot reliably capture microsecond-scale functions—they may not appear in any sample. Instead, use: (1) perf stat -e with cache miss or branch misprediction events that correlate with the function; (2) instruction counting via perf stat -e instructions; (3) trace the specific function with ftrace and its graph mode to measure individual calls; (4) consider if the function needs algorithm optimization rather than just measurement.

Hardware PMCs (Performance Monitoring Counters) are built into the CPU and count events like cycles, instructions, cache misses, and branch mispredictions—they require minimal overhead to read. Software counters are implemented by the kernel via tracepoints and kprobes; they count events like context switches, page faults, and scheduler decisions. Hardware counters provide lower overhead and higher precision for CPU-bound analysis; software counters provide visibility into kernel activities that have no hardware equivalent.

When a PMC overflows (reaches its maximum count), the kernel receives an interrupt and records a sample with the current instruction pointer. For continuous monitoring with overflow handling, use perf record -f to use frequency-based sampling where the kernel reprograms the counter after each sample rather than waiting for overflow. The perf_event_attr::watermark setting controls when the overflow interrupt fires. In multi-threaded programs, the kernel uses inherit settings to decide whether children inherit the parent's events.

perf uses a per-CPU ring buffer (struct ring_buffer) to transfer samples from kernel to user space. When perf records, itmmaps the buffer and the kernel writes samples directly into it without syscalls per sample—only a single mmapped region update. When the buffer is full, perf loses samples (in overwrite mode) or blocks (in per-CPU mode). Using perf record -m 2 increases the buffer size to reduce sample loss in high-frequency profiling scenarios.

ftrace excels at function-level tracing with minimal overhead—its function tracer can be enabled with just echo function > current_tracer. perf is better for hardware event counting and call graph analysis. Choose ftrace when you need: (1) function-level visibility with low overhead, (2) kernel function latency via function_graph, (3) tracepoint filtering with event parameters, (4) dynamic instrumentation with kprobe events. Choose perf when you need hardware metrics, call stacks, or userspace tracing.

Flame graphs show relative time spent in each function stack—wider boxes mean more samples. The vertical axis shows ancestry (caller on top, callee below); horizontal position has no meaning. To interpret: (1) find the widest blocks—these are your hottest paths; (2) read from root to leaf to understand call sequences; (3) compare flame graphs before/after changes to verify improvement; (4) look for "plateaus" of similar frame widths—these often indicate a bottleneck where all callers converge. The colors are typically cosmetic unless you enable hue-by-attribute.

On-CPU profiling samples where the CPU is actively executing code—useful for CPU-bound workloads where the bottleneck is computation. Off-CPU profiling (also called wakeup profiling) samples when threads are NOT running—useful for I/O-bound workloads, lock contention, or scheduling issues where threads block. Use perf record -a -g -- sleep 10 for on-CPU; use perf record -a -g --call-graph=fp -e sched:sched_switch -c 1 for off-CPU analysis. Off-CPU flame graphs often reveal "stacks lying down"—blocked threads show their wait channels (like futex, read, poll) as the blocking call, revealing I/O or lock bottlenecks.

LBR is a hardware feature that stores the last N (typically 16-32) branch records including source and destination addresses. Unlike standard sampling that interrupts periodically and captures only the current instruction pointer, LBR captures actual taken branches with their destinations. This is crucial because: (1) it eliminates speculation noise—branches that were mispredicted and flushed from pipeline don't appear as real work; (2) it provides precise call chains without frame pointers; (3) it enables perf record -j any,u -- branches to capture all branch events efficiently. Enable with perf record -j any,u -a and view with perf report --branch-history.

Cache misses don't directly measure memory latency—they measure miss events. When a cache miss occurs, the CPU waits (stalls) for data from memory, but a single cache-misses counter event doesn't tell you how long each stall lasted. To correlate misses with latency: (1) use perf stat -e cycles,cache-misses,cache-references and calculate miss rate; (2) combine with stalled-cycles-frontend and stalled-cycles-backend to see how much of the pipeline is waiting; (3) use perf record -e cache-misses -W with weight to capture miss latency; (4) consider using memory-latency events on Intel (cbo_X events) for actual memory access latency histograms.

CPI (Cycles Per Instruction) = cycles / instructions. A CPI of 1.0 means the CPU ideally completed one instruction per cycle (superscalar execution can exceed 1.0). CPI > 1 indicates stalls—typically: (1) CPI 1.0-2.0: normal for memory-bound workloads with some cache misses; (2) CPI 2.0-4.0: significant cache miss penalties or memory latency; (3) CPI > 4.0: severe memory bottleneck, possible cache thrashing or non-optimized memory access patterns. For multi-core systems, also check instructions per cycle (IPC)—if cores are parallelized, total cycles increase but each core may show different CPI. A high system-wide CPI with low CPU utilization indicates I/O waiting rather than computation stalls.

Dynamic module loading creates profiling challenges because symbols may not be available when perf isn't running. Solutions: (1) Build modules with debug symbols and install to /lib/modules/$(uname -r)/build—perf will find them automatically; (2) Use module.symbols file to map dynamic addresses to names after the fact; (3) Persistently enable tracing with ftrace before running workload—ftrace can capture module function entries even without symbol resolution; (4) Use BPF with kprobe on module functions: BPF programs attached to module symbols will auto-load when the module loads; (5) For post-mortem analysis, use perf record -k mono -g with mono (monotonic clock) for ordering.

AMD's IBS (Instruction-Based Sampling) provides more detailed profiling than standard PMCs. IBS periodically triggers based on: (strong>Fetches—instruction fetch cycles andop information; Ops—micro-ops retired and their characteristics. Unlike standard PMCs which count events (cache misses, branches), IBS provides exact instruction addresses that caused events. Benefits: (1) more precise hotspot identification without frame pointer dependency; (2) micro-op level visibility for AMD Zen architectures; (3) better handling of instruction cache misses. Access via perf record -e ibs_fetch/.../ -e ibs_op/.../. Intel equivalents are PEBS (Precise Event-Based Sampling) and LBR (Last Branch Record).

False sharing occurs when threads on different cores modify variables that share the same cache line, causing unnecessary cache invalidations. Profiling indicators: (1) High cache-misses with low memory traffic—many invalidations but not much data transferred; (2) perf stat showing high cache_misses and store_blocks ld_flood events; (3) High context switch rates between threads that access related data structures. Confirmation: use perf c2c record (cache-to-cache analysis) which shows cache line contention events including ld_l1hit, ld_l2hit, stores with RFO (Read-For-Ownership) that indicate cache line bouncing. Fix: pad data structures to ensure each thread's data is cache-line-aligned.

--call-graph dwarf (default on modern systems) uses DWARF debug information to unwind the stack—it reads saved registers and stack frame pointers from memory to reconstruct call chains. More accurate but requires debug symbols (-g at compile time) and has higher overhead. --call-graph fp uses frame pointer chaining—each stack frame contains a pointer to the previous frame. Faster but only works if code was compiled with -fno-omit-frame-pointer. Modern GCC defaults to -fomit-frame-pointer for optimized builds, making fp mode unreliable. For containerized environments where debug symbols may not be available locally, use --call-graph dwarf with separate debuginfo packages.

The perf_event_paranoid sysctl controls access to performance events for unprivileged users:

• -1: No restrictions (cap_sys_admin bypasses all checks)
• 0: Allow all user access to performance events
• 1: Disallow raw tracepoint access (still allows hardware PMCs)
• 2+: Disallow all performance events except cycle counts

In containers, this is often set to restrictive values. If perf_event_paranoid >= 2, most hardware counters won't work. Solutions: (1) run container with --cap=SYS_ADMIN (not recommended for security); (2) use perf record -P (privilege-level filtering) or profile in a VM; (3) use BPF with CAP_PERFMON (kernel 5.8+) which has finer-grained control.

BPF maps can exhibit staleness when the kernel is in the middle of updating them while userspace is reading. Key issues: (1) Lookup vs Update race: bpf_map_lookup_elem() may return a value that's being concurrently modified; (2) Incremental updates: when aggregating in-kernel, userspace might see partial results. Mitigations: (1) use BPF_F_NO_PREALLOC for atomic map allocation; (2) use bpf_percpu_hash for per-CPU aggregation which avoids cross-CPU synchronization; (3) implement double buffering in userspace—read from one map while the kernel writes to another, then swap; (4) use __sync_fetch_and_add for atomic increments. For production tracing, consider using ring_buffer (available in Linux 5.8+) which supports efficient, lock-free data transfer.