Async-Profiler: Low-Overhead CPU and Memory Profiling

Async-profiler has become the go-to tool for JVM profiling in production. Unlike traditional profilers that rely on JVMTI instrumentation with its associated overhead, async-profiler uses operating system-level instrumentation to capture stack traces without modifying running code. The result is profiling data that accurately represents what the JVM is doing, with overhead typically under 5% even for detailed CPU profiles.

This tool is not new, but it is still underutilized. Many teams struggle with production performance issues because their profilers introduce too much overhead to run on live systems. Async-profiler removes that barrier.

Introduction

Async-profiler is a production-grade profiling tool that captures CPU and memory allocation data without the overhead that makes traditional JVMTI-based profilers unusable in live systems. Where conventional profilers halt application threads to inspect their state, async-profiler relies on operating system signals and JVM internal APIs to collect stack traces with typically under 5% overhead. This makes it practical for investigating performance issues in production environments where introducing significant latency would mask the problems you are trying to diagnose.

The tool has become the de facto standard for JVM profiling because it produces flame graphs that make hot code paths immediately visible, works without code changes or restarts, and handles both CPU profiling and allocation analysis from the same agent. It addresses a persistent problem in Java production environments: the performance bottleneck you need to find only manifests under real load, yet adding profiling instrumentation changes the behavior you are trying to observe. By staying under 5% overhead even for detailed CPU captures, async-profiler allows you to profile production traffic without creating a self-fulfilling prophecy.

This post covers when async-profiler is the right tool and when alternatives like JProfiler or YourKit are better suited, explains how the agent uses OS signals and hardware counters to capture stack traces without modifying running code, and provides practical commands for CPU profiling, allocation analysis, and memory leak detection. You will also see real failure scenarios where async-profiler identified JIT compilation hotspots, lock contention, and allocation patterns causing GC pressure.

When to Use Async-Profiler / When Not to Use Async-Profiler

Async-profiler shines when you need accurate CPU flame graphs without halting application threads. It works well for identifying which code paths consume the most CPU, finding lock contention bottlenecks, and analyzing allocation patterns in production. The ability to profile without restart makes it invaluable for investigating issues in live systems.

Do not use async-profiler when you need method-level allocation byte counts or object retention graphs—YourKit handles those better. For very short-lived processes, the startup overhead may dominate. And if you need IDE integration for step-by-step debugging, JProfiler or IntelliJ profiler remain more convenient despite their overhead.

How Async-Profiler Works

graph TD
    subgraph "Operating System"
        PMU[Performance Monitoring Unit<br/>Hardware Counter Interrupts]
        Signals[OS Signals<br/>SIGPROF, SIGURG]
    end

    subgraph "JVM"
        JVMTI[JVMTI Agent<br/>Async-profiler Agent]
        CodeCache[Code Cache<br/>JIT Compiled Code]
        Threads[Java Threads<br/>Running State]
    end

    subgraph "Async-Profiler"
        Coll[Collector Thread<br/>Stack Walking]
        Buffer[Circular Event Buffer]
        Converter[Flame Graph Converter]
    end

    PMU -->|"Hardware events"| Signals
    Signals -->|"signal handler"| JVMTI
    JVMTI -->|"walk stacks"| Threads
    JVMTI --> Coll
    Coll --> Buffer
    Buffer --> Converter

Async-profiler registers signal handlers with the OS. When the PMU triggers an interrupt or the configured interval elapses, the OS delivers a signal to a Java thread. The signal handler captures the thread's stack trace by walking the stack using JVM internal APIs and native frame walking. This happens without cooperation from Java code itself, which is why overhead stays low. The collector thread aggregates these traces and converts them to flame graph format for analysis.

## Installation and Setup

```bash
# Download async-profiler release
wget https://github.com/async-profiler/async-profiler/releases/download/v2.9/async-profiler-2.9-linux-x64.tar.gz
tar xzf async-profiler-2.9-linux-x64.tar.gz

# Verify installation
cd async-profiler-2.9
./ profiler.sh --version

# Attach to running JVM (requires pid)
./ profiler.sh <pid> -o flamegraph.svg --duration 30

For Java 21 and later, async-profiler works with both x86 and ARM architectures. Older versions may require building from source for certain platforms.

CPU Profiling

CPU profiling captures stack traces at regular intervals to build a picture of where CPU time goes:

# Basic CPU profiling for 30 seconds
./profiler.sh -d 30 -e cpu <pid>

# CPU profiling with frame rate limit (reduce overhead further)
./profiler.sh -d 30 -e cpu --framebuf=4096 <pid>

# CPU profile with method filtering (only include specific packages)
./profiler.sh -d 30 -e cpu --include 'com/mycompany/**' --exclude 'java/**' <pid>

# Output as flame graph
./profiler.sh -d 30 -e cpu -o flamegraph <pid>

The -o flamegraph option produces an interactive SVG flame graph that you can open in a browser. The width of each frame represents the proportion of CPU time spent in that method. The vertical axis shows the call stack depth.

Allocation Profiling

Async-profiler can instrument object allocations without the massive overhead of JVMTI allocation profiling:

# Profile allocations over 30 seconds
./profiler.sh -d 30 -e alloc <pid>

# Allocation profiling with live object tracking
./profiler.sh -d 30 -e alloc --liveobjgraph <pid>

# Sample allocations by size (only track allocations >16KB)
./profiler.sh -d 30 -e alloc --alloc-thresh=16384 <pid>

Allocation profiling works by intercepting the slow path in object allocation—the code path taken when TLAB (Thread Local Allocation Buffer) is full or allocation requires JVM intervention. This is only a fraction of total allocations but provides statistically representative sampling with minimal overhead.

Memory Leak Detection with Heap Live Data

The --liveobjs mode tracks live objects that survive garbage collections:

# Capture live object set (requires Full GC to trigger)
./profiler.sh -d 30 --liveobjs <pid>

# Compare two heap dumps to find growing objects
./profiler.sh --liveobjs --base=<pid1> <pid2>

This mode walks the heap after a GC to find all live objects and their allocation sites. Comparing two captures reveals which objects are growing over time—strong evidence of memory leaks.

Reading Flame Graphs

A flame graph shows call stacks horizontally. Each wide block represents a method. The vertical stacking shows parent calls. Reading from bottom to top traces a specific call path. Reading horizontally shows which methods share the same parent.

# Sample flame graph output format (text version)
- profile
  - Runtime.optimize
    - Compiler.runCompilation
      - MyService.process [5000 samples]
        - MyServiceHelper.calculate [3000 samples]
        - MyServiceHelper.validate [2000 samples]
    - MyService.process [2000 samples]

The numbers in brackets show sample counts. In this example, MyService.process appears 7000 times total, but only 2000 of those samples show it directly under profile—the other 5000 samples show it called through Runtime.optimize and Compiler.runCompilation, indicating JIT compilation is happening during processing.

Production Failure Scenarios

Scenario 1: Identifying JIT Compilation Hotspots

A service exhibits periodic latency spikes that correlate with certain code paths being executed for the first time or after a while. The flame graph shows Runtime.compileMethod or Interpreter.compile consuming significant time.

This happens because JIT compilation is expensive. Methods that have not been compiled yet run through the interpreter. Methods that were compiled but not used recently may have their compiled code invalidated. The first request after a deployment always pays compilation cost.

Solution: Warm up the application before sending traffic. Use -XX:CompileCommand=option to force compilation of critical methods. Consider tiered compilation settings that prioritize frequently-used methods.

Scenario 2: Finding Lock Contention in Concurrent Code

An application shows high CPU utilization but low throughput, suggesting threads are waiting rather than working. The -e lock profiler mode captures lock contention:

./profiler.sh -d 30 -e lock <pid>

The resulting flame graph shows which code paths contend for locks most frequently. Often this reveals design issues where too many threads compete for the same lock, or where lock-free algorithms could replace locking implementations.

Scenario 3: Allocation Rate Problems

A service experiencing GC pressure shows allocation hot spots in flame graphs. The alloc mode reveals which code creates the most objects:

./profiler.sh -d 60 -e alloc -o flamegraph <pid>

The flame graph shows which call paths allocate memory most frequently. Focus optimization efforts on those paths—reducing unnecessary object creation directly reduces GC pressure.

Trade-off Table

Mode	Overhead	What It Shows	Limitations
CPU	2-5%	Hot code paths	Requires sampling interval tuning
Alloc	3-8%	Allocation hot spots	Only samples TLAB refill path
Lock	1-3%	Lock contention	Only shows contended locks
Live Objects	10-20%	Live object graph	Requires Full GC

Implementation Snippets

Programmatically Starting Async-Profiler

import java.lang.management.*;
import java.io.*;
import sun.management.*;

public class AsyncProfilerControl {

    public static void main(String[] args) throws Exception {
        // Get the VM management interface
        HotSpotDiagnosticMXBean vmDiag =
            ManagementFactory.getHotSpotDiagnosticMXBean();

        // The profiler runs as a native agent
        // Control it via JCMD for a running process:
        // jcmd <pid>Profiler.start duration=30s

        // For programmatic control, use Attach API
        String pid = ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
        System.out.println("Process ID: " + pid);

        // The profiler is controlled externally via the profiler.sh script
        // or by attaching with Attach API
        String profilerversion =
            vmDiag.getVMOption("CompilerConfigFile").getValue();
        System.out.println("Profiling requires external profiler.sh or API integration");
    }
}

Container Environment Profiling

#!/bin/bash
# Profile Java process in Kubernetes container
PID=$(pgrep -f "your-application.jar" | head -1)

if [ -z "$PID" ]; then
    echo "Java process not found"
    exit 1
fi

# Get container memory limits for appropriate frame buffer size
MEM_LIMIT=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null || echo "4294967296")
FRAMEBUF=$((MEM_LIMIT / 100))  # 1% of memory for frame buffer

# Run async-profiler inside container (requires container with async-profiler installed)
docker exec your-container \
    /opt/async-profiler/profiler.sh \
    -d 60 \
    -e cpu \
    --framebuf=$FRAMEBUF \
    $PID \
    -o flamegraph \
    > /tmp/flamegraph.svg

echo "Flame graph written to /tmp/flamegraph.svg"

Observability Checklist

Before concluding profiling is complete, verify these points. The profiling duration captured enough time to see stable patterns—30 seconds minimum for frequently-executed code. The sampling interval was appropriate—too frequent creates overhead, too sparse misses important events. The JVM was warmed up; cold code paths during profiling skew results. CPU and memory metrics correlated with profiling findings—confirms findings are real.

Also check that the JVM did not undergo GC during CPU profiling that could skew results, and that the workload during profiling matched the production behavior you are investigating.

Security and Compliance Notes

Async-profiler has significant capabilities that should be restricted. The ability to capture stack traces reveals application internals, including class names, method names, and code paths. This information aids attackers in understanding your application’s architecture.

On production systems, restrict profiler access to authorized operations personnel. Use Linux capabilities to limit which users can attach profilers. Consider profiling-only environments that do not run production traffic.

The --liveobjs mode reveals object graphs that may expose sensitive data structures. Handle these profiles with the same care as heap dumps.

Common Pitfalls / Anti-Patterns

The most common mistake is profiling for too short a duration. A 5-second profile captures only what happened in those 5 seconds. If traffic is bursty, you may miss important code paths. Always profile for at least 30 seconds, preferably several minutes or across multiple load cycles.

Another issue is incorrect interpretation of wide frames. A wide frame in a CPU flame graph shows the most samples, but that does not always mean the biggest problem. Look for unexpected patterns, like library code consuming significant time, which might indicate inefficient usage rather than a problem in your code.

Forgetting to exclude idle orGC threads skews results. JVM threads like GC task thread and VM Thread should typically be excluded from analysis unless you are investigating GC issues specifically.

Finally, not matching the workload during profiling. Profiling a development workload against production architecture gives misleading results. Ensure the system state during profiling represents what you are trying to diagnose.

Quick Recap Checklist

• Install async-profiler on your production hosts before you need it
• Use -e cpu for CPU hot spots, -e alloc for allocation issues
• Profile for at least 30 seconds, preferably during actual production load
• Read flame graphs from bottom to top to trace call paths
• Use --liveobjs to find memory leaks with allocation site information
• Exclude GC threads when investigating CPU issues
• Protect profiler access—it exposes application internals

Interview Questions

Further Reading

• async-profiler GitHub Repository — Official project page with releases and documentation
• async-profiler Wiki — Comprehensive wiki with usage examples and troubleshooting
• FlameGraph Project — Brendan Gregg’s flame graph tools that async-profiler integrates with
• [Profiling Java in Production](https:// brincomp.github.io/posts/redefining-profiling/) — Advanced techniques for production JVM profiling
• Java Performance Mastery — Book covering async-profiler and other JVM profiling tools

Conclusion

Async-profiler has become the standard for low-overhead production profiling because it uses OS-level signals rather than JVMTI instrumentation. For CPU profiling, use -e cpu with flame graph output. For allocation issues, use -e alloc. For lock contention, use -e lock. The --liveobjs mode provides memory leak detection with allocation sites.

The key to effective async-profiler usage is profiling during representative production load for at least 30 seconds. Read flame graphs from bottom to top to trace call paths, and exclude GC threads when investigating CPU issues. Async-profiler works without code changes or restarts, making it invaluable for investigating issues in live systems.