strace for Real Troubleshooting

Introduction¶

When a program fails with a useless error, hangs, or is mysteriously slow, strace shows you the exact system calls it's making — which file it can't open, which host it's connecting to, where it's blocked. The trick is filtering so you see the signal.

The essential invocations¶

What files is it touching?¶

strace -f -e trace=file -y -p <pid>
# -f  follow child processes
# -e trace=file   only file-related syscalls (open, stat, access, ...)
# -y  print the path behind each fd

Great for "config not found" — you'll see openat(AT_FDCWD, "/etc/app/config.yml", O_RDONLY) = -1 ENOENT and know it's looking in the wrong place.

What is it connecting to?¶

strace -f -e trace=network -y -p <pid>
# connect(3, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.9")}, 16) = -1 ECONNREFUSED

Why is it slow / where is it blocked?¶

strace -f -T -tt -p <pid>
# -T   time spent in each syscall (in <...>)
# -tt  wall-clock timestamp on each line

A single read(), poll(), futex(), or connect() with a huge <...> time is your answer — waiting on I/O, a lock, or a slow peer.

Summary mode — where does the time actually go?¶

strace -f -c -p <pid>      # then Ctrl-C after a representative period
# % time     seconds  usecs/call     calls    errors syscall
#  61.2     0.412000        4120       100        50 stat
#  22.1     0.149000        1490       100           read

stat with 50% errors and half the time = it's stat-ing files that don't exist in a loop (a bad search path, a cache miss storm).

Start a program under strace¶

strace -f -e trace=file -o /tmp/trace.log ./myapp --args

-o writes to a file so program stdout isn't mixed in.

Reading the output¶

• Return value after =. Negative with a name (-1 ENOENT, -1 EACCES, -1 ECONNREFUSED) is the failure and its reason.
• openat(... ) = 5 then later read(5, ...) — follow the fd number.
• = ? ERESTARTSYS (To be restarted ...) around a signal is normal.
• futex(..., FUTEX_WAIT, ...) for a long time = waiting on a lock/condvar (thread contention, or a deadlock if it never returns).

Verification and troubleshooting¶

• strace: attach: ptrace(PTRACE_SEIZE, ...): Operation not permitted — kernel.yama.ptrace_scope is 1+ (default on Ubuntu). Run as root, or sudo sysctl kernel.yama.ptrace_scope=0 (temporarily), or start the process under strace instead of attaching. In a container you need --cap-add=SYS_PTRACE and often --security-opt seccomp=unconfined.
• Output is a wall of poll/epoll_wait/futex — the process is idle, waiting for work. Filter with -e trace=file or -e trace=network to see only what you care about, or use -c summary mode.
• strace makes the program crawl — every syscall now context-switches to the tracer. Normal; it's a diagnostic, not a permanent thing. For lower overhead on syscall-heavy processes use perf trace or bpftrace.
• Nothing interesting in the trace, still hangs — the block might be inside the kernel (a D-state task on I/O) where strace can't see it. Check cat /proc/<pid>/stack and wchan instead.
• Statically linked / stripped binary, unclear calls — strace still shows syscalls (it's kernel-boundary), that doesn't change. For library calls use ltrace.
• Multi-threaded, output interleaved — -f traces threads too; add the PID/TID prefix with -ff -o prefix to split per-thread files.

Related tools and reading¶

• On-site: Linux commands reference, Port Checker.
• Related posts: Finding what is using a port, Understanding Linux load average.