Trace kernel code tips

Small Talk about Dynamically Trace Code

Viller Hsiao <villerhsiao@gmail.com>

Dec. 14, 2014

12/14/14 2/38

Attribution – ShareAlike 3.0

You are free

● to copy, distribute, display, and perform the work

● to make derivative works

● to make commercial use of the work

Under the following conditions

– Attribution. You must give the original author credit.

– Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.

For any reuse or distribution, you must make clear to others the license terms of this work.

● Any of these conditions can be waived if you get permission from the copyright holder.

Your fair use and other rights are in no way affected by the above.

License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode

Corrections, suggestions, contributions and translations are welcome!

Rights to Copycopyright © 2014 Viller Hsiao

12/14/14 3/38

Who am I ?

Viller Hsiao

source url: http://goo.gl/n7NAfS

Embeded Linux / RTOS engineer

2007 ~ Present Realtek Corp. 2003 ~ 2007 BenQ/Qisda

12/14/14 4/38

An Embedded Programmer

source url: http://ext.pimg.tw/aabb52043/1367377582-3526768649_n.png

makefile menuconfigmakefile

gcc

armcc

ld objdump

memorymanagement Linker

script

HardwareSpec

uclibc

newlib

glibc

musl

uboot

kbuild

12/14/14 5/38

Topic Today

How to trace C sources environment

We have cross reference tools.ctags, cscope, LXR, source insight ... and so on

BUT

12/14/14 6/38

Difficulties of Tracing C/C++ Sources

● Preprocessor related● Too much #if/#elif/#endif ● Too much level of macros● Configurable included files

12/14/14 7/38

Difficulties of Tracing C/C++ Sources (Cont.)

● Runtime related● Some debugging and error handling stuff● Multi-task environment● Hard to realize HW state change

Different task context

12/14/14 8/38

Topic Today

How to trace C sources environment

Dynamically

Trace code from runtime execution result(Mainly in linux kernel source)

12/14/14 9/38

Tracing from real execution log

Provides clues to trace code

http://www.clker.com/cliparts/c/c/7/p/7/4/earth-hi.pnghttp://goo.gl/4PnwFe

12/14/14 10/38

Agenda Today

● Build environment● Config system● Makefile / kbuild● Toolchain tips

● Runtime trace● Qemu log● Linux ftrace

12/14/14 11/38

menuconfig

● Too much words in menuconfig● Hard to map definition to code world (CONFIG_XXX)

12/14/14 12/38

menuconfig Tips – Search

● Press '/' to search specific configurations

12/14/14 13/38

menuconfig Tips – Search (Cont.)

Dependency

Path in menu

● Search result

Kconfig path

12/14/14 14/38

menuconfig Tips – Result output

● Linux/.config## Automatically generated file; DO NOT EDIT.# Linux/arm 3.14.25 Kernel Configuration#CONFIG_ARM=yCONFIG_SYS_SUPPORTS_APM_EMULATION=y

## General setup#CONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""# CONFIG_COMPILE_TEST is not setCONFIG_LOCALVERSION=""# CONFIG_LOCALVERSION_AUTO is not setCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_LZ4=yCONFIG_KERNEL_GZIP=y# CONFIG_KERNEL_LZMA is not set# CONFIG_KERNEL_XZ is not set

## Automatically generated file; DO NOT EDIT.# Linux/arm 3.14.25 Kernel Configuration#CONFIG_ARM=yCONFIG_SYS_SUPPORTS_APM_EMULATION=y

## General setup#CONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""# CONFIG_COMPILE_TEST is not setCONFIG_LOCALVERSION=""# CONFIG_LOCALVERSION_AUTO is not setCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_LZ4=yCONFIG_KERNEL_GZIP=y# CONFIG_KERNEL_LZMA is not set# CONFIG_KERNEL_XZ is not set

12/14/14 15/38

Tips of Tracing makefile

● make V=1● make --dry-run

12/14/14 16/38

Tips of Tracing makefile (Cont.)

● make --debug

12/14/14 17/38

Tips of Tracing Shell Script

● #!/bin/sh “-x”

● Result

#!/bin/sh -x

FOO=something_produced_from_command

if [ x${FOO} != x'' ]; then echo "get foo"else echo "no foo"fi

#!/bin/sh -x

FOO=something_produced_from_command

if [ x${FOO} != x'' ]; then echo "get foo"else echo "no foo"fi

+ FOO=something_produced_from_command+ '[' xsomething_produced_from_command '!=' x ']'+ echo 'get foo'get foo

+ FOO=something_produced_from_command+ '[' xsomething_produced_from_command '!=' x ']'+ echo 'get foo'get foo

12/14/14 18/38

Tips of Tracing Macros/Definitions

● gcc options: “-save-temps” or “-E”

● Use astyle to beauty result

$ astyle -A8 main.i$ astyle -A8 main.i

$ gcc -save-temps -o main main.c

$ lsmain main.c main.i main.o main.s Makefile

$ gcc -save-temps -o main main.c

$ lsmain main.c main.i main.o main.s Makefile

12/14/14 19/38

Tips of Tracing Macros/Definitions (Cont.)

12/14/14 20/38

Tips of Structure Definitions

● Point to Real definition in the headers

# 5 "include/linux/linkage.h" 2# 1 "include/linux/stringify.h" 1# 6 "include/linux/linkage.h" 2# 1 "include/linux/export.h" 1# 26 "include/linux/export.h"

struct kernel_symbol{ unsigned long value; const char *name;};

# 5 "include/linux/linkage.h" 2# 1 "include/linux/stringify.h" 1# 6 "include/linux/linkage.h" 2# 1 "include/linux/export.h" 1# 26 "include/linux/export.h"

struct kernel_symbol{ unsigned long value; const char *name;};

12/14/14 21/38

Kbuild Tips

● Add compile flags for some object

● Remove compile flags for some object

● All subdirectory

CFLAGS_foo.o = -O0 -g -save-tempCFLAGS_foo.o = -O0 -g -save-temp

CFLAGS_REMOVE_foo.o = -OsCFLAGS_REMOVE_foo.o = -Os

subdir-ccflags-y = -save-tempssubdir-ccflags-y = -save-temps

12/14/14 22/38

Qemu Tips

Setup the search rule

● Qemu log (“-d op -D /path/to/qemu.log”)

12/14/14 23/38

Qemu Tips (Cont.)

Example: get first user space entry point

Setup the search rule

● Qemu log + regular expression

12/14/14 24/38

Qemu Tips (Cont.)

● Other tips● Qemu log + disassembly (objdump)

● Qemu + gdb ( + qemu log)– Breakpoint / stack trace– Qemu options

● -s or -gdb tcp::2345: enable gdb port● -S: freeze CPU at startup

12/14/14 25/38

Ftrace – Overview

● Kernel function tracer ( > 2.6.28 ?)● Record execution log of kernel function● Implemented by “-pg”

– Use dynamic ftrace to reduce overhead– Add “-g -O0” to export more information

● Accessing API – debugfs– /path/to/debug/tracing/

mount -t debugfs none /sys/kernel/debugmount -t debugfs none /sys/kernel/debug

12/14/14 26/38

Ftrace – Configs

12/14/14 27/38

Ftrace – Steps

[tracing]$ echo “*mtd*” > set_ftrace_filter

[tracing]$ echo “*blk*” >> set_ftrace_filter

[tracing]$ echo “*:mod:ext3” >> set_ftrace_filter

[tracing]$ echo function > current_tracer

[tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on;

[tracing]$ cat trace

[tracing]$ echo “*mtd*” > set_ftrace_filter

[tracing]$ echo “*blk*” >> set_ftrace_filter

[tracing]$ echo “*:mod:ext3” >> set_ftrace_filter

[tracing]$ echo function > current_tracer

[tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on;

[tracing]$ cat trace

12/14/14 28/38

Ftrace – Available Tracer

[tracing] $ cat available_tracers

function

function_graph nop

[tracing] $ cat available_tracers

function

function_graph nop

12/14/14 29/38

ftrace

● function tracer

12/14/14 30/38

ftrace

● function tracer

12/14/14 31/38

ftrace

● function_graph tracer

12/14/14 32/38

ftrace

● function_graph tracer

Taskswitch

12/14/14 33/38

ftrace

● Better front-end● trace-cmd● kernelshark

12/14/14 34/38

Ftrace for Code Tracing

● Ftrace provides clues of● Per-module view● Top-down view● Procedure excution flow● Tasks/Threads model relationship

12/14/14 35/38

Ftrace – Other tracer

● Trace event● ftrace + tracepoint

● Probe event● ftrace + kprobe

● Other tracer● Stack tracer● Irqoff latency tracer● Syscall tracer

● And so forth … but not the topic today

12/14/14 36/38

Conclusion

● Trace code● Cross reference tool helps● Drill-down trace

– Start from interface functions● Filesystem hooks: open/read/write/ioctl ...● probe/exit● ISR entry● Architecture interface

– Global functions: EXPORT_SYMBOLS● Execution log provide some hints, too.

– Log from build flow– Qemu instruction trace log– Kernel ftrace

12/14/14 37/38

Reference

[1] Building embedded linux demo case

i. Sherif Mousa (Mar 26, 2014), “Building Embedded Linux”

ii. Linaro wiki, “cross-NG”

[2] Linux-3.14 documentation,

i. “kbuild/makefile.txt”

ii. “trace/ftrace.txt”

[3] 池田宗廣，大岩尚宏，島本裕志，竹步晶雄，平松雅巳 (Jan. 2014), “Linux kernel hacks”

[4] man of

i. astyle

ii. qemu

iii. Gnu make

12/14/14 Viller Hsiao

THE END