The Path to Portable Kernel Instrumentation: The Kerninst API · April 29, 2003 -1- Kerninst API The Path to Portable Kernel Instrumentation: The Kerninst API Alex Mirgorodskii [email protected]

April 29, 2003 Kerninst API-1-

The Path to Portable Kernel Instrumentation:The Kerninst API

Alex [email protected]

Computer Sciences Department

University of Wisconsin-Madison


What is Kerninst API?

read(...)OS Kernel

User SpaceC++ library for writing OSkernel instrumentation tools

libkerninstAPI

Instrumentation

tool



• Generates instrumentation code

OS Kernel


Instrumentation

tool

Cnt++

read(...)

libkerninstAPI




• Loads it into the kernel

ld [r1],r2

add r2,1,r2

st r2,[r1]

OS Kernel


Instrumentation

tool

read(...)

libkerninstAPI




• Loads it into the kernel

• Modifies the kernel image tocall the code

OS Kernel


Instrumentation

tool

read(...)

libkerninstAPI

ld [r1],r2

add r2,1,r2

st r2,[r1]


Kerninst API Clients

Kperfmon: monitorkernel performance

Kernel

Kerninst API Kerninst API

NEC Tracer: insertcalls to tracing

routines at interestingplaces in the kernel

Hooks Interface

User SpaceOutliner: optimize the

kernel on the fly toreduce the I-cache

miss rate

I-$

Kerninst API

call Tracekfunc( ) kfunc( ) optfn( )

startTimer

stopTimer

kfunc( )


Basic Functionality

• Instrument– Construct code snippets

– Insert them in the kernel

• Browse through kernel resources– Kernel modules, functions and basic blocks

• Inspect/modify kernel data from the user space– Peek at snippet’s data

“Dyninst for the kernel”


Key Highlights• Architecture-independent interface

– Use the same instrumentation tool on any platform

– Ports to x86 and IBM Power are under way

• Low-overhead instrumentation– Sophisticated analysis for efficient code generation

– Single-branch instrumentation: great for profiling uses!

• Convenience– No need for kernel recompilation or reboot

• Compatibility with Dyninst API– Similar syntax

– Both libraries can co-exist in the same tool


Example: Incrementing a Counter

Cnt++

OS Kernel

User Space

Instrumentation tool

read(...)

libkerninstAPI

insert snippet sample data

main() {kapi_manager kmgr;kmgr.attach("hostname", portnum);kapi_point pt = kmgr.findPoint(”unix”, ”read”, ENTRY);kptr_t addr = kmgr.malloc(sizeof(kapi_int_t));kapi_int_variable intCounter(addr);

arith_expr addOne(kapi_assign, intCounter, arith_expr(kapi_plus, intCounter, kapi_const_expr(1))); int snippet_handle = kmgr.insertSnippet(addOne, pt);

kapi_int_t userCtr = 0;while (userCtr < 1000) {

sleep(1); kmgr.memcopy(addr, &userCtr, sizeof(kapi_int_t)); printf(“%d\n”, userCtr);

}kmgr.removeSnippet(snippet_handle);

}






}






}






}






}






}

=

i +

i 1






}






}






}






}


Advanced Features

• Support for remote operation– Instrument the kernel of another machine over the Net

• Support for low-overhead kernel data access– Syscall-free access to snippets’ data from the user space

• Support for reading hardware counters from theinstrumentation code– Dyninst does this by calling external functions (PAPI)

• Value-added library of useful instrumentations


Remote Operation: History

Kernel

Kperfmon

01101

01101

110011

Host A

Kerninst

daemon

Host B

• Moves the cost of the GUI, analysis and visualization toanother host

• Performs code analysis once, in the daemon


Remote Operation: Kerninst API

• Moves the cost of the GUI, analysis and visualization toanother host

• Performs code analysis once, in the daemon

• Can monitor multiple machines with one tool

Kernel

Instrumentation

tool

Kerninst API

01101

01101

110011

Host A

Kerninst

daemon

Host B


Low-Overhead Kernel Data Access

• kmgr.malloc allocates memory from the spacevisible both by the kernel and the client (mmap’ed)

• No system call overhead to access the data

• Plans to support it in Dyninst as well

Shared Segment

load [user_addr],reg

Kernel store reg,[kernel_addr]

User Tool


Accessing Hardware Counters

iMisses = arith_expr(kapi_assign, intCounter,

kapi_hwcounter_expr(ICACHE_MISSES));snippet_handle = kmgr.insertSnippet(iMisses, point);

=

i hwctr

Read hardware counters from the instrumentation

• No PAPI for the kernel

• Efficient: no function call overhead


Value-added library of usefulinstrumentations

Primitives implemented on top of the API:

• Virtualized timers– Do not run while the thread is switched-out

– Useful for most hardware counter events

• Tracking of calls through function pointers– Find callees invoked from a given call site

– Can walk the dynamic call graph of the kernel


Current Efforts

• New Architectures!– x86 and IBM Power

• Optimized remote communication– Lower resource utilization

• New Clients– Cross-kernel-boundary performance analysis


New Architectures: Status

Kernel-level support routines

Trap-based instrumentation

Code analysis

Jump-based instrumentation

ETA

Done

Done

In Progress

To Be Done

6 Months

To Be Done

Not required

In Progress

To Be Done

12 Months

x86 Power

• Contact Mike Brim <[email protected]> andIgor Grobman <[email protected]> for more details


• Most data structures are duplicated between thedaemon and the tool– Takes long time to ship the data to the client

– Excessive memory consumption (extra 20MB)

Kernel

Host A Host BKerninstdInstrumentation Tool

Remote Communication: Present

Kerninst API

Control

Flow Graph Register

Analysis Info

Register

Analysis Info

Control

Flow Graph


New RPC interface between the tool and the daemon

• Thin and efficient

• Code analysis results are kept on the daemon side

Kernel

Host A Host BKerninstd

Control

Flow Graph Register

Analysis Info

Instrumentation Tool

Remote Communication: Future

Kerninst API


main( )

A( ) B( )

open( ) read( ) seek( )

Application

Kernel

Dyninst API

Instrumentation

Tool

Kerninst API

Cross-Kernel-Boundary Analysis

open32() read32( ) seek32( )

namei( )

lock( )copyin( )ufs_open()Come See the Demo

Tomorrow!

main( )

open32()

A( )

open( )

ufs_open()

namei( )


Summary

• Kerninst API– Platform-independent

– Efficient

– Compatible with Dyninst

• For the SPARC version send a request to:[email protected]

• Ports to x86 and Power are underway

Documents

The Path to Portable Kernel Instrumentation: The Kerninst API · April 29, 2003 -1- Kerninst API The Path to Portable Kernel Instrumentation: The Kerninst API Alex Mirgorodskii [email protected]