ubuntu+systemtap进行Linux内核和用户空间开发测试

Systemtap 是学习内核一个必不可少的工具，它不仅可以侦测内核空间的相关信息，还可以侦测用户空间的信息，是研究内核源代码、优化系统性能及调试诊断系统问题的一个必备工具。相关原理信息可参见附录https://wiki.ubuntu.com/Kernel/Systemtap及“Linux 下的一个全新的性能测量和调式诊断工具 Systemtap, 第 3 部分: Systemtap”

http://www.ibm.com/developerworks/cn/linux/l-cn-systemtap3/#main

1 Systemtap安装

1.1 Apt-get自动安装systemtap

用系统工具自动安装systemtap时，用户不用考虑各个软件包的依赖关系，系统会自动根据当前内核版本下载相应的软件包。

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudoapt-get install systemtap

。。

The following extra packages will beinstalled:

libdw1 systemtap-commonsystemtap-runtime

The following NEW packages will beinstalled:

libdw1 systemtap systemtap-common systemtap-runtime

。

Get:1 http://us.archive.ubuntu.com/ubuntu/quantal-updates/main libdw1 i386 0.153-1ubuntu1.1 [216 kB]

。。

如上可知，系统自动安装了相关的4个软件包libdw1 systemtap-common systemtap-runtime

1.2 源代码编译安装systemtap

有时候系统自动安装的不一定能用，这个时候需要升级，此时只能通过下载源代码自己编译

1.2.1 卸载通过apt-get自动安装的systemtap

首先要卸载系统之前安装的相关package，避免后续带来其他副作用。

dd@ubuntu:/mnt/hgfs/systemtap$ sudo apt-getremove systemtap systemtap-common systemtap-runtime

。。

Removing systemtap ...

Removing systemtap-common ...

Removing systemtap-runtime ...

Processing triggers for man-db ...

1.2.2 下载systemtap源码包

下载相应版本的源码包。源码包并不是越新越好，因为他们之间有一定的依赖关系，最简单的是按照内核源码包发布的时间来选择systemtap。

https://fedorahosted.org/releases/e/l/elfutils/

ftp://sources.redhat.com/pub/systemtap/releases/

1.2.3 编译安装

dd@ubuntu:/mnt/hgfs/systemtap/systemtap-2.0$./configure --with-elfutils=../elfutils-0.156 --prefix=/usr/local

。。。。。

configure: ./configure ‘--with-elfutils=../elfutils-0.156‘‘--prefix=/usr/local‘ --prefix=/home/dd/systemtap-2.0-4155

configure: Running systemtap uninstalled, entirely out of the buildtree,

configure: is not supported.

重启系统之后再执行，仍然如此，不知是否有其他副作用

dd@ubuntu:/mnt/hgfs/systemtap/systemtap-2.0$sudo make

。。

ln -fs libdw.so libdw.so.1

ln: failed to create symbolic link `libdw.so.1‘: Operation notsupported

make[4]: *** [libdw.so] Error 1

windows不支持符号链接，因此system包不能放在windows目录编译

dd@ubuntu:~/systemtap/systemtap-2.0$ sudomake

。。。

m4 -Di386 -DDISASSEMBLER../../../elfutils-0.156/libcpu/defs/i386 > i386_defsT

/bin/bash: m4: command not found

1.3 安装内核debug info

System调试内核或者application，必须有debuginfo，因此必须安装内核debug symbol。具体参见附录。

2 Hello World基本测试

Systemtap安装完毕后，进行简单的测试。如果能打印出helloworld则认为基本功能OK。

dd@ubuntu:/usr/src$ sudo stap -ve ‘probebegin { log("hello world") exit() }‘

Pass 1: parsed user script and 81 libraryscript(s) using 22640virt/13560res/2192shr kb, in 80usr/10sys/93real ms.

Pass 2: analyzed script: 1 probe(s), 2function(s), 0 embed(s), 0 global(s) using 22904virt/14088res/2264shr kb, in0usr/0sys/3real ms.

Pass 3: translated to C into"/tmp/stapNgfERv/stap_202537bef6262d2233e5759ff826c9d3_744_src.c"using 22904virt/14364res/2508shr kb, in 0usr/0sys/0real ms.

Pass 4: compiled C into"stap_202537bef6262d2233e5759ff826c9d3_744.ko" in4760usr/900sys/6360real ms.

Pass 5: starting run.

hello world

Pass 5: run completed in 0usr/20sys/359realms.

3 内核函数测试

3.1 基本功能

dd@ubuntu:/usr/src$ sudo stap -e ‘probekernel.function("sys_open") {log("hello world") exit()}‘

hello world

3.2 函数调用栈

dd@ubuntu:/mnt/hgfs/systemtap/example$ cat./timer.stp

#!/usr/bin/stap

probekernel.function("do_timer").return {

print_backtrace();

printf("\n");

exit();

}

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./timer.stp

Returning from: 0xc1091520 : do_timer+0x0/0x40 [kernel]

Returning to : 0xc1097e4a : tick_do_update_jiffies64+0xca/0x130 [kernel]

0xc1098074 : tick_sched_timer+0xc4/0xd0[kernel]

0xc1068f70 : __run_hrtimer+0x70/0x190 [kernel]

0xc1069be7 : hrtimer_interrupt+0xe7/0x2a0 [kernel]

0xc15d0769 :smp_apic_timer_interrupt+0x59/0x8d [kernel]

0xc15c9815 : apic_timer_interrupt+0x31/0x38[kernel]

0xc1036f35 : native_safe_halt+0x5/0x10[kernel]

0xc1017b66 : default_idle+0x46/0x190 [kernel]

0xc1018716 : cpu_idle+0xb6/0xe0 [kernel]

0xc159f715 : rest_init+0x5d/0x68 [kernel]

0xc18ae9be : start_kernel+0x35d/0x363 [kernel]

0xc18ae303 : i386_start_kernel+0xa6/0xad[kernel]

4 内核模块测试

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap ./vmx_inject_irq.stp

[sudo] password for dd:

semantic error: while resolving probepoint: identifier ‘kernel‘ at ./vmx_inject_irq.stp:1:7

source: probe kernel.function("vmx_inject_irq") {

semantic error: no match

Pass 2: analysis failed. Try again with another ‘--vp 01‘ option.

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ cat/proc/kallsyms | grep vmx_inject_irq

00000000 t vmx_inject_irq [kvm_intel]

kvm_intel以模块形式加载，其symbol不能以kernel.function形式访问。

Ubuntu平台上需要修正动态module的debug info信息

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catfix-module.sh

#!/bin/sh

for file in `find /usr/lib/debug -name‘*.ko‘ -print`

buildid=`eu-readelf-n $file| grep Build.ID: | awk ‘{print $3}‘`

dir=`echo$buildid | cut -c1-2`

fn=`echo$buildid | cut -c3-`

mkdir-p /usr/lib/debug/.build-id/$dir

ln-s $file /usr/lib/debug/.build-id/$dir/$fn

ln-s $file /usr/lib/debug/.build-id/$dir/${fn}.debug

done

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -l ‘module("*kvm*").function("*")‘ >kvm-module-sysm.log

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catkvm-module-sysm.log | grep vmx_inject_irq

module("kvm_intel").function("vmx_inject_irq@/build/buildd/linux-3.5.0/arch/x86/kvm/vmx.c:4036")

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catvmx_inject_irq.stp

probemodule("kvm_intel").function("vmx_inject_irq") {

printf("-------------------------------------\n")

print_backtrace()

printf("-------------------------------------\n")

}

很多符号都无法解析啊？？？？？？？根本原因待分析。

-------------------------------------

0xf86e4230 : vmx_inject_irq+0x0/0xe0[kvm_intel]

0xf998e694 [kvm]

0xc10654cd (inexact)

0xf997b25b [kvm] (inexact)

0xf998f11f [kvm] (inexact)

0xf997922b [kvm] (inexact)

0xc1075dee (inexact)

0xc1075ebf (inexact)

0xc1099a12 (inexact)

0xc109be08 (inexact)

0xf9978dd0 [kvm] (inexact)

0xc115e8fa (inexact)

0xc1069c65 (inexact)

0xc109c120 (inexact)

0xc115ee88 (inexact)

0xc15cff5f (inexact)

0xc15c0000 (inexact)

-------------------------------------

5 用户态监控Utrace

5.1 Utrace patch

用户态监控需要utrace支持，但utrace默认只有redhat系列的kernel支持，如fedora等。其他发行版Linux需要patch

Systemtap的README中有关于utrace支持的描述：

- Consider applying the utrace kernelpatches, if you wish to probe

user-space applications. http://sourceware.org/systemtap/wiki/utrace

Orif your kernel is near 3.5, apply the uprobes and related patches

(see NEWS). Or if your kernel is>= 3.5, enjoy the built-in uprobes.

- Build the kernel using your normalprocedures. Enable

CONFIG_DEBUG_INFO, CONFIG_KPROBES, CONFIG_RELAY, CONFIG_DEBUG_FS,

CONFIG_MODULES, CONFIG_MODULE_UNLOAD, CONFIG_UTRACE if able

- % make modules_install install headers_install

- Boot into the kernel.

Ubuntu 要支持utrace有两种方式：

1. 自己给 Ubuntu 自带的老 kernel 应用 utrace 补丁，并重新编译。但好像redhat提供的patch不能访问了

http://people.redhat.com/roland/utrace/

Ubuntu 11.10 (Oneiric)上编译带utrace补丁的内核

http://chaoslawful.iteye.com/blog/1463564

2. 将 kernel 升级到官方最新的 3.5 或以上的版本。最新的kernel 默认包含了 uprobes 机制，不再需要 utrace 补丁了。

Kernel hacking --->

[*] Tracers --->

[*] Enable uprobes-based dynamic events

新的内核里已经没有选项CONFIG_UTRACE，选择utrace相关的选项即可，包括CONFIG_ARCH_SUPPORTS_UPROBESCONFIG_UPROBES CONFIG_UPROBE_EVENT CONFIG_KPROBES CONFIG_DEBUG_FS 等

dd@ubuntu:/mnt/hgfs/systemtap/example$ cat/boot/config-3.5.0-17-generic | grep UPROBE

CONFIG_ARCH_SUPPORTS_UPROBES=y

CONFIG_UPROBES=y

CONFIG_UPROBE_EVENT=y

dd@ubuntu:/mnt/hgfs/systemtap/example$ cat/boot/config-3.5.0-17-generic | grep KPROBE

CONFIG_KPROBES=y

CONFIG_HAVE_KPROBES=y

# CONFIG_KPROBES_SANITY_TEST is not set

CONFIG_KPROBE_EVENT=y

如果不是必须用某一个版本，建议直接升级到3.5以后的内核版本，如Ubuntu 12.10

5.2 函数跟踪测试

首先写了一个用户态的测试函数，可参考如下

http://blog.csdn.net/astrotycoon/article/details/8142588

Utrace调试的程序需要在编译时加 -g 选项

dd@ubuntu:/mnt/hgfs/systemtap/example$./utrace-test

Obtained 8 stack frames.nm

./utrace-test() [0x80485c1]

[0xb77a3400]

./utrace-test() [0x804869f]

./utrace-test() [0x80486ac]

./utrace-test() [0x80486b6]

./utrace-test() [0x80486eb]

/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75ff4d3]

./utrace-test() [0x80484b1]

有函数调用栈，但是没有函数名称，莫非是编译优化导致无符号表？经查需要传递(-rdynamic)，-rdynamic可用来通知链接器将所有符号添加到动态符号表中

dd@ubuntu:/mnt/hgfs/systemtap/example$ gcc-g -rdynamic utrace-test.c -o utrace-test

dd@ubuntu:/mnt/hgfs/systemtap/example$./utrace-test

Obtained 7 stack frames.

./utrace-test(print_trace+0x1f) [0x80487ab]

./utrace-test(dummy_function_3+0xb)[0x8048840]

./utrace-test(test2+0xb) [0x804884d]

./utrace-test(test1+0xb) [0x804885a]

./utrace-test(main+0xb) [0x8048867]

/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75f44d3]

./utrace-test() [0x80486c1]

OK，能够正确打印出函数调用顺序，参考基准已经建立起来了。

如下测试显示stap可以监测到待调试程序utrace-test的函数列表。

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap -l ‘process("./utrace-test").function("*")‘

process("/mnt/hgfs/systemtap/example/utrace-test").function("dummy_function_3@/mnt/hgfs/systemtap/example/utrace-test.c:26")

process("/mnt/hgfs/systemtap/example/utrace-test").function("main@/mnt/hgfs/systemtap/example/utrace-test.c:39")

process("/mnt/hgfs/systemtap/example/utrace-test").function("print_trace@/mnt/hgfs/systemtap/example/utrace-test.c:7")

process("/mnt/hgfs/systemtap/example/utrace-test").function("test1@/mnt/hgfs/systemtap/example/utrace-test.c:34")

process("/mnt/hgfs/systemtap/example/utrace-test").function("test2@/mnt/hgfs/systemtap/example/utrace-test.c:30")

当任意一个函数被调用时，打印其函数名及参数

dd@ubuntu:/mnt/hgfs/systemtap/example$ catutrace.stp

#!/usr/bin/stap

probeprocess("./utrace-test").function("*") {

printf("%s(%s)\n", probefunc(), $$parms);

}

实际检测到的函数调用顺序与实际匹配

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./utrace.stp

_start()

__libc_csu_init()

__i686.get_pc_thunk.bx()

_init()

frame_dummy()

register_tm_clones()

main()

test1()

test2()

dummy_function_3()

print_trace()

__do_global_dtors_aux()

deregister_tm_clones()

_fini()

5.3 打印用户态调用栈

利用print_backtrace打印监测点的函数调用栈。

dd@ubuntu:/mnt/hgfs/systemtap/example$ catutrace-stack.stp

#!/usr/bin/stap

probeprocess("./utrace-test").function("dummy_function_3"){

printf("%s(%s)\n", probefunc(),$$parms);

print_backtrace();

printf("\n");

}

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./utrace-stack.stp

semantic error: while resolving probepoint: identifier ‘process‘ at ./utrace.stp:3:7

source: probeprocess("./utrace-test").function("dummy_function_3").return{

semantic error: process return probes not available withinode-based uprobes

Pass 2: analysis failed. Try again with another ‘--vp 01‘ option.

去掉return，再执行

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./utrace-stack.stp

WARNING: no or bad debug frame hdr

WARNING: No binary search table for eh frame, doing slow linear searchfor stap_3fb2bcd721ab82a6de577773eff766b_10596

dummy_function_3()

0xf8887e10 [stap_3fb2bcd721ab82a6de577773eff766b_10596+0xbe10/0x0]

0xf8888209[stap_3fb2bcd721ab82a6de577773eff766b_10596+0xc209/0x0]

0xf88868b0[stap_3fb2bcd721ab82a6de577773eff766b_10596+0xa8b0/0x0]

0xc113eb3f

0xc104e51b (inexact)

0xc161ac9d (inexact)

没有打印出用户态的调用堆栈，具体原因待分析。

6 Systemtap加载KO异常，Unknown symbol inmodule

6.1 异常现象

加载动态编译出来的module时失败Unknown symbol inmodule

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -ve ‘probe begin { log("hello world") exit() }‘

。。

Pass 4: compiled C into"stap_aeecf973d1b00387ff715fbb9a761ae9_688.ko" in6010usr/1080sys/10345real ms.

Pass 5: starting run.

Error inserting module‘/tmp/stapoo63sX/stap_aeecf973d1b00387ff715fbb9a761ae9_688.ko‘: Unknown symbolin module

WARNING: /usr/bin/staprun exited withstatus: 1

Pass 5: run completed in 0usr/0sys/11realms.

Pass 5: run failed. Try again with another ‘--vp 00001‘ option.

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -e ‘probe kernel.function("sys_open") {log("helloworld") exit()}‘

Error inserting module‘/tmp/stapPq4Mh5/stap_b3efd67e86a8df14579c7f419ddcd07f_826.ko‘: Unknown symbolin module

WARNING: /usr/bin/staprun exited withstatus: 1

Pass 5: run failed. Try again with another ‘--vp 00001‘ option

6.2 分析异常原因

查看内核log，分析Unknown symbol inmodule的意义，这种问题通常都是某个symbol无法找到。

dd@ubuntu:~/systemtap/systemtap-2.5$ dmesg| tail

…

[ 76.855961] stap_880c6f2fc0076129ea6feb7db23da703_899: Unknown parameter `_stp_bufsize‘

“-k”参数保留运行过程中生产的文件，查找线索

dd@ubuntu:~$ sudo stap -k -ve ‘probe begin{ log("hello world") exit() }‘

。。

Pass 5: starting run.

Error inserting module‘/tmp/staplCMn6Z/stap_4270.ko‘: Unknown symbol in module

WARNING: /usr/bin/staprun exited withstatus: 1

Pass 5: run completed in 0usr/0sys/3realms.

Pass 5: run failed. Try again with another ‘--vp 00001‘ option.

Keeping temporary directory"/tmp/staplCMn6Z"

dd@ubuntu:~$ sudo nm/tmp/staplCMn6Z/stap_4270.ko | grep _stp_bufsize

00000032 r __mod__stp_bufsize76

00000050 r __mod__stp_bufsizetype75

00000000 r __param__stp_bufsize

00000515 r __param_str__stp_bufsize

00000130 b _stp_bufsize

“--vp00005”启动systemtap的调试选项获得更多的log。具体用法可参考http://blog.163.com/digoal@126/blog/static/163877040201391434530674/

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap --vp 00005 -ve‘probe begin{printf("hello world\n") exit()}‘

。。。

Pass 5: starting run.

Running /usr/local/bin/staprun -v -v -v -v -v-R /tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko

staprun:parse_modpath:386

。。。

staprun:insert_module:71 inserting module/tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko

staprun:insert_module:97module options: _stp_bufsize=0

staprun:insert_module:105 module pathcanonicalized to ‘/tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko‘

。。

staprun:insert_module:183 Modulestap_333448f0b355beeb59e41a501a5421f_6340 inserted from file/tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko

ERROR:Couldn‘t insert module ‘/tmp/stapwlfJOn/stap_333448f0b355beeb59e41a501a5421fd_817.ko‘:Unknown symbol in module

Spawn waitpid result (0x100): 1

WARNING: /usr/local/bin/staprun exited withstatus: 1

Pass 5: run completed in 0usr/0sys/2real ms.

Pass 5: run failed. [man error::pass5]

Running rm -rf /tmp/stapwlfJOn

Spawn waitpid result (0x0): 0

Removed temporary directory"/tmp/stapwlfJOn"

dd@ubuntu:~/systemtap/systemtap-2.5$ grep -nr"_stp_bufsize" ./

Binary file ./staprun/staprun matches

Binary file ./staprun/staprun-staprun.omatches

./staprun/staprun.c:190: /* Add the _stp_bufsize option. */

./staprun/staprun.c:192: "_stp_bufsize=%d",buffer_size))

./runtime/transport/transport.c:75:static int_stp_bufsize;

./runtime/transport/transport.c:76:module_param(_stp_bufsize,int, 0);

./runtime/transport/transport.c:77:MODULE_PARM_DESC(_stp_bufsize,"buffer size");

./runtime/transport/transport.c:408: if (_stp_bufsize) {

./runtime/transport/transport.c:409: unsigned size = _stp_bufsize * 1024 *1024;

./runtime/transport/ring_buffer.c:83: unsigned long buffer_size = _stp_bufsize* 1024 * 1024;

./runtime/transport/transport.h:48:static int_stp_bufsize;

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudomodinfo stapoMK4aC/stap_12809.ko [sudo] password for dd:

filename: stapoMK4aC/stap_12809.ko

。

parm: _stp_bufsize:buffer size (int)

从上面的代码看，模块参数_stp_bufsize确实存在，且加载的过程没有问题啊，很奇怪为什么会fail。难道是版本不匹配？尝试卸载2.0版本，安装2.5版本，依然有问题

6.3 Workaround

于是尝试分析其内在原理，看是否能真正fix或者workaround这个问题。

./runtime/transport/transport.c接收模块参数_stp_bufsize，但是从代码看并没有特别的用处；另外通过“staprun:insert_module:97module options: _stp_bufsize=0”知道./staprun/staprun.c:192实际传入的参数值是0，因此可去掉special_options参数试试

static int insert_stap_module(privilege_t*user_credentials)

{

charspecial_options[128] = {0};

#if 0

/* Add the _stp_bufsize option. */

if (snprintf_chk(special_options,sizeof (special_options),

"_stp_bufsize=%d", buffer_size))

return -1;

#endif

stap_module_inserted= insert_module(modpath, special_options,

modoptions,

assert_stap_module_permissions,

user_credentials);

if (stap_module_inserted != 0)

err("Error insertingmodule ‘%s‘: %s\n", modpath, moderror(errno));

returnstap_module_inserted;

}

重新编译加载后，一切正常。

7 error: implicit declaration offunction ‘unregister_uprobe’

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap -ve ‘probe process("./utrace-test").function("*"){printf("-------------------------------------\n") print_backtrace()printf("-------------------------------------\n")}‘

。。。

In file included from/tmp/stapRI0noH/stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325_src.c:184:0:

/usr/share/systemtap/runtime/uprobes-inode.c:In function ‘stp_inode_uprobes_unreg’:

/usr/share/systemtap/runtime/uprobes-inode.c:94:3:error: implicit declaration of function ‘unregister_uprobe’[-Werror=implicit-function-declaration]

/usr/share/systemtap/runtime/uprobes-inode.c:In function ‘stp_inode_uprobes_reg’:

/usr/share/systemtap/runtime/uprobes-inode.c:107:3:error: implicit declaration of function ‘register_uprobe’ [-Werror=implicit-function-declaration]

cc1: all warnings being treated as errors

make[1]: ***[/tmp/stapRI0noH/stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325_src.o] Error 1

make: *** [_module_/tmp/stapRI0noH] Error 2

WARNING: make exited with status: 2

Pass 4: compiled C into"stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325.ko" in280usr/170sys/681real ms.

Pass 4: compilation failed. Try again with another ‘--vp 0001‘ option.

https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1075772

stap fails with: "error: implicitdeclaration of function ‘unregister_uprobe’"

dd@ubuntu:/mnt/hgfs/systemtap$ stap -V

Systemtap translator/driver (version 1.7/0.153 Debian version1.7-1ubuntu1 (quantal))

This is free software; see the source forcopying conditions.

enabled features: AVAHI LIBSQLITE3 NSSBOOST_SHARED_PTR TR1_UNORDERED_MAP NLS

内核中没有systemtap/runtime引用的API register_uprobe。在新版本的systemtap中已经fix了，因此需要下载源代码手动升级

dd@ubuntu:/mnt/hgfs/systemtap$ sudo grep"\suprobe.*register" /proc/kallsyms

c10fe1d0 T uprobe_register

c10fe430 T uprobe_unregister

升级到2.0版本之后，正常

dd@ubuntu:~/systemtap/systemtap-2.0$ stap-V

Systemtap translator/driver (version2.0/0.156, non-git sources)

This is free software; see the source forcopying conditions.

enabled features: TR1_UNORDERED_MAP NLS

8 参考资料

Systemtap WiKi

https://wiki.ubuntu.com/Kernel/Systemtap

A guide on how to install Systemtap on anUbuntu system

https://sourceware.org/systemtap/wiki/SystemtapOnUbuntu

Linux 下的一个全新的性能测量和调式诊断工具Systemtap, 第 3 部分: Systemtap

http://www.ibm.com/developerworks/cn/linux/l-cn-systemtap3/#main

使用systemtap调试Linux内核示例

http://lenky.info/archives/2013/02/2209

User-Space Probing

https://sourceware.org/systemtap/SystemTap_Beginners_Guide/userspace-probing.html

Dynamic Tracing with DTrace and SystemTap

https://gist.github.com/dhedlund/9635035

SystemTap，万能的性能测量和调式诊断工具

http://guiquanz.me/2013/04/21/systemtap-intro/

Systemtap : stap PROCESSING 5 stepsintroduce

http://blog.163.com/digoal@126/blog/static/163877040201391434530674/

ubuntu+systemtap进行Linux内核和用户空间开发测试,布布扣,bubuko.com