Discussion:
Bug#856474: stap: include runtime_defines.h not found
Jack Henschel
2017-03-01 12:53:14 UTC
Permalink
Package: systemtap
Version: 3.0-7
Severity: normal
Tags: stretch

Hi,

after I set up systemtap, I ran the example command provided in the documentation:
https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#testing
stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
/tmp/stapHaiI5M/stap_a31ee9d9da16ac912b90eb18969508b5_1547_src.c:10:29: fatal error: runtime_defines.h: No such file or directory
#include "runtime_defines.h"
For a more verbose log, please see the attachment.

-- System Information:
Debian Release: 9.0
APT prefers testing-debug
APT policy: (500, 'testing-debug'), (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages systemtap depends on:
ii libavahi-client3 0.6.32-2
ii libavahi-common3 0.6.32-2
ii libc6 2.24-9
ii libdw1 0.168-0.2
ii libelf1 0.168-0.2
ii libgcc1 1:6.3.0-6
ii libnspr4 2:4.12-6
ii libnss3 2:3.26.2-1
ii libsqlite3-0 3.16.2-2
ii libstdc++6 6.3.0-6
ii make 4.1-9.1
ii systemtap-common 3.0-7
ii systemtap-runtime 3.0-7

$ dpkg -l | grep linux-
ii linux-base 4.5 all Linux image base package
ii linux-compiler-gcc-6-x86 4.9.6-3 amd64 Compiler for Linux on x86 (meta-package)
ii linux-headers-4.9.0-1-amd64 4.9.6-3 amd64 Header files for Linux 4.9.0-1-amd64
ii linux-headers-4.9.0-1-common 4.9.6-3 all Common header files for Linux 4.9.0-1
ii linux-image-4.9.0-1-amd64 4.9.6-3 amd64 Linux 4.9 for 64-bit PCs (signed)
ii linux-image-4.9.0-1-amd64-dbgsym 4.9.6-3 amd64 Debug symbols for linux-image-4.9.0-1-amd64
ii linux-image-amd64 4.9+78 amd64 Linux for 64-bit PCs (meta-package)
ii linux-kbuild-4.9 4.9.6-3 amd64 Kbuild infrastructure for Linux 4.9
ii linux-libc-dev:amd64 4.9.6-3 amd64 Linux support headers for userspace development
ii util-linux-locales 2.29.1-1 all locales files for util-linux


$ dpkg -l | grep gcc
ii gcc 4:6.3.0-1 amd64 GNU C compiler
ii gcc-6 6.3.0-6 amd64 GNU C compiler
ii gcc-6-base:amd64 6.3.0-6 amd64 GCC, the GNU Compiler Collection (base package)
ii libgcc-6-dev:amd64 6.3.0-6 amd64 GCC support library (development files)
ii libgcc1:amd64 1:6.3.0-6 amd64 GCC support library
ii linux-compiler-gcc-6-x86 4.9.6-3 amd64 Compiler for Linux on x86 (meta-package)
Ritesh Raj Sarraf
2017-03-01 13:28:39 UTC
Permalink
Control: tag -1 +moreinfo
Post by Jack Henschel
https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.htm
l#testing
stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
GCC (6) complains upon compiling the C source code into the kernel module that
fatal error: runtime_defines.h: No such file or directory
#include "runtime_defines.h"
That file is provided by the systemtap-common package, which is properly marked
as a Depends for systemtap.

Over here, with a custom built kernel (Linux 4.10), I am getting compilation
errors, which are not relevant to the issue you've reported.

But in my logs, runtime_defines.h is properly detected. I'll try to run the
command on a Debian kernel to verify if it is broken or not.


- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Ritesh Raj Sarraf
2017-03-01 13:29:39 UTC
Permalink
Post by Ritesh Raj Sarraf
But in my logs, runtime_defines.h is properly detected. I'll try to run the
command on a Debian kernel to verify if it is broken or not.
Forgot to add the log.

```
***@learner:/tmp$ sudo stap -v -e 'probe vfs.read {printf("read performed\n");
exit()}'
[sudo] password for rrs: 
Pass 1: parsed user script and 118 library scripts using
106660virt/39568res/6268shr/33552data kb, in 200usr/20sys/258real ms.
Pass 2: analyzed script: 1 probe, 1 function, 4 embeds, 0 globals using
246904virt/181448res/7688shr/173796data kb, in 1810usr/660sys/4018real ms.
Pass 3: translated to C into
"/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c" using
246904virt/181640res/7880shr/173796data kb, in 10usr/50sys/60real ms.
In file included from /usr/share/systemtap/runtime/linux/runtime.h:209:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h: In function
‘__access_process_vm_’:
/usr/share/systemtap/runtime/linux/access_process_vm.h:36:61: error: passing
argument 6 of ‘get_user_pages_remote’ makes pointer from integer without a cast
[-Werror=int-conversion]
       ret = get_user_pages_remote (tsk, mm, addr, 1, write, 1, &page, &vma);
                                                             ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
./include/linux/mm.h:1267:6: note: expected ‘struct page **’ but argument is of
type ‘int’
 long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,
      ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/share/systemtap/runtime/linux/runtime.h:209:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h:36:64: error: passing
argument 7 of ‘get_user_pages_remote’ from incompatible pointer type [-
Werror=incompatible-pointer-types]
       ret = get_user_pages_remote (tsk, mm, addr, 1, write, 1, &page, &vma);
                                                                ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
./include/linux/mm.h:1267:6: note: expected ‘struct vm_area_struct **’ but
argument is of type ‘struct page **’
 long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,
      ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/share/systemtap/runtime/linux/runtime.h:209:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h:36:71: error: passing
argument 8 of ‘get_user_pages_remote’ from incompatible pointer type [-
Werror=incompatible-pointer-types]
       ret = get_user_pages_remote (tsk, mm, addr, 1, write, 1, &page, &vma);
                                                                       ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from
/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.c:25:
./include/linux/mm.h:1267:6: note: expected ‘int *’ but argument is of type
‘struct vm_area_struct **’
 long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,
      ^~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
scripts/Makefile.build:294: recipe for target
'/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.o' failed
make[1]: *** [/tmp/stapJQUBQF/stap_8ef46d39a073456b4762098be7ff765b_1501_src.o]
Error 1
Makefile:1490: recipe for target '_module_/tmp/stapJQUBQF' failed
make: *** [_module_/tmp/stapJQUBQF] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_8ef46d39a073456b4762098be7ff765b_1501.ko" in
13140usr/1330sys/15462real ms.
Pass 4: compilation failed.  [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
2017-03-01 / 18:52:38 ♒♒♒  ☹  => 1  
```

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Jack Henschel
2017-03-01 13:45:27 UTC
Permalink
Post by Ritesh Raj Sarraf
That file is provided by the systemtap-common package, which is properly marked
as a Depends for systemtap.
I know, the package is installed and the files are present:
$ find /usr/share/systemtap/ -iname *runtime_defines.h*
/usr/share/systemtap/runtime/runtime_defines.h
/usr/share/systemtap/runtime/linux/runtime_defines.h
Post by Ritesh Raj Sarraf
gcc-6 -Wp,-MD,/tmp/stapcukYUi/.stap_a31ee9d9da16ac912b90eb18969508b5_1547_aux_0.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/6/include -I/usr/src/linux-headers-4.9.0-1-common/arch/x86/include -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated -I/usr/src/linux-headers-4.9.0-1-common/include -I./include -I/usr/src/linux-headers-4.9.0-1-common/arch/x86/include/uapi -I/usr/src/linux-headers-4.9.0-1-common/include/uapi -I./include/generated/uapi -include /usr/src/linux-headers-4.9.0-1-common/include/linux/kconfig.h -I/usr/src/linux-headers-4.9.0-1-common//tmp/stapcukYUi -I/tmp/stapcukYUi -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -DCC_HAVE_ASM_GOTO -I/usr/src/linux-headers-4.9.0-1-common/include2/asm/mach-default -I/lib/modules/4.9.0-1-amd64/build -include /tmp/stapcukYUi/stapconf_0763f53e6c686e80bdfa9e79a1380e9b_711.h -freorder-blocks -fasynchronous-unwind-tables -Wframe-larger-than=512 -fno-ipa-icf -Wno-unused -Werror -I/usr/src/linux-headers-4.9.0-1-common/"/usr/share/systemtap/runtime" -DMODULE -DKBUILD_BASENAME='"stap_a31ee9d9da16ac912b90eb18969508b5_1547_aux_0"' -DKBUILD_MODNAME='"stap_a31ee9d9da16ac912b90eb18969508b5_1547"' -c -o /tmp/stapcukYUi/.tmp_stap_a31ee9d9da16ac912b90eb18969508b5_1547_aux_0.o /tmp/stapcukYUi/stap_a31ee9d9da16ac912b90eb18969508b5_1547_aux_0.c
/tmp/stapcukYUi/stap_a31ee9d9da16ac912b90eb18969508b5_1547_src.c:10:29: fatal error: runtime_defines.h: No such file or directory
#include "runtime_defines.h"
Is the argument `-I/usr/src/linux-headers-4.9.0-1-common/"/usr/share/systemtap/runtime"` valid?
Post by Ritesh Raj Sarraf
Over here, with a custom built kernel (Linux 4.10), I am getting compilation
errors, which are not relevant to the issue you've reported.
Indeed, those errors do not have anything in common.
Ritesh Raj Sarraf
2017-03-01 16:58:55 UTC
Permalink
Control: tag -1 +confirmed -moreinfo
Post by Ritesh Raj Sarraf
But in my logs, runtime_defines.h is properly detected. I'll try to run the
command on a Debian kernel to verify if it is broken or not.
I have been able to reproduce this on a Debian kernel.

***@chutzpah:~$ stap-
prep                                                          
22:25 ♒♒♒   ☺    
***@chutzpah:~$ stap -v -e 'probe vfs.read {printf("read performed\n");
exit()}'   
Pass 1: parsed user script and 119 library scripts using
115056virt/39720res/6372shr/33400data kb, in 130usr/0sys/340real ms.
Pass 2: analyzed script: 1 probe, 1 function, 4 embeds, 0 globals using
256132virt/182128res/7752shr/174476data kb, in 1770usr/160sys/6312real ms.
Pass 3: translated to C into
"/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c" using
256132virt/182376res/8000shr/174476data kb, in 0usr/0sys/7real ms.
/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal
error: runtime_defines.h: No such file or directory
 #include "runtime_defines.h"
                             ^
compilation terminated.
/usr/src/linux-headers-4.9.0-2-common/scripts/Makefile.build:298: recipe for
target '/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.o' failed
make[3]: *** [/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.o]
Error 1
make[3]: *** Waiting for unfinished jobs....
/usr/src/linux-headers-4.9.0-2-common/Makefile:1507: recipe for target
'_module_/tmp/stapUHT1si' failed
make[2]: *** [_module_/tmp/stapUHT1si] Error 2
Makefile:150: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:8: recipe for target 'all' failed
make: *** [all] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538.ko" in
7560usr/630sys/10583real ms.
Pass 4: compilation failed.  [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
22:26 ♒♒♒    ☹  => 1  

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Frank Ch. Eigler
2017-03-01 17:07:03 UTC
Permalink
Hi -
Post by Ritesh Raj Sarraf
I have been able to reproduce this on a Debian kernel.
[...]
/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal
error: runtime_defines.h: No such file or directory
 #include "runtime_defines.h"
                             ^
compilation terminated.
[...]
We've received intermittent reports of this sort of thing happening on
a few non-RH platforms. A common element seemed to be some
nonstandard kernel build tree environment - some sort of customized
out-of-tree module builder/processing script or makefile, which was
confused by quotations in the stap-synthesized Makefile. We have not
gotten to the bottom of it yet, but it didn't seem like a stap bug.

https://sourceware.org/ml/systemtap/2017-q1/msg00025.html

- FChE
Ritesh Raj Sarraf
2017-03-01 17:33:35 UTC
Permalink
Post by Frank Ch. Eigler
We've received intermittent reports of this sort of thing happening on
a few non-RH platforms.  A common element seemed to be some
nonstandard kernel build tree environment - some sort of customized
out-of-tree module builder/processing script or makefile, which was
confused by quotations in the stap-synthesized Makefile.  We have not
gotten to the bottom of it yet, but it didn't seem like a stap bug.
https://sourceware.org/ml/systemtap/2017-q1/msg00025.html
Thank you for replying quick. I realized your email after I sent another one.
That email has some details about how the Debian kernel is organized, especially
the linux-headers package split into 2 and installing files into 2 different
paths.

***@chutzpah:~$ ls /usr/src/
broadcom-sta-6.30.223.271/    linux-headers-4.9.0-2-common/
linux-headers-4.9.0-2-amd64/  linux-kbuild-4.9@
23:00 ♒♒♒   ☺    


IIRC, on the RH side, it used to be a single path. Maybe still is the same.

Similarly, I did not see this error on my custom kernel too.

***@learner:/var/tmp$ ls /usr/src/
linux-headers-4.10.1+/        linux-headers-4.9.0-2-common/  linux-kbuild-4.9@
linux-headers-4.9.0-2-amd64/  linux-headers-4.9.9+/
2017-03-01 / 23:00:52 ♒♒♒  ☺  
***@learner:/var/tmp$ uname -a
Linux learner 4.10.1+ #20 SMP Sun Feb 26 21:22:45 IST 2017 x86_64 GNU/Linux
2017-03-01 / 23:00:58 ♒♒♒  ☺  


I suspect the same, that this may be specific to Debian Linux kernel.
I have also had eBPF compiler failing on something similar (Debian Bug: 849841).

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Ritesh Raj Sarraf
2017-03-01 17:44:49 UTC
Permalink
Thanks to hints from Frank, I can see the culprit now.

- -Werror -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime" -DMODULE


The full compile command is:

gcc-6 -Wp,-MD,/tmp/staptZYmtX/.stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/6/include -I/usr/src/linux-headers-4.9.0-2-common/arch/x86/include -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated -I/usr/src/linux-headers-4.9.0-2-common/include -I./include -I/usr/src/linux-headers-4.9.0-2-common/arch/x86/include/uapi -I/usr/src/linux-headers-4.9.0-2-common/include/uapi -I./include/generated/uapi -include /usr/src/linux-headers-4.9.0-2-common/include/linux/kconfig.h -I/usr/src/linux-headers-4.9.0-2-common//tmp/staptZYmtX -I/tmp/staptZYmtX -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -DCC_HAVE_ASM_GOTO -I/usr/src/linux-headers-4.9.0-2-common/include2/asm/mach-default -I/lib/modules/4.9.0-2-amd64/build -include /tmp/staptZYmtX/stapconf_d8a215ec355839da11c41ac20d4e7e1e_702.h -freorder-blocks -fasynchronous-unwind-tables -Wframe-larger-than=512 -fno-ipa-icf -Wno-unused -Werror -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime" -DMODULE -DKBUILD_BASENAME='"stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0"' -DKBUILD_MODNAME='"stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538"' -c -o /tmp/staptZYmtX/.tmp_stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.o /tmp/staptZYmtX/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.c
/tmp/staptZYmtX/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal error: runtime_defines.h: No such file or directory
Post by Ritesh Raj Sarraf
Post by Frank Ch. Eigler
We've received intermittent reports of this sort of thing happening on
a few non-RH platforms.  A common element seemed to be some
nonstandard kernel build tree environment - some sort of customized
out-of-tree module builder/processing script or makefile, which was
confused by quotations in the stap-synthesized Makefile.  We have not
gotten to the bottom of it yet, but it didn't seem like a stap bug.
https://sourceware.org/ml/systemtap/2017-q1/msg00025.html
Thank you for replying quick. I realized your email after I sent another one.
That email has some details about how the Debian kernel is organized, especially
the linux-headers package split into 2 and installing files into 2 different
paths.
broadcom-sta-6.30.223.271/    linux-headers-4.9.0-2-common/
23:00 ♒♒♒   ☺    
IIRC, on the RH side, it used to be a single path. Maybe still is the same.
Similarly, I did not see this error on my custom kernel too.
linux-headers-4.9.0-2-amd64/  linux-headers-4.9.9+/
2017-03-01 / 23:00:52 ♒♒♒  ☺  
Linux learner 4.10.1+ #20 SMP Sun Feb 26 21:22:45 IST 2017 x86_64 GNU/Linux
2017-03-01 / 23:00:58 ♒♒♒  ☺  
I suspect the same, that this may be specific to Debian Linux kernel.
I have also had eBPF compiler failing on something similar (Debian Bug: 849841).
- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Jack Henschel
2017-03-01 18:48:57 UTC
Permalink
Post by Ritesh Raj Sarraf
Thanks to hints from Frank, I can see the culprit now.
-Werror -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime" -DMODULE
gcc-6 -Wp,-MD,/tmp/staptZYmtX/.stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/6/include -I/usr/src/linux-headers-4.9.0-2-common/arch/x86/include -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated -I/usr/src/linux-headers-4.9.0-2-common/include -I./include -I/usr/src/linux-headers-4.9.0-2-common/arch/x86/include/uapi -I/usr/src/linux-headers-4.9.0-2-common/include/uapi -I./include/generated/uapi -include /usr/src/linux-headers-4.9.0-2-common/include/linux/kconfig.h -I/usr/src/linux-headers-4.9.0-2-common//tmp/staptZYmtX -I/tmp/staptZYmtX -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -DCONFIG_X86_X32_ABI -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_FXSAVEQ=1 -DCONFIG_AS_SSSE3=1 -DCONFIG_AS_CRC32=1 -DCONFIG_AS_AVX=1 -DCONFIG_AS_AVX2=1 -DCONFIG_AS_AVX512=1 -DCONFIG_AS_SHA1_NI=1 -DCONFIG_AS_SHA256_NI=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -Wno-frame-address -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -pg -mfentry -DCC_USING_FENTRY -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -DCC_HAVE_ASM_GOTO -I/usr/src/linux-headers-4.9.0-2-common/include2/asm/mach-default -I/lib/modules/4.9.0-2-amd64/build -include /tmp/staptZYmtX/stapconf_d8a215ec355839da11c41ac20d4e7e1e_702.h -freorder-blocks -fasynchronous-unwind-tables -Wframe-larger-than=512 -fno-ipa-icf -Wno-unused -Werror -I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime" -DMODULE -DKBUILD_BASENAME='"stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0"' -DKBUILD_MODNAME='"stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538"' -c -o /tmp/staptZYmtX/.tmp_stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.o /tmp/staptZYmtX/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_aux_0.c
/tmp/staptZYmtX/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal error: runtime_defines.h: No such file or directory
I mentioned that in one of my earlier mails (#22).
Also I can confirm this not an issue with the _packaging_ of systemtap in Debian,
I downloaded and compiled the latest release tarball (3.1) directly from upstream and still got the same error.
Ritesh Raj Sarraf
2017-03-02 09:07:18 UTC
Permalink
Post by Jack Henschel
I mentioned that in one of my earlier mails (#22).
Sorry about that. I completely missed that one.

There's also another odd thing I noticed.

```
- -I/usr/src/linux-headers-4.9.0-1-common//tmp/stapcukYUi -I/tmp/stapcukYUi
```
Post by Jack Henschel
Also I can confirm this not an issue with the _packaging_ of systemtap in Debian,
I downloaded and compiled the latest release tarball (3.1) directly from
upstream and still got the same error.
Thanks for the information.

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Ritesh Raj Sarraf
2017-03-02 17:40:42 UTC
Permalink
Control: severity -1 grave
Control: tag -1 +help
Post by Ritesh Raj Sarraf
Post by Jack Henschel
Also I can confirm this not an issue with the _packaging_ of systemtap in Debian,
I downloaded and compiled the latest release tarball (3.1) directly from
upstream and still got the same error.
Thanks for the information.
I just pushed 3.1 to Experimental, the reason being 3.0 is going to be useless
for Debian Stretch. We need 3.1 to get it working proper with 4.9 kernel too.

This upload (3.1) still doesn't solve the runtime_defines.h issue with Debian
kernels. But, at least, we know now that the issue is something specific to
Debian kernels only. As is shown below, and also mentioned by Frank.

Given that this bug makes systemtap useless, I have marked the bug accordingly
to grave. I have also marked this for help because I don't really have that much
"free" time to investigate it right now.

@Jack: I would like to reassign this bug to the kernel team. But it'd be nice if
I could get more users confirm that the bug doesn't appear with non-Debian
kernels. Would you be in a position to pick systemtap from Experimental, and
build a pristine upstream kernel image (you need to clone Linus's tree and just
run the bindeb-pkg make target) and test the combination together ?


Here are my results.

***@learner:~$ stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 470 library scripts using
123812virt/48564res/6640shr/42124data kb, in 110usr/10sys/129real ms.
Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using
280416virt/206592res/7948shr/198728data kb, in 1400usr/190sys/1594real ms.
Pass 3: using cached
/home/rrs/.systemtap/cache/f8/stap_f83081804fadc6e30161178b5574195d_2502.c
Pass 4: using cached
/home/rrs/.systemtap/cache/f8/stap_f83081804fadc6e30161178b5574195d_2502.ko
Pass 5: starting run.
read performed
Pass 5: run completed in 10usr/30sys/530real ms.
2017-03-02 / 22:58:35 ♒♒♒  ☺  

***@learner:~$ uname -a
Linux learner 4.10.1+ #20 SMP Sun Feb 26 21:22:45 IST 2017 x86_64 GNU/Linux
2017-03-02 / 22:58:39 ♒♒♒  ☺  

vs

***@chutzpah:~/Community/Packaging/systemtap (master)$ stap -ve 'probe begin { log("hello world") exit () }'
Pass 1: parsed user script and 471 library scripts using 123528virt/48648res/6764shr/41840data kb, in 140usr/0sys/149real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 124452virt/49620res/6956shr/42764data kb, in 10usr/0sys/5real ms.
Pass 3: translated to C into "/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.c" using 124452virt/49620res/6956shr/42764data kb, in 0usr/0sys/1real ms.
/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.c:10:29: fatal error: runtime_defines.h: No such file or directory
#include "runtime_defines.h"
^
compilation terminated.
/usr/src/linux-headers-4.9.0-2-common/scripts/Makefile.build:298: recipe for target '/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.o' failed
make[3]: *** [/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.o] Error 1
make[3]: *** Waiting for unfinished jobs....
/usr/src/linux-headers-4.9.0-2-common/Makefile:1507: recipe for target '_module_/tmp/stapJNAO0g' failed
make[2]: *** [_module_/tmp/stapJNAO0g] Error 2
Makefile:150: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:8: recipe for target 'all' failed
make: *** [all] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_eed6170377f19ad002c0d2b53e5e7956_1067.ko" in 7810usr/760sys/9195real ms.
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
23:00 ♒♒♒ ☹ => 1

***@chutzpah:~/Community/Packaging/systemtap (master)$ uname -a
Linux chutzpah 4.9.0-2-amd64 #1 SMP Debian 4.9.13-1 (2017-02-27) x86_64 GNU/Linux
23:00 ♒♒♒ ☺

- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Jack Henschel
2017-03-02 20:47:57 UTC
Permalink
Post by Ritesh Raj Sarraf
I just pushed 3.1 to Experimental, the reason being 3.0 is going to be useless
for Debian Stretch. We need 3.1 to get it working proper with 4.9 kernel too.
Did I get that right: you are planning to ship SystemTap 3.1 with Stretch?
(If so: great, I'm all for it!)
Post by Ritesh Raj Sarraf
@Jack: I would like to reassign this bug to the kernel team. But it'd be nice if
I could get more users confirm that the bug doesn't appear with non-Debian
kernels. Would you be in a position to pick systemtap from Experimental, and
build a pristine upstream kernel image (you need to clone Linus's tree and just
run the bindeb-pkg make target) and test the combination together ?
Sure, I always wanted to build my own kernel, I guess I finally have a good reason to do so :-)
Jack Henschel
2017-03-03 15:43:18 UTC
Permalink
Can confirm, the stable kernel (compiled directly from kernel.org) works just fine (along with the included packaging) with SystemTap 3.1 from Debian experimental:

$ stap -V
Systemtap translator/driver (version 3.1/0.168, Debian version 3.1-1 (experimental))
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.10-rc8
enabled features: AVAHI LIBSQLITE3 NLS NSS
$ uname -a
Linux vm000949 4.10.1 #2 SMP Fri Mar 3 15:00:38 CET 2017 x86_64 GNU/Linux
$ stap -v -e 'probe vfs.read { printf("read performed\n"); exit() }'
Pass 1: parsed user script and 465 library scripts using 122332virt/46968res/6636shr/40644data kb, in 170usr/20sys/191real ms.
Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using 281772virt/207800res/7924shr/200084data kb, in 1860usr/330sys/2217real ms.
Pass 3: using cached /home/jack/.systemtap/cache/28/stap_284d447b34881d240eeab3f29fc1d9dc_2511.c
Pass 4: using cached /home/jack/.systemtap/cache/28/stap_284d447b34881d240eeab3f29fc1d9dc_2511.ko
Pass 5: starting run.
read performed
Pass 5: run completed in 10usr/0sys/430real ms.
Ritesh Raj Sarraf
2017-03-03 17:40:56 UTC
Permalink
Control: reassign -1 linux


Assigning it to the kernel team.

In this bug report, we have 2 confirmations of systemtap working fine with non-
Debian build Linux kernel (upstream kernel from Linus's tree built with the
bindeb-pkg make target)

systemtap relies on the running Linux kernel's header files to convert its
systemtap script into kernel object. With the Debian kernels, systemtap fails to
build the stap scripts.


Dear (Linux) Kernel Team:

We currently have systemtap 3.0 in Debian testing where this problem was
reported.

In Experimental, I have also pushed the 3.1 release. The 3.1 release is reported
to work perfectly in our tests with the custom built kernel. The problem is only
seen with the Debian built Linux kernels.

At this time, we do not know what difference in the Debian kernel is causing
this failure. Since this bug makes systemtap of no use at all, the bug is marked
"grave".


All other details (and many verbose logs) are present in the full bug report.
Post by Ritesh Raj Sarraf
Control: severity -1 grave
Control: tag -1 +help
Post by Ritesh Raj Sarraf
Post by Jack Henschel
Also I can confirm this not an issue with the _packaging_ of systemtap in Debian,
I downloaded and compiled the latest release tarball (3.1) directly from
upstream and still got the same error.
Thanks for the information.
I just pushed 3.1 to Experimental, the reason being 3.0 is going to be useless
for Debian Stretch. We need 3.1 to get it working proper with 4.9 kernel too.
This upload (3.1) still doesn't solve the runtime_defines.h issue with Debian
kernels. But, at least, we know now that the issue is something specific to
Debian kernels only. As is shown below, and also mentioned by Frank.
Given that this bug makes systemtap useless, I have marked the bug accordingly
to grave. I have also marked this for help because I don't really have that much
"free" time to investigate it right now.
@Jack: I would like to reassign this bug to the kernel team. But it'd be nice if
I could get more users confirm that the bug doesn't appear with non-Debian
kernels. Would you be in a position to pick systemtap from Experimental, and
build a pristine upstream kernel image (you need to clone Linus's tree and just
run the bindeb-pkg make target) and test the combination together ?
Here are my results.
Pass 1: parsed user script and 470 library scripts using
123812virt/48564res/6640shr/42124data kb, in 110usr/10sys/129real ms.
Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using
280416virt/206592res/7948shr/198728data kb, in 1400usr/190sys/1594real ms.
Pass 3: using cached
/home/rrs/.systemtap/cache/f8/stap_f83081804fadc6e30161178b5574195d_2502.c
Pass 4: using cached
/home/rrs/.systemtap/cache/f8/stap_f83081804fadc6e30161178b5574195d_2502.ko
Pass 5: starting run.
read performed
Pass 5: run completed in 10usr/30sys/530real ms.
2017-03-02 / 22:58:35 ♒♒♒  ☺  
Linux learner 4.10.1+ #20 SMP Sun Feb 26 21:22:45 IST 2017 x86_64 GNU/Linux
2017-03-02 / 22:58:39 ♒♒♒  ☺  
vs
log("hello world") exit () }'
Pass 1: parsed user script and 471 library scripts using
123528virt/48648res/6764shr/41840data kb, in 140usr/0sys/149real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using
124452virt/49620res/6956shr/42764data kb, in 10usr/0sys/5real ms.
Pass 3: translated to C into
"/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.c" using
124452virt/49620res/6956shr/42764data kb, in 0usr/0sys/1real ms.
/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.c:10:29: fatal
error: runtime_defines.h: No such file or directory
 #include "runtime_defines.h"
                             ^
compilation terminated.
/usr/src/linux-headers-4.9.0-2-common/scripts/Makefile.build:298: recipe for
target '/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.o'
failed
make[3]: ***
[/tmp/stapJNAO0g/stap_eed6170377f19ad002c0d2b53e5e7956_1067_src.o] Error 1
make[3]: *** Waiting for unfinished jobs....
/usr/src/linux-headers-4.9.0-2-common/Makefile:1507: recipe for target
'_module_/tmp/stapJNAO0g' failed
make[2]: *** [_module_/tmp/stapJNAO0g] Error 2
Makefile:150: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:8: recipe for target 'all' failed
make: *** [all] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_eed6170377f19ad002c0d2b53e5e7956_1067.ko" in
7810usr/760sys/9195real ms.
Pass 4: compilation failed.  [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
23:00 ♒♒♒    ☹  => 1  
Linux chutzpah 4.9.0-2-amd64 #1 SMP Debian 4.9.13-1 (2017-02-27) x86_64 GNU/Linux
23:00 ♒♒♒   ☺    
- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Gerald Turner
2017-03-03 19:21:51 UTC
Permalink
I modified systemtap buildrun.cxx so that the Makefile it produces
during Pass 3 doesn't have quotes around the include path, i.e.:

Before:

EXTRA_CFLAGS += -I"/usr/share/systemtap/runtime"

After:

EXTRA_CFLAGS += -I/usr/share/systemtap/runtime

Then stap works fine against the Debian kernel, running 4.10-1~exp1 at
the moment.

I cannot fathom how Debian's linux-headers packages would affect this
argument, but for the record, here's systemtap running gcc in Pass 3,
elided:

gcc-6 ... -I/usr/src/linux-headers-4.10.0-trunk-common/"/usr/share/systemtap/runtime" ...

And with buildrun.cxx modified so that quotations are removed from the
Makefile:

gcc-6 ... -I/usr/share/systemtap/runtime ...

How is /usr/src/linux-headers-4.10.0-trunk-common/ getting interpolated
into that path argument? Attached a Makefile produced by stap - maybe
somebody could guess why that interpolation occurs at line 125?

Apologies if I'm adding more noise than signal to the bug report. Frank
Ch. Eigler already addressed the quotation issue earlier in this bug
report.
--
Gerald Turner <***@unzane.com> Encrypted mail preferred!
OpenPGP: 4096R / CA89 B27A 30FA 66C5 1B80 3858 EC94 2276 FDB8 716D
Ben Hutchings
2017-03-04 01:39:58 UTC
Permalink
Control: tag -1 upstream patch

On Fri, 03 Mar 2017 23:10:56 +0530 Ritesh Raj Sarraf <***@debian.org> wrote:
[...]
 
We currently have systemtap 3.0 in Debian testing  where this problem was
reported.
 
In Experimental, I have also pushed the 3.1 release. The 3.1 release is reported
to work perfectly in our tests with the custom built kernel. The problem is only
seen with the Debian built Linux kernels.
[...]

I investigated this and found that it occurs when the kernel source and
object trees are separate (an "out-of-tree" build, not to be confused
with out-of-tree modules). We separate them in Debian kernel header
packages to avoid duplicating source files for each flavour.

When this is the case, the compiler is called in the root of the
object tree, and the kernel build system adjusts -I options in the
compiler flags to refer to subdirectories of the source tree if
necessary. Any directory name beginning with /, ./ or ../ is excluded
from this adjustment.

systemtap uses -I"/usr/share/systemtap/runtime", which ought to be
excluded... but make has no understanding of shell quoting, so it is
wrongly adjusted to something like
-I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime"
and runtime_defines.h cannot be found.

So my fix is to delete quotes before checking for the exclusions:

--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -206,7 +206,7 @@ hdr-inst := -f $(srctree)/scripts/Makefile.headersinst obj
# Prefix -I with $(srctree) if it is not an absolute path.
# skip if -I has no parameter
addtree = $(if $(patsubst -I%,%,$(1)), \
-$(if $(filter-out -I/% -I./% -I../%,$(1)),$(patsubst -I%,-I$(srctree)/%,$(1)),$(1)))
+$(if $(filter-out -I/% -I./% -I../%,$(subst $(quote),,$(subst $(squote),,$(1)))),$(patsubst -I%,-I$(srctree)/%,$(1)),$(1)))

# Find all -I options and call addtree
flags = $(foreach o,$($(1)),$(if $(filter -I%,$(o)),$(call addtree,$(o)),$(o)))
--- END ---

Ben.
--
Ben Hutchings
All the simple programs have been written, and all the good names
taken.
Ben Hutchings
2017-03-04 02:12:01 UTC
Permalink
On Sat, 2017-03-04 at 01:39 +0000, Ben Hutchings wrote:
[...]
Post by Ben Hutchings
I investigated this and found that it occurs when the kernel source and
object trees are separate (an "out-of-tree" build, not to be confused
with out-of-tree modules).  We separate them in Debian kernel header
packages to avoid duplicating source files for each flavour.
When this is the case, the compiler is called in the root of the
object tree, and the kernel build system adjusts -I options in the
compiler flags to refer to subdirectories of the source tree if
necessary.  Any directory name beginning with /, ./ or ../ is excluded
from this adjustment.
systemtap uses -I"/usr/share/systemtap/runtime", which ought to be
excluded... but make has no understanding of shell quoting, so it is
wrongly adjusted to something like
-I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime"
and runtime_defines.h cannot be found.
By the way, kbuild has been doing this ever since it started supporting
out-of-tree builds (around Linux 2.6.0). However, both the adjusted
and original -I options were used. Since Linux 4.8 only the adjusted
option is used.

Ben.
--
Ben Hutchings
All the simple programs have been written, and all the good names
taken.
Ritesh Raj Sarraf
2017-03-04 09:24:49 UTC
Permalink
Post by Ben Hutchings
[...]
Post by Ben Hutchings
I investigated this and found that it occurs when the kernel source and
object trees are separate (an "out-of-tree" build, not to be confused
with out-of-tree modules).  We separate them in Debian kernel header
packages to avoid duplicating source files for each flavour.
When this is the case, the compiler is called in the root of the
object tree, and the kernel build system adjusts -I options in the
compiler flags to refer to subdirectories of the source tree if
necessary.  Any directory name beginning with /, ./ or ../ is excluded
from this adjustment.
systemtap uses -I"/usr/share/systemtap/runtime", which ought to be
excluded... but make has no understanding of shell quoting, so it is
wrongly adjusted to something like
-I/usr/src/linux-headers-4.9.0-2-common/"/usr/share/systemtap/runtime"
and runtime_defines.h cannot be found.
By the way, kbuild has been doing this ever since it started supporting
out-of-tree builds (around Linux 2.6.0).  However, both the adjusted
and original -I options were used.  Since Linux 4.8 only the adjusted
option is used.
Thank you very much, Ben. I tested with your fix and it resolves the issue.
--
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Jack Henschel
2017-03-06 11:10:57 UTC
Permalink
Post by Ritesh Raj Sarraf
Thank you very much, Ben. I tested with your fix and it resolves the issue.
Can confirm, the patch fixes it.

Thank you Ben and Ritesh for your super quick actions! :-)

Greetings
Jack

Jack Henschel
2017-03-04 07:15:04 UTC
Permalink
Wow Ben, that was a really quick fix!

I'll test it on Monday and report back.
Ritesh Raj Sarraf
2017-03-01 17:26:48 UTC
Permalink
Hello Frank,

We have this bug report in Debian, where systemtap is looking for a local header
file rather than using the system provided one (provided by the systemtap-common
package).

The translate.cxx file writes the code. This looks like deliberately done, under
certain conditions. I am not sure what condition triggers this code.

Can you please provide any help ?

Running in verbose mode, it shows:

test -e include/generated/autoconf.h -a -e include/config/auto.conf ||
(                \
echo >&2;-                                                       \
echo >&2 "  ERROR: Kernel configuration is invalid.";           \
echo >&2 "         include/generated/autoconf.h or include/config/auto.conf are
missing.";\
echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix
it.";      \
echo >&2 ;                                                      \
/bin/false)


Those paths are provided by the linux-headers-xxx-amd64 (or equivalent arch)
package. In Debian, build/ and source/ paths are pointing to separate locations,
provided by separate package.

***@chutzpah:/lib/modules/4.9.0-2-amd64$ ls -l
total 4080
lrwxrwxrwx  1 root root      36 Feb 17 18:48 build -> /usr/src/linux-headers-
4.9.0-2-amd64/
drwxr-xr-x 12 root root    4096 Feb 20 22:57 kernel/
-rw-r--r--  1 root root 1005836 Mar  1 19:55 modules.alias
-rw-r--r--  1 root root  965121 Mar  1 19:55 modules.alias.bin
-rw-r--r--  1 root root    4027 Feb 27 21:28 modules.builtin
-rw-r--r--  1 root root    5485 Mar  1 19:55 modules.builtin.bin
-rw-r--r--  1 root root  393918 Mar  1 19:55 modules.dep
-rw-r--r--  1 root root  543473 Mar  1 19:55 modules.dep.bin
-rw-r--r--  1 root root     402 Mar  1 19:55 modules.devname
-rw-r--r--  1 root root  131711 Feb 27 21:28 modules.order
-rw-r--r--  1 root root     523 Mar  1 19:55 modules.softdep
-rw-r--r--  1 root root  489848 Mar  1 19:55 modules.symbols
-rw-r--r--  1 root root  605149 Mar  1 19:55 modules.symbols.bin
lrwxrwxrwx  1 root root      37 Mar  1 22:52 source -> /usr/src/linux-headers-
4.9.0-2-common/
drwxr-xr-x  3 root root    4096 Mar  1 19:55 updates/
22:52 ♒♒♒   ☺    


The full verbose log from stap is attached with this email.

I suspect it to be a Debian kernel problem but wanted to check with you first.


Thanks,
Ritesh
Post by Ritesh Raj Sarraf
Control: tag -1 +confirmed -moreinfo
Post by Ritesh Raj Sarraf
But in my logs, runtime_defines.h is properly detected. I'll try to run the
command on a Debian kernel to verify if it is broken or not.
I have been able to reproduce this on a Debian kernel.
prep                                                          
22:25 ♒♒♒   ☺    
exit()}'   
Pass 1: parsed user script and 119 library scripts using
115056virt/39720res/6372shr/33400data kb, in 130usr/0sys/340real ms.
Pass 2: analyzed script: 1 probe, 1 function, 4 embeds, 0 globals using
256132virt/182128res/7752shr/174476data kb, in 1770usr/160sys/6312real ms.
Pass 3: translated to C into
"/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c" using
256132virt/182376res/8000shr/174476data kb, in 0usr/0sys/7real ms.
/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.c:10:29: fatal
error: runtime_defines.h: No such file or directory
 #include "runtime_defines.h"
                             ^
compilation terminated.
/usr/src/linux-headers-4.9.0-2-common/scripts/Makefile.build:298: recipe for
target '/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.o' failed
make[3]: ***
[/tmp/stapUHT1si/stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538_src.o]
Error 1
make[3]: *** Waiting for unfinished jobs....
/usr/src/linux-headers-4.9.0-2-common/Makefile:1507: recipe for target
'_module_/tmp/stapUHT1si' failed
make[2]: *** [_module_/tmp/stapUHT1si] Error 2
Makefile:150: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:8: recipe for target 'all' failed
make: *** [all] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_7ef754ea40ea0990a9bfaf74753e8a9d_1538.ko" in
7560usr/630sys/10583real ms.
Pass 4: compilation failed.  [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
22:26 ♒♒♒    ☹  => 1  
--
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Frank Ch. Eigler
2017-03-01 17:30:14 UTC
Permalink
Hi -
[...] Running in verbose mode, it shows: [...] Those paths are
provided by the linux-headers-xxx-amd64 (or equivalent arch)
package. In Debian, build/ and source/ paths are pointing to
separate locations, provided by separate package. [...]
Could you provide a make V=1 transcript of any nontrivial out-of-tree
module (plus its Makefile) that builds & works on your kernel?

- FChE
Ritesh Raj Sarraf
2017-03-01 18:12:22 UTC
Permalink
Post by Frank Ch. Eigler
Hi -
[...]  Running in verbose mode, it shows: [...]  Those paths are
provided by the linux-headers-xxx-amd64 (or equivalent arch)
package. In Debian, build/ and source/ paths are pointing to
separate locations, provided by separate package. [...]
Could you provide a make V=1 transcript of any nontrivial out-of-tree
module (plus its Makefile) that builds & works on your kernel?
Here's a log attached. It was generated with V=1 through dkms on out-of-tree
module for broadcom-sta driver.
--
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Ritesh Raj Sarraf
2017-03-01 17:40:30 UTC
Permalink
This is specific to the linux-headers package and has nothing to do with
systemtap. Please ignore this one.
Post by Ritesh Raj Sarraf
test -e include/generated/autoconf.h -a -e include/config/auto.conf ||
(                \
echo >&2;-                                                       \
echo >&2 "  ERROR: Kernel configuration is invalid.";           \
echo >&2 "         include/generated/autoconf.h or include/config/auto.conf
are
missing.";\
echo >&2 "         Run 'make oldconfig && make prepare' on kernel src to fix
it.";      \
echo >&2 ;                                                      \
/bin/false)
- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Gerald Turner
2017-03-02 21:25:03 UTC
Permalink
FWIW, I attempted to kludge around the mangled include argument by
running stap with "-B 'EXTRA_CFLAGS += -I/usr/share/systemtap/runtime'".
This seems to work around the compilation failure of missing
runtime_defines.h in include path, however compiliation then fails due
to what appears to be API changes in Linux kernel 4.9.

For example:

In file included from /usr/share/systemtap/runtime/linux/runtime.h:209:0,
from /usr/share/systemtap/runtime/runtime.h:26,
from /tmp/stapqySw7o/stap_6ed5af627765aaab098e97da6b2931f0_2608_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h:50:13: error: too many arguments to function ‘get_user_pages’
ret = get_user_pages (tsk, mm, addr, 1, write, 1, &page, &vma);
^~~~~~~~~~~~~~

The prototype changed in Linux 4.9:

long get_user_pages(unsigned long start, unsigned long nr_pages,
int write, int force, struct page **pages,
struct vm_area_struct **vmas);

It looks like systemtap 3.1 is expecting the prototype in Linux versions
prior to 4.6:

long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
unsigned long start, unsigned long nr_pages,
int write, int force, struct page **pages,
struct vm_area_struct **vmas);

Log of systemtap 3.1 attached.
--
Gerald Turner <***@unzane.com> Encrypted mail preferred!
OpenPGP: 4096R / CA89 B27A 30FA 66C5 1B80 3858 EC94 2276 FDB8 716D
Ritesh Raj Sarraf
2017-03-03 11:40:51 UTC
Permalink
Hello Gerald,
Post by Gerald Turner
FWIW, I attempted to kludge around the mangled include argument by
running stap with "-B 'EXTRA_CFLAGS += -I/usr/share/systemtap/runtime'".
This seems to work around the compilation failure of missing
runtime_defines.h in include path, however compiliation then fails due
to what appears to be API changes in Linux kernel 4.9.
  In file included from /usr/share/systemtap/runtime/linux/runtime.h:209:0,
                   from /usr/share/systemtap/runtime/runtime.h:26,
                   from
  /usr/share/systemtap/runtime/linux/access_process_vm.h:50:13: error: too
many arguments to function ‘get_user_pages’
         ret = get_user_pages (tsk, mm, addr, 1, write, 1, &page, &vma);
               ^~~~~~~~~~~~~~
   long get_user_pages(unsigned long start, unsigned long nr_pages,
                               int write, int force, struct page **pages,
                               struct vm_area_struct **vmas);
It looks like systemtap 3.1 is expecting the prototype in Linux versions
This is very weird. I was so hoping that 3.1 would fix the interface change
related build failures. The 3.1 stap release worked fine for me locally on
custom build 4.10 kernel.

The 3.1 release notes also claim successful tests with 4.8 and 4.10 kernels.
Post by Gerald Turner
   long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
                       unsigned long start, unsigned long nr_pages,
                       int write, int force, struct page **pages,
                       struct vm_area_struct **vmas);
Log of systemtap 3.1 attached.
- --
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
Frank Ch. Eigler
2017-03-03 14:06:29 UTC
Permalink
Hi -
Post by Gerald Turner
FWIW, I attempted to kludge around the mangled include argument by
running stap with "-B 'EXTRA_CFLAGS += -I/usr/share/systemtap/runtime'".
This seems to work around the compilation failure of missing
runtime_defines.h in include path, however compiliation then fails due
to what appears to be API changes in Linux kernel 4.9.
Those errors all have systemtap-side porting logic associated with
them, a combination of compile-time (c #ifdefs) and build-time (kbuild
flags) conditionals. I suspect your workaround hid one problem and
created these others. We need to keep looking for the cause.

- FChE
Loading...