Debugging issue with gdbserver and a daemon on the target

[Laszlo Papp]

Hi,

I am having a bit of difficult time to get a breakpoint hit on my
daemon. My setup is as follows:

== Target ==

gdbserver --attach 192.168.0.32:2345 pid-of-my-daemon

== Host ==

./tmp/sysroots/x86_64-linux/usr/bin/armv5te-foo-linux-gnueabi/cgdb -d
./tmp/sysroots/x86_64-linux/usr/bin/armv5te-foo-linux-gnueabi/arm-foo-linux-gnueabi-gdb
-s ./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/usr/bin/.debug/foo
-e ./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/usr/bin/foo
-q -ex set sysroot
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs -ex
tar rem192.168.0.1:2345

Reading symbols from
/home/lpapp/Projects/Yocto/poky-dylan-9.0.1/build/tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/usr/bin/.debug/foo...done.
Remote debugging using 192.168.0.1:2345
Reading symbols from
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/librt.so.1...Reading
symbols from /home/lpapp/Projects/Yocto/poky-dylan-9.0.1/build/tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/root
fs/lib/.debug/librt-2.17.so...done.
done.
Loaded symbols for
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/librt.so.1
Reading symbols from
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libpthread.so.0...Reading
symbols from /home/lpapp/Projects/Yocto/poky-dylan-9.0.1/build/tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0
/rootfs/lib/.debug/libpthread-2.17.so...done.
done.
Loaded symbols for
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libpthread.so.0
Reading symbols from
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libgcc_s.so.1...Reading
symbols from /home/lpapp/Projects/Yocto/poky-dylan-9.0.1/build/tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/r
ootfs/lib/.debug/libgcc_s.so.1...done.
done.
Loaded symbols for
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libgcc_s.so.1
Reading symbols from
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libc.so.6...Reading
symbols from /home/lpapp/Projects/Yocto/poky-dylan-9.0.1/build/tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootf
s/lib/.debug/libc-2.17.so...done.
done.
Loaded symbols for
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libc.so.6
Reading symbols from
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/ld-linux.so.3...Reading
symbols from /home/lpapp/Projects/Yocto/poky-dylan-9.0.1/build/tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/r
ootfs/lib/.debug/ld-2.17.so...done.
done.
Loaded symbols for
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/ld-linux.so.3
Reading symbols from
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libnss_compat.so.2...Reading
symbols from /home/lpapp/Projects/Yocto/poky-dylan-9.0.1/build/tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0
-r0/rootfs/lib/.debug/libnss_compat-2.17.so...done.
done.
Loaded symbols for
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libnss_compat.so.2
Reading symbols from
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libnsl.so.1...Reading
symbols from /home/lpapp/Projects/Yocto/poky-dylan-9.0.1/build/tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/roo
tfs/lib/.debug/libnsl-2.17.so...done.
done.
Loaded symbols for
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/lib/libnsl.so.1
0x44ad26ec in select () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) set debug remote 1
(gdb) b foo
Sending packet: $maa1c,4#23...Packet received: 030052e1
Sending packet: $maa1c,4#23...Packet received: 030052e1
Sending packet: $maa1c,4#23...Packet received: 030052e1
Breakpoint 1 at 0xaa1c: file foo.c, line 99.
Sending packet: $qTStatus#49...Packet received:
(gdb) c
Continuing.
Sending packet: $qTStatus#49...Packet received:
Sending packet: $Z0,aa1c,4#6c...Packet received:
Packet Z0 (software-breakpoint) is NOT supported
Sending packet: $maa1c,4#23...Packet received: 030052e1
Sending packet: $Xaa1c,0:#44...Packet received: OK
binary downloading supported by target
Sending packet: $Xaa1c,4:\001#10...Packet received: OK
Sending packet: $m449e80c8,4#d6...Packet received: 1eff2fe1
Sending packet: $X449e80c8,4:\001#c3...Packet received: OK
Sending packet:
$QPassSignals:e;10;14;17;1a;1b;1c;21;24;25;2c;4c;#5f...Packet
received: OK
Sending packet: $vCont?#49...Packet received: vCont;c;C;s;S;t
Packet vCont (verbose-resume) is supported
Program received signal SIGINT, Interrupt.
0x44ad26ec in select () at ../sysdeps/unix/syscall-template.S:81
81 in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0 0x44ad26ec in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0002ac08 in bar (timeout=10, name=0x42f30 <yy_ec+56> "foo") at
src/socket.c:906
#2 0x0003284c in main (argc=0, argv=0x0) at src/bar.c:679
(gdb)


... And then I do some communication with the daemon where the foo
function is executed based on the logs, but the breakpoint is not hit.
I wished to try hardware breakpoints, but they are not presented on my
hardware.

Furthermore, if I use the same workflow on a binary that is
"one-shot", i.e. not running continuously as a daemon, the debugging
workflow for stopping at main works with exactly the aforementioned
software breakpoint issue.

I am completely clueless at this point. Do you know how I can debug a
daemon with gdbserver?

Cheers, L.

[Pedro Alves]

Post by Laszlo Papp
gdbserver --attach 192.168.0.32:2345 pid-of-my-daemon
(gdb) bt
#0 0x44ad26ec in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0002ac08 in bar (timeout=10, name=0x42f30 <yy_ec+56> "foo") at
src/socket.c:906
#2 0x0003284c in main (argc=0, argv=0x0) at src/bar.c:679
(gdb)
... And then I do some communication with the daemon where the foo
function is executed based on the logs, but the breakpoint is not hit.
I wished to try hardware breakpoints, but they are not presented on my
hardware.
Furthermore, if I use the same workflow on a binary that is
"one-shot", i.e. not running continuously as a daemon, the debugging
workflow for stopping at main works with exactly the aforementioned
software breakpoint issue.
I am completely clueless at this point. Do you know how I can debug a
daemon with gdbserver?

"daemon" and "select" makes me think "fork". If the daemon is handling
requests by forking a child, and then it's the child that calls 'foo',
then this is expected, as GDBserver doesn't know how to follow forks
currently. It's WIP, patches have been posted. Meanwhile, the usual
thing to do it to attach to the child process the daemon spawns instead
of the main daemon pid. You'll usually do that by adding a busyloop in
the child somewhere, like:

volatile int gdb_here;

while (!gdb_here)
sleep (1);

and after attaching to the child, do "print gdb_here = 1; continue".

Thanks,
Pedro Alves

[Laszlo Papp]

Post by Pedro Alves

Post by Laszlo Papp
gdbserver --attach 192.168.0.32:2345 pid-of-my-daemon
(gdb) bt
#0 0x44ad26ec in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0002ac08 in bar (timeout=10, name=0x42f30 <yy_ec+56> "foo") at
src/socket.c:906
#2 0x0003284c in main (argc=0, argv=0x0) at src/bar.c:679
(gdb)
... And then I do some communication with the daemon where the foo
function is executed based on the logs, but the breakpoint is not hit.
I wished to try hardware breakpoints, but they are not presented on my
hardware.
Furthermore, if I use the same workflow on a binary that is
"one-shot", i.e. not running continuously as a daemon, the debugging
workflow for stopping at main works with exactly the aforementioned
software breakpoint issue.
I am completely clueless at this point. Do you know how I can debug a
daemon with gdbserver?

"daemon" and "select" makes me think "fork". If the daemon is handling
requests by forking a child, and then it's the child that calls 'foo',
then this is expected, as GDBserver doesn't know how to follow forks
currently. It's WIP, patches have been posted. Meanwhile, the usual
thing to do it to attach to the child process the daemon spawns instead
of the main daemon pid. You'll usually do that by adding a busyloop in
volatile int gdb_here;
while (!gdb_here)
sleep (1);
and after attaching to the child, do "print gdb_here = 1; continue".
Thanks,
Pedro Alves

Thanks Pedro for the prompt reply. Unfortunately, I am already
attaching to the child right after the fork. I wonder if this can
happen if some source file was missing? Btw:

gdbserver --version
GNU gdbserver (GDB) 7.5.1

arm-polatis-linux-gnueabi-gdb --version
GNU gdb (GDB) 7.5.1

[Pedro Alves]

Post by Laszlo Papp

Post by Pedro Alves

Post by Laszlo Papp
gdbserver --attach 192.168.0.32:2345 pid-of-my-daemon
(gdb) bt
#0 0x44ad26ec in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0002ac08 in bar (timeout=10, name=0x42f30 <yy_ec+56> "foo") at
src/socket.c:906
#2 0x0003284c in main (argc=0, argv=0x0) at src/bar.c:679
(gdb)
... And then I do some communication with the daemon where the foo
function is executed based on the logs, but the breakpoint is not hit.
I wished to try hardware breakpoints, but they are not presented on my
hardware.
Furthermore, if I use the same workflow on a binary that is
"one-shot", i.e. not running continuously as a daemon, the debugging
workflow for stopping at main works with exactly the aforementioned
software breakpoint issue.
I am completely clueless at this point. Do you know how I can debug a
daemon with gdbserver?

"daemon" and "select" makes me think "fork". If the daemon is handling
requests by forking a child, and then it's the child that calls 'foo',
then this is expected, as GDBserver doesn't know how to follow forks
currently. It's WIP, patches have been posted. Meanwhile, the usual
thing to do it to attach to the child process the daemon spawns instead
of the main daemon pid. You'll usually do that by adding a busyloop in
volatile int gdb_here;
while (!gdb_here)
sleep (1);
and after attaching to the child, do "print gdb_here = 1; continue".
Thanks,
Pedro Alves

Thanks Pedro for the prompt reply. Unfortunately, I am already
attaching to the child right after the fork. I wonder if this can
happen if some source file was missing?

It shouldn't. Source files are only used for display. Where to
place a breakpoint is derived from the debug info in the binary.

I'd suggest just trying to step through the code instead of
putting a break at "foo", and see if that much works. On an
arm system, stepping is actually implemented with magic
breakpoints behind the scenes.

Post by Laszlo Papp
gdbserver --version
GNU gdbserver (GDB) 7.5.1
arm-polatis-linux-gnueabi-gdb --version
GNU gdb (GDB) 7.5.1

Knee-jerk reaction is to suggest a more recent GDB/GDBserver. Note
building these isn't very hard. There aren't that many
dependencies.

Thanks,
Pedro Alves

[Laszlo Papp]

Post by Pedro Alves

Post by Laszlo Papp

Post by Pedro Alves

Post by Laszlo Papp
gdbserver --attach 192.168.0.32:2345 pid-of-my-daemon
(gdb) bt
#0 0x44ad26ec in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x0002ac08 in bar (timeout=10, name=0x42f30 <yy_ec+56> "foo") at
src/socket.c:906
#2 0x0003284c in main (argc=0, argv=0x0) at src/bar.c:679
(gdb)
... And then I do some communication with the daemon where the foo
function is executed based on the logs, but the breakpoint is not hit.
I wished to try hardware breakpoints, but they are not presented on my
hardware.
Furthermore, if I use the same workflow on a binary that is
"one-shot", i.e. not running continuously as a daemon, the debugging
workflow for stopping at main works with exactly the aforementioned
software breakpoint issue.
I am completely clueless at this point. Do you know how I can debug a
daemon with gdbserver?

"daemon" and "select" makes me think "fork". If the daemon is handling
requests by forking a child, and then it's the child that calls 'foo',
then this is expected, as GDBserver doesn't know how to follow forks
currently. It's WIP, patches have been posted. Meanwhile, the usual
thing to do it to attach to the child process the daemon spawns instead
of the main daemon pid. You'll usually do that by adding a busyloop in
volatile int gdb_here;
while (!gdb_here)
sleep (1);
and after attaching to the child, do "print gdb_here = 1; continue".
Thanks,
Pedro Alves

Thanks Pedro for the prompt reply. Unfortunately, I am already
attaching to the child right after the fork. I wonder if this can
happen if some source file was missing?

It shouldn't. Source files are only used for display. Where to
place a breakpoint is derived from the debug info in the binary.
I'd suggest just trying to step through the code instead of
putting a break at "foo", and see if that much works. On an
arm system, stepping is actually implemented with magic
breakpoints behind the scenes.

Post by Laszlo Papp
gdbserver --version
GNU gdbserver (GDB) 7.5.1
arm-polatis-linux-gnueabi-gdb --version
GNU gdb (GDB) 7.5.1

Knee-jerk reaction is to suggest a more recent GDB/GDBserver. Note
building these isn't very hard. There aren't that many
dependencies.
Thanks,
Pedro Alves

Hmm, it seems that the stripped binary on the target and the one on
the host were out-of-sync. This is really strange since I have not
changed the source code. Seems different compilations still can get
out-of-sync for the same code so that when I rebuild the same source
code, I always need to update the binary on the target, too?

Anyway, now I only have problems with finding the sources file to view
them in cgdb. I do not know why it is wrong, but it seems to be. As
you can see the paths are set up for dwarf correctly:

readelf --debug-dump
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/usr/bin/.debug/foo
| grep DW_AT_comp_dir
<35> DW_AT_comp_dir :
/usr/src/debug///////////////////////////////////////////////////////////////////////////////////eglibc/2.17-r3/eglibc-2.17/libc/csu
<df> DW_AT_comp_dir : (indirect string, offset: 0x31):
/usr/src/debug/eglibc/2.17-r3/eglibc-2.17/libc/csu
<183> DW_AT_comp_dir :
/usr/src/debug///////////////////////////////////////////////////////////////////////////////////eglibc/2.17-r3/eglibc-2.17/libc/csu
<22d> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<102f> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<2e98> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<3e34> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<471a> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<4c5f> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<55b9> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<84b8> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<94d1> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<a1e9> DW_AT_comp_dir : (indirect string, offset: 0x749):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/bar
<af53> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<c382> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<c952> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<d281> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<dcc0> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<e093> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<e953> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<efd6> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<fb07> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<10cd9> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<127e6> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<132ab> DW_AT_comp_dir : (indirect string, offset: 0x3874):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/baz
<13bf3> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<1556d> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<162d4> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<17481> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<17900> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<18481> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<1903f> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<19432> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<19524> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<19fe1> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<1d2b4> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<1dc34> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<1f2c1> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<1fa4f> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<1fe13> DW_AT_comp_dir : (indirect string, offset: 0x585a):
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
<203e4> DW_AT_comp_dir : (indirect string, offset: 0x7cfa):
/usr/src/debug/flex/2.5.35-r3/flex-2.5.35
<2042a> DW_AT_comp_dir : (indirect string, offset: 0x31):
/usr/src/debug/eglibc/2.17-r3/eglibc-2.17/libc/csu
<2056c> DW_AT_comp_dir : (indirect string, offset: 0x7de0):
/usr/src/debug/eglibc/2.17-r3/eglibc-2.17/libc/io
<20833> DW_AT_comp_dir :
/usr/src/debug///////////////////////////////////////////////////////////////////////////////////eglibc/2.17-r3/eglibc-2.17/libc/csu
DW_AT_comp_dir DW_FORM_string
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_string
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_strp
DW_AT_comp_dir DW_FORM_string

... yet, gdb says src/bar.c cannot be found even though it should be
in the aforementioned
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
path, provided my sysroot setting is good above, but if that was not
good, it would not load the binaries anyway, right? So, if I set that
path one line above with the "-d" option to gdb, then the source file
can be viewed. What may be going on here?

Thanks in advance. I am so desperately lost. :(

[Pedro Alves]

Post by Laszlo Papp
Hmm, it seems that the stripped binary on the target and the one on
the host were out-of-sync. This is really strange since I have not
changed the source code. Seems different compilations still can get
out-of-sync for the same code so that when I rebuild the same source
code, I always need to update the binary on the target, too?

Ideally, if you used the exact same inputs, and the exact same tools,
and the same exact same tool options, the output is the same. You
can check that with md5sum, or some such.

Post by Laszlo Papp
Anyway, now I only have problems with finding the sources file to view
them in cgdb. I do not know why it is wrong, but it seems to be. As
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
... yet, gdb says src/bar.c cannot be found even though it should be
in the aforementioned
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
path, provided my sysroot setting is good above, but if that was not
good, it would not load the binaries anyway, right?

Correct.

Post by Laszlo Papp
So, if I set that
path one line above with the "-d" option to gdb, then the source file
can be viewed. What may be going on here?

That sounds like the expected behavior, as the source directory knobs
are independent from the sysroot setting. See:

https://sourceware.org/gdb/current/onlinedocs/gdb/Source-Path.html

Post by Laszlo Papp
Thanks in advance. I am so desperately lost. :(

I think I'm lost on which part you are lost. :-)

Thanks,
Pedro Alves

[Laszlo Papp]

Post by Pedro Alves

Post by Laszlo Papp
Hmm, it seems that the stripped binary on the target and the one on
the host were out-of-sync. This is really strange since I have not
changed the source code. Seems different compilations still can get
out-of-sync for the same code so that when I rebuild the same source
code, I always need to update the binary on the target, too?

Ideally, if you used the exact same inputs, and the exact same tools,
and the same exact same tool options, the output is the same. You
can check that with md5sum, or some such.

Thanks. I will check that next time.

Post by Pedro Alves

Post by Laszlo Papp
Anyway, now I only have problems with finding the sources file to view
them in cgdb. I do not know why it is wrong, but it seems to be. As
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
... yet, gdb says src/bar.c cannot be found even though it should be
in the aforementioned
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
path, provided my sysroot setting is good above, but if that was not
good, it would not load the binaries anyway, right?

Correct.

Thanks.

Post by Pedro Alves

Post by Laszlo Papp
So, if I set that
path one line above with the "-d" option to gdb, then the source file
can be viewed. What may be going on here?

That sounds like the expected behavior, as the source directory knobs
https://sourceware.org/gdb/current/onlinedocs/gdb/Source-Path.html

Heh, coincidentally I started my day with reading that before, but
thanks. At least, it is a confirmation that I am on the right track.
:)

Post by Pedro Alves

Post by Laszlo Papp
Thanks in advance. I am so desperately lost. :(

I think I'm lost on which part you are lost. :-)

Well, I am not sure why it does not work. Yocto generates the images
for me, but that is not important here as long as the dwarf
information looks correct after its operation in the provided readelf
output. So, provided that the dwarf information looks correct, and the
files are there, I am not sure why gdb cannot find src/bar.c in
.../git/meh (path above). I will check "show directories" what gdb
sees in my case, but if that does not help, I am not sure how to make
it work. Got a clue?

[Laszlo Papp]

Post by Laszlo Papp

Post by Pedro Alves

Post by Laszlo Papp
Hmm, it seems that the stripped binary on the target and the one on
the host were out-of-sync. This is really strange since I have not
changed the source code. Seems different compilations still can get
out-of-sync for the same code so that when I rebuild the same source
code, I always need to update the binary on the target, too?

Ideally, if you used the exact same inputs, and the exact same tools,
and the same exact same tool options, the output is the same. You
can check that with md5sum, or some such.

Thanks. I will check that next time.

Post by Pedro Alves

Post by Laszlo Papp
Anyway, now I only have problems with finding the sources file to view
them in cgdb. I do not know why it is wrong, but it seems to be. As
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
... yet, gdb says src/bar.c cannot be found even though it should be
in the aforementioned
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
path, provided my sysroot setting is good above, but if that was not
good, it would not load the binaries anyway, right?

Correct.

Thanks.

Post by Pedro Alves

Post by Laszlo Papp
So, if I set that
path one line above with the "-d" option to gdb, then the source file
can be viewed. What may be going on here?

That sounds like the expected behavior, as the source directory knobs
https://sourceware.org/gdb/current/onlinedocs/gdb/Source-Path.html

Heh, coincidentally I started my day with reading that before, but
thanks. At least, it is a confirmation that I am on the right track.
:)

Post by Pedro Alves

Post by Laszlo Papp
Thanks in advance. I am so desperately lost. :(

I think I'm lost on which part you are lost. :-)

Well, I am not sure why it does not work. Yocto generates the images
for me, but that is not important here as long as the dwarf
information looks correct after its operation in the provided readelf
output. So, provided that the dwarf information looks correct, and the
files are there, I am not sure why gdb cannot find src/bar.c in
.../git/meh (path above). I will check "show directories" what gdb
sees in my case, but if that does not help, I am not sure how to make
it work. Got a clue?

It seems to just show the default:

(gdb) show directories
Source directories searched: $cdir:$cwd
(gdb) show substitute-path
List of all source path substitution rules:
(gdb)

To be detailed, this is the path to the source file in question:

-> ./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/foo/foo.c

This is where gdb says:

(gdb) c
Continuing.

Breakpoint 1, foo (sp=0x5e968) at foo.c:99
99 foo.c: No such file or directory.
(gdb) n
100 in foo.c
(gdb)

Readelf says again:

-> e.g. /usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/foo

As you can see the two paths differ in
"./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs",
but that is set as sysroot when I run gdb. Do I also need to set this
as a substitute path, too, then?

[Laszlo Papp]

Post by Laszlo Papp

Post by Laszlo Papp

Post by Pedro Alves

Post by Laszlo Papp
Hmm, it seems that the stripped binary on the target and the one on
the host were out-of-sync. This is really strange since I have not
changed the source code. Seems different compilations still can get
out-of-sync for the same code so that when I rebuild the same source
code, I always need to update the binary on the target, too?

Ideally, if you used the exact same inputs, and the exact same tools,
and the same exact same tool options, the output is the same. You
can check that with md5sum, or some such.

Thanks. I will check that next time.

Post by Pedro Alves

Post by Laszlo Papp
Anyway, now I only have problems with finding the sources file to view
them in cgdb. I do not know why it is wrong, but it seems to be. As
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
... yet, gdb says src/bar.c cannot be found even though it should be
in the aforementioned
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
path, provided my sysroot setting is good above, but if that was not
good, it would not load the binaries anyway, right?

Correct.

Thanks.

Post by Pedro Alves

Post by Laszlo Papp
So, if I set that
path one line above with the "-d" option to gdb, then the source file
can be viewed. What may be going on here?

That sounds like the expected behavior, as the source directory knobs
https://sourceware.org/gdb/current/onlinedocs/gdb/Source-Path.html

Heh, coincidentally I started my day with reading that before, but
thanks. At least, it is a confirmation that I am on the right track.
:)

Post by Pedro Alves

Post by Laszlo Papp
Thanks in advance. I am so desperately lost. :(

I think I'm lost on which part you are lost. :-)

Well, I am not sure why it does not work. Yocto generates the images
for me, but that is not important here as long as the dwarf
information looks correct after its operation in the provided readelf
output. So, provided that the dwarf information looks correct, and the
files are there, I am not sure why gdb cannot find src/bar.c in
.../git/meh (path above). I will check "show directories" what gdb
sees in my case, but if that does not help, I am not sure how to make
it work. Got a clue?

(gdb) show directories
Source directories searched: $cdir:$cwd
(gdb) show substitute-path
(gdb)
-> ./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/foo/foo.c
(gdb) c
Continuing.
Breakpoint 1, foo (sp=0x5e968) at foo.c:99
99 foo.c: No such file or directory.
(gdb) n
100 in foo.c
(gdb)
-> e.g. /usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/foo
As you can see the two paths differ in
"./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs",
but that is set as sysroot when I run gdb. Do I also need to set this
as a substitute path, too, then?

Oh, and more thing to provide as much information as possible to you:

(gdb) info source
Current source file is foo.c
Compilation directory is
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/foo
Source language is c.
Compiled with DWARF 2 debugging format.
Does not include preprocessor macro info.
(gdb)

[Laszlo Papp]

Post by Laszlo Papp

Post by Laszlo Papp

Post by Laszlo Papp

Post by Pedro Alves

Post by Laszlo Papp
Hmm, it seems that the stripped binary on the target and the one on
the host were out-of-sync. This is really strange since I have not
changed the source code. Seems different compilations still can get
out-of-sync for the same code so that when I rebuild the same source
code, I always need to update the binary on the target, too?

Ideally, if you used the exact same inputs, and the exact same tools,
and the same exact same tool options, the output is the same. You
can check that with md5sum, or some such.

Thanks. I will check that next time.

Post by Pedro Alves

Post by Laszlo Papp
Anyway, now I only have problems with finding the sources file to view
them in cgdb. I do not know why it is wrong, but it seems to be. As
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
... yet, gdb says src/bar.c cannot be found even though it should be
in the aforementioned
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/meh
path, provided my sysroot setting is good above, but if that was not
good, it would not load the binaries anyway, right?

Correct.

Thanks.

Post by Pedro Alves

Post by Laszlo Papp
So, if I set that
path one line above with the "-d" option to gdb, then the source file
can be viewed. What may be going on here?

That sounds like the expected behavior, as the source directory knobs
https://sourceware.org/gdb/current/onlinedocs/gdb/Source-Path.html

Heh, coincidentally I started my day with reading that before, but
thanks. At least, it is a confirmation that I am on the right track.
:)

Post by Pedro Alves

Post by Laszlo Papp
Thanks in advance. I am so desperately lost. :(

I think I'm lost on which part you are lost. :-)

Well, I am not sure why it does not work. Yocto generates the images
for me, but that is not important here as long as the dwarf
information looks correct after its operation in the provided readelf
output. So, provided that the dwarf information looks correct, and the
files are there, I am not sure why gdb cannot find src/bar.c in
.../git/meh (path above). I will check "show directories" what gdb
sees in my case, but if that does not help, I am not sure how to make
it work. Got a clue?

(gdb) show directories
Source directories searched: $cdir:$cwd
(gdb) show substitute-path
(gdb)
-> ./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/foo/foo.c
(gdb) c
Continuing.
Breakpoint 1, foo (sp=0x5e968) at foo.c:99
99 foo.c: No such file or directory.
(gdb) n
100 in foo.c
(gdb)
-> e.g. /usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/foo
As you can see the two paths differ in
"./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs",
but that is set as sysroot when I run gdb. Do I also need to set this
as a substitute path, too, then?

(gdb) info source
Current source file is foo.c
Compilation directory is
/usr/src/debug/foo-git/AUTOINC+0c2cbe33e653afa335ca25156d293552001228fa-r0/git/foo
Source language is c.
Compiled with DWARF 2 debugging format.
Does not include preprocessor macro info.
(gdb)

This made it work, but is it the best thing to do?

-ex "set substitute-path /usr/src/debug/
./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs/usr/src/debug/"

Also, can someone verify that it is also supposed to work if I just
use directories as claimed in here? I cannot because it does not work
in my version, but it is not the latest!

http://stackoverflow.com/questions/1103161/gdb-searching-for-source-directories/1327343#comment35550119_1327343

If yes, which one to use?

[Pedro Alves]

Post by Laszlo Papp
As you can see the two paths differ in
"./tmp/work/foo-foo-linux-gnueabi/foo-core-image-dbg/1.0-r0/rootfs",
but that is set as sysroot when I run gdb.

sysroot is only for pointing gdb at a copy of the system root on
the target (the binaries), not sources.

Post by Laszlo Papp
Do I also need to set this as a substitute path, too, then?

Looks like it.

Thanks,
Pedro Alves