Debugging .so files / missing ".interp" section

[Karim Yaghmour]

I'm running into these weird problems trying to debug .so files with
some AOSPs and I'm wondering if this is just pilot error. Let me explain.

Basically I'm trying to set a breakpoint into some of the system
services' JNI using gdb and step from there. This works just fine in
AOSP 4.4 but not 4.3, 4.2 or 4.1.

Putting 4.3 aside for a moment, it seems the toolchains used to build
4.2. and 4.1 will create .so ELF files that don't have a ".interp"
section pointing to "/system/bin/linker"; though that section is there
for ELF binaries. The .so ELF files in 4.4 and 4.3 do have those
sections. Still, I wasn't able to get it to work in 4.3 because I'd get
a SIGTRAP, but that's another story (apparently has something to do with
gdb thread support being broken.)

Is anybody else seeing similar issues? Am I barking up the wrong tree?

P.S.: If you don't know what this problem is about, just try attaching
to the system_server's PID using "gdb --attach localhost:2345 PID" and
then use the AOSP cross-debugger in prebuilts/.... to set a breakpoint
at some valid line number in say com_android_server_LightsService.cpp
and tell gdb to continue. If you can go through this without gdb
complaining about unresolvable addresses or stopping because of a
SIGTRAP, you're probably ok.
--
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour

[Video Guy]

I ran into same issue with Android 4.0.3 (i.e. Icecream Sandwitch). I
couldn't get gdb from Eclipse working with system_server. But I was able to
attach and debug SurfaceFlinger process from same AOSP.

Can you check if you can attach to SurfaceFlinger and step through code?
If yes, then it is something to do with system_server only.

[Karim Yaghmour]

I'm pretty sure I can debug the surfaceflinger process. I was able to
attach to pretty much any standalone C daemon or command-line tool.
Those ones have the ".interp" section: if you do a "readelf -a
out/target/product/foobar/system/bin/* | grep "\.interp"", there should
be plenty of results. If you do the same on system/lib/*, you should get
zero results unless you're using 4.3 or 4.4.

It's good to know that others may be running into similar issues. And
I'm assuming someone within the Android team must've run into this too
because the toolchain was eventually fixed. Unless, as I said earlier,
this is all pilot error and we're just not doing something right.

P.S.: These observations are all done on ARM. I have no idea how the
toolchains for the other archs perform.
--
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour

Post by Video Guy
I ran into same issue with Android 4.0.3 (i.e. Icecream Sandwitch). I
couldn't get gdb from Eclipse working with system_server. But I was able
to attach and debug SurfaceFlinger process from same AOSP.
Can you check if you can attach to SurfaceFlinger and step through code?
If yes, then it is something to do with system_server only.
--
You received this message because you are subscribed to the Google
Groups "android-platform" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/android-platform.
For more options, visit https://groups.google.com/groups/opt_out.

[Video Guy]

One difference I can see between surfaceflinger and System_server is that
one is pure native process (i.e. SurfaceFlinger) where as the other one is
Java app with substantial amount Java code as well as lot of native code
instantiated through JNI.

I debugged J2SE style apps that load JNI from IDEs on dekstops. I was able
to put break points in Java code with one IDE and do the same in native JNI
libs using different IDE with both IDEs debugging the same process.

I wanted to step through libInput.so loaded by system_server. Tried for
more than a week. EClipse gdb (this is arm gdb debugging remote process
through gdbserver) couldn't load symbols nor recognize the breakpoints. I
gave up.

I hope someone from Google monitoring this forum could help.

Joe

[Video Guy]

I tried to debug again with Eclipse. When the gdb is attached, I see the
following errors in the log.

Cannot access memory at address 0x65746f6b
Cannot access memory at address 0x65746f6b
Cannot access memory at address 0x65746f6b
Cannot access memory at address 0x65746f6b

Even though I have LOCAL_CFLAGS include " -O0 -g ", the breakpoints aren't
recognized.

Joe

[Karim Yaghmour]

Yes, that's similar to what I was seeing. Check if the ".so" files in
your "/system/lib" have an ".interp" section. I bet they don't.
--
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour

Post by Video Guy
I tried to debug again with Eclipse. When the gdb is attached, I see the
following errors in the log.
Cannot access memory at address 0x65746f6b
Cannot access memory at address 0x65746f6b
Cannot access memory at address 0x65746f6b
Cannot access memory at address 0x65746f6b
Even though I have LOCAL_CFLAGS include " -O0 -g ", the breakpoints
aren't recognized.
Joe
--
You received this message because you are subscribed to the Google
Groups "android-platform" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/android-platform.
For more options, visit https://groups.google.com/groups/opt_out.

[Video Guy]

I don't see .interp section for any of the .so files under ../lib folder.
But I do see this for the executables though (i.e. the files under bin
folder).

I am not sure .interp is the problem. For example, surfaceflinger debugging
works fine even though the core .so file where most of the surface flinger
code is (linsurfaceflinger.so) has missing .interp section.

My guess is it is something to do with Java process vs native process.

[Karim Yaghmour]

Hmm. ok. I've got some homework then.

FWIW, I don't see the difference between native and Java. It's the same
Bionic linker either way. dalivk/vm/Native.cpp:dvmLoadNativeCode() does
a dlopen() as I'd expect to, which is the same thing the linker in
bionic/linker/ does.

But again, I'm clearly missing something.
--
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour

Post by Video Guy
I don't see .interp section for any of the .so files under ../lib
folder. But I do see this for the executables though (i.e. the files
under bin folder).
I am not sure .interp is the problem. For example, surfaceflinger
debugging works fine even though the core .so file where most of the
surface flinger code is (linsurfaceflinger.so) has missing .interp section.
My guess is it is something to do with Java process vs native process.
--
You received this message because you are subscribed to the Google
Groups "android-platform" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/android-platform.
For more options, visit https://groups.google.com/groups/opt_out.

[Video Guy]

Karim
I figured what the problem is. Instead of selecting system_server binary,
choose app_process binary.

Then I was able to attach to system_server and step through some of the C++
code in libinput.so.

I debug using remote debugging.

This is how I got it working.

1) Build app_process with debug symbols.
cd to frameworks/base/cmds/app_process and invoke mm like below.

mm -B LOCAL_STRIP_MODULE=false LOCAL_CFLAGS=" -O0 -g "

copy the app_process from
out/target/product/<productname>/symbols/system/bin folder to android
device /system/bin directory.

2) Lets say you want to step through code in frameworks/base/services/input
directory.
cd to that folder and invoke mm as in step (1).

There is fine print with LOCAL_CFLAGS. If the Android.mk file in the
directory has any line like "LOCAL_CFLAGS+=..." in it, passing it as part
of mm breaks the build. In those cases, you edit the Android.mk file and
add "-O0 -g" flag before invoking mm command.

Copy libinput.so to android device's /system/lib directory.

Reboot android device.

3) Adb connect and then adb shell into android device.

4) Invoke gdbserver like below.

gdbserver :10000 --attach `pidof system_server`

Now gdbserver is waiting for connection.


5) Create eclipse Remote debug configuration with the right gdb for target
device and connect to android device from your PC.
MAKE SURE YOU CHOOSE out/target/product/<product
name>/symbols/system/bin/app_process as binary instead of system_server.
This step is very important.

4) Now you can place breakpoints in libinput related .cpp/.c files and gdb
works reliably.

Post by Karim Yaghmour
Hmm. ok. I've got some homework then.
FWIW, I don't see the difference between native and Java. It's the same
Bionic linker either way. dalivk/vm/Native.cpp:dvmLoadNativeCode() does
a dlopen() as I'd expect to, which is the same thing the linker in
bionic/linker/ does.
But again, I'm clearly missing something.
--
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour

Post by Video Guy
I don't see .interp section for any of the .so files under ../lib
folder. But I do see this for the executables though (i.e. the files
under bin folder).
I am not sure .interp is the problem. For example, surfaceflinger
debugging works fine even though the core .so file where most of the
surface flinger code is (linsurfaceflinger.so) has missing .interp

section.

Post by Video Guy
My guess is it is something to do with Java process vs native process.
--
You received this message because you are subscribed to the Google
Groups "android-platform" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/android-platform.
For more options, visit https://groups.google.com/groups/opt_out.