Discussion:
[arch-general] debug package repositories again
Joe Julian
2015-08-12 19:36:22 UTC
Permalink
It's been hashed and rehashed, but I need to bring this up again. Every
other popular distro has a way of debugging without completely
rebuilding a package, saving countless combined hours.

Why can't we just have a way of installing the symbols?

When I find a bug, I don't want to spend another hour building python,
for instance. I want to install the symbols, open up gdb, and figure out
where the problem is. I'll report it, probably fix it, and we'll all be
happy.

Allan has already added the features necessary to make this possible.

Why is this so hard? What can I do to help make this a reality?

https://lists.archlinux.org/pipermail/arch-dev-public/2013-April/024812.html
https://lists.archlinux.org/pipermail/arch-general/2015-April/038794.html
Leonid Isaev
2015-08-13 00:02:34 UTC
Permalink
Post by Joe Julian
It's been hashed and rehashed, but I need to bring this up again. Every
other popular distro has a way of debugging without completely rebuilding a
package, saving countless combined hours.
Why can't we just have a way of installing the symbols?
Not enough manpower...
Post by Joe Julian
Allan has already added the features necessary to make this possible.
Why is this so hard? What can I do to help make this a reality?
Well, there has to be a separate [debug] repo which (most importantly) must be
kept in-sync with the main ones, or more packages in current repos. Then,
someone has to actually build those packages.

I would personally prefer that most packages come with debugging enabled by
default. Surely, there will be a performance cost, but speed is not crucial in
most cases.

Cheers,
--
Leonid Isaev
GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4
C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
Joe Julian
2015-08-13 00:09:27 UTC
Permalink
Post by Leonid Isaev
Post by Joe Julian
It's been hashed and rehashed, but I need to bring this up again. Every
other popular distro has a way of debugging without completely rebuilding a
package, saving countless combined hours.
Why can't we just have a way of installing the symbols?
Not enough manpower...
There would be no more manpower needed than there is now. Set the debug
option to makepkg and put the debug package created in the debug repo.
Post by Leonid Isaev
Post by Joe Julian
Allan has already added the features necessary to make this possible.
Why is this so hard? What can I do to help make this a reality?
Well, there has to be a separate [debug] repo which (most importantly) must be
kept in-sync with the main ones, or more packages in current repos. Then,
someone has to actually build those packages.
I would personally prefer that most packages come with debugging enabled by
default. Surely, there will be a performance cost, but speed is not crucial in
most cases.
Cheers,
There's no performance impact, just disk space and bandwidth.
Evan Penner
2015-08-13 13:11:58 UTC
Permalink
Post by Joe Julian
Post by Leonid Isaev
Post by Joe Julian
It's been hashed and rehashed, but I need to bring this up again. Every
other popular distro has a way of debugging without completely rebuilding a
package, saving countless combined hours.
Why can't we just have a way of installing the symbols?
Not enough manpower...
There would be no more manpower needed than there is now. Set the debug
option to makepkg and put the debug package created in the debug repo.
I'll submit a feature request on the bugtracker.
Post by Joe Julian
Post by Leonid Isaev
Allan has already added the features necessary to make this possible.
Post by Joe Julian
Why is this so hard? What can I do to help make this a reality?
Well, there has to be a separate [debug] repo which (most importantly) must be
kept in-sync with the main ones, or more packages in current repos. Then,
someone has to actually build those packages.
I would personally prefer that most packages come with debugging enabled by
default. Surely, there will be a performance cost, but speed is not crucial in
most cases.
Cheers,
There's no performance impact, just disk space and bandwidth.
Bandwidth is probably the main problem, although anyone who wants to debug
will probably be fine with that.
Leonid Isaev
2015-08-13 13:47:43 UTC
Permalink
Post by Evan Penner
Post by Joe Julian
Post by Leonid Isaev
I would personally prefer that most packages come with debugging enabled by
default. Surely, there will be a performance cost, but speed is not crucial in
most cases.
Cheers,
There's no performance impact, just disk space and bandwidth.
Bandwidth is probably the main problem, although anyone who wants to debug
will probably be fine with that.
I think you guys misunderstood me. The biggest problem IMHO with building debug
versions locally is not compiling itself, but setting up the environment. So, I
meant that packages come with debugging enabled (compiled with gcc -O0 -g and
perhaps ./configure options). This way, there will be not many new packages.

Of course, this is not a good idea for things like FF/Gnome/KDE because of a
slow-down, but a performance penalty for smaller programs like vim, links,
XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled
xfce4 desktop on a single-core Intel Atom based netbook).

Cheers,
--
Leonid Isaev
GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4
C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
Evan Penner
2015-08-13 13:55:11 UTC
Permalink
Post by Leonid Isaev
Post by Evan Penner
Post by Joe Julian
Post by Leonid Isaev
I would personally prefer that most packages come with debugging
enabled
Post by Evan Penner
Post by Joe Julian
Post by Leonid Isaev
by
default. Surely, there will be a performance cost, but speed is not crucial in
most cases.
Cheers,
There's no performance impact, just disk space and bandwidth.
Bandwidth is probably the main problem, although anyone who wants to
debug
Post by Evan Penner
will probably be fine with that.
I think you guys misunderstood me. The biggest problem IMHO with building debug
versions locally is not compiling itself, but setting up the environment. So, I
meant that packages come with debugging enabled (compiled with gcc -O0 -g and
perhaps ./configure options). This way, there will be not many new packages.
Of course, this is not a good idea for things like FF/Gnome/KDE because of a
slow-down, but a performance penalty for smaller programs like vim, links,
XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled
xfce4 desktop on a single-core Intel Atom based netbook).
Right, I got it now. So ideally, you'd want all packages to come with debug
flags enabled? I see how that would cause performance issues in larger
applications, I understand however the concern for those packages. The
problem is performance as you said. However, creating a new repository
would increase the Arch database by twofold. This is a concept the devs
will have to take a look at. For now, I guess, your only option is to
compile them yourself.
Joe Julian
2015-08-13 17:13:02 UTC
Permalink
Post by Evan Penner
Post by Leonid Isaev
Post by Evan Penner
Post by Joe Julian
Post by Leonid Isaev
I would personally prefer that most packages come with debugging
enabled
Post by Evan Penner
Post by Joe Julian
Post by Leonid Isaev
by
default. Surely, there will be a performance cost, but speed is not crucial in
most cases.
Cheers,
There's no performance impact, just disk space and bandwidth.
Bandwidth is probably the main problem, although anyone who wants to
debug
Post by Evan Penner
will probably be fine with that.
I think you guys misunderstood me. The biggest problem IMHO with building debug
versions locally is not compiling itself, but setting up the environment. So, I
meant that packages come with debugging enabled (compiled with gcc -O0 -g and
perhaps ./configure options). This way, there will be not many new packages.
Of course, this is not a good idea for things like FF/Gnome/KDE because of a
slow-down, but a performance penalty for smaller programs like vim, links,
XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled
xfce4 desktop on a single-core Intel Atom based netbook).
Right, I got it now. So ideally, you'd want all packages to come with debug
flags enabled? I see how that would cause performance issues in larger
applications,
False, see my previous reply.
Post by Evan Penner
I understand however the concern for those packages. The
problem is performance as you said. However, creating a new repository
would increase the Arch database by twofold.
False. Plus, the debug symbols *could* be in a separate package database
that would be manually enabled by the people with the skills to know how
to use them. People that actually use this distro professionally to
build some really cool stuff (plugging IO, my employer, here).
Post by Evan Penner
This is a concept the devs
will have to take a look at. For now, I guess, your only option is to
compile them yourself.
Joe Julian
2015-08-13 17:09:00 UTC
Permalink
Post by Leonid Isaev
Post by Evan Penner
Post by Joe Julian
Post by Leonid Isaev
I would personally prefer that most packages come with debugging enabled by
default. Surely, there will be a performance cost, but speed is not crucial in
most cases.
Cheers,
There's no performance impact, just disk space and bandwidth.
Bandwidth is probably the main problem, although anyone who wants to debug
will probably be fine with that.
I think you guys misunderstood me. The biggest problem IMHO with building debug
versions locally is not compiling itself, but setting up the environment. So, I
meant that packages come with debugging enabled (compiled with gcc -O0 -g and
perhaps ./configure options). This way, there will be not many new packages.
No, the biggest problem with building debug versions locally is that it
takes hours of developer time multiplied by every bug found. Why are we
wasting such a limited resource when it's so easy not to? Life is finite.
Post by Leonid Isaev
Of course, this is not a good idea for things like FF/Gnome/KDE because of a
slow-down, but a performance penalty for smaller programs like vim, links,
XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled
xfce4 desktop on a single-core Intel Atom based netbook).
Cheers,
What is this slow-down you keep talking about? I'm not asking to do away
with optimization, just give us a way to get the debug symbols without
rebuilding. The debug symbols are located in totally different sections
from the code/data sections. You can check it with objdump:

$ objdump -h a.out

a.out: file format elf64-x86-64

Sections:
Idx Name Size VMA LMA File
off Algn
0 .interp 0000001c 0000000000400200 0000000000400200
00000200 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
1 .note.ABI-tag 00000020 000000000040021c 000000000040021c
0000021c 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
2 .note.gnu.build-id 00000024 000000000040023c 000000000040023c
0000023c 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
3 .hash 00000018 0000000000400260 0000000000400260
00000260 2**3
CONTENTS, ALLOC, LOAD, READONLY, DATA
4 .gnu.hash 0000001c 0000000000400278 0000000000400278
00000278 2**3
CONTENTS, ALLOC, LOAD, READONLY, DATA
5 .dynsym 00000048 0000000000400298 0000000000400298
00000298 2**3
CONTENTS, ALLOC, LOAD, READONLY, DATA
6 .dynstr 00000038 00000000004002e0 00000000004002e0
000002e0 2**0
CONTENTS, ALLOC, LOAD, READONLY, DATA
7 .gnu.version 00000006 0000000000400318 0000000000400318
00000318 2**1
CONTENTS, ALLOC, LOAD, READONLY, DATA
8 .gnu.version_r 00000020 0000000000400320 0000000000400320
00000320 2**3
CONTENTS, ALLOC, LOAD, READONLY, DATA
9 .rela.dyn 00000018 0000000000400340 0000000000400340
00000340 2**3
CONTENTS, ALLOC, LOAD, READONLY, DATA
10 .rela.plt 00000018 0000000000400358 0000000000400358
00000358 2**3
CONTENTS, ALLOC, LOAD, READONLY, DATA
11 .init 00000018 0000000000400370 0000000000400370
00000370 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
12 .plt 00000020 0000000000400388 0000000000400388
00000388 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
13 .text 000001c8 00000000004003b0 00000000004003b0
000003b0 2**4
CONTENTS, ALLOC, LOAD, READONLY, CODE
14 .fini 0000000e 0000000000400578 0000000000400578
00000578 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
15 .rodata 00000004 0000000000400588 0000000000400588
00000588 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
16 .eh_frame_hdr 00000024 000000000040058c 000000000040058c
0000058c 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
17 .eh_frame 0000007c 00000000004005b0 00000000004005b0
000005b0 2**3
CONTENTS, ALLOC, LOAD, READONLY, DATA
18 .ctors 00000010 0000000000600630 0000000000600630
00000630 2**3
CONTENTS, ALLOC, LOAD, DATA
19 .dtors 00000010 0000000000600640 0000000000600640
00000640 2**3
CONTENTS, ALLOC, LOAD, DATA
20 .jcr 00000008 0000000000600650 0000000000600650
00000650 2**3
CONTENTS, ALLOC, LOAD, DATA
21 .dynamic 000001a0 0000000000600658 0000000000600658
00000658 2**3
CONTENTS, ALLOC, LOAD, DATA
22 .got 00000008 00000000006007f8 00000000006007f8
000007f8 2**3
CONTENTS, ALLOC, LOAD, DATA
23 .got.plt 00000020 0000000000600800 0000000000600800
00000800 2**3
CONTENTS, ALLOC, LOAD, DATA
24 .data 00000010 0000000000600820 0000000000600820
00000820 2**3
CONTENTS, ALLOC, LOAD, DATA
25 .bss 00000010 0000000000600830 0000000000600830
00000830 2**3
ALLOC
26 .comment 00000039 0000000000000000 0000000000000000
00000830 2**0
CONTENTS, READONLY
27 .debug_aranges 00000030 0000000000000000 0000000000000000
00000869 2**0
CONTENTS, READONLY, DEBUGGING
28 .debug_pubnames 0000001b 0000000000000000 0000000000000000
00000899 2**0
CONTENTS, READONLY, DEBUGGING
29 .debug_info 00000055 0000000000000000 0000000000000000
000008b4 2**0
CONTENTS, READONLY, DEBUGGING
30 .debug_abbrev 00000034 0000000000000000 0000000000000000
00000909 2**0
CONTENTS, READONLY, DEBUGGING
31 .debug_line 0000003b 0000000000000000 0000000000000000
0000093d 2**0
CONTENTS, READONLY, DEBUGGING
32 .debug_str 00000026 0000000000000000 0000000000000000
00000978 2**0
CONTENTS, READONLY, DEBUGGING
33 .debug_loc 0000004c 0000000000000000 0000000000000000
0000099e 2**0
CONTENTS, READONLY, DEBUGGING
You can see the extra sections (27 through 33). These sections won't be
loaded at runtime, so there won't be any performance penalty. Using gdb,
you can also examine them at runtime

$ gdb ./a.out
(gdb) break main
(gdb) run
(gdb) info files
// blah blah ....
Local exec file:
`/home/kghost/a.out', file type elf64-x86-64.
Entry point: 0x4003b0
0x0000000000400200 - 0x000000000040021c is .interp
0x000000000040021c - 0x000000000040023c is .note.ABI-tag
0x000000000040023c - 0x0000000000400260 is .note.gnu.build-id
0x0000000000400260 - 0x0000000000400278 is .hash
0x0000000000400278 - 0x0000000000400294 is .gnu.hash
0x0000000000400298 - 0x00000000004002e0 is .dynsym
0x00000000004002e0 - 0x0000000000400318 is .dynstr
0x0000000000400318 - 0x000000000040031e is .gnu.version
0x0000000000400320 - 0x0000000000400340 is .gnu.version_r
0x0000000000400340 - 0x0000000000400358 is .rela.dyn
0x0000000000400358 - 0x0000000000400370 is .rela.plt
0x0000000000400370 - 0x0000000000400388 is .init
0x0000000000400388 - 0x00000000004003a8 is .plt
0x00000000004003b0 - 0x0000000000400578 is .text
0x0000000000400578 - 0x0000000000400586 is .fini
0x0000000000400588 - 0x000000000040058c is .rodata
0x000000000040058c - 0x00000000004005b0 is .eh_frame_hdr
0x00000000004005b0 - 0x000000000040062c is .eh_frame
0x0000000000600630 - 0x0000000000600640 is .ctors
0x0000000000600640 - 0x0000000000600650 is .dtors
0x0000000000600650 - 0x0000000000600658 is .jcr
0x0000000000600658 - 0x00000000006007f8 is .dynamic
0x00000000006007f8 - 0x0000000000600800 is .got
0x0000000000600800 - 0x0000000000600820 is .got.plt
0x0000000000600820 - 0x0000000000600830 is .data
0x0000000000600830 - 0x0000000000600840 is .bss
// blah blah ....
So the only penalty is that you need extra disk space to store this
information.
Leonid Isaev
2015-08-13 17:58:28 UTC
Permalink
Post by Joe Julian
Post by Leonid Isaev
Post by Evan Penner
Bandwidth is probably the main problem, although anyone who wants to debug
will probably be fine with that.
I think you guys misunderstood me. The biggest problem IMHO with building debug
versions locally is not compiling itself, but setting up the environment. So, I
meant that packages come with debugging enabled (compiled with gcc -O0 -g and
perhaps ./configure options). This way, there will be not many new packages.
No, the biggest problem with building debug versions locally is that it
takes hours of developer time multiplied by every bug found. Why are we
wasting such a limited resource when it's so easy not to? Life is finite.
That's just pure theory. In practice, the compilation itself is a minor
inconvenience, unless you are talking about Gnome/KDE. But debugging those is
hopeless anyway :)

As a developer, you'll spend most time understanding the changes (looking at
the code), not compiling.
Post by Joe Julian
Post by Leonid Isaev
Of course, this is not a good idea for things like FF/Gnome/KDE because of a
slow-down, but a performance penalty for smaller programs like vim, links,
XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled
xfce4 desktop on a single-core Intel Atom based netbook).
Cheers,
What is this slow-down you keep talking about? I'm not asking to do away
with optimization, just give us a way to get the debug symbols without
rebuilding. The debug symbols are located in totally different sections from
I was under the impression that with C, -On (n > 0) is not recommended with -g.
Now, I don't know how much -O2 (for example) speeds things up compared to -O1
etc, but probably not much on small applications.

Granted, I think any compiler-level optimization is overrated, and I never
really saw any measurable effect of it. But I use Fortran (not C) for all my
projects.

Cheers,
--
Leonid Isaev
GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4
C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
Joe Julian
2015-08-13 18:09:46 UTC
Permalink
Post by Leonid Isaev
Post by Joe Julian
Post by Leonid Isaev
Post by Evan Penner
Bandwidth is probably the main problem, although anyone who wants to debug
will probably be fine with that.
I think you guys misunderstood me. The biggest problem IMHO with building debug
versions locally is not compiling itself, but setting up the environment. So, I
meant that packages come with debugging enabled (compiled with gcc -O0 -g and
perhaps ./configure options). This way, there will be not many new packages.
No, the biggest problem with building debug versions locally is that it
takes hours of developer time multiplied by every bug found. Why are we
wasting such a limited resource when it's so easy not to? Life is finite.
That's just pure theory. In practice, the compilation itself is a minor
inconvenience, unless you are talking about Gnome/KDE. But debugging those is
hopeless anyway :)
As a developer, you'll spend most time understanding the changes (looking at
the code), not compiling.
I'm not some newb, maybe google me before you make condescending statements.

There is a reason the other distros publish the debug symbols. It's
valuable and there's no valid reason not to.
Post by Leonid Isaev
Post by Joe Julian
Post by Leonid Isaev
Of course, this is not a good idea for things like FF/Gnome/KDE because of a
slow-down, but a performance penalty for smaller programs like vim, links,
XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled
xfce4 desktop on a single-core Intel Atom based netbook).
Cheers,
What is this slow-down you keep talking about? I'm not asking to do away
with optimization, just give us a way to get the debug symbols without
rebuilding. The debug symbols are located in totally different sections from
I was under the impression that with C, -On (n > 0) is not recommended with -g.
Now, I don't know how much -O2 (for example) speeds things up compared to -O1
etc, but probably not much on small applications.
Granted, I think any compiler-level optimization is overrated, and I never
really saw any measurable effect of it. But I use Fortran (not C) for all my
projects.
Cheers,
Yes, if you optimize you will lose some variable data when it's out of
scope. If it's actually about debugging and seeing what's in variables
and stepping through code, that's something that I would likely compile
on my own with O0. Most of the time it's about seeing where the deadlock
or segfault is and getting a meaningful stack trace.

I've seen quite a bit of a performance increase in some cases. GlusterFS
was being built with O0 back in the 2.0 days. Building with O2 proved
about a 30% increase in throughput.
Doug Newgard
2015-08-13 19:37:15 UTC
Permalink
On Thu, 13 Aug 2015 11:09:46 -0700
Post by Joe Julian
There is a reason the other distros publish the debug symbols. It's
valuable and there's no valid reason not to.
AFAIK, right now the reason it can't happen is because nobody has stepped up and
added support to devtools for putting split packages in different repos. Nobody
willing to do the work is a valid reason.

Doug
Joe Julian
2015-08-13 20:08:55 UTC
Permalink
Post by Doug Newgard
On Thu, 13 Aug 2015 11:09:46 -0700
Post by Joe Julian
There is a reason the other distros publish the debug symbols. It's
valuable and there's no valid reason not to.
AFAIK, right now the reason it can't happen is because nobody has stepped up and
added support to devtools for putting split packages in different repos. Nobody
willing to do the work is a valid reason.
Doug
Excellent. I'll tackle that then.
Damian Nowak
2015-08-13 21:06:00 UTC
Permalink
Post by Joe Julian
Post by Doug Newgard
AFAIK, right now the reason it can't happen is because nobody has stepped up and
added support to devtools for putting split packages in different repos. Nobody
willing to do the work is a valid reason.
Excellent. I'll tackle that then.
Good luck! Let us know about a progress. Or maybe you can create an issue in the bug
tracker and track the progress there? Anyways, would be great to have debug symbols
provided at some point. Debug symbols would have saved me a lot of time with reporting
problems to Mesa/Gallium3D bugtracker.
--
Damian Nowak
Florian Pritz
2015-08-15 13:42:24 UTC
Permalink
Post by Joe Julian
Post by Doug Newgard
AFAIK, right now the reason it can't happen is because nobody has stepped up and
added support to devtools for putting split packages in different repos. Nobody
willing to do the work is a valid reason.
Excellent. I'll tackle that then.
dbscripts also needs to be changed and I've started working on this
quite some time ago, but then had other things to do and I haven't
picked it up again since because the code is not pretty and it requires
a bit of time to get up to speed.

If you really, really want to work on this feel free to ping me on IRC
(Bluewind in #archlinux-projects would be best) so we can discuss it
in detail. My current progress can be seen here, though I'm not sure if
all the information there is still accurate:

https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite

Florian

Loading...