[RFC] Go (golang) packaging

[Michael Stapelberg]

Hi,

I am co-maintainer of the golang package and spent a few hours on trying
to figure out how to best create Debian packages for libraries and
programs which are implemented in Go.

I have documented my thoughts, conclusions and example packaging on:
http://wiki.debian.org/MichaelStapelberg/GoPackaging

Essentially, I propose that /usr/lib/gocode is used on Debian to store
the “src” and “pkg” folders which contain the .go files and compiled
versions (respectively) of Go libraries.

Furthermore, the package names should be e.g. “golang-codesearch” for
the library code.google.com/p/codesearch (and of course just
“codesearch” for the binaries).

To make go(1) find libraries when compiling, one normally sets the
environment variable $GOPATH to ~/gocode or similar. I propose to patch
Go in Debian so that it always appends /usr/lib/gocode when reading the
list of paths from $GOPATH.

Any feedback is appreciated. Please read the wiki page before you
comment, it contains more rationale than this email. Thanks.

[Paul Wise]

Post by Michael Stapelberg
Any feedback is appreciated. Please read the wiki page before you
comment, it contains more rationale than this email. Thanks.

Since golang apparently doesn't support dynamic linking, every package
built against a golang library will have to include an appropriate
Built-Using header. You will probably also want a lintian test for
this to autoreject anything without this header.

Upstream should switch to using the GNU triplet instead of linux_arm,
which is clearly not unique enough.

Is it possible to:

Support dynamic linking? That would avoid the whole rebuild-the-world
thing that statically-linked languages have.

Not require the full source code for libraries? C and other libraries
only require the interface headers.

[Michael Stapelberg]

Hi Paul,

Post by Paul Wise
Since golang apparently doesn't support dynamic linking, every package
built against a golang library will have to include an appropriate
Built-Using header. You will probably also want a lintian test for
this to autoreject anything without this header.

Thanks, I was not aware of this yet. Will have a closer look and
integrate it into my Wiki page.

Post by Paul Wise
Upstream should switch to using the GNU triplet instead of linux_arm,
which is clearly not unique enough.

I think so, too. Not sure if they think that’s doable at this point,
will ask them.

Post by Paul Wise
Support dynamic linking? That would avoid the whole rebuild-the-world
thing that statically-linked languages have.

Only when not using the “official” compiler (gc), e.g. gccgo has support
for dynamic linking.

Post by Paul Wise
Not require the full source code for libraries? C and other libraries
only require the interface headers.

AFAIK not, but I will add this to the list of questions I want to ask on
the Go mailing lists.

[Simon McVittie]

Post by Michael Stapelberg
Hi Paul,

Post by Paul Wise
Support dynamic linking? That would avoid the whole rebuild-the-world
thing that statically-linked languages have.

Only when not using the “official” compiler (gc), e.g. gccgo has support
for dynamic linking.

Where does gccgo look for Go sources (src)?

Where does gccgo look for Go "static libraries" (pkg)?

Where does gccgo look for Go dynamic libraries? (Presumably the same
places where gcc looks for C dynamic libraries?)

Post by Michael Stapelberg

Post by Paul Wise
Not require the full source code for libraries? C and other libraries
only require the interface headers.

AFAIK not, but I will add this to the list of questions I want to ask on
the Go mailing lists.

If a significant amount of code is going to be written in Go, it would
be useful to understand the classes of source change that do and don't
constitute an API break (well-understood for C, somewhat well-understood
for C++), and the classes of source change that do and don't constitute
an ABI break for gccgo.

Post by Michael Stapelberg
I propose to patch
Go in Debian so that it always appends /usr/lib/gocode when reading
the list of paths from $GOPATH.

Add /usr/local/lib/[$TUPLE/]gocode as "more important than"
/usr/lib/[$TUPLE/]gocode, perhaps? That's how /usr/local usually works
in Debian.

Post by Michael Stapelberg
Essentially, I currently believe that multi-arch does not make sense
for go, since we are only dealing with static binaries.

For multi-arch runtime libraries (in-scope for wheezy), sure, it's not
useful because there are no runtime libraries.

For multi-arch headers and development libraries (not formally in-scope
for wheezy, but some simpler packages already support it, and it's
likely to be a goal for jessie), it's useful to be multiarch so an
i386/amd64 system can have both golang-libhello:amd64 and
golang-libhello:i386, so that it can build both helloworld:amd64 and
helloworld:i386.

If golang has its own partial solution for multiarch but that solution
is actually insufficient for what we need (linux_arm for both armel and
armhf, etc.) and upstream are unwilling to make it sufficient, then I
don't see anything particularly wrong with
/usr/lib/arm-linux-gnueabi/golang/linux_arm/libhello.a. Sure, it's
redundant, but it works...

[Dmitrijs wrote:]

Post by Michael Stapelberg

Post by Paul Wise
/usr/lib/gocode/ for sources

/usr/[local/]share/gocode if they're arch-indep, surely? Or if they
*can* vary by arch but usually don't (like C headers),
/usr/local/lib/$TUPLE/gocode:/usr/local/share/gocode:/usr/lib/$TUPLE/gocode:/usr/share/gocode
or something?

Post by Michael Stapelberg
Furthermore, the package names should be e.g. “golang-codesearch” for
the library code.google.com/p/codesearch (and of course just
“codesearch” for the binaries).

Should the package with the "headers" and "static library" be
golang-codesearch-dev or something, in the hope that when Go grows the
ability to build dynamic libraries, the dynamic libraries can be called
golang-codesearch?

Post by Michael Stapelberg
For cross-compiling, it doesn’t matter which pre-compiled packages
you have installed (e.g. linux_amd64) because the .go source files

How "cheap" is recompiling everything? If it's sufficiently cheap, you
could just ship complete source and ignore the "static libraries"
entirely; or if it isn't, you'll benefit from proper multiarch.

S

[Michael Stapelberg]

Hi Simon,

Post by Simon McVittie
Where does gccgo look for Go sources (src)?
Where does gccgo look for Go "static libraries" (pkg)?
Where does gccgo look for Go dynamic libraries? (Presumably the same
places where gcc looks for C dynamic libraries?)

I can’t really answer these questions in depth, but the way I usually
invoke it is by using go(1)’s -compiler option, in which case the lookup
works the same as with the “original” go compiler.

I’m not sure whether Debian should support gccgo at all. Will ask the Go
mailing lists on what they think.

Post by Simon McVittie

Post by Michael Stapelberg
I propose to patch
Go in Debian so that it always appends /usr/lib/gocode when reading
the list of paths from $GOPATH.

Add /usr/local/lib/[$TUPLE/]gocode as "more important than"
/usr/lib/[$TUPLE/]gocode, perhaps? That's how /usr/local usually works
in Debian.

Mentioning /usr/local is a good point, but I’m not sure whether it’s
really a good idea: in the Go community, setting $GOPATH to a directory
within your home directory (e.g. ~/gocode) is normal and what everyone
uses. To install a third-party library, you just “go get” its URL, e.g.

go get code.google.com/p/codesearch

Which will then download the library to
~/gocode/src/code.google.com/p/codesearch.

Nobody will have libraries in /usr/local, because there is no facility
(no Makefiles) to install them there. The local directory is rather
specified by the user in $GOPATH — that is the idiomatic way.

Post by Simon McVittie
For multi-arch headers and development libraries (not formally in-scope
for wheezy, but some simpler packages already support it, and it's
likely to be a goal for jessie), it's useful to be multiarch so an
i386/amd64 system can have both golang-libhello:amd64 and
golang-libhello:i386, so that it can build both helloworld:amd64 and
helloworld:i386.

You cannot currently cross-compile an i386 binary on an amd64 host with
the Go packages we have in Debian. I consider this a separate issue.
Essentially, when we have a golang-go package that contains a version of
the compiler which has been built to support cross-compiling for a
particular architecture (say golang-go-i386), you can cross-compile
using the source code of the libraries, a pre-compiled version is not
necessary.

To state this clearly: The benefit of shipping the pkg/ files in Debian
packages at all is to avoid re-compilation every time the user builds
code that depends on the library. We could just as well leave out the
pkg/ files and everything would still work — just slower.

Post by Simon McVittie
Should the package with the "headers" and "static library" be
golang-codesearch-dev or something, in the hope that when Go grows the
ability to build dynamic libraries, the dynamic libraries can be called
golang-codesearch?

Yeah, I suppose we should name them -dev. AFAICT, there is some movement
into the shared library direction, see
https://groups.google.com/d/msg/golang-nuts/VS-Pt38097Q/MXMRFc7G26IJ
https://codereview.appspot.com/6926049/

Post by Simon McVittie
How "cheap" is recompiling everything? If it's sufficiently cheap, you
could just ship complete source and ignore the "static libraries"
entirely; or if it isn't, you'll benefit from proper multiarch.

See above. We could ship without the static libraries, but having them
speeds up compilation noticably in big projects.

[Paul Wise]

Post by Michael Stapelberg
Only when not using the “official” compiler (gc), e.g. gccgo has support
for dynamic linking.

Then we should use gccgo until the official compiler supports this.

Post by Michael Stapelberg
AFAIK not, but I will add this to the list of questions I want to ask on
the Go mailing lists.

Ok, please let us know about the results.

[Shawn]

Post by Paul Wise

Only when not using the “official” compiler (gc), e.g. gccgo has support
for dynamic linking.

Then we should use gccgo until the official compiler supports this.

go also doesn't support building as a shared library, because
#1 the go scheduler would have to have differn't bootstrapping code hooked
up (for gccgo it would be in libgo) and
#2 go uses split stacks[1].
But when it doesn't support building as a shared library, it will probably
be done with gccgo first.

Post by Paul Wise

AFAIK not, but I will add this to the list of questions I want to ask on
the Go mailing lists.

Ok, please let us know about the results.

Henceforth when a go program depends on a go library, those go libraries
are ALWAYS compiled in statically. Static linking causes many problems for
distributions like Debian, and therefore this presents a significant hurdle
to quality packaging of go in Debian. Source-only distribution is perhaps
possible, but then AFAIK go library writers are not necessarily as careful
as C library writers to maintain source compatibility.

[1] http://gcc.gnu.org/wiki/SplitStacks

Post by Paul Wise
-Shawn Landden

[Michael Stapelberg]

Hi Shawn,

Post by Shawn
Henceforth when a go program depends on a go library, those go
libraries are ALWAYS compiled in statically. Static linking causes
many problems for distributions like Debian, and therefore this

Can you please tell us which specific problems are caused by static
linking for Debian? I’m not aware of any.

I’m somewhat hesitant to say “Let’s just use gccgo in Debian” because
static linking is one of the advantages of Go (IMO, but also a shared
feeling by the community I think) and the gc compiler is fast, which is
another advertised advantage.

[Vincent Bernat]

Post by Michael Stapelberg

Post by Shawn
Henceforth when a go program depends on a go library, those go
libraries are ALWAYS compiled in statically. Static linking causes
many problems for distributions like Debian, and therefore this

Can you please tell us which specific problems are caused by static
linking for Debian? I’m not aware of any.

Fixin a bug (including a security bug) in a package means recompiling
all packages depending on this package.

[Guillem Jover]

Post by Michael Stapelberg

Post by Shawn
Henceforth when a go program depends on a go library, those go
libraries are ALWAYS compiled in statically. Static linking causes
many problems for distributions like Debian, and therefore this

Can you please tell us which specific problems are caused by static
linking for Debian? I’m not aware of any.

Well, that's surprising, I would have thought this has been discussed
at lenght in the past, in any case here are several reasons (I'm most
probably forgetting others):

* Changes in libraries might require recompiling all of their rdeps.
This includes:
- Bug fixes (security or otherwise, as pointed out in the thread).
- Performance improvements.
- Private dependencies, as they leak to rdeps. When a library uses
another library privately this dependency gets linked in directly
in all other rdeps, when that library stop depending on that
private dependency, all rdeps need to be rebuilt.
* We might need to keep the sources for all library instances that
have been linked into any other package, to comply with license
conditions (see the Built-Using comment on this thread too).
* Larger dependency chains. Because private libraries need to be
pulled in even by indirect dependencies, the sources and static
libraries need to be available at compile time, this means the
libfoo-dev dependencies and Build-Depends chains are going to
be larger.
* Produces huge binaries (and packages), which use more memory that
cannot be shared across processes (unlike shared libraries).

The first point alone, make such things a PITA to manage in a
distribution, and are a waste of buildd and people's time who need
to coordinate those things. The rest imply a waste of resources or
added complexity which could be avoided.

Post by Michael Stapelberg
I’m somewhat hesitant to say “Let’s just use gccgo in Debian” because
static linking is one of the advantages of Go (IMO, but also a shared
feeling by the community I think) and the gc compiler is fast, which is
another advertised advantage.

While it's true that static linking can produce varying degrees of faster
binaries depending on the conditions (architecture, amount of libraries,
etc), I don't think it's worth the tradeoff for the general case and I'm
having a hard time seeing a new compiled system language that does not
support shared libraries as a serious contender.

Thanks,
Guillem

[Michael Stapelberg]

Hi Guillem,

Thanks for your explanations, most points make sense to me. Two

Post by Guillem Jover
- Private dependencies, as they leak to rdeps. When a library uses
another library privately this dependency gets linked in directly
in all other rdeps, when that library stop depending on that
private dependency, all rdeps need to be rebuilt.

I’m not entirely sure I get your point. Why exactly do the rdeps need to
be rebuilt? Could you maybe provide an example to clarify?

Post by Guillem Jover
* We might need to keep the sources for all library instances that
have been linked into any other package, to comply with license
conditions (see the Built-Using comment on this thread too).

How is this any different from a “normal” dependency with a shared
library? Also, what license conditions do you mean specifically?

[Sune Vuorela]

Post by Guillem Jover
* We might need to keep the sources for all library instances that
have been linked into any other package, to comply with license
conditions (see the Built-Using comment on this thread too).

How is this any different from a ???normal??? dependency with a shared
library? Also, what license conditions do you mean specifically?

A 'normal' dependency with a shared library does not actually include
the actual code from the shared library in the application, only the
interface specification.

A static linking does include the actual code in the application.

Given the library is actually part of the distributed binary, we also
need to provide this code to comply with a) a copyleft license and b)
the dfsg.

/Sune

[Wouter Verhelst]

Post by Guillem Jover
- Private dependencies, as they leak to rdeps. When a library uses
another library privately this dependency gets linked in directly
in all other rdeps, when that library stop depending on that
private dependency, all rdeps need to be rebuilt.

Strictly speaking, if you're only using static libraries this is not
really true; once you've compiled something against a static library,
the static library might change in whatever way it sees fit, the
compiled binary will continue to work, with or without recompilation.

This isn't true if you're using a mix of shared and static libraries, of
course.

[...]

Post by Guillem Jover
* Produces huge binaries (and packages), which use more memory that
cannot be shared across processes (unlike shared libraries).

Also, if a significant body of code exists that uses a particular
library, this will have a significant influence on mirror archive size,
and installed size of Debian systems.

[...]

[Reinhard Tartler]

Post by Wouter Verhelst

Post by Guillem Jover
- Private dependencies, as they leak to rdeps. When a library uses
another library privately this dependency gets linked in directly
in all other rdeps, when that library stop depending on that
private dependency, all rdeps need to be rebuilt.

Strictly speaking, if you're only using static libraries this is not
really true; once you've compiled something against a static library,
the static library might change in whatever way it sees fit, the
compiled binary will continue to work, with or without recompilation.

Consider this from the application perspective: Say an application
links against a library libfoo.a. At some point, libfoo decides to
include compression support, and requires functionality from libz. No
problem for the library package maintainer; he just adds a
build-dependency on libz-dev, and uploads the package. At some point
the security team needs to update the application and finds the
package to FTBFS because libz is missing. The solution, of course, is
now to extend the build-dependencies of the application package.
However, this is not obvious and in any case more effort than a
binNMU.

Post by Wouter Verhelst
This isn't true if you're using a mix of shared and static libraries, of
course.

mixing shared and static libraries makes the situation no less
complicated, that's true.

[Michael Stapelberg]

Hi Reinhard,

Post by Reinhard Tartler
Consider this from the application perspective: Say an application
links against a library libfoo.a. At some point, libfoo decides to
include compression support, and requires functionality from libz. No
problem for the library package maintainer; he just adds a
build-dependency on libz-dev, and uploads the package. At some point
the security team needs to update the application and finds the
package to FTBFS because libz is missing. The solution, of course, is
now to extend the build-dependencies of the application package.
However, this is not obvious and in any case more effort than a
binNMU.

Thanks for your example. It occurs to me that the following way deals
with the problem:

The hypothetical “foo” package is packaged as “golang-foo-dev”.
The hypothetical application is packaged as “barapp”, which
Build-Depends on “golang-foo-dev”.

When golang-foo-dev starts depending on additional libraries, say
“golang-zlib-dev”, it not only adds that library to its Build-Depends,
but also to its Depends. This is okay because you cannot use
“golang-foo-dev” without having “golang-zlib-dev” installed (as you
explained), and normal users don’t get an unnecessary dependency because
“barapp” doesn’t Depend (only Build-Depends) on either of these
libraries anyway.

I hope my explanation was clear, the tl;dr is “static golang libraries
should mirror Build-Depends into Depends”. What do you think?

[Alastair McKinstry]

Post by Reinhard Tartler

Post by Wouter Verhelst

Post by Guillem Jover
- Private dependencies, as they leak to rdeps. When a library uses
another library privately this dependency gets linked in directly
in all other rdeps, when that library stop depending on that
private dependency, all rdeps need to be rebuilt.

Strictly speaking, if you're only using static libraries this is not
really true; once you've compiled something against a static library,
the static library might change in whatever way it sees fit, the
compiled binary will continue to work, with or without recompilation.

Consider this from the application perspective: Say an application
links against a library libfoo.a. At some point, libfoo decides to
include compression support, and requires functionality from libz. No
problem for the library package maintainer; he just adds a
build-dependency on libz-dev, and uploads the package. At some point
the security team needs to update the application and finds the
package to FTBFS because libz is missing. The solution, of course, is
now to extend the build-dependencies of the application package.
However, this is not obvious and in any case more effort than a
binNMU.

Yes, there are compile-time dependencies for any static library. We do
need to track
these. In practice we already have a mechanism in pkg-config, but this
is (I believe)
not properly formalised in Debian.

In the case you mention, if libfoo now depends on libz, adding a build
dependency
on libz-dev fixes the problem with libfoo.so as it will automatically
pull in libz.so
However, the packager should _also_ provide a pkg-config file and this
will have
a list of the dependencies , so
LIBS:=` pkg-config --static -libs foo`
does the right thing, and the updated libfoo-dev package will include
-lz on the libs line.

I think we should do the following:

(1) pkg-config files for libraries, in particular all those that ship
static libs, to be a
release goal for jessie.

It would be useful / interesting if pkg-config information could be used
to generate dependencies
for the libfoo-dev package: ie. if libz is _really_ needed for libfoo,
then libfoo-dev should depend
on libz-dev, to provide libz.a. But this is harder than it looks.

Consider the case of netcdf. This ships a libnetcdf.a static library,
and this may be used in scientific
codes for optimisation as well as "reliability". But in modern terms,
libnetcdf, which reads netcdf files, may also accept a URL as well as a
filename. In this case, it uses libcurl to retrieve the file, but pulls
in a huge trace of packages, from gnutls, ssl , pam packages, psql, msql
, etc in order to
get authentication. In practice this means I might build my scientific
binary with
libnetcdf.a static and dynamically link libcurl.so (which may never be
called in my
actual workloads) to avoid pulling in the whole distro.

So recursively pulling in all potentially needed static libs is not an
option, but examining all
binaries to find what static libs were used to generate it might be
problematic.

Post by Reinhard Tartler

Post by Wouter Verhelst
This isn't true if you're using a mix of shared and static libraries, of
course.

mixing shared and static libraries makes the situation no less
complicated, that's true.

regards
Alastair

[Sune Vuorela]

Post by Alastair McKinstry
(1) pkg-config files for libraries, in particular all those that ship
static libs, to be a
release goal for jessie.

rather get rid of static libs.

Post by Alastair McKinstry
It would be useful / interesting if pkg-config information could be used
to generate dependencies

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=534966

/Sune

[Florian Weimer]

Post by Sune Vuorela

Post by Alastair McKinstry
(1) pkg-config files for libraries, in particular all those that ship
static libs, to be a
release goal for jessie.

rather get rid of static libs.

We might want to extend static libraries with LTO data one day.
(We could even ship performance-criticial programs ready for LTO.)
Of course, this exacerbates the rebuild issue.

[Reinhard Tartler]

On Thu, Jan 3, 2013 at 12:17 PM, Alastair McKinstry

Post by Alastair McKinstry

Post by Reinhard Tartler

Post by Wouter Verhelst

Post by Guillem Jover
- Private dependencies, as they leak to rdeps. When a library uses
another library privately this dependency gets linked in directly
in all other rdeps, when that library stop depending on that
private dependency, all rdeps need to be rebuilt.

Strictly speaking, if you're only using static libraries this is not
really true; once you've compiled something against a static library,
the static library might change in whatever way it sees fit, the
compiled binary will continue to work, with or without recompilation.

Consider this from the application perspective: Say an application
links against a library libfoo.a. At some point, libfoo decides to
include compression support, and requires functionality from libz. No
problem for the library package maintainer; he just adds a
build-dependency on libz-dev, and uploads the package. At some point
the security team needs to update the application and finds the
package to FTBFS because libz is missing. The solution, of course, is
now to extend the build-dependencies of the application package.
However, this is not obvious and in any case more effort than a
binNMU.

Yes, there are compile-time dependencies for any static library. We do
need to track
these. In practice we already have a mechanism in pkg-config, but this
is (I believe)
not properly formalised in Debian.

and generally pretty broken: see e.g. #622931

IIRC, the pkg-config maintainer dislikes static linking and the
situation is that many if not most .pc files in Debian do not fully
declare all dependencies that would be required for static linking.

Post by Alastair McKinstry
In the case you mention, if libfoo now depends on libz, adding a build
dependency
on libz-dev fixes the problem with libfoo.so as it will automatically
pull in libz.so

Yeah, which does not really scale IMO. And doesn't solve the issue
that the application still needs to track the exact version of all
libraries that were used for linking, e.g. using the Built-Using
header.

Post by Alastair McKinstry
However, the packager should _also_ provide a pkg-config file and this
will have
a list of the dependencies , so
LIBS:=` pkg-config --static -libs foo`
does the right thing, and the updated libfoo-dev package will include
-lz on the libs line.
(1) pkg-config files for libraries, in particular all those that ship
static libs, to be a
release goal for jessie.

I would vote against this. It is really not worth the trouble.

[Wouter Verhelst]

Post by Reinhard Tartler

Post by Wouter Verhelst

Post by Guillem Jover
- Private dependencies, as they leak to rdeps. When a library uses
another library privately this dependency gets linked in directly
in all other rdeps, when that library stop depending on that
private dependency, all rdeps need to be rebuilt.

Strictly speaking, if you're only using static libraries this is not
really true; once you've compiled something against a static library,
the static library might change in whatever way it sees fit, the
compiled binary will continue to work, with or without recompilation.

Consider this from the application perspective: Say an application
links against a library libfoo.a. At some point, libfoo decides to
include compression support, and requires functionality from libz.

Ah, yes, but you're talking about /adding/ a dependency. Guillem's
example was about removing one. There's a difference.

[Florian Weimer]

Post by Wouter Verhelst
Strictly speaking, if you're only using static libraries this is not
really true; once you've compiled something against a static library,
the static library might change in whatever way it sees fit, the
compiled binary will continue to work, with or without recompilation.

My main worry is that, for example, a fix in another, otherwise
unrelated dependency prompts a rebuild, and this picks up behavioral
changes which haven't been visible before, but lingering in the static
library. Essentially, we end up with non-reproducible builds.

The way we currently do QA, it is important that we do not release
many packages with statically linked copies of outdated libraries. We
wouldn't have much chance spotting the impact of such lingering
changes once they are materializing. Non-reproducible builds are also
a software freedom issue (practically speaking).

[Michael Stapelberg]

Hi Florian,

Post by Florian Weimer
My main worry is that, for example, a fix in another, otherwise
unrelated dependency prompts a rebuild, and this picks up behavioral
changes which haven't been visible before, but lingering in the static
library. Essentially, we end up with non-reproducible builds.

Could you provide an example please? I don’t understand how this is
different with static linking than with dynamic linking yet.

[Florian Weimer]

Post by Michael Stapelberg

Post by Florian Weimer
My main worry is that, for example, a fix in another, otherwise
unrelated dependency prompts a rebuild, and this picks up behavioral
changes which haven't been visible before, but lingering in the static
library. Essentially, we end up with non-reproducible builds.

Could you provide an example please? I don’t understand how this is
different with static linking than with dynamic linking yet.

With dynamic linking, you pick up the behavior change along with
"apt-get upgrade", so I expect that we get much more testing of the
combination during development.

Technically, you are correct that we generally do not do purely static
linking—preprocessor macros can change, and so can the value of enum
constants. And with C++, there are many, many more language features
which escape dynamic linking. But I think the lingering change
phenomenon is still a valid concern with static linking.

(One approach which doesn't seem to have been discussed so far is
compile-on-installation, à la common-lisp-controller. Not sure if
this is a good idea. Linking-on-installation does not appear to be
sufficient.)

[Michael Stapelberg]

Hi Florian,

Post by Florian Weimer

Post by Michael Stapelberg
Could you provide an example please? I don’t understand how this is
different with static linking than with dynamic linking yet.

With dynamic linking, you pick up the behavior change along with
"apt-get upgrade", so I expect that we get much more testing of the
combination during development.

I’m still not sure I understand you. Could you please provide an
example? Participating in this discussion makes no sense for me until I
understand your point.

[Florian Weimer]

Post by Michael Stapelberg
Hi Florian,

Post by Florian Weimer

Post by Michael Stapelberg
Could you provide an example please? I don’t understand how this is
different with static linking than with dynamic linking yet.

With dynamic linking, you pick up the behavior change along with
"apt-get upgrade", so I expect that we get much more testing of the
combination during development.

I’m still not sure I understand you. Could you please provide an
example? Participating in this discussion makes no sense for me until I
understand your point.

Okay. Suppose you statically link into your application a copy of
OpenSSL and use it to make TLS connections, in a way that causes
OpenSSL to pick the highest supported TLS version. We upgrade OpenSSL
and add TLSv1.2 support to it (or we remove RC4 support). It turns
out that the application connects to a server which is broken and
refuses to handshake at version 1.2. This means that if we recompile
the application, linking against the current OpenSSL version and
incidentally enabling TLSv1.2 support, it ceases to work.

With dynamic linking, we'd see this immediately, as soon as OpenSSL
upgraded (say, because it has entered testing). With static linking,
we might never see it at all because a recompilation is necessary to
expose the non-working combination.

One might argue that the static case is actually better because it is
more predictable, but our post-release support model is heavily
dependent on minimal changes (because we cannot do full QA
post-release). Such minimal changes are impossible with static
linking, unless we aggressively recompile and relink reverse
dependencies as soon as possible, so that we have no lingering
outdated copies in a release.

I hope this clarifies things a bit.

[Paul Wise]

Post by Florian Weimer
One might argue that the static case is actually better because it is
more predictable, but our post-release support model is heavily
dependent on minimal changes (because we cannot do full QA
post-release). Such minimal changes are impossible with static
linking, unless we aggressively recompile and relink reverse
dependencies as soon as possible, so that we have no lingering
outdated copies in a release.
I hope this clarifies things a bit.

With Built-Using, we get a way to rebuild packages that embed parts of
other packages:

http://www.debian.org/doc/debian-policy/ch-relationships.html#s-built-using

Not sure if the buildd stuff will automatically schedule rebuilds or
if we will notice due to britney knowing about b-u and blocking
testing migrations, anyone got more details about b-u?

[Ansgar Burchardt]

Post by Paul Wise
With Built-Using, we get a way to rebuild packages that embed parts of
http://www.debian.org/doc/debian-policy/ch-relationships.html#s-built-using
Not sure if the buildd stuff will automatically schedule rebuilds or
if we will notice due to britney knowing about b-u and blocking
testing migrations, anyone got more details about b-u?

Built-Using only instructs dak to keep additional source packages in the
archive. This is done to comply with licenses that require the full
source to be available and with the DFSG requirement to provide source
when, for example, linking static libraries, using *-source packages
(e.g. gcc-avr uses the source from gcc-*-source) or embedding binaries
from other packages (e.g. debian-installer).

Nothing else uses it so far, but using it to make sure the references
source packages are not too outdated seems useful. Otherwise any upload
in stable might introduce significant changes (for example gcc-avr might
use a new upstream release of gcc-*-source).

Ansgar

[Florian Weimer]

Post by Paul Wise

Post by Michael Stapelberg
Only when not using the “official” compiler (gc), e.g. gccgo has support
for dynamic linking.

Then we should use gccgo until the official compiler supports this.

gccgo supports dynamic linking, but Go 1 API changes may break the
ABI. Struct sizes and and field offsets are compiled into reverse
dependencies, but the Go 1 API promise does not prevent changes in
this area.

There are ways to solve this (sizes and offsets have to be provided by
the dynamic linker), but there is a performance impact. I think this
has been done for GCJ.

[Matthias Klose]

Post by Paul Wise

Post by Michael Stapelberg
Only when not using the “official” compiler (gc), e.g. gccgo has support
for dynamic linking.

Then we should use gccgo until the official compiler supports this.

Post by Michael Stapelberg
AFAIK not, but I will add this to the list of questions I want to ask on
the Go mailing lists.

Ok, please let us know about the results.

Calling gc the "official" compiler seems to be misleading. gccgo in wheezy
supports the Go API 1.0, and the standard library 1.0.3, supports dynamic
linking, supports multiarch, makes the distinction for armel/armhf, and supports
cross-compilation. I don't mind having a second compiler in the archives like
clang, but please make sure that you are not bound to one. How does your
packaging proposal work with gccgo as the compiler? Did you consider making gc
an alternative (like cc, c++)?

Matthias

[Jakub Wilk]

Post by Matthias Klose
I don't mind having a second compiler in the archives

How generous of you!

[Michael Stapelberg]

Hi Matthias,

Post by Matthias Klose
Calling gc the "official" compiler seems to be misleading. gccgo in

That’s why I called it “official”, not official.

Post by Matthias Klose
wheezy supports the Go API 1.0, and the standard library 1.0.3,
supports dynamic linking, supports multiarch, makes the distinction
for armel/armhf, and supports cross-compilation. I don't mind having
a second compiler in the archives like clang, but please make sure
that you are not bound to one. How does your packaging proposal work
with gccgo as the compiler? Did you consider making gc an alternative
(like cc, c++)?

I am trying to evaluate how this would work. Here is what I tried:

mkdir -p /tmp/golang
export GOPATH=/tmp/golang
go get github.com/mstap/godebiancontrol
go install -v -compiler=gccgo github.com/mstap/godebiancontrol
ls -hlR /tmp/golang/pkg/

This results in the following file structure:

/tmp/golang/pkg/:
total 0
drwxr-xr-x 3 michael staff 60 2013-01-02 15:59 gccgo
drwxr-xr-x 3 michael staff 60 2013-01-02 15:59 linux_amd64

/tmp/golang/pkg/gccgo:
total 0
drwxr-xr-x 3 michael staff 60 2013-01-02 15:59 github.com

/tmp/golang/pkg/gccgo/github.com:
total 0
drwxr-xr-x 2 michael staff 60 2013-01-02 15:59 mstap

/tmp/golang/pkg/gccgo/github.com/mstap:
total 24K
-rw-r--r-- 1 michael staff 23K 2013-01-02 15:59 libgodebiancontrol.a

/tmp/golang/pkg/linux_amd64:
total 0
drwxr-xr-x 3 michael staff 60 2013-01-02 15:59 github.com

/tmp/golang/pkg/linux_amd64/github.com:
total 0
drwxr-xr-x 2 michael staff 60 2013-01-02 15:59 mstap

/tmp/golang/pkg/linux_amd64/github.com/mstap:
total 36K
-rw-r--r-- 1 michael staff 34K 2013-01-02 15:59 godebiancontrol.a

Now, obviously this is not dynamically linked. Neither does it use
different host triplets.

How do I make gccgo use dynamic linking?

[s***@gmail.com]

Post by Michael Stapelberg
Hi Matthias,

Post by Matthias Klose
Calling gc the "official" compiler seems to be misleading. gccgo in

That’s why I called it “official”, not official.

Post by Matthias Klose
wheezy supports the Go API 1.0, and the standard library 1.0.3,
supports dynamic linking, supports multiarch, makes the distinction
for armel/armhf, and supports cross-compilation. I don't mind having
a second compiler in the archives like clang, but please make sure
that you are not bound to one. How does your packaging proposal work
with gccgo as the compiler? Did you consider making gc an alternative
(like cc, c++)?

total 36K
-rw-r--r-- 1 michael staff 34K 2013-01-02 15:59 godebiancontrol.a
Now, obviously this is not dynamically linked. Neither does it use
different host triplets.
How do I make gccgo use dynamic linking?

I am not sure how or if lld works on .a files, but I am pretty sure that this binary is linked against libc6 and libgo1. If I invoke gccgo directly that is the result.

[Michael Stapelberg]

Hi Shawn,

Post by s***@gmail.com
I am not sure how or if lld works on .a files, but I am pretty sure
that this binary is linked against libc6 and libgo1. If I invoke gccgo
directly that is the result.

If by “this binary” you mean godebiancontrol.a, then no:

ldd /tmp/golang/pkg/linux_amd64/github.com/mstap/godebiancontrol.a
not a dynamic executable

[s***@gmail.com]

Post by Michael Stapelberg
Hi Shawn,

Post by s***@gmail.com
I am not sure how or if lld works on .a files, but I am pretty sure
that this binary is linked against libc6 and libgo1. If I invoke

gccgo

Post by s***@gmail.com
directly that is the result.

ldd /tmp/golang/pkg/linux_amd64/github.com/mstap/godebiancontrol.a
not a dynamic executable

.a files are just archives ( ar(1) i believe) of a bunch of .o files. They are not linked at all, either dynamically or statically. As gccgo uses libc6 and libgo1 and does not compile those every time you use it, you need libc6-dev and go Dev files to make any use of these .a files. You have to look at the resulting binaries.

[Michael Stapelberg]

Hi Shawn,

Post by s***@gmail.com
[...]
use of these .a files. You have to look at the resulting binaries.

But in this discussion we are talking about building _library_ packages,
not binaries. I would like to focus on the question of how to build a Go
package (such as github.com/mstap/godebiancontrol) as a shared library,
so that it can later be used in a dynamically linked binary.

[Dmitrijs Ledkovs]

On 1 January 2013 15:44, Michael Stapelberg

Post by Michael Stapelberg
Hi,
I am co-maintainer of the golang package and spent a few hours on trying
to figure out how to best create Debian packages for libraries and
programs which are implemented in Go.
http://wiki.debian.org/MichaelStapelberg/GoPackaging
Essentially, I propose that /usr/lib/gocode is used on Debian to store
the “src” and “pkg” folders which contain the .go files and compiled
versions (respectively) of Go libraries.

What about multiarch?

/usr/lib/gocode/ for sources
/usr/lib/$triplet/gocode/ for compiled versions

This assumes that sources are identical across all architectures.

Regards,

Dmitrijs.

[Michael Stapelberg]

Hi Dmitrijs,

Post by Dmitrijs Ledkovs
What about multiarch?

I tried to address this on the wiki page, see
http://wiki.debian.org/MichaelStapelberg/GoPackaging#Multi-Arch.2Fcross-compiling

Essentially, I currently believe that multi-arch does not make sense for
go, since we are only dealing with static binaries. E.g. to run an i386
program, you don’t need any additional libraries:

$ file helloworld
helloworld: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped
$ uname -a
Linux midna 3.5.0 #1 SMP Sun Jul 22 11:15:13 CEST 2012 x86_64 GNU/Linux
$ ./helloworld
hello

Post by Dmitrijs Ledkovs
/usr/lib/gocode/ for sources
/usr/lib/$triplet/gocode/ for compiled versions

Go’s directory layout already includes the architecture, see
http://wiki.debian.org/MichaelStapelberg/GoPackaging#Resulting_package_file_lists

I have a feeling that you have not actually read the wiki page :-).

[Dmitrijs Ledkovs]

Post by Michael Stapelberg
Hi Dmitrijs,

Post by Dmitrijs Ledkovs
What about multiarch?

I tried to address this on the wiki page, see
http://wiki.debian.org/MichaelStapelberg/GoPackaging#Multi-Arch.2Fcross-compiling

I was more concerned about future-proof, e.g. in debian we have
distinct armel & armhf and currently with multiarch I can
cross-compile binaries for either target from my amd64 build-machine.
In debian we have far more ports than any other distros. It would be
nice if go could use gnu and/or debian triplets instead of one more
incomplete set for the same thing.

Post by Michael Stapelberg
Essentially, I currently believe that multi-arch does not make sense for
go, since we are only dealing with static binaries. E.g. to run an i386

I understand that users running the resulting binaries typically do
not need anything.
My interest in multi-arch support is for fast cross-compilation, i.e.
the same reason pre-compiled libraries are shipped for "native"
compilation.
I could use chroots, but having all those packages as "M-A: same"
would be lovely... which brings the argument back into the circle that
upstream doesn't define unique per-arch paths (at least not for arm*)

reference: https://wiki.ubuntu.com/MultiarchCross

Regards,

Dmitrijs.

[Bernhard R. Link]

Post by Michael Stapelberg
Furthermore, the package names should be e.g. “golang-codesearch” for
the library code.google.com/p/codesearch

Please also consider "codesearch-golang". Especially with longer package
names not everything can show the full name so the beginning should be
the more important information.

Post by Michael Stapelberg
(and of course just “codesearch” for the binaries).

I assume s/binaries/sources/? And I'd suggest to just not policy the
source package names at all (as there really is no "one rule who fits
them all" and there is no reason to have a rule there at all).

Another question: Have you considered asking for a archive Section for
those packages? I guess with no special section yet all those packages
would be section libdevel as they are for static linking only, wouldn't
they?

Bernhard R. Link

[Paul Wise]

Post by Bernhard R. Link
Another question: Have you considered asking for a archive Section for
those packages? I guess with no special section yet all those packages
would be section libdevel as they are for static linking only, wouldn't
they?

http://wiki.debian.org/MichaelStapelberg/GoPackaging#TODO

[Michael Stapelberg]

Hi Bernhard,

Post by Bernhard R. Link
Please also consider "codesearch-golang". Especially with longer package
names not everything can show the full name so the beginning should be
the more important information.

Given that we already have python-* and ruby-*, I’d find golang-* more
consistent. Which specific tools/places do you have in mind that have to
truncate package names?

Post by Bernhard R. Link

Post by Michael Stapelberg
(and of course just “codesearch” for the binaries).

I assume s/binaries/sources/? And I'd suggest to just not policy the

No, I really meant binaries, as in “cgrep”, “cindex” and “csearch” in
this specific case.

Post by Bernhard R. Link
Another question: Have you considered asking for a archive Section for
those packages? I guess with no special section yet all those packages
would be section libdevel as they are for static linking only, wouldn't
they?

As pabs has pointed out, I did that, but the general rule of thumb is
that we want to have lots of packages first and an archive section
second.

[Bernhard R. Link]

Post by Michael Stapelberg
Given that we already have python-* and ruby-*, I’d find golang-* more
consistent.

We also have lib*-perl.
Ruby seems to have changed recently from lib*-ruby to ruby-*. (Does
anyone know of the reason that changed? For I only remember all the
reasons for the old naming scheme and see no reasons in favor of the new)

Post by Michael Stapelberg
Which specific tools/places do you have in mind that have to
truncate package names?

Most GUIs will truncate depending on the size of the window.

Post by Michael Stapelberg

Post by Bernhard R. Link

Post by Michael Stapelberg
(and of course just “codesearch” for the binaries).

I assume s/binaries/sources/? And I'd suggest to just not policy the

No, I really meant binaries, as in “cgrep”, “cindex” and “csearch” in
this specific case.

And what is the name of the binary package those programs will be in?

Post by Michael Stapelberg

Post by Bernhard R. Link
Another question: Have you considered asking for a archive Section for
those packages? I guess with no special section yet all those packages
would be section libdevel as they are for static linking only, wouldn't
they?

As pabs has pointed out, I did that, but the general rule of thumb is
that we want to have lots of packages first and an archive section
second.

Too avoid packaging mistakes and to have the wording ready I'd suggest
to already prepare the wording (I guess in the section later will be
everything that is is now in libdevel plus golong toolchain packages,
while everything providing programs to be in their respective section).

Bernhard R. Link

[Michael Stapelberg]

Hi Bernhard,

Post by Bernhard R. Link

Post by Michael Stapelberg

Post by Bernhard R. Link

Post by Michael Stapelberg
(and of course just “codesearch” for the binaries).

I assume s/binaries/sources/? And I'd suggest to just not policy the

No, I really meant binaries, as in “cgrep”, “cindex” and “csearch” in
this specific case.

And what is the name of the binary package those programs will be in?

“codesearch”, as I wrote.

Post by Bernhard R. Link

Post by Michael Stapelberg

Post by Bernhard R. Link
Another question: Have you considered asking for a archive Section for
those packages? I guess with no special section yet all those packages
would be section libdevel as they are for static linking only, wouldn't
they?

As pabs has pointed out, I did that, but the general rule of thumb is
that we want to have lots of packages first and an archive section
second.

Too avoid packaging mistakes and to have the wording ready I'd suggest
to already prepare the wording (I guess in the section later will be
everything that is is now in libdevel plus golong toolchain packages,
while everything providing programs to be in their respective section).

Which wording do you mean?

[Thomas Koch]

Post by Bernhard R. Link

Post by Michael Stapelberg
Given that we already have python-* and ruby-*, I’d find golang-* more
consistent.

We also have lib*-perl.

and lib*-java, but the java packaging might not be the best place to look for
best practices? (Since we're few people.)

I'm also following this thread to look for ideas for the D language which has
similar packaging characteristics like go as it seems.

Regards,

Thomas Koch, http://www.koch.ro