Bug#868527: want sbuild --no-source or something

[Ian Jackson]

Package: sbuild
Version: 0.73.0-4
Severity: wishlist

It would be nice if it were easier to use sbuild with a gitish
downstream workflow which does not produce "3.0 (quilt)" source
packages.

For example, consider this scenario:

dgit clone foo stretch
cd foo
hack hack hack
git commit
git merge upstream/stable
hack hack hack
git commit
gbp dch
sbuild -A
dpkg -iGOEB ../*.deb

This is similar to the recommendatioms in dgit-user(7), except that we
are trying to use sbuild.

The above attempt will probably fail.

This is because the package "foo" is probably "3.0 (quilt)", according
to debian/source/format. sbuild will try to build a source package,
and dpkg-source will want the user's changes to be "committed" as
patches in debian/patches. The user won't have done that and probably
doesn't want to. Worse, in the general case, changes might be made to
the git tree which dpkg-source cannot represent.

The solution is to not build a source package. The user who adopts a
gitish workflow probably doesn't want one. However, sbuild needs a
source package because that's how it transfers the package into the
chroot.

I suggest that sbuild grow an option which generates a dummy source
package in some hidden or temporary place. I think the actual source
package generation can be done with something like this rune (and that
this will always succeed):
dpkg-source -Zgzip -z0 --format=1.0 -sn
(but I have not yet tested this myself).

I don't have much of an opinion what the sbuild option should be
called. --no-source is a possibility. Or maybe it should be inferred
from the other options and the fact that sbuild is working from a
directory tree - although that would be an incompatible change.

In the meantime, I think users can use
sbuild -A --dpkg-source-opts='-Zgzip -z0 --format=1.0 -sn'
However, that sets up a hazard: it produces a dummy 1.0 native tarball
package as if it were a build product. But that native tarball
package is broken: it can be extracted and built, but it cannot be
re-packed, without again providing special optionsn to dpkg-source.

Thanks,
Ian.

[Johannes Schauer]

Quoting Ian Jackson (2017-07-16 14:32:17)

Post by Ian Jackson
It would be nice if it were easier to use sbuild with a gitish
downstream workflow which does not produce "3.0 (quilt)" source
packages.
[snip]
The above attempt will probably fail.
This is because the package "foo" is probably "3.0 (quilt)", according
to debian/source/format. sbuild will try to build a source package,
and dpkg-source will want the user's changes to be "committed" as
patches in debian/patches. The user won't have done that and probably
doesn't want to. Worse, in the general case, changes might be made to
the git tree which dpkg-source cannot represent.
The solution is to not build a source package. The user who adopts a
gitish workflow probably doesn't want one. However, sbuild needs a
source package because that's how it transfers the package into the
chroot.

Indeed, the task to solve is how to transfer the source into the chroot.

But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right thing to do?
Would it not be more fitting to use some git-based command to exchange the
data? For example using a git bundle? This would then make sure that everything
git knows about is transferred into the chroot. Sbuild can make sure that git
gets installed together with build-essential and is thus able to git unbundle
the file.

What do you think?

Thanks!

cheers, josch

[Raphael Hertzog]

Hi,

Post by Johannes Schauer
Indeed, the task to solve is how to transfer the source into the chroot.
But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right thing to do?

No, it's clearly a hack and one that might have side effects due to
differences (for example in set of ignored files) between format 1.0
and format 3.0 (quilt).

To me the right approach is to extend sbuild to support alternative ways
to copy the unpacked source tree into the build chroot. Your suggestion
to rely on a git bundle is a possibility. A plain tar of the build
directory is another approach.

IMO this goes in the right direction of entirely eliminating source
packages in the long term.

Cheers,

[Ian Jackson]

Post by Raphael Hertzog

Post by Johannes Schauer
Indeed, the task to solve is how to transfer the source into the chroot.
But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right thing to do?

No, it's clearly a hack and one that might have side effects due to
differences (for example in set of ignored files) between format 1.0
and format 3.0 (quilt).

Indeed. I think it's the best recommendation, though, with the
current tooling. Unless you can think of a better option, I think I
should recommend this hack to stretch users.

You are right that it has differences in the ignored files.
Specifically, this rune transfers the complete contents of the working
tree, including the .git directory, into the build environment. I
doubt that is likely to cause trouble in practice.

Ian.

[Johannes Schauer]

Hi,

Post by Ian Jackson

Post by Raphael Hertzog

Post by Johannes Schauer
Indeed, the task to solve is how to transfer the source into the chroot.
But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right thing to do?

No, it's clearly a hack and one that might have side effects due to
differences (for example in set of ignored files) between format 1.0
and format 3.0 (quilt).

Indeed. I think it's the best recommendation, though, with the
current tooling. Unless you can think of a better option, I think I
should recommend this hack to stretch users.
You are right that it has differences in the ignored files.
Specifically, this rune transfers the complete contents of the working
tree, including the .git directory, into the build environment. I doubt that
is likely to cause trouble in practice.

It has been more than two years since the last message in this bugreport. As
dgit becomes more and more mature, maybe we should revisit what sbuild could do
to improve on the current situation?

For example I would not be opposed to a --dgit switch which would then do the
right thing as it knows about the state of a source tree that dgit is able to
work with and how it can (and should) be transported into an sbuild chroot.

Personally, I never needed the dpkg-source workaround for any of my packages
with dgit but if anybody wants to propose a patch then I'd like to review and
include it into sbuild.

Thanks!

cheers, josch

[Sean Whitton]

Hello Johannes,

Post by Johannes Schauer
It has been more than two years since the last message in this bugreport. As
dgit becomes more and more mature, maybe we should revisit what sbuild could do
to improve on the current situation?
For example I would not be opposed to a --dgit switch which would then do the
right thing as it knows about the state of a source tree that dgit is able to
work with and how it can (and should) be transported into an sbuild chroot.

Thank you for raising this topic again.

dgit can be used with several different branch layouts. We probably
don't want to copy that knowledge from dgit into sbuild, so perhaps what
we want is for dgit to pass to sbuild the name of a git tree, which
sbuild will then try to build?

An issue with this is passing uncommitted changes to sbuild. But
perhaps someone who wants to do that is probably best using the existing
sbuild wrapper where you just type 'sbuild' from the root of the source
tree.

[Johannes Schauer]

Hi,

Quoting Sean Whitton (2019-09-02 01:00:12)

Post by Sean Whitton

Post by Johannes Schauer
It has been more than two years since the last message in this bugreport.
As dgit becomes more and more mature, maybe we should revisit what sbuild
could do to improve on the current situation?
For example I would not be opposed to a --dgit switch which would then do the
right thing as it knows about the state of a source tree that dgit is able to
work with and how it can (and should) be transported into an sbuild chroot.

Thank you for raising this topic again.
dgit can be used with several different branch layouts. We probably
don't want to copy that knowledge from dgit into sbuild, so perhaps what
we want is for dgit to pass to sbuild the name of a git tree, which
sbuild will then try to build?
An issue with this is passing uncommitted changes to sbuild. But
perhaps someone who wants to do that is probably best using the existing
sbuild wrapper where you just type 'sbuild' from the root of the source tree.

I just had another idea. Since the issue at hand is, that you need a way to
transfer the source tree into sbuild and sbuild currently only supports source
packages, why can dgit not be amended to build a temporary source package from
the git tree it has in a temporary location which it then hands to sbuild or
any other builder?

I mean, a dgit users will call "dgit sbuild" instead of just "sbuild" anyways,
no?

Thanks!

cheers, josch

[Sean Whitton]

Hello,

Post by Johannes Schauer
I just had another idea. Since the issue at hand is, that you need a way to
transfer the source tree into sbuild and sbuild currently only supports source
packages, why can dgit not be amended to build a temporary source package from
the git tree it has in a temporary location which it then hands to sbuild or
any other builder?

This is precisely what it already does -- I thought your interest here
was in removing the intermediate dpkg-source step.

[Johannes Schauer]

Hi,

Quoting Sean Whitton (2019-09-02 20:15:23)

Post by Johannes Schauer
I just had another idea. Since the issue at hand is, that you need a way to
transfer the source tree into sbuild and sbuild currently only supports
source packages, why can dgit not be amended to build a temporary source
package from the git tree it has in a temporary location which it then
hands to sbuild or any other builder?

This is precisely what it already does -- I thought your interest here was in
removing the intermediate dpkg-source step.

okay now I'm confused. I read [1] and it made me understand that users somehow
have to remember this horribly complicated sbuild command in certain
circumstances.

[1] https://lists.debian.org/***@chiark.greenend.org.uk

I came back to this bug because I find it horrible for users having to remember
this rune. But now you say that dgit is already taking care of doing the right
thing?

Thanks!

cheers, josch

[Sean Whitton]

Hello,

Post by Johannes Schauer
okay now I'm confused. I read [1] and it made me understand that users somehow
have to remember this horribly complicated sbuild command in certain
circumstances.
I came back to this bug because I find it horrible for users having to remember
this rune. But now you say that dgit is already taking care of doing the right
thing?

I'm sorry, I think I confused two dgit+sbuild topics. Let's continue
the discussion in response to Ian's message. Thanks again for your
interest.

[Ian Jackson]

Post by Johannes Schauer
I just had another idea. Since the issue at hand is, that you need a
way to transfer the source tree into sbuild and sbuild currently
only supports source packages, why can dgit not be amended to build
a temporary source package from the git tree it has in a temporary
location which it then hands to sbuild or any other builder?

Hi.

Thanks for looking at this bug again. I have been meaning to reply to
your messages for some days. Sorry for the delay, which is partly
caused by family stuff and partly by me traveling a lot (right now I
am in Madrid for the GNU Hackers Meeting...)

I'm afraid this is going to be a long mail.


I don't think Sean is right about the existence of functionality in
dgit which addresses this issue.

dgit is a bidirectional geteway between a dsc-style archive (and .dscs
in general) and git. When it is generating a .dsc, it tries to
generate a sane .dsc. dgit (at least in its existing operation modes)
cannot be used with git trees which are not representable as sane
.dscs.

It is *possible* to convert any git tree into a .dsc which will unpack
into that tree. But in the general case that conversion to a .dsc
requires several additional options to dpkg-source - the hideous rune
which is the subject of this bug report and which we are trying to get
rid of.

The resulting .dsc is insane. I don't want to try to completely
define "sane" in this context, but ISTM that a necessary condition for
sanity of a .dsc is that it is a fixed point under dpkg-source -x &&
dpkg-source -b. With one of these insane .dscs, dpkg-source -x works
but the subsequent dpkg-source -b fails (because it lacks the hideous
override options).


So let me step back a bit. I think it would be worth starting with an
investigation of what the problems are that this bug is about, and
then think about the best solution approach, intended new behaviour
and implementation strategy, and the names of the new stuff.

There are three things that seem to me like they are a significant
nuisance to a dgit user using a dgit view, with the current
arrangements. I fact they affect all users who are using pure git
workflow based on what Debian publishes, unless those users take
special care to maintain quilt patch queues up to date etc. - even if
those users don't ever run dgit (ie they got a git branch from
somewhere). I want users to be able to just use git and not have to
care about quilt etc.

The issues I see using sbuild in this situation are:

1. The need to pass a long, silly-looking and unergonomic option
to sbuild.

2. The fact that a .dsc is left lying around.

3. The fact that the .dsc left lying around is a beartrap: it cannot
be unpacked and repacked without again using weird options, and
must not be uploaded to anything like a Debian-style archive.

There are other possible concerns which I think are far less important
and possibly not important at all, but which I mention here only
because I thought of them (so that if someone else thinks they are
important they can be prompted to say so):

4. I'm not sure if making a 1.0 native gzip -1 dsc is as fast as other
possible approaches for copying the source into the chroot.

5. Possibly unclarity and infelicity surrounding untracked working
tree files and/or uncommitted changes.

6. Whether the .git directory should be passed into the chroot and
thus be present during the build. (I think yes - see below.)


Of these concerns, (3) follows inevitably from (2), in the sense that
if we want to deal with git trees generated from arbitrary packages,
and then modified in aribtrary ways (as envisaged in dgit-user(7)),
the .dsc might be insane, causing (3). Avoiding (2) avoids (3).

So we need to tackle (1) and (2).


It would of course be possible to write some wrapper program which:
- contains the horrid rune
- makes a .dsc somwhere private
- passes that .dsc to sbuild

There are a number of ways in which that would seem suboptimal to me:

- Such a wrapper program might have to interact with sbuild's option
parser (eg try to filter or understand options to sbuild). At the
very least, certain sbuild options don't make sense but maybe we
can just tell users who pass silly combinations and get strange
behaviours to please not do that.

- This would still do the work of building a source package, which
is needless copying etc. This is consideration (4) above.
Generating a .dsc makes us vulnerable to bugs and infelicities
in dpkg-source.

- The generated .dsc would presumably have to be in a directory in
/tmp or a temp directory under .git which is kind of inconvenient
from an implementation point of view (and we might leave it behind
if we don't manage to run our cleanup - ie we become less
crash-only). More generally the overall data flow is more
complicated than it needs to be.

- Thin wrapper programs are confusing to users and should be avoided
where possible. I know Debian packaging workflows are full of thin
wrappers - but this is indeed confusing. For example, any time you
see someone give a talk or a demo, or try to help them with some
packaging problem, you have some good chance that they will use
some wrapper or small utility you haven't heard of. This makes
everything hard to follow. I used to think that this was just me
being ignorant but since incidentally having to become an expert on
Debian git practices I have become convinced that there is so much
of this stuff no-one could be expected to know it all.

So I think it would be best to teach sbuild another way of
transferring source into the chroot, ideally without making an
unnecessary additional copy of it.

To my mind the obvious approach would be to make a new git tree in the
chroot and "git push" to it. (We want to copy the git objects because
some gittish workflows want a working .git directory, for example to
encode the built commitid somewhere in the output. This is obviously
not done in Debian but it would make sense for some downstream or
ad-hoc workflows. This does imply that git would have to be installed
in the chroot, by sbuild presumably.)

I haven't looked at the internals of sbuild at all. Do you think this
would be fairly straightforward ? Is it desirable ?

For the new option name(s) I suggest:
--git-ref=<somthing git-rev-parse understands>
--git equivalent to --git-ref=HEAD


As for naming etc.:

I appreciate that you are seeing this through a dgit lens. I can see
why, and dgit adoption is certainly a reason to think about this again
now. But I don't think this is actually "dgit stuff":

dgit's purpose is to help bridge the gap between source packages and
git. If you do not have an actual desire for source packages then you
should not get involved with dgit. (And if your involvement with
source packages is solely that your upstream publishes dscs, then you
use dgit clone/fetch to access it but then work solely in git.)

Ie, you can start with a pure git tree containing a debian/rules and
so forth, and modify it and build it, and so on, without involving
dgit. dgit is complicated and users who don't need to deal with dscs
shouldn't need to wrestle with it. (Ideally eventually dgit would
render itself obsolete...)

So for this reason I think any new sbuild option should *not* be named
after dgit. Likewise, if we decide that the best approach is the
wrapper script, I think the sbuild wrapper is probably best not being
function of dgit.


If we go for a sbuild wrapper I'm not sure where it should live. I
have been putting various things that aren't to do with dgit.deb or
/usr/bin/dgit in src:dgit because of the test suite I have there. But
testing such a wrapper script doesn't really seem to involve most of
the stuff that the src:dgit test suite is good at. Mostly it would
probably involve stuff that the sbuild test suite probably knows about
already. So maybe it should live in src:sbuild...


HTH.

Regards,
Ian.

[Ian Jackson]

Post by Johannes Schauer
Indeed, the task to solve is how to transfer the source into the chroot.
But is $(dpkg-source -Zgzip -z0 --format=1.0 -sn) really the right
thing to do? Would it not be more fitting to use some git-based
command to exchange the data? For example using a git bundle? This
would then make sure that everything git knows about is transferred
into the chroot. Sbuild can make sure that git gets installed
together with build-essential and is thus able to git unbundle the
file.
What do you think?

That would be a perfectly fine answer from my point of view. However,
it has a few different behaviours. I'm not sure what is best.

Things I thought of:

1. If there are files which are ignored, or uncommitted, this would
use the committed files. I think this is a benefit.

2. Using git this way means git is definitely in the chroot.
But a package might build differently. For example, it might
have a stealth b-dep on git, which this approach wouldn't detect.
I think this is a downside of using git.

3. The git approach cannot be easily used with a different VCS.

4. Building and unpacking a git bundle is pointless work when we know
that the destination completely trusts the source, and does not
have any objects already. A tarball would do.

From the point of view of a dgit user, item 1 is quite important and
IMO outweighs the other considerations.

Regards,
Ian.

[Johannes Schauer]

Quoting Ian Jackson (2017-07-17 11:31:07)

Post by Ian Jackson
That would be a perfectly fine answer from my point of view. However,
it has a few different behaviours. I'm not sure what is best.
1. If there are files which are ignored, or uncommitted, this would
use the committed files. I think this is a benefit.
2. Using git this way means git is definitely in the chroot.
But a package might build differently. For example, it might
have a stealth b-dep on git, which this approach wouldn't detect.
I think this is a downside of using git.

The only assumption sbuild should make about the chroot is, that all
Essential:yes packages are installed and that apt is installed and able to
install more packages. Starting from this situation sbuild should be able to
install anything that's needed to carry out the build like build-essential. It
is not impossible to only temporarily install git for unpacking the bundle and
to then remove and autoremove it before dpkg-buildpackage is run.

Post by Ian Jackson
3. The git approach cannot be easily used with a different VCS.

The switch could be git-specific in which case it's up to the user to make sure
that the underlying VCS is really git.

If the new switch is not git-specific then sbuild could auto-detect whether
it's git (and then do git-ish stuff) or not (and then just make a tarball).

Post by Ian Jackson
4. Building and unpacking a git bundle is pointless work when we know that
the destination completely trusts the source, and does not have any
objects already. A tarball would do.

Indeed a git bundle is overkill because dpkg-buildpackage inside the chroot is
not interested in the history stored in the bundle. But just using "tar" would
not have the advantages of item 1. So "git archive" should be used together
with its disadvantages (and workaround) from item 3.

Post by Ian Jackson
From the point of view of a dgit user, item 1 is quite important and IMO
outweighs the other considerations.

Agreed.

I wonder what the best interface for this feature would be. Even more if we
consider this a long term feature, existing git workflows as well as dgit...

Thanks!

cheers, josch

[Ian Jackson]

Post by Johannes Schauer
Quoting Ian Jackson (2017-07-17 11:31:07)

Post by Ian Jackson
4. Building and unpacking a git bundle is pointless work when we know that
the destination completely trusts the source, and does not have any
objects already. A tarball would do.

Indeed a git bundle is overkill because dpkg-buildpackage inside the
chroot is not interested in the history stored in the bundle. But
just using "tar" would not have the advantages of item 1. So "git
archive" should be used together with its disadvantages (and
workaround) from item 3.

Post by Ian Jackson
3. The git approach cannot be easily used with a different VCS.

The switch could be git-specific in which case it's up to the user
to make sure that the underlying VCS is really git.
If the new switch is not git-specific then sbuild could auto-detect whether
it's git (and then do git-ish stuff) or not (and then just make a tarball).

That would be OK. That would leave open the possibility of using
other vcs's tarball export functions.

In this context this project came to mind:
http://www.greenend.org.uk/rjk/vcs/

Post by Johannes Schauer
I wonder what the best interface for this feature would be. Even more if we
consider this a long term feature, existing git workflows as well as dgit...

Yes. I didn't want to make some wrapper in dgit for this because
1. we already have too many wrappers in our build tools 2. my doctrine
is that dgit should be necessary only if source packages are involved.

Regards,
Ian.