Do not make gratuitous source uploads just to provoke the buildds!

[Steve Langasek]

Changelog entry from a package that has just arrived in incoming:

gnucash (1.8.10-8) unstable; urgency=high
.
* high urgency upload because the fix for critical bug 291632 didn't get
into testing because of recompilation bugs, those later fixes were
uploaded with urgency low, and the package is still badly stuck.
People need the bug fix for 291632 and this will move it along. No
changes to the source.

No, Thomas, this will NOT move it along. As I explained to you in response
to your last message to debian-release, the missing builds have been
awaiting fixes to our buildd capacity on a pair of architectures, and
gnucash 1.8.10-7 was already in the Needs-Build queue on both of those
architectures (as you could see from
http://people.debian.org/~igloo/status.php?packages=gnucash, if you're not
used to interpreting the pages on buildd.debian.org). The reason gnucash
has not been built yet on arm or mipsel is because there is a backlog of
over a hundred packages on each of these architectures.

Re-uploading a package to provoke a buildd response is counterproductive,
*particularly* when the package is already in Needs-Build on the missing
architectures. Re-uploading doesn't change its position in the queue, but
it *does* force buildds for all the other archs to needlessly rebuild the
package. This is why the answer to your previous email was "please be
patient".

[Thomas Bushnell BSG]

Post by Steve Langasek
Re-uploading a package to provoke a buildd response is counterproductive,
*particularly* when the package is already in Needs-Build on the missing
architectures. Re-uploading doesn't change its position in the queue, but
it *does* force buildds for all the other archs to needlessly rebuild the
package. This is why the answer to your previous email was "please be
patient".

Unfortunately, the queue ordering policy is unclear. I was guessing
that the priority of the upload would have something to do with
queueing policy.

Since the all but one of the other arch buildd's have empty
needs-build queues, it is harmless to force them to execute a
recompile and costs no scarce resources. I did check this before
uploading.

I made an upload because a related package (grisbi) just seemed to get
compiled by all the buildd's in a nifty two-day round trip time. It
was uploaded March 10, compiled by most buildds on the 10th, and by
arm and mipsel on the 11th. I concluded that the queue must take note
of priority or something like that.

Perhaps it got through the queue because the grisbi upload fixed an
serious RC bug. Well, the gnucash one fixes a critical RC bug, but
that isn't indicated in the changelog. Maybe I should add that?

If, perhaps, there was a clear indication of the buildd ordering
policy, then it could be properly used. Until then, I go on the basis
of guesswork.

Thomas

[Jeroen van Wolffelaar]

Post by Thomas Bushnell BSG
If, perhaps, there was a clear indication of the buildd ordering
policy, then it could be properly used. Until then, I go on the basis
of guesswork.

You were *told*[1] to wait. Do not fall back to guesswork when someone
knowingly like Steve Langasek tells you what to do. Ignoring what gets
told to you by a release manager, and instead doing some uninformed
guesswork is not helping the release.

--Jeroen

[1] http://lists.debian.org/debian-release/2005/03/msg00047.html

[Thomas Bushnell BSG]

Post by Jeroen van Wolffelaar

Post by Thomas Bushnell BSG
If, perhaps, there was a clear indication of the buildd ordering
policy, then it could be properly used. Until then, I go on the basis
of guesswork.

You were *told*[1] to wait. Do not fall back to guesswork when someone
knowingly like Steve Langasek tells you what to do. Ignoring what gets
told to you by a release manager, and instead doing some uninformed
guesswork is not helping the release.
[1] http://lists.debian.org/debian-release/2005/03/msg00047.html

You are misreading that message.

In that message, Steve is advising me to wait for the buildd's to
clean up the xfree86-common chroot mess, and saying not that a
reupload does nothing, but instead is saying that further email to the
buildd maintainers is unnecessary.

Thomas

[Steve Langasek]

Post by Thomas Bushnell BSG

Post by Steve Langasek
Re-uploading a package to provoke a buildd response is counterproductive,
*particularly* when the package is already in Needs-Build on the missing
architectures. Re-uploading doesn't change its position in the queue, but
it *does* force buildds for all the other archs to needlessly rebuild the
package. This is why the answer to your previous email was "please be
patient".

Unfortunately, the queue ordering policy is unclear. I was guessing
that the priority of the upload would have something to do with
queueing policy.

Yes, it is unclear. The reality is that upload priority does not contribute
to queue ordering. There's been sufficient confusion on this point that I
had to check the wanna-build code for myself to be sure of it, and I'm aware
that confusion persists for many.

Post by Thomas Bushnell BSG
Since the all but one of the other arch buildd's have empty
needs-build queues, it is harmless to force them to execute a
recompile and costs no scarce resources. I did check this before
uploading.

It is not harmless; it costs buildd admin time to review/process the build
logs, and if the libraries available in unstable change on one or more
architectures between the time the package was previously built and the next
time it's built, there can be additional delay resulting from waiting on
those other libraries to transition to testing.

Post by Thomas Bushnell BSG
I made an upload because a related package (grisbi) just seemed to get
compiled by all the buildd's in a nifty two-day round trip time. It
was uploaded March 10, compiled by most buildds on the 10th, and by
arm and mipsel on the 11th. I concluded that the queue must take note
of priority or something like that.
Perhaps it got through the queue because the grisbi upload fixed an
serious RC bug. Well, the gnucash one fixes a critical RC bug, but
that isn't indicated in the changelog. Maybe I should add that?

The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
sorted by:

- target suite
- package priority
- package section
- package name

I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

[Thomas Bushnell BSG]

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I see, the problem is clearly that gnucash is in the "Extra" priority
instead of the "Optional" section where it belongs. I'll request the
ftpmasters to change it.

Thomas

[Andreas Barth]

Post by Thomas Bushnell BSG

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I see, the problem is clearly that gnucash is in the "Extra" priority
instead of the "Optional" section where it belongs. I'll request the
ftpmasters to change it.

Our goal is that the queue gets empty from time to time, and so,
priority shouldn't prevent a package from being built.


Cheers,
Andi

[Hamish Moffatt]

Post by Andreas Barth
Our goal is that the queue gets empty from time to time, and so,
priority shouldn't prevent a package from being built.

How often should the queue be emptied, or when will an architecture be
declarared not-keeping-up?

According to
http://people.debian.org/~igloo/needs-build-graph/index.php?days=30
arm hasn't been empty in over 3 weeks, and it's getting worse still.
s390 is also rising steeply.


Hamish

[Bastian Blank]

Post by Hamish Moffatt
s390 is also rising steeply.

gcc problems and non-responding ftp-masters for the new buildd.

Bastian

[Andreas Barth]

Post by Hamish Moffatt

Post by Andreas Barth
Our goal is that the queue gets empty from time to time, and so,
priority shouldn't prevent a package from being built.

How often should the queue be emptied, or when will an architecture be
declarared not-keeping-up?

In light of
http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html
the release architecture must have N+1 buildds where N is the number
required to keep up with the volume of uploaded packages
at least once per day for etch.

Post by Hamish Moffatt
According to
http://people.debian.org/~igloo/needs-build-graph/index.php?days=30
arm hasn't been empty in over 3 weeks, and it's getting worse still.
s390 is also rising steeply.

It is not only a question of days, but also, what expectations we have
for the architecture. If we e.g. know that in the next days a buildd
will be switched on again (as the buildd crashed, and the local admin is
away), we are of course a bit more tolerant. Also, vacations are
accepted.


Cheers,
Andi

[Goswin von Brederlow]

Post by Andreas Barth

Post by Hamish Moffatt

Post by Andreas Barth
Our goal is that the queue gets empty from time to time, and so,
priority shouldn't prevent a package from being built.

How often should the queue be emptied, or when will an architecture be
declarared not-keeping-up?

In light of
http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html
the release architecture must have N+1 buildds where N is the number
required to keep up with the volume of uploaded packages
at least once per day for etch.

That means no more m68k. Given that some packages compile up to 12
days there will be plenty of times the queue doesn't empty once per
day.

Post by Andreas Barth

Post by Hamish Moffatt
According to
http://people.debian.org/~igloo/needs-build-graph/index.php?days=30
arm hasn't been empty in over 3 weeks, and it's getting worse still.
s390 is also rising steeply.

It is not only a question of days, but also, what expectations we have
for the architecture. If we e.g. know that in the next days a buildd
will be switched on again (as the buildd crashed, and the local admin is
away), we are of course a bit more tolerant. Also, vacations are
accepted.

I would like to see some stats showing on how many days in the last
year an arch reached 0 needs-build. I highly doubt that any arch
managed to do it every day troughout the last year.

If w-b had some aging algorithm instead of a fixed order we could say
an arch may not have more than 1 week lag for more than 1 week or
something.

Post by Andreas Barth
Cheers,
Andi

MfG
Goswin

[Andreas Barth]

Post by Goswin von Brederlow

Post by Andreas Barth

Post by Hamish Moffatt

Post by Andreas Barth
Our goal is that the queue gets empty from time to time, and so,
priority shouldn't prevent a package from being built.

How often should the queue be emptied, or when will an architecture be
declarared not-keeping-up?

In light of
http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html
the release architecture must have N+1 buildds where N is the number
required to keep up with the volume of uploaded packages
at least once per day for etch.

That means no more m68k. Given that some packages compile up to 12
days there will be plenty of times the queue doesn't empty once per
day.

needs-build can be empty even if packages _are_ currently building.

Post by Goswin von Brederlow
I would like to see some stats showing on how many days in the last
year an arch reached 0 needs-build. I highly doubt that any arch
managed to do it every day troughout the last year.

You know why goals are important? 0 needs-build is definitly a goal we
should work to.


Cheers,
Andi

[Goswin von Brederlow]

Post by Andreas Barth

Post by Goswin von Brederlow

Post by Andreas Barth

Post by Hamish Moffatt

Post by Andreas Barth
Our goal is that the queue gets empty from time to time, and so,
priority shouldn't prevent a package from being built.

How often should the queue be emptied, or when will an architecture be
declarared not-keeping-up?

In light of
http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html
the release architecture must have N+1 buildds where N is the number
required to keep up with the volume of uploaded packages
at least once per day for etch.

That means no more m68k. Given that some packages compile up to 12
days there will be plenty of times the queue doesn't empty once per
day.

needs-build can be empty even if packages _are_ currently building.

Not with "N may not be > 2". Say 3 mozilla clones get uploaded. Each
of the N+1 buildds grabs one and ist busy for 5 days. If any other
package gets uploaded in that time the queue will not be empty.

Unless you take packages even while building. But I'm sure a long
"building" queue on the buildds would be cheating and not count.

Post by Andreas Barth

Post by Goswin von Brederlow
I would like to see some stats showing on how many days in the last
year an arch reached 0 needs-build. I highly doubt that any arch
managed to do it every day troughout the last year.

You know why goals are important? 0 needs-build is definitly a goal we
should work to.

I disagree. 0 needs-build once a day is a bad line to draw. Saying
packages must be build a day before they become testing candidates
would be a better line. But that would require a non starving queue to
mean anything but 0 needs-build.

I don't see any great harm with packages getting build 5 days late if
they have to wait 10 days for testing. As long as they do get build on
time.


But what do I know, I'm not an RM. So lets thing about the criterion:

Strictly requiring 0 needs-builds every day means the buildd must have
enough power to cope even with huge upload peeks and if one of the
buildds fail at a peak time no arch will cope with that. Obviously
some leaway will have to be given for arch to temporarily not meat the
criterion, say 0 needs-build on 75% of all days and no more than 3
consecuitve failures wihtout special circumstances or something of
that sort. Right?

Or do you realy want to remove i386 from the release if it fails 0
needs-build 10 times before etch release?

Post by Andreas Barth
Cheers,
Andi
--
http://home.arcor.de/andreas-barth/
PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C

MfG
Goswin

[Steve Langasek]

Post by Goswin von Brederlow

Post by Andreas Barth

Post by Goswin von Brederlow
I would like to see some stats showing on how many days in the last
year an arch reached 0 needs-build. I highly doubt that any arch
managed to do it every day troughout the last year.

You know why goals are important? 0 needs-build is definitly a goal we
should work to.

I disagree. 0 needs-build once a day is a bad line to draw. Saying
packages must be build a day before they become testing candidates
would be a better line. But that would require a non starving queue to
mean anything but 0 needs-build.
I don't see any great harm with packages getting build 5 days late if
they have to wait 10 days for testing. As long as they do get build on
time.

Ah, but packages that are being uploaded with urgency=low are usually not a
concern for the release team at all; it's the *high*-urgency uploads that
normally demand the release team's attention, and these are precisely the
ones that slower build architectures are going to have a harder time
building within the purgatory interval (2 days, not 10, for high-urgency
packages). To say nothing of the fact that urgency is not currently
considered for package build ordering, which means that it's very possible
for a high-urgency upload to go for 2 days without being tried at all.

Post by Goswin von Brederlow
Strictly requiring 0 needs-builds every day means the buildd must have
enough power to cope even with huge upload peeks and if one of the
buildds fail at a peak time no arch will cope with that. Obviously
some leaway will have to be given for arch to temporarily not meat the
criterion, say 0 needs-build on 75% of all days and no more than 3
consecuitve failures wihtout special circumstances or something of
that sort. Right?
Or do you realy want to remove i386 from the release if it fails 0
needs-build 10 times before etch release?

Andreas never said anything about this being the criterion for RC
architectures. He said it should be a *goal*.

[David Schmitt]

Post by Goswin von Brederlow

Post by Andreas Barth

Post by Hamish Moffatt

Post by Andreas Barth
Our goal is that the queue gets empty from time to time, and so,
priority shouldn't prevent a package from being built.

How often should the queue be emptied, or when will an architecture be
declarared not-keeping-up?

In light of
http://lists.debian.org/debian-devel-announce/2005/03/msg00012.html
the release architecture must have N+1 buildds where N is the number
required to keep up with the volume of uploaded packages
at least once per day for etch.

That means no more m68k. Given that some packages compile up to 12
days there will be plenty of times the queue doesn't empty once per
day.

Perhaps that was a slight misunderstanding: the Nybbles only require "the
release architecture must have N+1 buildds where N is the number required to
keep up with the volume of uploaded packages" with N <= 2.

The part about emptying once per day was only added by Andreas.

Considering the effects of a twelve-day build of something big like KDE, GNOME
or X: delays in security updates, shlib-deps, build-depends and testing
migration, I can see the roots of the requirements on buildds.


Regards, David

[Matthew Palmer]

[Probably going a bit off track for -release; MFT to -devel]

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I'm trying to work out why package *section* matters at all. Package name
is a bit odd, too, but including the section in there is just totally whack.
Upload priority would be nice to sort by, but I think the queue needs to be
as FIFO as possible for fairness and "principle of least surprise" sake.

Now I have this urge to go and make surgery on w-b.

- Matt

[Steve Langasek]

Post by Matthew Palmer
[Probably going a bit off track for -release; MFT to -devel]

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I'm trying to work out why package *section* matters at all. Package name
is a bit odd, too, but including the section in there is just totally whack.

Consider that libraries have their own section. Certain package sections
are (on the whole) more central to the dependency graph than others, so it's
to our advantage to order those first to reduce the need for give-backs or
dep-waits.

Post by Matthew Palmer
Upload priority would be nice to sort by, but I think the queue needs to be
as FIFO as possible for fairness and "principle of least surprise" sake.
Now I have this urge to go and make surgery on w-b.

Given that the w-b maintainers disagree, changing the code is not the hard
part. The system is designed such that it only really works properly if the
buildds drain the Needs-Build queue on a regular basis. This doesn't seem
terribly robust to me, but it's not my call.

[Hamish Moffatt]

Post by Steve Langasek

Post by Matthew Palmer
I'm trying to work out why package *section* matters at all. Package name
is a bit odd, too, but including the section in there is just totally whack.

Consider that libraries have their own section. Certain package sections
are (on the whole) more central to the dependency graph than others, so it's
to our advantage to order those first to reduce the need for give-backs or
dep-waits.

libs, devel and some others makes sense, but the others all have a
defined order too according to recent discussion on debian-release.
Of course this is never apparently when things are running smoothly with
all the buildds.

Is there anything that can be done to help with arm/mipsel?
I think I read that new machines/owners aren't welcome currently anyway.

Hamish

[Martin Zobel-Helas]

Hi Hamish,

Post by Hamish Moffatt
Is there anything that can be done to help with arm/mipsel?

<ironic>
Not uploading any new packages *g*
</ironic>

As it has been said earlier, the machines are back, but just need to
catch up. So just waiting might be the easiest thing to do.


Greetings
Martin

[Ingo Juergensmann]

Post by Martin Zobel-Helas
As it has been said earlier, the machines are back, but just need to
catch up. So just waiting might be the easiest thing to do.

More machines can catch up faster than few can do.
When one machine out of a dozen machines is unavailable, it has less impact
than one machine failure out of two machines, although the chances will
raise that a machine will fail somewhen with more machines. But the impact
is less critical...
I can't understand why no additional machines are being accepted.

[Martin Zobel-Helas]

Hi Ingo,

Post by Ingo Juergensmann

Post by Martin Zobel-Helas
As it has been said earlier, the machines are back, but just need to
catch up. So just waiting might be the easiest thing to do.

More machines can catch up faster than few can do.
When one machine out of a dozen machines is unavailable, it has less impact
than one machine failure out of two machines, although the chances will
raise that a machine will fail somewhen with more machines. But the impact
is less critical...
I can't understand why no additional machines are being accepted.

Please clarify this with the wanna-build admin(s).

Greetings
Martin

[Ingo Juergensmann]

Post by Martin Zobel-Helas

Post by Ingo Juergensmann

Post by Martin Zobel-Helas
As it has been said earlier, the machines are back, but just need to
catch up. So just waiting might be the easiest thing to do.

More machines can catch up faster than few can do.
When one machine out of a dozen machines is unavailable, it has less impact
than one machine failure out of two machines, although the chances will
raise that a machine will fail somewhen with more machines. But the impact
is less critical...
I can't understand why no additional machines are being accepted.

Please clarify this with the wanna-build admin(s).

Been there, done that.
The short answer: "We don't want anyone else to play in our playground!"
The longer answer: "More machines mean more work for the the buildd admin.
Additional buildd admins for those archs are not wanted, because that would
need communication between the buildd admins."

Because w-b admins are the same persons as the buildd admin for certain
archs which give regular problems, you can guess, where the real problem
lies.

[Martijn van Oosterhout]

On Sat, 12 Mar 2005 14:42:33 +0100, Ingo Juergensmann

Post by Ingo Juergensmann
Been there, done that.
The short answer: "We don't want anyone else to play in our playground!"
The longer answer: "More machines mean more work for the the buildd admin.
Additional buildd admins for those archs are not wanted, because that would
need communication between the buildd admins."
Because w-b admins are the same persons as the buildd admin for certain
archs which give regular problems, you can guess, where the real problem
lies.

Given that there is all this discussion about putting the
responsibility for keeping up, security updates and releasing back
onto the porters, maybe they should also be given the right to manage
and setup as many autobuilders as they see fit.

[Andreas Barth]

Post by Ingo Juergensmann

Post by Martin Zobel-Helas
As it has been said earlier, the machines are back, but just need to
catch up. So just waiting might be the easiest thing to do.

More machines can catch up faster than few can do.
When one machine out of a dozen machines is unavailable, it has less impact
than one machine failure out of two machines, although the chances will
raise that a machine will fail somewhen with more machines. But the impact
is less critical...

If you take a look at
Loading Image..., you can e.g. see for
mipsel exactly at which date the slow and at which the fast machine
became available again. The fast machine alone is able to keep up with
the usual upload rates.


Cheers,
Andi

[Ingo Juergensmann]

Post by Andreas Barth

Post by Ingo Juergensmann
More machines can catch up faster than few can do.
When one machine out of a dozen machines is unavailable, it has less impact
than one machine failure out of two machines, although the chances will
raise that a machine will fail somewhen with more machines. But the impact
is less critical...

If you take a look at
http://buildd.debian.org/stats/graph2-week-big.png, you can e.g. see for
mipsel exactly at which date the slow and at which the fast machine
became available again. The fast machine alone is able to keep up with
the usual upload rates.

And? Where's the conflicting part of what I've said?

If you go back just long enough in time, you would see that there was a time
when kullervo had no problems with keeping up at all. And then you see that
arrakis helped kullervo in doing that job.
And coming back to today, you'll notice that m68k has no problems with
keeping up at all.

Spot the differences to other archs yourself.

BTW: don't forget to apply the passwords to your buildds for status updates.

[Hamish Moffatt]

Post by Martin Zobel-Helas
Hi Hamish,

Post by Hamish Moffatt
Is there anything that can be done to help with arm/mipsel?

As it has been said earlier, the machines are back, but just need to
catch up. So just waiting might be the easiest thing to do.

arm doesn't appear to be catching up. New packages are being uploaded at
least as quickly as they're being built. My package geda-gschem has not
really changed its placement in the last week; actually I think it's
slipped further down the list.


Hamish

[Goswin von Brederlow]

Post by Steve Langasek

Post by Matthew Palmer
[Probably going a bit off track for -release; MFT to -devel]

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I'm trying to work out why package *section* matters at all. Package name
is a bit odd, too, but including the section in there is just totally whack.

Consider that libraries have their own section. Certain package sections
are (on the whole) more central to the dependency graph than others, so it's
to our advantage to order those first to reduce the need for give-backs or
dep-waits.

Build-Depends should be set automaticaly as Dep-Wait for every source
upload. That would reduce a lot of needless work for both machines and
admins.

Post by Steve Langasek

Post by Matthew Palmer
Upload priority would be nice to sort by, but I think the queue needs to be
as FIFO as possible for fairness and "principle of least surprise" sake.

I think package urgency isn't considered because people would abuse it
to get their packages build faster, or so someone nameless fears.

Remember that the buildd queue is not FIFO at all. The queue has a
completly static order. Any changes to the queue are just packages
hiding because they are not "needs-build". I consider that the biggest
flaw of all in wanna-build.

Post by Steve Langasek

Post by Matthew Palmer
Now I have this urge to go and make surgery on w-b.

Yes please. Packages should Dep-Wait automatically, should be ordered
by '(age * alpha) + beta' with alpha / beta set by at least the
package priority and upload urgency. Optionaly also the number of
Build-Depends and revers Build-Depends on the package.

Post by Steve Langasek
Given that the w-b maintainers disagree, changing the code is not the hard
part. The system is designed such that it only really works properly if the
buildds drain the Needs-Build queue on a regular basis. This doesn't seem
terribly robust to me, but it's not my call.

If you can convince the w-b admins to allow changes I could send you
patches. Having a queue that will hapily starve packages is quite
unfair to maintainers. Next you know x* will be renamed to a* just to
get it build faster.

Post by Steve Langasek
--
Steve Langasek
postmodern programmer

MfG
Goswin

[Thomas Bushnell BSG]

Post by Goswin von Brederlow
Remember that the buildd queue is not FIFO at all. The queue has a
completly static order. Any changes to the queue are just packages
hiding because they are not "needs-build". I consider that the biggest
flaw of all in wanna-build.

This is news to me.

It means that when one is told "just wait, your package will get
rebuilt"; it is not necessarily true at all. There is no upper bound
at all on time to wait for building, and that's a disaster. People
should stop repeating the fiction then that "just wait" means "your
package will eventually get built".

Thomas

[Steve Langasek]

Post by Thomas Bushnell BSG

Post by Goswin von Brederlow
Remember that the buildd queue is not FIFO at all. The queue has a
completly static order. Any changes to the queue are just packages
hiding because they are not "needs-build". I consider that the biggest
flaw of all in wanna-build.

This is news to me.
It means that when one is told "just wait, your package will get
rebuilt"; it is not necessarily true at all. There is no upper bound
at all on time to wait for building, and that's a disaster. People
should stop repeating the fiction then that "just wait" means "your
package will eventually get built".

Er, packages *do* eventually get built; they just don't get built in any
kind of FIFO order. I don't particularly care for this arrangement myself
(it means there are plenty of times that a high-priority bug in a
low-priority package stays on the release team's watchlist for far too
long), but I don't have any proof that a different queue ordering would
actually work better for the project, and the buildd admins *are* committed
to keeping up with the queue even though hardware circumstances sometimes
prevent it from time to time.

[Thomas Bushnell BSG]

Post by Steve Langasek
Er, packages *do* eventually get built; they just don't get built in any
kind of FIFO order.

This is not true. The current system has an unbounded wait time. For
example, the effect of the Bug Squashing Party, which causes a bunch
of uploads to be queued, forces "extra" priority packages to make no
progress towards building, because all those optional packages shove
right in ahead. This is true even when the "extra" priority package
is fixing a severity critical bug, and the optional packages are
fixing only, say, "important" bugs.

As evidence, I note that gnucash is moving *backwards* in the queue,
and as long as more uploads happen, it will continue to. Therefore,
"just wait" doesn't actually mean that anything will happen.

Thomas

[Matthew Palmer]

Post by Steve Langasek

Post by Thomas Bushnell BSG

Post by Goswin von Brederlow
Remember that the buildd queue is not FIFO at all. The queue has a
completly static order. Any changes to the queue are just packages
hiding because they are not "needs-build". I consider that the biggest
flaw of all in wanna-build.

This is news to me.
It means that when one is told "just wait, your package will get
rebuilt"; it is not necessarily true at all. There is no upper bound
at all on time to wait for building, and that's a disaster. People
should stop repeating the fiction then that "just wait" means "your
package will eventually get built".

Er, packages *do* eventually get built; they just don't get built in any
kind of FIFO order.

Er, no. Unless there's some sort of aging process (not yet described in the
threads here) which will result in an extra package called zappa in section
x11 from eventually being promoted above a package aardvark in section
admin, it is entirely possible that package will never be built. All it
requires is for the rate of new packages entering the queue before zappa to
be equal to or greater than the rate of packages leaving the queue due to
having been built or removed.

Practically, buildd admins can notice a longer-than-usual queue and throw
hardware at the problem, and that seems to work well enough, and we could
reduce the rate of package inflow through various means, but the problem
still remains -- the queue prioritisation *can* lead to starvation. I'm not
advocating that it be on the top of anyone's todo list to fix it, because we
have relatively effective workarounds, but it's not healthy to say the
problem does not exist, either.

- Matt

[Thomas Bushnell BSG]

Post by Matthew Palmer
Practically, buildd admins can notice a longer-than-usual queue and throw
hardware at the problem, and that seems to work well enough, and we could
reduce the rate of package inflow through various means, but the problem
still remains -- the queue prioritisation *can* lead to starvation. I'm not
advocating that it be on the top of anyone's todo list to fix it, because we
have relatively effective workarounds, but it's not healthy to say the
problem does not exist, either.

What are these "relatively effective workarounds"? I recall recently
hearing that the buildd admins don't want extra machines. So what
then?

[Matthew Palmer]

Post by Thomas Bushnell BSG

Post by Matthew Palmer
Practically, buildd admins can notice a longer-than-usual queue and throw
hardware at the problem, and that seems to work well enough, and we could
reduce the rate of package inflow through various means, but the problem
still remains -- the queue prioritisation *can* lead to starvation. I'm not
advocating that it be on the top of anyone's todo list to fix it, because we
have relatively effective workarounds, but it's not healthy to say the
problem does not exist, either.

What are these "relatively effective workarounds"?

Not being a buildd admin, I have no idea as to the specifics, but I infer
the existence of these workarounds due to the fact that, on the whole, the
build queues don't appear to suffer from hideous starvation problems.

Post by Thomas Bushnell BSG
I recall recently hearing that the buildd admins don't want extra
machines. So what then?

Presumably that was "we don't want extra buildds administered by random
people who we have no control over", a policy which has been discussed to
death before. Nowhere have I seen a buildd admin say "we're happy not
having enough machines to keep up with demand".

- Matt

[Jeroen van Wolffelaar]

Post by Matthew Palmer

Post by Thomas Bushnell BSG
What are these "relatively effective workarounds"?

Not being a buildd admin, I have no idea as to the specifics, but I infer
the existence of these workarounds due to the fact that, on the whole, the
build queues don't appear to suffer from hideous starvation problems.

Apparantly an explanation is needed here:
- normally, buildd power is more than new packages getting uploaded
(otherwise, queue would only be growing, and never reduce)
- as a consequence, normally the Needs-Build queue size will become zero
frequently, only not if there are temporary buildd power issues
The graph at [1] illustrates this nicely, you can see that for example
somewhere around 15 feb 2005 all architecture's Needs-Build queue was
zero, and that only s390, mipsel and arm have not reached zero for the
last few days
- if Needs-Build is zero, all packages that are in this state are build,
including package zzzzz from x11-extra. As long as the queue does
reach zero from time to time, there is no starvation.

If the queue is non-zero for a longer time, there is a problem in buildd
machine power, and the wanna-build admin has choosen to in this case
allocate the buildd power that remains to the building of packages that
are of higher priority, regardless of their age in the queue. The
allocation of a scarce resource is almost by definition a trade-off, and
this is the decision that has been made.

--Jeroen

[1] http://people.debian.org/~igloo/needs-build-graph/index.php?days=30

[Thomas Bushnell BSG]

Post by Jeroen van Wolffelaar
If the queue is non-zero for a longer time, there is a problem in buildd
machine power, and the wanna-build admin has choosen to in this case
allocate the buildd power that remains to the building of packages that
are of higher priority, regardless of their age in the queue. The
allocation of a scarce resource is almost by definition a trade-off, and
this is the decision that has been made.

First off, I think much confusion has been caused by using the word
queue here. A queue is a FIFO list; if there isn't even the least bit
FIFO in its management, which seems to be the case, then it shouldn't
be called a queue. If it were not called a queue, I would not have
made many wrong assumptions, and I think others too, to assume that of
course some kind of FIFO processing was happening. So PLEASE change
the name; stop calling it a queue.

I can see excellent reasons why age in the list shouldn't matter. But
package "priority" and "section" are extremely poor bases to decide
what the actual importance of a package is. I think the three most
critical factors are whether the package closes bugs, and the priority
of the bugs it closes (counting all the bugs closed between the
current unstable version for that arch and the upload being
considered); the stated priority of the upload itself, whether low,
medium, or high; and *particular* cases of section and priority. It
makes sense to have Required and Standard packages go first; it makes
sense to have libraries go first.

Thomas

[Wouter Verhelst]

Post by Thomas Bushnell BSG

Post by Jeroen van Wolffelaar
If the queue is non-zero for a longer time, there is a problem in buildd
machine power, and the wanna-build admin has choosen to in this case
allocate the buildd power that remains to the building of packages that
are of higher priority, regardless of their age in the queue. The
allocation of a scarce resource is almost by definition a trade-off, and
this is the decision that has been made.

First off, I think much confusion has been caused by using the word
queue here. A queue is a FIFO list; if there isn't even the least bit
FIFO in its management, which seems to be the case, then it shouldn't
be called a queue. If it were not called a queue, I would not have
made many wrong assumptions, and I think others too, to assume that of
course some kind of FIFO processing was happening. So PLEASE change
the name; stop calling it a queue.

None of the documentation calls it a 'queue', in fact; only people not
really involved in buildd stuff do.

Post by Thomas Bushnell BSG
I can see excellent reasons why age in the list shouldn't matter. But
package "priority" and "section" are extremely poor bases to decide
what the actual importance of a package is.

Why would that be the case? You're telling me you think gnome-games is
way more important than gcc for us to build?
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Thomas Bushnell BSG]

Post by Wouter Verhelst
None of the documentation calls it a 'queue', in fact; only people not
really involved in buildd stuff do.

Does that include you? In two recent messages, you referred to it as
a queue.

Post by Wouter Verhelst

Post by Thomas Bushnell BSG
I can see excellent reasons why age in the list shouldn't matter. But
package "priority" and "section" are extremely poor bases to decide
what the actual importance of a package is.

Why would that be the case? You're telling me you think gnome-games is
way more important than gcc for us to build?

No. I'm saying that when priorities are marked badly, the results are
disastrous. I'm also saying that a static ordering produces a
perverse incentive, especially when a priority is marked badly, not to
upload fixes for any other package.

Thomas

[Wouter Verhelst]

Post by Thomas Bushnell BSG

Post by Wouter Verhelst
None of the documentation calls it a 'queue', in fact; only people not
really involved in buildd stuff do.

Does that include you? In two recent messages, you referred to it as
a queue.

Yes, that's correct; I shouldn't have. Sorry :-)
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Wouter Verhelst]

Post by Thomas Bushnell BSG

Post by Matthew Palmer
Practically, buildd admins can notice a longer-than-usual queue and throw
hardware at the problem, and that seems to work well enough, and we could
reduce the rate of package inflow through various means, but the problem
still remains -- the queue prioritisation *can* lead to starvation. I'm not
advocating that it be on the top of anyone's todo list to fix it, because we
have relatively effective workarounds, but it's not healthy to say the
problem does not exist, either.

What are these "relatively effective workarounds"?

Ask people on port-specific mailinglists to manually build a package for
you (not every Debian Developer on those are buildd admins). Build the
package on the port machine available to all Debian Developers (no, that
indeed doesn't work for all packages). Find someone with a machine of
the target architecture and build it on there yourself. Etcetera.
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Andreas Barth]

Post by Matthew Palmer

Post by Steve Langasek
Er, packages *do* eventually get built; they just don't get built in any
kind of FIFO order.

Er, no. Unless there's some sort of aging process (not yet described in the
threads here) which will result in an extra package called zappa in section
x11 from eventually being promoted above a package aardvark in section
admin, it is entirely possible that package will never be built. All it
requires is for the rate of new packages entering the queue before zappa to
be equal to or greater than the rate of packages leaving the queue due to
having been built or removed.

If that happens for a too long period, we might consider such an
architecture to be too slow to keep up, and will eventually discuss
about kicking it out of the architectures we wait for testing migration
at all, or even kicking it out of testing at all. Not waiting for such
an arch has happened and might happen again.


Cheers,
Andi

[Mike Fedyk]

Post by Andreas Barth

Post by Matthew Palmer

Post by Steve Langasek
Er, packages *do* eventually get built; they just don't get built in any
kind of FIFO order.

Er, no. Unless there's some sort of aging process (not yet described in the
threads here) which will result in an extra package called zappa in section
x11 from eventually being promoted above a package aardvark in section
admin, it is entirely possible that package will never be built. All it
requires is for the rate of new packages entering the queue before zappa to
be equal to or greater than the rate of packages leaving the queue due to
having been built or removed.

If that happens for a too long period, we might consider such an
architecture to be too slow to keep up, and will eventually discuss
about kicking it out of the architectures we wait for testing migration
at all, or even kicking it out of testing at all. Not waiting for such
an arch has happened and might happen again.

I think it makes sense to shorten the list of arches we wait upon for
testing migration, but I fail to see the usefulness of removing an arch
from testing.

You may not see the usefulness of testing, but I run production servers
on it because there are just too many packages that are too old and
buggy or not available in the old stable release. Yes, I've only used
i386 and PPC, but that should not exclude that functionality from the
users of those architectures that are not as popular (even if you need a
log scale to see their relative popularity).

Mike

[Andreas Barth]

Post by Mike Fedyk

Post by Andreas Barth
If that happens for a too long period, we might consider such an
architecture to be too slow to keep up, and will eventually discuss
about kicking it out of the architectures we wait for testing migration
at all, or even kicking it out of testing at all. Not waiting for such
an arch has happened and might happen again.

I think it makes sense to shorten the list of arches we wait upon for
testing migration, but I fail to see the usefulness of removing an arch
from testing.

If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)


Cheers,
Andi

[Thiemo Seufer]

Post by Andreas Barth

Post by Mike Fedyk

Post by Andreas Barth
If that happens for a too long period, we might consider such an
architecture to be too slow to keep up, and will eventually discuss
about kicking it out of the architectures we wait for testing migration
at all, or even kicking it out of testing at all. Not waiting for such
an arch has happened and might happen again.

I think it makes sense to shorten the list of arches we wait upon for
testing migration, but I fail to see the usefulness of removing an arch
from testing.

If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)

Removing only the affected packages of that arch instead of the whole
thing looks more sensible. Of course this assumes the core packages
are kept in shape.


Thiemo

[Mike Fedyk]

Post by Andreas Barth

Post by Mike Fedyk

Post by Andreas Barth
If that happens for a too long period, we might consider such an
architecture to be too slow to keep up, and will eventually discuss
about kicking it out of the architectures we wait for testing migration
at all, or even kicking it out of testing at all. Not waiting for such
an arch has happened and might happen again.

I think it makes sense to shorten the list of arches we wait upon for
testing migration, but I fail to see the usefulness of removing an arch
from testing.

If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)

That would be understandable with the intention to release all arches at
the same time. In this case the release should be at a different time
*if* that arch is in a releasable state. Otherwise, it is not released.

The point is that propagating to testing is very useful, and it still
leaves the status of that arch to the porters. With testing there for
SCC arches, it everyone will just point to the porters for that arch if
there is a propagation problem.

Mike

[Andreas Barth]

Post by Mike Fedyk

Post by Andreas Barth
If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)

That would be understandable with the intention to release all arches at
the same time.

I was speaking about release archs (i.e. the archs where the release
team has to keep the strings together). For others archs, things may be
different.


Cheers,
Andi

[Goswin von Brederlow]

Post by Andreas Barth

Post by Mike Fedyk

Post by Andreas Barth
If that happens for a too long period, we might consider such an
architecture to be too slow to keep up, and will eventually discuss
about kicking it out of the architectures we wait for testing migration
at all, or even kicking it out of testing at all. Not waiting for such
an arch has happened and might happen again.

I think it makes sense to shorten the list of arches we wait upon for
testing migration, but I fail to see the usefulness of removing an arch
from testing.

If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)

Not if each arch has it's own source tracking. And you already need
that for snapshot fixes.

Non release archs should be released by the porters alone (no burden
to RMs) with a minimum of arch specific versions or patches. There
should be a strong encouragement to remove software instead of
patching it when it comes close to the actual release so when the port
does release (after the main release) it is based on stable source for
everything but last minute flaws in essential packages. Maintaining
those patches in case of security updates or for the point releases
again should lie with the porters.


The reason why I favour this is that I have in mind that some archs
will be too slow, they won't be able to keep up every day. But given
an extra month they can compile the backlog or kick out out-of-sync
software and release seperately with a nearly identical source
tree. The remaining source changes can (and basically must for
legal reasons) be contained on the binary CD/DVD set and in a branch
of the scc.d.o tree.

Take for example m68k. It will always be at risk of lagging a few days
behind because some packages do build a few days. It is always
out-of-sync longer than the other archs but it is not getting worse,
it is just a step behind. That is totaly different than arm or s390
taking a deep dive getting some 300+ package backlog and having
packages stuck for a month.

If an arch has enough developers on it to keep stuff building, and
that means supplying patches to unstable fast and early enough to get
them into testing and ultimately stable I see no reason why the arch
shouldn't release. Make some rule to limit out-of-sync, say no more
than 1% sources differences to stable for the release.

Any problems with that?

Post by Andreas Barth
Cheers,
Andi
--
http://home.arcor.de/andreas-barth/
PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C

MfG
Goswin

[David Schmitt]

On Thursday 17 March 2005 20:22, Goswin von Brederlow wrote:

[very sensible suggestions removed]

Post by Goswin von Brederlow
Any problems with that?

Not with the procedure in itself. I just want to chip in, that it is (not
only) my opinion, that a REGULAR Debian release cannot allow delaying
security updates and there might be issues caused by delaying builds across
library transitions.

Perhaps there will be a need for defining ALMOSTREGULAR.


Regards, David

[Peter 'p2' De Schrijver]

Post by Goswin von Brederlow

Post by Andreas Barth

Post by Mike Fedyk

Post by Andreas Barth
If that happens for a too long period, we might consider such an
architecture to be too slow to keep up, and will eventually discuss
about kicking it out of the architectures we wait for testing migration
at all, or even kicking it out of testing at all. Not waiting for such
an arch has happened and might happen again.

I think it makes sense to shorten the list of arches we wait upon for
testing migration, but I fail to see the usefulness of removing an arch
from testing.

If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)

Not if each arch has it's own source tracking. And you already need
that for snapshot fixes.
Non release archs should be released by the porters alone (no burden
to RMs) with a minimum of arch specific versions or patches. There
should be a strong encouragement to remove software instead of
patching it when it comes close to the actual release so when the port
does release (after the main release) it is based on stable source for

Why would a port release after the main release ? Why, if debian doesn't
care about the non-release archs, would the porters even bother to
follow the release arch sources and not just release whenever they
like ? They don't gain anything by following the main release.

Post by Goswin von Brederlow
everything but last minute flaws in essential packages. Maintaining
those patches in case of security updates or for the point releases
again should lie with the porters.
The reason why I favour this is that I have in mind that some archs
will be too slow, they won't be able to keep up every day. But given
an extra month they can compile the backlog or kick out out-of-sync
software and release seperately with a nearly identical source
tree. The remaining source changes can (and basically must for
legal reasons) be contained on the binary CD/DVD set and in a branch
of the scc.d.o tree.
Take for example m68k. It will always be at risk of lagging a few days
behind because some packages do build a few days. It is always
out-of-sync longer than the other archs but it is not getting worse,
it is just a step behind. That is totaly different than arm or s390
taking a deep dive getting some 300+ package backlog and having
packages stuck for a month.
If an arch has enough developers on it to keep stuff building, and
that means supplying patches to unstable fast and early enough to get
them into testing and ultimately stable I see no reason why the arch
shouldn't release. Make some rule to limit out-of-sync, say no more
than 1% sources differences to stable for the release.
Any problems with that?

Yes. It doesn't make sense. Either debian releases with all archs, or
every arch releases on its own. The latter is favoured by the current
proposal and will diminish debian's value. The former is the way to go.
Scalability problems need to be solved by improving infrastructure or
procedures as appropriate. A middle ground between both release
approaches is not sensible as it will still make the ports dependent on
a possibly suboptimal upstream tree without having the benefit of debian
security updates. Ie. ports get all the disadvantages of a synchronized
release without getting any of the advantages. That's just plain unfair.

Cheers,

Peter (p2).

[Thiemo Seufer]

Peter 'p2' De Schrijver wrote:
[snip]

Post by Peter 'p2' De Schrijver
Why would a port release after the main release ?

Probably to fix up a few remaining arch-specific bugs.

Post by Peter 'p2' De Schrijver
Why, if debian doesn't
care about the non-release archs, would the porters even bother to
follow the release arch sources and not just release whenever they
like ? They don't gain anything by following the main release.

Except the possibility to profit from the release team's efforts,
and to create an actually supported release. It is not reasonable
to believe a small porter team can do security updates for a
unstable snapshot when a task of similiar size already overloads
the stable security team.

[snip]

Post by Peter 'p2' De Schrijver

Post by Goswin von Brederlow
If an arch has enough developers on it to keep stuff building, and
that means supplying patches to unstable fast and early enough to get
them into testing and ultimately stable I see no reason why the arch
shouldn't release. Make some rule to limit out-of-sync, say no more
than 1% sources differences to stable for the release.

1% is huge when it is glibc. :-)
I think the yes / no decision about a ports release should be finally
done by the security team, because they are the ones to get the work
thrown at. A "no" would mean it can be still distributed along with
the released ports, but it would signal the port's users that they have
to ask the porter team for security updates (which would then lag a bit).

Post by Peter 'p2' De Schrijver

Post by Goswin von Brederlow
Any problems with that?

Yes. It doesn't make sense. Either debian releases with all archs, or
every arch releases on its own. The latter is favoured by the current
proposal and will diminish debian's value. The former is the way to go.
Scalability problems need to be solved by improving infrastructure or
procedures as appropriate. A middle ground between both release
approaches is not sensible as it will still make the ports dependent on
a possibly suboptimal upstream tree without having the benefit of debian
security updates. Ie. ports get all the disadvantages of a synchronized
release without getting any of the advantages. That's just plain unfair.

I believe there _is_ some space between those extremes, if the port's
source is synced enough with the main release.


Thiemo

[Peter 'p2' De Schrijver]

Post by Thiemo Seufer
Except the possibility to profit from the release team's efforts,
and to create an actually supported release. It is not reasonable
to believe a small porter team can do security updates for a
unstable snapshot when a task of similiar size already overloads
the stable security team.

No. As neither the release team, not the security team will take
non-release archs into account.

Cheers,

Peter (p2).

[Goswin von Brederlow]

Post by Peter 'p2' De Schrijver

Post by Thiemo Seufer
Except the possibility to profit from the release team's efforts,
and to create an actually supported release. It is not reasonable
to believe a small porter team can do security updates for a
unstable snapshot when a task of similiar size already overloads
the stable security team.

No. As neither the release team, not the security team will take
non-release archs into account.
Cheers,
Peter (p2).

You can still benefit from their work. The stable sources are
hopefully in sync with each other, the software is known to work.

Most importantly there are source CD/DVD sets out there for it. That
saves a lot of working and server space. I don't think Debian should
support having a complete fork, every port can do that on their own.

As for security stuff probably all fixes can be reused as is by the
porters if they have the sources in sync with stable. They just have
to setup a wanna-build and buildd and they get the fixes quickly after
Debian releases them. The more sources are out of sync the more likely
it is a vulnerable package has a different version which means extra
work.

But I would hope there could be some working together like one porter
joining the security team and including non release archs in the DSAs
_if_ they compile their packages on time.

MfG
Goswin

[David Schmitt]

Post by Peter 'p2' De Schrijver

Post by Goswin von Brederlow

Post by Andreas Barth

Post by Mike Fedyk

Post by Andreas Barth
If that happens for a too long period, we might consider such an
architecture to be too slow to keep up, and will eventually discuss
about kicking it out of the architectures we wait for testing
migration at all, or even kicking it out of testing at all. Not
waiting for such an arch has happened and might happen again.

I think it makes sense to shorten the list of arches we wait upon for
testing migration, but I fail to see the usefulness of removing an
arch from testing.

If we don't wait for an arch, it gets out-of-sync quite soon, and due
to e.g. legal requirements, we can't release that arch. (In other
words, if an arch is too long ignored for testing, we should remove it,
as we can't release it in any case.)

Not if each arch has it's own source tracking. And you already need
that for snapshot fixes.
Non release archs should be released by the porters alone (no burden
to RMs) with a minimum of arch specific versions or patches. There
should be a strong encouragement to remove software instead of
patching it when it comes close to the actual release so when the port
does release (after the main release) it is based on stable source for

Why would a port release after the main release ? Why, if debian doesn't
care about the non-release archs, would the porters even bother to
follow the release arch sources and not just release whenever they
like ? They don't gain anything by following the main release.

Post by Goswin von Brederlow
everything but last minute flaws in essential packages. Maintaining
those patches in case of security updates or for the point releases
again should lie with the porters.
The reason why I favour this is that I have in mind that some archs
will be too slow, they won't be able to keep up every day. But given
an extra month they can compile the backlog or kick out out-of-sync
software and release seperately with a nearly identical source
tree. The remaining source changes can (and basically must for
legal reasons) be contained on the binary CD/DVD set and in a branch
of the scc.d.o tree.
Take for example m68k. It will always be at risk of lagging a few days
behind because some packages do build a few days. It is always
out-of-sync longer than the other archs but it is not getting worse,
it is just a step behind. That is totaly different than arm or s390
taking a deep dive getting some 300+ package backlog and having
packages stuck for a month.
If an arch has enough developers on it to keep stuff building, and
that means supplying patches to unstable fast and early enough to get
them into testing and ultimately stable I see no reason why the arch
shouldn't release. Make some rule to limit out-of-sync, say no more
than 1% sources differences to stable for the release.
Any problems with that?

Yes. It doesn't make sense. Either debian releases with all archs, or
every arch releases on its own. The latter is favoured by the current
proposal and will diminish debian's value. The former is the way to go.
Scalability problems need to be solved by improving infrastructure or
procedures as appropriate.

Porters who have worked on getting an arch to REGUALR status are in a much
better position (demonstrated commitment, technical aptness and
experiencewise) to solve those problems than random-joe-developer.

Always remember that the main reason that it is easier for a porters team to
release within the (current) Debian framework than outside is that _others_
do work for them.




Regards, David

[Peter 'p2' De Schrijver]

Post by David Schmitt
Porters who have worked on getting an arch to REGUALR status are in a much
better position (demonstrated commitment, technical aptness and
experiencewise) to solve those problems than random-joe-developer.

I have no idea what you're trying to say here.

Post by David Schmitt
Always remember that the main reason that it is easier for a porters team to
release within the (current) Debian framework than outside is that _others_
do work for them.

That has been the case up to now, but won't be the case after sarge.

Cheers,

Peter (p2).

[David Schmitt]

Post by Peter 'p2' De Schrijver

Post by David Schmitt
Porters who have worked on getting an arch to REGUALR status are in a
much better position (demonstrated commitment, technical aptness and
experiencewise) to solve those problems than random-joe-developer.

I have no idea what you're trying to say here.

Post by David Schmitt
Always remember that the main reason that it is easier for a porters team
to release within the (current) Debian framework than outside is that
_others_ do work for them.

That has been the case up to now, but won't be the case after sarge.

Indeed that is quite the point I wanted to make with the first sentence.
Please excuse my inability to express that clearly. I will try again:

Up until and including the sarge release, central figures in release
management did most of the work that was needed between a arch-fixed upload
to unstable and a stable release. Reading the Vancouver-proposal, I come to
the conclusion that they recognise that they are not able to make a timely
release (sarge demonstrated that quite clearly).

Therefore they propose certain criteria a arch should fulfil that they are
willing to support a release for the arch. If you try to visualise the
effects of that I hope that this will happen:

1) people realize that $arch won't be REGULAR for etch because the people
working on a release don't want to handhold it through testing and
autobuilding is too slow to properly keep up.

2) people realize that whining and bitching on debian-devel won't convince
the people working on a release that $arch is worth the work

3a) people create $arch-specific-release-mechanism on ports.d.o and learn much
about the pain of keeping $arch in sync with REGULAR development.

3b) people create security-$arch and take care of packages which are affected
by DSAs but are not yet fixed in $arch

4) by going through 3a and 3b people demonstrate commitment and technical
aptness as well as gather experience regarding the release cycle. This makes
them perfect release and security assistants for $arch.


As far as I can tell, Debian on the whole is shortly before ending phase 2 and
I have already seen people work on 3a.

And if anyone dares to object to point 4 that the current release team should
get their act together, because they could "just improve $releae-mechanism"
and release etch with all 15+ arches within the next two years really hasn't
understood how Debian works[1].


If this pace keeps up I am convinced that etch will have more than four
REGULAR arches.


Regards, David

[1] ... and should send in some of the stuff he smokes.

[Wouter Verhelst]

Post by Andreas Barth
If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)

Is that statement based on experience of current behaviour, or rather on
expected behaviour?

(Currently, architectures only get ignored if they're lagging behind
anyway, so then the above statement is obviously true. When
architectures are being ignored for other reasons, the above isn't as
obvious anymore)
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Andreas Barth]

Post by Wouter Verhelst

Post by Andreas Barth
If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)

Is that statement based on experience of current behaviour, or rather on
expected behaviour?

It's based on experience.

Post by Wouter Verhelst
(Currently, architectures only get ignored if they're lagging behind
anyway, so then the above statement is obviously true. When
architectures are being ignored for other reasons, the above isn't as
obvious anymore)

It's probably true that an arch that is always ignored doesn't get out
of sync as bad as an ignored arch now. However, every arch is sometimes
a bit out-of-sync on one or another package, and I doubt that it will
really work. But well, of course this could be tried out if it seems to
be the best solution.


Cheers,
Andi

[Mike Fedyk]

Post by Andreas Barth
If we don't wait for an arch, it gets out-of-sync quite soon, and due to
e.g. legal requirements, we can't release that arch. (In other words, if
an arch is too long ignored for testing, we should remove it, as we
can't release it in any case.)

Why not include diffs of the source for the arches that required porting
before the packages would compile and work properly for that arch to
comply with this legal requirement? This would allow for Tier2 arches
to be released with a later point release or some other schedule.

Mike

[Goswin von Brederlow]

[remove -release, nothing they can do about it]

Post by Steve Langasek

Post by Thomas Bushnell BSG

Post by Goswin von Brederlow
Remember that the buildd queue is not FIFO at all. The queue has a
completly static order. Any changes to the queue are just packages
hiding because they are not "needs-build". I consider that the biggest
flaw of all in wanna-build.

This is news to me.
It means that when one is told "just wait, your package will get
rebuilt"; it is not necessarily true at all. There is no upper bound
at all on time to wait for building, and that's a disaster. People
should stop repeating the fiction then that "just wait" means "your
package will eventually get built".

Er, packages *do* eventually get built; they just don't get built in any

The only way to get the last package in the queue (zvbi or something)
build is when the queue is empty. Unfortunately some archs are on the
border of being too slow which means the time between empty queues is
long.

The effect of being (too) slow is that some few package will not be
build for weeks/month instead of all packages being build a few days
late as a simple FIFO would do.

Post by Steve Langasek
kind of FIFO order. I don't particularly care for this arrangement myself
(it means there are plenty of times that a high-priority bug in a
low-priority package stays on the release team's watchlist for far too
long), but I don't have any proof that a different queue ordering would
actually work better for the project, and the buildd admins *are* committed
to keeping up with the queue even though hardware circumstances sometimes
prevent it from time to time.
--
Steve Langasek
postmodern programmer

Check
Loading Image...

Queue not empty since:

arm Feb 18th
mipsel Feb 26th
s390 Mar 6th
mips Mar 10th (possibly)

That means (just an example) that an upload of zsh on Feb. 18th didn't
get build at all on arm while the 5 uploads of ash all got build.

Does that sound fair? Just because ash starts with an 'a' it gets
prefered treatment?


I see the point of sorting by priority to get the essentials build
with priority in case of backlogs.

I don't see a point in sorting by section as base is already done
through priority and an automatic Dep-Wait would get libs build
first.

And last I feel the sorting by name is actualy harmfull. That should
be exchanged with the time of upload, i.e. FIFO if the rest matches.
We all know FIFO isn't the best but it is simple, fair, predictable
and does not starve. I think it would avoid a lot of those "Why am I
stuck in the buildd queue?" questions.

MfG
Goswin

[Torsten Landschoff]

Post by Goswin von Brederlow
And last I feel the sorting by name is actualy harmfull. That should
be exchanged with the time of upload, i.e. FIFO if the rest matches.
We all know FIFO isn't the best but it is simple, fair, predictable
and does not starve. I think it would avoid a lot of those "Why am I
stuck in the buildd queue?" questions.

I second that.

Greetings

Torsten

[Wouter Verhelst]

Post by Thomas Bushnell BSG

Post by Goswin von Brederlow
Remember that the buildd queue is not FIFO at all. The queue has a
completly static order. Any changes to the queue are just packages
hiding because they are not "needs-build". I consider that the biggest
flaw of all in wanna-build.

This is news to me.
It means that when one is told "just wait, your package will get
rebuilt"; it is not necessarily true at all. There is no upper bound
at all on time to wait for building, and that's a disaster.

This paragraph assumes nobody ever looks the length of the needs-build
queue of his architecture, and nobody feels bad when the queue is longer
than normal. That idea would be hilarious if it wasn't sad.

In reality, the upper bound is determined by motivated porters who try
hard to avoid the queue ever to grow too long, day after day.

Post by Thomas Bushnell BSG
People
should stop repeating the fiction then that "just wait" means "your
package will eventually get built".

Why, if it is true?
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Thomas Bushnell BSG]

Post by Wouter Verhelst

Post by Thomas Bushnell BSG
It means that when one is told "just wait, your package will get
rebuilt"; it is not necessarily true at all. There is no upper bound
at all on time to wait for building, and that's a disaster.

This paragraph assumes nobody ever looks the length of the needs-build
queue of his architecture, and nobody feels bad when the queue is longer
than normal. That idea would be hilarious if it wasn't sad.

There is no need-build queue; please don't call it that because it
causes people to misunderstand it. "Queue" means "FIFO", at least to
some degree, where needs-build isn't FIFO in any degree.

I have plenty of evidence that the buildd maintainers look at the
length and feel bad when it's long. But looking at it and feeling bad
aren't sufficient to get a package built.

Post by Wouter Verhelst
In reality, the upper bound is determined by motivated porters who try
hard to avoid the queue ever to grow too long, day after day.

I have no doubt about their good intentions and trying hard.

Thomas

[Goswin von Brederlow]

Post by Wouter Verhelst

Post by Thomas Bushnell BSG

Post by Goswin von Brederlow
Remember that the buildd queue is not FIFO at all. The queue has a
completly static order. Any changes to the queue are just packages
hiding because they are not "needs-build". I consider that the biggest
flaw of all in wanna-build.

This is news to me.
It means that when one is told "just wait, your package will get
rebuilt"; it is not necessarily true at all. There is no upper bound
at all on time to wait for building, and that's a disaster.

This paragraph assumes nobody ever looks the length of the needs-build
queue of his architecture, and nobody feels bad when the queue is longer
than normal. That idea would be hilarious if it wasn't sad.
In reality, the upper bound is determined by motivated porters who try
hard to avoid the queue ever to grow too long, day after day.

The queue length is bound and easily detected if it grows too long.

But do you notice when the same packages keep showing up at the end of
the queue for weeks? The queue can be as small as 1 package inbetween
and that 1 package could still never get build.

Eventualy someone notices, usualy the maintainer, and a new "why isn't
my package build yet?" flame starts.

Post by Wouter Verhelst

Post by Thomas Bushnell BSG
People
should stop repeating the fiction then that "just wait" means "your
package will eventually get built".

Why, if it is true?

It usualy is. It might not be. And it can be an awfully long wait.

The last one is the problem. The first two not.

MfG
Goswin

[Thijs Kinkhorst]

Post by Goswin von Brederlow

Post by Steve Langasek
People
should stop repeating the fiction then that "just wait" means "your
package will eventually get built".

It usualy is. It might not be. And it can be an awfully long wait.
The last one is the problem. The first two not.

This could easily be prevented: the priority of a package could be
increased if it is longer than T in the queue (or if it's really long even
give it top priority over any other package). You would get the situation
that higher priority packages get built first, but from time to time a
lower-but-long-waiting package gets built to prevent starvation. Seems
like a fair situation to me.


Thijs

[Kalle Kivimaa]

[debian-release dropped]

Post by Goswin von Brederlow
But do you notice when the same packages keep showing up at the end of
the queue for weeks? The queue can be as small as 1 package inbetween
and that 1 package could still never get build.

Just out of curiosity, what is the record time for any package on any
arch to have been on the list (or, what is the longest ever time any
arch has had a non-empty list)?

[Andreas Barth]

Post by Matthew Palmer

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I'm trying to work out why package *section* matters at all. Package name
is a bit odd, too, but including the section in there is just totally whack.

Because we want packages in base to be preferred, as well as packages in
libs.


Cheers,
Andi

[Matthew Palmer]

Post by Andreas Barth

Post by Matthew Palmer

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I'm trying to work out why package *section* matters at all. Package name
is a bit odd, too, but including the section in there is just totally whack.

Because we want packages in base to be preferred, as well as packages in
libs.

I think I slightly misunderstood the "ordering by section" bit -- I was
assuming an alphabetical ordering by section. So once base and libs have
had their priority building, what's the ordering after that? Alphabetical
by section, or does it go straight into a alphabetical by package name?

- Matt

[Kurt Roeckx]

Post by Matthew Palmer

Post by Andreas Barth
Because we want packages in base to be preferred, as well as packages in
libs.

I think I slightly misunderstood the "ordering by section" bit -- I was
assuming an alphabetical ordering by section. So once base and libs have
had their priority building, what's the ordering after that? Alphabetical
by section, or does it go straight into a alphabetical by package name?

There is a list in wanna-build for each section.

The start of it looks like:
libs
debian-installer
base
devel
shells
perl
python


If you want to full list, look at the source.


Kurt

[Andreas Barth]

Post by Matthew Palmer
I think I slightly misunderstood the "ordering by section" bit -- I was
assuming an alphabetical ordering by section. So once base and libs have
had their priority building, what's the ordering after that? Alphabetical
by section, or does it go straight into a alphabetical by package name?

It is a highly ordered list, more or less libs+base first, than devel, shells,
perl, python. After that graphics, admin, utils. Just to look at the
other side of the sorting order, at the end of the list is hamradio,
non-US and embedded. The large ones like gnome and kde are in the middle
of the list. Please see wanna-builds source for the full list.


Cheers,
Andi

[Hamish Moffatt]

Post by Andreas Barth
It is a highly ordered list, more or less libs+base first, than devel, shells,
perl, python. After that graphics, admin, utils. Just to look at the
other side of the sorting order, at the end of the list is hamradio,
non-US and embedded. The large ones like gnome and kde are in the middle
of the list. Please see wanna-builds source for the full list.

Given how low hamradio (and the like) are prioritised, I suggest that we
get smarter about 'tesing' and omit some sections on some architectures.

Frankly there are not likely to be any users for hamradio on s390, mips*
arm, or m68k. Nor electronics. Instead those architectures just prevent
the migration to testing for those packages, for no good reason.

I'm in favour of building all packages on all architectures (when time
permits), and FTBFS bugs should be release-critical. But do we need to
actually make those packages critical for release? For zero users?

Having half your packages prioritised last in the build queue is rather
insulting to be honest..

Hamish

[Thomas Bushnell BSG]

Post by Hamish Moffatt
Given how low hamradio (and the like) are prioritised, I suggest that we
get smarter about 'tesing' and omit some sections on some architectures.

I don't think those sections are causing the problem. There are also
not so many uploads for them. The problem from my perspective is that
"optional" packages all have to be built before any "extra" packages
get considered. That's what's hosing gnucash.

Removing a few sections also wouldn't help the perverse incentives
that the current system offers.

Thomas

[Hamish Moffatt]

Post by Thomas Bushnell BSG

Post by Hamish Moffatt
Given how low hamradio (and the like) are prioritised, I suggest that we
get smarter about 'tesing' and omit some sections on some architectures.

I don't think those sections are causing the problem. There are also
not so many uploads for them. The problem from my perspective is that

True, but that isn't making my electronics packages build any faster.

Post by Thomas Bushnell BSG
"optional" packages all have to be built before any "extra" packages
get considered. That's what's hosing gnucash.

hamradio is only one notch above "extra" in the pecking order anyway.

I'm very pleased to hear this will be less important for etch anyway.

Hamish

[Ingo Juergensmann]

Post by Hamish Moffatt
Given how low hamradio (and the like) are prioritised, I suggest that we
get smarter about 'tesing' and omit some sections on some architectures.
Frankly there are not likely to be any users for hamradio on s390, mips*
arm, or m68k. Nor electronics. Instead those architectures just prevent
the migration to testing for those packages, for no good reason.

How can archs prevent the migration when the software is already uptodate,
f.e. ax25-tools?
http://unstable.buildd.net/cgi/package_status?unstable_all_pkg=ax25-tools&searchtype=go

Post by Hamish Moffatt
I'm in favour of building all packages on all architectures (when time
permits), and FTBFS bugs should be release-critical. But do we need to
actually make those packages critical for release? For zero users?

Let the user decide.
But a user can only use a package, when it's there. Hence the name user...

Post by Hamish Moffatt
Having half your packages prioritised last in the build queue is rather
insulting to be honest..

Instead of considering dropping archs or excluding archs from building, you
should consider improving the buildd process. The current wanna-build is
known to have many drawbacks. It's an ancient program that doesn't fit any
longer on todays needs. Patching it to death doesn't help much, imho.

http://www.buildd.net/files/Multibuild-Draft.pdf

[Hamish Moffatt]

Post by Ingo Juergensmann

Post by Hamish Moffatt
Given how low hamradio (and the like) are prioritised, I suggest that we
get smarter about 'tesing' and omit some sections on some architectures.
Frankly there are not likely to be any users for hamradio on s390, mips*
arm, or m68k. Nor electronics. Instead those architectures just prevent
the migration to testing for those packages, for no good reason.

How can archs prevent the migration when the software is already uptodate,
f.e. ax25-tools?
http://unstable.buildd.net/cgi/package_status?unstable_all_pkg=ax25-tools&searchtype=go

Yes. So I haven't uploaded any of those lately. I have one ready but
don't want to confuse the issue for sarge.

How about geda-gschem? Waiting on arm for a couple of weeks now.
Holding up migration of all of geda* on all architectures.

I couldn't work out where wanna-build CVS is hosted so I couldn't
actually check the order to see where electronics fits in.

Post by Ingo Juergensmann
Instead of considering dropping archs or excluding archs from building, you
should consider improving the buildd process. The current wanna-build is
known to have many drawbacks. It's an ancient program that doesn't fit any
longer on todays needs. Patching it to death doesn't help much, imho.

But right now, arm (for example) just cannot build all the packages that
need building. Changing the order won't help. Only adding machines or
removing packages will help.


Hamish

[Ingo Juergensmann]

Post by Hamish Moffatt
How about geda-gschem? Waiting on arm for a couple of weeks now.
Holding up migration of all of geda* on all architectures.
I couldn't work out where wanna-build CVS is hosted so I couldn't
actually check the order to see where electronics fits in.

Yes, it's not easy to find and sometimes it migrates from host to host
even... Maybe it's at newraff or at db.d.o?

Post by Hamish Moffatt

Post by Ingo Juergensmann
Instead of considering dropping archs or excluding archs from building, you
should consider improving the buildd process. The current wanna-build is
known to have many drawbacks. It's an ancient program that doesn't fit any
longer on todays needs. Patching it to death doesn't help much, imho.

But right now, arm (for example) just cannot build all the packages that
need building. Changing the order won't help. Only adding machines or
removing packages will help.

Yes, adding machines will help, but this becomes a problem, when the
in-person union of wanna-build and buildd admin for that arch don't want you
to play in his Sandkasten... That's a human problem, not a hardware or arch
specific problem.

[Andreas Barth]

Post by Hamish Moffatt

Post by Andreas Barth
It is a highly ordered list, more or less libs+base first, than devel, shells,
perl, python. After that graphics, admin, utils. Just to look at the
other side of the sorting order, at the end of the list is hamradio,
non-US and embedded. The large ones like gnome and kde are in the middle
of the list. Please see wanna-builds source for the full list.

Given how low hamradio (and the like) are prioritised, I suggest that we
get smarter about 'tesing' and omit some sections on some architectures.

We won't omit some sections on some architectures, but all sections on
some architectures :)


Cheers,
Andi

[Hamish Moffatt]

Post by Andreas Barth

Post by Hamish Moffatt
Given how low hamradio (and the like) are prioritised, I suggest that we
get smarter about 'tesing' and omit some sections on some architectures.

We won't omit some sections on some architectures, but all sections on
some architectures :)

That's a solution too!

Well done to the release and ftpmaster teams for the plan for etch.
You made some tough and possibly unpopular decisions, but I think
they were the right decisions for the project.

Hamish

[Thomas Bushnell BSG]

Post by Andreas Barth
Because we want packages in base to be preferred, as well as packages in
libs.

I think that is a given, but it's not uploads to base and libs that
are hosing the recompilation of gnucash at present.

I think it's worth looking at the perverse incentives that the current
system offers. (A perverse incentive is one that rewards people for
doing the wrong thing.)

My top release priority is getting my own packages in shape, which
means closing all release critical bugs and fixing all the important
bugs which can be. Gnucash in stable and testing currently has a very
serious RC bug which can cause massive data loss, *in an accounting
application* where such data loss is all the more serious.

My second release priority is doing what I can to fix RC bugs in other
packages, and clean up and monitor the QA packages as best as I can.

But I will not be fixing any RC bugs in other packages or making any
QA uploads, because nearly every such package comes ahead of gnucash
in the list, and not because they are base and libs, but because they
are optional and gnucash has the wrong priority (extra). Any work I
do on my second release priority will delay my top release priority.
I believe that taking care of my own packages' bugs should be my top
priority--as it should be for every DD--and if I do any uploads of
other packages, it will delay that first priority.

So the current system is creating a perverse incentive for me to sit
on my hands, and only fix bugs in other packages once gnucash has
*finally* gotten rebuilt, which may well be three months from the date
the bug was fixed.

The bug was reported January 21; I confirmed the bug, implemented the
fix, and uploaded the fixed package the same day. This, it seems to
me, is what should happen for such a dangerous bug. It is now nearly
two months later, and the fix still isn't in testing. And I will do
nothing to further hose gnucash users by delaying more the bug's entry
into sarge.

In the past two days, gnucash has slipped from being 90th on s390 to
being 189th, and has moved essentially not at all on arm and mipsel.
It may well be another month before it actually gets rebuilt. Any
upload I do for any other package, any bug fix I suggest which some
other maintainer uploads, further hoses my primary responsibility.

I want a system where I can fix bugs in other packages while I'm
waiting, and upload them, *without* it hosing my primary
responsibility.

Thomas

[Wouter Verhelst]

Post by Matthew Palmer
I'm trying to work out why package *section* matters at all.

This is simply an attempt to avoid as much
needs-build->building->dep-wait cycles as possible; packages that are
usually build-dependencies are built before packages that are usually
build-depending.
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Matthias Urlichs]

Post by Matthew Palmer
I think the queue
needs to be as FIFO as possible for fairness and "principle of least
surprise" sake.

See my patch on d-d (also mailed to the ftpmasters), which inserts "age in
queue" (actually, timestamp of last status change, but that's
more-or-less equivalent) right before "name" in the sort order.

[Wouter Verhelst]

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite

- previous compilation state (already built packages are prioritized
above packages never built for the target architecture)

Post by Steve Langasek
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I agree with the w-b maintainers. The queue order is only interesting in
the case where there is a backlog; in other cases, packages are usually
built rather fast. In the case where there is a backlog, those trying to
fix the architecture (usually those that are working on it) should be in
charge of deciding what package gets built first, not the maintainer of
a random package -- /all/ package builds are urgent if there's a
backlog.
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Thomas Bushnell BSG]

Post by Wouter Verhelst
I agree with the w-b maintainers. The queue order is only interesting in
the case where there is a backlog; in other cases, packages are usually
built rather fast. In the case where there is a backlog, those trying to
fix the architecture (usually those that are working on it) should be in
charge of deciding what package gets built first, not the maintainer of
a random package -- /all/ package builds are urgent if there's a
backlog.

First, there is no queue. It's a list, but not a queue.

I agree that we need policy here, but a serious problem is that
currently when there is a backlog, maintainers of penalized packages
have a perverse incentive not to fix any bugs in any *other* package
because it will cause the penalty on their package to continue.

[Steve Langasek]

Post by Wouter Verhelst

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite

- previous compilation state (already built packages are prioritized
above packages never built for the target architecture)

Yep, remembered that one after I sent the message.

Post by Wouter Verhelst

Post by Steve Langasek
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I agree with the w-b maintainers. The queue order is only interesting in
the case where there is a backlog; in other cases, packages are usually
built rather fast. In the case where there is a backlog, those trying to
fix the architecture (usually those that are working on it) should be in
charge of deciding what package gets built first, not the maintainer of
a random package -- /all/ package builds are urgent if there's a
backlog.

Well, my objection is basically the same as Thomas's here -- all package
builds are *not* equally urgent, and in fact, we have an "urgency" field
in uploads that expresses this fact quite clearly. Certainly there's
some danger of abuse by uploaders, but there are dozens of other things
that maintainers *could* abuse right now and are only stopped from doing
because they *shouldn't* do them.

I wouldn't be bothered by porters choosing how to order builds, if the
ordering they chose more closely corresponded to what the release team
(and britney) want it to be. :)

[Wouter Verhelst]

Post by Steve Langasek
Well, my objection is basically the same as Thomas's here -- all package
builds are *not* equally urgent,

Of course not, that is exactly my point.

But from the POV of a package's maintainer, all fixes are more or less
urgent. If some fixes weren't necessary, the upload wouldn't have been
there in the first place.

Post by Steve Langasek
and in fact, we have an "urgency" field
in uploads that expresses this fact quite clearly. Certainly there's
some danger of abuse by uploaders, but there are dozens of other things
that maintainers *could* abuse right now and are only stopped from doing
because they *shouldn't* do them.
I wouldn't be bothered by porters choosing how to order builds, if the
ordering they chose more closely corresponded to what the release team
(and britney) want it to be. :)

I from my side wouldn't mind if someone from the release team would ask
me to prioritize a build[1] if necessary; but I feel irky at the thought
of allowing other people to prioritize their packages' builds over
others, because that *will* make sure that eventually, those people that
do what is actually the right thing will have to wait for all eternity
for their packages to be built.

[1] this is technically possible, but only in a kindof hackish way, by
manually adding the package to a buildd's REDO file and (also manually)
setting it to 'building' by that host.
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Steve Langasek]

Post by Wouter Verhelst

Post by Steve Langasek
Well, my objection is basically the same as Thomas's here -- all package
builds are *not* equally urgent,

Of course not, that is exactly my point.
But from the POV of a package's maintainer, all fixes are more or less
urgent. If some fixes weren't necessary, the upload wouldn't have been
there in the first place.

I find this line of reasoning fairly incomprehensible; the "more" or "less"
aspect of the urgency is relevant here. We obviously have a system for
classifying the severity of bugs in packages, and it's possible to relate
these bug severities to the urgency field in uploads; even assuming it does
get abused by maintainers, how would considering urgency for package build
ordering be worse than what we have now given that it should only be an
issue in either case when the buildds are not working the way they should?

Post by Wouter Verhelst

Post by Steve Langasek
and in fact, we have an "urgency" field
in uploads that expresses this fact quite clearly. Certainly there's
some danger of abuse by uploaders, but there are dozens of other things
that maintainers *could* abuse right now and are only stopped from doing
because they *shouldn't* do them.
I wouldn't be bothered by porters choosing how to order builds, if the
ordering they chose more closely corresponded to what the release team
(and britney) want it to be. :)

I from my side wouldn't mind if someone from the release team would ask
me to prioritize a build[1] if necessary; but I feel irky at the thought
of allowing other people to prioritize their packages' builds over
others, because that *will* make sure that eventually, those people that
do what is actually the right thing will have to wait for all eternity
for their packages to be built.
[1] this is technically possible, but only in a kindof hackish way, by
manually adding the package to a buildd's REDO file and (also manually)
setting it to 'building' by that host.

Yes, I don't think it scales very well to either have the release team
asking for this, or for the buildd maintainers to be fielding such manual
requests. If anything, the current workaround options (ignoring select
out-of-date binaries for an arch) seem more efficient.

[Thomas Bushnell BSG]

I don't think the possibility of something like that being abused is as
strange as you seem to imply. As proof of that statement, I faintly
remember someone doing a gratuitous source upload just to provoke the
buildds...

Of course, there was no abuse here about priority; the package was
incorrectly in "extra".

[Wouter Verhelst]

the "more" or "less" aspect of the urgency is relevant here. We
obviously have a system for classifying the severity of bugs in
packages, and it's possible to relate these bug severities to the
urgency field in uploads; even assuming it does get abused by
maintainers,

I don't think the possibility of something like that being abused is as
strange as you seem to imply. As proof of that statement, I faintly
remember someone doing a gratuitous source upload just to provoke the
buildds...

how would considering urgency for package build ordering be worse than
what we have now given that it should only be an issue in either case
when the buildds are not working the way they should?

It would be worse in that it would increase the impact of a re-upload.
Not only would it trigger a rebuild on all architectures, it would now
suddenly also throw the build ordering around, possibly worsening the
problem that prompted the gratuitous upload in the first place by not
building urgent (in build-dependency order) packages first.

Post by Wouter Verhelst
[1] this is technically possible, but only in a kindof hackish way, by
manually adding the package to a buildd's REDO file and (also manually)
setting it to 'building' by that host.

Yes, I don't think it scales very well to either have the release team
asking for this, or for the buildd maintainers to be fielding such manual
requests. If anything, the current workaround options (ignoring select
out-of-date binaries for an arch) seem more efficient.

Whatever you say; I don't mind either way. The offer still stands :)
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Steve Langasek]

the "more" or "less" aspect of the urgency is relevant here. We
obviously have a system for classifying the severity of bugs in
packages, and it's possible to relate these bug severities to the
urgency field in uploads; even assuming it does get abused by
maintainers,

I don't think the possibility of something like that being abused is as
strange as you seem to imply. As proof of that statement, I faintly
remember someone doing a gratuitous source upload just to provoke the
buildds...

On the contrary, I do expect there would be a certain amount of abuse of
such a system -- I just can't imagine such abuse reaching a level where it
constitutes a worse failure scenario than the one we currently have.
*Particularly* if we apply appropriate social pressures against abusers.

how would considering urgency for package build ordering be worse than
what we have now given that it should only be an issue in either case
when the buildds are not working the way they should?

It would be worse in that it would increase the impact of a re-upload.
Not only would it trigger a rebuild on all architectures, it would now
suddenly also throw the build ordering around, possibly worsening the
problem that prompted the gratuitous upload in the first place by not
building urgent (in build-dependency order) packages first.

But, uploads impact the ordering of Needs-Build all the time; I don't see
why that would generally be any worse than the status quo.

[Goswin von Brederlow]

Post by Wouter Verhelst

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite

- previous compilation state (already built packages are prioritized
above packages never built for the target architecture)

Post by Steve Langasek
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I agree with the w-b maintainers. The queue order is only interesting in
the case where there is a backlog; in other cases, packages are usually
built rather fast. In the case where there is a backlog, those trying to
fix the architecture (usually those that are working on it) should be in
charge of deciding what package gets built first, not the maintainer of
a random package -- /all/ package builds are urgent if there's a
backlog.

Since you think an empty queue is normal why then fight changes to the
queue order? If it is empty most of the time as you say the specific
order hardly matters. Packages will be build within 15 minutes of
their upload no matter what order as they will be the only packae in
the queue.

As you say, the oder is only relevant when there is a backlog and that
is currently a badly starving algorithm.

The problem is that a backlog is more and more the normal day to day
business while an empty queue is rare. Obviously not for every arch
but always some archs.

MfG
Goswin

[Wouter Verhelst]

Post by Goswin von Brederlow

Post by Wouter Verhelst

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite

- previous compilation state (already built packages are prioritized
above packages never built for the target architecture)

Post by Steve Langasek
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I agree with the w-b maintainers. The queue order is only interesting in
the case where there is a backlog; in other cases, packages are usually
built rather fast. In the case where there is a backlog, those trying to
fix the architecture (usually those that are working on it) should be in
charge of deciding what package gets built first, not the maintainer of
a random package -- /all/ package builds are urgent if there's a
backlog.

Since you think an empty queue is normal why then fight changes to the
queue order?

You misunderstood. I don't fight generic changes to the order; I just
don't think it would be a good thing that any random developer could
prioritize his pet package.
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Goswin von Brederlow]

Post by Wouter Verhelst

Post by Goswin von Brederlow

Post by Wouter Verhelst

Post by Steve Langasek
The queue ordering is entirely automatic, and AIUI the queue(s) is (are)
- target suite

- previous compilation state (already built packages are prioritized
above packages never built for the target architecture)

Post by Steve Langasek
- package priority
- package section
- package name
I personally believe it would be beneficial to prioritize by upload urgency
as well (probably as a sort criterion between package priority and package
section), but the w-b maintainers disagree.

I agree with the w-b maintainers. The queue order is only interesting in
the case where there is a backlog; in other cases, packages are usually
built rather fast. In the case where there is a backlog, those trying to
fix the architecture (usually those that are working on it) should be in
charge of deciding what package gets built first, not the maintainer of
a random package -- /all/ package builds are urgent if there's a
backlog.

Since you think an empty queue is normal why then fight changes to the
queue order?

You misunderstood. I don't fight generic changes to the order; I just
don't think it would be a good thing that any random developer could
prioritize his pet package.

My suggestion is to use an aging algorithm with "time * alpha +
beta". Aspects of a source would then affect alpha and beta.

Example:

- Essential: yes adds 100 to beta and 5 to alpha.
- Urgency: critical adds 10 to beta and 1 to alpha

The package with the highest score gets build first. Alpha makes a
package advance the queue faster overtaking other packages while beta
makes packages start further up in the queue.

By adjusting the changes to alpha and beta each aspect of a source can
be finely tuned to have some affect on the queue. So random developer
can influence the priority by setting the urgency but not so much as
to abuse the power and deadlock other packages.


Instead of the urgency of uploads the number of bugs (weighted by
priority) could be used instead or on top of that. Or the number of
depends, revers depends, dep-waits, .... A lot of aspects could be
added up to set alpha and beta for each package.

Even alpha=1, beta=-<current static queue position> and time in
minutes would be an improvement. That would mean that after 6 days any
package would be build before a fresh upload of the most essential
package.

MfG
Goswin

[Anthony DeRobertis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Wouter Verhelst wrote:

| You misunderstood. I don't fight generic changes to the order; I just
| don't think it would be a good thing that any random developer could
| prioritize his pet package.
|

Any random developer already has root on X thousand debian systems. We
can trust him with that, but not with helping determine build order?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCN1FE+z+IwlXqWf4RArSmAJwLA1aiNaTQtVzgXWYcua2061CsbACfUTnG
Ye6KWZ+xsscs+7VRHRTL8Dw=
=kHlW
-----END PGP SIGNATURE-----

[Wouter Verhelst]

Post by Anthony DeRobertis
| You misunderstood. I don't fight generic changes to the order; I just
| don't think it would be a good thing that any random developer could
| prioritize his pet package.
|
Any random developer already has root on X thousand debian systems. We
can trust him with that, but not with helping determine build order?

I'm afraid it won't actually be 'helping'.

I've seen enough of "My package isn't built and it's urgent and the
buildd admins suck!" to expect what would happen.

That's not to say that a request to prioritize a package is to be
ignored; however, the power of deciding which packages get built first
should be with those that actually build the packages, rather than with
those who want their packages to be built. The former are expected to be
following the larger picture; the latter are not.
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Christian Hammers]

Hello Wouter

Post by Wouter Verhelst
That's not to say that a request to prioritize a package is to be
ignored; however, the power of deciding which packages get built first
should be with those that actually build the packages, rather than with
those who want their packages to be built. The former are expected to be
following the larger picture; the latter are not.

Given that the only really relevant thing is when the package enters
testing, which already can be changed by the maintainers, the only thing
a maintainer can wish is to have his package build within the 2 or 5 days,
or? As only 2 days might be a problem, wouldn't just prefering high+security
uploads be enough to make everybody happy without disrupting the buildd
order too much?

friendly,

-christian-

[Anthony DeRobertis]

Post by Wouter Verhelst
That's not to say that a request to prioritize a package is to be
ignored; however, the power of deciding which packages get built first
should be with those that actually build the packages, rather than with
those who want their packages to be built. The former are expected to be
following the larger picture; the latter are not.

The latter, however, also know a good deal more about the particular
details of the upload. It'd seem to at least make sense to use
critical/medium/low as part of the sort criteria, for example before
alphabetical. Maybe even ahead of standard/optional/extra.

[Martijn van Oosterhout]

Post by Wouter Verhelst

Post by Anthony DeRobertis
| You misunderstood. I don't fight generic changes to the order; I just
| don't think it would be a good thing that any random developer could
| prioritize his pet package.
|
Any random developer already has root on X thousand debian systems. We
can trust him with that, but not with helping determine build order?

I'm afraid it won't actually be 'helping'.
I've seen enough of "My package isn't built and it's urgent and the
buildd admins suck!" to expect what would happen.

Aah, so the trick should be to implement this on the quiet. Tweak the
values according to have it's progressing. Eventually you should see a
reduction in complaints. It wouldn't be hard to be able to guarentee
an upper bound for building, say two months. Once people stop
complaining you can tell them. Maybe they won't feel so inclined to
fiddle when they can see it works.

Post by Wouter Verhelst
That's not to say that a request to prioritize a package is to be
ignored; however, the power of deciding which packages get built first
should be with those that actually build the packages, rather than with
those who want their packages to be built. The former are expected to be
following the larger picture; the latter are not.

As far as I can tell, buildd admins don't spend all day prioritising
builds manually anyway. We're dealing with computers here, they don't
care how complicated the calculations are. Decide where your
priorities are and implement it. If you don't like to hear people
complain, add a rule saying "if package is older then one month, build
it now". It cost you nothing and saves heaps of grief. As a bonus,
repeatedly uploading new versions would keep resetting the counter, so
they'd be encouraged to upload less.

The current process of indefinitly starving "extra" packages is just silly.

[Marc Haber]

Post by Thomas Bushnell BSG
Unfortunately, the queue ordering policy is unclear. I was guessing
that the priority of the upload would have something to do with
queueing policy.
Since the all but one of the other arch buildd's have empty
needs-build queues, it is harmless to force them to execute a
recompile and costs no scarce resources. I did check this before
uploading.

Did you also check how much network traffic would be wasted by rolling
out the unnecessary package to all mirrors?

Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834
Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835

[Andreas Barth]

Post by Thomas Bushnell BSG
Since the all but one of the other arch buildd's have empty
needs-build queues, it is harmless to force them to execute a
recompile and costs no scarce resources. I did check this before
uploading.

Even in the case the buildds otherwise idleing, it costs time of the
buildd admin to check the log and sign the upload.

Post by Thomas Bushnell BSG
If, perhaps, there was a clear indication of the buildd ordering
policy, then it could be properly used. Until then, I go on the basis
of guesswork.

The ordering applied is (in this order, first difference wins)
- >= standard
- not uncompiled
- priority of the package
- priority of the section
- name

That all is BTW visible from the code (or you could just ask).


Cheers,
Andi

[Andreas Barth]

Post by Andreas Barth

Post by Thomas Bushnell BSG
If, perhaps, there was a clear indication of the buildd ordering
policy, then it could be properly used. Until then, I go on the basis
of guesswork.

The ordering applied is (in this order, first difference wins)
- >= standard
- not uncompiled
- priority of the package
- priority of the section
- name

Which goes in front of that ordering is that all buildds check
wanna-build in the order of the priority of the suites, i.e. security
uploads first, than stable, ..., but only take packages not in noauto,
and, if all queues are empty, also take up packages in noauto_weak in
the same order (both are individual settings at each buildd).


Cheers,
Andi

[Wouter Verhelst]

Post by Thomas Bushnell BSG

Post by Steve Langasek
Re-uploading a package to provoke a buildd response is counterproductive,
*particularly* when the package is already in Needs-Build on the missing
architectures. Re-uploading doesn't change its position in the queue, but
it *does* force buildds for all the other archs to needlessly rebuild the
package. This is why the answer to your previous email was "please be
patient".

Unfortunately, the queue ordering policy is unclear. I was guessing
that the priority of the upload would have something to do with
queueing policy.

As is explained on
<http://www.debian.org/devel/buildd/wanna-build-states>, this is not the
case.

Post by Thomas Bushnell BSG
Since the all but one of the other arch buildd's have empty
needs-build queues, it is harmless to force them to execute a
recompile and costs no scarce resources.

If everyone would reason like that, then it will cost scarce resources.
This is a bogus argument.

Post by Thomas Bushnell BSG
I did check this before
uploading.
I made an upload because a related package (grisbi) just seemed to get
compiled by all the buildd's in a nifty two-day round trip time.

You were lucky that time around, then.

Post by Thomas Bushnell BSG
It
was uploaded March 10, compiled by most buildds on the 10th, and by
arm and mipsel on the 11th. I concluded that the queue must take note
of priority or something like that.

It does not.

[...]

Post by Thomas Bushnell BSG
If, perhaps, there was a clear indication of the buildd ordering
policy, then it could be properly used. Until then, I go on the basis
of guesswork.

That indication is there, and it mainly boils down to 'buildd builds
packages in a more or less predefined order which a maintainer has no
direct influence on'. Of course, we can massage the ordering if
required, but that is only done in exceptional cases.

If your problem is 'my package will not migrate to testing!', then you
are wrong, too. There are precedents for release managers forcibly
moving packages to testing, even if the architectures are not in sync;
there are precedents for an architecture with a huge backlog being
temporarily ignored for the testing migration.

In any case, re-uploading to force a package rebuild is /always/ the bad
thing to do.
--
EARTH
smog | bricks
AIR -- mud -- FIRE
soda water | tequila
WATER
-- with thanks to fortune

[Steve Langasek]

[please don't cc: me on this thread, one copy is plenty, thanks; and
please don't cc: debian-release unless there's a specific reason it's
on-topic there, which explaining wanna-build is not. ;)]

Post by Wouter Verhelst
[...]

Post by Thomas Bushnell BSG
If, perhaps, there was a clear indication of the buildd ordering
policy, then it could be properly used. Until then, I go on the basis
of guesswork.

That indication is there, and it mainly boils down to 'buildd builds
packages in a more or less predefined order which a maintainer has no
direct influence on'. Of course, we can massage the ordering if
required, but that is only done in exceptional cases.
If your problem is 'my package will not migrate to testing!', then you
are wrong, too. There are precedents for release managers forcibly
moving packages to testing, even if the architectures are not in sync;
there are precedents for an architecture with a huge backlog being
temporarily ignored for the testing migration.

Yes, though we're generally avoiding pushing packages into testing right
now because it's not clear that the t-p-u queue is picking up
out-of-date packages for building, particularly for archs that are
currently hardware strapped; so waiting on the package to build for all
archs in unstable may be the *quickest* way to get the package in sync
in testing. It's an ugly trade-off, but I think we've erred on the side
of caution for long enough and will probably be more aggressive with
buildd-stalled RC fixes going forward.