usrmerge -- plan B?

[Adam Borowski]

Hi!
There's been another large bout of usrmerge issues recently, and a bunch of
new problems have been identified. The point of the whole idea has been
questioned, as well.

Thus, I'd propose holding our horses and thinking a bit. I'd also like to
suggest an alternate approach:

* let's scrap the usrmerge package
* move binaries to /usr/bin one by one, without hurry. If it takes 10
years, so what?
* /bin would be left populated with nothing but /bin/sh, /bin/bash and
whatever else POSIX demands.


Note that usrmerge isn't really related to dropping support for split fs
without initrd. That's already done, and was a case of trading a sizeable
drop of developer effort for the loss of a rare user setup. I for one
consider it an okay deal -- some disagree, but it's a separate matter.
Somehow, most of usrmerge flamewars are about that topic.

What we are discussing here is the plan to 1. move everything from /bin to
/usr/bin[1] and 2. "ln -s /usr/bin /bin".

That move turned out to be problematic (see #914204, and individual FTBFS/
uninstallability/unupgradeability bugs). Even the proposed workaround, to
disable usrmerge on buildds, doesn't really work -- a lot of binary uploads
are still built on developers' machines, and remember that there's a whole
world besides DDs -- being able to compile packages is a core freedom for
many of our users, and they usually lack the knowledge how to troubleshoot
this particular problem.

Another question is, why?

14:18 < bunk> What are the "of course" benefits of merged /usr for a user?
For a distribution I see the benefits of moving to merged /usr
only, but I honestly don't know in what cases "Installing the
usrmerge package will solve your problem." would be a correct
answer to a user problem.

... which hasn't received an answer.

The only writeup pro the move I've seen pointed at is:
https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
which describes other users, but provides no compelling arguments. Its
main reason is compatibility with Solaris -- which hasn't been relevant for
a long long time. Even the other distribution (Fedora) that has done the
split is rapidly following Solaris into obscurity (the whole RPM world has
gone to 20% of web servers from 65% a few years ago, other server uses seem
to be alike[2], Red Hat has been gobbled up). This leaves mostly claim #4:

# Fact: The /usr merge makes sharing the vendor-supplied OS resources
# between a host and networked clients as well as a host and local
# light-weight containers easier and atomic. Snapshotting the OS becomes a
# viable option. The /usr merge also allows making the entire
# vendor-supplied OS resources read-only for increased security and
# robustness.

-- which is untrue as a system with /etc and /var out of sync with /usr is
broken. There are attempts of reducing /etc but I have seen nothing about
/var.

Thus, it seems to me that the plan A for usrmerge has serious downsides for
dubious benefits. What about the plan B I described above?


Meow!

[1]. For simplicity I'm not mentioning /sbin.
[2]. It's hard to find good data on anything but web servers.

[Domenico Andreoli]

Post by Adam Borowski
Hi!

Hi,

Post by Adam Borowski
Thus, it seems to me that the plan A for usrmerge has serious downsides for
dubious benefits. What about the plan B I described above?

My preferred plan B is the one allowing to release Buster on time with most confidence. It's for our users, if we mess up with them the show is over.

thanks,
Domenico

[Adam Borowski]

Post by Domenico Andreoli

Post by Adam Borowski
Thus, it seems to me that the plan A for usrmerge has serious downsides for
dubious benefits. What about the plan B I described above?

My preferred plan B is the one allowing to release Buster on time with
most confidence. It's for our users, if we mess up with them the show is
over.

Yeah, those would be:
• my plan B (no immediate effort pre-Buster), or
• scrapping the whole thing (no effort at any time)
with less confidence:
• making usrmerge Essential (large amount of effort, known issues)
completely, utterly unacceptable:
• status quo -- supporting both

Note that my big objection isn't against usrmerge itself (even though I'm
unhappy about it), it's about having both merged and unmerged systems.
Thus, if we decide to commit to merging, we need to have a flag day and
merge _all_ supported systems. The only technical way to do so I can
think of would be uploading usrmerge set to Essential then immediately
upgrading all buildds. Even binary uploads built on developers' machines
would require checking. But yeah, it's doable.

The slew of build failures and broken packages in the last few days should
be enough to explain _why_ we need to either go all the way in or all the
way out. I bet there's at least an order or two of magnitude more problems
that would appear on rebuilding the world then carefully testing. For
which, a short time before freeze, we simply don't have a chance.

Thus, either usrmerge Essential or not included in Buster -- no middle way.


Meow!

[Hans]

Hi folks,

I am following this thread qwith great interest. And I also read about
usrmerge, too.

However, this did not answer my question, so I like to aksk here:

Will usrmerge break my system?

I have /usr on a seperate partition and I have it encrypted with luks.
Of course I am running a standard kernwel with initrd.

The same configuration is running on further computers.

Thanks for any aanswer.

Best regards

Hans

[Adam Borowski]

Post by Hans
Will usrmerge break my system?
I have /usr on a seperate partition and I have it encrypted with luks.
Of course I am running a standard kernwel with initrd.

No it won't. You already use an initrd, thus at no point the system is
running from a filesystem that has an incomplete set of executables.


Meow.

[Hans]

Am Mittwoch, 21. November 2018, 11:34:47 CET schrieb Adam Borowski:
Thanks Adam for the fast response.
I am looking forward to usrmerge, as I see only advantages at the moment.

Best

Hans

Post by Adam Borowski

Post by Hans
Will usrmerge break my system?
I have /usr on a seperate partition and I have it encrypted with luks.
Of course I am running a standard kernwel with initrd.

No it won't. You already use an initrd, thus at no point the system is
running from a filesystem that has an incomplete set of executables.
Meow.

[Hans]

Am Mittwoch, 21. November 2018, 11:34:47 CET schrieb Adam Borowski:
Thanks Adam for the fast response.
I am looking forward to usrmerge, as I see only advantages at the moment.

Best

Hans

Post by Adam Borowski

Post by Hans
Will usrmerge break my system?
I have /usr on a seperate partition and I have it encrypted with luks.
Of course I am running a standard kernwel with initrd.

No it won't. You already use an initrd, thus at no point the system is
running from a filesystem that has an incomplete set of executables.
Meow.

[Russ Allbery]

Post by Adam Borowski
Thus, either usrmerge Essential or not included in Buster -- no middle way.

I think I agree with this.

My impression is that most of the problems with usrmerge are because we're
trying to support a halfway position that Red Hat did not try to support,
where some systems are merged and some aren't. This creates a ton of
complicated edge cases and inconsistencies, and now we're looking at
making invasive changes to package build systems to try to fix those edge
cases. This feels like a vast waste of developer time and resources that
could be put to better purposes.

If we just force-merge every system on upgrade, none of those
inconsistencies matter, and I do believe we could successfully complete
that process (with some bumps, of course). But this halfway slow
migration is death by a thousand cuts.

I think there are some arguments to be made for just force-merging and
moving on, but they're mostly "tidiness" arguments (letting everyone
recycle the brain cells they're currently using to keep track of whether
something is in /bin or /usr/bin and try to make decisions about this,
getting rid of all the compatibility symlinks, permanently eliminating all
minor bugs from omitting /bin or /sbin from PATH, and so forth). Those
are real benefits that I don't think we should underestimate, but I'm not
sure they're strong enough benefits to create a bunch of hard feelings and
frustration and anger inside the project. They're also mostly benefits
for packagers; there aren't a ton of benefits for users here.

If we're *not* going to commit to force-merging all systems, I think we
should just stop, or at least slow way down, because there are a *lot* of
edge cases and it's going to require a lot of work to go through them.
I'm very dubious of the viability of any strategy that requires people
override the paths to binaries in their debian/rules file to undo Autoconf
auto-probing, for example.

It feels to me like we should decide as a project whether we're going to
do the same thing Red Hat did and just do the merge and be done with it,
or whether we're going to do a much slower migration by some more robust
strategy of (for example) moving each binary out of /bin manually, but
either way, the current strategy does not seem viable to me.

[Michael Stone]

Post by Russ Allbery
If we just force-merge every system on upgrade, none of those
inconsistencies matter, and I do believe we could successfully complete
that process (with some bumps, of course).

I think that's likely to be the most painful upgrade since a.out. We'd
need one heck of a lot of testing that needs to have already started
unless we want to push back buster.

[Russ Allbery]

Post by Michael Stone

Post by Russ Allbery
If we just force-merge every system on upgrade, none of those
inconsistencies matter, and I do believe we could successfully complete
that process (with some bumps, of course).

I think that's likely to be the most painful upgrade since a.out. We'd
need one heck of a lot of testing that needs to have already started
unless we want to push back buster.

This seems like too high of a level of pessimism given that the usrmerge
package already implements this sort of force-merge and some people have
it installed and don't seem to be running into a bunch of bugs. The last
round of problems was on systems without it installed, because packages
generated that way weren't backward-compatible, not with the forward
direction of forcing the merge.

That said, if we do want to go down this path, getting as many people as
possible to install usrmerge now and make sure it doesn't break anything
(or report bugs if it does) seems like a good idea. (Just don't build
Debian packages for upload on the resulting system; use a build chroot
instead.)

[Michael Stone]

Post by Russ Allbery
This seems like too high of a level of pessimism given that the usrmerge
package already implements this sort of force-merge and some people have
it installed and don't seem to be running into a bunch of bugs. The last
round of problems was on systems without it installed, because packages
generated that way weren't backward-compatible, not with the forward
direction of forcing the merge.
That said, if we do want to go down this path, getting as many people as
possible to install usrmerge now and make sure it doesn't break anything
(or report bugs if it does) seems like a good idea. (Just don't build
Debian packages for upload on the resulting system; use a build chroot
instead.)

How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage
whatsoever to doing so. I'd also expect those to be the systems most
likely to have issues.

[Marco d'Itri]

Post by Michael Stone
How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage whatsoever

Actually I have quite a lot personally, with exactly zero problems.
On some of them I also enjoy advantages of merged-/usr, like having
multiple containers share the same /usr.

[Michael Stone]

Post by Marco d'Itri

Post by Michael Stone
How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage whatsoever

Actually I have quite a lot personally, with exactly zero problems.
On some of them I also enjoy advantages of merged-/usr, like having
multiple containers share the same /usr.

So these are machines you've had around since like lenny (with a bunch
of people admining them), and you just ran usrmerge? The part that's
confusing me is turning something like that into a container. I'm not
talking about running it on new systems, that's obviously easy to test.

[Matthias Klumpp]

Post by Marco d'Itri

Post by Michael Stone
How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage whatsoever

Actually I have quite a lot personally, with exactly zero problems.
On some of them I also enjoy advantages of merged-/usr, like having
multiple containers share the same /usr.

Ditto, I did the same with about 8 machines about two weeks ago with no issues.
It feels to me that there are a lot of people just assuming issues
will happen without data to back it up.
With the reproducible builds testing for changes based on usrmerge vs
non-usrmerge builds now, I hope at least for the "how does changing
the build chroots affect compatibility of built packages" we'll have
reliable data soon.

(At the moment I don't actually see the upcoming doom - a few packages
broke, bugs were fixed, life goes on. If it turns out that we are not
able to cope with new bugs in time, we can always change decisions
later).

Cheers,
Matthias

[Russ Allbery]

Post by Matthias Klumpp
(At the moment I don't actually see the upcoming doom - a few packages
broke, bugs were fixed, life goes on. If it turns out that we are not
able to cope with new bugs in time, we can always change decisions
later).

The reason why I personally jumped into this thread is that I don't think
the initial fix proposed for R was good engineering. We should not be
doing that sort of fragile machinations in Autoconf configuration. It
will end in tears. We will miss some other binary later, or Autoconf will
change flags or how it probes, and we won't catch the breakage.

Doing this check in reproducible-builds definitely helps allievate my
concerns as a backstop, but this is still fragile and we don't have a
tight test/fix cycle. And, in general, I'm dubious of a path where we
support building a package on both merged and unmerged systems and then
allow running it on both merged and unmerged systems. There are so many
places that binary paths creep into software build processes, and so many
different upstream software build processes to analyze.

[Michael Stone]

Post by Russ Allbery
Doing this check in reproducible-builds definitely helps allievate my
concerns as a backstop, but this is still fragile and we don't have a
tight test/fix cycle. And, in general, I'm dubious of a path where we
support building a package on both merged and unmerged systems and then
allow running it on both merged and unmerged systems. There are so many
places that binary paths creep into software build processes, and so many
different upstream software build processes to analyze.

I'm likewise generally dubious of a process where we try to merge two
directories into one directory during a system upgrade as a side effect
of installing a package. I've just seen too many strange things on real
systems. If not supporting legacy installs isn't an option, we never
should have started this.

[Russ Allbery]

Post by Michael Stone

Post by Russ Allbery
Doing this check in reproducible-builds definitely helps allievate my
concerns as a backstop, but this is still fragile and we don't have a
tight test/fix cycle. And, in general, I'm dubious of a path where we
support building a package on both merged and unmerged systems and then
allow running it on both merged and unmerged systems. There are so
many places that binary paths creep into software build processes, and
so many different upstream software build processes to analyze.

I'm likewise generally dubious of a process where we try to merge two
directories into one directory during a system upgrade as a side effect
of installing a package. I've just seen too many strange things on real
systems. If not supporting legacy installs isn't an option, we never
should have started this.

I don't believe supporting legacy installs *without doing the migration*
is an option, or at least an option that we should take. We could
theoretically make it work, but the ongoing burden to packagers and to
our testing infrastructure would be high. It would be yet another obscure
and irritating thing that you'd have to mess with when packaging and that
was very hard to test, and we already have way too many of those.

So yes, this is exactly the decision before us.

[Michael Stone]

Post by Russ Allbery
I don't believe supporting legacy installs *without doing the migration*
is an option, or at least an option that we should take. We could
theoretically make it work, but the ongoing burden to packagers and to
our testing infrastructure would be high. It would be yet another obscure
and irritating thing that you'd have to mess with when packaging and that
was very hard to test, and we already have way too many of those.
So yes, this is exactly the decision before us.

Then this needs to be a very explicit (and much better advertised)
decision, and it needs a much, much better implementation. Pulling in
usrmerge during an upgrade isn't going to cut it--we'd need some kind of
pre-upgrade check that tells people what they need to fix before we
break it. Designing this in a hurry less than two months before we start
freezing seems incredibly ambitious.

[Didier 'OdyX' Raboud]

Post by Michael Stone
Then this needs to be a very explicit (and much better advertised)
decision, and it needs a much, much better implementation.
The current usrmerge package has no test mode, will bail with a partially-
converted system if it runs into problems, and has no way to revert the
process.

Sorry to be blunt about this, but have you reported these? Sniping at (any)
package without making the problems you see visible to others (through bugs)
is not really helpful.

Post by Michael Stone
Pulling in usrmerge during an upgrade isn't going to cut it--we'd need some
kind of pre-upgrade check that tells people what they need to fix before we
break it. Designing this in a hurry less than two months before we start
freezing seems incredibly ambitious.

usrmerge is in the archive for 3+ years now. What seems to be needed now is
for a lot of us to actually _try_ it, find and report bugs, and get this
through.

Don't forget that a specificity of our bug report system is that the only
measure of "it worked without issues" that we have is popcon; we only get a
measure of how much things fail, not how good they work:

https://qa.debian.org/popcon.php?package=usrmerge

(Funnily enough, it seems to have had a recent spike…)

Cheers,
OdyX

[Stephan Seitz]

Post by Didier 'OdyX' Raboud
usrmerge is in the archive for 3+ years now. What seems to be needed now
is for a lot of us to actually _try_ it, find and report bugs, and get
this through.

Why, if it was intended as an optional package for people who want
a merged /usr? If I don’t need or want a merged /usr I won’t install and
test the package.

The problem is that people are now talking to forcing the package on
everyone.

Shade and sweet water!

Stephan

[Marco d'Itri]

Post by Didier 'OdyX' Raboud
Sorry to be blunt about this, but have you reported these? Sniping at (any)

No, they have not. There is a lot of handwaving in this thread but very
few results of actual tests.

After creating again unmerged chroots for the buildds the only bugs
left, archive-wide, are #860523 (an abandoned package with no reverse
dependencies, there is even a patch in the BTS) and #913883 (iptables
mixing manually-created symlinks and diversions).
A few bugs in other packages have been reported and fixed over the years:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=usrmerge;users=***@linux.it

So we have:
- very few bugs exposed in other packages
- a track record of them being fixed

These are facts. People being worried if it would work or not on their
own system are just showing emotions, which are usually not relevant in
engineering decisions.

Post by Didier 'OdyX' Raboud
Don't forget that a specificity of our bug report system is that the only
measure of "it worked without issues" that we have is popcon; we only get a
https://qa.debian.org/popcon.php?package=usrmerge

This is only partially relevant because after the conversion there is no
reason to keep the package around and it can just be removed.

[Bjørn Mork]

Post by Marco d'Itri

Post by Didier 'OdyX' Raboud
Sorry to be blunt about this, but have you reported these? Sniping at (any)

No, they have not. There is a lot of handwaving in this thread but very
few results of actual tests.

"Migration is not (easily) reversible" was reported as important in
2016: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848626

It was immediately demoted to wishlist with this explanation:

"If and when the package will be automatically installed for some reason
then a warning will be appropriate.
usrmerge has extra priority, is not a dependency of anything and I am
not aware of anybody ever installing it by mistake."

The bug was later closed when a deconf question was added.

The fact that you redefined the topic of the bug does not mean that it
was solved. Your whining about missing bug reports on this subject is
simply arrogant. Or ignorant. Or both.


Bjørn

[Marco d'Itri]

Post by Bjørn Mork
"Migration is not (easily) reversible" was reported as important in
2016: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848626

This is not a bug: nothing is misbehaving, so classifying this as
wishlist is correct.
I even doubt that it would be technically feasible to implement
conversion rollback, and since the main concern here was doing the
conversion by mistake I solved it with the debconf question.

[Alexander E. Patrakov]

Post by Russ Allbery
The reason why I personally jumped into this thread is that I don't think
the initial fix proposed for R was good engineering. We should not be
doing that sort of fragile machinations in Autoconf configuration. It
will end in tears. We will miss some other binary later, or Autoconf will
change flags or how it probes, and we won't catch the breakage.

We could catch some of it automatically by creating /dontuse/bin and
/dontuse/usr/bin as symlinks to /bin and /usr/bin on all buildds (and
the sbin equivalents, too), adding that first to the PATH, and
auto-rejecting any package that has these strings in it. I understand
that there will be lots of FTBFS from doing that, so it may or may not
be a good idea, especially because this covers only "we won't catch the
breakage" point, but not "we should not be doing that".

[Alexander E. Patrakov]

Post by Marco d'Itri

Post by Michael Stone
How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage whatsoever

Actually I have quite a lot personally, with exactly zero problems.
On some of them I also enjoy advantages of merged-/usr, like having
multiple containers share the same /usr.

I agree with the argument in this thread that usrmerge should be either
Essential or not in Buster at all. I have no Debian systems in
production, but the company that I have a contract with has some Ubuntu
systems, and we are considering (as in: there is a successful PoC) the
use of OSTree for deployment of some production systems. OSTree folks
recommend merged /usr, so I do install usrmerge in the LXC container
that is being used as a "golden" tree for deployment. And it works quite
well for us with just a few extra moves and symlinks, and extlinux as
the boot loader (to work around removal of GRUB2-related files from the
Debian package).

So: my opinion is that I would prefer usrmerge in Essential in Buster,
for the benefit of people who use OSTree for deployment, and to reduce
divergence between Debian and Ubuntu plans.

[Alexander E. Patrakov]

Post by Alexander E. Patrakov

Post by Marco d'Itri

Post by Michael Stone
How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage whatsoever

Actually I have quite a lot personally, with exactly zero problems.
On some of them I also enjoy advantages of merged-/usr, like having
multiple containers share the same /usr.

I agree with the argument in this thread that usrmerge should be either
Essential or not in Buster at all. I have no Debian systems in
production, but the company that I have a contract with has some Ubuntu
systems, and we are considering (as in: there is a successful PoC) the
use of OSTree for deployment of some production systems. OSTree folks
recommend merged /usr, so I do install usrmerge in the LXC container
that is being used as a "golden" tree for deployment. And it works quite
well for us with just a few extra moves and symlinks, and extlinux as
the boot loader (to work around removal of GRUB2-related files from the
Debian package).
So: my opinion is that I would prefer usrmerge in Essential in Buster,
for the benefit of people who use OSTree for deployment, and to reduce
divergence between Debian and Ubuntu plans.

Based on reading the other messages in this thread, I changed my
opinion. The new opinion is that any of the following outcomes is OK,
with no specific preference:

1. mandatory usrmerge in Buster
2. usrmerge available from somewhere for installation in Buster, but not
installed on buildds, and dpkg-buildpackage should refuse to build
anything if /usr is merged and the user does not say "please".

[Russ Allbery]

Post by Michael Stone
How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage
whatsoever to doing so. I'd also expect those to be the systems most
likely to have issues.

Yup, lack of testing is definitely a valid point.

[Jonathan Dowland]

Post by Michael Stone
How many long-running production systems do you think people have run
usrmerge on? I'd guess close to zero, since there is no advantage
whatsoever to doing so. I'd also expect those to be the systems most
likely to have issues.

Long-running production systems would presumably be running Stable. We
need testing or unstable systems to try usrmerge. I don't think even
reporting on the success of usrmerge on current-Stable¹ would be
necessarily useful.

My gut feeling is we should do this transition; and we should not target
our next release but the one after.



¹ I can't remember our own code names any more.

[Marco d'Itri]

Post by Jonathan Dowland
Long-running production systems would presumably be running Stable. We
need testing or unstable systems to try usrmerge. I don't think even
reporting on the success of usrmerge on current-Stable¹ would be
necessarily useful.

I use merged-/usr without any issues on many stable systems, both new
and upgraded.

[Michael Stone]

Post by Marco d'Itri
I use merged-/usr without any issues on many stable systems, both new
and upgraded.

Again, how many weren't systems you're responsible for? I have no doubt
that you took care of the problems that you encountered personally when
you wrote the tool. I've seen a lot of problems on systems in the wild
that don't exist on my systems, but I don't expect that every one of our
users is a debian developer; sometimes other people solve problems in a
way that we didn't really expect or plan for them to.

Let's restate this so it can stand clearly on its own:

"We ran into some unanticipated issues when we usrmerged our build
systems, and we think the way to fix that is to force merge every one of
our users' systems."

[Ian Jackson]

Post by Michael Stone
"We ran into some unanticipated issues when we usrmerged our build
systems, and we think the way to fix that is to force merge every one of
our users' systems."

That does seem to be the position that is being advanced. It's quite
striking, isn't it ? At the very least more comprehensive thought and
planning is needed. And very probably more time.

Ian.

[Russ Allbery]

Post by Ian Jackson

Post by Michael Stone
"We ran into some unanticipated issues when we usrmerged our build
systems, and we think the way to fix that is to force merge every one
of our users' systems."

That does seem to be the position that is being advanced.

I think this is an unfair characterization of the thread, and adds
considerably more heat than light.

We ran into unanticipated problems *with supporting full portability of
packages between usrmerged systems and non-usrmerged systems*. (And I'm
not sure the problems were really that unanticipated -- that problem is
hard.) We're now evaluating what that means for the overall goal of
supporting systems with merged /usr. One possibility is to not attempt to
support that use case, which can be done in two ways: never merging /usr,
or merging all systems.

In this sort of discussion, it's very important to keep distinct the point
we're discussing and our positions on that point, and try to maintain our
agreement on what is even under discussion while we disagree.

To support my side of that, I promise to not mischaracterize consensus on
*what we're debating* as consensus on *what the outcome should be*, and
keep those two things entirely separate.

[Ian Jackson]

Post by Russ Allbery

Post by Ian Jackson

Post by Michael Stone
"We ran into some unanticipated issues when we usrmerged our build
systems, and we think the way to fix that is to force merge every one
of our users' systems."

That does seem to be the position that is being advanced.

I think this is an unfair characterization of the thread, and adds
considerably more heat than light.

I'm afraid I see it as totally fair. (And I do very much hesitate to
disagree with you since you are usually right about everything.)

Post by Russ Allbery
We ran into unanticipated problems *with supporting full portability of
packages between usrmerged systems and non-usrmerged systems*.

It was very clear that what was proposed, some months ago, was
*optional* usrmerge, which necessarily involves cross-compatibility.
It was on that basis that there were few objections. Thus the
original design promise from usrmerge proponents, when these changes
were last discussed here, was indeed full portability of packages.
(Whether the usrmerge proponents realised that or not.)

Post by Russ Allbery
(And I'm not sure the problems were really that unanticipated --
that problem is hard.)

No doubt there were misgivings in several quarters. I had my own
misgivings. But I kept them to myself, in a spirit of not picking an
unnecessary fight. After all if something is optional then I can just
not opt in. If someone wants to do something that might break their
system then why not let them ? They can keep all the pieces.

But when I said `unanticipated issues' I meant, of course,
unanticipated by the usrmerge designers and implementers. Yes,
those problems were unanticipated problems *of full portability*
rather than of usrmerge as an abstract principle. But the usrmerge
designers and implementers either (i) did not realise that their
actions depended on full portability, or (ii) did not realise how hard
it would be. That's why there was no plan for how to achieve it.


My position as a usrmerge sceptic, of letting them get on with doing
their thing, now seems to have been unwise. The idea that it would be
optional mutated, without proper discussion, and without a transition
plan, into it being the default for new installs.

And now there are serious suggestions that to solve this lack of
foresight, lack of proper planning, lack of proper consultation, and
lack of attention to detail, we should press ahead even faster and
make the new scheme mandatory ?

On a narrow technical level I can see why that is attractive. But in
the wider world, when a particular project team has caused lossage due
to poor technical judgement, excessive haste, and at least a perceived
reluctance to accomodate feedback, it is rarely the right response to
be *less* critical of their plans and recommendations, and push on
harder.

Especially if that risks alienating objectors who will rightly feel
that they would have predicted a mess, if anyone had been prepared to
listen to them.


I'm sorry to have to put these things this way. This will no doubt
read as a criticism of the people rather than of the code, which is
difficult and undesirable. But, whether the problems are cultural, or
whatever, the dispute here is not only about technical details.

We cannot resolve the risk/reward tradeoff of usrmerge with a
definitely correct answer because we do not have, and will never have,
all the necessary information. We cannot know how many systems in the
field would break from a forced usrmerge; even if we were to go ahead
with that we would probably never know the real cost.

Necessarily this is going to be a matter of judgement and guesswork.
The question then: whose judgement and guesswork ?


I would like to contrast the approach here to the way that the
AppArmor team have conducted their transition. Throughout they have
been alert to problems; sought feedback; avoided commitment; and
generally brought the vast majority of the Debian community along with
them. They have gained not only rough consensus but what looks like
real consent (insofar as such a thing might be possible collectively).


On the narrower question it must surely be obvious that it is far too
late to try to do a forced usrmerge of all systems in buster.
Certainly we should not be discussing forced usrmerge as anything but
a theoretical option, given that there is not even any transition plan
for it and the buster transition freeze is weeks away.

The default should be changed back not just in our buildds but also
for ordinary new installs. usrmerge proponents can use the time
between now and the release of buster to refine their understanding of
the issues, do the necessary research, and write a transition plan.

We can then have this discussion again early in the bullseye release
cycle. If we must.

Ian.

[Russ Allbery]

Post by Ian Jackson
It was very clear that what was proposed, some months ago, was
*optional* usrmerge, which necessarily involves cross-compatibility. It
was on that basis that there were few objections. Thus the original
design promise from usrmerge proponents, when these changes were last
discussed here, was indeed full portability of packages. (Whether the
usrmerge proponents realised that or not.)

This is a much better summary of the thread, and I wish that you would
have said this instead of claiming incorrectly that those same people are
the ones advocating for a full merge of all systems.

*I've* been one of the people advocating for that on this thread, and I've
not previously been involved in the usrmerge proposal at all. I'm not
sure how many of the other people advocating it have been involved in the
project. Neither Marco nor Simon have taken a position that I've seen,
and so far as I can tell are still interested in the optional approach.

I'm stating the advantages of fully converting Debian to merged /usr for
engineering reasons: I don't think the existing optional migration is
going to work very well or be supportable. On this point, I'm
*disagreeing* with the usrmerge maintainer, who holds the opposite view.

Can you see from this why I think your and Michael's collapsing of nuance
into a claim that the usrmerge proponents are trying to react to a setback
by moving faster is unfair and harmful? It's reducing a more complex
discussion to a simple us vs. them narrative.

Post by Ian Jackson
But when I said `unanticipated issues' I meant, of course, unanticipated
by the usrmerge designers and implementers. Yes, those problems were
unanticipated problems *of full portability* rather than of usrmerge as
an abstract principle. But the usrmerge designers and implementers
either (i) did not realise that their actions depended on full
portability, or (ii) did not realise how hard it would be. That's why
there was no plan for how to achieve it.

I generally agree with this. I also don't think this is particularly
unusual, or should be considered some sort of major failing. A lot of us,
particularly on volunteer projects, are not going to get our analysis
right the first time. That's why we stop, and think, and adjust when
something unexpected goes wrong, like now.

Post by Ian Jackson
My position as a usrmerge sceptic, of letting them get on with doing
their thing, now seems to have been unwise. The idea that it would be
optional mutated, without proper discussion, and without a transition
plan, into it being the default for new installs.

I agree with wanting more discussion and more of a plan before making it
the default for new installs, and I'm skeptical this is a good idea for
buster.

Post by Ian Jackson
And now there are serious suggestions that to solve this lack of
foresight, lack of proper planning, lack of proper consultation, and
lack of attention to detail, we should press ahead even faster and make
the new scheme mandatory ?

Again, I think you're mixing up two separate issues in a way that adds
more noise than clarity. There are two separate decisions here:

* Do we want to back off from supporting an optional switch and instead
pursue converting all Debian installations to merged /usr?

* If we do want to do that, *when* should we do that, and what should the
timeline look like?

The question of "faster" is about the second point, not the first one.
Personally, I think I agree with the comments here that it's too late to
do this for buster, and if we're going to work towards fully merging /usr,
this is something to tackle for buster+1 (preferrably by significant
testing very early in the cycle).

The first question is still open. Various people have stated their
opinions, but I'm not sure we're having a proper debate about it because
it keeps getting mixed into other issues, such as people's opinion of the
planning and communication strategy of people working on usrmerge, or what
the timeline could look like.

I understand that those are all issues that matter. What I'm objecting to
is (unintentionally, I think) muddling them all together into a giant,
upsetting mess of conflict. I don't blame you for finding that upsetting
-- when everything is mixed together like that, it's all very upsetting!
That's why we need to break this down into separate points of
disagreement, separate the engineering decisions from the communication
and community decisions, and figure out where we actually disagree and
what the most constructive path forward is.

Post by Ian Jackson
Necessarily this is going to be a matter of judgement and guesswork.
The question then: whose judgement and guesswork ?

Are we not a community? We're providing input as a community in this
thread, providing some judgement and guesswork from lots of different
contributors.

I agree that at some point someone needs to step forward to pull that all
together into a set of possible strategies to move forward, but I don't
feel like we've even separated the strands of things we're arguing about
yet, let alone clearly explored the options.

Post by Ian Jackson
We can then have this discussion again early in the bullseye release
cycle. If we must.

That idea makes me wince. This will just result in the same thing
happening again. Let's have the discussion *now*, when the problems are
fresh in our mind, and then defer *action* to early in the bullseye
release cycle (which I suspect is likely to happen anyway given how long
it usually takes us to sort through questions of large migrations like
this).

[Marco d'Itri]

Post by Russ Allbery
I'm stating the advantages of fully converting Debian to merged /usr for
engineering reasons: I don't think the existing optional migration is
going to work very well or be supportable. On this point, I'm
*disagreeing* with the usrmerge maintainer, who holds the opposite view.

I am not disagreeing with you at all. I just do not want to start
another fight with the people who are scared by changes.

Post by Russ Allbery
I agree with wanting more discussion and more of a plan before making it
the default for new installs, and I'm skeptical this is a good idea for
buster.

But nobody so far has explained exactly what could be done better by
waiting more time.

[Ian Jackson]

Post by Russ Allbery
This is a much better summary of the thread, and I wish that you would
have said this instead of claiming incorrectly that those same people are
the ones advocating for a full merge of all systems.
If you are seriously concerned with the small issuses caused by the
transition to merged-/usr then I have a better proposal.
Plan C: just stop supporting non-merged-/usr systems since these
problems are caused by having to support both, and there is no real
benefit in doing that other than pleasing the few people who are scared
by changes.

Ian.

[Ansgar Burchardt]

Post by Russ Allbery
This is a much better summary of the thread, and I wish that you would
have said this instead of claiming incorrectly that those same people are
the ones advocating for a full merge of all systems.
If you are seriously concerned with the small issuses caused by the
transition to merged-/usr then I have a better proposal.
Plan C: just stop supporting non-merged-/usr systems since these
problems are caused by having to support both, and there is no real
benefit in doing that other than pleasing the few people who are scared
by changes.

What is this quote supposed to tell us?

Ansgar

[Matthias Klumpp]

Am Fr., 23. Nov. 2018 um 13:45 Uhr schrieb Ian Jackson

Post by Russ Allbery
This is a much better summary of the thread, and I wish that you would
have said this instead of claiming incorrectly that those same people are
the ones advocating for a full merge of all systems.
If you are seriously concerned with the small issuses caused by the
transition to merged-/usr then I have a better proposal.
Plan C: just stop supporting non-merged-/usr systems since these
problems are caused by having to support both, and there is no real
benefit in doing that other than pleasing the few people who are scared
by changes.

For this I actually wonder: Why don't we just do that and enforce the
usrmerge on unstable?
I think we are caring too much about the stability of the unstable
suite, and absolutely miss the chance of just running experiments to
gather data on the feasibility of changes.
What we could do in this case is for example to just make the usrmerge
migration mandatory for users of the unstable suite and see what kind
of issues and how many of them users will actually encounter. The
suite is called unstable afterall :-)
If there are actual issues encountered, we can always revert a change
and not have it go into stable, but in any case we will get a lot
better testing on the migration and a lot more data on whether there
actually are any issues.

Since new installations will be usrmerged by default anyway and there
is no reason or easy way to swich back to a split-usr system, I think
the issues related to a split-usr-system will go away in the long run
anyway. During distribution upgrades there is a lot that can be wrong
and a lot of stuff the administrator needs to care about (config file
changes, different featuresets of tools, software being removed from
the archive, ...), so if the usrmerge package has a sensible fallback
with information to the system administrator on what to do in case of
an error, and works for 98% of all users anyway, I see no reason not
to try using it and save us from an eternal transition period or pain
of maintaining two configurations.

Cheers,
Matthias

[Stephan Seitz]

Post by Matthias Klumpp
If there are actual issues encountered, we can always revert a change

And how do you revert this change? As far as I have understand you can’t
remove the usrmerge package and have your system in the old state again.

As others in the thread have mentioned they don’t see the risk with new
installations but with old systems with different admins and third-party
software.

Post by Matthias Klumpp
anyway. During distribution upgrades there is a lot that can be wrong
and a lot of stuff the administrator needs to care about (config file

Right, and it means he has enough to do and doesn’t want to debug the
usrmerge. I don’t want to have a usrmerge at a dist-upgrade. You don’t
really know the sequence of the package updates. I think the risk is too
big to have a partial upgraded system.

Post by Matthias Klumpp
with information to the system administrator on what to do in case of
an error, and works for 98% of all users anyway, I see no reason not

If the update of 2% of our systems won’t work because of failing usrmerge
I would be asked if Debian is the right distribution.

Shade and sweet water!

Stephan

[Hans]

And how do you revert this change? As far as I have understand you can’t
remove the usrmerge package and have your system in the old state again.

Making an image of the whole hard drive is always a good idea.

As others in the thread have mentioned they don’t see the risk with new
installations but with old systems with different admins and third-party
software.

Changes are always coming with a risk. However, ones who fear, should try on a
canary. This is, what I would do. Ok, this might be a little more work, but it
is is worth.

Right, and it means he has enough to do and doesn’t want to debug the
usrmerge. I don’t want to have a usrmerge at a dist-upgrade. You don’t
really know the sequence of the package updates. I think the risk is too
big to have a partial upgraded system.

So, that is, where "testing" is standing for. Testing before release....

If the update of 2% of our systems won’t work because of failing usrmerge
I would be asked if Debian is the right distribution.

Good point. However, this should be told to Microsoft admins or their
deciders.

Shade and sweet water!
Stephan

I think, we have now 2-3 moths for testing until freeze begins. Shouldn't this
time not beeing well used, to eliminate all issues?

Best

Hans

[Stephan Seitz]

Post by Hans

Post by Stephan Seitz
And how do you revert this change? As far as I have understand you
can’t remove the usrmerge package and have your system in the old
state again.

Making an image of the whole hard drive is always a good idea.

While this is easy done if the system is a VMware VM I know my fellow
admins well enough that they will simply cancel the update because they
are not interested in debugging the upgrade process for a single package
they don’t need.

Post by Hans
So, that is, where "testing" is standing for. Testing before release....

You know I don’t have doubts that you won’t have problems (or very few)
if you do a new testing installation. I don’t like Marco’s attitude but
he is doing a good job. If he has tested his package as he said this
isn’t the problem.

But Debian/testing doesn’t mean testing on real servers which have
„survived” different Debian releases, where different admins copied files
from one location to another, or with some third-party software.

Post by Hans

Post by Stephan Seitz
If the update of 2% of our systems won’t work because of failing
usrmerge I would be asked if Debian is the right distribution.

Good point. However, this should be told to Microsoft admins or their
deciders.

;-) But while there are no alternative Windows distributions Debian isn’t
the only one and can be changed.

Shade and sweet water!

Stephan

[Matthias Klumpp]

Am Fr., 23. Nov. 2018 um 14:47 Uhr schrieb Stephan Seitz

Post by Matthias Klumpp
If there are actual issues encountered, we can always revert a change

And how do you revert this change? As far as I have understand you can’t
remove the usrmerge package and have your system in the old state again.

You don't - it's unstable, for testing these things. If it breaks, you
file a bug and fix the system.

As others in the thread have mentioned they don’t see the risk with new
installations but with old systems with different admins and third-party
software.

Post by Matthias Klumpp
anyway. During distribution upgrades there is a lot that can be wrong
and a lot of stuff the administrator needs to care about (config file

Right, and it means he has enough to do and doesn’t want to debug the
usrmerge. I don’t want to have a usrmerge at a dist-upgrade. You don’t
really know the sequence of the package updates. I think the risk is too
big to have a partial upgraded system.

For the sequence of installations, the APT maintainers can shed light
on what the proper plan could be to ensure the usrmerge update happens
at the right time.

Post by Matthias Klumpp
with information to the system administrator on what to do in case of
an error, and works for 98% of all users anyway, I see no reason not

If the update of 2% of our systems won’t work because of failing usrmerge
I would be asked if Debian is the right distribution.

There are always unforseen issues to be expected when upgrading. And
at the moment, the only issues that are known when installing the
usrmerge package is when there are different binaries with the same
name in /bin and /usr/bin (or /sbin), and I really don't think that
this is actually a likely scenario.
For these cases though maybe the usrmerge script could ask the admin
on what to do to handle these particular binaries, instead of failing.

I am not strongly advocating for going down this route and actually
migrating all systems on update, but I do not want us to dismiss that
option because we are scared that something might go wrong without
actually knowing that there are unfixable cases where the update might
inevitably break on older installations. Instead, I would rather want
to try out the migration on a bigger scale - potentially and
temporarily break unstable (with a warning, of course), instead of
discussing over and over again potential issues that might not
actually be real and delaying a useful change because of that.

(TBH, for the buster release not switching the buildds to usrmerge and
keep debootstrap/the installer to install in an usrmerged
configuration and then do the final switch during bullseye seems
sensible and I don't see any issue this would cause. Of course if the
reproducible-builds test turns out that we only need to fix a small
amount of packages to make the usrmerge happen on buildds as well,
switching them as well could make sense still)

Cheers,
Matthias

[Michael Stone]

Post by Matthias Klumpp
For these cases though maybe the usrmerge script could ask the admin
on what to do to handle these particular binaries, instead of failing.

Maybe, as I suggested upthread, there could be a preview mode in which
the admin could be told what would happen, alerted if the process is
going to fail, and given some guidance as to what to do--*before*
blindly pulling the trigger. We could even ask people to report on
whether the preview mode predicts issues on their systems in order to
gather data--which I think is far more likely than people letting us
know whether the process broke (past tense) their systems (for the sake
of science).

[Gabriel F. T. Gomes]

Post by Michael Stone

Post by Matthias Klumpp
For these cases though maybe the usrmerge script could ask the admin
on what to do to handle these particular binaries, instead of
failing.

Maybe, as I suggested upthread, there could be a preview mode in which
the admin could be told what would happen, alerted if the process is
going to fail, and given some guidance as to what to do--*before*
blindly pulling the trigger. We could even ask people to report on
whether the preview mode predicts issues on their systems in order to
gather data--which I think is far more likely than people letting us
know whether the process broke (past tense) their systems (for the
sake of science).

Has this option been given enough attention? It sounds appealing in
two ways: 1. it would work as an incentive for people who want to
install usrmerge (be it because they are curious about the results or
because they wish to help), but are not in a mood to reinstall their
systems *if* the merge causes trouble; 2. it would serve as a place to
collect the feedback and summarize the potential problems.

Maybe I'm just asking because I'm one of those not in a mood to
reinstall, but it does sound appealing to me.

[Marco d'Itri]

Post by Gabriel F. T. Gomes
Has this option been given enough attention? It sounds appealing in

Yes. I would love to implement a --dry-run mode, but I still need to
figure out how much complex it would be and if it could actually cover
all cases.

[Stephan Seitz]

Post by Matthias Klumpp
There are always unforseen issues to be expected when upgrading. And

Of course, and since a dist-upgrade will bring newer software you may
already have to fix configuration files.

Post by Matthias Klumpp
at the moment, the only issues that are known when installing the
usrmerge package is when there are different binaries with the same
name in /bin and /usr/bin (or /sbin), and I really don't think that
this is actually a likely scenario.

I don’t know but I’ve seen enough strange installations.

It would be good if the usrmerge package would do a dry-run as part of
the installation. If there are duplicate files the list will be printed
(or mailed) and the installation will fail without breaking the whole
upgrade process. The the admin can fix the problem later.

Shade and sweet water!

Stephan

[Adam Borowski]

Post by Matthias Klumpp
Am Fr., 23. Nov. 2018 um 14:47 Uhr schrieb Stephan Seitz

Post by Matthias Klumpp
If there are actual issues encountered, we can always revert a change

And how do you revert this change? As far as I have understand you can’t
remove the usrmerge package and have your system in the old state again.

You don't - it's unstable, for testing these things. If it breaks, you
file a bug and fix the system.

Fix -- not completely restore from backups. This is not an option with
usrmerge; we have severity:critical for bugs with such a fallout.

And yeah, bugs of this severity prevent a package from being a part of the
next stable release.


Meow!

[Daniele Nicolodi]

Hello Adam,

Post by Adam Borowski

Post by Matthias Klumpp
Am Fr., 23. Nov. 2018 um 14:47 Uhr schrieb Stephan Seitz

Post by Matthias Klumpp
If there are actual issues encountered, we can always revert a change

And how do you revert this change? As far as I have understand you can’t
remove the usrmerge package and have your system in the old state again.

You don't - it's unstable, for testing these things. If it breaks, you
file a bug and fix the system.

Fix -- not completely restore from backups. This is not an option with
usrmerge; we have severity:critical for bugs with such a fallout.

I've seen this repeated multiple times in this thread, where does your
assessment that the usrmerge package can leave the system in a non
working state comes from?

In my experience, usrmerge stops the merge if it encounters some
unforeseen difficulty and hints to how to resolve the problem. Once the
problem has been resolved, usrmerge can be run again and it will pick up
from where it left.

Is your experience different? Can you be more precise on the kind of
errors or system breakage you encountered?

Thank you.

Best,
Daniele

[Theodore Y. Ts'o]

Post by Russ Allbery

Post by Ian Jackson
My position as a usrmerge sceptic, of letting them get on with doing
their thing, now seems to have been unwise. The idea that it would be
optional mutated, without proper discussion, and without a transition
plan, into it being the default for new installs.

I agree with wanting more discussion and more of a plan before making it
the default for new installs, and I'm skeptical this is a good idea for
buster.

Given that one of my packages had a bug filed against it that was
caused by usrmerge, and while I *can* fix it, I am also getting a bit
skeptical about trying to rush the usrmerge for buster --- and
**definitely** if it is a mandatory merge. I'd be OK usrmerge being
in buster so long as it has a big huge, fat warning of the form, "if
it breaks your system you get to keep both pieces". So if things
break in usrmerge system on Buster, they would ***not*** be RC, at
least not for the next 3 months.

Post-buster, I agree doing a mandatory usrmerge transition makes
sense, and this should be done very early in the development cycle,
and not at the very end of the development cycle.

Post by Russ Allbery
That idea makes me wince. This will just result in the same thing
happening again. Let's have the discussion *now*, when the problems are
fresh in our mind, and then defer *action* to early in the bullseye
release cycle (which I suspect is likely to happen anyway given how long
it usually takes us to sort through questions of large migrations like
this).

Agreed, completely. If we leave usrmerge in buster as a "use at your
own risk", then the people who are the most passionate can try it on
their production systems. That will hopefully give us more feedback,
which will significantely reduce risk for bullseye.

- Ted

[Marco d'Itri]

Again, how many weren't systems you're responsible for? I have no doubt that
the tool. I've seen a lot of problems on systems in the wild that don't

No: what I meant is that I did not encounter any problems.

I reject the narrative that switching to a merged-/usr system generally
causes problems. So far nobody reported anything significant.

[Ian Jackson]

Post by Marco d'Itri
So far nobody reported anything significant.

I hear there was a major free software project who accidentally
usrmerged their build systems and discovered that they then built
broken packgaes.

Ian.

[Marco d'Itri]

Post by Ian Jackson
I hear there was a major free software project who accidentally
usrmerged their build systems and discovered that they then built
broken packgaes.

And it was quickly fixed, so no big deal.

[Ian Jackson]

Post by Marco d'Itri

Post by Ian Jackson

Post by Marco d'Itri
So far nobody reported anything significant.

I hear there was a major free software project who accidentally
usrmerged their build systems and discovered that they then built
broken packgaes.

And it was quickly fixed, so no big deal.

I think this allows us to calibrate what you consider `anything
significant'.

There is tradeoff here between risk of breakage, and reduction of
future work (as most clearly explained by Russ). The argument that is
being made is that the risk is low because of a lack of reports of
breakage.

Others have observed that systems most likely to experience trouble
will not have been upgraded. For example, chiark was first installed
with Debian 0.93R5 in 1993. Obviously I haven't usrmerged it. No-one
sensible in my position would do so. So the `lack of reports'
probably stems from a lack of contact of this proposal with the more
difficult parts of the real world.

But what we see here is that the `lack of reports' somehow ignores a
problem sufficiently serious to generate more than one RC bug in
Debian itself, and to require the usrmerge to be reverted at least
temporarily. (Bear in mind of course that happily our build machines
*can* be reverted because they are frequently re-imaged. Others may
not be so lucky.)

Ian.

[Ansgar Burchardt]

Post by Ian Jackson

Post by Marco d'Itri

Post by Ian Jackson

Post by Marco d'Itri
So far nobody reported anything significant.

I hear there was a major free software project who accidentally
usrmerged their build systems and discovered that they then built
broken packgaes.

And it was quickly fixed, so no big deal.

I think this allows us to calibrate what you consider `anything
significant'.

Causing problems with a few packages is not a significant problem. We
should stop upgrading GCC to newer versions otherwise as doing so
generates tons of RC bugs every time.

Post by Ian Jackson
There is tradeoff here between risk of breakage, and reduction of
future work (as most clearly explained by Russ). The argument that is
being made is that the risk is low because of a lack of reports of
breakage.

Moving files around in such a matter that they are still available in
the old location (via a symlink) is not a very invasive change, so
there is only a small risk of problems. That matches what was reported
so far.

Very few changes come with zero problems.

Post by Ian Jackson
Others have observed that systems most likely to experience trouble
will not have been upgraded. For example, chiark was first installed
with Debian 0.93R5 in 1993. Obviously I haven't usrmerged it. No-
one sensible in my position would do so.

Why should "originally installed in 1993" make moving files from one
location to another more difficult? It's not different than having to
add LSB init headers to local init scripts (or just replace them with
systemd units these days), having to purge leftover conffiles from
removed packages or similar changes on upgrades.

If the system is prone to breakage on upgrades in general, I would
expect anyone sensible to fix that.

Ansgar

[Russ Allbery]

Post by Ansgar Burchardt
Moving files around in such a matter that they are still available in
the old location (via a symlink) is not a very invasive change, so there
is only a small risk of problems.

I think it's fair to note that our past experience in Debian doesn't
really support this. I've run into multiple problems in unstable with
uninstallable packages due to various bugs in this sort of change, most
recently with iptables. We repeatedly get the details of this change
wrong in various subtle ways that create issues in some upgrade paths and
not others.

This may be acceptable temporary breakage, and I don't think any of it
made it into stable (and it usually doesn't even make it into testing),
but if we're going to do a lot of this, I think we need better tools, such
as declarative support in packaging metadata that tells dpkg to do the
right thing, so that we can lean on a single, well-tested, robust
implementation.

[Ansgar Burchardt]

Post by Russ Allbery

Post by Ansgar Burchardt
Moving files around in such a matter that they are still available in
the old location (via a symlink) is not a very invasive change, so there
is only a small risk of problems.

I think it's fair to note that our past experience in Debian doesn't
really support this. I've run into multiple problems in unstable with
uninstallable packages due to various bugs in this sort of change, most
recently with iptables. We repeatedly get the details of this change
wrong in various subtle ways that create issues in some upgrade paths and
not others.

Well, the iptables case is different: those are compat symlinks created
by the package's maintainer scripts, not the /bin -> /usr/bin symlinks
that merged-/usr sets up.

If we want to support packages such as iptables moving binaries from
/{s,}bin to /usr/{s,}bin while setting up compat symlinks on
non-merged-/usr systems, it might be useful to have a dh-usrmerge
package creating the maintainer scripts. (Also for some files below
/lib, but most libraries could just be moved without compat symlinks.)

This should be similar to what OpenSUSE has done; see the section around
`#UsrMerge` on https://en.opensuse.org/openSUSE:Usr_merge

This could also be seen as a slower path to merged-/usr: programs such
as `sed` would be in both /bin and /usr/bin and hard-coding either would
be fine (as with merged-/usr, but not without). Less static files would
be on a read-write root file system (if /usr is a separate read-only
fs).

In case a later Debian release would merged-/usr mandatory, the
conversion process would be less problematic as no files would be left
to move (just replace individual symlinks with a directory symlink).

Ansgar

[Russ Allbery]

Post by Ansgar Burchardt
If we want to support packages such as iptables moving binaries from
/{s,}bin to /usr/{s,}bin while setting up compat symlinks on
non-merged-/usr systems, it might be useful to have a dh-usrmerge
package creating the maintainer scripts. (Also for some files below
/lib, but most libraries could just be moved without compat symlinks.)
This should be similar to what OpenSUSE has done; see the section around
`#UsrMerge` on https://en.opensuse.org/openSUSE:Usr_merge
This could also be seen as a slower path to merged-/usr: programs such
as `sed` would be in both /bin and /usr/bin and hard-coding either would
be fine (as with merged-/usr, but not without). Less static files would
be on a read-write root file system (if /usr is a separate read-only
fs).

Yes, this exactly, thank you. This is a much better description of what I
was seeing as an alternative to the current usrmerge approach, assuming a
(not yet decided) project consensus that we want to do something like
usrmerge in the long run.

This approach will obviously take forever, and be another one of those
Debian transitions that someone five years from now has to do a bunch of
work to finally finish, assuming that happens at all. But it would be
backward-compatible the entire way.

[Simon McVittie]

Post by Ansgar Burchardt
If we want to support packages such as iptables moving binaries from
/{s,}bin to /usr/{s,}bin

To be honest, I'm not sure whether we do want this. We should be careful,
at least.

Now that we don't support booting without /usr[1], it is no longer
necessary to move an executable from /bin to /usr/bin if it gains
a dependency on a library that is in /usr/lib, and similarly, it is
unnecessary to move executables in the opposite direction to make them
available in early boot.

Unlike merged /usr, doing this move doesn't make things like containers
any simpler until literally every executable in /bin has undergone this
transition; the work happens now, but the benefit doesn't happen until
the transition is complete. If the compat symlink isn't created
carefully, it can easily break merged-/usr systems.

However, it can easily cause the same issue that started this thread -
when a dependent package detects "the" path to the iptables executable,
it will get different answers depending on $PATH order, just like it would
on a merged /usr system. That's fine if we can guarantee that iptables
exists in both places, but during a partial upgrade, we can't count on
that being the case without a versioned dependency on the moved iptables,
which I assume nobody is seriously intending to mass-bug-file?

For iptables, maybe that's no big deal, because not many packages depend
on it and hard-code its path, but we have evidence from our accidental
merged-/usr-buildds experiment (and its subsequent intentional equivalent
in reproducible-builds) that the paths to programs like grep, sed and
(of course) sh are more widespread.

(Everything I've said about /bin applies equally to /sbin of course,
I just didn't want to keep typing /{s,}bin.)

I'm not sure yet what the best plan for merged /usr is. I would definitely
like to make sure it's at least possible to continue to use merged
/usr for special-purpose systems (particularly containers and embedded
systems), even if it comes with major caveats like "can't reliably build
Debian packages suitable for other systems"; I personally think everyone
should be using sbuild or equivalent, either on a buildd or locally,
to build "release-quality" packages suitable for distribution to other
systems *anyway*, but I know that view isn't necessarily universal.

For at least special-purpose systems, merged /usr seems to work fine with
stretch, and I was able to get it working in an Ubuntu 12.04 derivative
by backporting a single-digit number of changes, so that particular genie
has been out of the bottle for quite some time anyway.

smcv

[1] in the sense that if /usr is separate, we require it to be mounted
by the initramfs

[Ian Jackson]

Post by Simon McVittie
I'm not sure yet what the best plan for merged /usr is. I would definitely
like to make sure it's at least possible to continue to use merged
/usr for special-purpose systems (particularly containers and embedded
systems), even if it comes with major caveats like "can't reliably build
Debian packages suitable for other systems";

To be very clear: I have no problem with this at all.

Post by Simon McVittie
I personally think everyone
should be using sbuild or equivalent, either on a buildd or locally,
to build "release-quality" packages suitable for distribution to other
systems *anyway*, but I know that view isn't necessarily universal.

"Suitable for distribution to other systems" is rather a moveable
feast. I absolutely agree if you mean formal publication as part of
some kind of release.

But I'm sure all of us have on occasion done ad-hoc builds and then
copied the .deb somewhere else to install it. Indeed my own
experience is that during development I rarely use a chroot. I think
someone should be able to build some software on their own computer
and give the binaries to a friend, without having to set up a chroot.

I also think that setting up a chroot should be made easier and that
more people should use chroots. I don't think these views conflict.

Post by Simon McVittie
For at least special-purpose systems, merged /usr seems to work fine with
stretch, and I was able to get it working in an Ubuntu 12.04 derivative
by backporting a single-digit number of changes, so that particular genie
has been out of the bottle for quite some time anyway.

Would it be helpful to make some of this explicit in Debian policy ?

IMO binary packages shipped by Debian should certainly support
installation on both merged-usr and separate-usr systems.

And I wouldn't object to a rule that our source packages must build
`correctly' on both such systems, subject to the caveat that the
results from a merged-usr build are not of general applicability and
should be used only in a closed environment where all the target
systems are also merged-usr.

Does that make sense ?

Ian.

[W. Martin Borgert]

Post by Ansgar Burchardt
This could also be seen as a slower path to merged-/usr: programs such
as `sed` would be in both /bin and /usr/bin and hard-coding either would
be fine (as with merged-/usr, but not without). Less static files would
be on a read-write root file system (if /usr is a separate read-only
fs).
In case a later Debian release would merged-/usr mandatory, the
conversion process would be less problematic as no files would be left
to move (just replace individual symlinks with a directory symlink).

Reminds me of the long /usr/doc -> /usr/share/doc transition in potato
times. And we did not even have dh in those days. Sounds good to me!

[Holger Levsen]

Post by W. Martin Borgert
Reminds me of the long /usr/doc -> /usr/share/doc transition in potato
times. And we did not even have dh in those days. Sounds good to me!

ITYM s#potato#slink, potato, woody, sarge, etch and lenny#

(Started in 1999 and finally fully finished on Sep 25th 2008 with
closing #322762.)

So I don't think this transition is such a positive example :-D

[W. Martin Borgert]

Post by Holger Levsen

Post by W. Martin Borgert
Reminds me of the long /usr/doc -> /usr/share/doc transition in potato
times. And we did not even have dh in those days. Sounds good to me!

ITYM s#potato#slink, potato, woody, sarge, etch and lenny#
(Started in 1999 and finally fully finished on Sep 25th 2008 with
closing #322762.)
So I don't think this transition is such a positive example :-D

Why not positive? Yes, it took some time, but it worked out
without breakage. As we say: Gut Ding will Weile haben! :—)
(for non-Germans: "it takes time to do a thing well")

[Marco d'Itri]

Post by Ansgar Burchardt
Well, the iptables case is different: those are compat symlinks created
by the package's maintainer scripts, not the /bin -> /usr/bin symlinks
that merged-/usr sets up.

Actually iptables is different because it mixes (incorrectly)
compatibility symlinks and diversions.

Correctly creating just compatibility symlinks is trivial and has been
documented for years on the wiki.

Post by Ansgar Burchardt
In case a later Debian release would merged-/usr mandatory, the
conversion process would be less problematic as no files would be left
to move (just replace individual symlinks with a directory symlink).

No: it would not be any easier than it is now because every Debian
system can already be converted automatically. Having usrmerge move
files /installed by packages/ is well tested.

[Michael Stone]

Post by Ansgar Burchardt
Moving files around in such a matter that they are still available in
the old location (via a symlink) is not a very invasive change, so
there is only a small risk of problems. That matches what was reported
so far.

That's not actually what happens: the files are still available in the
old location *if and only if the process is fully successful*. If there
are any issues you have a partially migrated system in which the files
are *not* still available in the old location, and which cannot be
reverted back to the original state.

[Marco d'Itri]

That's not actually what happens: the files are still available in the old
location *if and only if the process is fully successful*. If there are any
issues you have a partially migrated system in which the files are *not*
still available in the old location, and which cannot be reverted back to
the original state.

This is not true: the convert-usrmerge program was designed to keep the
system consistent at all times, even while it is being run, by creating
symlinks when binaries are being moved.
In the worst case it will fail explaining that some local change (in
a directory which should not have been modified by the local admin, BTW)
needs to be addressed by the local admin and then it can be restarted
and continue its work.
And if it will fail due to some other unexpected bug then it will still
leave behind a working system.

Maybe you could actually look at the code and/or try it?
You can even test it on a live system by booting it again in kvm with
a snapshot.

[Matthew Vernon]

Post by Marco d'Itri
In the worst case it will fail explaining that some local change (in
a directory which should not have been modified by the local admin, BTW)
needs to be addressed by the local admin and then it can be restarted
and continue its work.

Could you expand on this? I'm responsible for enough systems with enough
"interesting historical artifacts" that I'd bet that pretty much every
directory has been modified on at least some of them.

Matthew

[Marco d'Itri]

Post by Matthew Vernon

Post by Marco d'Itri
In the worst case it will fail explaining that some local change (in
a directory which should not have been modified by the local admin, BTW)
needs to be addressed by the local admin and then it can be restarted
and continue its work.

Could you expand on this? I'm responsible for enough systems with enough
"interesting historical artifacts" that I'd bet that pretty much every
directory has been modified on at least some of them.

usrmerge cannot convert a system which has two different binaries (not
symlinks) with the same name in {/usr,}/s?bin/, because it cannot know
which one should be kept.
This condition does not happen on regular systems: only if somebody
manually copied something to {/usr,}/s?bin/, which is not supposed to
happen.

[Didier 'OdyX' Raboud]

Post by Ian Jackson
There is tradeoff here between risk of breakage, and reduction of
future work (as most clearly explained by Russ). The argument that is
being made is that the risk is low because of a lack of reports of
breakage.
Others have observed that systems most likely to experience trouble
will not have been upgraded. For example, chiark was first installed
with Debian 0.93R5 in 1993. Obviously I haven't usrmerged it. No-one
sensible in my position would do so.

Of course you do have backups. For the sake of the argument, would you
consider trying an install of usrmerge on a restored backup VM somewhere?

Maybe I'm overlooking something, but it seems that installing usrmerge on a
production system is going to be a _much_ less demanding task proceeding with
a stable upgrade.

Cheers,
OdyX

[Ian Jackson]

Post by Didier 'OdyX' Raboud
Of course you do have backups. For the sake of the argument, would you
consider trying an install of usrmerge on a restored backup VM somewhere?

I could do it. But, frankly, this is quite a lot of work. I think
the work (throughout the Debian ecosystem) to test this properly far
outweighs the minor benefits.

And if I do test it and find a lot of bugs then am I going to be
expected to report them all in detail ? That is also a ton of work.
Am I then going to be expected to retest when the bugs are allegedly
fixed ? I think this is just outsourcing the pain of bad design
choices, frankly.

If we must get to merged-usr on all systems eventually, Adam's
proposed transition plan is much better.

Ian.

[Marco d'Itri]

Post by Russ Allbery
I think there are some arguments to be made for just force-merging and
moving on, but they're mostly "tidiness" arguments (letting everyone

No, they are not that at all. I have never argued about "tidiness", nor
did anybody else that is working to support merged-/usr.
A merged-/usr is an enabler for new features, like stateless systems,
clones, appliances and better containers.
Simon McVittie today was patient enough to explain some of them.

Post by Russ Allbery
strategy of (for example) moving each binary out of /bin manually, but

As I explained, this cannot really work.

[Russ Allbery]

Post by Marco d'Itri

Post by Russ Allbery
I think there are some arguments to be made for just force-merging and
moving on, but they're mostly "tidiness" arguments (letting everyone

No, they are not that at all. I have never argued about "tidiness", nor
did anybody else that is working to support merged-/usr. A merged-/usr
is an enabler for new features, like stateless systems, clones,
appliances and better containers. Simon McVittie today was patient
enough to explain some of them.

I read all of Simon's message, which was excellent and which I greatly
appreciate. The arguments he listed there are, however, the arguments I
was classifying as "tidiness" arguments.

We shouldn't get caught up on terminology, since whether or not my
adjective is correct isn't really the point. The point is that every use
case he describes is possible without usrmerge. It's just more
complicated because it requires manipulating multiple paths instead of
one, which requires a bit more ingenuity and fragility (via symlink
redirections and the like) to make atomic.

I completely understand why merged /usr makes it easier, and removes a lot
of avoidable bugs. But it is not a *requirement* for doing any of the
things that he spells out in that message. There is still a tradeoff
here.

Post by Marco d'Itri

Post by Russ Allbery
strategy of (for example) moving each binary out of /bin manually, but

As I explained, this cannot really work.

Of course it can work. It will just take forever and produce a bunch of
minor bugs along the way and won't be something you'll be able to wait for
before tackling other goals.

In reality, it might well mean just giving up on merging entirely because
people won't be willing to sustain the effort over that length of time.

Personally, I consider separate /bin and /usr/bin to be a historic
artifact whose remaining useful properties have become obsolete or been
replaced with other, better techniques (such as mounting /usr in
initramfs), and we'd all be better off following Red Hat and Arch, dealing
with the pain of merging every system, and then saving ourselves the time
and energy we spend on this separation. It definitely makes some system
design patterns simpler. I also agree that this isn't a diversity
question; so far as I know, there aren't any remaining viable use cases
for split /bin and /usr/bin that work in Debian today or have any hope of
working in a Debian of the future. /bin is already useless without /usr
mounted, and is basically certain to remain so. The remaining argument is
just that the transition is difficult and may not be worth the effort.

But it's not just my opinion that matters. I think we need to decide this
somehow as a project, whether via the TC or via GR or something, because
there's a real disagreement here over whether we can or should
force-upgrade all Debian systems and I don't believe doing something short
of that is going to be supportable.

[Jeremy Bicha]

Post by Russ Allbery
But it's not just my opinion that matters. I think we need to decide this
somehow as a project, whether via the TC or via GR or something, because
there's a real disagreement here over whether we can or should
force-upgrade all Debian systems and I don't believe doing something short
of that is going to be supportable.

If we are going to have a vote, we don't have a lot of time. I think
that if Debian were to choose to convert all systems to usrmerge for
the Buster release, it ought to be complete in Buster before the
Transition Freeze which is scheduled for January 12. [1]

[1] https://release.debian.org/

Thanks,
Jeremy Bicha

[Russ Allbery]

Post by Jeremy Bicha

Post by Russ Allbery
But it's not just my opinion that matters. I think we need to decide
this somehow as a project, whether via the TC or via GR or something,
because there's a real disagreement here over whether we can or should
force-upgrade all Debian systems and I don't believe doing something
short of that is going to be supportable.

If we are going to have a vote, we don't have a lot of time. I think
that if Debian were to choose to convert all systems to usrmerge for
the Buster release, it ought to be complete in Buster before the
Transition Freeze which is scheduled for January 12. [1]
[1] https://release.debian.org/

We could also skip buster, hold usrmerge out of the buster release as a
not-yet-supported configuration (or ship it with a bunch of disclaimers; I
have no strong opinion either way), and then aim for the next release, and
in the meantime double down on trying to get as much testing of usrmerge
systems as possible.

That's going to be a disappointing delay for a lot of people, I'm sure,
but it's still better for them than never doing this at all.

[Michael Stone]

Post by Jeremy Bicha

Post by Russ Allbery
But it's not just my opinion that matters. I think we need to decide this
somehow as a project, whether via the TC or via GR or something, because
there's a real disagreement here over whether we can or should
force-upgrade all Debian systems and I don't believe doing something short
of that is going to be supportable.

If we are going to have a vote, we don't have a lot of time. I think
that if Debian were to choose to convert all systems to usrmerge for
the Buster release, it ought to be complete in Buster before the
Transition Freeze which is scheduled for January 12. [1]

No way we can hit that date with a forced merge. Where are the testing
reports showing that this actually works on a broad range of systems?
And now we're getting into the winter holiday season? Pushing that now
would be nuts.

[Ian Jackson]

Post by Michael Stone
No way we can hit that date with a forced merge. Where are the testing
reports showing that this actually works on a broad range of systems?
And now we're getting into the winter holiday season? Pushing that now
would be nuts.

I would also like to point out that change planning is the job of
someone who wants to change something. That includes not only:
- figuring out what the new state should look like
- developing tools to switch individual systems etc.
which has been done but also:
- planning what changes need to happen throughout Debian and
its ecosystem, in what order, to avoid lossage
- writing that plan up and getting it reviewed by everyone who
is likely to have useful input
- specifically covering what kinds of residual lossage can be
expected, and what the plan is for handling that
- in a case like this, getting at least a semiformal sign-off
from the Release Team
- throughout, engaging positively with sceptics, to reassure
them (rather than pressing on or dismissing concerns)
none of which seem to have been done.

We should back off at the very least until we have a transition plan
which we can discuss and possibly get rough consensus on. Personally
what I have seen so far gives me little confidence.

Ian.

[Marco d'Itri]

Post by Ian Jackson
I would also like to point out that change planning is the job of

I planned for everything that I could foresee.
I did not think about the buildds issue, but this is hardly the first
time that some complex software/process has bugs that nobody tought
about until it was deployed.
Actually I believe that the fact that this could be solved quickly and
with a trivial change is a great argument in favour of the quality of my
plan and work for switching to merged-/usr.

Post by Ian Jackson
We should back off at the very least until we have a transition plan
which we can discuss and possibly get rough consensus on. Personally
what I have seen so far gives me little confidence.

You should have asked for an explicit plan three years ago when I first
announced that I was working on this. At this point you are just
creating arbitrary requirements that would delay forever this change.

The reality is that the usrmerge conversion program has been available
for three years and all the conversion problems that were unconvered by
users have been addressed.
If you have further doubts about its general viability then it is up to
you to report specific issues.

[Michael Stone]

Post by Marco d'Itri
You should have asked for an explicit plan three years ago when I first
announced that I was working on this. At this point you are just
creating arbitrary requirements that would delay forever this change.

Three years ago we were told this was to enable people to optionally do
something, not that all users would be forced to migrate. That's a
pretty substantial change.

[Marco d'Itri]

Post by Michael Stone
Three years ago we were told this was to enable people to optionally do
something, not that all users would be forced to migrate. That's a pretty
substantial change.

At this point there is no plan to force anybody to migrate. It has just
been argued that it would be more convenient.

[Vincent Bernat]

Post by Marco d'Itri
Actually I believe that the fact that this could be solved quickly and
with a trivial change is a great argument in favour of the quality of my
plan and work for switching to merged-/usr.

Thank you for that! My workstation was switched to merged-/usr without
issue three years ago. It is running the same Debian unstable
installation since 2002.

[Holger Levsen]

Post by Russ Allbery
But it's not just my opinion that matters. I think we need to decide this
somehow as a project, whether via the TC or via GR or something, because
there's a real disagreement here over whether we can or should
force-upgrade all Debian systems and I don't believe doing something short
of that is going to be supportable.

why do you think we need to decide it via TC or GR, without letting
whoever is responsible for the affected packages decide?

How is that any different (from a procedural POV) than say changing the
default gpg keylength?

(Or are you just forseeing that someone will want to go to the TC?)

As a user, I couldnt care less if usrmerge is enabled or not. 0 visible
differences. /bin is a link to /usr/bin, the sun still rises in the
morning.

Or what am I missing?

[Michael Stone]

Post by Holger Levsen
Or what am I missing?

The possibility that your system will break? The current usrmerge
package has no test mode, will bail with a partially-converted system if
it runs into problems, and has no way to revert the process. A sysadmin
putting the same file into /bin and /usr/bin will cause the process to
abort. I've seen a heck of a lot of systems in the wild where someone
copies a file all over the place because some stupid 3rd party software
requried it or told them it was a good idea or because they weren't
quite sure what the problem was and maybe this would fix it.
Historically debian has had a well-deserved reputation for very reliable
upgrades--let's not mess this up for something with such ambiguous
benefits as a forced usrmerge.

If there has been serious discussion of making this mandatory before
now, I missed it. https://wiki.debian.org/UsrMerge itself calls just
*recommending* the process a future item.

[Russ Allbery]

Post by Holger Levsen

Post by Russ Allbery
But it's not just my opinion that matters. I think we need to decide
this somehow as a project, whether via the TC or via GR or something,
because there's a real disagreement here over whether we can or should
force-upgrade all Debian systems and I don't believe doing something
short of that is going to be supportable.

why do you think we need to decide it via TC or GR, without letting
whoever is responsible for the affected packages decide?

Because we have no idea what packages are affected (possibly a lot). It's
not just packages that ship files in /lib, /bin, and /sbin (although
that's already lots); it's every package that uses something in those
directories.

Personally, I think the chances of breakage are reasonably small and I
think Marco has already done a ton of work to try to find that breakage,
but it's a very sweeping, global change, which is what makes it
challenging.

Post by Holger Levsen
(Or are you just forseeing that someone will want to go to the TC?)
As a user, I couldnt care less if usrmerge is enabled or not. 0 visible
differences. /bin is a link to /usr/bin, the sun still rises in the
morning.

If we can decide via consensus that no one objects to the goal and it's
just a matter of getting enough testing to make usrmerge essential (and
working out the timing for when we have enough testing), great. Given the
general trend of the threads, I'm dubious we'll get there, but consider me
+1 for that position if it helps.

[Guillem Jover]

Hi!

Post by Adam Borowski
• making usrmerge Essential (large amount of effort, known issues)

Oh dear, no, please!

First off, as I've said in the past, I have no problem whatsoever with
the concept, I think it brings both advantages (cleans up the FHS,
makes some things easier, etc) and disadvantages (makes Debian perhaps
a worse development environment for upstreams that want to target other
systems, easier to miss pathnames that should not be under /usr leaking
over, or a proper conversion making it not possible to keep unmerged
/usr for those that would prefer that, for whatever reason, etc).

As a proof-of-concept to make it possible to test the thing, to unearth
bugs, as a quick way to deploy on throwaway containers, or even as a
pragmatic way to switch ones system if there's a need for that no
matter what, right now, the usrmerge package is great.

But I do have a very big problem with us even considering using it as
a way to deploy this change as part of our standard installation or
upgrade. This is a hack, it's subverting the dpkg view of the world,
and while this is intended to be supported somehow from dpkg PoV, I
consider that to be on best-effort basis. For example I still have
not merged the branch [Q] to make dpkg-query cope with the symlink
stuff, as I was waiting for some feedback, and it still has some
conceptual problems. Making usrmerge Essential is the "we cannot be
bothered to do a proper transition and move things to their proper
place" kind of plan. :(

[Q] <https://git.hadrons.org/git/debian/dpkg/dpkg.git/log/?h=pu/query-map-pathnames>

Please, if we decide we want to do merged /usr, let's do it properly.
I'm still quite appalled that debootstrap has been change *again*
(after it got reverted for stretch) to default to use this method for
new installations…

Thanks,
Guillem

[Marco d'Itri]

On Nov 20, Adam Borowski <***@angband.pl> wrote:

If you are seriously concerned with the small issuses caused by the
transition to merged-/usr then I have a better proposal.
Plan C: just stop supporting non-merged-/usr systems since these
problems are caused by having to support both, and there is no real
benefit in doing that other than pleasing the few people who are scared
by changes.

Post by Adam Borowski
* move binaries to /usr/bin one by one, without hurry. If it takes 10
years, so what?

If it takes 10 years then we will have to wait 10 years to deliver an
enabling technology for important features.
Also, I seriously question that this would be practical: moving a binary
requires coordination with all other packages using it, while the switch
to a merged-/usr is transparent.
If you believe that a 10 years timeframe for a change is totally OK then
you obviously do not care about it, so you what you are really arguing
for is doing nothing.

Post by Adam Borowski
* /bin would be left populated with nothing but /bin/sh, /bin/bash and
whatever else POSIX demands.

There are no benefits from a merged-/usr scheme as long as there are
system binaries outside of /usr.

Post by Adam Borowski
Another question is, why?

It has been documented here long ago: https://wiki.debian.org/UsrMerge .

Post by Adam Borowski
main reason is compatibility with Solaris -- which hasn't been relevant for

No, it's not the main reason. It's not even an interesting reason, it's
just an example showing that this kind of setup has been tested for
years and is nothing new.
You are misconstructing the arguments in favour of merged-/usr to be
able to dismiss them easily.

Post by Adam Borowski
a long long time. Even the other distribution (Fedora) that has done the
split is rapidly following Solaris into obscurity (the whole RPM world has
gone to 20% of web servers from 65% a few years ago, other server uses seem

WTF? Fedora is not relevant, it's RHEL that matters and it switched to
a merged-/usr. If you are seriously claiming that RHEL is "fading into
obscurity" then we obviously lack a common ground to discuss anything.

Post by Adam Borowski
# Fact: The /usr merge makes sharing the vendor-supplied OS resources
# between a host and networked clients as well as a host and local
# light-weight containers easier and atomic. Snapshotting the OS becomes a
# viable option. The /usr merge also allows making the entire
# vendor-supplied OS resources read-only for increased security and
# robustness.
-- which is untrue as a system with /etc and /var out of sync with /usr is
broken. There are attempts of reducing /etc but I have seen nothing about
/var.

These are few examples of the features that a merged-/usr system
enables. /etc and /var do not just get "out of sync", so your argument
is wrong.

[Adam Borowski]

Post by Marco d'Itri
If you are seriously concerned with the small issuses caused by the
transition to merged-/usr then I have a better proposal.
Plan C: just stop supporting non-merged-/usr systems since these
problems are caused by having to support both, and there is no real
benefit in doing that other than pleasing the few people who are scared
by changes.

Yes, that's much better than status quo. Plenty of risks and unneeded
work, but it at least has a chance. Supporting both is madness.

Post by Marco d'Itri

Post by Adam Borowski
* move binaries to /usr/bin one by one, without hurry. If it takes 10
years, so what?

If it takes 10 years then we will have to wait 10 years to deliver an
enabling technology for important features.

Uh... what "enabling technology", what "important features"?

Despite repeated requests, I have yet to hear a compelling reason. All the
talk is about "how" not "why".

A cost-vs-benefits analysis is needed.

Post by Marco d'Itri
Also, I seriously question that this would be practical: moving a binary
requires coordination with all other packages using it, while the switch
to a merged-/usr is transparent.

Well, can use a symlink if you really want to move a binary. It could use
a common debhelper script. But I have doubts even that makes much point.

Post by Marco d'Itri
If you believe that a 10 years timeframe for a change is totally OK then
you obviously do not care about it, so you what you are really arguing
for is doing nothing.

If you want me to care about it, please explain what can I gain.

Post by Marco d'Itri

Post by Adam Borowski
* /bin would be left populated with nothing but /bin/sh, /bin/bash and
whatever else POSIX demands.

There are no benefits from a merged-/usr scheme as long as there are
system binaries outside of /usr.

And those benefits would be...?

There's a reference to "system snapshotting" for which merged-/usr is
neither sufficient nor necessary. I religiously use system snapshotting for
many years, without issues.

Post by Marco d'Itri

Post by Adam Borowski
Another question is, why?

It has been documented here long ago: https://wiki.debian.org/UsrMerge .

It talks a lot about "how" without a word "why".

Post by Marco d'Itri

Post by Adam Borowski
main reason is compatibility with Solaris -- which hasn't been relevant for

No, it's not the main reason. It's not even an interesting reason, it's
just an example showing that this kind of setup has been tested for
years and is nothing new.
You are misconstructing the arguments in favour of merged-/usr to be
able to dismiss them easily.

Sure, please then provide a man not built of straw whom I can fight.

Post by Marco d'Itri

Post by Adam Borowski
a long long time. Even the other distribution (Fedora) that has done the
split is rapidly following Solaris into obscurity (the whole RPM world has
gone to 20% of web servers from 65% a few years ago, other server uses seem

WTF? Fedora is not relevant, it's RHEL that matters and it switched to
a merged-/usr. If you are seriously claiming that RHEL is "fading into
obscurity" then we obviously lack a common ground to discuss anything.

I am seriously claiming that RHEL is in the place Solaris was in 2010.
Rapidly falling user share (like Solaris, it was ubiquitous in the past!),
acquired by a company known for wringing dry a small number of lucratious
customers -- just like Oracle. This very scenario has repeated in the past
for a number of other Unices. Some developers (including The Lennart)
already sound like they're mulling jumping ship <rumour warning>.

If going from 65% to 20% usage (including Fedora, CentOS and SLES) in the
only category we can reasonably measure isn't "serious", I don't know what
is.

But, we're not RHEL, we're not Fedora. We're not even Devuan nor Ubuntu,
so there's no need to look at anything but: "will this benefit us? our
users?".

Post by Marco d'Itri

Post by Adam Borowski
# Fact: The /usr merge makes sharing the vendor-supplied OS resources
# between a host and networked clients as well as a host and local
# light-weight containers easier and atomic. Snapshotting the OS becomes a
# viable option. The /usr merge also allows making the entire
# vendor-supplied OS resources read-only for increased security and
# robustness.
-- which is untrue as a system with /etc and /var out of sync with /usr is
broken. There are attempts of reducing /etc but I have seen nothing about
/var.

These are few examples of the features that a merged-/usr system
enables. /etc and /var do not just get "out of sync", so your argument
is wrong.

Ok, please tell us more about those features then. "Why" not "how".


(But at least we both agree there's no way to support both merged and
unmerged -- so this flamewar already made a step forward.)


Meow!

[Matthew Vernon]

Hi,

Adam Borowski <***@angband.pl> writes:

FTR, I agree with the broad thrust of your mail; I'm not sure I've seen
a convincing case for /usr-merge yet.

Post by Adam Borowski
I am seriously claiming that RHEL is in the place Solaris was in 2010.

But I want to take issue with this. RHEL is moderately-widely used,
because if you want to get RH support (e.g. for OpenStack) then you have
to be running RHEL. It's interesting to note what RHEL are up to, I'd
say, but I wouldn't go out of my way to be compatible with them - after
all, there's a reason we're .deb based not .rpm ;-)

I'd suggest we talk about the benefits or otherwise of /usr-merge and
leave RHEL out of it.

Regards,

Matthew

[Stephan Seitz]

Debian, is, btw, also losing quickly for not keeping pace with the
world around it.

Or maybe it scares people away with its bullshit decisions.

Stephan

[Svante Signell]

On Wed, 21 Nov 2018 10:52:22 +0100, Adam Borowski

Post by Adam Borowski
I am seriously claiming that RHEL is in the place Solaris was in 2010.
Rapidly falling user share (like Solaris, it was ubiquitous in the past!),
acquired by a company known for wringing dry a small number of lucratious
customers -- just like Oracle. This very scenario has repeated in the past
for a number of other Unices. Some developers (including The Lennart)
already sound like they're mulling jumping ship <rumour warning>.

RHEL is also the template for CentOS which has a huge user base.
Debian, is, btw, also losing quickly for not keeping pace with the
world around it.

In my opinion it is the other way around, rpm-based systems are decreasing.

YMWV

[Jeremy Stanley]

Post by Svante Signell

On Wed, 21 Nov 2018 10:52:22 +0100, Adam Borowski

Post by Adam Borowski
I am seriously claiming that RHEL is in the place Solaris was
in 2010. Rapidly falling user share (like Solaris, it was
ubiquitous in the past!), acquired by a company known for
wringing dry a small number of lucratious customers -- just
like Oracle. This very scenario has repeated in the past for
a number of other Unices. Some developers (including The
Lennart) already sound like they're mulling jumping ship
<rumour warning>.

RHEL is also the template for CentOS which has a huge user base.
Debian, is, btw, also losing quickly for not keeping pace with
the world around it.

In my opinion it is the other way around, rpm-based systems are decreasing.

At least for the corner of the tech sector I've been spending time
in recently, it seems like persistent systems are becoming
increasingly rare and piecemeal package-based software distributions
in general are being abandoned (or at best becoming implementation
details in some image build automation). Pointing fingers at other
distros isn't productive behavior, and certainly isn't a way to keep
ours relevant. It's like fighting over the last slice of cake on a
sinking ship.

[Jonathan Dowland]

Did RHEL do that from 7.x to 7.x+1 (where updates are supported)? Or
from x.y to z.0 where ther recommended migration way is a reinstall?

I don't know the answer to when or how this happens, but my local RHEL
VM is on 7.5 and doesn't have a merged /usr. It was, however, running a
much earlier version of 7.x originally. I can't remember which.

I'll upgrade to 7.6 and see what happens.

[Jonathan Dowland]

Post by Jonathan Dowland
I'll upgrade to 7.6 and see what happens.

I've upgraded and /bin -> /usr/bin

However I didn't correctly check this before so it might have already
happened.

[Ian Jackson]

Post by Marco d'Itri

Post by Adam Borowski
Another question is, why?

It has been documented here long ago: https://wiki.debian.org/UsrMerge .

I looked at that page and it has no rationale.

Post by Marco d'Itri
You are misconstructing the arguments in favour of merged-/usr to be
able to dismiss them easily.

I suspect you meant to refer to this:
https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/

I have read that. IMO the arguments are extremely diffuse and
certainly do not justify the level of effort necessary to do this in a
rapid fashion (or, IMO, at all).


In this context it is important to realise that there would be serious
social costs to pressing ahead with making this mandatory: rightly or
wrongly, a significant minority of users and downstreams would
strongly dislike this change. The nastiness and fighting which will
result if we press forward with this would far outweigh any minor
technical benefits of this simplification.

(In this context I want to repeat what Adam said about mounting /usr
in the initramfs. That has removed the practical need to care, most
of the time, about whether a particular thing is in / or /usr. So we
have already got most of the techical benefits.)


I have a perception that this change is part of a programme of
`tidying up'. I have seen controversial changes proposed elsewhere,
which also seem largely motivated by a sense of `tidiness'.

As a general rule I think this is harmful. `Tidying things up' is
fine if no-one cares about them. If someone does care about them then
usually it is better to keep them, even if it is a small amount of
extra work, to maintain Debian as the most flexible base for
everyone's work.


I don't intend to do a point-by-point rebuttal of the whole
`TheCaseForTheUsrMerge' page, but it leads with `Compatibility', so I
will deal with the section `Compatibility: The Gist of It'.

1. "... scripts/programs written for other Unixes or other Linuxes and
ported to your distribution will no longer need fixing for the file
system paths of the binaries called, which is otherwise a major
source of frustration ..."

The author of this text is obviously extremely easily frustrated.
Personally I have very rarely encountered any situation where the fact
that a particular binary was in /bin vs /usr/bin was any source of
frustration or even any trivial difficulty. For interpreters, there
is generally a conventional #! line that everyone uses. Most of the
rest of the time programs use PATH.

There *is* trouble arounding executable locations: but it isn't / vs
/usr. It's sbin vs libexec; and /usr vs /usr/local. But usrmerge
does nothing to help those.

For modern programs everything tends to be in /usr[/local] anyway; for
older programs the compatibility arrangements are already in place.

2. Solaris compatibility

Well, this is a real argument. But, seriously, do we care ?

3. "Maintaining the /usr split requires non-trivial project-specific
handling in the upstream build system".

I am not sure exactly what this point is. AFAICT it is simply
complaining that if your package wants to ship files in / rather than
/usr, you need to do something in your *.install files or whatever.
This is true. But it is not an argument in favour of making /bin a
symlink to /usr/bin.

Now that we mount /usr in the initramfs this can often be dispensed
with, along the lines of Adam's "plan B". This can be done on a
package-by-package basis as the maintainer considers that the costs of
maintaining things in /bin becomes too high. A maintainer can also
leave behind compatibility symlinks (the set of which is of course
closed).

4. "Improved compatibility with current upstream development"

I can't even figure out what this means. Upstream programs that want
to ship everything in /usr are not a problem.


Ian.

[Simon McVittie]

Post by Adam Borowski
There's been another large bout of usrmerge issues recently

These are one example of a wider class of issues, which Debian is going
to keep running into, and solving, as long as we're a binary distribution
(as opposed to a source distribution like Gentoo):

Many packages probe properties of the build system at build-time, and
then assume that those properties remain true for the host system.
(I'm using the usual GNU cross-compiling terms here: you build a package
on the build system and run it on the host system.)

Clearly, we value the advantages of being a binary distribution enough
that we're willing to deal with these issues (if we didn't, we'd be
using Gentoo or something).

Cross-compiling is the place where this becomes most obvious, but even
without cross-compiling, this class of issues includes -march=native:
assume that the host system's CPU will be the same as (or better than)
the build system's. It also includes "automagic" dependencies: assume
that the host system will (want to) have all the optionally-used runtime
libraries whose -dev packages happen to be present on the build system
(see recent Policy bugs discussing use of Build-Conflicts and/or explicit
--disable-foo to make this not happen).

Merged or unmerged /usr is just another one on the list of properties
of build systems that host systems are not guaranteed to match.

We already have a series of written or unwritten rules about what is
"close enough" that detecting it in the build system and assuming it will
be the same in the host system is OK, and what is not "close enough".
For instance, we build i386 packages for >= i686 hardware on SSE-capable
x86_64 CPUs, we build packages for end-user {stretch + stretch-updates
+ security} systems in chroots that only have stretch available, and
we build packages in chroots that have "full-fat" perl available and
expect them to work on systems with only the Essential perl-base; but
in general we don't expect to be able to build packages for sid systems
in stretch chroots.

I think we're going in the right direction with merged /usr, just not
always in the right order. Unintended consequences of debootstrap and
buildd changes meant that transitioning buildd chroots to be merged-/usr
accidentally happened early, but they should have been the very last
thing to be transitioned.

Note that the failure mode is one way round: building on a merged-/usr
system, and using on a non-merged-/usr system, can fail; but because
of the design of merged /usr (with the compat symlinks in /), I am not
aware of any examples of building on a non-merged-/usr system, and using
on a merged-/usr system, being problematic.

Post by Adam Borowski
* let's scrap the usrmerge package

I think you're conflating the usrmerge package with merged /usr. The
usrmerge package is one concrete implementation of a way to convert
existing installations to merged /usr, but you can also get a merged-/usr
system by installing into a skeleton filesystem where /bin, /sbin,
/lib* are already symlinks into /usr (as recent debootstrap does),
or by moving directories around yourself (as my prototype for making
Flatpak runtimes out of Debian packages does). This particular genie is
very much no longer in its bottle.

If we have any level of support for merged /usr, then the usrmerge
package is a really useful way to codify "here is how you transition from
unmerged to merged, if that's what you want to do". I used it for the
recent reproducible-builds test improvements to avoid having to build
separate unmerged-/usr and merged-/usr base tarballs, for instance.

Post by Adam Borowski
a system with /etc and /var out of sync with /usr is
broken. There are attempts of reducing /etc but I have seen nothing about
/var.

This is partially true, but isn't the whole story. A system or container
needs its /etc and /var to be *based on* one that is "close enough" to its
static files, but the point of /etc and /var being separate from the
static files (/usr, in a merged-/usr system) is that they are mutable.

One common deployment model for machines with an immutable /usr is
for /usr to contain a starting point for the system's /etc and /var
(in OSTree the recommendation is /usr/etc and /usr/var, in Lennart's
stateless-system design article it's /usr/share/factory/{etc,var}), and
for the system to populate /etc and /var from those if they don't already
exist. I think OSTree even has infrastructure for doing a three-way
merge of (old /usr/etc, new /usr/etc, deployed /etc), although I don't
know how well it works in practice.

Efforts towards making stateless systems have tended to concentrate on
/etc over /var because /etc contains more critical files for a minimal
system: you can get by without /var/lib/dpkg if you don't plan to upgrade,
install, remove or list packages, but you won't usually get far without
at least /etc/fstab, /etc/nsswitch.conf and /etc/passwd.

It's also often not necessary in practice for /etc and /var to *precisely*
match the installed packages: if that was required, then backing up and
restoring /etc and /var couldn't work (not even well enough to reinstall
the packages to get back to being in sync), and dpkg's default to keep
sysadmin-modified conffiles would be problematic. The more special-purpose
and limited the root, the less /etc and /var you will need, and the
less closely they need to match the static files: a complex development
desktop system needs a lot of /etc and /var, and needs them to match the
static files rather closely, but the same is not true for a single-purpose
container or embedded device that gets upgraded atomically.

I've noticed that upstreams who are interested in stateless systems
tend to be moving towards /etc and /var being as empty as possible, as
decoupled from the static files as possible, and populated as "lazily"
as possible, with files created as they are needed rather than up-front,
and re-created as required if missing, rather than assuming that a
maintainer script that ran once at install time did it. systemd-tmpfiles
(see tmpfiles.d(5)) is very helpful when writing in this style.

Post by Adam Borowski
Another question is, why?

There are a few reasons, and which ones are the most significant depend
which of several related questions you're asking. You might be asking
why we want merged /usr to be *possible*; or you might be asking why we
want it to be encouraged/default on end user systems; or you might be
asking why we want it on Debian developer systems. (Of course, all
Debian developers are Debian users, and some Debian users are Debian
developers, so there is overlap.)

There are probably more reasons than I know about, but here are the
ones that spring to mind:

Making merged /usr possible
===========================

The work necessary to allow building and booting a merged-/usr Debian
system can be summed up as: don't assume that /bin and /usr/bin (etc.)
are distinct directories.

As a starting point, I hope we agree that /bin, /sbin, /lib* and /usr
(excluding /usr/local which is special) are all more similar than they
are different? They're all essentially the same class of files: static,
read-only during normal operation, and replaced as atomically as possible
during upgrades instead of being modified incrementally. Unlike /etc,
they are not modified by the sysadmin; unlike /var, they are not modified
by automated systems, other than OS upgrade mechanisms; and unlike both
/etc and /var, their contents depend only on the set of package versions
installed, not on the identity and history of the system (in principle
a particular {package: version} map should result in a reproducible
merged-/usr regardless of how many upgrades took place between initial
installation and the current situation, although I'm sure there are
bugs). So it makes some conceptual and operational sense to bundle them
together as a single atomic unit. (This is why people who want a simpler
FHS have chosen to advocate merged /usr, and not the opposite approach
that Debian hurd-i386 tried for a while, in which /usr was a symlink to /
and static files were unified into the root filesystem.)

/usr/local can be an exception, depending on how it's used, but in
containers and embedded systems it often doesn't make sense at all,
and on systems where a non-empty /usr/local makes sense it can be a
separate filesystem, or a symlink to /var/usrlocal or similar, if
desired.

We want merged /usr to be possible because it's a significant
simplification for reliable special-purpose systems. If you're
developing a consumer "appliance" that needs to be used by people who
don't know it runs Unix, or a piece of infrastructure that needs to
be reliable, then it needs to minimize opportunities for filesystem
corruption, and be able to recover as gracefully as possible.

One way to do that, if you don't need persistent OS-level state or
configuration changes, is to have a completely stateless system. In
the initramfs, mount a tmpfs as the new root, create a simple skeleton
filesystem, mount /usr in it, populate enough of /etc and /var to operate
(perhaps by copying from immutable template files in /usr), maybe mount
/home or /srv for user data if that makes sense, and pivot to the new
root. This is a lot simpler if /usr is all one blob and you don't need
to bring in /bin, /sbin, /lib* from somewhere else.

Another way is to populate a persistent root filesystem where
configuration and state can be changed, but be prepared to reformat it if
it gets corrupted. Mounting /usr read-only minimizes opportunities for
filesystem corruption, but that works best if crucial binaries in /bin,
/sbin, /lib are also on /usr (or some parallel read-only filesystem,
but why would you want the complexity of two read-only filesystems that
have to be kept in sync across upgrades when you could just have one?)
If the root filesystem gets corrupted, or if the user just wants to
reset configuration and state to the "factory" situation, the recovery
path is to reformat the root filesystem and re-populate it from a
template.

In either of those situations, for best robustness you'll probably
want to upgrade using "A/B" /usr filesystems, as seen in OSTree, rauc,
recent Android phones and so on: boot from filesystem A, download and
install the new system into filesystem B, reboot into filesystem B,
wait for the next update to be released, install it into filesystem
A, reboot into filesystem A and repeat. This means you always have a
known-working filesystem to fall back on (if the most recently updated
filesystem doesn't boot, use the other one), and is much simpler if all
your static files are in a single /usr filesystem. If you have persistent
state then you'll also need at least one persistent root filesystem,
perhaps also in an A/B pair; but that's still better than having /bin,
/sbin, /lib* also be separate, or mixing up the static /bin, /sbin,
/lib* with your persistent state.

Another reason to want merged /usr is if you have a large number
of containers (Docker, lxc, Flatpak, whatever) with the same static
files but distinct persistent state. The static files can be shared
to save space, but that's only safe if you bind-mount them into the
container as a read-only filesystem, so that a compromised container
can't modify other containers' static files. The state is "personal"
to the container, so it shouldn't be shared: even though it *started*
as a copy of a template /etc and /var that (as you said) correspond
to the static files, it has probably been modified while running the
container. If we want Debian to be a suitable base for such containers,
then merged /usr needs to be possible. (Note that this does not require
the host system to also be merged-/usr.)

Finally, the situations below can't work unless merged /usr is possible.

Merged /usr on end-user systems
===============================

The main reason to prefer merged /usr *on end user systems* is that it's a
simplification that removes avoidable bugs. You wouldn't design a machine
that is mostly held together with 12mm bolts, but uses half-inch bolts
for a few components. In particular, this gives us an advantage that I
will admit is somewhat circular: it makes packages work as intended,
even if they are not using the canonical paths that would exist on
a non-merged-/usr system. A package with a #!/usr/bin/sh script, or
conversely, a #!/bin/perl script, would fail on unmerged-/usr but would
work fine on a merged-/usr system. This removes a class of avoidable
bugs: you don't have to know that sh is canonically in /bin and perl is
canonically in /usr/bin, and neither does your upstream. These bugs will
tend to be especially prevalent in software developed by an upstream
author on a merged-/usr system. As long as we support non-merged-/usr,
they are bugs, and we can put work into reporting them and fix them like
any other bug; but if we can declare them not to be bugs, then a class
of work that would otherwise need to be done goes away, and we can spend
our time solving more interesting problems.

Another reason to want merged /usr on end user systems is that we're
increasingly seeing container technologies that remap the filesystem,
like bubblewrap (which, for example, is used to sandbox GNOME
thumbnailers so that file parsing bugs in thumbnailers can't result
in arbitrary code execution with full privileges). This typically reuses
the host system's static files, mounted read-only in the sandbox, which
is much more straightforward if the host system's static files can
all be found in /usr: you don't need to bind-mount /bin, /sbin, /lib*
separately. Straightforward code is more likely to get written (I hope
we can agree that sandboxing thumbnailers is better than not sandboxing
them) and is more likely to be bug-free.

Another, more minor, reason to support (or even recommend or require)
merged /usr is that it sends an undeniable message that booting without
/usr is definitely no longer something that we aim to support.

Finally, using merged /usr on end user systems proves that it is, and
continues to be, possible (in general, things that aren't regularly tested
don't work).

Merged /usr on developer systems
================================

Developers are users, so the reasons above apply. In particular, many
developers use the next release of Debian, so using merged /usr on
developers systems proves that it will continue to be possible in the
next Debian release.

If we reach a point where packages don't need to install any files to
the rootfs (all installed files going in /usr), then those packages'
maintainers benefit by not having to go behind their build system's back
to implement the /usr split (see systemd, dbus, and pre-buster versions
of glib2.0 for examples of packages that get extra complexity from the
need to separate files between the root and /usr; in systemd the
complexity is partially upstream, in dbus and glib2.0 it's all
downstream).

Why merged /usr instead of gradually moving files out of the root?
==================================================================

Post by Adam Borowski
* move binaries to /usr/bin one by one, without hurry. If it takes 10
years, so what?

Every time we move a file from the rootfs to /usr (let's say we move
/bin/ping to /usr/bin/ping), we have a risk of bugs where a dependent
package has hard-coded the old canonical path (in this case /bin/ping)
and now needs to use the new canonical path.

On a merged-/usr system, both /bin/ping and /usr/bin/ping work
equally well; hard-coding either of those paths will work. (Of course,
hard-coding the one that doesn't exist on a non-merged-/usr system won't
work on a non-merged-/usr system, but merged-/usr systems aren't going
to solve bugs that only exist on non-merged-/usr systems by some sort
of action-at-a-distance.)

We can avoid those bugs by creating a symbolic link, until /bin etc. are
entirely populated by symbolic links into /usr/bin etc., but then
we've taken a long time, a lot of maintainer-script code, and probably
a lot of NMUs to arrive at something that is functionally equivalent to
merged /usr being mandatory, without the simplicity of merged /usr's O(1)
symlinks in the root directory. Taking a long time and higher complexity
to achieve the same functional result does not sound very appealing to me.

In some OSs, like Fedora and Arch Linux, there was a flag day that made
merged /usr go from unsupported to mandatory, and that was that. Of
course, being Debian, we have made life hard for ourselves by supporting
both ways for an extended period of time.

Post by Adam Borowski
* /bin would be left populated with nothing but /bin/sh, /bin/bash and
whatever else POSIX demands.

It's not just about what POSIX demands, but also about what
interoperability with other distros and OSs requires. #! in scripts is
one of the problem cases where hard-coding absolute paths is required; we
can sidestep that with "#!/usr/bin/env perl", but as well as potentially
causing issues with locally-installed /usr/local/bin/perl, you'll notice
that has just swapped one absolute path for another :-)

On a merged-/usr system, both /bin/sh and /usr/bin/sh work; so do
/usr/bin/env and /bin/env. (In each case, the former path is the only one
that can be expected to work on a non-merged-/usr system.)

Away from #!, in general, upstreams (somewhat rationally) prefer
to hard-code absolute paths into installed files rather than
searching PATH at runtime, because their target platforms are not
all as sensible as ours: they also have to cope with platforms
where /bin/sed is old/bad/broken and you actually want to use
/opt/gnu/latest/new/newer/bin/gsed or something, but you can't just
add /opt/gnu/latest/new/newer/bin to PATH because that would break OS
scripts that rely on the behaviour of /bin/sed.

smcv

[Holger Levsen]

Hi Simon,

On Wed, Nov 21, 2018 at 02:05:42PM +0000, Simon McVittie wrote:
[...]

Post by Adam Borowski
Another question is, why?

[...]

thank you very much for explaining in detail why usrmerge is sensible
and the right thing to do for Debian, the universal OS.

I'll link your mail on wiki.debian.org/UsrMerge now and encourage
everyone to read it.

[Svante Signell]

Post by Holger Levsen
Hi Simon,
[...]

Post by Adam Borowski
Another question is, why?

[...]
thank you very much for explaining in detail why usrmerge is sensible
and the right thing to do for Debian, the universal OS.
I'll link your mail on wiki.debian.org/UsrMerge now and encourage
everyone to read it.

Unfortunately Simon writes too long mails. Who can even thake the time to read
all these verbalism. Things could be written more condensed. Please, can
somebody summarize his verbosity! Maybe he is a politcian?

[Andreas Henriksson]

Post by Svante Signell
Unfortunately Simon writes too long mails. Who can even thake the time to read
all these verbalism. Things could be written more condensed. Please, can
somebody summarize his verbosity! Maybe he is a politcian?

Here's a dumbed down narrative for you:
You're all sadly uninformed, ignorant and clueless. Please let
me spell it out for you:
<... details people don't care about snipped...>
Can y'all now please stop posting stupid shit about this topic?

(And as if for nothing else this mail is proof of the answer is a
sound "NO!" from the peanut gallery.)

The only people who have time to read the full mail are the ones who
actually want to learn.

Regards,
Andreas Henriksson

PS. Thanks to smcv for being naive enough to think that providing
knowledge is a useful thing to do in a food/shit-fight. Please keep up
your useful work of trying raise the bar for discussions in debian.

PPS. For gods sake do NOT even think about CCing me on this thread.

[Ian Jackson]

Svante's message was pretty bad but yours is even worse.
Would everyone please stop it.

Ian.

[Philip Hands]

Svante Signell <***@gmail.com> writes:

...

Who can even thake the time to read all these verbalism.

This is a criticism of an attempt to educate, made from a position of
ignorance. I don't think we need we need this on our mailing lists.

Things could be written more condensed.

Given that Simon was replying to a mail that was complaining about a
lack of detailed justification, criticising him for providing a series
of detailed justifications is deranged. If you don't like reading about
the details, you should have killed the thread at the first mail.

...

Maybe he is a politcian?

I guess that was supposed to be a joke, but sadly it reads like an ad
hominem attack (as did your opening sentence for that matter: the one
about Simon writing long emails, which I trimmed in an attempt to
accommodate your tiny attention span).

Cheers, Phil.

[Jonathan Dowland]

Post by Svante Signell
Unfortunately Simon writes too long mails. Who can even thake the time
to read all these verbalism. Things could be written more condensed.
Please, can somebody summarize his verbosity! Maybe he is a politcian?

I don't think this is a constructive comment.

[Holger Levsen]

package: usrmerge
severity: wishlist

Post by Stephan Seitz

Post by Matthias Klumpp
at the moment, the only issues that are known when installing the
usrmerge package is when there are different binaries with the same
name in /bin and /usr/bin (or /sbin), and I really don't think that
this is actually a likely scenario.

I don’t know but I’ve seen enough strange installations.

indeed, life is strange.

Post by Stephan Seitz
It would be good if the usrmerge package would do a dry-run as part of the
installation. If there are duplicate files the list will be printed (or
mailed) and the installation will fail without breaking the whole upgrade
process. The the admin can fix the problem later.

filing this idea as a wishlist bug against the usrmerge package.

btw: what do you think about providing usrmerge in stretch-backports?