Discussion:
Improving the offline updates user experience
Stephen Gallagher
2014-09-12 14:46:46 UTC
Permalink
== The Problem ==

It is very common for users to have systems with encrypted root
partitions (or even just /var and /etc). This may be due to a personal
concern for their data or a corporate policy mandating full-disk
encryption. Disk encryption requires a password (or other more
complicated credentials) be be presented just after the kernel is
booted and before the drives can be mounted and their data read.

With the current implementation of the offline updates in Fedora, this
leads to a very unpleasant user experience when updating. We offer two
ways to perform offline updates in the default user environment of
Fedora Workstation: "Install updates and reboot" or "Install updates
and shut down".

With "Install updates and reboot", the behavior is as follows:

1) The system shuts down and initiates an ACPI reboot.
2) The system presents the kernel boot menu and then starts the
updater kernel.
3) The system presents the user with a password prompt for the disk
encryption (possibly more than one, if the system is configured with
different passwords for different partitions).
4) The offline updates occur.
5) The system shuts down and initiates an ACPI reboot.
6) The system presents the kernel boot menu and then starts the
standard (possibly updated) kernel.
7) The system presents the user with a password prompt for the disk
encryption (possibly more than one, if the system is configured with
different passwords for different partitions).
8) The system completes booting.

During this experience, the user has been required to enter their disk
encryption password *twice*. The same is true for the "Install and
shut down" case, except that the two passwords are separated by some
actual wallclock time.

== Proposed Improvements ==

We could significantly improve this situation by allowing the system
to drop directly from the interactive system into the updater
environment without doing a full reboot or relaunching the kernel.

Lennart, would it be possible to set up a special systemd target for
performing updates that would essentially stop all processes except
for systemd and then apply the updates?

In an ideal world, it would then also be possible after update is
completed to restore operation to the standard boot targets of systemd
so that the system comes back up without having to perform a total
reboot. The exceptional case would of course be that in which either
the kernel, libc or systemd[1] needed to be updated, in which case a
reboot could be performed.

In this scenario, we can reduce the number of encrypted disk
challenges to at most a single one, and that only if absolutely
minimal plumbing packages saw an update.

I'd very much like to hear from the plumbers on this matter.


[1] I'm told that this might not be necessary; that systemd can
re-exec itself to pick up its own updates. That would reduce the scope
presumably to "only the kernel" forcing reboots.
Nathanael d. Noblet
2014-09-12 15:16:40 UTC
Permalink
Post by Stephen Gallagher
== Proposed Improvements ==
We could significantly improve this situation by allowing the system
to drop directly from the interactive system into the updater
environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for
performing updates that would essentially stop all processes except
for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is
completed to restore operation to the standard boot targets of systemd
so that the system comes back up without having to perform a total
reboot. The exceptional case would of course be that in which either
the kernel, libc or systemd[1] needed to be updated, in which case a
reboot could be performed.
In this scenario, we can reduce the number of encrypted disk
challenges to at most a single one, and that only if absolutely
minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
Yeah, I almost never use the reboot & install method. 90% of the
packages being installed/updated seem foolish to need a reboot to
update. I typically do a yum update manually and then if I notice
glibc/kernel/systemd or other big packages do a reboot.

All my systems have disk encryption since some of our projects could
potentially include people's private information. The latest way of
updating is just plain annoying for an unknown gain.

I would *love* if it was improved.
--
Nathanael
Richard Hughes
2014-09-12 15:47:42 UTC
Permalink
Post by Nathanael d. Noblet
Yeah, I almost never use the reboot & install method. 90% of the
packages being installed/updated seem foolish to need a reboot to
update.
I've been called worse that foolish I guess...
Post by Nathanael d. Noblet
I typically do a yum update manually and then if I notice
glibc/kernel/systemd or other big packages do a reboot.
That's just not safe. Have you ever had firefox open and done a
firefox update? Widgets start disappearing, redraws start having weird
artifects and then after a little while it just crashes. Other
applications like LibreOffice behave the same. Anyone that says things
like "the old version of the library stays in memory" obviously hasn't
actually done much real-world GUI programming in the last decade, or
runs any kind of secure desktop system. The *only* way to do this
securely and safely in the system we have now is in a clean pre-boot
environment, which is sad and crap UX, but still nevertheless true.
When we have application sandboxing and a stable OS platform to use,
we can certainly do something more sane, but until then we're just
hacking around the problem.

What we could do is do updates on shutdown by basically killing
everything except PID 1, and then restart everything, but even then
that relies on no systemd or kernel updates being present. Solving the
real problem is much harder (OS + Apps split) but we also need to slow
the firehose of needless updates.

Richard
Nathanael d. Noblet
2014-09-12 16:33:13 UTC
Permalink
Post by Richard Hughes
Post by Nathanael d. Noblet
Yeah, I almost never use the reboot & install method. 90% of the
packages being installed/updated seem foolish to need a reboot to
update.
I've been called worse that foolish I guess...
Post by Nathanael d. Noblet
I typically do a yum update manually and then if I notice
glibc/kernel/systemd or other big packages do a reboot.
That's just not safe. Have you ever had firefox open and done a
firefox update? Widgets start disappearing, redraws start having weird
artifects and then after a little while it just crashes. Other
applications like LibreOffice behave the same. Anyone that says things
like "the old version of the library stays in memory" obviously hasn't
actually done much real-world GUI programming in the last decade, or
runs any kind of secure desktop system. The *only* way to do this
securely and safely in the system we have now is in a clean pre-boot
environment, which is sad and crap UX, but still nevertheless true.
When we have application sandboxing and a stable OS platform to use,
we can certainly do something more sane, but until then we're just
hacking around the problem.
So I don't use Firefox anymore but I do know back in the day if we had
FF open when we updated it would do a double request for each page/form.
However when updating we just restarted FF and it would work fine after
that. I've never noticed any other issues than FF but like I said I
don't use it anymore.

Granted that doesn't matter obviously we don't want that kind of
behaviour.

I am curious though. Everyone says the only way to do it securely and
safely is with nothing running. Why can't updates be applied with stuff
running prior to a reboot?
Reindl Harald
2014-09-12 16:37:07 UTC
Permalink
Post by Nathanael d. Noblet
So I don't use Firefox anymore but I do know back in the day if we had
FF open when we updated it would do a double request for each page/form.
However when updating we just restarted FF and it would work fine after
that. I've never noticed any other issues than FF but like I said I
don't use it anymore.
Granted that doesn't matter obviously we don't want that kind of
behaviour.
I am curious though. Everyone says the only way to do it securely and
safely is with nothing running. Why can't updates be applied with stuff
running prior to a reboot?
1 out of a million cases needs offline updates

really - the only good at it is that you can stick
at using YUM and decide what you have to do at your
own - rarely updates really require a reboot

* lsof | grep DEL | grep /usr and restart services on servers
* logout/login and the above on workstations

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140912/86ad649d/attachment-0001.sig>
Till Maas
2014-09-12 16:58:57 UTC
Permalink
Post by Reindl Harald
* lsof | grep DEL | grep /usr and restart services on servers
There is a convenient script called "needs-restarting" to do this.
Checkrestart on debian even suggest potential init scripts that can be
used to restart the services.
Reindl Harald
2014-09-12 17:09:44 UTC
Permalink
Post by Till Maas
Post by Reindl Harald
* lsof | grep DEL | grep /usr and restart services on servers
There is a convenient script called "needs-restarting" to do this.
Checkrestart on debian even suggest potential init scripts that can be
used to restart the services
never worked relieable here on multiple machines

it often showed nothing where i knew the thing
which should be restarted without looking and
"lsof" proved it

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140912/f4d10ae7/attachment.sig>
Miroslav Suchý
2014-09-15 07:50:36 UTC
Permalink
Post by Reindl Harald
never worked relieable here on multiple machines
it often showed nothing where i knew the thing
which should be restarted without looking and
"lsof" proved it
I am one of those guys who refuse to reboot after each upgrade (and it works for me) and needs-restarting is ugly and
insufficient to me.

Therefore I initiated this project:
https://github.com/FrostyX/tracer
http://copr.fedoraproject.org/coprs/frostyx/tracer/

It is still not finished and ready for announcement, but if you are looking for some other way than offline-upgrade,
this might be worth of participating.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
Richard W.M. Jones
2014-09-15 08:06:00 UTC
Permalink
Post by Miroslav Suchý
Post by Reindl Harald
never worked relieable here on multiple machines
it often showed nothing where i knew the thing
which should be restarted without looking and
"lsof" proved it
I am one of those guys who refuse to reboot after each upgrade (and
it works for me) and needs-restarting is ugly and insufficient to
me.
https://github.com/FrostyX/tracer
http://copr.fedoraproject.org/coprs/frostyx/tracer/
It is still not finished and ready for announcement, but if you are
looking for some other way than offline-upgrade, this might be worth
of participating.
It wasn't clear to me how tracer works for non-C programs.

However there was some Red Hat only discussion recently about how to
do this for Python programs, with minimal overhead. Below I'm just
reproducing a technique (untested) that I think will work for Python.

It requires a small patch to the Python interpreter, and a similar
patch to any other language interpreters (eg. Perl, Ruby).

Rich.

-------
For each module (*.py or *.pyc) that it imports, have it mmap the
first page of that file into its memory.

The mmap would be PROT_NONE because it's not actually used, and the
associated file descriptor should be closed.

This will appear in /proc/PID/maps, with a "(deleted)" flag if the
underlying file gets deleted (and hence the process needs restarting).

The cost should be almost nothing:

- 4K of virtual memory, no real memory
- an extra mmap syscall on import
- an extra segment in the kernel's VM AVL
-------

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
Miroslav Suchý
2014-09-15 08:24:38 UTC
Permalink
Post by Richard W.M. Jones
It wasn't clear to me how tracer works for non-C programs.
https://github.com/FrostyX/tracer/commit/4abfc4ecbc6d1d4cd89b7162e1ba3f63088db3ff

Which basicaly checkout output of `ps` and if there is e.g. python as executable, it will check for arguments and use those.
But I agree that interpreted languages are problem, because they open the file, read it and close the handler.
So there are no footsteps to track.
Post by Richard W.M. Jones
However there was some Red Hat only discussion recently about how to
do this for Python programs, with minimal overhead. Below I'm just
reproducing a technique (untested) that I think will work for Python.
It requires a small patch to the Python interpreter, and a similar
patch to any other language interpreters (eg. Perl, Ruby).
Rich.
-------
For each module (*.py or *.pyc) that it imports, have it mmap the
first page of that file into its memory.
The mmap would be PROT_NONE because it's not actually used, and the
associated file descriptor should be closed.
This will appear in /proc/PID/maps, with a "(deleted)" flag if the
underlying file gets deleted (and hence the process needs restarting).
- 4K of virtual memory, no real memory
- an extra mmap syscall on import
- an extra segment in the kernel's VM AVL
-------
Very nice.
Is there some bugzilla RFE report for this? Or should I file it?
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
Vít Ondruch
2014-09-15 08:57:13 UTC
Permalink
Every of the script is based on assumption that you already read some
library/unit whatever. But that is not enough. I wonder how you want to
detect that you need restart in case that I have something like this:

$ ls
foo.rb
bar.rb

$ cat foo.rb

def some_function
require 'bar'
end

And now

1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb

There is no universal and reliable way how to detect this scenario IMO.


Vít
Post by Richard W.M. Jones
Post by Miroslav Suchý
Post by Reindl Harald
never worked relieable here on multiple machines
it often showed nothing where i knew the thing
which should be restarted without looking and
"lsof" proved it
I am one of those guys who refuse to reboot after each upgrade (and
it works for me) and needs-restarting is ugly and insufficient to
me.
https://github.com/FrostyX/tracer
http://copr.fedoraproject.org/coprs/frostyx/tracer/
It is still not finished and ready for announcement, but if you are
looking for some other way than offline-upgrade, this might be worth
of participating.
It wasn't clear to me how tracer works for non-C programs.
However there was some Red Hat only discussion recently about how to
do this for Python programs, with minimal overhead. Below I'm just
reproducing a technique (untested) that I think will work for Python.
It requires a small patch to the Python interpreter, and a similar
patch to any other language interpreters (eg. Perl, Ruby).
Rich.
-------
For each module (*.py or *.pyc) that it imports, have it mmap the
first page of that file into its memory.
The mmap would be PROT_NONE because it's not actually used, and the
associated file descriptor should be closed.
This will appear in /proc/PID/maps, with a "(deleted)" flag if the
underlying file gets deleted (and hence the process needs restarting).
- 4K of virtual memory, no real memory
- an extra mmap syscall on import
- an extra segment in the kernel's VM AVL
-------
Rich.
Miroslav Suchý
2014-09-15 12:06:39 UTC
Permalink
Post by Vít Ondruch
Every of the script is based on assumption that you already read some
library/unit whatever. But that is not enough. I wonder how you want to
$ ls
foo.rb
bar.rb
$ cat foo.rb
def some_function
require 'bar'
end
And now
1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb
Indeed, would foo.rb and bar.rb comes from different packages, then there is really no way.
Post by Vít Ondruch
There is no universal and reliable way how to detect this scenario IMO.
Well, if you are operator of nuclear power plant, then I understand the need of reboot after each upgrade.

But *I* do not want to reboot after each upgrade. Those crashes will be 0.00001% of all crashes on my workstation, which
is less PITA than rebooting because I upgraded 'foo-doc' package.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
Richard Hughes
2014-09-16 08:50:17 UTC
Permalink
Post by Miroslav Suchý
But *I* do not want to reboot after each upgrade. Those crashes will be
0.00001% of all crashes on my workstation
I think you might change your mind when it's you're the one that has
to triage those ABRT-reported bugs. Also, consoling users with lost
data gets tiring really quick.

Richard
Reindl Harald
2014-09-16 09:04:08 UTC
Permalink
Post by Richard Hughes
Post by Miroslav Suchý
But *I* do not want to reboot after each upgrade. Those crashes will be
0.00001% of all crashes on my workstation
I think you might change your mind when it's you're the one that has
to triage those ABRT-reported bugs. Also, consoling users with lost
data gets tiring really quick
well, users which want full control typically disable
or even uninstall ABRT - as long offline updates are
not becoming mandatory all is fine

[harry at srv-rhsoft:~]$ rpm -qa | grep -i ABRT
[harry at srv-rhsoft:~]$

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140916/d7397ea1/attachment.sig>
Richard W.M. Jones
2014-09-15 12:28:56 UTC
Permalink
Post by Vít Ondruch
Every of the script is based on assumption that you already read some
library/unit whatever. But that is not enough. I wonder how you want to
$ ls
foo.rb
bar.rb
$ cat foo.rb
def some_function
require 'bar'
end
And now
1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its
needs properly in its RPM dependencies?

It would still have been broken even with a reboot.

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
Reindl Harald
2014-09-15 12:40:34 UTC
Permalink
Post by Richard W.M. Jones
Post by Vít Ondruch
1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its
needs properly in its RPM dependencies?
It would still have been broken even with a reboot
no - why should it?

'foo' is loaded in memory, updated and now has different dependencies
no longer require 'bar.rb' but your running version still do

yes, such corner cases are possible, regardless no reason
for go the windows way and reboot after any minor update

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140915/c2d070a3/attachment.sig>
Richard W.M. Jones
2014-09-15 12:44:10 UTC
Permalink
Post by Reindl Harald
Post by Richard W.M. Jones
Post by Vít Ondruch
1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its
needs properly in its RPM dependencies?
It would still have been broken even with a reboot
no - why should it?
'foo' is loaded in memory, updated and now has different dependencies
no longer require 'bar.rb' but your running version still do
Please read closely. 'foo' has *not* been updated.

If 'foo' had been updated, we would have spotted it and restarted that
process using my technique outlined in the previous email.

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
Reindl Harald
2014-09-15 12:49:34 UTC
Permalink
Post by Richard W.M. Jones
Post by Reindl Harald
Post by Richard W.M. Jones
Post by Vít Ondruch
1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its
needs properly in its RPM dependencies?
It would still have been broken even with a reboot
no - why should it?
'foo' is loaded in memory, updated and now has different dependencies
no longer require 'bar.rb' but your running version still do
Please read closely. 'foo' has *not* been updated.
If 'foo' had been updated, we would have spotted it and restarted that
process using my technique outlined in the previous email
cross deps coming in my mind
foo -> library -> library -> library

* the first maybe already loaded
* also loaded the second one in a previous call
* that version relies on teh third one for some operations
* in a update the deps have changed

so you may have a mix with different dep-chains in memory
and some parts used the first time from disk with unexpected
results



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140915/9da8ba0b/attachment.sig>
Richard W.M. Jones
2014-09-15 13:10:43 UTC
Permalink
Post by Reindl Harald
Post by Richard W.M. Jones
Post by Reindl Harald
Post by Richard W.M. Jones
Post by Vít Ondruch
1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken? ie. Not expressing its
needs properly in its RPM dependencies?
It would still have been broken even with a reboot
no - why should it?
'foo' is loaded in memory, updated and now has different dependencies
no longer require 'bar.rb' but your running version still do
Please read closely. 'foo' has *not* been updated.
If 'foo' had been updated, we would have spotted it and restarted that
process using my technique outlined in the previous email
cross deps coming in my mind
foo -> library -> library -> library
* the first maybe already loaded
* also loaded the second one in a previous call
* that version relies on teh third one for some operations
* in a update the deps have changed
so you may have a mix with different dep-chains in memory
and some parts used the first time from disk with unexpected
results
I have absolutely no idea what you are talking about.

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
Vít Ondruch
2014-09-15 14:07:39 UTC
Permalink
Post by Richard W.M. Jones
Post by Vít Ondruch
Every of the script is based on assumption that you already read some
library/unit whatever. But that is not enough. I wonder how you want to
$ ls
foo.rb
bar.rb
$ cat foo.rb
def some_function
require 'bar'
end
And now
1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken?
They might come from different packages. Or there might be also another
level of requires, where bar.rb requires by baz.rb. In case that bar.rb
stays and baz.rb is removed, you still cannot predict that this will
fail in the future, since neither of these files was loaded before.

Or there might be another example of code with similar issues:

$ cat foo.rb

def some_function
$files.each {|f| require f}
end

$files = Dir.glob('*.rb')

I.e. during initialization, you list available files and you want to
load them later, but at that moment, they are not there already.
Post by Richard W.M. Jones
ie. Not expressing its
needs properly in its RPM dependencies?
We typically express dependencies between packages, the separate library
files are typically considered just implementation detail. But I agree
that it would be more precise to express dependencies between separate
library files, instead of packages. Anyway, this is case of dynamic
loading, so I am not sure how you would specify the dependencies.


Vít
Richard W.M. Jones
2014-09-15 14:51:34 UTC
Permalink
Post by Vít Ondruch
Post by Richard W.M. Jones
Post by Vít Ondruch
Every of the script is based on assumption that you already read some
library/unit whatever. But that is not enough. I wonder how you want to
$ ls
foo.rb
bar.rb
$ cat foo.rb
def some_function
require 'bar'
end
And now
1) I run some application, which loads my foo.rb file.
2) I later update the package which removes bar.rb file.
3) And I call some_function which fails due to missing bar.rb
How is this not 'foo' simply being broken?
They might come from different packages.
OK, in which case the package that needs bar was broken because it
didn't express that need in its dependencies, nor guard against the
possibility that bar was not installed.
Post by Vít Ondruch
Or there might be also another
level of requires, where bar.rb requires by baz.rb. In case that bar.rb
stays and baz.rb is removed, you still cannot predict that this will
fail in the future, since neither of these files was loaded before.
In which case bar.rb was similarly broken, for the same reason as
above.
Post by Vít Ondruch
$ cat foo.rb
def some_function
$files.each {|f| require f}
end
$files = Dir.glob('*.rb')
I.e. during initialization, you list available files and you want to
load them later, but at that moment, they are not there already.
This code is still by any measure broken. There are lots of ways that
such code could fail to work.

Plainly what I'm trying to say is: If the potentially insecure code
has been loaded into a Python process, and the python interpreter has
the small modification that I suggested, then we will be able to
detect that the insecure code is loaded into memory and flag the
process/service as needing to be restarted.

That's all.

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
Lennart Poettering
2014-10-21 20:08:20 UTC
Permalink
Post by Reindl Harald
Post by Nathanael d. Noblet
So I don't use Firefox anymore but I do know back in the day if we had
FF open when we updated it would do a double request for each page/form.
However when updating we just restarted FF and it would work fine after
that. I've never noticed any other issues than FF but like I said I
don't use it anymore.
Granted that doesn't matter obviously we don't want that kind of
behaviour.
I am curious though. Everyone says the only way to do it securely and
safely is with nothing running. Why can't updates be applied with stuff
running prior to a reboot?
1 out of a million cases needs offline updates
really - the only good at it is that you can stick
at using YUM and decide what you have to do at your
own - rarely updates really require a reboot
* lsof | grep DEL | grep /usr and restart services on servers
Well, some deps are not visible like that, because they do not involve
continuous mappings or open fds.

Moreover, it won't help you much anyway, as some daemons are not
restarble right now, most prominently dbus-daemon.

And strictly speaking as you cannot restart all daemons at the very
same instant, or even at the same instant as you install the new files
and remove the old ones you will always have races where daemons might
make use of resources or interfaces that are either newer than what
they expect or older.

offline updates are really about make updates fully reliable. Yes, in
most cases a "yum update" during runtime works well enough, and yes, I
usually do my updates that way too. But I am actually able to help
myself if something goes wrong. And so are you.

Offline updates are more for the cases where things need to be
reliable, because no well educated admin is available to instantly fix
things. Possibly because the machine is used by noobs only, or because
the machine is buried somewhere under the see, or where so many
instances of the machine are running that a human admins don't scale.

Hope that makes some sense.

Lennart
--
Lennart Poettering, Red Hat
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http:
Reindl Harald
2014-10-21 20:21:36 UTC
Permalink
Post by Lennart Poettering
Post by Reindl Harald
1 out of a million cases needs offline updates
really - the only good at it is that you can stick
at using YUM and decide what you have to do at your
own - rarely updates really require a reboot
* lsof | grep DEL | grep /usr and restart services on servers
Well, some deps are not visible like that, because they do not involve
continuous mappings or open fds.
may be true but in practice no problem over many years
Post by Lennart Poettering
Moreover, it won't help you much anyway, as some daemons are not
restarble right now, most prominently dbus-daemon
you repeat that again and again while i restart dbus over years on
headless machines for web/file/db-servers and frankly before F15 even
messagebus was completly disabled on all that machines
Post by Lennart Poettering
And strictly speaking as you cannot restart all daemons at the very
same instant, or even at the same instant as you install the new files
and remove the old ones you will always have races where daemons might
make use of resources or interfaces that are either newer than what
they expect or older.
interesting is that not so long ago there where just not much such
dependencies - mandatory presence of dbus is very recent

other services like some webapp talking to a db-server? frankly i wrote
10 years ago db-layers to wait and retry so you can restart the db
server after an update
Post by Lennart Poettering
offline updates are really about make updates fully reliable. Yes, in
most cases a "yum update" during runtime works well enough, and yes, I
usually do my updates that way too. But I am actually able to help
myself if something goes wrong. And so are you.
true
Post by Lennart Poettering
Offline updates are more for the cases where things need to be
reliable, because no well educated admin is available to instantly fix
things. Possibly because the machine is used by noobs only, or because
the machine is buried somewhere under the see, or where so many
instances of the machine are running that a human admins don't scale
Hope that makes some sense
yes, but keep in mind not introduce more and more dependencies to make
them mandatory somewhere in the future
Miroslav Suchý
2014-10-22 06:48:59 UTC
Permalink
Post by Lennart Poettering
Offline updates are more for the cases where things need to be
reliable, because no well educated admin is available to instantly fix
things.
I will print it an pin up on my notice board.

And the implication is that offline updates are not for readers of ***@lists.fedoraproject.org
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Con
Matthew Miller
2014-10-22 11:47:25 UTC
Permalink
Post by Miroslav Suchý
Post by Lennart Poettering
Offline updates are more for the cases where things need to be
reliable, because no well educated admin is available to instantly
fix things.
I will print it an pin up on my notice board.
And the implication is that offline updates are not for readers of
I don't think that's very fair, without the context. First, of course,
we're developing for more than just ourselves. And second, this isn't a
reversible statement: just because offline updates primarily target one
user type doesn't mean that other user types can't or shouldn't use it.

Why do I care about this? The non-techie user wants less rebooting too.
I'd love to see the updates toolchain get more smarts about recognizing
when an update is "safe" (or at least "safer") and not reboot in those
cases. That's possible but would be an investment of work. In the
meantime, "offline updates if you want simple and guaranteed", "use
yum or dnf online if you're able to handle unlikely but possible
consequences" seems workable enough.
--
Matthew Miller
<***@fedoraproject.org>
Fedora Project Leader
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct:
Richard W.M. Jones
2014-09-12 21:36:53 UTC
Permalink
Post by Nathanael d. Noblet
I am curious though. Everyone says the only way to do it securely and
safely is with nothing running. Why can't updates be applied with stuff
running prior to a reboot?
There's no reason, apart from the kernel. You're doing it right already.

Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
Reindl Harald
2014-09-12 21:40:34 UTC
Permalink
Post by Richard W.M. Jones
Post by Nathanael d. Noblet
I am curious though. Everyone says the only way to do it securely and
safely is with nothing running. Why can't updates be applied with stuff
running prior to a reboot?
There's no reason, apart from the kernel. You're doing it right already
well, some libraries like glibc leads also in better reboot
since they are used by nearly any component, but for 99% of
all updates a reboot is a joke since you easily can restart
servcices with nearly zero downtime

systemctl condrestart service1 service2 service3

condrestart since i distribute such commands over the whole
infrsastructure after tests and deployment and not every
installed service is enabled on all hosts

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140912/f025dd13/attachment-0001.sig>
Chris Murphy
2014-09-13 00:32:44 UTC
Permalink
Post by Richard Hughes
The *only* way to do this
securely and safely in the system we have now is in a clean pre-boot
environment,
Mostly clean post-boot environment, with the system we have now?
Post by Richard Hughes
What we could do is do updates on shutdown by basically killing
everything except PID 1, and then restart everything, but even then
that relies on no systemd or kernel updates being present.
Even if the system is fully rebooted once, after the update is done in a post-boot environment, it's half the reboots needed now.

One step up from this would be a way for packages to contain metadata indicating they need a reboot after an update; if none require it, then isolate graphical.target rather than a reboot.


Chris Murphy
Fred New
2014-09-13 08:59:34 UTC
Permalink
On Sat, Sep 13, 2014 at 3:32 AM, Chris Murphy <lists at colorremedies.com>
Post by Chris Murphy
Post by Richard Hughes
The *only* way to do this
securely and safely in the system we have now is in a clean pre-boot
environment,
Mostly clean post-boot environment, with the system we have now?
Post by Richard Hughes
What we could do is do updates on shutdown by basically killing
everything except PID 1, and then restart everything, but even then
that relies on no systemd or kernel updates being present.
Even if the system is fully rebooted once, after the update is done in a
post-boot environment, it's half the reboots needed now.
One step up from this would be a way for packages to contain metadata
indicating they need a reboot after an update; if none require it, then
isolate graphical.target rather than a reboot.
One step up from this would be something like a kpatch process in rpm
combined with packaged metadata that replaces in-memory modules so that
reboots wouldn't be necessary. Yeh, probably impossible.

Fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140913/ba2f6878/attachment.html>
Ian Pilcher
2014-09-13 23:01:32 UTC
Permalink
Post by Fred New
One step up from this would be something like a kpatch process in rpm
combined with packaged metadata that replaces in-memory modules so that
reboots wouldn't be necessary. Yeh, probably impossible.
This has almost certainly already been considered by people smarter than
me, but it occurs to me that there's no reason that kexec couldn't some-
how pass LUKS/dm-crypt keys to the new kernel.
--
========================================================================
Ian Pilcher arequipeno at gmail.com
-------- "I grew up before Mark Zuckerberg invented friendship" --------
========================================================================
Corey Sheldon
2014-09-13 23:26:10 UTC
Permalink
the keys are in kernelspace IIRC and thus updated / passed on initrd
/initramfs updates and kernel updates

Corey W Sheldon
Freelance IT Consultant, Multi-Discipline Tutor
310.909.7672
www.facebook.com/1stclassmobileshine
Post by Ian Pilcher
Post by Fred New
One step up from this would be something like a kpatch process in rpm
combined with packaged metadata that replaces in-memory modules so that
reboots wouldn't be necessary. Yeh, probably impossible.
This has almost certainly already been considered by people smarter than
me, but it occurs to me that there's no reason that kexec couldn't some-
how pass LUKS/dm-crypt keys to the new kernel.
--
========================================================================
Ian Pilcher arequipeno at gmail.com
-------- "I grew up before Mark Zuckerberg invented friendship" --------
========================================================================
--
devel mailing list
devel at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140913/c0b6e698/attachment.html>
Roberto Ragusa
2014-09-16 09:36:52 UTC
Permalink
Post by Richard Hughes
That's just not safe. Have you ever had firefox open and done a
firefox update? Widgets start disappearing, redraws start having weird
artifects and then after a little while it just crashes. Other
applications like LibreOffice behave the same. Anyone that says things
like "the old version of the library stays in memory" obviously hasn't
actually done much real-world GUI programming in the last decade, or
runs any kind of secure desktop system.
You are basically saying that "modern" software is just breaking
things which were perfectly solved decades ago.
If applications would just use libraries correctly, the kernel
would be able to let parts of deleted files be available for lazy
loading.
--
Roberto Ragusa mail at robertoragusa.it
Richard Hughes
2014-09-16 09:48:02 UTC
Permalink
Post by Roberto Ragusa
If applications would just use libraries correctly, the kernel
would be able to let parts of deleted files be available for lazy
loading.
Sure, as long as all[1] the resources were either open()d when the
user started the program, or linked in as resources into the library
itself then it works fine. You can certainly design an application
that does this correctly, it's just that very few do. A lot of GNOME
apps compile in resources into the binary itself, using GResource, and
this very much helps live updating.

Richard

[1] Including help files, all clipart, all icons, all .ui files, and
all the .css for all the dialogs.
Zdenek Kabelac
2014-09-16 09:55:01 UTC
Permalink
Post by Richard Hughes
Post by Roberto Ragusa
If applications would just use libraries correctly, the kernel
would be able to let parts of deleted files be available for lazy
loading.
Sure, as long as all[1] the resources were either open()d when the
user started the program, or linked in as resources into the library
itself then it works fine. You can certainly design an application
that does this correctly, it's just that very few do. A lot of GNOME
apps compile in resources into the binary itself, using GResource, and
this very much helps live updating.
Just a thought - but wouldn't be better spend time to enlighten Gnome/Firefox
developers how to write applications in a way the could be upgraded runtime,
instead of forcing users to reboot machines which seems seriously ridiculous.

And if we can't fix Gnome - then let's have there daemon/systemd service
waiting in background till the 'problematic' app is still running - and run
upgrade when the app is finally closed (eventually kindly ask user to close
the app in case of some CVE....)

Zdenek
Richard Hughes
2014-09-16 10:21:31 UTC
Permalink
Post by Zdenek Kabelac
Just a thought - but wouldn't be better spend time to enlighten
Gnome/Firefox developers how to write applications in a way the could be
upgraded runtime
So, it's not just the application, it's every application and D-Bus
service the application uses. Even glibc opens files-as-resources at
runtime. The much bigger issues is if you're using a D-Bus service
like most applications seem to do (and most use quite a few system and
session, directly and indirectly) then you've also got to co-ordinate
and handle changing D-Bus API (which typically isn't versioned). This
means you've got to restart the system DBus service at the right time
in the on-line update, which means co-ordinating with any other open
sessions on the system. Most desktop apps don't just sit in islands on
their own. It's probably okay to live-update powertop on a single-user
system when it's not being run, but anything much more integrated than
that just isn't going to work very well.

Richard
Reindl Harald
2014-09-16 10:26:18 UTC
Permalink
Post by Richard Hughes
Post by Zdenek Kabelac
Just a thought - but wouldn't be better spend time to enlighten
Gnome/Firefox developers how to write applications in a way the could be
upgraded runtime
So, it's not just the application, it's every application and D-Bus
service the application uses. Even glibc opens files-as-resources at
runtime. The much bigger issues is if you're using a D-Bus service
like most applications seem to do (and most use quite a few system and
session, directly and indirectly) then you've also got to co-ordinate
and handle changing D-Bus API (which typically isn't versioned). This
means you've got to restart the system DBus service at the right time
in the on-line update, which means co-ordinating with any other open
sessions on the system. Most desktop apps don't just sit in islands on
their own. It's probably okay to live-update powertop on a single-user
system when it's not being run, but anything much more integrated than
that just isn't going to work very well
and the other side of the story are some hundret dist-upgrades i
made in the past years with YUM (in a screen session for safety)
while just continue to browse the web and read / answer email

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140916/ecbcdad0/attachment.sig>
Richard Hughes
2014-09-16 10:33:52 UTC
Permalink
Post by Reindl Harald
and the other side of the story are some hundret dist-upgrades i
made in the past years with YUM (in a screen session for safety)
while just continue to browse the web and read / answer email
So you're saying my technical analysis is incorrect because it happens
to work for you? I've triaged many bugs to do with online and offline
update failures, and if we're going to say that we actually care about
the users data, it becomes increasingly hard to defend the "old" way
of doing it. I'm sure I could find numerous bugs numbers where doing
an online update made the session/terminal crash which of course
leaves you with duplicate packages on your system which may or may not
be fixable.

Richard
Przemek Klosowski
2014-09-16 12:29:11 UTC
Permalink
I've triaged many bugs to do with online and offline update failures,
and if we're going to say that we actually care about the users data,
it becomes increasingly hard to defend the "old" way of doing it. I'm
sure I could find numerous bugs numbers where doing an online update
made the session/terminal crash which of course leaves you with
duplicate packages on your system which may or may not be fixable.
Richard
OK, but this is means that we painted ourselves in the
corner---something is wrong if my Android phone, which I don't have to
reboot for updates, has higher uptime than my computer.

We are in a bind: on one hand, the best security practice is to upgrade
daily to avoid emerging vulnerabilities; on the other hand daily reboots
aren't really a nutritious alternative. Something has to give---which
one do we do:

- create a separate daily security upgrade stream, curated to not
require reboots if at all possible

- follow Microsoft and do a fixed 'patch Tuesday' schedule instead of
ASAP updates

- rewrite Linux or at least Gnome/DBus for safe updates :)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20140916/a587c701/attachment.html>
Josh Boyer
2014-09-16 12:50:36 UTC
Permalink
On Tue, Sep 16, 2014 at 8:29 AM, Przemek Klosowski
I've triaged many bugs to do with online and offline update failures, and
if we're going to say that we actually care about the users data, it becomes
increasingly hard to defend the "old" way of doing it. I'm sure I could find
numerous bugs numbers where doing an online update made the session/terminal
crash which of course leaves you with duplicate packages on your system
which may or may not be fixable. Richard
OK, but this is means that we painted ourselves in the corner---something is
wrong if my Android phone, which I don't have to reboot for updates, has
higher uptime than my computer.
You don't have to restart your phone for app updates, but the app
itself is restarted. For Android updates, it does require you to
reboot the phone. Android updates happen rather infrequently. Your
analogy is slightly flawed.
We are in a bind: on one hand, the best security practice is to upgrade
daily to avoid emerging vulnerabilities; on the other hand daily reboots
aren't really a nutritious alternative. Something has to give---which one
- create a separate daily security upgrade stream, curated to not require
reboots if at all possible
This has been suggested before and it isn't a terrible idea. It
requires people to work on it though.
- follow Microsoft and do a fixed 'patch Tuesday' schedule instead of ASAP
updates
There have been suggestions around having "levels" of updates. A
firehose level, which is the equivalent of today, and then a batched
level, etc. Also needs people to work on it.

josh
Richard Hughes
2014-09-16 12:56:14 UTC
Permalink
On 16 September 2014 13:29, Przemek Klosowski
OK, but this is means that we painted ourselves in the corner---something is
wrong if my Android phone, which I don't have to reboot for updates, has
higher uptime than my computer.
Right, we certainly have. By not having an OS / app layering policy
we've basically lumped everything into the OS layer. We're now as good
as Android, when you upgrade releases that you basically do an offline
update like we do in GNOME.

The only way to do application online updating (IMO) is to solve four things:

* Defining an OS platform
* Not breaking platform API
* Sandboxing
* Per-user applications

Richard
Petr Pisar
2014-09-16 13:35:46 UTC
Permalink
Post by Richard Hughes
The much bigger issues is if you're using a D-Bus service
like most applications seem to do (and most use quite a few system and
session, directly and indirectly) then you've also got to co-ordinate
and handle changing D-Bus API (which typically isn't versioned).
Maybe it's time to version the API.

Look at microkernel based systems which utilize messaging heavily and
the API consumers (applications or another subsystems) have to be
prepared for spurious API provider (server) restarts.

A server can refuse a message any time (especially if it does not
understand the request). Simple operations are usualy implemented as
a sequence of requests and responses where initial request obtains
a descriptor (a session, a handle) and subsequent requests passe it as
context (a session) identifier. If the server is restarted in between,
the context identifier becomes unrecognized and it's up to the
application to deal with it.

Just the fact that somebody calls functions over dbus instead of jumping
into a shared library do not free them from maintaing API.

-- Petr
Lennart Poettering
2014-10-22 10:27:47 UTC
Permalink
Post by Petr Pisar
Post by Richard Hughes
The much bigger issues is if you're using a D-Bus service
like most applications seem to do (and most use quite a few system and
session, directly and indirectly) then you've also got to co-ordinate
and handle changing D-Bus API (which typically isn't versioned).
Maybe it's time to version the API.
Look at microkernel based systems which utilize messaging heavily and
the API consumers (applications or another subsystems) have to be
prepared for spurious API provider (server) restarts.
A server can refuse a message any time (especially if it does not
understand the request). Simple operations are usualy implemented as
a sequence of requests and responses where initial request obtains
a descriptor (a session, a handle) and subsequent requests passe it as
context (a session) identifier. If the server is restarted in between,
the context identifier becomes unrecognized and it's up to the
application to deal with it.
Just the fact that somebody calls functions over dbus instead of jumping
into a shared library do not free them from maintaing API.
Well, the theory for this might be great. But reality tells us that
code that isn't regularly tested tends to be broken. Hence: the
assumption it would be reasonably possible to comprehensively test all
kinds of updates between any combination of software versions,
executed in any order, simultaneously with the user using the machine,
then is simply wrong. You explode the test matrix, and without testing
such upgrades will never be reliable. The offline update logic is
about making the test matrix smaller, and adding determinism where it
normally is missing.

Lennart
--
Lennart Poettering, Red Hat
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproje
Zdenek Kabelac
2014-09-16 14:21:15 UTC
Permalink
Post by Richard Hughes
Post by Zdenek Kabelac
Just a thought - but wouldn't be better spend time to enlighten
Gnome/Firefox developers how to write applications in a way the could be
upgraded runtime
So, it's not just the application, it's every application and D-Bus
service the application uses. Even glibc opens files-as-resources at
So it's time to fix D-Bus then as well!

If something is broken - it will not get fixed by hiding broken design behind
reboot&upgrade.

Guys it's 21st. century - it's so much broken idea to reboot 'machine'
because I want to upgrade i.e. Firefox - that I simply don't have words for it.

If you can't fix the Firefox (or some other broken tool) and you want to allow
install of new version - then you need to allow i.e. parallel installs of such
packages - it's that simple - I've been doing this more then 15 years ago at
University - so really nothing new here...

While the old-firefox is in use - new one could be installed in parallel
and the old one is removed when the last user closes app - of course
this has another problem with dependency hell - again solvable thing - look at
i.e. NixOS.

Just fix things in their core instead of hiding problem in pointless reboots.

Have any of those 'inventors' of reboot&update ever managed multiuser system?

This 'reboot' idea is just 'somehow' usable maybe for a single seat single
user desktop - but not anywhere else.

Has Fedora given up Unix ??

Zdenek
Miloslav Trmač
2014-09-16 14:39:10 UTC
Permalink
----- Original Message -----
Post by Zdenek Kabelac
Post by Richard Hughes
Post by Zdenek Kabelac
Just a thought - but wouldn't be better spend time to enlighten
Gnome/Firefox developers how to write applications in a way the could be
upgraded runtime
So, it's not just the application, it's every application and D-Bus
service the application uses. Even glibc opens files-as-resources at
So it's time to fix D-Bus then as well!
If something is broken - it will not get fixed by hiding broken design behind
reboot&upgrade.
Quite true.
Post by Zdenek Kabelac
If you can't fix the Firefox (or some other broken tool) and you want to allow
install of new version - then you need to allow i.e. parallel installs of such
packages - it's that simple - I've been doing this more then 15 years ago at
University - so really nothing new here...
Well, what we would need is:
1. Ability to keep multiple versions (both ABI-compatible and ABI-incompatible) of a single application or library or service installed and running at the same time.
2. Ability to detect which processes depend on which versions of which components.
3. Ability to automatically restart such processes without loosing state (either completely transparently or with some user notification for GUIs).

This has all been done before, and can be done again. (And it would make at least half of the userbase clamoring for containers what they need, without playing ugly complex nontransparent namespace games.) But let’s be clear about it, 1. means completely changing our filesystem layout, and 3. means changing our process model to go way beyond int main(...).

It is technically possible, it is the right thing to do. Do we want to do it and can we do it?
Post by Zdenek Kabelac
This 'reboot' idea is just 'somehow' usable maybe for a single seat single
user desktop - but not anywhere else.
Has Fedora given up Unix ??
The Unix history is actually closer to “edit header files to match your hardware, then rebuild the kernel and userpace with (make world), and reboot“. Packaged applications with an ISV source and an update stream separate from the OS have certainly been built on top of Unix but have never been a major design focus. Arguably the whole point of a “Linux distribution” has been to get closer to the BSD-style single-kernel-and-userspace distribution updated always as a whole.
Mirek
Zdenek Kabelac
2014-09-16 15:01:50 UTC
Permalink
Post by Miloslav Trmač
----- Original Message -----
Post by Zdenek Kabelac
Post by Richard Hughes
Post by Zdenek Kabelac
Just a thought - but wouldn't be better spend time to enlighten
Gnome/Firefox developers how to write applications in a way the could be
upgraded runtime
So, it's not just the application, it's every application and D-Bus
service the application uses. Even glibc opens files-as-resources at
So it's time to fix D-Bus then as well!
If something is broken - it will not get fixed by hiding broken design behind
reboot&upgrade.
Quite true.
Post by Zdenek Kabelac
If you can't fix the Firefox (or some other broken tool) and you want to allow
install of new version - then you need to allow i.e. parallel installs of such
packages - it's that simple - I've been doing this more then 15 years ago at
University - so really nothing new here...
1. Ability to keep multiple versions (both ABI-compatible and ABI-incompatible) of a single application or library or service installed and running at the same time.
Other distributions allow to install multiple version of same libraries - I've
never understood the Fedora policy to recompile whole Fedora when a new
version of library is released.

This policy design is quite 'show-stopper' for anything like this...
Post by Miloslav Trmač
2. Ability to detect which processes depend on which versions of which components.
We already managed to brought in systemd....
Post by Miloslav Trmač
3. Ability to automatically restart such processes without loosing state (either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of
unused packages would do the trick here - doing here state-full design is much
more complex thing....

The primary use-case to target should be to allow reinstall user packages
while they are in use.
Post by Miloslav Trmač
This has all been done before, and can be done again. (And it would make at least half of the userbase clamoring for containers what they need, without playing ugly complex nontransparent namespace games.) But let’s be clear about it, 1. means completely changing our filesystem layout, and 3. means changing our process model to go way beyond int main(...).
It is technically possible, it is the right thing to do. Do we want to do it and can we do it?
Fedora made not much useful /usr move thing - so maybe it's time to think and
design something really useful.

As mentioned there are OSes which do handle things much better...

And surprisingly even Systemd guru realized there is something broken with
current filesystem layout - except solving it with Btrfs is not really a fix...
Post by Miloslav Trmač
Post by Zdenek Kabelac
Has Fedora given up Unix ??
The Unix history is actually closer to “edit header files to match your hardware, then rebuild the kernel and userpace with (make world), and reboot“. Packaged applications with an ISV source and an update stream separate from the OS have certainly been built on top of Unix but have never been a major design focus. Arguably the whole point of a “Linux distribution” has been to get closer to the BSD-style single-kernel-and-userspace distribution updated always as a whole.
My view rather is - Fedora is taking feature-by-feature away from my box.

Zdenek
Miloslav Trmač
2014-09-16 15:08:56 UTC
Permalink
----- Original Message -----
Post by Zdenek Kabelac
Post by Miloslav Trmač
1. Ability to keep multiple versions (both ABI-compatible and
ABI-incompatible) of a single application or library or service installed
and running at the same time.
Other distributions allow to install multiple version of same libraries
AFAIK only when the ABI (soname) is different. We also need to allow for the case when the ABI stays the same but the internal implementation changes (e.g. changing a format of a resource file that is used by the library, making old processes incompatible with the newly installed resource).
Post by Zdenek Kabelac
Post by Miloslav Trmač
2. Ability to detect which processes depend on which versions of which components.
We already managed to brought in systemd....
I can’t see how systemd helps. See the other discussions about Python/Ruby modules that leave no obvious trace of their dependency after being loaded into memory.
Post by Zdenek Kabelac
Post by Miloslav Trmač
3. Ability to automatically restart such processes without loosing state
(either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of
unused packages would do the trick here - doing here state-full design is much
more complex thing....
Because otherwise you end up with an old version of Firefox running for 60 days (or however long a laptop can run with suspends and no restarts; most people about never quit their browser), and that version of Firefox keeping an old version of a system-wide daemon running for 60 days as well..
Post by Zdenek Kabelac
And surprisingly even Systemd guru realized there is something broken with
current filesystem layout - except solving it with Btrfs is not really a fix...
The sad thing is that adding more workarounds like namespaces and the like really might be easier than agreeing on making a real change and getting it done :( But we will live with the costs forever.
Mirek
Zdenek Kabelac
2014-09-16 15:26:09 UTC
Permalink
Post by Miloslav Trmač
----- Original Message -----
Post by Zdenek Kabelac
Post by Miloslav Trmač
1. Ability to keep multiple versions (both ABI-compatible and
ABI-incompatible) of a single application or library or service installed
and running at the same time.
Other distributions allow to install multiple version of same libraries
AFAIK only when the ABI (soname) is different. We also need to allow for the case when the ABI stays the same but the internal implementation changes (e.g. changing a format of a resource file that is used by the library, making old processes incompatible with the newly installed resource).
That's what I mean - when something changes in the library - soname should
change. It's quite broken to release same version of library with the same
name when there is different internal implementation. It's something library
package maintainer should take care of...

There is also support for versioned symbols.
Post by Miloslav Trmač
Post by Zdenek Kabelac
Post by Miloslav Trmač
2. Ability to detect which processes depend on which versions of which components.
We already managed to brought in systemd....
I can’t see how systemd helps. See the other discussions about Python/Ruby modules that leave no obvious trace of their dependency after being loaded into memory.
It has similar complexity and maybe packages could be described just like
services - so similar thing could be possibly reused here ?

Anyway - I've not been thinking too deeply about this...
Post by Miloslav Trmač
Post by Zdenek Kabelac
Post by Miloslav Trmač
3. Ability to automatically restart such processes without loosing state
(either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of
unused packages would do the trick here - doing here state-full design is much
more complex thing....
Because otherwise you end up with an old version of Firefox running for 60 days (or however long a laptop can run with suspends and no restarts; most people about never quit their browser), and that version of Firefox keeping an old version of a system-wide daemon running for 60 days as well..
Sure if user is able to run Firefox for 60 days (though my usually goes out
of RAM in just a week or so, if it's not crashing earlier...) - then he is
happy user.

I guess something may show to user some nice gui dialog warning like - 'Hey
there is a new version installed - so 'restart' your browser to get new cool
feature' (FF normally already detects upgrades and suggest restart)

I don't see nothing wrong with this case - it's per user and doesn't require
upgrade - and if I've something important is running in my browser I could
delay restart to the moment I don't care.
Post by Miloslav Trmač
Post by Zdenek Kabelac
And surprisingly even Systemd guru realized there is something broken with
current filesystem layout - except solving it with Btrfs is not really a fix...
The sad thing is that adding more workarounds like namespaces and the like really might be easier than agreeing on making a real change and getting it done :( But we will live with the costs forever.
Look at NixOS - there are no namespace or btrfs requirements....

Zdenek
Miloslav Trmač
2014-09-16 15:36:43 UTC
Permalink
----- Original Message -----
Post by Zdenek Kabelac
Post by Miloslav Trmač
----- Original Message -----
Post by Zdenek Kabelac
Post by Miloslav Trmač
1. Ability to keep multiple versions (both ABI-compatible and
ABI-incompatible) of a single application or library or service installed
and running at the same time.
Other distributions allow to install multiple version of same libraries
AFAIK only when the ABI (soname) is different. We also need to allow for
the case when the ABI stays the same but the internal implementation
changes (e.g. changing a format of a resource file that is used by the
library, making old processes incompatible with the newly installed
resource).
That's what I mean - when something changes in the library - soname should
change. It's quite broken to release same version of library with the same
name when there is different internal implementation.
No. Let me try to explain the issue in detail.

libfoo-1.0.0 and libfoo-1.0.1 both use /usr/share/libfoo/magic_database, but the internal format has changed between the versions. It is _internal_ so changing the soname would be both unnecessary and _inappropriate_ (breaking users). Looking at current packaging, neither RPM nor debian would install libfoo-1.0.0 and libfoo-1.0.1 in parallel. Looking at the filesystem (i.e. assuming that we can change packaging), we may have both /usr/lib*/foo-1.0.0.so and /usr/lib*/foo-1.0.1.so installed on the filesystem, but that /usr/share/libfoo/magic_database can only have only one of the versions. (OK, in this case it could be argued that it is the upstream’s fault and that they should version the magic_database path if they change the internal format, to explicitly design for parallel installs. But now consider the distribution patching the library (changing “release” as opposed to “version” in RPM terminology); upstream can’t reasonably for that, parallel installation of same-version-different-release packages must be a distribution-focused filesystem design.
Post by Zdenek Kabelac
Post by Miloslav Trmač
Post by Zdenek Kabelac
Post by Miloslav Trmač
2. Ability to detect which processes depend on which versions of which components.
We already managed to brought in systemd....
I can’t see how systemd helps. See the other discussions about Python/Ruby
modules that leave no obvious trace of their dependency after being loaded
into memory.
It has similar complexity and maybe packages could be described just like
services - so similar thing could be possibly reused here ?
Extra manual dependency information that would get obsolete?
Post by Zdenek Kabelac
Post by Miloslav Trmač
Post by Zdenek Kabelac
Post by Miloslav Trmač
3. Ability to automatically restart such processes without loosing state
(either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of
unused packages would do the trick here - doing here state-full design is much
more complex thing....
Because otherwise you end up with an old version of Firefox running for 60
days (or however long a laptop can run with suspends and no restarts; most
people about never quit their browser), and that version of Firefox
keeping an old version of a system-wide daemon running for 60 days as
well..
Sure if user is able to run Firefox for 60 days (though my usually goes out
of RAM in just a week or so, if it's not crashing earlier...) - then he is
happy user.
Until an exploit _that they have already installed an update for_ steals their data.
Post by Zdenek Kabelac
I guess something may show to user some nice gui dialog warning like - 'Hey
there is a new version installed - so 'restart' your browser to get new cool
feature' (FF normally already detects upgrades and suggest restart)
No, that’s not good enough. The OS should restart it for the user (other OSes already can do this). (In the case of user-invisible components there is no other option, and for user-visible GUIs it is also the right thing to do.)
Mirek
Zdenek Kabelac
2014-09-16 16:26:48 UTC
Permalink
Post by Miloslav Trmač
----- Original Message -----
Post by Zdenek Kabelac
Post by Miloslav Trmač
Post by Zdenek Kabelac
Post by Miloslav Trmač
2. Ability to detect which processes depend on which versions of which components.
We already managed to brought in systemd....
I can’t see how systemd helps. See the other discussions about Python/Ruby
modules that leave no obvious trace of their dependency after being loaded
into memory.
It has similar complexity and maybe packages could be described just like
services - so similar thing could be possibly reused here ?
Extra manual dependency information that would get obsolete?
Runtime removal of unused packages might be fun.
It could be something like 'fstrim' tool to run through cron...
Post by Miloslav Trmač
Post by Zdenek Kabelac
Post by Miloslav Trmač
Post by Zdenek Kabelac
Post by Miloslav Trmač
3. Ability to automatically restart such processes without loosing state
(either completely transparently or with some user notification for GUIs).
I'm not quite sure why we would need restart - simply delayed lazy release of
unused packages would do the trick here - doing here state-full design is much
more complex thing....
Because otherwise you end up with an old version of Firefox running for 60
days (or however long a laptop can run with suspends and no restarts; most
people about never quit their browser), and that version of Firefox
keeping an old version of a system-wide daemon running for 60 days as
well..
Sure if user is able to run Firefox for 60 days (though my usually goes out
of RAM in just a week or so, if it's not crashing earlier...) - then he is
happy user.
Until an exploit _that they have already installed an update for_ steals their data.
Post by Zdenek Kabelac
I guess something may show to user some nice gui dialog warning like - 'Hey
there is a new version installed - so 'restart' your browser to get new cool
feature' (FF normally already detects upgrades and suggest restart)
No, that’s not good enough. The OS should restart it for the user (other OSes already can do this). (In the case of user-invisible components there is no other option, and for user-visible GUIs it is also the right thing to do.)
As long as Firefox can't restart 'unnoticeable' while playing youtube video -
this is not an option - when admins updates machine - it just can't kill every
users' running firefox.

It's fine to place a warning somewhere and require restart with some 'many
hours' timeout - but for almost all "Firefox" updates - it's good enough to
restart it just sometimes latest - and for those 'have to restart' - it still
upon admins policy - not upon Fedora package maintainer....

Zdenek
Matthias Clasen
2014-09-16 15:29:51 UTC
Permalink
Post by Zdenek Kabelac
Has Fedora given up Unix ??
This thread has gone quite far out into the weeds. It started with a
fairly concrete question: can we improve the offline update experience
by requiring only a single reboot, instead of two ?

I'd still be interested in hearing Lennarts opinion on this.


Matthias
Lukáš Tinkl
2014-09-16 20:02:36 UTC
Permalink
Post by Matthias Clasen
Post by Zdenek Kabelac
Has Fedora given up Unix ??
This thread has gone quite far out into the weeds. It started with a
fairly concrete question: can we improve the offline update experience
by requiring only a single reboot, instead of two ?
That's the core of the problem, while other systems have been trying for
years to get rid of this nonsense (updates requiring reboots), we are
moving the opposite direction. Technical reasons aside(*), this is NOT
right for our users.


(*) I agree offline updates are the safest way technically but that
should be only regarded as a workaround for a more fundamental problem,
lying deep below the desktop level.
--
Lukáš Tinkl <ltinkl at redhat.com>
Senior Software Engineer - KDE desktop team, Brno
KDE developer <lukas at kde.org>
Red Hat Inc. http://cz.redhat.com
Bastien Nocera
2014-09-17 09:54:03 UTC
Permalink
----- Original Message -----
Post by Lukáš Tinkl
Post by Matthias Clasen
Post by Zdenek Kabelac
Has Fedora given up Unix ??
This thread has gone quite far out into the weeds. It started with a
fairly concrete question: can we improve the offline update experience
by requiring only a single reboot, instead of two ?
That's the core of the problem, while other systems have been trying for
years to get rid of this nonsense (updates requiring reboots), we are
moving the opposite direction. Technical reasons aside(*), this is NOT
right for our users.
(*) I agree offline updates are the safest way technically but that
should be only regarded as a workaround for a more fundamental problem,
lying deep below the desktop level.
There are no desktop systems that can do live application updates. Applications
on mobile OSes get killed (and state is saved) when you update them.

All those OSes require reboots when updating the OS.
Miroslav Suchý
2014-09-17 11:58:09 UTC
Permalink
Post by Bastien Nocera
All those OSes require reboots when updating the OS.
Define OS.

Firefox is definitely not OS. While systemd is OS.
I am fine with reboot after systemd upgrade, but not after upgrading Firefox.

And there we have lots of packages in between those two. Where you would draw a line? What is OS? (And this brought us
to core of Fedora.next discussion :) )
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
Lennart Poettering
2014-10-22 10:31:53 UTC
Permalink
Post by Miroslav Suchý
Post by Bastien Nocera
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS.
I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
Well, on Fedora and Unixes the apps are just part of the OS, they are
all dropped into /usr, and not isolated out.

It would be great if we could nicely isolate the apps from the OS so
that we can restart the apps independently from the OS, but this
requires isolating things first.

We are working on app sandboxes, and they will make this available,
but it's the traditional Linux model cannot really deliver this.

Lennart
--
Lennart Poettering, Red Hat
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Co
Stephen John Smoogen
2014-10-22 14:31:19 UTC
Permalink
Post by Lennart Poettering
Post by Miroslav Suchý
Post by Bastien Nocera
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS.
I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
Well, on Fedora and Unixes the apps are just part of the OS, they are
all dropped into /usr, and not isolated out.
It would be great if we could nicely isolate the apps from the OS so
that we can restart the apps independently from the OS, but this
requires isolating things first.
We are working on app sandboxes, and they will make this available,
but it's the traditional Linux model cannot really deliver this.
Well it depends on the traditional model that you are going to refer to. In
the model of the Unix systems I had to set up in the late 1980's and the
1990's... there was a separation in that items for the OS were in the root
directories (say /bin and /sbin), and stuff that was not OS critical but
user critical were in /usr/{bin,sbin}. And then local critical were in /opt
or /usr/local depending on the OS.

We (Linux distributions) lost that distinction sometime in the past decade
and then undid it completely with the various /usr/ merges. [This isn't
meant to open that can of worms.. the distinction was broken before the
merge.. the merge just made it clear.]

I am not sure how to best move from here. A complete reinvent of
hierarchies is always tempting.. but it has always been the deathknell of
every OS that has done it in the past due to chasm crossing issues (too
much old stuff needing old things causing any benefits from new stuff to be
actual detriments). Doing a more thorough job of packaging items so that
system only items were in /bin,/lib, etc has never worked because too many
things sit between the two. [its a user component AND a system component!]
At best I can say it comes down to operating systems are too damn
complicated and I need to go back to potato farming :)
Post by Lennart Poettering
Lennart
--
Lennart Poettering, Red Hat
--
devel mailing list
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
--
Stephen J Smoogen.
Tom Rivers
2014-10-22 14:45:33 UTC
Permalink
Post by Lennart Poettering
It would be great if we could nicely isolate the apps from the OS so
that we can restart the apps independently from the OS, but this
requires isolating things first.
Isn't the differentiation between kernel space and user space sufficient
for identifying what is the OS and what is an application? I know
Windows blurred those lines during the Browser Wars with Netscape, but
to my knowledge that separation still exists in Linux.


Tom
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduc
drago01
2014-10-22 14:58:11 UTC
Permalink
It would be great if we could nicely isolate the apps from the OS so that
we can restart the apps independently from the OS, but this requires
isolating things first.
Isn't the differentiation between kernel space and user space sufficient for
identifying what is the OS and what is an application? I know Windows
blurred those lines during the Browser Wars with Netscape, but to my
knowledge that separation still exists in Linux.
No the OS is more than just a kernel.
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code o
Tom Rivers
2014-10-22 15:12:53 UTC
Permalink
Post by drago01
No the OS is more than just a kernel.
Kernel Space contains more than just the kernel. It also contains
device drivers, kernel extensions, and other privileged processes that
require full system access. User Space exists as a barrier to keep
applications separate from the OS itself.


Tom
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of
Michael Stahl
2014-10-22 19:07:33 UTC
Permalink
Post by Miroslav Suchý
Post by Bastien Nocera
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS.
I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
the important point in that case is not reboot after upgrading Firefox
but *before* upgrading Firefox, which means that at the time of the
upgrade no Firefox will be running and potentially crashing because one
of the 100s of DSOs it loads on-demand has changed in an incompatible way.

there used to be quite a few ABRT crashes reported against desktop
applications with impossible looking stack traces (although with the
automatic micro-reports it's less of a problem nowadays as they are
easier to ignore), and sometimes the reporter gives feedback that the
application was running across a yum update...
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora C
Ian Malone
2014-10-22 22:35:00 UTC
Permalink
Post by Michael Stahl
Post by Miroslav Suchý
Post by Bastien Nocera
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS.
I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
the important point in that case is not reboot after upgrading Firefox
but *before* upgrading Firefox, which means that at the time of the
upgrade no Firefox will be running and potentially crashing because one
of the 100s of DSOs it loads on-demand has changed in an incompatible way.
there used to be quite a few ABRT crashes reported against desktop
applications with impossible looking stack traces (although with the
automatic micro-reports it's less of a problem nowadays as they are
easier to ignore), and sometimes the reporter gives feedback that the
application was running across a yum update...
While it can be a bit confusing the first time it happens to you, the
solution is just to start and stop firefox again in that case. If it
the goal is just to tidy ABRT crash reports (and I'm not sure it is)
then forcing reboots on users wouldn't be very kind.
--
imalone
http://ibmalone.blogspot.co.uk
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduc
Michael Scherer
2014-10-26 13:43:09 UTC
Permalink
Post by Ian Malone
Post by Michael Stahl
Post by Miroslav Suchý
Post by Bastien Nocera
All those OSes require reboots when updating the OS.
Define OS.
Firefox is definitely not OS. While systemd is OS.
I am fine with reboot after systemd upgrade, but not after upgrading Firefox.
the important point in that case is not reboot after upgrading Firefox
but *before* upgrading Firefox, which means that at the time of the
upgrade no Firefox will be running and potentially crashing because one
of the 100s of DSOs it loads on-demand has changed in an incompatible way.
there used to be quite a few ABRT crashes reported against desktop
applications with impossible looking stack traces (although with the
automatic micro-reports it's less of a problem nowadays as they are
easier to ignore), and sometimes the reporter gives feedback that the
application was running across a yum update...
While it can be a bit confusing the first time it happens to you, the
solution is just to start and stop firefox again in that case. If it
the goal is just to tidy ABRT crash reports (and I'm not sure it is)
then forcing reboots on users wouldn't be very kind.
Beyond cleaning crash reports and reducing them, there is also the perception
of users.

If we want to have the same reputation as some older windows with "this software
crashed and I do not know why" from our users, then having regular random crashes
after a random delay after upgrade is the way to go. If we still aim for "the software
is solid and stable", then we should try to reduce the random crash.
--
Michael Scherer
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Con
Miroslav Suchý
2014-10-23 08:56:00 UTC
Permalink
Post by Michael Stahl
the important point in that case is not reboot after upgrading Firefox
but*before* upgrading Firefox, which means that at the time of the
upgrade no Firefox will be running and potentially crashing because one
of the 100s of DSOs it loads on-demand has changed in an incompatible way.
The upgrade can last one hour (more or less). I'm not going to wait for upgrade and looking at black screen (albeit with
progress bar). When upgrading - I save critical work in progress, and start reading articles from my RSS queue.
And if Firefox crash (which happens very very rarely) I lost nothing, but get one hour of my life for free.

To be constructive: I can imagine displaying dialog:
Going to upgrade your system. Do you want to do it offline?
* Yes - most safe way
* No - there is non-trivial chance that some running application would crash (proceed on your own risk)

And let the user decide if he is power user or not.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-
Richard Hughes
2014-10-23 10:43:29 UTC
Permalink
Post by Miroslav Suchý
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with
F21. Is this on fairly up-to-date SSD hardware? Are any specific
packages taking longer than the others?

Richard
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.or
Daniel P. Berrange
2014-10-23 10:52:26 UTC
Permalink
Post by Richard Hughes
Post by Miroslav Suchý
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with
F21. Is this on fairly up-to-date SSD hardware? Are any specific
packages taking longer than the others?
Don't assume everyone is using SSDs with Fedora....

Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/c
Reindl Harald
2014-10-23 11:00:42 UTC
Permalink
Post by Daniel P. Berrange
Post by Richard Hughes
Post by Miroslav Suchý
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with
F21. Is this on fairly up-to-date SSD hardware? Are any specific
packages taking longer than the others?
Don't assume everyone is using SSDs with Fedora....
that assumption is sadly a common mistake on many places and leads to
implement things less efficient because "it's fast enough on 'fairly'
recent hardware)

frankly it will take many years before i replace my 4x2 TB RAID10 disks
by 4x2 TB SSD just because the price and no i won't re-install from
scratch to just have the OS only on SSD and my virtual machines (which
are the reason for that large disks) stay on slower disks
Reindl Harald
2014-10-23 10:57:42 UTC
Permalink
Post by Richard Hughes
Post by Miroslav Suchý
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with
F21. Is this on fairly up-to-date SSD hardware? Are any specific
packages taking longer than the others?
an hour was surely exaggerated

but there was not so long ago a bug in some package updateing caches
which took on a hyperfast RAID10 machine with enabled barriers up to 10
minutes for a ordinary KDE update

before it got fixid without barriers (which are are default) a few
seconds - you don't know if and how often such things may get triggered
inside a update transaction (like the useless ldconfig and systemd
reloads after every single related package)

# https://bugzilla.redhat.com/show_bug.cgi?id=1052173#c32
export PKGSYSTEM_ENABLE_FSYNC=0
Miroslav Suchý
2014-10-24 10:00:28 UTC
Permalink
Post by Reindl Harald
Post by Richard Hughes
Post by Miroslav Suchý
The upgrade can last one hour (more or less).
An hour?! Most of my offline updates take a few tens of seconds with
F21. Is this on fairly up-to-date SSD hardware? Are any specific
packages taking longer than the others?
an hour was surely exaggerated
Not at all.
I'm not updating daily. I upgraded my machine IIRC 2-3 weeks ago. So lets benchmark it and provide you real data.

My machine have classic magnetic disk, however in SW RAID1.
Timing cached reads: 12236 MB in 2.00 seconds = 6124.59 MB/sec
Timing buffered disk reads: 412 MB in 3.00 seconds = 137.12 MB/sec
Fedora 21, 16 GB RAM (2GB free), 8 CPU cores, swap available but none used

I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download.
Download lasted 3mins 20secs.
Then installation started and since beginning "transaction started" till the end lasted exactly 53 minutes.
No specific package is blocking the process, dnf was chewing packages one by one in steady pace. Veryfing phase lasted
~4 minutes, so it means approximately 3 second per package, which is what I am seeing on screen.

And this is nothing exceptional. I see similar times across all machines I maintain. When I'm updating box of my mother
(old EeeBox, updating aprox every 3 months) then the time is usually 3 hours (however ~1 hour is just download phase).
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/
Mathieu Bridon
2014-10-24 10:02:55 UTC
Permalink
Post by Miroslav Suchý
I'm not updating daily. I upgraded my machine IIRC 2-3 weeks ago. So lets benchmark it and provide you real data.
My machine have classic magnetic disk, however in SW RAID1.
Timing cached reads: 12236 MB in 2.00 seconds = 6124.59 MB/sec
Timing buffered disk reads: 412 MB in 3.00 seconds = 137.12 MB/sec
Fedora 21, 16 GB RAM (2GB free), 8 CPU cores, swap available but none used
I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download.
Download lasted 3mins 20secs.
Then installation started and since beginning "transaction started" till the end lasted exactly 53 minutes.
No specific package is blocking the process, dnf was chewing packages one by one in steady pace. Veryfing phase lasted
~4 minutes, so it means approximately 3 second per package, which is what I am seeing on screen.
And this is nothing exceptional. I see similar times across all machines I maintain. When I'm updating box of my mother
(old EeeBox, updating aprox every 3 months) then the time is usually 3 hours (however ~1 hour is just download phase).
More than anything, doesn't this just shows that we simply push way too
many updates in Fedora?
--
Mathieu
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/co
Reindl Harald
2014-10-24 11:04:00 UTC
Permalink
Post by Mathieu Bridon
Post by Miroslav Suchý
I'm not updating daily. I upgraded my machine IIRC 2-3 weeks ago. So lets benchmark it and provide you real data.
My machine have classic magnetic disk, however in SW RAID1.
Timing cached reads: 12236 MB in 2.00 seconds = 6124.59 MB/sec
Timing buffered disk reads: 412 MB in 3.00 seconds = 137.12 MB/sec
Fedora 21, 16 GB RAM (2GB free), 8 CPU cores, swap available but none used
I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download.
Download lasted 3mins 20secs.
Then installation started and since beginning "transaction started" till the end lasted exactly 53 minutes.
No specific package is blocking the process, dnf was chewing packages one by one in steady pace. Veryfing phase lasted
~4 minutes, so it means approximately 3 second per package, which is what I am seeing on screen.
And this is nothing exceptional. I see similar times across all machines I maintain. When I'm updating box of my mother
(old EeeBox, updating aprox every 3 months) then the time is usually 3 hours (however ~1 hour is just download phase).
More than anything, doesn't this just shows that we simply push way too
many updates in Fedora?
no

* first: the above is Rawhide/Alpha
* second: the reason i run Fedora and not Debian/RHEL is fast updates
* third: nobody should apply updates every 3 weeks
but that above is Alpha and so no "production" machine
Richard Hughes
2014-10-24 10:59:16 UTC
Permalink
Post by Miroslav Suchý
I run "dnf upgrade" and I have been offered 853 packages and 1.3 GB to download.
I'm pretty sure F21 for the last few weeks is not representative of a
normal installed system over a 2 week period. 853 must be close to
every package on your box...

Richard
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Con
Miroslav Suchý
2014-10-24 13:01:08 UTC
Permalink
Post by Richard Hughes
853 must be close to
every package on your box...
Nope.
$ rpm -qa |wc -l
4337
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-cond
Dennis Gilmore
2014-11-07 05:42:40 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 23 Oct 2014 10:56:00 +0200
Post by Miroslav Suchý
Post by Michael Stahl
the important point in that case is not reboot after upgrading
Firefox but*before* upgrading Firefox, which means that at the
time of the upgrade no Firefox will be running and potentially
crashing because one of the 100s of DSOs it loads on-demand has
changed in an incompatible way.
The upgrade can last one hour (more or less). I'm not going to wait
for upgrade and looking at black screen (albeit with progress bar).
When upgrading - I save critical work in progress, and start reading
articles from my RSS queue. And if Firefox crash (which happens very
very rarely) I lost nothing, but get one hour of my life for free.
I have had some of my arm systems take 6-8 hours and even more to update
storage is a class 10 microsd card. the system was Fedora 21 and about
2-3 weeks worth of updates to be applied, about 700 packages.

Dennis
Post by Miroslav Suchý
Going to upgrade your system. Do you want to do it offline?
* Yes - most safe way
* No - there is non-trivial chance that some running application
would crash (proceed on your own risk)
And let the user decide if he is power user or not.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBAgAGBQJUXFvUAAoJEH7ltONmPFDRwUYQAIEcuR8dyKQehxMk3eiRf2lh
Grq2//ndIBW1dtwQevd0edVmqaQEJ1m9TDNk2WuU1mGUFUeKFpSLUtPimOwgdLpD
V8ls6flIkrdqvmdN7ki0ulGZT18I4Pwc+dH0JdzdkoyaYbB4BpHpwfwPinH1hJDn
23mlaLJNVcKFyFTtn/pcYDKeHIEMrbBuBBTEEGLTn12T50Y7pdrlq97l5BwwS00m
p69XGfeup6Jx3lLQhiYKmioq/eTIGXnLsEomMnd/os5R3P5nqFq7bgRZg5yBtsNn
F8i/TSaqd/QdOomzR2WA2EQAA06Hz2Is4mfeZAELbEFE/1q309PrqJ6yuZhmMDCs
Fcq+qHVMrn9GclHlkL79DQcmlkM9C6WaFzX0x7uDm6hvk5td4+xyqSUtfWvCgQBR
z/YVfB07T+Y7kGltKXCUt3Su+WSKYWeyh5TRaOsvpIa/rwsINNExCUAKgRRvNdHX
IuWL6dzSGvtDN+8ztqd8eVLOVzZnfORNCz8JykOwMOXhM0qGeKxhCanrgwqNellY
fNg1M3LaPp4bLxhi0dhxJxpQuKt7ONc29Npty9MWZ1IFosikDDUevHGZcwiHnxm6
GAByS49Q4Y65w4xAz+Q2uEWZQT087zkiH+FvaZs5Ldgvk+WQEO4V2/l+8xn/0JI+
KdkbTlKL9s2RIgBl6TYs
=cmts
-----END PGP SIGNATURE-----
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject
Stephen Gallagher
2014-09-17 12:17:29 UTC
Permalink
Post by Lukáš Tinkl
Post by Matthias Clasen
Post by Zdenek Kabelac
Has Fedora given up Unix ??
This thread has gone quite far out into the weeds. It started
with a fairly concrete question: can we improve the offline
update experience by requiring only a single reboot, instead of
two ?
That's the core of the problem, while other systems have been
trying for years to get rid of this nonsense (updates requiring
reboots), we are moving the opposite direction. Technical reasons
aside(*), this is NOT right for our users.
While that may be a wonderfully ideal state to reach (and certainly
worth discussion), I agree with Matthias that this has gone far afield
of my original question.

I'd like to try to tie this thread back to answering a single
question: Without a fundamental redesign of the way things work, which
would be a VERY large and long-term effort, can we improve the user
experience for people *today* with fairly little effort?

If the answer to that question is "yes", is there anything
fundamentally wrong with adding that workaround while we solve the
situation in a more complete manner?

I think this is a great example of the adage "perfect is the enemy of
the good". We have a situation that everyone seems to agree is less
than ideal and we have two proposed solutions: one that probably
requires only a few weeks of work and testing and one that probably
requires a fundamental change to the way the system works and
therefore could take years to accomplish.

Ask yourself this question: are you as a user patient enough to wait
for a perfect solution or are you happy with incremental improvement?
Post by Lukáš Tinkl
(*) I agree offline updates are the safest way technically but
that should be only regarded as a workaround for a more fundamental
problem, lying deep below the desktop level.
Stephen Gallagher
2014-10-02 11:53:58 UTC
Permalink
Post by Stephen Gallagher
== The Problem ==
It is very common for users to have systems with encrypted root
partitions (or even just /var and /etc). This may be due to a personal
concern for their data or a corporate policy mandating full-disk
encryption. Disk encryption requires a password (or other more
complicated credentials) be be presented just after the kernel is
booted and before the drives can be mounted and their data read.
With the current implementation of the offline updates in Fedora, this
leads to a very unpleasant user experience when updating. We offer two
ways to perform offline updates in the default user environment of
Fedora Workstation: "Install updates and reboot" or "Install updates
and shut down".
1) The system shuts down and initiates an ACPI reboot.
2) The system presents the kernel boot menu and then starts the
updater kernel.
3) The system presents the user with a password prompt for the disk
encryption (possibly more than one, if the system is configured with
different passwords for different partitions).
4) The offline updates occur.
5) The system shuts down and initiates an ACPI reboot.
6) The system presents the kernel boot menu and then starts the
standard (possibly updated) kernel.
7) The system presents the user with a password prompt for the disk
encryption (possibly more than one, if the system is configured with
different passwords for different partitions).
8) The system completes booting.
During this experience, the user has been required to enter their disk
encryption password *twice*. The same is true for the "Install and
shut down" case, except that the two passwords are separated by some
actual wallclock time.
== Proposed Improvements ==
We could significantly improve this situation by allowing the system
to drop directly from the interactive system into the updater
environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for
performing updates that would essentially stop all processes except
for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is
completed to restore operation to the standard boot targets of systemd
so that the system comes back up without having to perform a total
reboot. The exceptional case would of course be that in which either
the kernel, libc or systemd[1] needed to be updated, in which case a
reboot could be performed.
In this scenario, we can reduce the number of encrypted disk
challenges to at most a single one, and that only if absolutely
minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
[1] I'm told that this might not be necessary; that systemd can
re-exec itself to pick up its own updates. That would reduce the scope
presumably to "only the kernel" forcing reboots.
I'm bumping this thread to get comments from Lennart (CCed). There's a
lot of chatter in the conversation, but I'd very much like to hear an
answer to the specific questions I posed in this first email.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20141002/d93c0ca8/attachment.sig>
Stephen Gallagher
2014-10-13 15:26:50 UTC
Permalink
Post by Stephen Gallagher
Post by Stephen Gallagher
== The Problem ==
It is very common for users to have systems with encrypted root
partitions (or even just /var and /etc). This may be due to a personal
concern for their data or a corporate policy mandating full-disk
encryption. Disk encryption requires a password (or other more
complicated credentials) be be presented just after the kernel is
booted and before the drives can be mounted and their data read.
With the current implementation of the offline updates in Fedora, this
leads to a very unpleasant user experience when updating. We offer two
ways to perform offline updates in the default user environment of
Fedora Workstation: "Install updates and reboot" or "Install updates
and shut down".
1) The system shuts down and initiates an ACPI reboot.
2) The system presents the kernel boot menu and then starts the
updater kernel.
3) The system presents the user with a password prompt for the disk
encryption (possibly more than one, if the system is configured with
different passwords for different partitions).
4) The offline updates occur.
5) The system shuts down and initiates an ACPI reboot.
6) The system presents the kernel boot menu and then starts the
standard (possibly updated) kernel.
7) The system presents the user with a password prompt for the disk
encryption (possibly more than one, if the system is configured with
different passwords for different partitions).
8) The system completes booting.
During this experience, the user has been required to enter their disk
encryption password *twice*. The same is true for the "Install and
shut down" case, except that the two passwords are separated by some
actual wallclock time.
== Proposed Improvements ==
We could significantly improve this situation by allowing the system
to drop directly from the interactive system into the updater
environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for
performing updates that would essentially stop all processes except
for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is
completed to restore operation to the standard boot targets of systemd
so that the system comes back up without having to perform a total
reboot. The exceptional case would of course be that in which either
the kernel, libc or systemd[1] needed to be updated, in which case a
reboot could be performed.
In this scenario, we can reduce the number of encrypted disk
challenges to at most a single one, and that only if absolutely
minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
[1] I'm told that this might not be necessary; that systemd can
re-exec itself to pick up its own updates. That would reduce the scope
presumably to "only the kernel" forcing reboots.
I'm bumping this thread to get comments from Lennart (CCed). There's a
lot of chatter in the conversation, but I'd very much like to hear an
answer to the specific questions I posed in this first email.
Trying one more time to get Lennart to chime in here.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20141013/b34064da/attachment-0001.sig>
Stephen Gallagher
2014-10-21 18:43:59 UTC
Permalink
Post by Stephen Gallagher
Post by Stephen Gallagher
Post by Stephen Gallagher
== The Problem ==
It is very common for users to have systems with encrypted root
partitions (or even just /var and /etc). This may be due to a personal
concern for their data or a corporate policy mandating full-disk
encryption. Disk encryption requires a password (or other more
complicated credentials) be be presented just after the kernel is
booted and before the drives can be mounted and their data read.
With the current implementation of the offline updates in Fedora, this
leads to a very unpleasant user experience when updating. We offer two
ways to perform offline updates in the default user environment of
Fedora Workstation: "Install updates and reboot" or "Install updates
and shut down".
1) The system shuts down and initiates an ACPI reboot.
2) The system presents the kernel boot menu and then starts the
updater kernel.
3) The system presents the user with a password prompt for the disk
encryption (possibly more than one, if the system is configured with
different passwords for different partitions).
4) The offline updates occur.
5) The system shuts down and initiates an ACPI reboot.
6) The system presents the kernel boot menu and then starts the
standard (possibly updated) kernel.
7) The system presents the user with a password prompt for the disk
encryption (possibly more than one, if the system is configured with
different passwords for different partitions).
8) The system completes booting.
During this experience, the user has been required to enter their disk
encryption password *twice*. The same is true for the "Install and
shut down" case, except that the two passwords are separated by some
actual wallclock time.
== Proposed Improvements ==
We could significantly improve this situation by allowing the system
to drop directly from the interactive system into the updater
environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for
performing updates that would essentially stop all processes except
for systemd and then apply the updates?
In an ideal world, it would then also be possible after update is
completed to restore operation to the standard boot targets of systemd
so that the system comes back up without having to perform a total
reboot. The exceptional case would of course be that in which either
the kernel, libc or systemd[1] needed to be updated, in which case a
reboot could be performed.
In this scenario, we can reduce the number of encrypted disk
challenges to at most a single one, and that only if absolutely
minimal plumbing packages saw an update.
I'd very much like to hear from the plumbers on this matter.
[1] I'm told that this might not be necessary; that systemd can
re-exec itself to pick up its own updates. That would reduce the scope
presumably to "only the kernel" forcing reboots.
I'm bumping this thread to get comments from Lennart (CCed). There's a
lot of chatter in the conversation, but I'd very much like to hear an
answer to the specific questions I posed in this first email.
Trying one more time to get Lennart to chime in here.
I spotted on another mailing list that Lennart is using a different
email address these days. I'm now trying to reach him on that one
instead.
Lennart Poettering
2014-10-21 20:02:08 UTC
Permalink
Post by Stephen Gallagher
It is very common for users to have systems with encrypted root
partitions (or even just /var and /etc). This may be due to a
personal
Nitpicking: we currently do not support split-off /etc on
Fedora/Dracut. /var may be split out though.
Post by Stephen Gallagher
We could significantly improve this situation by allowing the system
to drop directly from the interactive system into the updater
environment without doing a full reboot or relaunching the kernel.
Lennart, would it be possible to set up a special systemd target for
performing updates that would essentially stop all processes except
for systemd and then apply the updates?
Well, the entire logic we have in place for offline updates is to
ensure that they are applied in a well-defined very clean, minimal
environment. Actually, that's not just the logic that is in place,
that's the whole reason for having them at all!

I mean, if it is OK to allow leaking runtime state into the upgrade
process we could just run the updates in parallel to normal runtime,
how we always did things.

Or in even otherwords: offline updates are supposed to make update
strictly reliable, and for that I figure a full reset and fresh boot,
with everything initialized to the boot-up defaults is the closest if
not only option.

That all said, we have been discussing supporting a new reboot-mode in
systemd, which would essentially shut down userspace, until only PID 1
remains, and then immediately start userspace again on the same
kernel. This would be pretty close to what you are asking for
here... However, so far we have quite some concerns about adding this,
precisely for the reason that it pretends to be a reset of everything
to the boot-up defaults, but actually isn't, as a ton of runtime state
is retained in /sys and /proc/sys and other runtime objects.
Post by Stephen Gallagher
[1] I'm told that this might not be necessary; that systemd can
re-exec itself to pick up its own updates. That would reduce the scope
presumably to "only the kernel" forcing reboots.
Well, you cannot really do this atomically. Offline updates are about
reliability. But you cannot restart userspace reliably during
runtime. Think about this: daemons talk to each other, supporting a
variety of APIs and API versions. If we want to do an update, then
we'd have to restart all daemons at the same instant to ensure that no
daemon starts talking with another daemon at the wrong time where it
might be confused by either a too old or too new daemon. However,
restarten them all at once is not possible. Moreover, some daemons
cannot be restarted at all (for example dbus-daemon; and journald is
difficult too, as it will lose connections to all other daemon's
stdout/stderr if you do, even though it other works fine).

Hence: I'd really recommend leaving offline updates as is.

I see the problem though and acknowledge the fact that entering the
passwords multiple times is annoying. Interestingly hw hdd encryption
is nicer there as the keys are not flushed out on resets... Maybe
that's actually a strategy to adopt here: upload the encryption keys
into the firmware as efi vars, and then pull them out on next boots or
so (assuming that efi vars can be marked to survive soft reboots
without making them fully persistent...)

I understand this is all unsatisfying. Maybe fixing related problems
might improve the situation a bit though. For example, I find it quite
annoying if we always ask for both a hdd passphrase and a user
passphrase at boot. It has been a long-time todo list item for systemd
to add the hdd passphrases (with strict expiry) to the kernel keyring
and then optionally pull them out there for gdm's autologin mode, so
that they can be used for decrypting the gnome keyring and
such. However, so far we haven't worked on that yet...

Lennart
--
Lennart Poettering, Red Hat
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproje
Roberto Ragusa
2014-10-22 12:11:57 UTC
Permalink
Post by Lennart Poettering
Maybe
that's actually a strategy to adopt here: upload the encryption keys
into the firmware as efi vars, and then pull them out on next boots or
so (assuming that efi vars can be marked to survive soft reboots
without making them fully persistent...)
Hmmm, surrendering your encryption keys to the only software
part which you do not have control on?
--
Roberto Ragusa mail at robertoragusa.it
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-
Lennart Poettering
2014-10-22 13:18:52 UTC
Permalink
Post by Roberto Ragusa
Post by Lennart Poettering
Maybe
that's actually a strategy to adopt here: upload the encryption keys
into the firmware as efi vars, and then pull them out on next boots or
so (assuming that efi vars can be marked to survive soft reboots
without making them fully persistent...)
Hmmm, surrendering your encryption keys to the only software
part which you do not have control on?
The firmware runs at the highest priviliges anyway, it can do whatever
it wants... You don't have to "surrender" you keys to it. If it wants
them it can take them anyway...

Lennart
--
Lennart Poettering, Red Hat
--
devel mailing list
***@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: ht
Loading...