[arch-general] Updating linux-lts initcpios... error: command failed to execute correctly

Post by Ralph Corderoy
Hi,
This morning's `pacman -Su' had a problem building the fallback initcpio
image for 4.14.79-1-lts. Yesterday's /var/log/pacman.log shows
[2018-11-06 10:45] [ALPM] upgraded linux-lts (4.14.78-1 -> 4.14.79-1)
[2018-11-06 10:45] [ALPM] running '60-linux-lts.hook'...
[2018-11-06 10:45] [ALPM] running '90-linux-lts.hook'...
that all went well.
Today, package `linux-lts' wasn't upgraded but 90-linux-lts.hook was
triggered. 60-linux-lts.hook wasn't.

The 60- hook is:
Description = Updating linux-lts module dependencies...

and its purpose is to run depmod whenever you update modules.

The 90- hook is:
Description = Updating linux-lts initcpios...

and its purpose is to run mkinitcpio whenever you update either the
kernel, or else any file which is part of the mkinitcpio configuration.

$ pacman -Qo /usr/lib/initcpio/
/usr/lib/initcpio/ is owned by archiso 39-1
/usr/lib/initcpio/ is owned by btrfs-progs 4.19-1
/usr/lib/initcpio/ is owned by cryptsetup 2.0.5-1
/usr/lib/initcpio/ is owned by device-mapper 2.02.182-1
/usr/lib/initcpio/ is owned by dmraid 1.0.0.rc16.3-11
/usr/lib/initcpio/ is owned by lvm2 2.02.182-1
/usr/lib/initcpio/ is owned by mdadm 4.0-1
/usr/lib/initcpio/ is owned by mkinitcpio 25-1
/usr/lib/initcpio/ is owned by mkinitcpio-busybox 1.29.3-1
/usr/lib/initcpio/ is owned by systemd 239.300-1

As you can see, there's lots of things which can do so!

This explains why one ran and the other did not.

Post by Ralph Corderoy
pacman's TTY output complained
`error: command failed to execute correctly' and the list of `Running'
build hooks didn't complete. The same steps for package `linux'
immediately after didn't have the same problem.

This means mkinitcpio itself got borked, but rerunning it with a
different kernel preset works okay... why, remains to be seen.

Post by Ralph Corderoy
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
-> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
==> Starting build: 4.14.79-1-lts
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
â error: command failed to execute correctly

So, the fallback image for linux-lts seems to have simply died halfway
through (but the main image generated okay).

Post by Ralph Corderoy
I was hoping pacman.log might have more detail, but it doesn't even
record that there was an error. I think that's a bug.
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> Starting build: 4.14.79-1-lts
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2018-11-07 11:18] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: wd719x
[2018-11-07 11:18] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: aic94xx
[2018-11-07 11:19] [ALPM] running '90-linux.hook'...
[2018-11-07 11:19] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'

Well, the log only contains a copy of the output from the hook which you
can look up after the fact, so it definitely won't have *more* input. As
for logging the failure of the hook to complete, I guess you could open
a ticket on the bugtracker for this (and/or post to the pacman-dev
mailing list if that floats your boat).

Post by Ralph Corderoy
Where should I look for clues why the command failed?
And after that, is attempting another build of the image as simple as
`/usr/bin/mkinitcpio -p linux-lts' as root?

This looks like it is another occurrence of
https://bugs.archlinux.org/task/60396

To test, you could try reinstalling a package that provides a mkinitcpio
configuration, and test resizing the terminal to see if you get the same
error.

Then, try installing pacman-git from the AUR or from my unofficial repo
hosted at
https://wiki.archlinux.org/index.php/Unofficial_user_repositories#eschwartz

The fix should be in pacman-git, and this should no longer crash.

--
Eli Schwartz
Bug Wrangler and Trusted User

Ralph Corderoy

2018-11-07 15:06:37 UTC

Hi Eli,

Post by Eli Schwartz via arch-general
$ pacman -Qo /usr/lib/initcpio/
/usr/lib/initcpio/ is owned by archiso 39-1

...

Thanks, it hadn't occurred to me to use -Qo on a directory before.

Post by Eli Schwartz via arch-general
Well, the log only contains a copy of the output from the hook which
you can look up after the fact, so it definitely won't have *more*
input.

Understood. I was thinking along the lines of how a ./configure writes
to an agreed file descriptor other than stdout/stderr for more verbose
detail, that kind of thing, but there weren't extra messages to record
anyway given below suggests it's a SIGPIPE. Would the exit status of
the hook have reflected that signal by being 128+13?

Post by Eli Schwartz via arch-general
As for logging the failure of the hook to complete, I guess you could
open a ticket on the bugtracker for this

Okay. https://bugs.archlinux.org/task/60735

Post by Eli Schwartz via arch-general
This looks like it is another occurrence of
https://bugs.archlinux.org/task/60396

It's very likely. I know I did maximise the X terminal part-way through
pacman's run to unwrap lines.

Post by Eli Schwartz via arch-general
To test...

I'll skip that, sorry. Way behind with life as it is at the moment.
I've looked at the bug report and the patch and it seems sound.

Post by Ralph Corderoy
And after that, is attempting another build of the image as simple
as `/usr/bin/mkinitcpio -p linux-lts' as root?

I did this and it seems to have gone well.

Thanks for the prompt answers.

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

Eli Schwartz via arch-general

2018-11-07 15:19:09 UTC

Post by Ralph Corderoy
Hi Eli,

Post by Eli Schwartz via arch-general
$ pacman -Qo /usr/lib/initcpio/
/usr/lib/initcpio/ is owned by archiso 39-1

...
Thanks, it hadn't occurred to me to use -Qo on a directory before.

It's pretty nifty, yeah. Manpage and --help output does say "file(s)",
though -- maybe we should change that.

Post by Eli Schwartz via arch-general
Well, the log only contains a copy of the output from the hook which
you can look up after the fact, so it definitely won't have *more*
input.

Yes, pacman just checks if the exit status of the hook was something
indicating an error.

Post by Eli Schwartz via arch-general
As for logging the failure of the hook to complete, I guess you could
open a ticket on the bugtracker for this

Okay. https://bugs.archlinux.org/task/60735

Thanks. <3

Post by Eli Schwartz via arch-general
This looks like it is another occurrence of
https://bugs.archlinux.org/task/60396

It's very likely. I know I did maximise the X terminal part-way through
pacman's run to unwrap lines.

Post by Eli Schwartz via arch-general
To test...

I'll skip that, sorry. Way behind with life as it is at the moment.
I've looked at the bug report and the patch and it seems sound.

Sure, I wasn't expecting problems -- it's been tested already. :)

But of course you are free to use my git repo either way -- it's
verified by my packaging key already, it contains cool new features, and
I am good about keeping it up to date.

Anyway, it's planned to be added to a maintenance release for pacman
5.1.2, hopefully soon.

Post by Ralph Corderoy
And after that, is attempting another build of the image as simple
as `/usr/bin/mkinitcpio -p linux-lts' as root?

I did this and it seems to have gone well.

Forgot to mention, but yeah, running the command that the hook was
supposed to run, will usually work. :D

If it doesn't work, it would be because the hook specifies NeedsTargets
and therefore expects to receive a list of files on standard input,
listing the files which triggered the hook.

Post by Ralph Corderoy
Thanks for the prompt answers.

No problem! :)

--
Eli Schwartz
Bug Wrangler and Trusted User

l***@gmx.fr

2018-11-07 18:50:27 UTC

Post by Ralph Corderoy
Hi,
This morning's `pacman -Su' had a problem building the fallback initcpio

Right command is pacman -Syu

Post by Ralph Corderoy
image for 4.14.79-1-lts. Yesterday's /var/log/pacman.log shows
[2018-11-06 10:45] [ALPM] upgraded linux-lts (4.14.78-1 -> 4.14.79-1)
[2018-11-06 10:45] [ALPM] running '60-linux-lts.hook'...
[2018-11-06 10:45] [ALPM] running '90-linux-lts.hook'...

Ralph Corderoy

2018-11-07 22:46:37 UTC

Hi leoutation,

Post by l***@gmx.fr

Post by Ralph Corderoy
This morning's `pacman -Su' had a problem building the fallback initcpio

Right command is pacman -Syu

Not in my case as I've previously done a `pacman -Syuw' during `free
bandwidth' hours, thanks to cron. Later on, when I'm at the keyboard
and it's a convenient time for breakage, I upgrade to what's already
been downloaded manually and definitely don't want `-y'. :-)

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

Eli Schwartz via arch-general

2018-11-07 23:16:15 UTC

Post by Ralph Corderoy
Hi leoutation,

Post by l***@gmx.fr

Post by Ralph Corderoy
This morning's `pacman -Su' had a problem building the fallback initcpio

Right command is pacman -Syu

That's dangerous because you could forget and run pacman -S pkgname. Not
saying you would, and in fact I've done -Sywu all the time myself...

But, just because I do it doesn't mean I should. ;)

Instead, I eventually did some fiddling with the checkupdates script,
which can be seen here:
https://git.archlinux.org/users/eschwartz/pacman-contrib.git/commit/?id=80be77ce7d3e1ba8eee42c56623e01f0c45235cd

This will eventually make its way to pacman-contrib upstream, but it's
pending some pacman changes which are still in git and only very
recently landed.

In the meantime you can also use
https://github.com/eli-schwartz/dotfiles/blob/master/bin/cacheupdates

Just run cacheupdates the way you'd run checkupdates, but if you specify
the argument "-d" (or "--download") then it will also try to download
any pending updates into the cache, still using the temporary copy of
the syncdbs.

--
Eli Schwartz
Bug Wrangler and Trusted User

Ralph Corderoy

2018-11-08 17:25:13 UTC

Hi Eli,

Post by Ralph Corderoy
Not in my case as I've previously done a `pacman -Syuw' during `free
bandwidth' hours, thanks to cron. Later on, when I'm at the
keyboard and it's a convenient time for breakage, I upgrade to
what's already been downloaded manually and definitely don't want
`-y'. :-)

That's dangerous because you could forget and run pacman -S pkgname.

AIUI, if I did then the version of pkgname it would try to download to
install would be that snap-shotted when cron ran `pacman -Syuw'. And
that's a problem because that version might not be available for
download any more if a later one has been made available? Similarly,
with packages pkgname depends on. Is there another problem I don't
know?

I actually tend to: sudo -i at 1:23 <<<'pacman -Syw pkgname' as it's
normally not urgent.

Can an attempt to fetch an old, now missing, package still occur even
with `pacman -Sy pkgname' because the remote database has been altered,
and old versions removed, between the -y's refresh and fetching a long
list of packages?

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

Eli Schwartz via arch-general

2018-11-08 17:37:30 UTC

Post by Ralph Corderoy
Hi Eli,

That's dangerous because you could forget and run pacman -S pkgname.

-_-

No, the problem is that you'll accidentally do a partial update.

The package itself might very well be cached already, but if not, you'll
get a 404 "cannot find package, I will now error without doing anything".

That's not "dangerous". Dangerous is when you break your system by
installing incompatible packages via a "successful" partial update.

Post by Ralph Corderoy
I actually tend to: sudo -i at 1:23 <<<'pacman -Syw pkgname' as it's
normally not urgent.
Can an attempt to fetch an old, now missing, package still occur even
with `pacman -Sy pkgname' because the remote database has been altered,
and old versions removed, between the -y's refresh and fetching a long
list of packages?

It won't be old and now missing, because you used -Sy and it will
refresh *again*.

But if you don't use -y, then trying to install a package first looks in
the cache, where it may not need to be fetched...

--
Eli Schwartz
Bug Wrangler and Trusted User

Ralph Corderoy

2018-11-08 18:05:58 UTC

Hi Eli,

Dangerous is when you break your system by installing incompatible
packages via a "successful" partial update.

Ah, right. I `pacman -S foo'. foo depends on libbar. libbar 1.0 is
already installed because xyzzy needs it. The earlier `pacman -Syuw'
updated the local database to know libbar 1.1 is available thus
installing foo upgrades libbar. xyzzy remains at the old version,
incompatible with libbar 1.1.

Post by Ralph Corderoy
Can an attempt to fetch an old, now missing, package still occur
even with `pacman -Sy pkgname' because the remote database has been
altered, and old versions removed, between the -y's refresh and
fetching a long list of packages?

It won't be old and now missing, because you used -Sy and it will
refresh *again*.

But isn't it multiple stages with a race condition?

1 Update /var/lib/pacman/sync/core.db
2 Update /var/lib/pacman/sync/extra.db
3 Update /var/lib/pacman/sync/community.db
4 Update /var/lib/pacman/sync/multilib.db
5 Create /var/cache/pacman/pkg/foo-1.0.pkg.tar.xz
6 Create /var/cache/pacman/pkg/bar-2.0.pkg.tar.xz
7 Create /var/cache/pacman/pkg/xyzzy-3.0.pkg.tar.xz

Can the remote core.db be updated between 1 and 6 because bar 2.1 is
available? Do bar-2.0.pkg.tar.xz and bar-2.1.pkg.tar.xz ever exist at
the same time? Perhaps the fetch of bar-2.0.pkg.tar.xz might fail
because it's been removed by the time it's attempted.

This is nothing to do with my original question. :-) I guess I'm
asking how is a consistent multi-file database of indexes and `records'
presented? It might be that the rare times this happens, the user just
runs `pacman -Sy pkgname' again.

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

Eli Schwartz via arch-general

2018-11-08 19:32:44 UTC