[systemd-devel] Starting a service at shutdown time, with requirements

[Colin Walters]

I'm working on: https://github.com/ostreedev/ostree/issues/545
TL;DR:
libostree is an image-like update system, and I want to take some
actions at system shutdown time, specifically performing a "snapshot+merge"
of /etc after most other services are shut down. The way ostree
handles /etc (IMO a lot more correctly) is a major distinguishing point from a lot
of other image-like systems out there.

What I have so far is this (distilled into a simple easily tested unit not tied to ostree):

[Unit]
Description=cgwalters-shutdown
DefaultDependencies=no
After=shutdown.target
Before=final.target
# FIXME
# Before=sysroot.mount
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'sleep 5 && ls -ald /sysroot/ostree/repo'
[Install]
WantedBy=final.target

I modeled this after `halt-local.service`. But as you
might infer from the FIXME, the problem I'm hitting is that the /sysroot
mount is unmounted before this service is started. I think the problem

Given two units with any ordering dependency between them, if one unit is shut down and the other is started up, the shutdown is ordered before the start-up.

Which does sort of make sense to me in general; but is problematic for
this "start service on shutdown" case. I was looking a bit at the way plymouth
hooks into shutdown, but it seemed complex, and as plymouth doesn't (AFAIK)
have any real dependencies on the filesystem state I'm not sure it matches what I need.

I can think of ways to hack around this, by e.g. having the service start up on
shutdown, take a snapshot of a private mount namespace, have it be excluded
from the Conflicts=shutdown.target, then have a unit that triggers before final.target
(like the above) that sends an IPC to actually do what I want. But that adds a notable
amount of complexity, and I'd like to have this process feel as obvious/transparent as possible.

Another way I've thought about handling this is to basically invert things so that
we have a "stub" unit that starts on bootup, and its ExecStop does the real work:

[Unit]
Description=cgwalters-startstop
ConditionPathExists=/run/ostree-booted
DefaultDependencies=no
RequiresMountsFor=/sysroot
After=basic.target
Before=multi-user.target
Conflicts=final.target
Before=final.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/true
ExecStop=/usr/bin/sh -c 'sleep 5 && ls -ald /ostree/repo'

[Install]
WantedBy=multi-user.target

Which seems to work but feels...a bit ugly?

[Mantas Mikulėnas]

Post by Colin Walters
I'm working on: https://github.com/ostreedev/ostree/issues/545
libostree is an image-like update system, and I want to take some
actions at system shutdown time, specifically performing a "snapshot+merge"
of /etc after most other services are shut down. The way ostree
handles /etc (IMO a lot more correctly) is a major distinguishing point from a lot
of other image-like systems out there.
[Unit]
Description=cgwalters-shutdown
DefaultDependencies=no
After=shutdown.target
Before=final.target
# FIXME
# Before=sysroot.mount
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'sleep 5 && ls -ald /sysroot/ostree/repo'
[Install]
WantedBy=final.target
I modeled this after `halt-local.service`. But as you
might infer from the FIXME, the problem I'm hitting is that the /sysroot
mount is unmounted before this service is started. I think the problem

Given two units with any ordering dependency between them, if one unit

is shut down and the other is started up, the shutdown is ordered before
the start-up.
Which does sort of make sense to me in general; but is problematic for
this "start service on shutdown" case. I was looking a bit at the way plymouth
hooks into shutdown, but it seemed complex, and as plymouth doesn't (AFAIK)
have any real dependencies on the filesystem state I'm not sure it matches what I need.
I can think of ways to hack around this, by e.g. having the service start up on
shutdown, take a snapshot of a private mount namespace, have it be excluded
from the Conflicts=shutdown.target, then have a unit that triggers before final.target
(like the above) that sends an IPC to actually do what I want. But that adds a notable
amount of complexity, and I'd like to have this process feel as
obvious/transparent as possible.
Another way I've thought about handling this is to basically invert things so that
[Unit]
Description=cgwalters-startstop
ConditionPathExists=/run/ostree-booted
DefaultDependencies=no
RequiresMountsFor=/sysroot
After=basic.target
Before=multi-user.target
Conflicts=final.target
Before=final.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/true
ExecStop=/usr/bin/sh -c 'sleep 5 && ls -ald /ostree/repo'
[Install]
WantedBy=multi-user.target
Which seems to work but feels...a bit ugly?

Currently this is the recommended method; it's a little ugly indeed but
fits quite well into the shutdown process. Recent systemd versions should
let you skip the dummy ExecStart.

Post by Colin Walters
--

Mantas Mikulėnas <***@gmail.com>
Sent from my phone

[Simon McVittie]

Post by Colin Walters
Another way I've thought about handling this is to basically invert things so that
we have a "stub" unit that starts on bootup, and its ExecStop does the real
work
Currently this is the recommended method; it's a little ugly indeed but fits
quite well into the shutdown process. Recent systemd versions should let you
skip the dummy ExecStart.

The unattended-upgrades tool on Debian/Ubuntu works like this when
configured to install updates during shutdown (the same idea as systemd's
"reboot to a special mode, install updates, reboot again", but with fewer
reboots). I can confirm that in recent-ish systemd (v232 in Debian 9,
although not the older v215 in Debian 8) the ExecStart is unnecessary.

smcv

[Lennart Poettering]

Post by Colin Walters
Another way I've thought about handling this is to basically invert things so that

This is the correct and recommended way to do this.

Post by Colin Walters
[Unit]
Description=cgwalters-startstop
ConditionPathExists=/run/ostree-booted
DefaultDependencies=no
RequiresMountsFor=/sysroot
After=basic.target
Before=multi-user.target
Conflicts=final.target
Before=final.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/true

You can drop this line. On current systemd versions services without
ExecStart= are perfectly fine, as long as they have ExecStop=,
precisely to cover this usecase nicely.

Post by Colin Walters
ExecStop=/usr/bin/sh -c 'sleep 5 && ls -ald /ostree/repo'
[Install]
WantedBy=multi-user.target
Which seems to work but feels...a bit ugly?

I think it's relatively pretty, as it means you have to "start"
something explicitly so that it can run code at shutdown. I'd claim
this is a semantical benefit, not a malus.

Lennart

[Colin Walters]

Post by Lennart Poettering

Post by Colin Walters
Another way I've thought about handling this is to basically invert things so that

This is the correct and recommended way to do this.

Thanks!

Post by Lennart Poettering
I think it's relatively pretty, as it means you have to "start"
something explicitly so that it can run code at shutdown. I'd claim
this is a semantical benefit, not a malus.

Right. It does make sense also to me that basically we want to
"hold a reference" to any resources required (in this case the /sysroot mount).

But so `halt-local.service` is just busted? I guess no one really cares
that much about it today, I may see though about submitting a PR to add
a warning not to use it as a model.

[Lennart Poettering]

Post by Colin Walters

Post by Lennart Poettering

Post by Colin Walters
Another way I've thought about handling this is to basically invert things so that

This is the correct and recommended way to do this.

Thanks!

Post by Lennart Poettering
I think it's relatively pretty, as it means you have to "start"
something explicitly so that it can run code at shutdown. I'd claim
this is a semantical benefit, not a malus.

Right. It does make sense also to me that basically we want to
"hold a reference" to any resources required (in this case the /sysroot mount).
But so `halt-local.service` is just busted? I guess no one really cares
that much about it today, I may see though about submitting a PR to add
a warning not to use it as a model.

So basically, during shutdown there's this moment of singularity,
where we switch from "negative" operation to "positive" operation,
i.e. where we switch from *stopping* running stuff and move into
*starting* the final shutdown code. The halt-local.service stuff as well
as systemd-poweroff.service and similar are among the latter.

Of course, you can decide to plug in your stuff into that phase of the
boot too, but if so you can't really have any deps anymore, you are
basically running with *everything* already turned off. And in most
cases this is not what you want: you actually want some stuff around
still, for example in your case sysroot.mount and suchlike.


Lennart