[systemd-devel] How to specify dynamic services/requirements

Discussion:

Luiz Angelo Daros de Luca

2017-09-13 04:08:22 UTC

Hello,

I'm facing a problem with Xen machines that depends on MD devices.
As I'm new to systemd world, I might not be seeing the clean solution.
That's why I'm asking for an advice.

MD devices are automatically detect by udev. If device state is safe,
/dev/md/myraid is started. However, if the state is not safe (like a raid
with a missing disk), udev starts mdadm-last-***@.timer that tries to
start the device anyway after 30s. As mdadm-last-resort conflicts with the
device presence, it will not run if the required disk appears before 30s.
Even with mdadm-last-resort running, MD device might still be usable,
although degraded.

Xen VM are started by xendomains.service, that simply calls a shell script
like in SysV times. It start a bunch of VM in sequence. xendomains.service
has only generic dependencies that let it run at a very early stage.

Now the problem: If for any reason, a MD device takes some seconds to
appear (or even 30s as the last resort), xendomains will fail to start any
machine that depends on that MD device.

I'm extending manually xendomains.service to depend on a series of MD
devices, that fixed the start order problem. However, I created new
problems. First I had to frequently regenerate those "Requires" as machines
are frequently started/migrated between hosts (I also consider using
systemd generators). But worse, whenever a single MD device permanently
fails, xendomains is never started and all those VM that does not use the
failed MD never start.

I though that maybe I could use instances (***@vm1.service) to
launch VM individually, each of them depending on those devices it uses.
However, these instances would have to be dynamically generated based on
its configuration (systemd generators from /etc/xen/vm/xxx.cfg?), either on
boot, shutdown or simply periodically. The stop procedure will still be the
same, calling the SysV script as systemd will not know about VM (re)started
after boot.

I even though about simply create an alternative xendomains.service that
does not depend on any MD device and launch it using a systemd timer if the
xendomains that depends on MD devices isn't started after 30s. It looks
ugly but it might work.

Regards,

--
Luiz Angelo Daros de Luca
***@gmail.com

Jérémy Rosen

2017-09-13 07:29:13 UTC

Permalink

Post by Luiz Angelo Daros de Luca
Hello,
I'm facing a problem with Xen machines that depends on MD devices.
As I'm new to systemd world, I might not be seeing the clean solution.
That's why I'm asking for an advice.
MD devices are automatically detect by udev. If device state is safe,
/dev/md/myraid is started. However, if the state is not safe (like a
tries to start the device anyway after 30s. As mdadm-last-resort
conflicts with the device presence, it will not run if the required
disk appears before 30s. Even with mdadm-last-resort running, MD
device might still be usable, although degraded.
Xen VM are started by xendomains.service, that simply calls a shell
script like in SysV times. It start a bunch of VM in sequence.
xendomains.service has only generic dependencies that let it run at a
very early stage.

I would cut that script in multiple services... but you mention doing
that below, so I'm just saying I think it's the right way to go :)

Post by Luiz Angelo Daros de Luca
Now the problem: If for any reason, a MD device takes some seconds to
appear (or even 30s as the last resort), xendomains will fail to start
any machine that depends on that MD device.
I'm extending manually xendomains.service to depend on a series of MD
devices, that fixed the start order problem. However, I created new
problems. First I had to frequently regenerate those "Requires" as
machines are frequently started/migrated between hosts (I also
consider using systemd generators). But worse, whenever a single MD
device permanently fails, xendomains is never started and all those VM
that does not use the failed MD never start.
launch VM individually, each of them depending on those devices it
uses. However, these instances would have to be dynamically generated
based on its configuration (systemd generators from
/etc/xen/vm/xxx.cfg?), either on boot, shutdown or simply
periodically. The stop procedure will still be the same, calling the
SysV script as systemd will not know about VM (re)started after boot.

you could create a template (***@.service) and use drop-ins to do
per-instance overrides
(/etc/systemd/system/***@vm1.service.d/append.conf) that would
allow you to individualize each domain while keeping the common parts

I would find a way to prevent a domain with no corresponding to start
(maybe by not defining a mandatory key in the template) to make sure no
rogue domains are created...

Post by Luiz Angelo Daros de Luca
I even though about simply create an alternative xendomains.service
that does not depend on any MD device and launch it using a systemd
timer if the xendomains that depends on MD devices isn't started after
30s. It looks ugly but it might work.
Regards,
--
Luiz Angelo Daros de Luca
_______________________________________________
systemd-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

--
Logo <http://www.smile.fr/>

20 rue des Jardins
92600 AsniÃšres-sur-Seine
www.smile.fr <http://www.smile.fr/>
*JÃ©rÃ©my ROSEN*
Architecte technique
Email : ***@smile.fr <mailto:***@smile.fr>
Tel : +33141402967

Facebook <https://www.facebook.com/smileopensource> Google%2B
<http://fr.slideshare.net/SmileOpenSource/presentations> LinkedIn
<https://www.linkedin.com/company/smile> Twitter
<https://twitter.com/GroupeSmile>

bandeaux_mail
<http://www.smile.fr/Offres-services/Offres/Ingenierie?utm_source=signature&utm_medium=email&utm_campaign=signature>

eco Pour la planÃšte, n'imprimez ce mail que si c'est nÃ©cessaire

Luiz Angelo Daros de Luca

2017-09-15 05:27:18 UTC

Permalink

Thanks JÃ©rÃ©my,

That was what I needed. I was afraid to be headed to the wrong way. Now
that I know what to do, I need to know where and when.

that would allow you to individualize each domain while keeping the common
parts
I would find a way to prevent a domain with no corresponding to start
(maybe by not defining a mandatory key in the template) to make sure no
rogue domains are created...

I would use a template like ***@.service in order to specify common
stuff. However, the config of which vm are enabled (autostart) is only
available at boot time. Is a systemd generator the way to go? Is is there
another way to define it?

Also, should I use a service patch (like append.conf you mentioned) or
simply directly create /run/.../***@vm1.service?

It would be trivial to define a template ***@.service and start
services like "systemctl start ***@vm1.service", that would read
/etc/xen/vm/vm1 and launch it. I just don't know how I could define dynamic
dependency for that instance depending on the content of /etc/xen/vm/vm1.
Execstartpre can check requirements and prevent a service to run but not
wait for a resource to be available. It would be something like
ExecRequires=.

Instances seems to be created only when asked directly (systemctl start
***@vm1.service), which might be triggered by something like udev
event. Is there anything like
ExecListStartInstances=, ExecListRunningInstances=?
Generators can create something like that but they are triggered only at
boot and daemon-reload. It would be interesting to somehow run "something
like generators" when start/stop/status is called.

From what I know about systemd, I'm thinking of something like a generator
that will create services for all autostart and saved vm. The current
script-based xendomains.service will run after all ***@.service
instances. This way, a new enabled-after-boot vm would be started with
"systemctl restart xendomains". I could use the existence of
/etc/xen/auto/vm or /var/lib/xen/save/vm as requirement
for ***@vm.service, in order to avoid to start a disabled-after-boot
vm. However, this does not look like an elegant solution.

Regards,

Luiz Angelo Daros de Luca
***@gmail.com

Jérémy Rosen

2017-09-15 08:02:32 UTC

Permalink

Post by Luiz Angelo Daros de Luca
Thanks JÃ©rÃ©my,
That was what I needed. I was afraid to be headed to the wrong way.
Now that I know what to do, I need to know where and when.
overrides
would allow you to individualize each domain while keeping the
common parts
I would find a way to prevent a domain with no corresponding to
start (maybe by not defining a mandatory key in the template) to
make sure no rogue domains are created...
common stuff. However, the config of which vm are enabled (autostart)
is only available at boot time. Is a systemd generator the way to go?
Is is there another way to define it?
Also, should I use a service patch (like append.conf you mentioned) or

both method work... the first one allow to add stuff to the common
template, the second to override more easily in the case of a VM that is
totally different from the others...

Post by Luiz Angelo Daros de Luca
read /etc/xen/vm/vm1 and launch it. I just don't know how I could
define dynamic dependency for that instance depending on the content
of /etc/xen/vm/vm1. Execstartpre can check requirements and prevent a
service to run but not wait for a resource to be available. It would
be something like ExecRequires=.

A generator is probably the way to go... generators can create ANY
systemd configuration files, including templates, drop-ins and .wants
directory

You probably already found that, but in case you didn't, you should read
the DESCRIPTION section of man:systemd.unit It describes all the ways
you can add files to configure units

In particular, if you want a service started at boot time, you need to
create a default.target.wants/ directory and add a symbolic link to the
unit to start in there. Populating such a directory at boot time with a
generator might be the best approch for you...

Post by Luiz Angelo Daros de Luca
Instances seems to be created only when asked directly (systemctl
like udev event. Is there anything like
ExecListStartInstances=, ExecListRunningInstances=?

see my answer above. There is no such thing, but generators can do the
equivalent by creating a .wants link

(as a side note, I don't know if you could do a "systemctl enable
--runtime" from a generator. maybe someone else here would know)

Post by Luiz Angelo Daros de Luca
Generators can create something like that but they are triggered only
at boot and daemon-reload. It would be interesting to somehow run
"something like generators" when start/stop/status is called.

what would be your use-case ?
status is not "an event" nothing real happens on status
start/stop can propagate stuff based on their parameters, but the
services themselves are not meant to be dynamic...

That being said, it might be possible to dinamically add dependencies to
a unit via dbus and/or systemctl set-property, but I have never tried
it, so you'll need to try and see

Post by Luiz Angelo Daros de Luca
From what I know about systemd, I'm thinking of something like a
generator that will create services for all autostart and saved vm.
The current script-based xendomains.service will run after
vm would be started with "systemctl restart xendomains". I could use
the existence of /etc/xen/auto/vm or /var/lib/xen/save/vm as
disabled-after-boot vm. However, this does not look like an elegant
solution.

Your requirements seems very complex (and, I have to admit, a bit hard
to understand through e-mail) but I think that with a good understanding
of templates and generators it can be solved elegantly

* use templates to define how to start vms (common parts in the
template, particularities in the instances)
* add all boot-time depedencies via .wants directory via a generator
(maybe usint systemctl enable if it can be run at generator-time, but
i'm not sure it's possible)

Hope this helps

JÃ©rÃ©my

Post by Luiz Angelo Daros de Luca
Regards,
--
Luiz Angelo Daros de Luca