[systemd-devel] runtime directories for services vs. tmpfiles

[Michael Biebl]

Hi,

an interesting issue was raised as part of reviewing a patch for
iodione [1], a system service which needs a runtime directory. We
thought this might need further dicussion, so reposting the issue to
systemd-devel:

For system services needing a runtime directory, we basically have two
(three) options nowadays
1/ use ExecStartPre=/usr/bin/mkdir /var/run/foo
2/ use a tmpfile snippet
(3/ let the daemon create the runtime directory itself)

In [1] we recommended the the usage of 1/ over 2/.
The main reason for that were, that systemd-tmpfiles properly handles
applying security policies, like SELinux labels, and it avoids
spawning unnecessary processes (every ExecStartPre=/usr/bin/mkdir is a
separate process)

Zbyszek is arguing, that splitting the configuration over two files, a
tmpfile and a service file, complicates things for the administrator,
as things are no longer in a single place.
He also argues, that tmpfiles are active, independently of the
service, which needs them. Which can lead to unused runtime
directories.

One suggestion that came up was, to specficy runtime directories in a
declarative fashion in the .service file itself, which means it would
be only be created if the service itself is enabled.

I think this idea warrants further discussion, so I'm posting it here.

Michael


[1] http://lists.alioth.debian.org/pipermail/pkg-systemd-maintainers/Week-of-Mon-20130715/000147.html

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Michael Biebl]

Post by Michael Biebl
For system services needing a runtime directory, we basically have two
(three) options nowadays
1/ use ExecStartPre=/usr/bin/mkdir /var/run/foo
2/ use a tmpfile snippet
(3/ let the daemon create the runtime directory itself)
In [1] we recommended the the usage of 1/ over 2/.

Bah, sorry for the confusion: I meant 2/ over 1/. I.e. we recommended
to use a tmpfiles snippet over ExecStartPre

[Colin Guthrie]

Post by Michael Biebl
Hi,
an interesting issue was raised as part of reviewing a patch for
iodione [1], a system service which needs a runtime directory. We
thought this might need further dicussion, so reposting the issue to
For system services needing a runtime directory, we basically have two
(three) options nowadays
1/ use ExecStartPre=/usr/bin/mkdir /var/run/foo
2/ use a tmpfile snippet
(3/ let the daemon create the runtime directory itself)
In [1] we recommended the the usage of 2/ over 1/.
The main reason for that were, that systemd-tmpfiles properly handles
applying security policies, like SELinux labels, and it avoids
spawning unnecessary processes (every ExecStartPre=/usr/bin/mkdir is a
separate process)
Zbyszek is arguing, that splitting the configuration over two files, a
tmpfile and a service file, complicates things for the administrator,
as things are no longer in a single place.
He also argues, that tmpfiles are active, independently of the
service, which needs them. Which can lead to unused runtime
directories.
One suggestion that came up was, to specficy runtime directories in a
declarative fashion in the .service file itself, which means it would
be only be created if the service itself is enabled.
I think this idea warrants further discussion, so I'm posting it here.

Hmmm, I can certainly see the argument, but it would be a shame to have
multiple ways to achieve the same goal through different places too (it
makes the documentation case much more confusing).

There is also an argument that on first install, one has to call
"systemd-tmpfiles --create $BASENAME" in a %post script to ensure the
necessary dirs are created. If it was all "built in" to the service this
would be fine.

So I see a few options.

1. Bake it into systemd unit syntax and handle it internally.
2. Bake it in as a noop directive and have a generator parse it out and
then write out /run/tmpfiles.d/ snippets and let systemd-tmpfiles work
with that.
3. Keep the status quo.

I don't really like option 2 there - seems unnecessary overhead. But
then option one (which presumably would see the tmpfiles code being
libified and linked to from systemd itself) doesn't seem amazing too.

I guess I don't really have a major problem with either 1 or 3 to be
honest. I can see arguments for both.

Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

[Tom Gundersen]

Hi Michael,

Post by Michael Biebl
an interesting issue was raised as part of reviewing a patch for
iodione [1], a system service which needs a runtime directory. We
thought this might need further dicussion, so reposting the issue to
For system services needing a runtime directory, we basically have two
(three) options nowadays
1/ use ExecStartPre=/usr/bin/mkdir /var/run/foo
2/ use a tmpfile snippet
(3/ let the daemon create the runtime directory itself)
In [1] we recommended the the usage of 1/ over 2/.
The main reason for that were, that systemd-tmpfiles properly handles
applying security policies, like SELinux labels, and it avoids
spawning unnecessary processes (every ExecStartPre=/usr/bin/mkdir is a
separate process)

I think the current way of doing this (letting tmpfiles or the daemon
itself handle it) is for the best, for the reasons you mention. Also
because it means we only have one tool to create folders during early
boot.

Post by Michael Biebl
Zbyszek is arguing, that splitting the configuration over two files, a
tmpfile and a service file, complicates things for the administrator,
as things are no longer in a single place.

Hm, that is a fair point, but we also have the dual argument: if we
add a new place to configure runtime dirs without removing the old
(which I suppose we can't), the admin will anyway have to look in both
places to figure out what is going on...

Post by Michael Biebl
He also argues, that tmpfiles are active, independently of the
service, which needs them. Which can lead to unused runtime
directories.

I suppose the concern here is speed? If it can be shown that there is
a real speed advantage of not creating unneeded directories I guess
that's something worth discussing (I must admit that would surprise me
though).

-t

[Lennart Poettering]

Post by Michael Biebl
Hi,
an interesting issue was raised as part of reviewing a patch for
iodione [1], a system service which needs a runtime directory. We
thought this might need further dicussion, so reposting the issue to
For system services needing a runtime directory, we basically have two
(three) options nowadays
1/ use ExecStartPre=/usr/bin/mkdir /var/run/foo
2/ use a tmpfile snippet
(3/ let the daemon create the runtime directory itself)
In [1] we recommended the the usage of 2/ over 1/.
The main reason for that were, that systemd-tmpfiles properly handles
applying security policies, like SELinux labels, and it avoids
spawning unnecessary processes (every ExecStartPre=/usr/bin/mkdir is a
separate process)

I tend to agree with this.

Post by Michael Biebl
Zbyszek is arguing, that splitting the configuration over two files, a
tmpfile and a service file, complicates things for the administrator,
as things are no longer in a single place.
He also argues, that tmpfiles are active, independently of the
service, which needs them. Which can lead to unused runtime
directories.

I am not too concerned about unused runtime directories. After all this
is not something that would (or even could) grow without bounds. There
will never be more than O(n) runtime directores for n being the number
of packages you have installed (or had installed).

We will always need setting up of tmpfiles during boot, as many clients
expect certain directories to be existant in /run without any specific
daemon running. Thus, we couldn't move this stuff into service files ---
we could only allow both.

However, I am not convinced that allowing tmpfiles to be denoted in unit
files would be a good thing though, because it sounds unnecessary. It
really sounds as if the daemons which need that should rather create the
directories on their own before making use of them. This should
generally be the most robust solution. And in such a case there's no
need to denote anything about this in the unit files at all...

I mean, this is the general recommendation anyway: if possible just fix
the daemon to create the dirs as it needs it. Use tmpfiles only if
that's not feasible, maybe because the daemon never runs privileged code
or because the dirs need to exist at boot already.

Something I'd love to see though is if we could make it easier to apply
tmpfiles stuff automatically on package installation. More specifically,
I'd like an RPM macro to be added that handles this, and which is
sufficient for packagers to call for their tmpfiles. ALl in order to
ensure that the user can invoke the services right after package
installation without requiring a reboot.

Post by Michael Biebl
One suggestion that came up was, to specficy runtime directories in a
declarative fashion in the .service file itself, which means it would
be only be created if the service itself is enabled.

The way I see tmpfiles is that they don't actually do anything active,
they just reserve some space where something else then does something
active. It just sets up working areas, and doesn't actually play in
them. As such the impact, the cost of tmpfiles is minimal. It's like
installing a package and then not using it, i.e. you have stuff lying
around that is dead, but prepares everything for the time when you
actually want to run it, not more. If you want to get rid of the runtime
dirs (or the normal packaged files) you should just remove the package
in question...

I hope this makes some sense?

Lennart

[Colin Guthrie]

Post by Lennart Poettering
Something I'd love to see though is if we could make it easier to apply
tmpfiles stuff automatically on package installation. More specifically,
I'd like an RPM macro to be added that handles this, and which is
sufficient for packagers to call for their tmpfiles. ALl in order to
ensure that the user can invoke the services right after package
installation without requiring a reboot.

I've gone around the block a few times on this.

1. We used file triggers (which we've supported for a while with various
patches to rpm but which think are only just now coming to fedora via
official support in upstream rpm - I forget the details tho', so could
be wrong), but this was a bad idea as these triggers ran too late after
%post. As %post might need these folders to start/restart the daemon, I
had to abandon that.

2. I used to recommend just running "systemd-tmpfiles --create" but this
meant that the /run/nologin file was created which meant you couldn't
login... ugg. Don't do that in any macro please :D

3. Finally I settled on:

%_tmpfilesdir /usr/lib/tmpfiles.d
%_tmpfilescreate() /usr/bin/systemd-tmpfiles --create %{1}.conf \
%{nil}


In spec files I just put "%_tmpfilescreate %{name}" in %post in 99% of
cases and a custom basename for when the file name does not match the
package name in some rare cases.

I'm sure the macro could be tweaked to put %{name} in magically if no
argument is supplied..

HTHs (and if you have no preference as to the name, that you can use the
one I already use ;)

Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

[Lennart Poettering]

Post by Colin Guthrie
%_tmpfilesdir /usr/lib/tmpfiles.d
%_tmpfilescreate() /usr/bin/systemd-tmpfiles --create %{1}.conf \
%{nil}

This looks pretty close to what I'd like to see in place. Note that
%_tmpfilesdir is already set in macros.systemd as we ship it upstream.

My suggestion would be this however:

_tmpfiles_create() \
/usr/bin/systemd-tmpfiles --create %{1} >/dev/null 2>&1 || : \
%{nil}

I thik that people should specify file names rather than package names
(just to keep this closer to the underlying command and to make the
multiple drop-in case less exotic, and since we require the suffixe for
the unit macros too) and ignore errors. Also, add an underscore since
our other macros use the same naming style.

Post by Colin Guthrie
In spec files I just put "%_tmpfilescreate %{name}" in %post in 99% of
cases and a custom basename for when the file name does not match the
package name in some rare cases.
I'm sure the macro could be tweaked to put %{name} in magically if no
argument is supplied..

Hmm, I'd like such an automatism, but I'd really prefer if we could come
up with some scheme to automatically determine all tmpfiles snippets in
the package and apply them all automatically. But I am not sure how that
could be done with current RPM.

In fact, I'd actually like to do the same for the %systemd_post macros,
and suchlike, where people tend to be pretty bad at always listing all
unit files correctly.

Anyway, does that RPM macro sound good to you? If so, let's add this to
macros.systemd and I'll ask our Fedora guys to add this to the
guidelines.

Lennart

[Colin Guthrie]

Post by Lennart Poettering
Anyway, does that RPM macro sound good to you?

Sure, seems close enough :)

I can do a mass update to all our packages anyway so the slight change
in syntax isn't a problem.

Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

[Zbigniew Jędrzejewski-Szmek]

Post by Colin Guthrie

Post by Lennart Poettering
Anyway, does that RPM macro sound good to you?

Sure, seems close enough :)
I can do a mass update to all our packages anyway so the slight change
in syntax isn't a problem.

Hm, can we take a step back for a moment? It seems that the rpm macros
are a fairly complicated solution, and they also don't carry over into
debian or arch. User mode sessions also will not work with rpm macros.

I'm sure we can get them to work, but I'd like to explore the idea of
moving that into unit files first.

Post by Colin Guthrie
I am not too concerned about unused runtime directories. After all this
is not something that would (or even could) grow without bounds. There
will never be more than O(n) runtime directores for n being the number
of packages you have installed (or had installed).

Sure, but this n can be around a couple hundred I'd image when debian
is fully converted and installed. For efficiency this is probably
unmesurable, but as an admin I'd prefer not to have hundreds of empty
dirs in /run.

Post by Colin Guthrie
We will always need setting up of tmpfiles during boot, as many clients
expect certain directories to be existant in /run without any specific
daemon running. Thus, we couldn't move this stuff into service files ---
we could only allow both.

Yes, I'm not arguing for removing boot time tmpfiles.

Post by Colin Guthrie
However, I am not convinced that allowing tmpfiles to be denoted in unit
files would be a good thing though, because it sounds unnecessary. It
really sounds as if the daemons which need that should rather create the
directories on their own before making use of them. This should
generally be the most robust solution. And in such a case there's no
need to denote anything about this in the unit files at all...

Well, we seem to be using the tmpfiles mechanism quite a lot. And
there are good reasons for that: we get a lot of flexibility with a
powerful syntax, and actually there's significant downsides to putting
all this logic in daemons, e.g. support for various xattr
mechanisms. We want to simplify daemons. Also, as you mentioned, some
stuff runs unpriviledged.

Also, we could grow a mechanism to *remove* dirs after the daemon is
stopped. Putting it in rpm specific mechanism removes this possibility.

Post by Colin Guthrie
I mean, this is the general recommendation anyway: if possible just fix
the daemon to create the dirs as it needs it. Use tmpfiles only if
that's not feasible, maybe because the daemon never runs privileged code
or because the dirs need to exist at boot already.
Something I'd love to see though is if we could make it easier to apply
tmpfiles stuff automatically on package installation. More specifically,
I'd like an RPM macro to be added that handles this, and which is
sufficient for packagers to call for their tmpfiles. ALl in order to
ensure that the user can invoke the services right after package
installation without requiring a reboot.

Zbyszek

[Tom Gundersen]

On Tue, Jul 16, 2013 at 6:34 PM, Zbigniew Jędrzejewski-Szmek

Post by Zbigniew Jędrzejewski-Szmek
Hm, can we take a step back for a moment? It seems that the rpm macros
are a fairly complicated solution, and they also don't carry over into
debian or arch. User mode sessions also will not work with rpm macros.

For what it is worth we have been calling "systemd-tmpfiles --create
$pkgname.conf" in post-install for a long time in Arch (since the
beginning I think), and it doesn't seem to be a big problem. However,
if you have some idea for simplifying it somehow, I'd be all for that
of course :-)

Maybe establish an implicit link between foo.service and
tmpfiles.d/foo.conf, which would allow calling systemd-tmpfiles
--create/--clean before/after start/stop? Not sure if it is really
worth it, and not sure there won't be regressions though...

-t

[Lennart Poettering]

Post by Zbigniew Jędrzejewski-Szmek

Post by Colin Guthrie

Post by Lennart Poettering
Anyway, does that RPM macro sound good to you?

Sure, seems close enough :)
I can do a mass update to all our packages anyway so the slight change
in syntax isn't a problem.

Hm, can we take a step back for a moment? It seems that the rpm macros
are a fairly complicated solution, and they also don't carry over into
debian or arch. User mode sessions also will not work with rpm macros.

Well, it's easy to come up with similar macros for debhelper or arch...

User session don't really need this, as there as user daemons usually
create the dirs they need on their own and there are no privileges
involved which might forbid this.

I have commite the RPM macro fro now, since either way we will need it
for the packages where the dirs need to exist all the time.

Post by Zbigniew Jędrzejewski-Szmek

Post by Colin Guthrie
I am not too concerned about unused runtime directories. After all this
is not something that would (or even could) grow without bounds. There
will never be more than O(n) runtime directores for n being the number
of packages you have installed (or had installed).

Sure, but this n can be around a couple hundred I'd image when debian
is fully converted and installed. For efficiency this is probably
unmesurable, but as an admin I'd prefer not to have hundreds of empty
dirs in /run.

On my (pretty much fully converted) Fedora I currently have 20 tmpfiles
snippets around. I doubt on an everage Debian machine this would grow
much larger. May 40 or so, but that's still not much.

Post by Zbigniew Jędrzejewski-Szmek

Post by Colin Guthrie
However, I am not convinced that allowing tmpfiles to be denoted in unit
files would be a good thing though, because it sounds unnecessary. It
really sounds as if the daemons which need that should rather create the
directories on their own before making use of them. This should
generally be the most robust solution. And in such a case there's no
need to denote anything about this in the unit files at all...

Well, we seem to be using the tmpfiles mechanism quite a lot. And
there are good reasons for that: we get a lot of flexibility with a
powerful syntax, and actually there's significant downsides to putting
all this logic in daemons, e.g. support for various xattr
mechanisms. We want to simplify daemons. Also, as you mentioned, some
stuff runs unpriviledged.

I want daemons to be robust. So if possible I think daemons should do
these things on their own. This also has benefits for systems like selinux
where the label for files can be determined using the normal security
transitions, rather then databse checks like we currently need to do
it...

Post by Zbigniew Jędrzejewski-Szmek
Also, we could grow a mechanism to *remove* dirs after the daemon is
stopped. Putting it in rpm specific mechanism removes this possibility.

I can see how this could be nice but it kinda reminds me of the
situation regarding removing UIDs from /etc/passwd after package
deinstallation, which is soemthing that is simply very hard to ever get
rid, and which is why Fedora is not doing this at all...

Lennart

[Jóhann B. Guðmundsson]

Post by Lennart Poettering
On my (pretty much fully converted) Fedora I currently have 20 tmpfiles
snippets around. I doubt on an everage Debian machine this would grow
much larger. May 40 or so, but that's still not much.

Well we have only migrated what 400 components out of ca 550 Debian has
twice that much ( around 1100 - 1200 I think ) but yes that number seems
to fairly accurate which means it might actually be better pay off to
simple fix those 20 daemons that require the tmpfiles in the first place.

JBG

[Lennart Poettering]

Post by Lennart Poettering

Post by Zbigniew Jędrzejewski-Szmek

Post by Colin Guthrie
I can do a mass update to all our packages anyway so the slight change
in syntax isn't a problem.

Hm, can we take a step back for a moment? It seems that the rpm macros
are a fairly complicated solution, and they also don't carry over into
debian or arch. User mode sessions also will not work with rpm macros.

Well, it's easy to come up with similar macros for debhelper or arch...
User session don't really need this, as there as user daemons usually
create the dirs they need on their own and there are no privileges
involved which might forbid this.
I have commite the RPM macro fro now, since either way we will need it
for the packages where the dirs need to exist all the time.

I discussed this a bit more with Kay on the phone. Here's what we'd propose:

I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
specific new unit file settings:

RuntimeDirectory=
RuntimeDirectoyMode=

If RuntimeDirectory= is set we'd create it and chown() it to the UID/GID
set with User= and Group=. We'd apply the mode specified in
RuntimeDirectoryMode= to it.

We'd even allow multiple runtime directories to be specified.

I want this to be specific to the purpose of running daemons so I would
even go es far as generating a parse error if the specified directory is
not within /run. We wouldn't do any of the fancier things that tmpfiles
does, because we want people to use tmpfiles for that. For example, if
automatic clean-up needs to take place this information needs to be in
tmpfiles, not in a service unit.

We would hook this up with the same logic that currently handles
PrivateTmp= and remove the the runtime directory when the service goes
down.

With this in place we could probably cover 60% of the current users of
tmpfiles or so.

Putting this all together we'd recommend:

a) in the best case make your daemon create all runtime dirs as
necessary

b) if that's not possible (for example, due to rpivs), then use
RuntimeDirectory= instead.

c) If that's not enough (for example, because you need your dirs around
even if the daemon is not running) , then drop in a tmpfiles snippet
instead, and hook it in via the new RPM macro (or counterparts for
non-RPM distributions) so that it is executed at package
installation time

More specific to Fedora, there's still the question whether these dirs
in /run should be tracked by RPM or not. For this I think I am leaning towards:

d) List the runtime dirs you have in the RPM spec file, but %ghost
them. This is nice so that the directory can be easily traced back to
the package that created them.

Opinions? Suggestions?

Lennart

[Tomasz Torcz]

Post by Lennart Poettering

Post by Lennart Poettering

Post by Zbigniew Jędrzejewski-Szmek

Post by Colin Guthrie
I can do a mass update to all our packages anyway so the slight change
in syntax isn't a problem.

Hm, can we take a step back for a moment? It seems that the rpm macros
are a fairly complicated solution, and they also don't carry over into
debian or arch. User mode sessions also will not work with rpm macros.

Well, it's easy to come up with similar macros for debhelper or arch...
User session don't really need this, as there as user daemons usually
create the dirs they need on their own and there are no privileges
involved which might forbid this.
I have commite the RPM macro fro now, since either way we will need it
for the packages where the dirs need to exist all the time.

I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=
If RuntimeDirectory= is set we'd create it and chown() it to the UID/GID
set with User= and Group=. We'd apply the mode specified in
RuntimeDirectoryMode= to it.

There are daemons which do, in order:
1) start as root
2) open some root-only resources, like network sockets
3) change to less priviledge user
4) if all OK, write PID file or some state in /run
5) do the work

Those daemons can't have User= specified, because 2) would fail.
And because they must start as root, systemd can't know what
chown() to do on runtime dir.

So what's the solution?
a) declare those daemons incompatible with RuntimeDirectory* stuff
- nothing changes, they continue to use separate tmpfile .conf
b) fix the daemons?

[Kay Sievers]

Post by Tomasz Torcz

Post by Lennart Poettering
If RuntimeDirectory= is set we'd create it and chown() it to the UID/GID
set with User= and Group=. We'd apply the mode specified in
RuntimeDirectoryMode= to it.

1) start as root
2) open some root-only resources, like network sockets
3) change to less priviledge user
4) if all OK, write PID file or some state in /run
5) do the work
Those daemons can't have User= specified, because 2) would fail.
And because they must start as root, systemd can't know what
chown() to do on runtime dir.
So what's the solution?
a) declare those daemons incompatible with RuntimeDirectory* stuff
- nothing changes, they continue to use separate tmpfile .conf
b) fix the daemons?

B, sure, when it's private to the daemon and it run as root, it should
prepare its own environment itslef. It's just weird to assume someone
else does that for the daemon.

And it's just slightly less broken to use a tmpfiles to work around
such a broken daemon init. :)

Stuff is different though if the directory is some kind of an API,
then tmpfiles are sure fine, and creating them at boot is the right
thing.

Kay

[Jóhann B. Guðmundsson]

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=

What exactly are we trying to solve here which should not be fixed
upstream in the daemons themselves?

JBG

[Zbigniew Jędrzejewski-Szmek]

Post by Jóhann B. Guðmundsson

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=

What exactly are we trying to solve here which should not be fixed
upstream in the daemons themselves?

It was written in the part you removed:

Putting this all together we'd recommend:
a) in the best case make your daemon create all runtime dirs as
necessary
b) if that's not possible (for example, due to privs), then use
RuntimeDirectory= instead.
...

Zbyszek

[Kay Sievers]

On Tue, Jul 16, 2013 at 9:28 PM, "Jóhann B. Guðmundsson"

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=

What exactly are we trying to solve here which should not be fixed upstream
in the daemons themselves?

Stuff that uses User= cannot be fixed in the daemon itself, at the
time it runs, systemd itself has already taken away the privileges to
create these directories.

Kay

[Jóhann B. Guðmundsson]

Post by Kay Sievers
On Tue, Jul 16, 2013 at 9:28 PM, "Jóhann B. Guðmundsson"

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=

What exactly are we trying to solve here which should not be fixed upstream
in the daemons themselves?

Stuff that uses User= cannot be fixed in the daemon itself, at the
time it runs, systemd itself has already taken away the privileges to
create these directories.

We increase unit complexity with each option we add so I would suggest
unless it's absolutely necessary to just stick with just tmpfiles to
handle this.

JBG

[Tomasz Torcz]

Post by Jóhann B. Guðmundsson

Post by Kay Sievers
On Tue, Jul 16, 2013 at 9:28 PM, "Jóhann B. Guðmundsson"

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=

What exactly are we trying to solve here which should not be fixed upstream
in the daemons themselves?

Stuff that uses User= cannot be fixed in the daemon itself, at the
time it runs, systemd itself has already taken away the privileges to
create these directories.

We increase unit complexity with each option we add so I would
suggest unless it's absolutely necessary to just stick with just
tmpfiles to handle this.

I second Jóhann. As a packager, I never had problems with separate
configuration of runtime dir. Additionally, runtime dirs hardly need
customizations by admin. It's not worth exposing them in unit file.

[Harald Hoyer]

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=

What exactly are we trying to solve here which should not be fixed upstream in
the daemons themselves?

If RuntimeDirectory would support variable substitutions, this feature could not
be provided with systemd-tmpfiles.

[Michael Biebl]

Post by Harald Hoyer
If RuntimeDirectory would support variable substitutions, this feature could not
be provided with systemd-tmpfiles.

That would indeed be a nice feature.


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Andrey Borzenkov]

В Tue, 16 Jul 2013 20:26:23 +0200

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=
If RuntimeDirectory= is set we'd create it and chown() it to the UID/GID
set with User= and Group=. We'd apply the mode specified in
RuntimeDirectoryMode= to it.
We'd even allow multiple runtime directories to be specified.

What if different directories need different modes? I'm afraid it will
become hard to express in unit file.

[Lennart Poettering]

Post by Andrey Borzenkov

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=
If RuntimeDirectory= is set we'd create it and chown() it to the UID/GID
set with User= and Group=. We'd apply the mode specified in
RuntimeDirectoryMode= to it.
We'd even allow multiple runtime directories to be specified.

What if different directories need different modes? I'm afraid it will
become hard to express in unit file.

All complex cases would be handled by tmpfiles as before.

This is not supposed to be a 100% solution. It would be a > 70% solution
though, if I look at the tmpfiles snippets I have installed on my local
machine.

Lennart

[Michael Biebl]

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=
If RuntimeDirectory= is set we'd create it and chown() it to the UID/GID
set with User= and Group=. We'd apply the mode specified in
RuntimeDirectoryMode= to it.

What about daemons which drop privileges on their own? Shouldn't we
provide a directive to set the directory owner/group?
If this scheme is not flexible enough to cover the vast majority of
all cases (for services), then I fear we'd end up half of the services
using RuntimeDirectory, the other half a tmpfile. And that imho would
be even more confusing.

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Michael Biebl]

Post by Michael Biebl
If this scheme is not flexible enough to cover the vast majority of
all cases (for services), then I fear we'd end up half of the services
using RuntimeDirectory, the other half a tmpfile. And that imho would
be even more confusing.

Could we have some stats from Fedora and/or Arch which have done a
(complete) migration to systemd, about how many services/packages
currently use a tmpfile and what they use in the tmpfile, i.e. if the
proposed scheme from Lennart would suffice say for 90+ % of them

Michael
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Lennart Poettering]

Post by Michael Biebl

Post by Michael Biebl
If this scheme is not flexible enough to cover the vast majority of
all cases (for services), then I fear we'd end up half of the services
using RuntimeDirectory, the other half a tmpfile. And that imho would
be even more confusing.

Could we have some stats from Fedora and/or Arch which have done a
(complete) migration to systemd, about how many services/packages
currently use a tmpfile and what they use in the tmpfile, i.e. if the
proposed scheme from Lennart would suffice say for 90+ % of them

Fedora is not completely converted, but here are some stats.

The number of packages with tmpfiles:

$ repoquery --whatprovides '/usr/lib/tmpfiles.d/*' --qf "%{name}" | sort -u | wc -l
59

The full list is here: http://ur1.ca/eonvl

The number of packages with service files:

$ repoquery --whatprovides '/usr/lib/systemd/system/*' --qf "%{name}" | sort -u | wc -l
602

The full list is here: http://ur1.ca/eonxq

The number of holdouts with init scripts:

$ repoquery --whatprovides '/etc/rc.d/init.d/*' --qf "%{name}" | sort -u | wc -l
167

This is highly misleading however as it is permitted by the fedora
policy to continue sysv scripts if this is done in a separate package
from the main one. Let's manually sort out these cases:

$ repoquery --whatprovides '/etc/rc.d/init.d/*' --qf "%{name}" | sort -u | egrep -v '(-sysvinit|-initscript|-sysv)$' | wc -l
139

The full list is here: http://ur1.ca/eoo02

This means: ~81% of the packages have been converted from sysv to
systemd. And ~10% of the converted packages make use of tmpfiles.

Now, my rpm/yum-fu is a bit too limited to easily figure out what
precisely they use of the tmpfiles functionality.

This is Fedora 19.

Lennart

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jul 18, 2013 at 02:51:09AM +0200, Lennart Poettering wrote:
[snip repoquery magic]

Post by Lennart Poettering
This means: ~81% of the packages have been converted from sysv to
systemd. And ~10% of the converted packages make use of tmpfiles.
Now, my rpm/yum-fu is a bit too limited to easily figure out what
precisely they use of the tmpfiles functionality.

So, as the person who originally porposed adding tmpfiles to units,
I'm now more or less convinced that it's too late and not worth it.
Since the tmpfiles snippets have been written, and a big chunk is
done, it would be too confusing to add duplicate functionality.
People pointed out problems with the limited
RuntimeDirectory=/RuntimeDirectoryMode= proposal, and Lennart and
Kay are against the original idea to add tmpfiles snippets directly.
The unproven benefits are not worth it, and we certainly have other
things waiting on the table. So let's close this thread.

Zbyszek

[Kay Sievers]

On Thu, Jul 18, 2013 at 4:50 AM, Zbigniew Jędrzejewski-Szmek

Post by Zbigniew Jędrzejewski-Szmek
[snip repoquery magic]

Post by Lennart Poettering
This means: ~81% of the packages have been converted from sysv to
systemd. And ~10% of the converted packages make use of tmpfiles.
Now, my rpm/yum-fu is a bit too limited to easily figure out what
precisely they use of the tmpfiles functionality.

So, as the person who originally porposed adding tmpfiles to units,
I'm now more or less convinced that it's too late and not worth it.
Since the tmpfiles snippets have been written, and a big chunk is
done, it would be too confusing to add duplicate functionality.
People pointed out problems with the limited
RuntimeDirectory=/RuntimeDirectoryMode= proposal, and Lennart and
Kay are against the original idea to add tmpfiles snippets directly.
The unproven benefits are not worth it, and we certainly have other
things waiting on the table. So let's close this thread.

Sounds good to me.

Thanks,
Kay

[Jóhann B. Guðmundsson]

Post by Lennart Poettering
Fedora is not completely converted, but here are some stats.
$ repoquery --whatprovides '/etc/rc.d/init.d/*' --qf "%{name}" | sort -u | egrep -v '(-sysvinit|-initscript|-sysv)$' | wc -l
139

Well even that number is not accurate since we have several migrated
components being stuck in bugzilla and not being packaged and shipped
which is the biggest problem we are faced with in the migration process
( and have been faced with from the start ) as in not the actual
migration but to get maintainers of those components, package and ship
the units ( and quite few of those remaining ones seem to unmaintained ).

If everything goes as planned we should finish migrating those this
release cycle ( F20 ).

JBG

[Lennart Poettering]

Post by Michael Biebl

Post by Lennart Poettering
I'd be very conservative regarding adding full tmpfiles support into
unit files directly. Instead, I'd suggest adding two very minimal, very
RuntimeDirectory=
RuntimeDirectoyMode=
If RuntimeDirectory= is set we'd create it and chown() it to the UID/GID
set with User= and Group=. We'd apply the mode specified in
RuntimeDirectoryMode= to it.

What about daemons which drop privileges on their own? Shouldn't we
provide a directive to set the directory owner/group?

If they drop privs on their own they *really* should also just create
the runtime dirs along with it, after all they are privileged first. And
most already do actually.

Lennart

[Michael Biebl]

Post by Zbigniew Jędrzejewski-Szmek

Post by Lennart Poettering
I am not too concerned about unused runtime directories. After all this
is not something that would (or even could) grow without bounds. There
will never be more than O(n) runtime directores for n being the number
of packages you have installed (or had installed).

Sure, but this n can be around a couple hundred I'd image when debian
is fully converted and installed. For efficiency this is probably
unmesurable, but as an admin I'd prefer not to have hundreds of empty
dirs in /run.

That seems to be misconception: Even when all 1200+ packages currently
shipping SysV init scripts, have native systemd support, this doesn't
mean that all those 1200 packages are actually installed.
I think it's even less of an issue for Debian, as we usually only
install stuff when needed and not in a disabled state.

I expect that a typical desktop installation has around 30 system
services, and I'm pretty sure it's rather uncommon to have more then 5
disabled system services.



--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Michael Biebl]

Post by Michael Biebl
I think it's even less of an issue for Debian, as we usually only
install stuff when needed and not in a disabled state.

So the unused directories in /run are something I don't expect to be
an issue for Debian.

As for tmpfiles support:
In Debian we already have pretty nice support and debhelper
(dh_installinit) scans the installed files and creates postinst
snippet like

if [ -d /run/systemd/system ] ; then
systemd-tmpfiles --create /usr/lib/tmpfiles.d/xconsole.conf

Post by Michael Biebl
/dev/null || true

fi

which is run before the daemon is started.

So, also not an issue for Debian.

That leaves the only point: the configuration is split between two
files. Tbh, I'm not too thrilled duplicating part of the tmpfiles
functionality in .service files itself.
What we could do though, similar to Documentation, reference a tmpfile
which belongs to the service.
So when you run systemctl status foo.service, it would show the
corresponding file.

This would it more discoverable for the admin and he'd see which file to modify.

Michael


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Lennart Poettering]

Post by Colin Guthrie

Post by Lennart Poettering
Anyway, does that RPM macro sound good to you?

Sure, seems close enough :)
I can do a mass update to all our packages anyway so the slight change
in syntax isn't a problem.

As mentioned, I commited this now, since we need this in any way. Could
you check if this works for you, please?

Lennart

[Kay Sievers]

On Tue, Jul 16, 2013 at 6:01 PM, Lennart Poettering

Post by Lennart Poettering
Hmm, I'd like such an automatism, but I'd really prefer if we could come
up with some scheme to automatically determine all tmpfiles snippets in
the package and apply them all automatically. But I am not sure how that
could be done with current RPM.
In fact, I'd actually like to do the same for the %systemd_post macros,
and suchlike, where people tend to be pretty bad at always listing all
unit files correctly.

Hmm, an rpmlint check for stuff like this maybe is the first step? At
least people who care and look at that would fix their stuff?

Kay

[Lennart Poettering]

Post by Kay Sievers
On Tue, Jul 16, 2013 at 6:01 PM, Lennart Poettering

Post by Lennart Poettering
Hmm, I'd like such an automatism, but I'd really prefer if we could come
up with some scheme to automatically determine all tmpfiles snippets in
the package and apply them all automatically. But I am not sure how that
could be done with current RPM.
In fact, I'd actually like to do the same for the %systemd_post macros,
and suchlike, where people tend to be pretty bad at always listing all
unit files correctly.

Hmm, an rpmlint check for stuff like this maybe is the first step? At
least people who care and look at that would fix their stuff?

Makes sense. Added to the TODO list for now.

Lennart

[Colin Guthrie]

Post by Kay Sievers
On Tue, Jul 16, 2013 at 6:01 PM, Lennart Poettering

Post by Lennart Poettering
Hmm, I'd like such an automatism, but I'd really prefer if we could come
up with some scheme to automatically determine all tmpfiles snippets in
the package and apply them all automatically. But I am not sure how that
could be done with current RPM.
In fact, I'd actually like to do the same for the %systemd_post macros,
and suchlike, where people tend to be pretty bad at always listing all
unit files correctly.

Actually rereading this bit above, I can see the desire for automated
snippets, but are there not some cases where you would want to avoid
calling %systemd_post macros for some shipped units? e.g. they may only
be used internally by others (Requires=, Also= and similar). I think
such automation might not be good. (just look at the NFS units which I
think we've finally cleaned up a bit in Mageia - enough to actually work
at least).

Post by Kay Sievers
Hmm, an rpmlint check for stuff like this maybe is the first step? At
least people who care and look at that would fix their stuff?

Misc was looking at doing most of that. Not sure what you mean about
naming tho'... do you just mean the folder name (etc vs usr) or just not
picking very nice unit names?

For the folder, I put rpmlint checks in Mageia a while ago to ban units,
tmpfiles and udev rules from shipping in /etc.

And another to do with non-ghost files in /run or /var/run.

I had it in my head that I'd spoken to Misc about this at the time I
wrote them but now I have a paranoid fear that maybe I didn't... They
are simple enough anyway, so if they are not upstream and have instead
been reimplemented then it's no great loss, but apologies if forgot to
ping you at the time Misc (and I've seen you twice in person since then
too!!)

Patches here. Can't check if it's upstream yet as rpmlint.zarb.org is
down for now...

http://svnweb.mageia.org/packages/cauldron/rpmlint/current/SOURCES/

Col
--
Colin Guthrie
colin(at)mageia.org
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

[Lennart Poettering]

Post by Colin Guthrie

Post by Kay Sievers
On Tue, Jul 16, 2013 at 6:01 PM, Lennart Poettering

Post by Lennart Poettering
Hmm, I'd like such an automatism, but I'd really prefer if we could come
up with some scheme to automatically determine all tmpfiles snippets in
the package and apply them all automatically. But I am not sure how that
could be done with current RPM.
In fact, I'd actually like to do the same for the %systemd_post macros,
and suchlike, where people tend to be pretty bad at always listing all
unit files correctly.

Actually rereading this bit above, I can see the desire for automated
snippets, but are there not some cases where you would want to avoid
calling %systemd_post macros for some shipped units? e.g. they may only
be used internally by others (Requires=, Also= and similar). I think
such automation might not be good. (just look at the NFS units which I
think we've finally cleaned up a bit in Mageia - enough to actually work
at least).

No. You really want the macros for all units, always.

The macros use the preset information to enable units if that's
what the vendor or admin wants. This preset information should be what
decides what is enabled and what is not, not the packager who accidently
forgot to invoke "systemctl preset" on one of his units.

The unit names also matter to shutdown running units before uninstalling
them. For that too you really need the list of all units the package
installs and not just the ones the package might have though about
listing.

So yeah, we want that the scritps process *all* units the package
installs, not just some subset. Only that guarantees correct behaviour
regarding presets and regarding stopping things when we uninstall.

Of coruse, this is not obvious to people, which is why many packages
don't get this right. With the automatic logic we'd avoid this confusion.

Post by Colin Guthrie

Post by Kay Sievers
Hmm, an rpmlint check for stuff like this maybe is the first step? At
least people who care and look at that would fix their stuff?

Misc was looking at doing most of that. Not sure what you mean about
naming tho'... do you just mean the folder name (etc vs usr) or just not
picking very nice unit names?

The rpmlint should just check that all units a package ships are listed
as arguments for the RPM macro.

Post by Colin Guthrie
For the folder, I put rpmlint checks in Mageia a while ago to ban units,
tmpfiles and udev rules from shipping in /etc.

That's a very good choice.

Post by Colin Guthrie
And another to do with non-ghost files in /run or /var/run.

Also a good choice!

Lennart

[Michael Scherer]

Post by Colin Guthrie

Post by Kay Sievers
On Tue, Jul 16, 2013 at 6:01 PM, Lennart Poettering

Post by Lennart Poettering
Hmm, I'd like such an automatism, but I'd really prefer if we could come
up with some scheme to automatically determine all tmpfiles snippets in
the package and apply them all automatically. But I am not sure how that
could be done with current RPM.
In fact, I'd actually like to do the same for the %systemd_post macros,
and suchlike, where people tend to be pretty bad at always listing all
unit files correctly.

Actually rereading this bit above, I can see the desire for automated
snippets, but are there not some cases where you would want to avoid
calling %systemd_post macros for some shipped units? e.g. they may only
be used internally by others (Requires=, Also= and similar). I think
such automation might not be good. (just look at the NFS units which I
think we've finally cleaned up a bit in Mageia - enough to actually work
at least).

Post by Kay Sievers
Hmm, an rpmlint check for stuff like this maybe is the first step? At
least people who care and look at that would fix their stuff?

Misc was looking at doing most of that. Not sure what you mean about
naming tho'... do you just mean the folder name (etc vs usr) or just not
picking very nice unit names?
For the folder, I put rpmlint checks in Mageia a while ago to ban units,
tmpfiles and udev rules from shipping in /etc.
And another to do with non-ghost files in /run or /var/run.
I had it in my head that I'd spoken to Misc about this at the time I
wrote them but now I have a paranoid fear that maybe I didn't... They
are simple enough anyway, so if they are not upstream and have instead
been reimplemented then it's no great loss, but apologies if forgot to
ping you at the time Misc (and I've seen you twice in person since then
too!!)
Patches here. Can't check if it's upstream yet as rpmlint.zarb.org is
down for now...

We are now on sf.net.

And someone already contacted me to get the patches, I said i was ok on
the principle, but from the code point of view, it could have been
refactored ( but I didn't do it yet );

IE having a structure like :
['regexp','name-of-exception','message'], have it exposed in the
configuration and have a generic module using this, so every
distribution could add them.

IIRC, Suse do have this kind of patchs, but I didn't merged them yet.