[systemd-devel] Persistent timers delay Type=idle units

[Thomas Bächler]

According to [1], when a persistent timer runs its service on boot, it
delays startup. That itself isn't a problem, but it delays Type=idle
units, in particular, ***@tty1.service, prohibiting login until it
finishes.

I hooked up OnCalendar=daily/Persistent=true timers to Type=oneshot
units that may take a few seconds to run. Obviously, I'd rather have the
getty service not wait for those on boot.

I can't think of a proper solution now, any ideas?

[1]
https://mailman.archlinux.org/pipermail/arch-general/2014-April/035935.html

[Lennart Poettering]

Post by Thomas Bächler
According to [1], when a persistent timer runs its service on boot, it
delays startup.

Humm? What precisely do you mean by "delays bootup"? Just scheduling a
timer unit should have about zero effect on boot times... If it does
this would be a bug.

What does "systemctl list-jobs" print when this happens? (i.e. when the
bootup is supposedly delayed?)

Lennart

[Thomas Bächler]

Post by Lennart Poettering

Post by Thomas Bächler
According to [1], when a persistent timer runs its service on boot, it
delays startup.

Humm? What precisely do you mean by "delays bootup"? Just scheduling a
timer unit should have about zero effect on boot times... If it does
this would be a bug.

From the description that I linked in the first post, it seems that the
unit it started as soon as the timer is started. I don't know if "delay
bootup" is the right expression, but the Type=idle units, like getty,
don't start until the timer's unit finished starting up.

Post by Lennart Poettering
What does "systemctl list-jobs" print when this happens? (i.e. when the
bootup is supposedly delayed?)

I'd have to build a test-case for this, since I have not seen any
negative effect myself yet (I can't do this until later this week). I'm
CC'ing the original reporter, maybe he can give more information.

[Tom Gundersen]

Post by Thomas Bächler
I'm
CC'ing the original reporter, maybe he can give more information.

I think you forgot to do that...

-t

[Thomas Bächler]

Post by Tom Gundersen

Post by Thomas Bächler
I'm
CC'ing the original reporter, maybe he can give more information.

I think you forgot to do that...

Strange stuff - he is listed in CC in the mail I received, but his
address is missing on-list.

[Leonid Isaev]

Hi,

I'll try to answer all questions at once...

On Tue, 22 Apr 2014 09:37:29 +0200

Post by Lennart Poettering
Humm? What precisely do you mean by "delays bootup"? Just scheduling a
timer unit should have about zero effect on boot times... If it does
this would be a bug.

This is right for the "usual", e.g. OnBootSec= timers.

Just to give a little background, archlinux has recently replaced daily cron
jobs (like indexing of man and locate databases) with OnCalendar= timers:
https://mailman.archlinux.org/pipermail/arch-dev-public/2014-March/026044.html .

[Lennart Poettering]

Post by Leonid Isaev

Post by Lennart Poettering
Humm? What precisely do you mean by "delays bootup"? Just scheduling a
timer unit should have about zero effect on boot times... If it does
this would be a bug.

This is right for the "usual", e.g. OnBootSec= timers.
Just to give a little background, archlinux has recently replaced daily cron
https://mailman.archlinux.org/pipermail/arch-dev-public/2014-March/026044.html .

Awesome! This sounds great!

[Andrey Borzenkov]

В Wed, 23 Apr 2014 05:57:39 +0200

You have services that are to be started by timers that take a long time
to complete. THe timers have been configured to be persistent. If the
system comes up and the timestamp files suggest that the timers need to
be triggered immediately this is done, adding the service execution time
to the bootup time. This is normally not a problem except when there's
some other bootup service that uses Type=idle which will then be
affected by these long running services...
Did I get this right?
Hmm, this sounds nasty. I wodner what we can do about it...

Provide "boot completed" indication?

systemd already provides "starting" and "running" states. Which
logically implies that bootup is finished when "starting" is
replaced by "running".

Add new timer condition which fires at this point? This will stop
misusing Idle for poor man's replacement of proper "boot completed"
event.

[Lennart Poettering]

Post by Andrey Borzenkov
В Wed, 23 Apr 2014 05:57:39 +0200

You have services that are to be started by timers that take a long time
to complete. THe timers have been configured to be persistent. If the
system comes up and the timestamp files suggest that the timers need to
be triggered immediately this is done, adding the service execution time
to the bootup time. This is normally not a problem except when there's
some other bootup service that uses Type=idle which will then be
affected by these long running services...
Did I get this right?
Hmm, this sounds nasty. I wodner what we can do about it...

Provide "boot completed" indication?
systemd already provides "starting" and "running" states. Which
logically implies that bootup is finished when "starting" is
replaced by "running".

Well, it works the other way round. "running" is entered as soon as
there are no jobs anymore...

Lennart

[Leonid Isaev]

Hi,

On Wed, 23 Apr 2014 05:57:39 +0200

[...]
You have services that are to be started by timers that take a long time
to complete. THe timers have been configured to be persistent. If the
system comes up and the timestamp files suggest that the timers need to
be triggered immediately this is done, adding the service execution time
to the bootup time. This is normally not a problem except when there's
some other bootup service that uses Type=idle which will then be
affected by these long running services...
Did I get this right?

Yes. Of course, the meaning of "long" really depends... but it can be ~10 sec.

Hmm, this sounds nasty. I wodner what we can do about it...
Maybe we should add a new setting PersistentExtraSec= to timer units or
so which allows delaying these kind of timers by an extra margin. Would
this work for you?

Yes, I think so. Actually, that's what Thomas proposed on arch-general...

Post by Lennart Poettering
What does "systemctl list-jobs" print when this happens? (i.e. when the
bootup is supposedly delayed?)

I'll have to test this, but I'll speculate that list-jobs will show nothing
by the time I login, because it takes about 30 sec for me to enter login
credentials...

Use "systemctl enable debug-shell"...

OK will do. I didn't know about the debug shell.

Thanks,
L.

[Lennart Poettering]

Post by Leonid Isaev

Hmm, this sounds nasty. I wodner what we can do about it...
Maybe we should add a new setting PersistentExtraSec= to timer units or
so which allows delaying these kind of timers by an extra margin. Would
this work for you?

Yes, I think so. Actually, that's what Thomas proposed on
arch-general...

Hmm, thinking about this: the problem is actually not restricted to
persistent timers, any timer that has OnBootSec=10ms or so is also
affected by this, should the boot take longer than 10ms...

Another option might be to change what Type=idle means: instead of
making it wait until the job queue is empty it might be better to simply
make it wait until the default job is finished (with other words, until
graphical.target is reached). That way it doesn't matter what services
are added in by timers... Somehow this appears like the better solution
to me. This probably also means changing the manager state machine, and
splitting its "starting" state into two: one state for the time until
the default target is not yet reached, and a second state for the time
until the job queue is actually empty.

I added this to the TODO list now. I'd be happy to take a patch for this
though!

Lennart

[Andrey Borzenkov]

В Wed, 23 Apr 2014 20:30:35 +0200

Post by Lennart Poettering

Post by Leonid Isaev

Hmm, this sounds nasty. I wodner what we can do about it...
Maybe we should add a new setting PersistentExtraSec= to timer units or
so which allows delaying these kind of timers by an extra margin. Would
this work for you?

Yes, I think so. Actually, that's what Thomas proposed on
arch-general...

Hmm, thinking about this: the problem is actually not restricted to
persistent timers, any timer that has OnBootSec=10ms or so is also
affected by this, should the boot take longer than 10ms...
Another option might be to change what Type=idle means: instead of
making it wait until the job queue is empty it might be better to simply
make it wait until the default job is finished (with other words, until
graphical.target is reached).

I think it was discussed in the past and it was exact reason Idle was
introduced - because default.target is not accurate representation of
finished startup sequence.

Is it technically possible to track jobs that resulted from
dependency closure of default.target? I.e. all units pulled in
(directly or indirectly) by default target? This would probably be more
accurate approximation of "at the end of startup" and automatically
fix problem discussed here. It would also provide more or less useful
"startup finished" synchronization point.

Post by Lennart Poettering
That way it doesn't matter what services
are added in by timers... Somehow this appears like the better solution
to me. This probably also means changing the manager state machine, and
splitting its "starting" state into two: one state for the time until
the default target is not yet reached, and a second state for the time
until the job queue is actually empty.
I added this to the TODO list now. I'd be happy to take a patch for this
though!
Lennart