Discussion:
[systemd-devel] exim4 only queues mails sent by systemd service
Kamil Jońca
2018-09-23 20:48:51 UTC
Permalink
It is something strange with sending mails from systemd system service:
assume we have service file /etc/systemd/system/mailtest.service:

--8<---------------cut here---------------start------------->8---
[Unit]
Description="Test maili"
[Service]
#User=kjonca
NoNewPrivileges=false
Type=oneshot
ExecStart=-zsh -c 'echo xxx|mail news'
ExecStart=-zsh -c 'echo xxx|mutt -F /dev/null -s subject -e \'set copy=no\' kjonca'
--8<---------------cut here---------------end--------------->8---

When I call
sudo systemctl start mailtest.service
Two messages are put in exim queue, but not deliveried immediately.
Why? What am I missing?
Morevoer this service run as "--user" service works as expected - mails
are delivered at once.
Does systemd interfere with processes (suid/sgid? file access limitations?)
KJ
--
http://wolnelektury.pl/wesprzyj/teraz/
Hold the MAYO & pass the COSMIC AWARENESS ...
Lennart Poettering
2018-09-24 08:32:30 UTC
Permalink
Post by Kamil Jońca
--8<---------------cut here---------------start------------->8---
[Unit]
Description="Test maili"
[Service]
#User=kjonca
NoNewPrivileges=false
Type=oneshot
ExecStart=-zsh -c 'echo xxx|mail news'
ExecStart=-zsh -c 'echo xxx|mutt -F /dev/null -s subject -e \'set copy=no\' kjonca'
--8<---------------cut here---------------end--------------->8---
When I call
sudo systemctl start mailtest.service
Two messages are put in exim queue, but not deliveried immediately.
Why? What am I missing?
Uh, this looks like something you need to ask the exim community,
systemd can't make exim mail queueing decisions, that's entirely
internal to exim.

One question though: are you sure you have started the exim service
properly beforehand? I am pretty sure exim won't process the mail
queue if it's not running...

Lennart
--
Lennart Poettering, Red Hat
Kamil Jońca
2018-09-24 08:47:12 UTC
Permalink
Post by Lennart Poettering
Post by Kamil Jońca
--8<---------------cut here---------------start------------->8---
[Unit]
Description="Test maili"
[Service]
#User=kjonca
NoNewPrivileges=false
Type=oneshot
ExecStart=-zsh -c 'echo xxx|mail news'
ExecStart=-zsh -c 'echo xxx|mutt -F /dev/null -s subject -e \'set copy=no\' kjonca'
--8<---------------cut here---------------end--------------->8---
When I call
sudo systemctl start mailtest.service
Two messages are put in exim queue, but not deliveried immediately.
Why? What am I missing?
Uh, this looks like something you need to ask the exim community,
systemd can't make exim mail queueing decisions, that's entirely
internal to exim.
Another puzzle, when I put "strace" in front of mutt - mail is delivered
immediately.
So it I suspect something with SUID/SGID binaries called from systemd,
but I do not know how to check it :(
Post by Lennart Poettering
One question though: are you sure you have started the exim service
properly beforehand? I am pretty sure exim won't process the mail
queue if it's not running...
Everything works as expected:
- mails from cron
- mails from fetchmail
- mails from "--user" services
KJ
--
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
Imagination is more important than knowledge.
-- Albert Einstein
Mantas Mikulėnas
2018-09-24 09:04:27 UTC
Permalink
Post by Kamil Jońca
Post by Kamil Jońca
--8<---------------cut here---------------start------------->8---
[Unit]
Description="Test maili"
[Service]
#User=kjonca
NoNewPrivileges=false
Type=oneshot
ExecStart=-zsh -c 'echo xxx|mail news'
ExecStart=-zsh -c 'echo xxx|mutt -F /dev/null -s subject -e \'set
copy=no\' kjonca'
Post by Kamil Jońca
--8<---------------cut here---------------end--------------->8---
When I call
sudo systemctl start mailtest.service
Two messages are put in exim queue, but not deliveried immediately.
Why? What am I missing?
Uh, this looks like something you need to ask the exim community,
systemd can't make exim mail queueing decisions, that's entirely
internal to exim.
One question though: are you sure you have started the exim service
properly beforehand? I am pretty sure exim won't process the mail
queue if it's not running...
exim's a bit oldschool, and whenever you pipe a message to 'sendmail', it
immediately forks a worker to deliver the message synchronously, regardless
of the main daemon running.

(I think this behavior is slightly ugly. It can be disabled by setting
'queue_only=yes', but... it's probably necessary because the main daemon
only processes its queue every x minutes and won't notice queued messages
for a while.)
--
Mantas Mikulėnas
Lennart Poettering
2018-09-24 13:18:15 UTC
Permalink
Post by Mantas Mikulėnas
Post by Lennart Poettering
Uh, this looks like something you need to ask the exim community,
systemd can't make exim mail queueing decisions, that's entirely
internal to exim.
One question though: are you sure you have started the exim service
properly beforehand? I am pretty sure exim won't process the mail
queue if it's not running...
exim's a bit oldschool, and whenever you pipe a message to 'sendmail', it
immediately forks a worker to deliver the message synchronously, regardless
of the main daemon running.
Uh, what? Are you saying exim is forking off privileged daemon code
from unprivileged user command invocations? Christ, that's ugly. They
really really shouldn't do that.
Post by Mantas Mikulėnas
(I think this behavior is slightly ugly. It can be disabled by setting
'queue_only=yes', but... it's probably necessary because the main daemon
only processes its queue every x minutes and won't notice queued messages
for a while.)
If I got the above right then this isn't just "slightly" ugly, it's
really awful. Daemon code should not be run from unprivileged user
contexts, not in today's world.

It appears to me exim should figure out some way how clients such as
'sendmail' invocations can trigger queue dispatching some other way,
for example, by making exim listen on some IPC of some form, or using
inotify or anything else.

Lennart
--
Lennart Poettering, Red Hat
Kamil Jońca
2018-09-24 13:29:09 UTC
Permalink
Post by Lennart Poettering
Post by Mantas Mikulėnas
Post by Lennart Poettering
Uh, this looks like something you need to ask the exim community,
systemd can't make exim mail queueing decisions, that's entirely
internal to exim.
One question though: are you sure you have started the exim service
properly beforehand? I am pretty sure exim won't process the mail
queue if it's not running...
exim's a bit oldschool, and whenever you pipe a message to 'sendmail', it
immediately forks a worker to deliver the message synchronously, regardless
of the main daemon running.
Uh, what? Are you saying exim is forking off privileged daemon code
from unprivileged user command invocations? Christ, that's ugly. They
Yes. exim is suid root to deliver mails.
Post by Lennart Poettering
really really shouldn't do that.
But they do.
Post by Lennart Poettering
It appears to me exim should figure out some way how clients such as
'sendmail' invocations can trigger queue dispatching some other way,
for example, by making exim listen on some IPC of some form, or using
inotify or anything else.
IIRC postfix is written that way, but I want to use exim, as it is more
configurable.
KJ
--
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
Samsung Wave to badziewie ...
Mantas Mikulėnas
2018-09-24 16:51:39 UTC
Permalink
Post by Mantas Mikulėnas
Post by Lennart Poettering
Post by Mantas Mikulėnas
Post by Lennart Poettering
Uh, this looks like something you need to ask the exim community,
systemd can't make exim mail queueing decisions, that's entirely
internal to exim.
One question though: are you sure you have started the exim service
properly beforehand? I am pretty sure exim won't process the mail
queue if it's not running...
exim's a bit oldschool, and whenever you pipe a message to 'sendmail',
it
Post by Lennart Poettering
Post by Mantas Mikulėnas
immediately forks a worker to deliver the message synchronously,
regardless
Post by Lennart Poettering
Post by Mantas Mikulėnas
of the main daemon running.
Uh, what? Are you saying exim is forking off privileged daemon code
from unprivileged user command invocations? Christ, that's ugly. They
Yes. exim is suid root to deliver mails.
Post by Lennart Poettering
really really shouldn't do that.
But they do.
Post by Lennart Poettering
It appears to me exim should figure out some way how clients such as
'sendmail' invocations can trigger queue dispatching some other way,
for example, by making exim listen on some IPC of some form, or using
inotify or anything else.
IIRC postfix is written that way, but I want to use exim, as it is more
configurable.
KJ
When I was writing exim systemd units for Arch a few years ago, I
experimented with using queue_only=true and no permanent daemon at all, but
triggering the queue runner via systemd.path units (start as soon as spool
is non-empty) and timers (to replace the usual -q15m).

.path units are inotify-based and can start exim as soon as
/usr/bin/sendmail puts something in the queue.

This didn't work well enough IIRC, but if it did, then it'd provide almost
postfix-like architecture.
--
Mantas Mikulėnas
Mantas Mikulėnas
2018-09-24 17:06:38 UTC
Permalink
Post by Mantas Mikulėnas
Post by Mantas Mikulėnas
Post by Lennart Poettering
Post by Mantas Mikulėnas
Post by Lennart Poettering
Uh, this looks like something you need to ask the exim community,
systemd can't make exim mail queueing decisions, that's entirely
internal to exim.
One question though: are you sure you have started the exim service
properly beforehand? I am pretty sure exim won't process the mail
queue if it's not running...
exim's a bit oldschool, and whenever you pipe a message to 'sendmail',
it
Post by Lennart Poettering
Post by Mantas Mikulėnas
immediately forks a worker to deliver the message synchronously,
regardless
Post by Lennart Poettering
Post by Mantas Mikulėnas
of the main daemon running.
Uh, what? Are you saying exim is forking off privileged daemon code
from unprivileged user command invocations? Christ, that's ugly. They
Yes. exim is suid root to deliver mails.
Post by Lennart Poettering
really really shouldn't do that.
But they do.
Post by Lennart Poettering
It appears to me exim should figure out some way how clients such as
'sendmail' invocations can trigger queue dispatching some other way,
for example, by making exim listen on some IPC of some form, or using
inotify or anything else.
IIRC postfix is written that way, but I want to use exim, as it is more
configurable.
KJ
When I was writing exim systemd units for Arch a few years ago, I
experimented with using queue_only=true and no permanent daemon at all, but
triggering the queue runner via systemd.path units (start as soon as spool
is non-empty) and timers (to replace the usual -q15m).
.path units are inotify-based and can start exim as soon as
/usr/bin/sendmail puts something in the queue.
This didn't work well enough IIRC, but if it did, then it'd provide almost
postfix-like architecture.
Or just making 'sendmail' send a SIGALRM to the main daemon would do the
job perfectly well, I suspect...
--
Mantas Mikulėnas
Kamil Jońca
2018-09-24 18:34:37 UTC
Permalink
Mantas Mikulėnas <***@gmail.com> writes:

[...]
This didn't work well enough IIRC, but if it did, then it'd provide almost postfix-like architecture.
Or just making 'sendmail' send a SIGALRM to the main daemon would do the job perfectly well, I suspect...
--
Mantas Mikulėnas
But I still does not know, where is the problem, why exim doest not play
well with systemd ...
KJ
--
http://wolnelektury.pl/wesprzyj/teraz/
If you love someone, set them free.
If they don't come back, then call them up when you're drunk.
Lennart Poettering
2018-09-24 18:38:30 UTC
Permalink
Post by Kamil Jońca
This didn't work well enough IIRC, but if it did, then it'd provide almost postfix-like architecture.
Or just making 'sendmail' send a SIGALRM to the main daemon would do the job perfectly well, I suspect...
But I still does not know, where is the problem, why exim doest not play
well with systemd ...
Here's an educated guess: your script terminates, so that that systemd
decides your service has ended. In such a case it kills any left-over
processes of the service, and this will include the exim process
forked off into the bg, because it is attributed to your script's
context.

Thus the exim queue runner might be forked off, but it wil be
immediately terminated by systemd again...

It's one of the reasons why exim shouldn't do what it is doing
there. Running priv daemon code as forked bg children from differently
privileged user code is just a massively bad idea.

Lennart
--
Lennart Poettering, Red Hat
Kamil Jońca
2018-09-24 19:13:19 UTC
Permalink
Post by Lennart Poettering
Post by Kamil Jońca
This didn't work well enough IIRC, but if it did, then it'd provide almost postfix-like architecture.
Or just making 'sendmail' send a SIGALRM to the main daemon would do the job perfectly well, I suspect...
But I still does not know, where is the problem, why exim doest not play
well with systemd ...
Here's an educated guess: your script terminates, so that that systemd
decides your service has ended. In such a case it kills any left-over
processes of the service, and this will include the exim process
forked off into the bg, because it is attributed to your script's
context.
I made some tests (ie.add sleep at end of exec line),
and this confirms your explanation.
But ... why this is working as --user service? pure luck?

KJ
--
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
Beware the new TTY code!
Mantas Mikulėnas
2018-09-24 19:14:42 UTC
Permalink
Post by Lennart Poettering
Post by Kamil Jońca
Post by Mantas Mikulėnas
This didn't work well enough IIRC, but if it did, then it'd
provide almost postfix-like architecture.
Post by Lennart Poettering
Post by Kamil Jońca
Post by Mantas Mikulėnas
Or just making 'sendmail' send a SIGALRM to the main daemon would do
the job perfectly well, I suspect...
Post by Lennart Poettering
Post by Kamil Jońca
But I still does not know, where is the problem, why exim doest not play
well with systemd ...
Here's an educated guess: your script terminates, so that that systemd
decides your service has ended. In such a case it kills any left-over
processes of the service, and this will include the exim process
forked off into the bg, because it is attributed to your script's
context.
I made some tests (ie.add sleep at end of exec line),
and this confirms your explanation.
But ... why this is working as --user service? pure luck?
Maybe the --user manager doesn't have privileges to kill a setuid-root
process.
--
Mantas Mikulėnas
Lennart Poettering
2018-09-24 19:17:44 UTC
Permalink
Post by Mantas Mikulėnas
Post by Lennart Poettering
Post by Kamil Jońca
Post by Mantas Mikulėnas
This didn't work well enough IIRC, but if it did, then it'd
provide almost postfix-like architecture.
Post by Lennart Poettering
Post by Kamil Jońca
Post by Mantas Mikulėnas
Or just making 'sendmail' send a SIGALRM to the main daemon would do
the job perfectly well, I suspect...
Post by Lennart Poettering
Post by Kamil Jońca
But I still does not know, where is the problem, why exim doest not play
well with systemd ...
Here's an educated guess: your script terminates, so that that systemd
decides your service has ended. In such a case it kills any left-over
processes of the service, and this will include the exim process
forked off into the bg, because it is attributed to your script's
context.
I made some tests (ie.add sleep at end of exec line),
and this confirms your explanation.
But ... why this is working as --user service? pure luck?
Maybe the --user manager doesn't have privileges to kill a setuid-root
process.
Precisely.

Lennart
--
Lennart Poettering, Red Hat
Kamil Jońca
2018-09-24 19:31:30 UTC
Permalink
Post by Kamil Jońca
Post by Lennart Poettering
Post by Kamil Jońca
This didn't work well enough IIRC, but if it did, then it'd provide almost postfix-like architecture.
Or just making 'sendmail' send a SIGALRM to the main daemon would do the job perfectly well, I suspect...
But I still does not know, where is the problem, why exim doest not play
well with systemd ...
Here's an educated guess: your script terminates, so that that systemd
decides your service has ended. In such a case it kills any left-over
processes of the service, and this will include the exim process
forked off into the bg, because it is attributed to your script's
context.
I made some tests (ie.add sleep at end of exec line),
and this confirms your explanation.
But ... why this is working as --user service? pure luck?
KJ
And another question: is it a way to make systemd not to kill background
processes?
KJ
--
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
Let us never negotiate out of fear, but let us never fear to negotiate.
-- John F. Kennedy
Kamil Jońca
2018-09-24 19:48:05 UTC
Permalink
***@o2.pl (Kamil Jońca) writes:

[...]
Post by Kamil Jońca
And another question: is it a way to make systemd not to kill background
processes?
Sorry for premature question. I just read about KillMode setting.
KJ
--
http://wolnelektury.pl/wesprzyj/teraz/
"Maybe we can get together and show off to each other sometimes."
Dan Nicholson
2018-09-24 20:07:53 UTC
Permalink
On Mon, Sep 24, 2018 at 1:38 PM Lennart Poettering
Post by Lennart Poettering
Post by Kamil Jońca
This didn't work well enough IIRC, but if it did, then it'd provide almost postfix-like architecture.
Or just making 'sendmail' send a SIGALRM to the main daemon would do the job perfectly well, I suspect...
But I still does not know, where is the problem, why exim doest not play
well with systemd ...
Here's an educated guess: your script terminates, so that that systemd
decides your service has ended. In such a case it kills any left-over
processes of the service, and this will include the exim process
forked off into the bg, because it is attributed to your script's
context.
Thus the exim queue runner might be forked off, but it wil be
immediately terminated by systemd again...
This is exactly what exim does when run via sendmail (which mail and
mutt likely use behind the scenes). I had a very similar setup to the
OP using exim on one of our servers and was perplexed why the queue
wasn't being flushed immediately. Here's what I wrote our git history:

I had naively thought that sendmail used SMTP to send a mail to the
local MTA. That's actually not the case for either exim or postfix,
which use internal procedures to queue and send the mail. Normally
that's not a problem except when sendmail is run by a systemd service.
Then the way that processes are spawned to queue and send the mail have
to match the service type. In the case of exim, the sending process
actually forks from sendmail, which systemd kills before it can run if
the service type is set to oneshot.

I ended up just writing python script to send the mail via SMTP to the
localhost.

--
Dan
Reindl Harald
2018-09-24 20:30:58 UTC
Permalink
Post by Dan Nicholson
This is exactly what exim does when run via sendmail (which mail and
mutt likely use behind the scenes). I had a very similar setup to the
OP using exim on one of our servers and was perplexed why the queue
I had naively thought that sendmail used SMTP to send a mail to the
local MTA. That's actually not the case for either exim or postfix,
which use internal procedures to queue and send the mail. Normally
that's not a problem except when sendmail is run by a systemd service.
Then the way that processes are spawned to queue and send the mail have
to match the service type. In the case of exim, the sending process
actually forks from sendmail, which systemd kills before it can run if
the service type is set to oneshot.
and be happy about it!

other than with the dumb native "sendmail" you don't lose mails just
because the service won't start for whatever reason

see what happens otherwise

https://www.reddit.com/r/vmware/comments/7mmh3o/vcenter_server_appliance_65_update_1cd_email/

you lose alert mails just because that idiotic configuration was wrong,
with postfix after you got it starting again you get all your
system-mails in the meantime submitted with the "sendmail-interface" aka
maildrop

Michael Biebl
2018-09-24 09:03:25 UTC
Permalink
Post by Kamil Jońca
--8<---------------cut here---------------start------------->8---
[Unit]
Description="Test maili"
[Service]
#User=kjonca
NoNewPrivileges=false
Type=oneshot
ExecStart=-zsh -c 'echo xxx|mail news'
ExecStart=-zsh -c 'echo xxx|mutt -F /dev/null -s subject -e \'set copy=no\' kjonca'
--8<---------------cut here---------------end--------------->8---
When I call
sudo systemctl start mailtest.service
Two messages are put in exim queue, but not deliveried immediately.
Why? What am I missing?
Fwiw, I tried your example service file, works for me.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Kamil Jońca
2018-09-24 09:11:13 UTC
Permalink
[...]
Post by Michael Biebl
Fwiw, I tried your example service file, works for me.
To be clear:
Message was delivered at once;
not put in queue, and delivered in queue run?
KJ
--
http://wolnelektury.pl/wesprzyj/teraz/
<Mercury> LordHavoc: I'm already insane.
<Coderjoe> damn straight. or curvy, crooked, or what have you
Loading...