Understanding "unknown" in Received Header

[Rich Shepard]

I'm apparently still missing a critical part of understanding what is
happening during the SMTP exchange when a remote client wants to have a sent
message accepted by my MTA for internal delivery. Your patient explanation is
appreciated.

It is almost always a contrived hostname with the real IP address that
accompany the spam that continue to get through to my in-box. This is what
I've been trying to reject correctly; I still don't "get it."

On page 59 of Ralf and Patrick's "The Book of Postfix" I read,

"Client

"The client is the machine sending mail; Postfix will either log hostname
and IP, or "unknown" (if the hostname cannot be determined using DNS
lookups). Postfix gets the client's IP address from the kernel's TCP/IP
stack, and gets name from DNS or /etc/hosts before SMTP communication takes
place. This allows Postfix to impose restrictions if the client's IP address
and the hostname during SMTP communication don't match."

I thought that the restrictions I added to /etc/postfix/main.cf would catch
these mis-matched IP addresses and hostnames. They're not. So, I added a
header_check,

/^Received: .*unknown \[/ reject Rule H9

but was advised that this is quite risky and can result in rejections of
legitimate mail.

The two questions I would like answered are:

1) Under what circumstances would legitimate mail not reverse resolve the
IP address to the MX or A record hostname in the DNS?

2) What is the appropriate way to test for, and reject, mis-matched IP
addresses and hostnames?

I feed all these messages to SpamAssassin but the rest of the headers and
body are sufficiently varied that it doesn't catch this situation.

TIA,

Rich

[Keith Matthews]

On Fri, 14 Oct 2005 09:24:38 -0700 (PDT)

Post by Rich Shepard
I thought that the restrictions I added to /etc/postfix/main.cf would catch
these mis-matched IP addresses and hostnames. They're not. So, I added
a header_check,
/^Received: .*unknown \[/ reject Rule H9
but was advised that this is quite risky and can result in rejections
of legitimate mail.
1) Under what circumstances would legitimate mail not reverse resolve the
IP address to the MX or A record hostname in the DNS?

Basically any source that is connected by DSL (or in some cases cable).
Just about every ISP out there insists on setting up the reverse records
to something like <IP>.dsl.<ispname>, regardless of whether the address
is static or dynamic. Seeing as many SMEs are being tempted to run their
own domains by the broadband providers (especially by BT here in the UK)
that means that a lot of quite legitimate sources will not have a proper
reverse pointer.

If you want an example look up my details.

Depending on your correspondence partners this may or may not be of
importance to you.

Post by Rich Shepard
2) What is the appropriate way to test for, and reject,
mis-matched IP
addresses and hostnames?

It's quite difficult to do it sensibly unless you can afford to reject
all mail from broadband connected sources. I've managed to get a
situation where the only spam I've received in the last 3 months has
come via open mailing lists. It's hard and means rejecting all mail to
list subscription accounts (e.g. this one) unless it comes from the list
server (and that in turn means having a separate account for each list).
If you are subscribed to a sourceforge list it's even more complex as
they use sender verification and you have to allow from the list's
recieve-side as well.

If it's appropriate to you, it may be better to identify countries that
give rise to spam and don't raise any legitimate mail to you and block
them completely. It's very effective but may well mean that you have to
make sudden changes if your legitmate correspondence partners change.

Dial-up lists (recently discussed here at some length) will help as long
as they are properly set up, unfortunately many are not and assume that
broadband=dynamic addressing. Again it's a matter of YMMV, personally I
find them not worth the effort.

[Noel Jones]

This post might be inappropriate. Click to display it.

[Rich Shepard]

Post by Keith Matthews
Basically any source that is connected by DSL (or in some cases cable).
Just about every ISP out there insists on setting up the reverse records to
something like <IP>.dsl.<ispname>, regardless of whether the address is
static or dynamic. Seeing as many SMEs are being tempted to run their own
domains by the broadband providers (especially by BT here in the UK) that
means that a lot of quite legitimate sources will not have a proper reverse
pointer.

Thank you, Keith.

My experience with DSL has been that those of us with static IP addresses
have those associated with our domain names. I can see the situation being
different with cable subscribers, however. I did not realize that we may be
the exceptions rather than the rule.

Post by Keith Matthews

Post by Rich Shepard
2) What is the appropriate way to test for, and reject, mis-matched IP
addresses and hostnames?

It's quite difficult to do it sensibly unless you can afford to reject all
mail from broadband connected sources.

No, that's not practical or reasonable for me, either.

Post by Keith Matthews
If it's appropriate to you, it may be better to identify countries that
give rise to spam and don't raise any legitimate mail to you and block them
completely. It's very effective but may well mean that you have to make
sudden changes if your legitmate correspondence partners change.

I've been working on this approach for several years now. What prompted my
question earlier this week (which resulted in a long thread) was having
mismatched spam mail show up in my inbox when the first two bytes of the IP
address _are_ supposed to be rejected.

Rich

[Ralf Hildebrandt]

Post by Rich Shepard
I'm apparently still missing a critical part of understanding what is
happening during the SMTP exchange when a remote client wants to have a
sent message accepted by my MTA for internal delivery. Your patient
explanation is appreciated.
It is almost always a contrived hostname with the real IP address that
accompany the spam that continue to get through to my in-box. This is what
I've been trying to reject correctly; I still don't "get it."
On page 59 of Ralf and Patrick's "The Book of Postfix" I read,
"Client
"The client is the machine sending mail; Postfix will either log
hostname and IP, or "unknown" (if the hostname cannot be determined
using DNS lookups). Postfix gets the client's IP address from the
kernel's TCP/IP stack, and gets name from DNS or /etc/hosts before SMTP
communication takes place. This allows Postfix to impose restrictions
if the client's IP address and the hostname during SMTP communication
don't match."
I thought that the restrictions I added to /etc/postfix/main.cf would
catch these mis-matched IP addresses and hostnames.

Which one did you add?

Post by Rich Shepard
They're not. So, I added a
header_check,
/^Received: .*unknown \[/ reject Rule H9
but was advised that this is quite risky and can result in rejections of
legitimate mail.

Just like setting the appropriate restrictions in main.cf. It lies in
the nature of what you request.

Post by Rich Shepard
1) Under what circumstances would legitimate mail not reverse resolve the
IP address to the MX or A record hostname in the DNS?

* Incompetent DNS admin
* bad ISP
* multi homed hosts

Post by Rich Shepard
2) What is the appropriate way to test for, and reject, mis-matched IP
addresses and hostnames?

reject_unknown_client

[Rich Shepard]

I think you're confusing the HELO hostname with the client hostname. In a
properly configured server they will usually be the same (multi-homed and
load-balanced hosts can be an exception), but many, many legit system are
misconfigured.

Noel,

I do believe that you're correct in my confusion.

Here's an example...
Received: from dinamic.biada.org (unknown [62.81.151.161])
by mgate2.vbhcs.org (Postfix) with ESMTP id EE60B797813

This is what I'm seeing in the spam that gets through.

In the above, "dinamic.biada.org" is the HELO name given by the other mail
system. 62.81.151.161 is the IP address of the client. Postfix does a "host
62.81.151.161" which show this client has a PTR of
"161-151-81-62.libre.auna.net". Postfix then does a "host
161-151-81-62.libre.auna.net" and finds that name doesn't exist (we
expected an answer of 62.81.151.151), so this client is labeled as
"unknown". (this isn't *exactly* what postfix does, but it illustrates what
happens, and shows you how to test it.)

Thank you for a really clear explanation of the process that I did not
fully understand before.

Some legit hosts will be "unknown", mostly people who have no control over
their reverse DNS entry. This is common in some countries, also common for
dialup/dynamic connections. And of course some are just misconfigured.

OK. I'm now fully convinced that I'm being too restrictive.

That's too broad a brush as it looks at ALL the headers. You could use
reject_unknown_client to reject clients that postfix considers unknown, but
even that may reject legit mail you want. Maybe try "warn_if_reject
reject_unknown_client" for a while to see how much legit mail you would
drop. You can add a whitelist to accept those you do want.

Excellent advice. I did not have reject_unknown_client in main.cf, but I do
have reject_unknown_sender_domain and reject_unknown_recipient_domain. I'll
do it cautiously as you recommend.

HTH.

Very much so. Thanks.

Rich

[Ralf Hildebrandt]

This post might be inappropriate. Click to display it.

[Keith Matthews]

On Fri, 14 Oct 2005 10:27:59 -0700 (PDT)

Post by Rich Shepard

Post by Keith Matthews
Basically any source that is connected by DSL (or in some cases
cable). Just about every ISP out there insists on setting up the
reverse records to something like <IP>.dsl.<ispname>, regardless of
whether the address is static or dynamic. Seeing as many SMEs are
being tempted to run their own domains by the broadband providers
(especially by BT here in the UK) that means that a lot of quite
legitimate sources will not have a proper reverse pointer.

Thank you, Keith.
My experience with DSL has been that those of us with static IP
addresses
have those associated with our domain names. I can see the situation
being different with cable subscribers, however. I did not realize
that we may be the exceptions rather than the rule.

To the best of my knowledge only one UK ISP is seriously willing to set
up proper reverse DNS. They are one of the really small ones and do not
handle dynamic IP at all.

BT are famous for claiming that reverse DNS is not necessary.

[mouss]

Post by Rich Shepard
I'm apparently still missing a critical part of understanding what is
happening during the SMTP exchange when a remote client wants to have a sent
message accepted by my MTA for internal delivery. Your patient
explanation is
appreciated.
It is almost always a contrived hostname with the real IP address that
accompany the spam that continue to get through to my in-box. This is what
I've been trying to reject correctly; I still don't "get it."
On page 59 of Ralf and Patrick's "The Book of Postfix" I read,
"Client
"The client is the machine sending mail; Postfix will either log hostname
and IP, or "unknown" (if the hostname cannot be determined using DNS
lookups). Postfix gets the client's IP address from the kernel's TCP/IP
stack, and gets name from DNS or /etc/hosts before SMTP communication takes
place. This allows Postfix to impose restrictions if the client's IP address
and the hostname during SMTP communication don't match."
I thought that the restrictions I added to /etc/postfix/main.cf would catch
these mis-matched IP addresses and hostnames. They're not. So, I added a
header_check,
/^Received: .*unknown \[/ reject Rule H9
but was advised that this is quite risky and can result in rejections of
legitimate mail.

Because if the sender is inside a private domain with no dns (or with
msiconfigure dns,...), his own MTA will set the rdns to unknown.
Received is a "trace" header, and should rarely (if ever) be used to
reject mail.

Post by Rich Shepard
1) Under what circumstances would legitimate mail not reverse resolve the
IP address to the MX or A record hostname in the DNS?

the MX is a different thing. many organizations have an outbound MTA
that is different from their MX.
The checks you posted previously are used to reject senders with a "bad"
MX (such as thos setting their MX to 127.0.0.1...).

As for the "unknown" client, this is not generally considered a spam sign.

Post by Rich Shepard
2) What is the appropriate way to test for, and reject, mis-matched IP
addresses and hostnames?

This may cause false positives. but if you really want that, use
reject_unknown_client (or put "unknown" in a client access map).

Post by Rich Shepard
I feed all these messages to SpamAssassin but the rest of the headers and
body are sufficiently varied that it doesn't catch this situation.

even SA doesn't care about "unknown". because this isn't a real sign for
spam. you may add a rule for that, but make sure to only match your
Received header (to avoid matching headers generated inside the sender's
domain).