Why marked as spam, although in address book?

[rabarberski]

A couple of months ago I cleaned out my complete spam folder (to save some space).
Since then, quite some good messages have been arriving in my spam folder, that were previously not. I repeatly started marking them as "Not spam", but even some months later this didn't help.
So, I added some of the sender address to my address book, but some still end up in my Spam folder.
Why could that be?

If I look at the raw content of a message marked as spam (spam score 5.4) I see that the X-Spam-known-sender field says "no" but it /is/ correctly recognized as being in my address book (i.e. in the group "whitelist"). So why is it saying "no" to "known-sender". The sender address is my address book as *@<subdomain>.<domain>.com


X-Spam-known-sender: no ("Email failed DMARC policy for domain"), in-addressbook,
10c8717d-4508-4dc9-a564-4880a5854db6 ("whitelist")
X-Spam-score: 5.4

[n5bb]

As your post shows, that email message failed the DMARC policy published for the domain. DMARC allows the owner of a domain to specify which outgoing SMTP servers are allowed to send messages using their domain in the From address. The purpose is to reduce spam and especially phishing attacks. For example:
https://support.google.com/mail/answer/2451690?hl=en
https://dmarc.org/stats/alexa-top-sites/dmarc/

Yahoo publishes a p=reject DMARC policy for their yahoo.com domain, so if you have a yahoo.com email account and send email using that From address from any server other than those specified by Yahoo, your mail will be rejected by any recipient email system who implements the DMARC rules.

If you want to test this, enter the From domain in this tool:
https://www.misk.com/tools/#dns/
Microsoft.com specifies p=quarantine, so messages sent from the wrong server will be quarantined by recipient email systems (which is usually the spam folder).

Each recipient email system can choose to use the full DMARC recommendations, ignore DMARC, or use some modified policy. In this case, it appears that FastMail has quarantined the message because it was sent from the wrong server. Certain types of forwarding or redirection can also cause this to occur.

Bill

[jhollington]

The note that the email failed the DMARC policy for the domain appears to be the key here. This obviously factors into the spam score, and somehow overrides any whitelisting you may have in place. Without knowing the sending domain and other details, it's not possible to know exactly what DMARC policy is in place that might have triggered this, but assuming that a policy or mail server is correctly configured, it's usually a good sign that the message could be spoofed.

Whether this makes sense or not is open to some debate, but to me it makes sense that if a message fails the DMARC policy for the sending domain a whitelisting based on the sender simply being in your address book wouldn't necessarily be trustworthy, as the sender could be forged. In other words, you may whitelist "bob@gmail.com" but if you're getting an email that says it's from Bob, but didn't actually come through Gmail's servers, you can't be certain it's really from Bob, as opposed to somebody pretending to be Bob.

To be fair, this sort of thing actually happens a lot more than you think with the number of malware and phishing emails out there. It seems to have dropped off in recent years as people (and ISPs, and companies like Microsoft) secure their email clients and computers more diligently, but it wasn't uncommon for viruses to propagate themselves through email by extracting information from the address book of somebody you know, finding your name in it, and then sending out emails that appear to be "From" them in the hopes that you'd actually fall for the trap by thinking the message is from somebody you know.

[rabarberski]

Thanks for the explanation. I have it for:
nieuwsbrief@info.bol.com (a big Dutch online store)
Return-Path: <nieuwsbrief@info.bol.com>

Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
by sloti28t09 (Cyrus fastmail-fmwheezy42417-13919-git-fastmail-13919) with LMTPA;
Wed, 26 Oct 2016 05:45:37 -0400
X-Cyrus-Session-Id: sloti28t09-651032-1477475137-2-13967142697992196466
X-Sieve: CMU Sieve 2.4
X-Spam-known-sender: no ("Email failed DMARC policy for domain"), in-addressbook,
10c8717d-4508-4dc9-a564-4880a5854db6 ("whitelist")
X-Spam-score: 5.4
...
Received-SPF: fail
(info.bol.com: Sender is not authorized by default to use 'nieuwsbrief@info.bol.com' in 'mfrom' identity (mechanism '-all' matched))
receiver=mx1.messagingengine.com;
identity=mailfrom;
envelope-from="nieuwsbrief@info.bol.com";
helo=SNT004-OMC1S31.hotmail.com;
client-ip=65.55.90.42
...

invitations@linkedin.com

Return-Path: <invitations@linkedin.com>
Received: from compute4.internal (compute4.nyi.internal [10.202.2.44])
by sloti28t09 (Cyrus fastmail-fmwheezy42417-13919-git-fastmail-13919) with LMTPA;
Tue, 25 Oct 2016 14:20:04 -0400
X-Sieve: CMU Sieve 2.4
X-Spam-known-sender: no ("Email failed DMARC policy for domain"), in-addressbook,
10c8717d-4508-4dc9-a564-4880a5854db6 ("whitelist")
X-Spam-score: 13.7
...
Received-SPF: softfail
(linkedin.com: Sender is not authorized by default to use 'invitations@linkedin.com' in 'mfrom' identity, however domain is not currently prepared for false failures (mechanism '~all' matched))
receiver=mx4.messagingengine.com;
identity=mailfrom;
envelope-from="invitations@linkedin.com";
helo=COL004-OMC4S1.hotmail.com;
client-ip=65.55.34.203
...

I am especially surprised for linkedin.com, not really a small domain...

I do havee to state that my main address is a hotmail address that is forwarded to my fastmail account.

[jhollington]

It's your forwarding from Hotmail that's doing you in.

As the headers you've posted show, what's basically happening is this:

• A message from nieuwsbrief@info.bol.com goes to your Hotmail account.
• Hotmail accepts it and forwards it on to your Fastmail address.
• Fastmail sees a message from nieuwsbrief@info.bol.com so it looks up the SPF and DMARC records for the info.bol.com domain (in DNS). Those records list the servers that are allowed to send mail from info.bol.com. Hotmail's server (in this case, SNT004-OMC1S31.hotmail.com) is naturally not on that list.
• Fastmail compares the FROM address with the list of servers that are "authorized" to send mail from that address. They don't match, SPF fails and the message is considered to be spam as it's not coming from where it's supposed to.

In the case of info.bol.com, the SPF record uses "-ALL" which means that the list of servers is authoritative — unless a server is on that list, it is NEVER allowed to send mail from anybody @info.bol.com. Hence you see an outright SPF "FAIL." LinkedIn uses a slightly more relaxed approach with "~ALL" which means that the list of servers is not authoritative — other servers might be used to send mail from @linkedin.com, so that's an SPF "softfail" and is generally handled less seriously than an outright "fail."

Under normal circumstances, this is a good thing — it's exactly how you want Fastmail to work, as you wouldn't want somebody sending a message to you from "invitations@linkedin.com" if it was really just some guy with a Hotmail address forging a LinkedIn email to phish for your password.

However, for people forwarding mail, this is actually what SRS Rewriting is for (see Fastmail's help on SRS Rewriting — this applies to forwarding messages FROM FastMail TO another service, so it won't solve your problem, but it explains the concept behind it). I doubt that Hotmail supports such a feature, but if it does you could enable that to allow the forwarded messages to be delivered properly to Fastmail and avoid these issues.

[lane]

Your problem is from the forwarding from Hotmail, as has just been suggested. Both info.bol.com and linkedin.com have DMARC records. DMARC is a facility that allows senders to make a request to a receiving email system as to what to do when the receiving system fails certain verification tests for the authenticity of the sender. Roughly (I may not have all the words right), at least one of the following tests must be met: (1) spf is tested against the envelope sender ("smtp mailfrom"), it passes, and the domain of the envelope sender is the same as the domain (or subdomain if "relaxed" is allowed) in the visible "From:" field you see in your mail client; or, (2) a DKIM encryption code is evaluated and passes, which shows the message has not been modified, and the domain of the signer of the encryption agrees with the visible "From:" domain.

The DMARC record of info.bol.com asks that failures be put in spam, and linkedin.com asks that failures be rejected entirely (though Fastmail does not do that, but does raise the spam score substantailly).

In the case of Hotmail (aka outlook.com), SPF cannot succeed, even if the sender is rewritten, since it will not agree with the sender in the "From:". And Hotmail/Outlook.com at the present time modifies the message slightly in a way that results in DKIM failure. This is going to be fixed, but they are having some trouble doing so. See the writeup on the issue by the Microsoft chap who works with these things: Why does my email from Facebook, that I forward from my outlook.com account, get rejected?

Particularly, follow the comments to see the current status of the fix. I am eagerly awaiting it myself.

[lane]

Conclusion: you need to either wait out the fix, or do something else:

1. You could set the spam score required to move something to Spam much higher. Downside might be extra spam in your Inbox.
2. You could have Fastmail POP the mail down from Hotmail instead of using forwarding. Downside: you no longer get nearly instant delivery.

[BritTim]

In theory, custom sieve code could be used to resolve the problem completely. It could check for the specific case of "known sender" being ignored for DKIM reasons, and apply a higher threshold for spam in that situation. Bit messy, of course.

[n5bb]

The ugly truth is that forwarding in this manner just isn't compatible with current techniques for spam prevention. So the best solution is to log into the sources of those emails and change the email address they use to your FastMail address. I also suggest that you use a subdomain address for two reasons:

• Any message sent to that subdomain address is guaranteed to be from that service, unless that service sells your address or there is a security breach. In any of these cases, since you only use that subdomain address for that one purpose, you know about security problems immediately when spam arrives at that subdomain address.
• Incoming messages from that service can be easily filed or otherwise processed by rules. Since many services (such as banks) often use more than one From address, in this situation all you need to look for in a rule is a single destination (delivery) address.

So if you owned the email address rab@example.com, you could use linkedin@rab.example.com as a subdomain address. The received message would by default be delivered to rab+linkedin@example.com, so if you had a "LinkedIn" folder it would receive those messages.

Bill

[rabarberski]

Thanks (a lot) for all the great answers. Very helpful !
I will look into it in more detail this weekend.

[lane]

Originally Posted by n5bb

The ugly truth is that forwarding in this manner just isn't compatible with current techniques for spam prevention.

You said it in only 18 words, Bill. I like your way better (see The decline and fall of forwarding for additional technical discussion). The best advice about forwarding now seems to be, "Don't do it."

[David]

Originally Posted by lane

You said it in only 18 words, Bill. I like your way better (see The decline and fall of forwarding for additional technical discussion). The best advice about forwarding now seems to be, "Don't do it."

That would not be the case (I am hoping) when forwarding directly from a domain name registrar.....

[n5bb]

Originally Posted by David

That would not be the case (I am hoping) when forwarding directly from a domain name registrar.....

The DMARC restriction involves information placed in the DNS records by the domain owner (controller of those records for that domain). The domain registrar is going to have their own sending server, so they don't automatically get a reprieve. The DNS records for a domain are often not hosted (published) by the registrar. For instance, my personal domain is registered through NameSecure, but my DNS records are hosted/published by FastMail. Even if I let NameSecure host the DNS records, I as the domain owner have full control over the contents (not the DNS host or domain registrar).

So it makes no difference which registrar was used for yahoo.com -- any email sent with an address at the yahoo.com domain from any server not listed in their DMARC, SPF, and DKIM mechanisms will be rejected by the recipient email server if they strictly follow DMARC policy for that domain.

So as I said, the ugly truth is that automatic forwarding (as it is normally implemented, maintaining the original From address) is not compatible with current techniques of spam prevention. The most secure email transfer is made as follows:

• The message originator uses a secure connection to an email provider which supports their domain.
• The originating server is authorized by the DNS records (DMARC, SPF, and DKIM) to send mail for the domain of the From address in the message.
• DKIM signing is employed by the sending system.
• The sending and receiving transport servers use a secure connection.
• The receiving server applies DMARC rules on all received messages.
• The receiving server only allows access to the mail store via a secured connection.

This insures that the message is not modified in transit (DKIM), that the sending server is authorized to send for that domain (SPF), and that the message authenticity when received is handled as desired by the owner of the sending domain (DMARC).

Bill

[David]

Originally Posted by n5bb

The DMARC restriction involves information placed in the DNS records by the domain owner (controller of those records for that domain). The domain registrar is going to have their own sending server, so they don't automatically get a reprieve. The DNS records for a domain are often not hosted (published) by the registrar. For instance, my personal domain is registered through NameSecure, but my DNS records are hosted/published by FastMail. Even if I let NameSecure host the DNS records, I as the domain owner have full control over the contents (not the DNS host or domain registrar).

So it makes no difference which registrar was used for yahoo.com -- any email sent with an address at the yahoo.com domain from any server not listed in their DMARC, SPF, and DKIM mechanisms will be rejected by the recipient email server if they strictly follow DMARC policy for that domain.

So as I said, the ugly truth is that automatic forwarding (as it is normally implemented, maintaining the original From address) is not compatible with current techniques of spam prevention. The most secure email transfer is made as follows:

• The message originator uses a secure connection to an email provider which supports their domain.
• The originating server is authorized by the DNS records (DMARC, SPF, and DKIM) to send mail for the domain of the From address in the message.
• DKIM signing is employed by the sending system.
• The sending and receiving transport servers use a secure connection.
• The receiving server applies DMARC rules on all received messages.
• The receiving server only allows access to the mail store via a secured connection.

This insures that the message is not modified in transit (DKIM), that the sending server is authorized to send for that domain (SPF), and that the message authenticity when received is handled as desired by the owner of the sending domain (DMARC).

Bill

Thanks Bill: I only have two domain names (that are important to me) - one is registered at Namecheap, the other at baremetal (a Canadian provider) - I am falling behind, re the technicalities (of email in general) and need to get up to scratch. many thanks for the info.

[jhollington]

Originally Posted by David

Thanks Bill: I only have two domain names (that are important to me) - one is registered at Namecheap, the other at baremetal (a Canadian provider) - I am falling behind, re the technicalities (of email in general) and need to get up to scratch. many thanks for the info.

If you own the domain names, is there some reason you can't simply point the MX records directly to FastMail?You'd have to have at least a "Standard" account (under the new plans — I think "Enhanced" was the minimum under the old set of plans) to host your own domain, but it would be a MUCH cleaner way of doing this then relying on forwarding.