Gmail silently discards forwarded email for sender domains with SPF record "-all"

[petergh]

I have notced that Gmail sometimes(?) discards email forwarded mail if the sender domain has an SPF record that contains "-all". The real problem is that Gmail accepts the message but never delivers it, not even to the spam folder. This happens even if the forwarding service, in this case Fastmail, uses SRS rewriting.

Does anyone know a way around this?

Peter

[William9]

What using the SPF soft fail "~all" ?

[petergh]

I have no control over the SPF records of the domains from which I receive mail. It's these domains that have -all, not my own domain.

[n5bb]

Are you sure that the messages are not being delivered to your Gmail account but were delivered to your Fastmail account? If you send test messages from your Gmail account to your Fastmail forwarding address, they will not appear in your Gmail Inbox. You must look in All Mail in your Gmail account to see these messages, which are tagged as from me. Do you also deliver them to your Fastmail account so you can see them there?

I sent various test messages to Fastmail aliases, one which used SRS and another which did not use SRS. All of these were delivered to my Gmail Inbox, even when I sent with From set to something @ example.org, a standard test address which has SPF set to v=spf1 -all (which rejects all messages from that domain with hardfail).

It appears to me that Gmail accepts messages which fail SPF with either softfail or hardfail as long as DKIM passes. I can't easily generate a test message with DKIM failing, so I'm not sure what happens if DKIM fails.

So my guess is one of these is happening:

• The message is hiding in your Gmail account (but not in Inbox) or is being filtered in some manner.
• The message isn't arriving at the proper Fastmail alias which targets your Gmail account.
• DKIM (or some reputation issue other than SPF) is causing the message to be rejected.

Bill

[petergh]

Hi Bill,

The messages (and there are several, from several different senders) are delivered to a Fastmail account as well as forwarded to Gmail, so I can tell for certain when a message hasn't made it to Gmail. I know about the Gmail->Fastmail->Gmail issue, so I send test messages from an external provider, e.g. Outlook.com.

I tried your test with example.org as well, and sure enough it made it through to Gmail. I guess that rules out guesses 1 & 2, and leaves some undocumented rule that silently discards some messages. Le sigh.

Peter

[n5bb]

Peter, you might look for a lack of successful DKIM signing on messages delivered to Fastmail but not making it to Gmail. Google might block messages which SPF hardfaiil if they have no other reputation features.

If a Google rejects them at the SMTP acceptance stage, I would assume that if SRS was used that Fastmail would in form the sender (not the FM account). Do you know if any sender receives a message disposition notice from Google?

If you can come up with some relatable test that fails, Fastmail could examine their logs to see how the Google receiving server is responding.

Unfortunately, I can find no currently maintained list showing SPF and DKIM policies for various email services.

Bill

[DavidJ]

petergh,

I used to use Gmail even for semi-serious mail, and what made me move was that Gmail would sometimes discard good emails which I wanted to receive completely silently - no notification, nothing in the spam folder.

I never did the research to try to find out which emails were being rejected; for all I know, it may have been some problem with authentifications, though the emails involved were from reputable sources who used reputable providers.

Good luck.
David.

[lane]

Gmail will sometimes discard without notice email which fails DMARC. I believe this normally happens only with major companies which have arranged with Gmail to do this, e.g., Paypal, banks, but perhaps Gmail does it more widely now.

For those unfamiliar with it, DMARC is a way for the owner of a domain to protect the domain name (like Paypal, which used to be frequently spoofed), by publishing a policy as to what to do with email that fails SPF or DKIM. Although it is more complicated than this, roughly speaking if (1) the domain owner publishes a DMARC policy, and (2) the email does not pass either SPF or DKIM on the domain in the From field (not Return-Path), and (3) the policy says to discard email which fails DMARC, then the domain owner is asking the receiving email system (e.g., Gmail, Hotmail) to discard the message. Other policies can be published too, for example to quarantine the message.

I'm not sure that this is actually happening to you, though. As I think someone mentioned, if you use your own account to send to yourself on Gmail, its duplicate suppression policy will ensure that you never see the received copy.

[petergh]

Let me just reiterate: I am *not* sending from my Gmail account to myself. I am sending test messages from an external account, so suppressed duplicates are not the issue here.

It seems like I'm not the only one experiencing this issue, judging by some of the posts in this thread on WHT:

http://www.webhostingtalk.com/showthread.php?t=1247647

I just wish I could find some kind of documentation that will tell me exactly when Gmail will silently gobble up a message but not deliver it, but I guess that would aide spammers, too, so probably no luck with that.

[FredOnline]

Have you checked in All Mail for the missing e-mail?

[petergh]

Yes, and it's not there.

[n5bb]

As lane said, DMARC provides a way for a sender domain to specify what they would like to happen for messages which fail both DKIM and SPF.

• SPF can fail during forwarding (although SRS forwarding should solve that issue). DKIM shouldn't be affected by forwarding (unless the content or subject header is modified). So if a message is sent with both SPF and DKIM and these pass at the receiver, it will probably pass the DMARC test.
• DMARC suggests that the destination system by default use the policy specified by the appropriate DNS TXT record (which for Gmeail is
  "v=DMARC1; p=none; rua=mailto:mailauth-reports@google.com".
• The main DMARC policies which can be specified by the sender are:
  • none (do not treat the message differently due to DMARC, although reports may be sent to the sending domain) - This is specified by the above Gmail DMARC policy on their sent mail.
  • quarantine (place the message in a spam folder or otherwise hide it from the Inbox)
  • reject (cancel the message at the SMTP receiving level with a 550 response to the sending server)
• Although the sender DMARC policy can be published in the domain DNS records, a destination system can choose to use the sender DMARC policy, ignore that policy, or modify the policy.
  • For example, if the sending server is on a reject list or the message has other spammy characteristics, Gmail may reject the message. Forwarding complicates this process, since it's not clear whether the headers can be trusted when the message was apparently accepted by another server.

Peter, are your test messages sent directly to a Gmail account (without forwarding) delivered properly? Have you tried with messages sent from various source systems? Obviously, the vast majority of non-spam messages sent to Gmail are delivered properly, so there must be something about the domains or servers originating the messages (assuming that the message subject and contents aren't suspicious).

Bill

[lane]

Originally Posted by petergh

I have notced that Gmail sometimes(?) discards email forwarded mail if the sender domain has an SPF record that contains "-all". The real problem is that Gmail accepts the message but never delivers it, not even to the spam folder. This happens even if the forwarding service, in this case Fastmail, uses SRS rewriting.

It does sound, from your subsequent comments, that this is not a DMARC issue. I was not aware that Gmail would quietly drop any messages without a bounce to the sender; in fact, this is why I gave up Hotmail a number of years ago.

Is it possible somehow that Gmail is in fact giving a bounce response, but it is not making it back to yourself at the original test account? I do have one address (my wife's) at my domain forwarded with SRS from Fastmail to Google Apps (essentially Gmail), so I tried a test. I sent a test message with an executable attachment (which will be refused by Gmail) from an Outlook.com account to that address at my domain (hosted at Fastmail), which directs one copy to a Fastmail subfolder for insurance purposes and sends a second with SRS to Google Apps. The copy to Google Apps was indeed refused with a bounce message back to Fastmail (it was sent to the address SRS had rewritten as "SRS0=mr/t=WD=outlook.com=xxxxxx@srs.messagingengine.com") and Fastmail properly sent it back to the originating Outlook.com account. So the bounce message appeared in the original sending account, and everything worked properly as far as I can see.

So there are many links in the chain for a forwarded message:

1. The original message is sent to Fastmail. Fastmail will drop it quietly under certain circumstances, e.g., sent from a dial-up server, perhaps after greylisting.
2. Then Fastmail should perform SRS and forward it to Gmail. Perhaps Gmail may drop it quietly at this point, and that is very disturbing.
3. If the message is unacceptable to Gmail, say because it has an unacceptable attachment (exe or html file etc.) as in my test case, or because of an issue with SPF "-all", Gmail should generate a bounce at the smtp stage.
4. Gmail's refusal should be read by the sending Fastmail server at the smtp negotiation stage with the Gmail server, and Fastmail's server should generate a bounce reply to the recipient defined by SRS.
5. Fastmail should unwind the SRS recipient and send the bounce back to the original sender.
6. The original sending mail system might deliver the bounce or drop it.

I am not sophisticated enough to think of tests that you could perform that might unwind each of these possible points of failure. But it is certainly possible that Gmail is not dropping it quietly at step #2 and instead one of the other links is failing. Nevertheless, I understand your concern, and of course, unless you pay for a business Google Apps account, there is no one at home at Gmail to talk to about it.

[petergh]

Thanks for your valuable input, everyone, and lane and Bill in particular. It's much appreciated.

I've managed to find a message that was delivered to my wife's Fastmail account, but never made it to Gmail. These are the full headers:

Return-Path: <international@emarsys.net>
Received: from compute2.internal (compute2.nyi.mail.srv.osa [10.202.2.42])
	 by sloti19t10 (Cyrus git2.5+0-git-fastmail-9538) with LMTPA;
	 Sun, 15 Dec 2013 02:26:11 -0500
X-Sieve: CMU Sieve 2.4
X-Spam-score: 1.6
X-Spam-hits: BAYES_50 0.8, HTML_IMAGE_RATIO_04 0.556, HTML_MESSAGE 0.001,
  MIME_HTML_ONLY 0.723, RP_MATCHES_RCVD -0.435, T_FRT_CONTACT 0.01,
  T_KHOP_FOREIGN_CLICK 0.01, LANGUAGES da, BAYES_USED global,
  SA_VERSION 3.3.2
X-Spam-source: IP='91.211.240.12', Host='pmta40012.emarsys.net', Country='AT',
  FromHeader='dk', MailFrom='net'
X-Spam-charsets: subject='UTF-8', html='utf-8'
X-Resolved-to: mywife@fastmail.fm
X-Delivered-to: mywife@ourdomain.com
X-Mail-from: international@emarsys.net
Received: from mx4 ([10.202.2.203])
  by compute2.internal (LMTPProxy); Sun, 15 Dec 2013 02:26:11 -0500
Received: from pmta40012.emarsys.net (pmta40012.emarsys.net [91.211.240.12])
	by mx4.messagingengine.com (Postfix) with ESMTP id 109B13C00AA
	for <mywife@ourdomain.com>; Sun, 15 Dec 2013 02:26:11 -0500 (EST)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=key2; d=nyheder.bilka.dk;
 h=To:Subject:From:MIME-Version:List-Id:List-Unsubscribe:Content-Type:Content-transfer-encoding:Message-ID:Date; i=mail@nyheder.bilka.dk;
 bh=8dcOBdySHD1kR+eDzgs2xKB0cJ0=;
 b=DBaPWAZBXbvm0ZlqLvk90uTQnPUWEJvMjQhM6SFGj8qChJDzqysMeeBNJX0Rs31mNA4ngnBDKcJ1
   w9jvP1f6p3eFSHCmKRb0fNO9EUoAzIwzz26v5OJHJJI1ZZRcRV+Pttk9eatdY7/+D3nBLdwMIW3n
   oCjvtd/6Z+XfsRpfDbc=
DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=key2; d=nyheder.bilka.dk;
 b=g8UxAeLzF5G4ym07fWoM5rN4Q/075P4/yvb5QCK9CCg2h0rfdElJyFl73HINwB1wkuG5M7MsYGH1
   97/IEsOxIQcbTT3zVQUaElwW8SIWrA04815BNvcVO1EKZRayS9qb6PBhmcMnhHU8Opy8pVwoNZwX
   tSWfnkT7OGN8DzyqYi4=;
Received: from (91.194.249.233) by pmta40012.emarsys.net id hllcp616nd8n for <mywife@ourdomain.com>; Sun, 15 Dec 2013 08:26:09 +0100 (envelope-from <international@emarsys.net>)
To: <mywife@ourdomain.com>
Subject: 
 Det Store Gaveguide | Shop julegaver til hele familien i dag | Dagens
 JuleNetkup | Inspiration til gaverne til ham, hende,
 =?UTF-8?Q?b=C3=B8rnene=20og=20teenageren?=
X-Mailer: class SMTPMail
From: "Bilkas netbutik" <mail@nyheder.bilka.dk>
MIME-Version: 1.0
X-EMarSys-Identify: 121595857_758214_94641
List-Id: 121595857 <Bilkas netbutik>
X-EMarSys-Environment: international
X-CSA-Complaints: whitelist-complaints@eco.de
List-Unsubscribe: <http://link.bilka.dk/u/lu.php?lu=nbsaA:xBPF:,pW:nJ5qDH2n3z>
Content-Type: text/html; charset=utf-8
Content-transfer-encoding: quoted-printable
Message-ID: <0.1.47.935.1CEF966ECBAC760.0@pmta40012.emarsys.net>
Date: Sun, 15 Dec 2013 08:26:09 +0100

As I understand from lane, DMARC looks at the domain in the From: header, which in this case is "nyheder.bilka.dk". This (sub)domain does indeed have a published DMARC record with instructions to reject invalid messages (p=reject), as you can see here:

https://dmarcian.com/dmarc-inspector/nyheder.bilka.dk

However, pasting the message into an online DKIM validator (e.g., https://9vx.org/~dho/dkim_validate.php), it looks like the signature checks out.

Since forwarding from FM doesn't modify the DKIM signature, it should also check out at Gmail's end.

I'm unable to tell if the sender of this particular message received a bounce or not. They may have, or they may not.

Any thoughts?

[petergh]

Just for the sake of clearing up any doubt, here's a link to Gmail's DMARC page that clearly states what is required for a message to fail or pass the DMARC test:

https://support.google.com/a/answer/2466580?hl=en

"A message must fail both SPF and DKIM checks to also fail DMARC. A single check failure using either technology allows the message to pass DMARC."