Header Order

[David MacQuigg]

I would like to get some opinions from folks who can read email headers, but are not necessarily email experts. The question is - in placing a new header, should we follow strict chronological sequence, or follow the common convention of always making the Received header lowest in the group of headers prepended by our Mail Transfer Agent (MTA). Here is an example from our current mailflow:

Received: from fence.pobox.com (fence.pobox.com [208.210.124.76]) by
wand.pobox.com (Postfix) with ESMTP id EDBE424BD for
<07422030@mailstore.pobox.com>; Thu, 16 Nov 2006 06:28:33 -0500 (EST)
Received: from fence.pobox.com (localhost [127.0.0.1]) by fence.pobox.com
(Postfix) with ESMTP id 716F2F0A8 for <07422030@mailstore.pobox.com>; Thu, 16
Nov 2006 06:28:27 -0500 (EST)
Delivered-To: xxx@pobox.com
Received: from fence (localhost [127.0.0.1]) by fence.pobox.com (Postfix)
with ESMTP id D99961161C for <xxx@pobox.com.07422030.000.icgmh>; Thu, 16
Nov 2006 06:27:33 -0500 (EST)
X-Pobox-Pass: SRS0=itbTd7ph=E4=pco-iraq.net=xxx@open-mail.org is
whitelisted
Received: from open-mail.org (open-mail.org [207.210.221.26]) by
fence.pobox.com (Postfix) with ESMTP id D913F116FE for <xxx@pobox.com>;
Thu, 16 Nov 2006 06:24:17 -0500 (EST)
X-SpamScore: -14.0
Received: from CANCER.pco-iraq.net (cancer.pco-iraq.net [64.191.210.175]) by
open-mail.org (8.13.1/8.13.1) with ESMTP id kAGBKtIf002229 for
<xxx@box67.com>; Thu, 16 Nov 2006 06:20:57 -0500
X-Authent: 64.191.210.175 pco-iraq.net PTR1 PASS ratings=(2,9,None) <=====
Received: from pavo.pco-iraq.net ([64.191.210.183]) by CANCER.pco-iraq.net
with Microsoft SMTPSVC(6.0.3790.1830); Thu, 16 Nov 2006 14:22:16 +0300
Received: from FORNAX.pco-iraq.net ([172.16.5.106]) by pavo.pco-iraq.net with
Microsoft SMTPSVC(6.0.3790.1830); Thu, 16 Nov 2006 14:20:52 +0300

The new header is marked with <=====. That and the two above it are our headers. The order of events is
1) We get a request for a mail session: HELO this is pco-iraq.net.
2) We authenticate the alleged sender 'pco-iraq.net' against the incoming IP address, and reject forgeries.
3) We prepend our new X-Authent header with the authentication results and the senders ratings.
4) We then receive the message headers and bodies, and prepend a Received header to mark that event.
5) If the sender is not whitelisted, the message goes to our spam filter, and we prepend the results with an X-SpamScore header.

The experts are telling me - no this is wrong. The header order should be:
X-SpamScore: -14.0
X-Authent: 64.191.210.175 pco-iraq.net PTR1 PASS ratings=(2,9,None) <=====
Received: from CANCER.pco-iraq.net (cancer.pco-iraq.net [64.191.210.175]) by
open-mail.org (8.13.1/8.13.1) with ESMTP id kAGBKtIf002229 for
<xxx@box67.com>; Thu, 16 Nov 2006 06:20:57 -0500

There is no RFC standard mandating this order. It appears we are free to chose any order we want as long as we keep them together in a block. So the experts appear to be favoring what seems more familiar - Received header always at the bottom.

My intent in placing the X-Authent header at the bottom was to follow a simple rule (strict chronology) that even non-experts will be able to follow - Everything above X-Authent is trusted. Everything below is not.

So the tradeoff appears to be simplicity for non-experts vs following a convention expected by experts.

One of my pet peeves is mail forwarders who re-order existing headers - a clear violation of standards with no apparent need (making the headers pretty, I guess)!

[Scott Kitterman]

Actually there is no RFC mandating header order. The fact that DKIM requires header order to maintained was noted as a potiential issue for the DKIM-base specification during the IETF last call.

The part that I find confusing is that you mark receipt of the mail from the outside world twice (and in your example from different IP addresses). That is where I think you go wrong.

If your successive received headers looked something like this, then I think no one would complain:

Received: from mailout00.controlledmail.com (localhost [127.0.0.1])
by mailout00.controlledmail.com (Postfix) with ESMTP id C92C85CC066;
Tue, 28 Nov 2006 04:59:13 +0000 (UTC)
Received: from [75.195.212.193] (193.sub-75-195-212.myvzw.com [75.195.212.193])
(using SSLv3 with cipher RC4-MD5 (128/128 bits))
(No client certificate requested)
by mailout00.controlledmail.com (Postfix) with ESMTP id 36E675CC03F;
Tue, 28 Nov 2006 04:59:07 +0000 (UTC)

[David MacQuigg]

I did find one statement relating to header order:

RFC-2821, Sect 4.4 Trace Information
An Internet mail program MUST NOT change a Received: line that was
previously added to the message header. SMTP servers MUST prepend
Received lines to messages; they MUST NOT change the order of
existing lines or insert Received lines in any other location.

That still leaves open the question of whether we should follow strict chronological order within our block of headers, or establish some kind of "canonical form" with the Received line always at the bottom of the block. We certainly won't mess with any existing headers, even if they are out of order.

I don't understand what you mean by "mark reciept of the mail twice .. from different IP addresses". There is only one IP address in the three header lines I added - 64.191.210.175. In this example, authentication is done when we first get the request for a mail session: HELO this is pco-iraq.net, transmitting from 64.191.210.175 in the heart of sunny Baghdad . At this point no messages have been received. If pco-iraq.net is forged, we reject the entire session.

Receipt of the messages occurs *after* this authentication check. Each message is marked with the same X-Authent header, a different Received header for each message, and possibly other headers, if for example, the message is routed through a spam filter, or additional authentication checks (SPF, DKIM, etc.). If the sender is whitelisted (authenticated and reputable), the messages go straight to the recipients inbox. In that case all you will see is the X-Authent header, and a Received header. In this example, we had no ratings on pco-iraq.net, so it went to the spam filter, and the X-SpamScore shows it was clean.

[hadaso]

Originally posted by macquigg
... folks who can read email headers, but are not necessarily email experts.

I think no one ever devised a better characterization of us here!

I really don't have an opinion one way or the other, and the RFCs probably allow you to add headers in any order as long as they are prepended and one of them is the correct "Received header. However, I have a slight tendency to agree with the opinion of the "experts", and it seems to me to agree with the "chronological order".

When one sees a "Received" header, one assumes that everything below that is exactly what has been received from the sender listed in the "Received" header. In your example the topmost line below the "Received" header (the "X-Authent" header) was not part of the received message, but instead it has been prepended to the received message. So logically what I see here is that you:
1. make a test, and record the result (in a form that you can prepend to the email message)
2. you receive the message, and record the details is a standard way ("Received header")
3. you prepend the info from 1,
4. you prepend the info from 2.
Your logic says that 3 should happen before 4 because perhaps you are actually doing the prepending internally by appending the message received at stage 2 to the line created in stage 1, so it happens before you have the "Received header". My logic is that the way you are expected to do it is as part of stage 2 you finish it by immediately prepending the "Received" header, and only then you have finished the process of receipt and can apply other processes such as adding info about the authentication that justified the receipt. The fact that internally you append the message to the header and not prepend the header should not matter. The end result shouild be as if there was a complete message received and only then headers prepended. Another point of view: if you internally "prepend headers" by first puting them and then appending the received message (perhaps line by line as it is being received) then following this logic the first step in receiving an email message is putting a "Received" header recording the details of the transaction, and then appending the rest of the message. If you work this way then you have the header from stage 1, then the "Received" header from stage 2, and then the received message (unless the delivery fails and then everything including the two intial headers are dumped). Doing it this way probably means that the time stamp signifies the start of the "data" stage and not the completion of this stage of the SMTP session.

To sum it up, I think the "experts" point of view is based on the reasonable assumption that the string that is the message received by the DATA command is a single object, the result of the DATA command is a single object that is the previous object "wrapped" by adding a single "Received" header, and then this object is "wrapped" by adding addtional headers providing optional info, and this becomes the object that is the message reseived by the next server in the chain.

[Scott Kitterman]

Originally posted by macquigg I don't understand what you mean by "mark reciept of the mail twice .. from different IP addresses". There is only one IP address in the three header lines I added - 64.191.210.175. In this example, authentication is done when we first get the request for a mail session: HELO this is pco-iraq.net, transmitting from 64.191.210.175 in the heart of sunny Baghdad . At this point no messages have been received. If pco-iraq.net is forged, we reject the entire session.

I was reading it wrong...

I missed pavo.pco-iraq.net versus CANCER.pco-iraq.net.

X-SpamScore: -14.0
Received: from CANCER.pco-iraq.net (cancer.pco-iraq.net [64.191.210.175]) by
open-mail.org (8.13.1/8.13.1) with ESMTP id kAGBKtIf002229 for
<xxx@box67.com>; Thu, 16 Nov 2006 06:20:57 -0500
X-Authent: 64.191.210.175 pco-iraq.net PTR1 PASS ratings=(2,9,None) <=====
Received: from pavo.pco-iraq.net ([64.191.210.183]) by CANCER.pco-iraq.net
with Microsoft SMTPSVC(6.0.3790.1830); Thu, 16 Nov 2006 14:22:16 +0300
Received: from FORNAX.pco-iraq.net ([172.16.5.106]) by pavo.pco-iraq.net with
Microsoft SMTPSVC(6.0.3790.1830); Thu, 16 Nov 2006 14:20:52 +0300

The experts are right, but not because of any RFC. Where you put the header there is no way for a down stream processor to tell for SURE if it was added by CANCER.pco-iraq.net or by open-mail.org.

If you prepend the header above your received line, then if open-mail.org is trusted by the downstream processor it can rely on the fact that a trusted MTA added the header. Sorry I missed it the first time.

If you are using a Sendmail milter to do your processing, from what little I know about milters, you can control where it puts the header. I'm working on a similar prepend issue right now and with a Postfix policy service I have to make sure I only prepend once per message and not once per recipient when doing processing after rcpt to. I don't know if that's an issue for milters or not.

[David MacQuigg]

Excellent discussion! Thanks to both hadaso and Scott Kitterman for thoughtful comments. I am not convinced by the technical arguments (see below), but I am seeing a strong preferrence for making a Received line mark the border, regardless of actual chronological sequence. If Scott was confused by my unusual header order, others will be also.

Let's think in terms of what the user sees, and what we might do to amend RFC-2821. We are sacrificing a little simplicity for less deviation from what is expected by people who have seen many Received headers. We might think of a new "canonical form" for "Received blocks", with the Received line always at the bottom of the block.

Rule for users:
Find the topmost X-Authent: line, then the next Received: line below that. Draw a boundary line just below that Received: line. Everything above the boundary can be trusted. Anything below may be forged.

Rule if we follow "strict chronology":
Find the topmost X-Authent: line. That line and everything above it can be trusted. Anything below may be forged.

# proposed amendment to RFC-2821
existing text:
RFC-2821, Sect 4.4 Trace Information
An Internet mail program MUST NOT change a Received: line that was
previously added to the message header. SMTP servers MUST prepend
Received lines to messages; they MUST NOT change the order of
existing lines or insert Received lines in any other location.
add:
If multiple lines are to be prepended, they MUST be kept together
as a block, and there MUST be only one Received: line, which MUST
be the bottommost line in the block. The other prepended lines
MUST be above that, regardless of the chronological order in
which those lines were created.

Suggestions on either the rule or the amendment will be appreciated.

As for the technical arguments, I see some flaws:

Re hadaso's argument:
I would not put much significance on "internal order" of the prepended headers. In our milter, there is no internal order until they are actually created, when the message is being forwarded. Prior to that, the information is kept in variables with names like IP, ID, auth_method, auth_result. The "order of events" in my original post is not exactly what happens internally with regard to the creating of headers.

I'm not familiar with other MTA's, but there is a definite order of events in Sendmail (connect, helo, envfrom, envrcpt,
header, eoh, body, eom). I suppose other MTA's might have minor variations, like processing all incoming headers in one call rather than one line at a time.

Re Scott's argument:
I believe the trust requirements are the same regardless of where we place the X-Authent header. There should be one and only one such header, and any appearance of a second one is either a serious configuration error or an attempt at forgery. Authentication should be done at the incoming Border MTA, never by an MTA on the sender's side. The X-Authent header should always be present, even if it reports only an IP address and None for the sender's Identity.

I can imagine situations with multiple MTA's at the Border, some doing authentication, others not, all going to one downstream processor that looks for an authentication header, but doesn't distinguish between the different Border MTA's. Seems to me this is also in the category of serious configuration error. Maybe you could give an example of where this would be a setup we should support.

We need to think about the details of this example. How does the placement of the X-Authent header make any difference in what the downstream processor needs to do? That processor must have some list of trusted hosts, so it can find the bottommost trusted Received line. It must have some logic to detect a bogus Received line, even if that line lists a trusted host. Having found the correct Received line, it should be trivial to look for an authentication header in the expected position, above or below the Received line.

Sendmail is very flexible. We can do anything we want with the headers.

Again, I appreciate these comments, and I am leaning toward changing the header order, even without a compelling technical reason.

[Scott Kitterman]

Think in terms of a general SpamAssassin plug-in that takes advantage of your X-Authent header. It isn't going to know whether or not to expect the header or not (unless specifically configured).

Using X-Authent: as the border it would have to be:

a. Specially configured to know where to expect it.

b. Dependent on you to 100% strip the old header and replace it with the new one.

No, keeping in mind that not all mail an MDA might get would necessarily come through your service and this starts to get complex.

All that complexity is avoided by putting your header above Received:. Why would you possibly not want to do it that way?

[David MacQuigg]

Just trying to keep the rule simple for people who are new to reading headers. But we can make an exception to "strict chronology" for the Received header, putting it at the bottom of our block. That will satisfy what appears to be a strong preference among experienced "header readers", and will also avoid the problem you point out - a spam filter that has this expectation built in. I don't know of any that work that way now, but it is quite likely as these filters get more sophisticated in the never-ending "arms-race" with spammers. We want to minimize any conflict between our authentication filter and whatever other tools might be in a recipient's setup.

I have changed the header order so that X-Authent is now just above Received. Here is our header block in a test message sent via an A-rated forwarder:

X-Authent: 72.81.252.18 controlledmail.com QR1 PASS ratings=(9,9,110)
Received: from mailout02.controlledmail.com (mailout02.controlledmail.com
... [72.81.252.18]) by open-mail.org (8.13.1/8.13.1) with ESMTP id kAUCEMvD003769
... for <xxx@box67.com>; Thu, 30 Nov 2006 07:14:27 -0500

This is a simple case with no extra heades from additional authentications or spam filtering. See below for a more complex example.

(9,9,110) is (IDstatus, auth_status, IDrating) These numbers mean controlledmail.com is a registered ID, this ID authorizes the transmitter at 72.81.252.18, and this ID has a rating 110, a number we assign temporarily to A-rated senders. Once we have sufficient statistics on messages from this sender, the rating will be the expected number of good messages per spam.

Now for the more complex example. Here is our proposed Canonical Header Order:

X-spam-score-headers:
X-other-body-authentication-headers:
X-other-return-address-authentication-headers:
Received-SPF:
X-other-session-authentication-headers:
X-Authent:
Received:

Example:

X-SpamScore: -14.0
X-Originating-IP: [12.144.145.210]
Authentication-Results: mta150.mail.re3.yahoo.com from=scottrade.com; domainkeys=neutral (no sig)
X-Guessed-SPF: neutral
Received-SPF: none (mail.bmsi.com: 72.81.252.18 is neither permitted nor
... denied by domain of box67.com) client-ip=72.81.252.18;
... envelope-from=xxx@box67.com; helo=mailout02.controlledmail.com;
... receiver=mail.bmsi.com; identity=mailfrom;
X-Authent: 64.191.210.175 pco-iraq.net PTR1 PASS ratings=(2,9,None)
Received: from CANCER.pco-iraq.net (cancer.pco-iraq.net [64.191.210.175]) by
... open-mail.org (8.13.1/8.13.1) with ESMTP id kAGBKtIf002229 for
... <xxx@box67.com>; Thu, 16 Nov 2006 06:20:57 -0500

Comments or suggestions are welcome. If anyone wants to see what their headers will look like, send an email to macquigg at box67 dot com, and I will send it back with the headers preserved.

-- Dave

[Scott Kitterman]

I think that will work. The approach I've suggested is the way that SpamAssassin sorts out which headers are trusted and which ones aren't.

[hadaso]

I noticed today that FastMail puts the X-linkname header that records the POP=link an email was fetched from below the corresponding Received header that records the transfer. So this is one example of an email provider that inserts headers before a Received header.

[David MacQuigg]

This might make sense if they are following a "strictly chronological" rule. But I'm a little surprised that an MTA is using POP to transfer mail. Usually SMTP is the protocol between MTA's, and POP is used by the recipient to fetch mail from the destination MTA.
Could you show us a sample of these headers, and a description of what is happening at each step?

[hadaso]

Originally posted by David MacQuigg
This might make sense if they are following a "strictly chronological" rule. ... Could you show us a sample of these headers, and a description of what is happening at each step?

Here's an example (with real email addresses slightly munged). One thing worth noting is that X-linkname is a header prepended by MessagingEngine.com while X-Message-Status was added by Hotmail. About the only way to know this is to see many such headers and understand what each service is doing. What is happenning is that a user (myself) set a FastMail (a.k.a. MessagingEngine) account to periodically pull new email from a Hotmail account (so actually the header lies when it says the protocol is POP3). The fetched email is then injected into the incoming mail stream and delivered (using LMTP) to the IMAP server (which then might redirect it or a copy elsewhere depending on the user's filtering rules).

Return-Path: <books@honestperkpals.con>
Received: from imap1.internal (store1m.internal [10.202.10.1])
         by store1m.internal (Cyrus v2.3.7-fmsvn9682) with LMTPA;
         Mon, 20 Nov 2006 01:07:40 -0500
X-Sieve: CMU Sieve 2.3
X-Spam-score: 1.6
X-Spam-hits: HTML_MESSAGE, MIME_HTML_MOSTLY, MPART_ALT_DIFF, SARE_HEAD_HDR_XSIDPRA,
  X_MESSAGE_INFO
X-Resolved-to: hadaso@domain.tld
X-Mail-from: books@honestperkpals.con
Received: from services.msn.com [64.4.60.7] by messagingengine.com
  with POP3 for hadaso@domain.tld; Mon, 20 Nov 2006 01:07:38 -0500
X-LinkName: hotmail
X-Message-Status: sF:0
X-SID-PRA: University of Phoenix <books@honestperkpals.con>
X-Message-Info: LsUYwwHHNt2RlBaBm/dUNFMqf6y+DHODEmxfFPFGtm0=
Received: from honestperkpals.com ([208.100.6.244]) by bay0-mc1-f16.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2444);
         Sun, 19 Nov 2006 21:00:15 -0800
Received: by honestperkpals.com id hc4r8809bo4s; Sun, 19 Nov 2006 23:00:18 -0600 (envelope-from <books@honestperkpals.con>)
Reply-To: "University of Phoenix" <books@honestperkpals.con>
From: "University of Phoenix" <books@honestperkpals.con>
To: <hadaso@hotmail.con>
Subject: Earn your Degree

Originally posted by David MacQuigg
But I'm a little surprised that an MTA is using POP to transfer mail. Usually SMTP is the protocol between MTA's, and POP is used by the recipient to fetch mail from the destination MTA.

In this case Hotmail is assuming that they are the destination MTA but they are not. Anyway, I have not really checked the facts, but I think POP was designed to allow the delivery of email to email systems that are disconnected from the Network most of the time. That is, the email for the destination is routed to an accessible host, and then the MTA for final destination is setup to pull the email for its users from the server that holds them and delivers them to the users' mailboxes. In the early days mail was delivered to the computer where the user account was hosted and the user logged in and read the mail on that computer (such as by using the unix "mail" command). A system that was connected most of the time didn't need POP since mail could be delivered directly to the recipient's machine.

[DrStrabismus]

Actually it isn't pop3 anyway. Fastmail retreives from Hotmail using the HTTP/DAV-based protocol that Outlook Express uses. It's just that the software doesn't distingish between dav and pop when it writes the dummy received header.