C/R - can we do it right

David MacQuigg · 28 Nov 2007, 12:39 AM

Please let's avoid all religion and keep this thread short. Let's focus on what can be done to get the benefits of a Challenge/Response system, while avoiding the "bounce spam" problem. I apologize if this has already been discussed. I did not have time to read the entire previous discussion.

Here is a suggestion that might work: Deliver the challenge in an SMTP REJECT message: "Sorry, we do not recognize <example.com> as a reputable sender. We will send this message through our spam filters, but we cannot guarantee delivery. To check the status of your message, go to http://<quarantine-website>/<hash code>.

At this website, legitimate users can add a short header message - "Hi John, please put me on your whitelist.", and release the message from the quarantine. Or they can see that the recipient has already picked it out of the quarantine.

Key features of this method are:
1) No bounce spam (challenges to unauthenticated addresses).
2) No new protocols or changes on the sender side. It will work right now, with the Internet "as is".
3) Pressure on senders from their own customers. To ensure reliable delivery, they must become a reputable sender.

hadaso · 28 Nov 2007, 06:48 AM

The rejection message would have to be in response to end of data if the message is to be kept and quarantined.

The only problem I can see is if some mail clients (or MTAs creating bounce messages after failed delivery) don't display the text or the SMTP reject message properly or in a way that the sender would see it.

Also I don't know about the website: there can be other ways, such as sending an empty email to a special address. If this kind of thing becomes widely used with one dominant implementation then the responses can be automated by spammers. So diversity helps. On the other hand there is a similar system where the "release from quarantine" is automatic - greylisting - and spammers haven't bothered to overcome it.

I think BlueBottle people said somewhere on the forums that they experimented with this kind of thing.

xmailer · 28 Nov 2007, 07:25 AM

Quote:

Originally Posted by David MacQuigg

Please let's avoid all religion...

Seriously.

While I probably don't have much technical insight to offer, to summarily dismiss out of hand, on questionable "moral" grounds, a method of spam control with a proven high degree of efficacy, merely because it may not be "perfect" in every respect (which AFAIK NO method devised to date is) might seem an obvious case of throwing out the proverbial baby with the bathwater.

hadaso · 28 Nov 2007, 04:05 PM

Quote:

Originally Posted by David MacQuigg

Please let's avoid all religion ...

That's why I haven't provided a guess as to which email client/server software might be hiding the text accompanying SMTP reject messages from users...

David MacQuigg · 29 Nov 2007, 12:11 AM

> The rejection message would have to be in response to end of data if the message is to be kept and quarantined.

Correct. Challenges would be sent only for messages that pass our checks of IP Address, Heloname, Return Address, and Recipient, appear to have valid data, and perhaps pass a tempfail test as well (greylist). In addition, depending on recipient options, there may be a "slice off the top" of the quarantine, which the recipient will look at anyway, so no challenges are necessary. Only a small fraction of the mailflow will get challenges.

> The only problem I can see is if some mail clients (or MTAs creating bounce messages after failed delivery) don't display the text or the SMTP reject message properly or in a way that the sender would see it.

What the message author sees is not a "bounce message" from a distant MTA, but rather a "failure notice" from his own ESP. There is no replay of any content of the original message. In my mail client (Eudora) these notices appear at the bottom of the screen. "Can't send to 'test@example.com'. The server gave this reason: '550 5.0.0 ... text of reject message ... '."

There might be some blockage of these notices at large ESPs, where they want to minimize customer support. I haven't seen it at Yahoo, and they are very much in the "minimum support" business model. i.e. I do get the full text of any 550 REJECT which comes back. Has anyone seen such a failure at their service?

> Also I don't know about the website: there can be other ways, such as sending an empty email to a special address. If this kind of thing becomes widely used with one dominant implementation then the responses can be automated by spammers. So diversity helps. On the other hand there is a similar system where the "release from quarantine" is automatic - greylisting - and spammers haven't bothered to overcome it.

Good suggestion on the alternative to a website. We could have a REJECT message like:

"Sorry, we do not recognize <example.com> as a reputable sender. We will send this message through our spam filters, but we cannot guarantee delivery. To ensure delivery, send an empty message to H8zi2p@y5ngBi.net"

This would be easier than a website, but it seems like it will be *more* vulnerable to automated responses. With a website, you can have the sender do a little work solving a puzzle, proving they are human, etc.

I don't think spammers are going to make any effort to break our system, even if half the recipients on the Internet are using it. There are plenty of recipients out there with no effective protection, and the economics of the business require large distribution with little effort.

> I think BlueBottle people said somewhere on the forums that they experimented with this kind of thing.

Hearing the experience of others would be very helpful.

Scott Kitterman · 29 Nov 2007, 06:52 AM

This was discussed (and suggested) in the other thread as a resonable alternative to inconveniencing random third parties on the internet to keep your inbox clean and rejected.

Another alternative would be to only generate challenges on mail that passes some sort of email authentication/authorization (that's another religious war) test. The trick is that the test would have to cover the address you were sending the challenge to. As an example, I've no problem at all with sending challenges to mail that gets an SPF Pass as that mail is from a source authorized by that domain.

David MacQuigg · 29 Nov 2007, 10:57 AM

Even with an SPF Pass, there is still a small chance that the return address is forged. The SMTP Reject avoids even this small risk.

I have heard that with some senders, the SMTP reject message may not be returned to the message author. I would sure like to know if this is a significant problem, but nobody seems to know of any examples.

hadaso · 29 Nov 2007, 04:45 PM

Quote:

Originally Posted by David MacQuigg

...
I have heard that with some senders, the SMTP reject message may not be returned to the message author. I would sure like to know if this is a significant problem, but nobody seems to know of any examples.

When mail is forwarded and the final MTA rejects the forwarded mail, then the forwarding MTA creatred a rejection message and sends it to the SMTP MAIL FROM address, which on most spam would be the forged one (or the spammer's real gmail address, as I see happenning quite often lately...)

For instance: lots of people set their ISP mail to forward to Gmail. Then Gmail SMTP rejects any email with exe or zip files (reason provided: "illegal attachment" or something like that). The result is that ISPs produce backscatter whenever some virus tries to spread to an address that forwards to gmail. I've seen this happen quite often. It also happen quite often when email is forwarded to an account that's over quota.

David MacQuigg · 30 Nov 2007, 12:47 AM

Here is what I understand is happening in this situation:
. Spammer --> Zombie --> ~~~ --> Receiver --> MDA
The MDA (Mail Distribution Agent) sends an SMTP REJECT to the Receiver, and the Receiver generates a bounce message to a forged Return Address.
If the Receiver is our Border Patrol, there will be no bounce messages, only SMTP Reject messages. If we accept it, we deliver it, even if only to a spam bucket. In the rare situation where we get a Reject from our own recipient's MDA, we will immediately suspend the account, reject any further messages to that recipient, and attempt to contact the recipient via an alternate method.
What I'm looking for is any examples of Reject messages from the Receiver not being handled properly by a legitimate sender.

Scott Kitterman · 30 Nov 2007, 03:13 AM

Quote:

Originally Posted by David MacQuigg

Even with an SPF Pass, there is still a small chance that the return address is forged. The SMTP Reject avoids even this small risk.

What chance is that?

David MacQuigg · 30 Nov 2007, 05:57 PM

I don't have any numbers, but there is a small chance that an SPF record could include too many IP addresses, and thereby allow a spammer with an "authorized" IP address to forge any return address in that domain. A C/R system based on SMTP rejects avoids even that small chance of error.

Nonetheless, I agree with your statement that it is OK to send a challenge to an address that gets an SPF Pass. If this causes a problem, it should be fixed on the sender's side.

hadaso · 1 Dec 2007, 06:00 AM

SPF works on the level of domains, so spammers can forge any address within the domains of their ISP (or the ISP hosting the zombie PC they control if they have the zombie send through the ISP's servers).

David MacQuigg · 1 Dec 2007, 11:21 PM

I don't know what is meant by an address "within a domain". An "ISP" can provide various services, including network access, email transmission, and domain names. Let's use terms that are very specific. I propose "network owner", "station operator", and "domain-name owner". These can be the same individual or organization, or they can all be separate and unrelated.

A network owner is the person or organization to whom IP addresses are allocated. Network owners provide the "wire" on which our data is carried, and the routers which connect us to the rest of the Internet.

A station operator is the person or organization that owns the transmitter from which data on the open Internet originates. This may include zombies, whose owners have no idea what is happening.

A domain owner is the person or organization that registered the domain name. They may have no connection to network or station ownership.

I hope this will clarify our discussions.

Scott Kitterman · 2 Dec 2007, 04:16 AM

Right, but that's the domain owners responsibility to prevent that type of abuse. It's directly addressed in the security section of the SPF RFC. See http://www.openspf.org/RFC_4408#cross-user-forgery. If MTA operators (station operator in your terms) are not properly controlling submission of mail to their servers, there's nothing anyone else can really do.

For SPF, domain owners are involved because they control their domain's DNS entries.

hadaso · 2 Dec 2007, 04:48 AM

Quote:

Originally Posted by David MacQuigg

A station operator is the person or organization that owns the transmitter from which data on the open Internet originates. This may include zombies, whose owners have no idea what is happening..

THis one is not 100% clear to me. Say a zombie PC controlled by a 3rd party (a botnet herder) is set up by the 3rd party to use the email client software installed By the owner of the PC (and the credentials stored by this application if needed) to connect to a machine offered by the network owner to users of the network as an SMTP relay (the mode of operation most people use sending through the ISP's outgoing mail servers) then who is the station operator? Who is the transmitter? I see two stations: one is the zombie PC and the other is the ISP's relay, that is open to transmitters within the network (or open to those who can authenticate themselves).

An ISP is usually the owner of the network, the owner of some stations that transmit email outside of their network (email relays that are usually refered to as "outgoing mail servers") and owner of some domains. They usually assign station owners within their network some email addresses in some of the domains they own, allow them to relay mail through stations they own and setup SPF to say these addresses are allowed to transmit from those stations. Other domain owners the have station within the ISP's network use their stations to relay their email through the stations provided by the ISP (the network owner) and they also set up their SPF records to allow these stations (the ISPs relays) to send mail for their domain. So a spammer who has control of some stations within a network and can use them to gain use of the stations provided by the network owner (the ISP) to all network users to relay email through can forge email addresses in several domains that are owned by users of these relays including those domains used by the ISP to provide email addresses to subscribers and including domains who have SPF records that allow these transmitters because their owners use these shared transmitters.

I'm not sure it makes things clearer, but anyway the point is that most email users use shared transmitters and spammers can have access to these transmitters and can use them with all any domain owned by a user of these shared transmitters and pass SPF. Of course an ISP can police the use of their transmitters, but it's not clear that they have real interest in doing it. If they are big enough their users will not be blocked even if their transmitters send out lots of junk.

28 Nov 2007, 12:39 AM	#1
David MacQuigg Member Join Date: Aug 2006 Location: Tucson AZ Posts: 66 Representative of: Open-Mail.org	C/R - can we do it right Please let's avoid all religion and keep this thread short. Let's focus on what can be done to get the benefits of a Challenge/Response system, while avoiding the "bounce spam" problem. I apologize if this has already been discussed. I did not have time to read the entire previous discussion. Here is a suggestion that might work: Deliver the challenge in an SMTP REJECT message: "Sorry, we do not recognize <example.com> as a reputable sender. We will send this message through our spam filters, but we cannot guarantee delivery. To check the status of your message, go to http://<quarantine-website>/<hash code>. At this website, legitimate users can add a short header message - "Hi John, please put me on your whitelist.", and release the message from the quarantine. Or they can see that the recipient has already picked it out of the quarantine. Key features of this method are: 1) No bounce spam (challenges to unauthenticated addresses). 2) No new protocols or changes on the sender side. It will work right now, with the Internet "as is". 3) Pressure on senders from their own customers. To ensure reliable delivery, they must become a reputable sender.

28 Nov 2007, 06:48 AM	#2
hadaso Intergalactic Postmaster Join Date: Oct 2002 Location: Holon, Israel. Posts: 5,117	The rejection message would have to be in response to end of data if the message is to be kept and quarantined. The only problem I can see is if some mail clients (or MTAs creating bounce messages after failed delivery) don't display the text or the SMTP reject message properly or in a way that the sender would see it. Also I don't know about the website: there can be other ways, such as sending an empty email to a special address. If this kind of thing becomes widely used with one dominant implementation then the responses can be automated by spammers. So diversity helps. On the other hand there is a similar system where the "release from quarantine" is automatic - greylisting - and spammers haven't bothered to overcome it. I think BlueBottle people said somewhere on the forums that they experimented with this kind of thing.

29 Nov 2007, 12:11 AM	#5
David MacQuigg Member Join Date: Aug 2006 Location: Tucson AZ Posts: 66 Representative of: Open-Mail.org	> The rejection message would have to be in response to end of data if the message is to be kept and quarantined. Correct. Challenges would be sent only for messages that pass our checks of IP Address, Heloname, Return Address, and Recipient, appear to have valid data, and perhaps pass a tempfail test as well (greylist). In addition, depending on recipient options, there may be a "slice off the top" of the quarantine, which the recipient will look at anyway, so no challenges are necessary. Only a small fraction of the mailflow will get challenges. > The only problem I can see is if some mail clients (or MTAs creating bounce messages after failed delivery) don't display the text or the SMTP reject message properly or in a way that the sender would see it. What the message author sees is not a "bounce message" from a distant MTA, but rather a "failure notice" from his own ESP. There is no replay of any content of the original message. In my mail client (Eudora) these notices appear at the bottom of the screen. "Can't send to 'test@example.com'. The server gave this reason: '550 5.0.0 ... text of reject message ... '." There might be some blockage of these notices at large ESPs, where they want to minimize customer support. I haven't seen it at Yahoo, and they are very much in the "minimum support" business model. i.e. I do get the full text of any 550 REJECT which comes back. Has anyone seen such a failure at their service? > Also I don't know about the website: there can be other ways, such as sending an empty email to a special address. If this kind of thing becomes widely used with one dominant implementation then the responses can be automated by spammers. So diversity helps. On the other hand there is a similar system where the "release from quarantine" is automatic - greylisting - and spammers haven't bothered to overcome it. Good suggestion on the alternative to a website. We could have a REJECT message like: "Sorry, we do not recognize <example.com> as a reputable sender. We will send this message through our spam filters, but we cannot guarantee delivery. To ensure delivery, send an empty message to H8zi2p@y5ngBi.net" This would be easier than a website, but it seems like it will be more vulnerable to automated responses. With a website, you can have the sender do a little work solving a puzzle, proving they are human, etc. I don't think spammers are going to make any effort to break our system, even if half the recipients on the Internet are using it. There are plenty of recipients out there with no effective protection, and the economics of the business require large distribution with little effort. > I think BlueBottle people said somewhere on the forums that they experimented with this kind of thing. Hearing the experience of others would be very helpful.

29 Nov 2007, 06:52 AM	#6
Scott Kitterman Essential Contributor Join Date: Sep 2006 Location: Ellicott City, MD, USA Posts: 206 Representative of: ControlledMail.com	This was discussed (and suggested) in the other thread as a resonable alternative to inconveniencing random third parties on the internet to keep your inbox clean and rejected. Another alternative would be to only generate challenges on mail that passes some sort of email authentication/authorization (that's another religious war) test. The trick is that the test would have to cover the address you were sending the challenge to. As an example, I've no problem at all with sending challenges to mail that gets an SPF Pass as that mail is from a source authorized by that domain.

29 Nov 2007, 10:57 AM	#7
David MacQuigg Member Join Date: Aug 2006 Location: Tucson AZ Posts: 66 Representative of: Open-Mail.org	Even with an SPF Pass, there is still a small chance that the return address is forged. The SMTP Reject avoids even this small risk. I have heard that with some senders, the SMTP reject message may not be returned to the message author. I would sure like to know if this is a significant problem, but nobody seems to know of any examples.

30 Nov 2007, 12:47 AM	#9
David MacQuigg Member Join Date: Aug 2006 Location: Tucson AZ Posts: 66 Representative of: Open-Mail.org	Here is what I understand is happening in this situation: . Spammer --> Zombie --> ~~~ --> Receiver --> MDA The MDA (Mail Distribution Agent) sends an SMTP REJECT to the Receiver, and the Receiver generates a bounce message to a forged Return Address. If the Receiver is our Border Patrol, there will be no bounce messages, only SMTP Reject messages. If we accept it, we deliver it, even if only to a spam bucket. In the rare situation where we get a Reject from our own recipient's MDA, we will immediately suspend the account, reject any further messages to that recipient, and attempt to contact the recipient via an alternate method. What I'm looking for is any examples of Reject messages from the Receiver not being handled properly by a legitimate sender.

30 Nov 2007, 05:57 PM	#11
David MacQuigg Member Join Date: Aug 2006 Location: Tucson AZ Posts: 66 Representative of: Open-Mail.org	I don't have any numbers, but there is a small chance that an SPF record could include too many IP addresses, and thereby allow a spammer with an "authorized" IP address to forge any return address in that domain. A C/R system based on SMTP rejects avoids even that small chance of error. Nonetheless, I agree with your statement that it is OK to send a challenge to an address that gets an SPF Pass. If this causes a problem, it should be fixed on the sender's side.

1 Dec 2007, 06:00 AM	#12
hadaso Intergalactic Postmaster Join Date: Oct 2002 Location: Holon, Israel. Posts: 5,117	SPF works on the level of domains, so spammers can forge any address within the domains of their ISP (or the ISP hosting the zombie PC they control if they have the zombie send through the ISP's servers).

1 Dec 2007, 11:21 PM	#13
David MacQuigg Member Join Date: Aug 2006 Location: Tucson AZ Posts: 66 Representative of: Open-Mail.org	I don't know what is meant by an address "within a domain". An "ISP" can provide various services, including network access, email transmission, and domain names. Let's use terms that are very specific. I propose "network owner", "station operator", and "domain-name owner". These can be the same individual or organization, or they can all be separate and unrelated. A network owner is the person or organization to whom IP addresses are allocated. Network owners provide the "wire" on which our data is carried, and the routers which connect us to the rest of the Internet. A station operator is the person or organization that owns the transmitter from which data on the open Internet originates. This may include zombies, whose owners have no idea what is happening. A domain owner is the person or organization that registered the domain name. They may have no connection to network or station ownership. I hope this will clarify our discussions.

2 Dec 2007, 04:16 AM	#14
Scott Kitterman Essential Contributor Join Date: Sep 2006 Location: Ellicott City, MD, USA Posts: 206 Representative of: ControlledMail.com	Right, but that's the domain owners responsibility to prevent that type of abuse. It's directly addressed in the security section of the SPF RFC. See http://www.openspf.org/RFC_4408#cross-user-forgery. If MTA operators (station operator in your terms) are not properly controlling submission of mail to their servers, there's nothing anyone else can really do. For SPF, domain owners are involved because they control their domain's DNS entries.