Open-Source Reputation Service

[David MacQuigg]

Our Registry of Public Email Senders has passed its initial tests, and our domain-rating strategy is working better than expected! We have some preliminary reputation data, based on mail received by a few beta testers. See http://open-mail.org/WhatsNew There is almost no spam coming from the authorized transmitters of legitimate senders. I've seen only 3% spam in my inbox over the last few months, and almost all of this was due to a few senders, who took care of the problem quickly.

The next step is to build up the database behind our Registry to include many more sender domains, and more accurate statistical averages. If anyone has a small domain with maybe a few dozen recipients, and you need a "spam appliance" to sort incoming mail, let me know, and I can set you up with a server running our Border Patrol MTA (TM).

The database is also open to automated queries, if you wish to use your own border MTA. Just add the domain name in front of 'xmid.net'. e.g.
$ dig txt open-mail.org.xmid.net +short
"svc=X1:A mth=SPF+1,CSV+2 ip4=207.210.221.26"

The response packet says this domain has an A rating from service X1, for authentication it offers SPF and CSV, and there is only one address authorized to say HELO.

The plan is to have Registry operations supported by voluntary contributions from the largest receivers, and keep it always free for senders and small receivers.

[indyamail]

Can you explain how this would work exactly based on running our own MTA but adding the records (TXT) for your service. How many other email services world wide accept your system to receive or send emails - that way we know which sites would be best beneficial to this. Adding too many records will slow down dns checking and access, so its best to know how this will help me (since our email service's and scripts we offer to clients) decide how this will function and help users who dont use your hosting / antispam solutions directly and use their own MTAs for all emails but may send to one of your clients (who we don't want to have us blacklisted or spam flagged by)

Regards

[David MacQuigg]

Hello Indyamail,

As for integration of Registry services into other MTAs, we started working on a Framework package, which would provide core routines to work in any MTA, and do such things as parsing Registry records, but had to put that work on hold to spend full time on the Border Patrol package, which uses Sendmail as the MTA platform. If you are interested in working on this, I'll be glad to help. Postfix would be my next choice as an MTA platform.

A better alternative to get something running quickly is just treat the border MTA as a separate "appliance", and use it as a front end to your favorite MTA. You don't have to learn Sendmail for that purpose.

We are just getting started in offering Registry services. I think there may be some misunderstanding about the need for worldwide acceptance before there is any benefit. That has been the insurmountable hurdle for each of the authentication methods, the so-called "chicken-and-egg" problem. We are getting around that problem by not requiring senders to use any specific method. We accept whatever the sender has to offer (SPF, SID, CSV, PTR, DKIM, etc.) A sender that offers *no* authentication of any kind is probably not legitimate. Even in that situation, however, we provide a "default" Registry record based on the sender's entire IP allocation from a Regional Registry.

These default records are a key element of our design strategy. We believe they will motivate senders to get off their butts and publish accurate lists of their transmitter addresses. If they do nothing, they are responsible for all the spam zombies in their netrange. This will lower their reputation, causing some recipients to reject their mail, and motivating them to publish the addresses of those transmitters. Once they do that, we can reject the zombies, and accept mail only from their authorized transmitters. This will provide an immediate improvement in their reputation, and better delivery of their mail.

I think there may be another misunderstanding regarding "blacklisting" by our receivers. What our receivers do is essentially offer a "bypass" to the usual IP blacklists and statistical filters. See the diagram at http://open-mail.org/BorderPatrol.html. Reputable senders are "whitelisted", and the rest go through the same processing that they would have without our service. We offer legitimate senders a reward. We don't attempt to punish the ones who are not yet in our Registry.

For more on how the Registry works see http://open-mail.org/

-- Dave

[DavidJ]

Originally Posted by David MacQuigg

A sender that offers *no* authentication of any kind is probably not legitimate.

Does everyone else around here agree with this?

I happen not to use the email address assigned by my ISP, but I believe that if I did, I would be sending without any authentication of any kind. And it's the largest ISP in an admittedly tiny country. I also believe that many private, non-commercial owners of their own domains, including many of the members of this community, use no authentication at all. I suspect that when I use the email addresses associated with two of my Web-sites I am also using completely unauthenticated email.

My understanding has always been that authentication should be used ONLY as authentication, to make sure the purported sender really is the sender, BEFORE using some OTHER reputation-based (or other) system for mail filtering. The attitude that the lack of authentication is in itself considered suspicious bothers me. It is going to turn email from a world-changing technology for everyone, including ordinary nobodies, into a medium suited only for geeks who can set up their own transmission and authentication systems, or for companies with IT departments.

Another wasted could-have-been, and another victory for the spammers. I think that I just changed from an interested supporter of email authentication into someone who feels that when I transmit with SPF, as I normally do, it's because I'm being coerced. But maybe that's the whole point: I'm just a private individual, and it the world of email technologies neither my feelings nor my needs are of any significance. Google, Yahoo, and Microsoft will decide among themselves, and if I lose the use of a technology, it's my tough luck.

Just my $0.02.

Thanks.
D.

[David MacQuigg]

David,

I share your concern that email is increasingly dominated by a few large ESPs. That is one of the main reasons I am pursuing this open-source project, to allow the little guys to have access to a reputation system as good or better than what Google, Yahoo, and MSN do for themselves. Anyone who operates a transmitter for a small domain, knows the difficulty of dealing with these big companies. They mark all your mail as spam, even though you have never been on any blacklists. To get on their whitelists, you have to jump through a different set of hoops for each. Yahoo, for example, had me answer a bunch of questions about how to operate a mailing list. When I tried to tell them I was not operating a mailing list, it became clear I was talking with a robot! Same with Google. Send them a verified spam report, and you will get an automated denial "somebody must be forging our name".

The big companies like the status quo. It is much like the early days of email, when you had to have a separate account with each of the big ESPs, because they wouldn't even exchange email with each other, let alone the little guys! Like those early days, the big companies now claim the solution to the problem is very difficult, and they are working very hard at it. The real solution will also be like those early days - lots of small ESPs getting together, sharing a common method, and building a community large enough that the big guys will have to accept it, and eventually join it.

I'm a bit puzzled by your concern over the difficulty of offering authentication. I understand the problems with SPF, and that it is not appropriate in all situations. It sounds like the situation you describe, however, is not one of those few. Whatever your reason for not using SPF, you can still offer authentication using another method, or by just listing your authorized transmitters in our Registry. Perhaps we should work out the details off-line, and get back to this thread after we agree on a solution.

I appreciate your $0.02, even if it is pessimistic. It tells me I need to work harder at communication, perhaps a re-organization of the information at http://open-mail.org. We are not, for example, rejecting mail simply because the sender is unauthenticated. Our rejects are based entirely on the policies of either the sender or the individual recipient. Senders tell us to reject mail from any transmitter forging their HELO name. Recipients tell us to reject mail if the sender's reputation is below a certain threshold *AND* the message gets a low score in our spam filter.

We are doing everything we can to make our system easy for small domains. Your feedback will be appreciated.

-- Dave

[DavidJ]

Originally Posted by David MacQuigg

I'm a bit puzzled by your concern over the difficulty of offering authentication. I understand the problems with SPF, and that it is not appropriate in all situations. It sounds like the situation you describe, however, is not one of those few. Whatever your reason for not using SPF, you can still offer authentication using another method, or by just listing your authorized transmitters in our Registry.

I do use SPF with my main email domain, and with the two other domains I own which make it technically convenient. The reason I don't use it with my other two domains, though, is a very simple one, and probably a very common one, which is why it's bothering me: I don't have complete control over their DNS files, for perfectly legitimate reasons.

To take the most common reason: GoDaddy, the largest domain registrar, at least for small users, allows their vaunted "Total DNS" service only for domains which use their own DNS servers, i.e., domains which are associated with sites or email accounts hosted with them, or parked domains. I realize that I could use a third party DNS service just for the purpose of adding SPF, but that would be a pain in the [deleted]. In other words, a person who uses GoDaddy as his registrar and DNS provider, but not his host for any other services, has no way of using SPF. And there are a lot of us, because this is a common situation.

I also have two Web-sites, associated with the corresponding email addresses, hosted by a small hosting company which does not offer direct DNS control by control panel, so that I cannot easily add an SPF string. I realize that there are many solutions to this problem, such as, once again, using a third-party DNS service, or just asking the hosting company by email to add a TXT record with my string, but this is also a pain, especially when one considers that every time I change ESPs I'll have to change that record as well. And, although my hosting company is very cooperative, there are others who would just refuse, because they don't want to be bothered with manual maintenance.

I also suspect that there are many home users of small ESPs, usually their local ISP, who know absolutely nothing about authentication, and therefore don't even know that their small rural ISP provides no authentication method. Does that make all of their mail spam? Should they be punished for wanting to use email with no technical knowledge? That's what I meant by saying that if so, email is only for geeks and large corporations.

Originally Posted by David MacQuigg

you can still offer authentication using another method

What would you suggest? I was looking into the possibility of setting up an MTA on my desktop machine just so that I could use DKIM with it, but
a) that will also be a technical pain in the [deleted],
b) I can't find any free MTAs which support DKIM, and
c) if my mail originates from a machine with no reverse DNS entry, I'm going to have more of a deliverability problem than I had before.

I have no objection to continuing this discussion here, especially since others may have some ideas to contribute, but I'll try to PM you an email address which can use for me, in case you need it for some reason.

Thanks.
David.

[David MacQuigg]

Let's start a new thread "Outgoing Mail for Small Domains", and keep this one for discussion of reputation services.

[miley]

Isn't this just a CSV variant?
http://www.bbiw.net/CSV/draft-ietf-m...-intro-01.html
http://wiki.fastmail.fm/index.php/Ce...ver_Validation

I've long thought email will boil down to:
helo -- authenticate by CSV
Mail From: -- authenticate null senders by BATV (SPF is too broken to use in wide scale)
Rcpt to: -- hard coded authentication (ie, no open relays)
Data: DKIM authentication

[DavidJ]

Originally Posted by miley

Isn't this just a CSV variant?

I'm afraid that I don't see what they have to do with each other:
As far as I can tell, CSV is a receiver-side "validation" service, meant to be used as PART of a filtering system, I suppose.
David's "Registry of Internet Transmitters", if that's what we'll call it for the moment, is an authentication system, meant mainly to be used BEFORE a reputation system is called by a service such as CSV, or as part of an authentication system. As I understand it, it is intended to help receivers (who may also be transmitters) avoid having to check several different authentication systems. Instead of checking for an SPF record, a DKIM signature, Sender-ID, and so on, the receiver will have to check only one source, which already stores information based on all of the popular authentication systems. Once the receiver believes that it has authenticated the mail, it can use the information in a filter which uses sender reputation as a criterion, or with a local list-based filter, or whatever.
http://www.bbiw.net/CSV/draft-ietf-m...-intro-01.html
http://wiki.fastmail.fm/index.php/Ce...ver_Validation

Originally Posted by miley

SPF is too broken to use in wide scale

I wouldn't say that SPF is broken. It is used on both the sending and receiving sides by vast numbers of people and by large organizations. (I once heard of a company called Google....) SPF has it's faults, but so does every other authentication sytem. I also have my faults, but I don't consider myself "broken". Or maybe I am....

Thanks.
David.

[miley]

Originally Posted by DavidJ

I'm afraid that I don't see what they have to do with each other:
As far as I can tell, CSV is a receiver-side "validation" service, meant to be used as PART of a filtering system, I suppose.
David's "Registry of Internet Transmitters", if that's what we'll call it for the moment, is an authentication system, meant mainly to be used BEFORE a reputation system is called by a service such as CSV, or as part of an authentication system.

Actually the core of CSV is authentication for the HELO.

As I understand it, it is intended to help receivers (who may also be transmitters) avoid having to check several different authentication systems. Instead of checking for an SPF record, a DKIM signature, Sender-ID, and so on, the receiver will have to check only one source, which already stores information based on all of the popular authentication systems. Once the receiver believes that it has authenticated the mail, it can use the information in a filter which uses sender reputation as a criterion, or with a local list-based filter, or whatever.

huh? SPF, DKIM... are all authentication technologies and use extremely different mechanisms. How would a receiver check one thing to see if they all pass?

I wouldn't say that SPF is broken. It is used on both the sending and receiving sides by vast numbers of people and by large organizations. (I once heard of a company called Google....) SPF has it's faults, but so does every other authentication sytem. I also have my faults, but I don't consider myself "broken". Or maybe I am....

Thanks.
David.

I'm comfortable calling SPF broken. Email is designed to be a store and forward protocol, and SPF ignores that. I'm not aware of any large ISP/mail provider using it as spec'd. [And the spec was not ratified by IETF]. Neither Gmail, nor hotmail uses SPF per se. Gmail has their own algorithm (I think they call it best-guess SPF (http://www.ceas.cc/2006/19.pdf).

[miley]

Originally Posted by DavidJ

b) I can't find any free MTAs which support DKIM, and.

How about sendmail?
http://dkim.org/deploy/index.htm

[DavidJ]

Originally Posted by miley

How about sendmail?
http://dkim.org/deploy/index.htm

Sorry, I phrased that badly. I meant either a free MTA which I could use conveniently in some way on my desktop machine, a free public service whose MTA supports DKIM, or an inexpensive mail service with DKIM which is worth paying for.

It's quite likely that the servers which two of my Websites are on use sendmail - I think that I've even used it - but they would not be willing to set up DKIM for me, and in any case their SMTP service is quirky at best.

Actually, the ideal solution FOR ME would be Mercury Mail on my desktop machine if and when Mercury gets DKIM support. David Harris, the developer of Mercury Mail, says that if he sees "a groundswell of adoption among major players", he will add DKIM support to Mercury Mail.

Thanks.
David.

[DavidJ]

Originally Posted by miley

huh? SPF, DKIM... are all authentication technologies and use extremely different mechanisms. How would a receiver check one thing to see if they all pass?

I can't find my way around David MacQuigg's reorganized site, so I can't point you to the explanation. From what I remember, his system would read each one separately, and cache the results when appropriate. In other words - though I'm guessing here - it would read your SPF record using one of the normal ways of getting DNS information - ultimately from your own DNS record, and it would check the DKIM signature in the message whose authentication is now being requested. As far as I know, its only advantage is that it its a one-stop-shopping solution.

I wonder what time David MacQuigg wakes up.

Thanks.
David.

[David MacQuigg]

> I wonder what time David MacQuigg wakes up.

Usually 5AM, and I saw the flurry of posts, but today is very busy. Please be patient if my answers aren't clear or complete.

It has become clear to me (from this and other discussions) that I have not done a good job of communicating the essentials of what open-mail.org provides. I've been too caught up in the details, and my website is not clear and concise. I've re-written the home page at http://open-mail.org/home and I will try and find time to re-write some of the other pages. Suggestions are welcome. Send them by private mail, if they are not appropriate for this thread on reputation services.

On the issue of the relationship between our Registry of Internet Transmitters, and CSV or any other authentication method, we support all methods, including CSV, in the sense that we will at least include in the Registry record for a sender, a keyword indicating that the sender offers that method. So for example, the Registry record for open-mail.org says:
"svc=X1:A mth=SPF+1,CSV+2 ip4=207.210.221.26"
This means that open-mail.org has one ip4 address authorized to say "HELO this is open-mail.org". We also offer SPF and CSV authentication records, all in their proper places. We've given ourselves an A rating, of course.

The record published by the domain owner (from which the above record is derived) is even simpler:
$ dig txt _auth.open-mail.org +short
"helo=mx method=CSV,SPF"
This says - "If you want to know our authorized HELO addresses, look at our MX record. If the HELO test passes, and you want to do some more authentication, we offer CSV and SPF." Note that the domain owner controls everything about his Registry record except the ratings, and the numbers indicating how many additional DNS queries are required to run a particular method.

Options to the helo term include "a", "mx", and "spf", or just a list of IP blocks. We offer these options for methods like SPF or SID, where it is possible to "compile" the record to a list of IP addresses. For those who support only SPF, they can say "helo=spf method=spf", and from then on, just keep their SPF record up to date.

> its only advantage is that it its a one-stop-shopping solution.

While avoiding excess DNS queries is an important advantage of using the Registry, the critical difference is avoidance of some subtle problems with other authentication methods. An SPF record for example, must authorize not only a sender's own transmitters, but those of any forwarders that might carry their mail. Our HELO check avoids the "forwarding problem". When a sender publishes "helo=spf", that means we can use his SPF record to reject HELO forgeries, even if that record ends in ?all.

I hope this is a little more clear. Maybe I should add this to our FAQ.

-- Dave

[Scott Kitterman]

Originally Posted by DavidJ

Sorry, I phrased that badly. I meant either a free MTA which I could use conveniently in some way on my desktop machine, a free public service whose MTA supports DKIM, or an inexpensive mail service with DKIM which is worth paying for.

That's on my near-term TODO. If it tests out well and I'm willing to commit to production use, I'll let you know (I tested DK signing, but was never happy with the results).