Open-Source Reputation Service - Page 2

indyamail · 29 Dec 2007, 10:41 PM

Havent seen much since June...or any updates - should I assume the idea has been delayed?

David MacQuigg · 30 Dec 2007, 03:51 PM

Please see http://open-mail.org/WhatsNew for major updates, or send me a private message if you would like more detail on my mailing list.

Our Registry of Internet Transmitters (TM) and the Border Patrol MTA (TM) are working better than expected. Reputation systems work surprisingly well. See http://www.ceas.cc/2006/19.pdf if you have any doubt.

We are now at what I would call Stage 2 of our development. Stage 1 was proving technical feasibility: inbox spam less than 1% and no lost mail. Stage 2 is accumulating reputation data on a large number of senders: enough that our database is as good as any proprietary system. Stage 3 will be changing the world: getting enough people to use the Registry, that we will finally see an end to the "era of spam".

The delay is due to lack of interest from mail admins and others in the email community. Problems I have identified include: 1) They are satisfied with the existing proprietary systems, and don't have much interest in solving what they see as other people's problems, 2) They believe that spam is an insolvable problem, and are not willing to look at what we have, and 3) They are working on a competing system, or have a conflicting interest in the status quo.

I'm looking now for a business partner, someone who has an interest in marketing and business development, but who won't insist on making our Registry just another proprietary system. I'm thinking about some kind of hybrid business model - proprietary to get started, then back to our original goal of changing the world.

hadaso · 30 Dec 2007, 04:38 PM

Quote:

Originally Posted by David MacQuigg

... or have a conflicting interest in the status quo.

ISPs sell anti-spam services. They don't have to be very good since most customers are clueless and don't use email for any critical mission (or perhaps they have to let through a little so customers remember why they are paying for anti-spam). ISPs are also hosting spammers, so they are making money on both ends. Why would they want to end spam?

David MacQuigg · 3 Jan 2008, 05:05 AM

Hadaso, your observations are correct for ISPs in general, but it would help if we make a distinction between NSPs (Network Service Providers) and ESPs (Email Service Providers). Each has different incentives and motivations. An ISP can be either or both, but it still helps to think of them as separate in our discussions, and in focusing our efforts where we will make the most progress.

Network owners have a strong conflict of interest with anyone trying to put a dent in the worldwide flow of spam. Their profits depend on maximizing the traffic on their networks, and a recent study shows that over 90% of that traffic is now spam. http://www.circleid.com/posts/90_to_...cent_spam_2007 We need to avoid network owners, or at least not waste any time with them. The best we can do for them is them is a system that leaves the spam still flowing for that minority of recipients who really don't care if it comes their way. That might actually be a majority of non-business users.

We should also break out domain-name owners as a separate category. They have a strong interest in stopping spam connected with their names, which can only hurt their reputations. There is a wide range in the degree of this interest. Some owners care very little, and some care a lot. I can't think of any that would actually encourage spammers to use their name. The value of that name would soon be no more than a new unknown name, i.e. zero. We need to look for support among owners of reputable domain names.

Email Service Providers are caught in the middle. As Receivers, spam is nothing but a burden, and they would like to see it gone. As Senders, however, they have a small conflict. It costs them time and money to eliminate outgoing spam, but if spam becomes more than a small fraction of a Sender's traffic, that Sender will lose business due to their acquiring a bad reputation among Receivers. Senders will do only what they must to avoid a bad reputation. We need to encourage Senders to do the right thing - authenticate their transmitters, and keep them spam-free.

There is a small, but unavoidable cost in keeping a transmitter spam-free. Users must be authenticated, outgoing mail must be filtered and rate-limited, and spam reports must be dealt with promptly. Senders that make this effort (e.g. aol.com and yahoo.com) have driven their spammers away. There are plenty of other more spam-friendly services out there. See a sample of our data at http://open-mail.org/DomainRatings. There are huge differences in the % of spam coming from different ESPs.

So how can we encourage all ESPs to do the right thing? I believe the answer is in making sender reputation data widely and freely available, and providing more direct feedback from recipients to senders. Recipients of legitimate email, as a whole, have far more economic power than spammers. We need to harness that power and direct it in a way that will influence the behavior of senders. This is the guiding principle in the design of programs and services at Open-Mail.org. We gather spam reports from Receives using our Border Patrol MTA, and publish ratings of sender domains in our Registry of Internet Transmitters. Recipients set a threshold for how much spam they will tolerate from a sender. Senders with less spam are "whitelisted", and enjoy 100% delivery. The rest go through the usual gauntlet of statistical filters and blacklists.

So far our system is working very well. No spam in my inbox, and no lost messages. I can't see any reason it won't scale up to any level needed. I believe it will work even in an environment where the networks are *owned* by the spammers, which may not be too far from the current status quo.

DavidJ · 10 Jul 2008, 05:29 PM

Could you give us an update on the number of "transmitters" and receivers using your system? It might be better if you divided them into large and small providers.

I still don't quite understand: Are you only providing one-stop-shopping for the various sender authentication methods, or do you also have something to do with the more downstream stages of reputation and rating services?

Thanks.
D.

29 Dec 2007, 10:41 PM	#16
indyamail Member Join Date: May 2007 Posts: 33 Representative of: IndyaMail.com / SocketMail.com	Havent seen much since June...or any updates - should I assume the idea has been delayed?

30 Dec 2007, 03:51 PM	#17
David MacQuigg Member Join Date: Aug 2006 Location: Tucson AZ Posts: 66 Representative of: Open-Mail.org	Please see http://open-mail.org/WhatsNew for major updates, or send me a private message if you would like more detail on my mailing list. Our Registry of Internet Transmitters (TM) and the Border Patrol MTA (TM) are working better than expected. Reputation systems work surprisingly well. See http://www.ceas.cc/2006/19.pdf if you have any doubt. We are now at what I would call Stage 2 of our development. Stage 1 was proving technical feasibility: inbox spam less than 1% and no lost mail. Stage 2 is accumulating reputation data on a large number of senders: enough that our database is as good as any proprietary system. Stage 3 will be changing the world: getting enough people to use the Registry, that we will finally see an end to the "era of spam". The delay is due to lack of interest from mail admins and others in the email community. Problems I have identified include: 1) They are satisfied with the existing proprietary systems, and don't have much interest in solving what they see as other people's problems, 2) They believe that spam is an insolvable problem, and are not willing to look at what we have, and 3) They are working on a competing system, or have a conflicting interest in the status quo. I'm looking now for a business partner, someone who has an interest in marketing and business development, but who won't insist on making our Registry just another proprietary system. I'm thinking about some kind of hybrid business model - proprietary to get started, then back to our original goal of changing the world.

3 Jan 2008, 05:05 AM	#19
David MacQuigg Member Join Date: Aug 2006 Location: Tucson AZ Posts: 66 Representative of: Open-Mail.org	Hadaso, your observations are correct for ISPs in general, but it would help if we make a distinction between NSPs (Network Service Providers) and ESPs (Email Service Providers). Each has different incentives and motivations. An ISP can be either or both, but it still helps to think of them as separate in our discussions, and in focusing our efforts where we will make the most progress. Network owners have a strong conflict of interest with anyone trying to put a dent in the worldwide flow of spam. Their profits depend on maximizing the traffic on their networks, and a recent study shows that over 90% of that traffic is now spam. http://www.circleid.com/posts/90_to_...cent_spam_2007 We need to avoid network owners, or at least not waste any time with them. The best we can do for them is them is a system that leaves the spam still flowing for that minority of recipients who really don't care if it comes their way. That might actually be a majority of non-business users. We should also break out domain-name owners as a separate category. They have a strong interest in stopping spam connected with their names, which can only hurt their reputations. There is a wide range in the degree of this interest. Some owners care very little, and some care a lot. I can't think of any that would actually encourage spammers to use their name. The value of that name would soon be no more than a new unknown name, i.e. zero. We need to look for support among owners of reputable domain names. Email Service Providers are caught in the middle. As Receivers, spam is nothing but a burden, and they would like to see it gone. As Senders, however, they have a small conflict. It costs them time and money to eliminate outgoing spam, but if spam becomes more than a small fraction of a Sender's traffic, that Sender will lose business due to their acquiring a bad reputation among Receivers. Senders will do only what they must to avoid a bad reputation. We need to encourage Senders to do the right thing - authenticate their transmitters, and keep them spam-free. There is a small, but unavoidable cost in keeping a transmitter spam-free. Users must be authenticated, outgoing mail must be filtered and rate-limited, and spam reports must be dealt with promptly. Senders that make this effort (e.g. aol.com and yahoo.com) have driven their spammers away. There are plenty of other more spam-friendly services out there. See a sample of our data at http://open-mail.org/DomainRatings. There are huge differences in the % of spam coming from different ESPs. So how can we encourage all ESPs to do the right thing? I believe the answer is in making sender reputation data widely and freely available, and providing more direct feedback from recipients to senders. Recipients of legitimate email, as a whole, have far more economic power than spammers. We need to harness that power and direct it in a way that will influence the behavior of senders. This is the guiding principle in the design of programs and services at Open-Mail.org. We gather spam reports from Receives using our Border Patrol MTA, and publish ratings of sender domains in our Registry of Internet Transmitters. Recipients set a threshold for how much spam they will tolerate from a sender. Senders with less spam are "whitelisted", and enjoy 100% delivery. The rest go through the usual gauntlet of statistical filters and blacklists. So far our system is working very well. No spam in my inbox, and no lost messages. I can't see any reason it won't scale up to any level needed. I believe it will work even in an environment where the networks are owned by the spammers, which may not be too far from the current status quo.

10 Jul 2008, 05:29 PM	#20
DavidJ Senior Member Join Date: Dec 2002 Posts: 178	Could you give us an update on the number of "transmitters" and receivers using your system? It might be better if you divided them into large and small providers. I still don't quite understand: Are you only providing one-stop-shopping for the various sender authentication methods, or do you also have something to do with the more downstream stages of reputation and rating services? Thanks. D.