Stopping Abuse

[lavabit]

Were looking for tips on how to keep the abusers off our system. We've taken the usual steps of limiting outbound messages, limiting the number of accounts that can be registered per IP subnet, and requiring a CAPTCHA, but nothing seems to be enough.

Have any of the other free service admins developed strategies for locking out the abusive people?

Recently we've had a couple of people use our service for Russian dating scams, and eBay scams. Do you lock these people out without proof of abuse? How do you keep them from coming back?

Feel free to PM us if you don't want this info public.

[davy51]

set up an abuse center have customers enter scams and phishes and blacklist the addresses

it is time consuming but soon it becomes a task that stops a lot of abusers

watch your own customer list and the incomming and outgoing it sometimes gives you a clue to who is abusing

as for an automatic system i dont think anything is a %100 sure

[shaggy24]

There has been a number of email issues from Russia, especially on forums, i know that is not the case with you, but we have started to block entire subnets that originate from Russia for our forums. Might be a good idea to try that with your email service...

[jdtaylor]

I do personally black list a lot of the spam domains, and that is what I have been doing recently now my main email accounts are on the exchange email platform, this does seem to be pretty effective in weeding out the main spam domain names, which have been spamming me, and this with a little tweaking daily, seems to actually improve the situation, as I can't block IP Addresses, which I would, for known spammers if I could, domain blocking, and very stringent spam and virus killers is a good move.

Also for Ebay and Paypal, frauds, please report it directly to spoof AT ebay. com and spoof AT paypal. com respectively they can then take action to get the sites shut down and get the relevant enforcement agencies involved for faking and trying to you know rip off the well known brand names.

Mod: Fixed "live" email address to avoid spambots.

[lavabit]

When we lock out an abuser, we also lock out the /24 they registered from. If we see multiple subnets registering abusive accounts, we lock out the class B. In one case, we even locked out a Class A from (from an African ISP). While we think this has helped, the abusers keep coming. We'll do are best.

[Havokmon]

While we think this has helped, the abusers keep coming. We'll do are best.

Not even that will stop them. I recently spoke with a woman who had 14 charges on my service. A spammer had gotten her card data and upgraded multiple accounts in an attempt to avoid the free user restrictions in place.

I have restrictions on all account types, but no matter what you do, they will always try and counter it.

My primary 'stopper' is not a limit on outgoing messages, but a couple queues set up serially with a delay. If the number of messages in any of my queues exceeds a particular threshold, the queue is shut down, and I'm notified. I can clean out obvious spam, and remove the abusers. If it recurs, I'll wipe out all the signups from that IP address, and block it. Or remove all accounts that used that particular 'originating' email account - and possibly block mail to that domain, or just throw in some javascript to block that address in the signup. :P My reaction really just depends on the situation. The queue check is a great solution, because I can limit outgoing concurrency, create internal delays, and artificially 'stack up' email to actually see how much is going out at once.
I probably have 3-4 shutdowns a day. Cleanup is under a minute, unless I'm away and spam really stacks up...

Some spammers with valid emails in the body also don't like to have their mail 'returned to sender'.. So that's an option as well...

Rick

[jdtaylor]

Well, I am having problems with some spammers trying to 'fake' my domain in spam emails but I thought I had a very secure spf record set up blocking thius but every time I look at this it is either coming back with a netual result or isn't working when it should be and this is with using the spf wizard which is used by virturally everyone and Freeparking as the registar, but it just doesn't seem to be going up again since I tried to update the spf record, the problem came back when I had stopped it, and if anyone has an spf record set up they know which would stop this and just limit it to my ip address if someone can PM me, I'd be most grateful

[Havokmon]

Well, I am having problems with some spammers trying to 'fake' my domain in spam emails but I thought I had a very secure spf record set up blocking thius but every time I look at this it is either coming back with a netual result or isn't working

IMHO, SPF shouldn't be used how you're trying to use it. SPF should only be used as an additional spam scoring mechanism by the receiving server.

If they do set it up to block all email that comes from a non-SPF listed IP address, they will potentially drop forwarded email, or email 'accidentally' not sent through those hosts, or email that - at that moment - COULDN'T be sent through valid hosts.

I realize the - and ~ are supposed to account for that but, as any admin would know, user's don't always do what we expect

SPF is a good way to help admins score that email as spam, but it's not an all-encompassing solution due to other issues that can come up.

Rick

[lavabit]

What is your domain? I'd like to examine the SPF record myself.

As for your comment, SPF is only intended to prevent the spoofing of the MAIL FROM value during the SMTP session. This is the address used for sending bounce messages, and is intended to prevent scatter back. If the spammer is using a valid value in the MAIL FROM, but spoofing your domain in the From or Reply To headers, SPF won't help.

You might try publishing a DKIM record. Its possible to also specify that your domain 'always signs' outbound mail using DKIM. Some, though currently very few, will treat with suspicion incoming messages that aren't signed.

Our mail system is relatively unique, we allow each users to set their preferences individually. You can delete, reject, mark, or accept messages that fail either an SPF or DKIM. If your curious, I can post a break down of how many different users have what options set. We do it this way so that people who subscribe to mailing lists, or have mail forwarded can use looser settings than those who don't.

Finally, we recently decided to code our mail server such that it now only sends bounce messages if it can verify the MAIL FROM address using either SPF or DKIM. Same with auto-responses like vacation messages.

[lavabit]

Originally Posted by Havokmon

My primary 'stopper' is not a limit on outgoing messages, but a couple queues set up serially with a delay. If the number of messages in any of my queues exceeds a particular threshold, the queue is shut down, and I'm notified.

What SMTP server do you use? What scripts/config settings do you have in place for this? Just wondering if we could re-use your idea on our outbound SMTP servers. Our custom mail server relays outbound mail to a Postfix server. On the todo list is setting up a graph of the outbound queue, so we can look for spikes. But I don't know if its possible to set serial queues. Are your queues serial per user?

Long term we'd like to set up a point system. If a user sends out enough messages with URLs in the various URL blacklists, or a number of their messages bounce, then the account is flagged for review. But we haven't implemented this yet.

[Havokmon]

Originally Posted by lavabit

What SMTP server do you use?

I use qmail. To create multiple queues, I setup a second installation (/var/qmail2), and forward all mail to that installation via smtproutes.
Since qmail creates a single smtp connection for each RCPT TO (which at one time I shunned ), the number of emails in the second queue reflects the total number of current RCPT TOs. So I can do a qmail-stat, and see how many RCPT TOs are in the queue at any given time. And actually, to get more specific, I have a 3rd queue that all of those get forwarded to for final delivery. The 3rd queue, on 127.0.0.1 on the outgoing server, has a delay before HELO to keep small bursts of RCPT TOs from getting through.

So I listen on external IP (backup MX), internal network IP (2nd queue gets mail from front-end machines), loopback (3rd queue for delivery with smtp delay). Users don't submit to this server, everything that's outgoing gets funneled to it.

Originally Posted by lavabit

What scripts/config settings do you have in place for this? Just wondering if we could re-use your idea on our outbound SMTP servers. Our custom mail server relays outbound mail to a Postfix server.

I just have a custom perl script that runs every minute and compares the current queue size to a predetermined number. I adjust that number manually as traffic trends change. For example, due to the holiday season, a lot of people are sending out emails to a large number of recipients, so I need to adjust my numbers accordingly. That's easy to do going upwards, because you notice the queue getting shutdown with normal email. The trick is to still check what's going through the queue afterwards to ensure normal traffic hasn't gone down and spammers aren't getting through. I actually get nervous if it doesn't shut down on it's own for a day or two

It sounds like you're doing basically the same thing I am. You can just setup a 2nd postfix install, have one listen on 127.0.0.1 with delay. Have your current postfix install redirect everything it gets to 127.0.0.1 (which should do final delivery), and you can check queue size and shut them down however postfix does it. Hopefully it splits up RCPT TOs, or there's not much point. Doing a single combined delivery to Yahoo is nice, if all your users are trusted.

For the record, I don't technically shutdown my qmail queues, I just set remote concurrency to 0. If you completely shutdown qmail-send (the delivery part), the todo doesn't run and anything that comes into that queue via smtp isn't fully processed into where it can be deleted. So if you're spammer is still going strong, a bunch of garbage gets backed up. If you have 20,000 emails backed up, it takes a long time to process :/ Just stopping remote deliveries allows any residual spam to fall into the queue so when you do get to cleaning it up, it doesn't take so long to process.

I have another script that deletes the offending user and all emails in both queues with his login. That's why there are 2 queues on one machine - it's a lot easier to clean.

Originally Posted by lavabit

On the todo list is setting up a graph of the outbound queue, so we can look for spikes. But I don't know if its possible to set serial queues. Are your queues serial per user?

Just expand on your MRTG use. I have a nice set of graphs to see what's going on for everything from network traffic to spam caught to antivirus to queue sizes, smtp connections, and number of deliveries. Looks like you've already got the numbers consolidated, it shouldn't be too hard to get that into graph form.

I do not have per-user queues, everything goes through that one setup, EXCEPT for paid users. They go out a completely different machine, not quite as complex yet - it will be - but spammers don't use fraudulent cards too much, and when they do it's usually pretty obvious.
To split free users from paid users, the webmail automatically uses the 'authorized' smtp server based on their service level, and fat client users must specifically change their SMTP server to authorized. That server only allows those higher service levels to use SMTP Auth - without SMTP Auth, there is no relaying of email.

A nice addition for me would be an smtproutes-type function that directs emails based on the SMTPAuth name and not the RCPT TO's domain.. This is one area where qmail's modularity could hamper progress - but now that I've thought about it, maybe I can do it

Originally Posted by lavabit

Long term we'd like to set up a point system. If a user sends out enough messages with URLs in the various URL blacklists, or a number of their messages bounce, then the account is flagged for review. But we haven't implemented this yet.

I would go the other way. On my system, spammers don't spend time 'establishing' an account. My thoughts are to check signup date, and total bandwidth used since inception, and based on those numbers send them through another set of queues. With enough hardware I would have 3 sets of the above queues, each being checked in the same manner, but with different thresholds. That way if the 'new' free queues are abused, the established free users wouldn't suffer. Webmail I could do that with, I think I would create a 'usagelevel' field, then have webmail use a different SMTP server based on their usage level. Unfortunately if they're SMTP client users (which 90% of my users are), I'm not sure how I could redirect that at this point unless I can manage that smtproutes modification.. :/

IMHO, the more services that implement some sort of blocking like this to prevent user abuse, the better off we'll all be.

Rick

[Scott Kitterman]

Originally Posted by lavabit

Finally, we recently decided to code our mail server such that it now only sends bounce messages if it can verify the MAIL FROM address using either SPF or DKIM. Same with auto-responses like vacation messages.

How do you verify MAIL FROM using DKIM? DKIM has nothing to say about the envelope.

[lavabit]

Originally Posted by Scott Kitterman

How do you verify MAIL FROM using DKIM? DKIM has nothing to say about the envelope.

We parse the domain from the MAIL FROM address, and then check to see if the sender address in the header uses the same domain and is verifiable using DKIM. If it is, then the bounce message is sent.

[lavabit]

Originally Posted by Havokmon

I use qmail.

Incoming SMTP connections go to our custom SMTP server. This SMTP server scans the message using ClamAV, signs it using DKIM, and checks to make sure the user is under their outbound quota (along with a few other policy checks). Assuming everything passes, the message is relayed over an internal network to a Postfix server. Postfix then saves the message to disk, and is responsible for sending it out (or creating a bounce message).

Your theory regarding expanding user's quotas as they build 'reputation' is interesting. I've considered something similar in the past, but run into this problem: new user's often like to send out a bulk e-mail telling people about their new address. If the website says they can send out X messages, and the server blocks them before that limit is reached, they will either complain, or leave.

What I'd _like_ to do is run every message through a spam filter, like DSPAM, and then use points. If DSPAM thinks its spam, and the user sends out 200 of them, flag the account for review/locking, etc. The problem is DSPAM requires a lot of resources. (Particularly when it comes to storing the token information.) So processing every outbound message isn't an option right now. (We only let paid user's use DSPAM for that reason.)

Originally Posted by Havokmon

Just to expand on your MRTG use. I have a nice set of graphs to see what's going on for everything from network traffic to spam caught to antivirus to queue sizes, smtp connections, and number of deliveries. Looks like you've already got the numbers consolidated, it shouldn't be too hard to get that into graph form.

We use Cacti, and have our own set of graphs. The reason I'm not graphing any of the Postfix stats right now is I haven't figured out a good (secure) way of getting the information into Cacti over the network. I know its possible, I just haven't had time to research it. Integration with Net-SNMP is probably the best method, but I'm not sure how much work that involves. What I'd like to do, if I can figure out a way to monitor the queue size remotely, is configure Nagios to monitor it, and send out a notification if it goes over a threshold. Then I can start poking around if I get a notification.

We just switched over to using a new version of our incoming code for SMTP/IMAP/POP connections. Unlike past versions, this version does statistics tracking, and we recently started pulling that data into Cacti. It still needs some tweaking, but we plan to put some of those graphs up on the website.

Originally Posted by Havokmon

I have another script that deletes the offending user and all emails in both queues with his login.

I found a similar perl script on the web. It takes a regular expression and deletes all of the messages out of the Postfix queue that match the regular expression. Very useful when the spammer decides to register accounts like 'spam1', 'spam2', etc...

Originally Posted by Havokmon

I do not have per-user queues, everything goes through that one setup, EXCEPT for paid users. They go out a completely different machine, not quite as complex yet - it will be - but spammers don't use fraudulent cards too much, and when they do it's usually pretty obvious.

Good idea. Our new SMTP server supports using different pools of SMTP servers for paid/free users. Haven't turned it on yet, but plan too soon.

Originally Posted by Havokmon

I would have 3 sets of the above queues, each being checked in the same manner, but with different thresholds. That way if the 'new' free queues are abused, the established free users wouldn't suffer. Webmail I could do that with, I think I would create a 'usagelevel' field, then have webmail use a different SMTP server based on their usage level. Unfortunately if they're SMTP client users (which 90% of my users are), I'm not sure how I could redirect that at this point unless I can manage that smtproutes modification...

I like this idea. If it were me, I would set up a proxy server. The proxy server would handle the SMTP session until the AUTH is complete, then forward the connection to the appropriate outbound SMTP server based on your rules.

[Havokmon]

Originally Posted by lavabit

Incoming SMTP connections go to our custom SMTP server. This SMTP server scans the message using ClamAV, signs it using DKIM, and checks to make sure the user is under their outbound quota (along with a few other policy checks). Assuming everything passes, the message is relayed over an internal network to a Postfix server. Postfix then saves the message to disk, and is responsible for sending it out (or creating a bounce message).

Your theory regarding expanding user's quotas as they build 'reputation' is interesting. I've considered something similar in the past, but run into this problem: new user's often like to send out a bulk e-mail telling people about their new address. If the website says they can send out X messages, and the server blocks them before that limit is reached, they will either complain, or leave.

This would be in addition to the existing queueing system. All it would do is allow those users who are established to have a higher priority automatically. Everyone else would still fall under the current usage check. This is sort of in place already, as users that I see regularly causing the queue to shutdown have their mails deducted from the total. But it's currently a manual process.

Originally Posted by lavabit

What I'd _like_ to do is run every message through a spam filter, like DSPAM, and then use points. If DSPAM thinks its spam, and the user sends out 200 of them, flag the account for review/locking, etc. The problem is DSPAM requires a lot of resources. (Particularly when it comes to storing the token information.) So processing every outbound message isn't an option right now. (We only let paid user's use DSPAM for that reason.)

I tried that, and use SpamAssassin here. It really didn't work very well. It seems a good portion of the scoring comes from a hash of the complete message, including headers. I never had a spammer have any decent scores when they sent out from me, while 'normal' Spam scanning detects spam just fine. I also have a server that only does spam scanning, as it IS a huge resource hog.

Originally Posted by lavabit

We use Cacti, and have our own set of graphs. The reason I'm not graphing any of the Postfix stats right now is I haven't figured out a good (secure) way of getting the information into Cacti over the network. I know its possible, I just haven't had time to research it. Integration with Net-SNMP is probably the best method, but I'm not sure how much work that involves. What I'd like to do, if I can figure out a way to monitor the queue size remotely, is configure Nagios to monitor it, and send out a notification if it goes over a threshold. Then I can start poking around if I get a notification.

We just switched over to using a new version of our incoming code for SMTP/IMAP/POP connections. Unlike past versions, this version does statistics tracking, and we recently started pulling that data into Cacti. It still needs some tweaking, but we plan to put some of those graphs up on the website.

Nice thing about using an existing package is that work is already done for me All I need to do is save my own patches and re-apply them if I need to update anything. If the Cacti gathers the data with tcp, you could always use stunnel.

Originally Posted by lavabit

I found a similar perl script on the web. It takes a regular expression and deletes all of the messages out of the Postfix queue that match the regular expression. Very useful when the spammer decides to register accounts like 'spam1', 'spam2', etc...

That's exactly what I use. That kind of thing should be part of a mail distribution.

Originally Posted by lavabit

Good idea. Our new SMTP server supports using different pools of SMTP servers for paid/free users. Haven't turned it on yet, but plan too soon.

I like this idea. If it were me, I would set up a proxy server. The proxy server would handle the SMTP session until the AUTH is complete, then forward the connection to the appropriate outbound SMTP server based on your rules.

I'm not sure how postfix works, but that sounds overly complicated - try and redirect a connection during the SMTP session? Qmail uses a file called smtproutes to tell the delivery mechanism to override MX records, and deliver certain domains specifically to a particular machine. I use that feature, without a domain, to redirect all email received to an internal SMTP server where my queue checking happens. I would just use the Auth to have Qmail use a different SMTPRoutes file. The only problem is Qmails modularity may prevent that from being easy - but someone else may have already published a patch that would help here (say, replacing the MAIL FROM with the AUTH name in the queue might do it, so the username is easily available for the delivery module).

Rick