Blocking spam using APEWS.ORG

Something that is working came about because spammers ignore published procedure. Let me explain that, an email gets sent based on what the DNS records say about the domain name in question. If I have example.com then I may want to have a website that people can find either at example.com or www.example.com. We do that by creating a blank "A" host record and another for www thus creating the desired prefixes.

Email servers can be setup for a domain by creating "MX" records in the DNS and set the priority value for each. One basic system is to have 2 public IP addresses and create an MX record for each but give one a priority value of say 1, and the other a value of 2. The physical email server that is on the IP address corresponding to priority 1 will be the first recipient of domain emails. Only when sending servers find that server & IP address to be unreachable or busy will emails start to be tried at the second server & IP address corresponding to priority value 2. In this way you have published your preference of how sending email servers should attempt to deliver emails to your domain.

What I have found is that spammers send their spam to all MX IP addresses and the root "A" record IP address in addition. Taking the above example you would have a web server on one IP address and 2 email servers on another 2 different IP addresses making 3 external or public IP addresses in total. You expect web traffic to go the www IP address, and the majority of your emails to go MX1, with MX2 acting as backup or failover.

Nice idea but spammers don't follow your preferences since they are only interested in successful delivery. They will attempt to deliver spam to all 3 IP addresses and if your web server should have an email server program listening on port TCP 25, it will receive connections from spam sending bot infested computers etc.

My tip is to separate the traffic by design. Staying with the above simple setup, MX1 is the primary email server on one IP address and MX2 is the secondary. Your web server is on a 3rd IP address and does not handle inbound email therefore it does not have a MX record. That means that it should never receive email and depending on web server traffic, you could choose to have the web server handling outbound emails from the website and / or authorized domain users. It may be that due to high traffic levels you dedicate a 4th server to outbound emails and give it a different IP address to the above. The point is that inbound emails should only ever arrive for delivery at MX 1 or if busy, then MX2.

These days email Administrators must use a whitelist of trusted email server IP addresses. There are some very good databases online and I intend to cover that topic shortly for anyone still unaware or unsure but it is essential in order to avoid false positives. Setup your email server to check the connecting IP address against your whitelist and accept for delivery from all that are listed.

Any IP address that has a server connecting to your server and is not on that whitelist is unknown to you and therefore untrusted. The public whitelists have come about by Administrators sharing their trusted IP address details so if an IP address is not listed there, that means a lot of network Administrators do not trust the IP address either!

Starting with MX1, your primary email server, set that to accept all inbound email even if it comes from IP addresses that are not listed in the whitelist. Now have the email server check the IP address against the L2.APEWS.ORG either in realtime at the online database or your local copy that you obtained via RSYNC (another topic for the near future). If your server finds the connecting IP address to be listed at L2.APEWS.ORG, have the email program create a X-Header which can later be used in filtering. Your server then accepts the email for delivery and transfers all thsuch emails with the X-Header to the recipient's Spam or Junk folder.

You can configure your secondary email server MX2 in exactly the same way if you want to, or if the primary server MX1 is handling the majority of emails, you can set this one to reject emails from IP addresses that are listed in L2.APEWS.ORG. Your outbound email server and web server too, if it has an email program running, should be set to reject inbound emails that come from APEWS.ORG listed IP address space. You will only get a false positive if your whitelist is inadequate, remember that the EWS in APEWS stands for "early warning system". I intend to publish here the false positives that we get in the hope of them being delisted by the APEWS.ORG Administrators which helps everyone, more folks should do the same.

The above is currently working on several commercial servers with excellent results. Due to the whitelist on each email server followed by APEWS, 99% of spam is correctly identified. Spammers are getting the 550 error message (which they always ignore) but more importantly, failed delivery. These results are before any after-receipt filters or client side filters.

Do not put Yahoo, Hotmail, Gmail, and the other web-based email servers in your whitelist as we have found that they get used for list washing and can overwhelm your servers. You will find that APEWS.ORG do not have them listed either so you won't lose any emails from their senders. I recommend an alias list that handles mis-spelled email addresses by routing common errors to the correct user email address, and then reject all emails for unknown user names / email addresses. It's all about reputation now, trusted senders are more easily documented as they are so few.

L2.APEWS.ORG False Positive #1

Here is an example for you APEWS;

1 Wed 2011-10-19 18:28:12: [540:1999] Accepting SMTP connection from [50.28.15.113]
2 Wed 2011-10-19 18:28:12: [540:1999] Looking up PTR record for 50.28.15.113 (113.15.28.50.IN-ADDR.ARPA)
3 Wed 2011-10-19 18:28:13: [540:1999] D=113.15.28.50.IN-ADDR.ARPA TTL=(1200) PTR=[host.mudnworks.com]
4 Wed 2011-10-19 18:28:13: [540:1999] Gathering A-records for PTR hosts
5 Wed 2011-10-19 18:28:13: [540:1999] D=host.mudnworks.com TTL=(240) A=[50.28.15.113]
6 Wed 2011-10-19 18:28:13: [540:1999] --> 220 xxx.xxx.xxx ESMTP; Wed, 19 Oct 2011 18:28:13 -0500
7 Wed 2011-10-19 18:28:13: [540:1999] <-- EHLO host.mudnworks.com
8 Wed 2011-10-19 18:28:13: [540:1999] Performing reverse lookup on host.mudnworks.com (looking for 50.28.15.113)
9 Wed 2011-10-19 18:28:13: [540:1999] D=host.mudnworks.com TTL=(239) A=[50.28.15.113 ]
10 Wed 2011-10-19 18:28:13: [540:1999] --> 250-xxx.xxx.xxx Hello host.mudnworks.com, pleased to meet you
11 Wed 2011-10-19 18:28:13: [540:1999] --> 250-ETRN
12 Wed 2011-10-19 18:28:13: [540:1999] --> 250-AUTH=LOGIN
13 Wed 2011-10-19 18:28:13: [540:1999] --> 250-AUTH LOGIN CRAM-MD5
14 Wed 2011-10-19 18:28:13: [540:1999] --> 250-8BITMIME
15 Wed 2011-10-19 18:28:13: [540:1999] --> 250 SIZE 0
16 Wed 2011-10-19 18:28:13: [540:1999] <-- MAIL FROM: SIZE=6549
17 Wed 2011-10-19 18:28:13: [540:1999] Performing reverse lookup on yyy.yyy (looking for 50.28.15.113)
18 Wed 2011-10-19 18:28:13: [540:1999] D=yyy.yyy TTL=(240) A=[50.28.15.126]
19 Wed 2011-10-19 18:28:14: [540:1999] P=000 D=yyy.yyy TTL=(240) MX=[yyy.yyy] {50.28.15.126}
20 Wed 2011-10-19 18:28:14: [540:1999] Spam Blocker A-record resolution of [113.15.28.50.l2.apews.org] in progress (DNS Server: xxx.xxx.xxx.xxx)...
21 Wed 2011-10-19 18:28:14: [540:1999] Spam Blocker D=113.15.28.50.l2.apews.org TTL=(35) A=[127.0.0.2]
22 Wed 2011-10-19 18:28:14: [540:1999] APEWS listed, 99.7% certain it is spam
23 Wed 2011-10-19 18:28:14: [540:1999] Message will be accepted and X-RBL-Warning: header will be inserted.
24 Wed 2011-10-19 18:28:14: [540:1999] --> 250 , Sender ok
25 Wed 2011-10-19 18:28:14: [540:1999] <-- RCPT TO:
26 Wed 2011-10-19 18:28:14: [540:1999] --> 250 , Recipient ok
27 Wed 2011-10-19 18:28:14: [540:1999] <-- DATA
28 Wed 2011-10-19 18:28:14: [540:1999] --> 354 Enter mail, end with .
29 Wed 2011-10-19 18:28:14: [540:1999] --> 250 Ok, message saved
30 Wed 2011-10-19 18:28:15: [540:1999] <-- QUIT
31 Wed 2011-10-19 18:28:15: [540:1999] --> 221 See ya in cyberspace
32 Wed 2011-10-19 18:28:15: [540:1999] SMTP session successful, 5856 bytes transferred.
33 Wed 2011-10-19 18:28:15: [540:1999] Shuffling message(s) into proper queue(s)
34 Wed 2011-10-19 18:28:15: [540:1999] Message received from host.mudnworks.com [50.28.15.113] with SMTP for [Size 5841] {drive:\folder\localq\50000112311.msg}
Wed 2011-10-19 18:28:15: ----------

Line 1: I know that this email was solicited by the user and was only a single email that came from a website server in response to that user's input. It was found in the user's spam folder due to the use of the X-Header and a script as per my previous post. I have munged the header data but the connecting IP address and host name are real.

Line 2: Note the the use of reverse DNS (rDNS) lookup to establish whether a PTR record exists and matches for the connecting IP address. Advice: do not reject incoming emails based on this this alone because I know of several trusted senders (including government and other large institutions) that are not compliant and would result in false positives.

Line 20: Here is the DNS lookup to the L2.APEWS.ORG database in realtime, the connecting IP address is found to be listed and a comment is made to that effect. The mail server Administrator for the website on that IP address will have exactly that in his log too.

Line 23: The email server creates the X-Header entry which later causes the email to be placed into the user's spam folder.

Checking ARIN whois shows that the connecting IP address belongs to Liquidweb, not the best of reputations in my opinion. It will be interesting to see if anything happens with this listing. I will report back here if/when I see a change.

APEWS.ORG data usage

Due to the decline in use / effectiveness of Usenet for antispam, which formerly had newsgroups called news.admin.net-abuse.blocklisting and news.admin.net-abuse.sightings (both now inactive), here is a place where users of APEWS.ORG database can publish their experience and any problems or errors. On their website http://www.APEWS.ORG have asked for evidence of errors so that they can improve their data, and to my knowledge there is no one place that exists for the purpose.

In choosing an antispam solution for email servers, there are point scoring methods like SpamAssassin and there are Domain Name Server Block List (DNSBL). Operating my own business servers and not having the resources of perhaps larger corporations, I needed an approach that would seriously cut into the, at times, overwhelming number of inbound emails. I came across Apews.org shorly after SPEWS ceased to be updated and after finding other DNSBLs to be inadequate. However, Google for this DNSBL and the majority of information available suggests that the data represents an overly aggressive approach with too many false positives.

Rather than just believe what I found, especially since there was little evidence of the false positives, I decided to test the IP based L2.APEWS.ORG data which they recommend for scoring to be used in addition to other DNSBL services like Spamhaus, Spamcop, Sorbs etc via SpamAssassin or similar. Knowing your clients and having a good white list is essential these days and I doubt anyone would put an email server online without one. My own whitelist has come from more than 10 years of experience and accumulated knowledge thus testing inbound connections first against the whitelist, before the blacklist, produces almost zero errors.

During the last few years there have been very few statistics websites that compared True [TP] and False [FP] Positives from the use of a single DNSBL e.g. Spamhaus, Spamcop, UCEProtect or Sorbs etc. By 2008 it seemed that L2.APEWS.ORG had a very high [90%] spam catch rate [TP] together with a reducing level of false positives [FP] but the website operator didn't elaborate much on the FP, only referring to the mail stream as being USA based and including some marketing emails for products, services, and reviews of same. No other free DNSBL comes close from what I can see, http://www.UCEProtect.net have 3 databases and if all are used then their results appear to be about the same if not a little better, approx 1% to 1.5% higher TP currently.

I have configured my email servers to use the L2.APEWS.ORG DNSBL in realtime during the SMTP session, after first querying the whitelist. In the event that the IP address is unknown or untrusted and then found to be listed in L2.APEWS.ORG then the email servers don't reject the email, they just flag it as probable spam by the use of an X-Header which a script then uses to move the flagged emails to the user's spam / junk folder.

Note my criteria and requirements;

• Email servers are used for both local and remote users
• Users to/from UK, West/Central Europe, USA, India, Australia, also roaming users including Far East

Source of emails [almost all private emails are via free webmail providers]

• approx 40% are received are via Yahoo, Gmail, AOL & Hotmail users

• approx 40% are received via contracted email services [negligible spam received] e.g. messagelabs, psmtp, frontbridge, bigfish, postini, mxlogic etc

• approx 15% are from client owned corporate servers [negligible spam received] and includes many international and regional banks, USA / EU government departments etc

• approx 5% are newsletters and social networking contact [negligible spam received] e.g. reuters, alertnet, foxnews, cnn, nytimes, dartmail, collab, cheetah, ezinedirector, sun microsystems, symantec, linkedin, facebook, myspace, flickr, digg, naymz.com [chnaged back after being visible.me], mbox, j2global, iht, osac.gov, oecd, imf.org, worldbank, natgeo, dhl, ups, fedex, deutscheposte, usps, dealertime, shopping.com, amazon.com, aa.com, continental air, virgin, travelocity, hotel.com, cheaptickets, lufthansa etc
  

• hard/soft-ware suppliers & manufacturers e.g. HP, Dell, Cisco, Microsoft, Apple, Macromedia, Adobe, sourceforge etc

• Very few emails are received via ISP smtp servers / smart hosts [negligible spam received] e.g. rogers, rogerstelcom, earthlink, mindspring, prodigy, comcast, sprint, sprintlink, btinternet, bt.com, demon, shaw, shawcable, qwest, adelphia, bellatlantic, bell, bellglobal, bellsouth, bellnexxia, swbell, bellhosting, att, ownmail, telstra, megacity, free2surf, charter, level3, optus, sonic, orange, vodafone, pipex, t-online, dtag, t-mobil, cox, coxinternet, verizon, cogentco, blueyonder, bigpond, roadrunner, twtelecom, nortel etc
  

• Almost zero emails are received via domain Registrars [negligible spam received] e.g. networksolutions, netsol, register, joker, gandi, godaddy, tucows etc
  

• Complaints by email relating to abuse from my servers can be received to role accounts here from major dnsbl operators for each domain name hosted e.g. spamhaus, spamcop, sorbs, abuseat, ahbl, uceprotect, robtex, njabl, mail-abuse, uceb, abuse-net, whitelisted trusted sender servers

• all the above are regarded as trusted senders and as such have been whitelisted here

• the only spam received into user inboxes comes almost entirely from free webmail user accounts or unlisted IP addresses, True Positive is better than 99% because of a good whitelist
  

• all emails in a user's spam/junk folder have been found to be spam i.e. correctly identified and after running email client spam filters on the mailboxes. The FP% is extremely low, less than 0.05%.
  

In conclusion, the use of L2.APEWS.ORG has, for us, removed the spam problem to the extent that the few spam we do receive are via Yahoo, Gmail, AOL and Hotmail servers that we need to give access. It has been said that of the world's total daily email volume, approx 97% is unsolicited bulk email and our experience accords with that statistic. The remaining 3% of the world's total daily email volume is solicited and the above figures represent an approximate analysis of the source and/or nature as it pertains to our business mail stream. These are our findings and no warranty either express or implied exists regarding these findings since each mail stream is unique to the particular business or network.