L2.APEWS.ORG False Positive #4

This is only the fourth false positive in as many weeks, and it wasn't listed before as the client said it used to be in the inbox;

Mon 2011-11-28 17:33:49: [672:3108] Accepting SMTP connection from [129.228.5.23]
Mon 2011-11-28 17:33:49: [672:3108] Looking up PTR record for 129.228.5.23 (23.5.228.129.IN-ADDR.ARPA)
Mon 2011-11-28 17:33:49: [672:3108] D=23.5.228.129.in-addr.arpa TTL=(60) PTR=[mtv-newsletter4.mms.mtv.com]
Mon 2011-11-28 17:33:49: [672:3108] Gathering A-records for PTR hosts
Mon 2011-11-28 17:33:50: [672:3108] D=mtv-newsletter4.mms.mtv.com TTL=(1440) A=[129.228.5.23]
Mon 2011-11-28 17:33:50: [672:3108] --> 220 xxx.xxx.xxx ESMTP MDaemon 6.7.9; Mon, 28 Nov 2011 17:33:50 -0500
Mon 2011-11-28 17:33:50: [672:3108] <-- EHLO mtv-newsletter4.mms.mtv.com
Mon 2011-11-28 17:33:50: [672:3108] Performing reverse lookup on mtv-newsletter4.mms.mtv.com (looking for 129.228.5.23)
Mon 2011-11-28 17:33:50: [672:3108] D=mtv-newsletter4.mms.mtv.com TTL=(1440) A=[129.228.5.23]
Mon 2011-11-28 17:33:50: [672:3108] --> 250-xxx.xxx.xxx Hello mtv-newsletter4.mms.mtv.com, pleased to meet you
Mon 2011-11-28 17:33:50: [672:3108] --> 250-ETRN
Mon 2011-11-28 17:33:50: [672:3108] --> 250-AUTH=LOGIN
Mon 2011-11-28 17:33:50: [672:3108] --> 250-AUTH LOGIN CRAM-MD5
Mon 2011-11-28 17:33:50: [672:3108] --> 250-8BITMIME
Mon 2011-11-28 17:33:50: [672:3108] --> 250 SIZE 0
Mon 2011-11-28 17:33:50: [672:3108] <-- MAIL FROM:
Mon 2011-11-28 17:33:50: [672:3108] Performing reverse lookup on mms.mtv.com (looking for 129.228.5.23)
Mon 2011-11-28 17:33:50: [672:3108] D=mms.mtv.com TTL=(1440) A=[129.228.5.22]
Mon 2011-11-28 17:33:50: [672:3108] P=010 D=mms.mtv.com TTL=(1440) MX=[mailin.strongmail.west.mtvi.com] {129.228.1.185}
Mon 2011-11-28 17:33:50: [672:3108] Spam Blocker A-record resolution of [23.5.228.129.l2.apews.org] in progress (DNS Server: 192.168.1.2)...
Mon 2011-11-28 17:33:51: [672:3108] Spam Blocker D=23.5.228.129.l2.apews.org TTL=(35) A=[127.0.0.2]
Mon 2011-11-28 17:33:51: [672:3108] APEWS listed, 99.7% certain it is spam
Mon 2011-11-28 17:33:51: [672:3108] Message will be accepted and X-RBL-Warning: header will be inserted.
Mon 2011-11-28 17:33:51: [672:3108] --> 250 , Sender ok
Mon 2011-11-28 17:33:51: [672:3108] <-- RCPT TO:
Mon 2011-11-28 17:33:51: [672:3108] --> 250 , Recipient ok
Mon 2011-11-28 17:33:51: [672:3108] <-- DATA
Mon 2011-11-28 17:33:51: [672:3108] --> 354 Enter mail, end with .
Mon 2011-11-28 17:33:52: [672:3108] --> 250 Ok, message saved
Mon 2011-11-28 17:33:52: [672:3108] <-- QUIT
Mon 2011-11-28 17:33:52: [672:3108] --> 221 See ya in cyberspace
Mon 2011-11-28 17:33:52: [672:3108] SMTP session successful, 10320 bytes transferred.
Mon 2011-11-28 17:33:52: [672:3108] Shuffling message(s) into proper queue(s)
Mon 2011-11-28 17:33:52: [672:3108] Message received from mtv-newsletter4.mms.mtv.com [129.228.5.23] with SMTP for [Size 10309] {j:\localq\md00000000000.msg}
Mon 2011-11-28 17:33:52: ----------

As you can see from the headers, this is MTV's newsletter. Well, watch this space, we'll check in a day or two and report back.

L2.APEWS.ORG False Positive #3

Here is another false positive, nobody else have any then? Strange, so much chat about the amount of errors generated by using Apews yet we're finding very few false positives. These that we have found to date are without using a whitelist and before any client side filtering.

Mon 2011-11-28 07:55:47: [632:2869] Accepting SMTP connection from [176.9.30.45]
Mon 2011-11-28 07:55:47: [632:2869] --> 220 xxx.xxx.xxx ESMTP MDaemon 6.7.9; Mon, 28 Nov 2011 07:55:47 -0500
Mon 2011-11-28 07:55:47: [632:2869] <-- EHLO mail.enewsletters.travel
Mon 2011-11-28 07:55:47: [632:2869] --> 250-xxx.xxx.xxx Hello mail.enewsletters.travel, pleased to meet you
Mon 2011-11-28 07:55:47: [632:2869] --> 250-ETRN
Mon 2011-11-28 07:55:47: [632:2869] --> 250-AUTH=LOGIN
Mon 2011-11-28 07:55:47: [632:2869] --> 250-AUTH LOGIN CRAM-MD5
Mon 2011-11-28 07:55:47: [632:2869] --> 250-8BITMIME
Mon 2011-11-28 07:55:47: [632:2869] --> 250 SIZE 0
Mon 2011-11-28 07:55:48: [632:2869] <-- MAIL FROM:< bounce @ tma.travel > SIZE=75362 BODY=8BITMIME
Mon 2011-11-28 07:55:48: [632:2869] Spam Blocker A-record resolution of [45.30.9.176.l2.apews.org] in progress (DNS Server: 192.168.1.2)...
Mon 2011-11-28 07:55:48: [632:2869] Spam Blocker D=45.30.9.176.l2.apews.org TTL=(35) A=[127.0.0.2]
Mon 2011-11-28 07:55:48: [632:2869] APEWS listed, 99.7% certain it is spam
Mon 2011-11-28 07:55:48: [632:2869] Message will be accepted and X-RBL-Warning: header will be inserted.
Mon 2011-11-28 07:55:48: [632:2869] --> 250 < bounce @ tma.travel >, Sender ok
Mon 2011-11-28 07:55:48: [632:2869] <-- RCPT TO:
Mon 2011-11-28 07:55:48: [632:2869] --> 250 , Recipient ok
Mon 2011-11-28 07:55:48: [632:2869] <-- DATA
Mon 2011-11-28 07:55:48: [632:2869] --> 354 Enter mail, end with .
Mon 2011-11-28 07:55:50: [632:2869] --> 250 Ok, message saved
Mon 2011-11-28 07:55:50: [632:2869] <-- QUIT
Mon 2011-11-28 07:55:50: [632:2869] --> 221 See ya in cyberspace
Mon 2011-11-28 07:55:50: [632:2869] SMTP session successful, 75775 bytes transferred.
Mon 2011-11-28 07:55:50: [632:2869] Shuffling message(s) into proper queue(s)
Mon 2011-11-28 07:55:50: [632:2869] Message received from mail.enewsletters.travel [176.9.30.45] with SMTP for [Size 75762] {j:\localq\md0000000.msg}
Mon 2011-11-28 07:55:50: ----------

Our client said that the email was in the spam folder but is in fact a daily newsletter aimed at folks in the travel business. Looking on the http://www.apews.org website, the IP address itself is not listed but the /24 is suggesting that there is a spammer with an IP address close to that of the newsletter. Further checking of Whois shows this IP address belong to Hetzner, a German hosting business, who in our experience have issues like this quite often.

L2.APEWS.ORG False Positive #2

This is only the second FP that we have seen, and remember folks, we're using the L2.Apews.org balcklist straight "out-of-the-box" by allowing all connections, testing only the connecting IP address, and inserting an X-Header reference for Apews.org listed senders.

Thu 2011-11-24 16:57:53: [632:1914] Accepting SMTP connection from [50.56.45.130]
Thu 2011-11-24 16:57:53: [632:1914] Looking up PTR record for 50.56.45.130 (130.45.56.50.IN-ADDR.ARPA)
Thu 2011-11-24 16:57:53: [632:1914] Name server reports domain name unknown.
Thu 2011-11-24 16:57:53: [632:1914] --> 220 xxx.xxx.xxx ESMTP MDaemon 6.7.9; Thu, 24 Nov 2011 16:57:53 -0500
Thu 2011-11-24 16:57:54: [632:1914] <-- EHLO 356523-web2.x.com
Thu 2011-11-24 16:57:54: [632:1914] Performing reverse lookup on 356523-web2.x.com (looking for 50.56.45.130)
Thu 2011-11-24 16:57:54: [632:1914] Name server reports domain name unknown.
Thu 2011-11-24 16:57:54: [632:1914] --> 250-xxx.xxx.xxx Hello 356523-web2.x.com (may be forged), pleased to meet you
Thu 2011-11-24 16:57:54: [632:1914] --> 250-ETRN
Thu 2011-11-24 16:57:54: [632:1914] --> 250-AUTH=LOGIN
Thu 2011-11-24 16:57:54: [632:1914] --> 250-AUTH LOGIN CRAM-MD5
Thu 2011-11-24 16:57:54: [632:1914] --> 250-8BITMIME
Thu 2011-11-24 16:57:54: [632:1914] --> 250 SIZE 0
Thu 2011-11-24 16:57:54: [632:1914] <-- MAIL FROM:< admin @ x.com > SIZE=1834 BODY=8BITMIME
Thu 2011-11-24 16:57:54: [632:1914] Performing reverse lookup on x.com (looking for 50.56.45.130)
Thu 2011-11-24 16:57:54: [632:1914] D=x.com TTL=(5) A=[50.56.45.133]
Thu 2011-11-24 16:57:54: [632:1914] P=010 D=x.com TTL=(60) MX=[lore.ebay.com] {216.113.175.103}
Thu 2011-11-24 16:57:54: [632:1914] P=010 D=x.com TTL=(60) MX=[gort.ebay.com] {216.113.167.215}
Thu 2011-11-24 16:57:54: [632:1914] P=010 D=x.com TTL=(60) MX=[data.ebay.com] {66.135.195.180}
Thu 2011-11-24 16:57:54: [632:1914] Spam Blocker A-record resolution of [130.45.56.50.l2.apews.org] in progress (DNS Server: 192.168.1.2)...
Thu 2011-11-24 16:57:55: [632:1914] Spam Blocker D=130.45.56.50.l2.apews.org TTL=(35) A=[127.0.0.2]
Thu 2011-11-24 16:57:55: [632:1914] APEWS listed, 99.7% certain it is spam
Thu 2011-11-24 16:57:55: [632:1914] Message will be accepted and X-RBL-Warning: header will be inserted.
Thu 2011-11-24 16:57:55: [632:1914] --> 250 < admin @ x.com >, Sender ok
Thu 2011-11-24 16:57:55: [632:1914] <-- RCPT TO:
Thu 2011-11-24 16:57:55: [632:1914] --> 250 , Recipient ok
Thu 2011-11-24 16:57:55: [632:1914] <-- DATA
Thu 2011-11-24 16:57:55: [632:1914] --> 354 Enter mail, end with .
Thu 2011-11-24 16:57:55: [632:1914] --> 250 Ok, message saved
Thu 2011-11-24 16:57:55: [632:1914] <-- QUIT
Thu 2011-11-24 16:57:55: [632:1914] --> 221 See ya in cyberspace
Thu 2011-11-24 16:57:55: [632:1914] SMTP session successful, 1840 bytes transferred.
Thu 2011-11-24 16:57:55: [632:1914] Shuffling message(s) into proper queue(s)
Thu 2011-11-24 16:57:55: [632:1914] Message received from 356523-web2.x.com [50.56.45.130] < admin @ x.com > with SMTP for [Size 1829] {j:\localq\md00000000.msg}

The client found this email in his spam folder and it should not have been there. The IP address seems to be Ebay developers website using Rackspace web hosting so maybe Rackspace are listed rather than Ebay!

Coincidence or Conspiracy?

I thought it might be interesting to see what data there is on the internet that shows a comparison between blacklists. There is not that much to look at and a search on your favourite search engine will likely yield something like the following;

http://www.sdsc.edu/~jeff/spam/cbc.html is a list of performances but no graph or details of errors. The list is showing the top three as:
1st L2.Apews.org
2nd Zen.Spamhaus.org
3rd b.Barracudacentral.org

Interestingly it has Apews as being the best blacklist and as we know it is free to use for both business and personal. Spamhaus has many years providing antispam solutions but they also have subscription services, not everyone may use their data for free. The same is true for many other antispam solution providers therefore if the spam problem were to cease tomorrow, quite a few folks would be out of a job. In fact, anyone that earns money out of spam wants or even needs spam to continue.

Lets see what else we can find,
http://spamlinks.net/filter-dnsbl-lists.htm#local refers to a long out-dated L2.Apews.org data link namely that of a mirror that was formerly provided by Sorbs.net so the spamlinks.net website is not up to date.

http://www.declude.com/Articles.asp?ID=97 no mention of Apews.org in their list, not even historically, so not very accurate then.

http://www.dnsbl.info/dnsbl-list.php no mention of Apews.org, another not very accurate source.

http://www.spambouncer.org/reference/blocklists.shtml no mention of Apews.org, another not very accurate source.

http://www.nber.org/sys-admin/dnsbl-comparison.html refers to L1.Apews.org but not L2.Apews.org. L1 is a dataset containing domain names only and L2 is all IP addresses. We have found domain name blacklists to be virtually a waste of time for our servers and email flows.

http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists no mention of Apews.org, you would think that wikipedia would at least refer to it.

http://www.intra2net.com/en/support/antispam/ started off showing test data but then stopped after the Apews.org servers were attacked by DDOS stating that the blacklist is no longer available. There is even a special note on the Apews News web page to the websmaster of intra2net.com telling him that they are out of date with their facts.

http://www.techtheft.info/zones/?expand=50 no mention of Apews.org, another inaccurate source.

http://cbl.abuseat.org/faq.html does refer to Apews.org in the part about other blacklists but talks about high false positives which was true a couple years ago and before for not for a long time now. Perhaps the CBL Administrators will start posting their FP details here :-)

http://www.moensted.dk/spam/ does include Apews.org for tests

http://multirbl.valli.org/index.php does include Apews.org for tests

http://wiki.apache.org/spamassassin/DnsBlocklists no mention of Apews.org yet the product SpamAssassin is a scoring solution so it should be even more suited to the use of Apews.org data since the score value for a listing can be adjusted.

http://www.dnsbl.com Al Iverson started testing Apews.org data for a couple of years then just as the catch rate started to surpass existing blacklists, he stopped his testing. Our results show that just after he stopped is when the FP began to gradually reduce until it reached commercially acceptable levels (said to be approx 0.5% or not more than 1%) well over a year ago.

Anyone add to the above? We will keep looking...

RSync for a local copy of a DNSBL

Those email servers that are doing a DNSBL lookup in realtime may sometimes see a DNS timeout or similar. It can happen, reguests don't get answered before the time-out period so in that case the email software will usually ignore the job and continue as if the DNSBL had responded "not listed". The effect of these time-outs is that spam can be either passed for delivery and / or not marked with an X-Header. In short, inboxes see more spam.

Running a local copy of a DNSBL avoids this problem as the lookup requests are entirely on your own network, or even on the same server. That would ensure the continuity of access to the DNSBL data and keep inboxes free of spam. Maintaining local copies of databases, regardless of whether their data is changing frequently or not, can be tedious but not with RSYNC.

Rsync is a nice routine for downloading only changed data from a database host. Most if not all the DNSBL operators offer RSync and have instructions on their website for how to use it to obtain their data. L2.APEWS.ORG is also available by rsync. It is worth adding that many DNSBL data and services are provided totally free to all users.

I won't get into the installation and configuration of Rsync here, there are other places on the internet that adequately explain that. Unix and Linux users have probably already come across it, and Microsoft Windows users could install e.g. Cygwin. Check for compatibility with your particular operating system etc.

Antispam whitelist

There is always plenty of talk about how good or bad a blacklist, or blocklist, is with comments about the false positives generated by that list. These days, with spam at approx 96% of the total daily volume of email sent, no sane email Administrator would operate email servers without first using a whitelist and thereafter possibly filters in addition.

I have had excellent results from these guys;
http://www.whitelisted.org/
You may want to get your own email server listed on their database so that your emails have a better chance of successful delivery, see their website for instructions. The whitelist service seems to be associated with, or run by, http://wwwUCEProtect.net/ , a German blacklist operator.

UCEProtect actually have 3 main blacklists each with it's own listing criteria. Using all 3 blacklists together on your email server will require 3 blacklist entries for DNS lookups but the combined results are very close with those of L2.APEWS.ORG. UCEProtect.net may provide better results for European language based senders and receivers, results here suggest that APEWS.ORG data is particularly good for English.

The use of a whitelist is to exempt the need for blacklist checking, i.e. it is a list of trusted IP addresses. Any sender of an email from a whitelisted IP address can be trusted to connect and deliver their email without any further checking. There can not be any error due to a blacklist since one has not been consulted!

Any connecting IP address that is not whitelisted probably can not be trusted and therefore warrants further checking. Things like PTR records can be useful indicators but as mentioned previously, we see email service providers to governments using badly configured email servers where the reverse DNS does not match. Results here suggest to ignore PTR record checking and just do a blacklist DNS lookup, creating X-Headers for those connecting IP addresses that are blacklisted.

False positives can be seen to be a reflection of the quality of the whitelist being used. If the whitelist maintainer has their data accurate, it would not matter whether trusted email servers were listed in the blacklist or not. Fine tuning of data for both whitelists and blacklists is a coninuous job though once the bulk of the entries are in it is just a matter of adding the odd one at local level.