…and around it goes

I’ve been fighting spam zombies most of the winter.  We are currently hosting over 25,000 mail domains.  Some of these domains come from hosts that simply could not handle the volume of zombies and backscatter, or just dumped it all at the end user, some being hit with 30k bounces an hour backscatter.  Others are our own domains, some of which spammers seem to love.  All of which need some protection where turning off the catchall just won’t do, that’s where my milter comes into play.

This milter was started when sendmail first started featuring milters (we’ve been seeing zombies hit us hard for years and without the protection it would be nearly impossible to offer our service).  Since then it has evolved quite a bit.   The idea behind it is to dynamically identify infected end user machines spewing spam and block those while allowing all validly sent mail, including valid mail servers spewing spam, to still get through (the milter, anyway, then it’s user filters that will block).

It accomplishes this by evaluating a number of items: the helo, the host and number of other related hosts that hit and were identified as a zombie (ie. how many other 123-123-123-123.example.coms have attacked), how it hit the server (ie. slamming, number of concurrent attempted deliveries, etc), number of unknown users it attempted to mail, spamtraps of ours it hit, and more.  Using all of these variables it creates a profile and matches against it.

At first it just dynamically managed a blocklist.  This got cumbersome fast as we grew.  The blocklist had to move to a db and I had to learn to be a better and more efficient programmer.  It now utilizes a few databases, incorporates dynamic blocking, backscatter protection per account or domain, and more.  It also creates a profile of valid mail servers and matches against those as well so that they don’t get caught even if they match the rest of the zombie profile (ie. the server got infected).  The target is only zombies.  We’ve slowed on the number of zombies added, but still pump in a fair amount of new ones every day.  I do expect this to slow again, but I’ve zeroed the dynamic db a couple of times both to redesign it and to test it’s autolearning as I tweaked, so it’s still a rather large amount.

Perhaps one day I’ll write a web interface to the db.  It’s contents paint a very interesting map of infected end user machines, or spam zombies.  I’ve also been able to identify individual botnets by things as simple as the way they helo (type server40.welcometelecom.ru into Google and see what I mean, you’ll see it show up as the helo for a lot of spam with hostnames around the globe). Others are slightly more complex, being fed a list of helo values along with the standard list of from addresses to use or using the machine name, but they form a pattern in other ways over time and become identifiable as well.

My mind is already running with ways to query this data and provide live statistics (number of zombies per botnet, domain, etc) that can be drilled down all the way to the list of machines and date last seen.  Unfortunately that is side work, which takes a back seat to the day to day running of the service so I have no idea when, or really even if, it will end up searchable like that.  Right now I have perl scripts that do it and I use the results to further tweak the milter.

BTW: Greylisting is dead.  I declare so now.  Nearly all zombies I have identified return, even after getting 550 returns to their delivery attempts (I’d say all but don’t yet have positive proof of all, but I do have positive proof that those which return within a typical greylist period are well above 90%.  What I don’t know is if they are resending or sending a new blast).  So those 4xx errors you greylisters return will also have the same zombie return.  If you are greylisting you are likely now accomplishing little but delaying your mail.

Other Battles:

Been battling a rather serious (but not fatally serious) health issue this winter (stomach/bowel).  It’s had me bedridden a lot.  This put a delay in some new features that were planned (they are still planned, just delayed).  I did get a chance to improve a number of backbone things.  Added bandwidth and server power to the mail network, further improved automatic failover (finally addressing the issue of “what if Verizon comes and rips out all the copper to the building again?”), redesigned our DNS, and fixed a number of bugs, but nothing that can really be identified by users as “hey, here’s something new”.