Bayesian Filtering for Spam Detection
Spam is one of the most evil things most people deal with on a daily basis. There are many different techniques for detecting spam. One of the most used and most effective filters is called Bayesian filtering. It uses a statistical method to analyse the content of each mail to determine the probability of them being spam.
Knowledge about the specifics of this method are not needed to use a Bayesian filter. Just install any of the softwares that feature Bayesian filtering and your off. The filter will need training but in most products this is done simply by correcting the filter when it misses. Either by marking spam mails that got through the filter as spam or marking wanted mails as clean. As the filter learns the number of errors will decrease.
The reasoning behind Bayesian filtering is quite simple. All mails contain words. A fundamental assumption in Bayesian filtering is that spam contains words different from those in other mails. We then assign each word a factor. The factor tells us how often the word appears in spam versus other mails. Given a mail we extract a number of words and find the corresponding factors. We then use the collection of factors to calculate a probability. If this probability exceeds a certain limit the mail will be classified as spam.
A common way to initialise the factors is to give the filter something to learn from. You give the filter a number of spam mails and a number of wanted mails. The filter sifts through these mails and sets initial values for the factors for each word. The filter can later be perfected by correcting errors. When wanted mails are classified as spam and we correct the filter, the factors for the words in that mail are decreased. Similarly when a spam gets through the correction means that the corresponding factors are increased.
How spammers try to fool Bayesian filters
One way of fooling a Bayesian filter would be to use lots of words that has had no prior training in the filter. Factors for words are typically in the range of 0% to 99%. Where 0% means that it is highly unlikely that this word features in spam and 99% means that it is highly unlikely that the word features in wanted mails. New words are often given a factor of 50%. If the spammer can hit enough of new words the spam will get through.
It is here that bad spelling comes in. Given the number of spam promoting Viagra most users have gotten most users filters will classify any mail with Viagra in it as spam. But if the spammer instead writes Vlagra or /iägra or Víagrá there is a slim chance that no other spammer have used that combination before. Spellling badlly thuss seeams t0 bee á müst fór spammerrs.
For the Bayesian filter common words that are used in almost all mails will cause problems. Some filters deals with this by implementing an exclude list. Words on that list are always excluded from the calculation.
Does Bayesian filters work?
Yes it does, take SpamPal which is the software I use myself. It features multiple filters where Bayesian is one. The Bayesian filter even learns from when other filter classifies a mail as spam. In the box below is information from a spam I got recently, the Bayesian filter includes this information in the headers of the mail. The mail program then uses this information to remove spam from the inbox.
X-Bayesian-Result: Spam (100)
X-Bayesian-Words: absolutely 97 astounding 99 automatic 48 battery 84 diverse 77 energy 95 feelings 99 forest 99 geocities 40 goods 89 heart 83 identical 99 innocent 99 modules 50 nearly 86 number 76 ones 50 originals 50 outlooks 50 passed 47 poor 50 quartz 99 removal 37 roolexes 50 serial 40 spend 80 star 50 super 62 supply 98 their 75 there 69 through 81 watches 99 where 61 winding 99
The first row is the compound result for the Bayesian calculation. This number (100) shows that this mail is most certainly spam. The next row (or rows when we look at it here) shows what words/factors combinations where used in the calculation. We can see that spelling Rolexes as roolexes got the desired effect, a factor of 50% where used for that word. But the shear number of “almost only in spam” words gave it away. Next we look at a wanted mail.
X-Bayesian-Result: Clean (0)
X-Bayesian-Words: behalf 2 cause 82 company 86 confusion 16 contained 92 contains 81 developers 3 distribute 81 don’t 85 extension 13 feature 14 largest 83 license 15 listed 2 meant 86 membership 85 misleading 99 others 76 professional 84 program 9 purchase 13 shared 99 shareware 4 sites 18 software 22 source 18 submission 0 submitting 0 thank 23 these 77 think 77 update 10 users 9 version 8 windows 77
The first line tells us that this is a wanted mail. The list of words shows us just how well this works. While there are quite a few words that are mostly associated with spam most words have factors well below 50%.
Conclusion
Bayesian filtering works and it works well. When working in parallel with other filtering methods it can filter out most spam from most users inboxes.