Bayesian Statistics and Spam
One of the things about email that really chaps my hide is the amount of junk that I get everyday. Now, there are several products out there that offer spam protection and some of those are annoying as all get out. When I fire off a response to a mail list I don't want to have to visit a webpages, type in a hidden word, just so the person asking the question can get the answer from me. To me, foisting off the responsibility to the sender is not a clean way to handle spam.
About a year ago I stumbled across a website that was looking into Bayesian Model Averaging for sorting out spam. Now, you are asking "what the heck is a Bayesian Model Averaging?". To steal a definition:
"Bayesian Model Averaging is a technique designed to help account for the uncertainty inherent in the model selection process, something which traditional statistical analysis often neglects. By averaging over many different competing models, BMA incorporates model uncertainty into conclusions about parameters and prediction. BMA has been applied successfully to many statistical model classes including linear regression, generalized linear models, Cox regression models, and discrete graphical models, in all cases improving predictive performance."
I was able to jump on a project called SpamBayes, written in Python for Unix based systems. At the time, there was a sub-group porting it over to a Windows based application for Outlook. Once it got close to a stable release, the project went commercial and is now know as InBoxer. I've been using the SpamBayes betas for a little over a year and have recently switched to InBoxer. I cannot recommend this product highly enough.
It works 99.9% of the time. After a month of training the engine I rarely get spam or false positives. I go through the suspected spam folder about once a week. A quick scan, select them all, mark them as spam and I'm done. It even has the capability to mark a spam as good. Try it, I think you'll be really suprised.
Some further links: