2012-01-06T18:36:41Ztag:blog.limedaley.com,2018:/isso/thread/post/parcel-id-parsertag:blog.limedaley.com,2018:/isso/669/29092012-01-06T18:36:41Zjondaley<p>(Peter is commenting on my bayesian filter tests, that I figured I would do using an article that no one had previously commented on, so people would be less likely to see it...)</p> <p>Fortunately, LifeType was smart enough to filter out the hundreds of comments I made while testing, so you didn't see all of those... :)</p> <p>What I was really testing is that I've been getting tons of spam on my company blog and most of it has lots of common spammy terms, and so I'm not sure why the bayesian filter isn't filtering them out by itself. I think it is because there are lots of words, and so the really spammy words are getting reduced by the other words.</p> <p>I think I want a pseudo-blacklist, where I don't have to type in terms, but if the spammer uses words that it thinks are spammy, then more heavily emphasize those words in the score.</p> <p>Also, the spammy URLs should be caught more easily too.</p> <p>Also, it looks like there might be a problem due to the character set I have for mysql - a non-case-significant one, which there were some weird problems with case-significant user names in German a long time ago, and we probably should have looked into it more carefully.</p>tag:blog.limedaley.com,2018:/isso/669/29082012-01-06T17:51:10ZPeter V<p>Word significance is often scored using TfIdf, which is term frequency * inverse document frequency.</p>