fix AWL score confusion

[dmorton]

While we don't know directly what score AWL gives, we could either doa special case computation to subtract the other rules from the score or just display ??? for the score, which would be less confusing than an incorrect "1.0"

[rjl]

Funny that you should bring this up when you did--I effectively solved this problem yesterday in my patch-in-progress. It turns out there are some "undocumented features" of SpamAssassin that enable you to access extra information from the SpamAssassin PerMsgStatus object. As of SpamAssassin 3.0 and later, for example, you can use get_tag('TESTSSCORES',',') to return a list of the triggered rules and their scores, in the format AWL=-3.0,BAYES_00=-2.54, etc. This is easily parsed into rule-and-score data, so we can now record the actual score for all triggered rules, rather than having to look it up from the maia_rules table.

In consequence, I think it's time we bumped the minimum required version of SpamAssassin to 3.0, which was probably overdue anyway. There are some security issues with 2.64 and 2.55, mostly related to denial-of-service attack potential, but it's certainly advisable that people use the newest versions of SpamAssassin for the simple reason that they're more effective. Our incentive has to do with correcting the AWL problem, but that doesn't mean it isn't a good idea on other grounds as well.

[dmorton]

woot! That means we can drop the maia_rules table all together?

I'm sure another overhaul will be needed with the auto adjusting scores issue, but we'll cross that bridge later. Should we make another ticket to make configtest.pl look for SA > 3.0

[rjl]

No, we can't drop maia_sa_rules altogether, nor should we really want to, since that's where we store rule names and track rule hit counts. That's also where rule performance history data will be stored with Dynamic Score Balancing.

The benefit to this fix, though, is that for the purposes of displaying rule hit summaries in the Mail Viewer, we can use maia_sa_rules_triggered.score with confidence that it's been set properly, so the totals should add up correctly (assuming all of the triggered rules exist in maia_sa_rules).

[dmorton]

I guess I'm thinking, we don't need the rule 4 scores in maia_sa_rules, at least not for current functionality, but yes, I'm sure an altered functionality will be needed for DSB.

[rjl]

That's not quite true. Yes, the maia_sa_rules.rule_score_[0-3] columns and maia_config.sa_score_set are no longer used by viewmail.php now that we're getting rule scores directly from SpamAssassin, but the rulestats.php page still relies on this data to stock the Score column of the SpamAssassin Rules stats table.

On the other hand, eventually DSB will replace these things with a new mechanism that stores score values returned by SpamAssassin when rules are triggered (if the new score doesn't match the currently-stored value), so there will be a need for a single score column, e.g. maia_sa_rules.base_score. If you're eager to get rid of these other columns, we can certainly move ahead with at least this score-caching part of the DSB mechanism, and rulestats.php can draw its scores from this new column instead.

[rjl]

Implemented in [1103]. Dynamically-scored rules like AWL and FUZZY_OCR_* are now properly reported in the mail viewer, so mail viewer spam report summaries should now add up properly.

The maia_sa_rules.rule_score_[0-3] columns will eventually be removed and replaced with a single 'base_score' column (needed for rule stats tabulation). In the meantime, rule_score_3 is being used for that purpose. A future upgrade script can rename this column as necessary.

Note that this requires SpamAssassin 3.0 or newer, which everyone should be using by now anyway, and will become a minimum requirement in future Maia releases.