It's been a long hard few months in the anti-malware industry (which is why I haven't blogged outside work for a while): for some reason, all our conferences, workshops etc. all seem to be jammed into the last few months of the year. One event I'm always glad to get to is Virus Bulletin, the premier conference and networking opportunity for people in my speciality, but that was a month ago, so I guess it's a bit late to blog about it. I've just got back, though, from a meeting of AMTSO (Anti-Malware Testing Standards Organization), and that has me feeling more positive about the state of anti-malware testing than I have in quite a while.
Product testing (and especially detection testing) is the bete noire of the anti-malware industry. Once upon a time, when the threat landscape was a lot populated than today, it was all a little less fraught. If you found your product credited with a near-zero detection rate, there was a chance, if you managed to establish contact with the tester, to find out what was really happening.
Now, though, when anti-virus labs routinely receive 100,000 or more unique samples a day and we tend to assume a margin for error of +/- 10% to allow for regional bias, validation errors, and so on... The problem is, we tend to find it easier to tell people what they should be doing than to advise them on how to do it properly (or what we think of as properly...) However, the AMTSO meeting represents, I think, something of a coming of age for the representatives of the anti-malware industry taking part, not to mention the testers, reviewers, publishers and so on who are also taking part.
We’ve been working for some time on two major documents: one on "The Fundamental Principles of Testing" and one on "Best Practices for Dynamic Testing." So it was a joy it was to have the final versions of both documents unanimously approved on the last day of the conference. Neither is going to stop bad testing, but they'll go a long way towards giving people with a genuine interest in good testing (whether as a tester or as a consumer) some of the knowledge they need if standards are to be raised across the board. This is an excellent step forward in making available a vendor-agnostic informational resource, and there are several other resources on the way. (Unfortunately, I'm going to have to write some of them...)
David Harley CISSP FBCS CITP
Director of Malware Intelligence, ESET LLC
