English journalist Kevin Townsend asks Is the anti-virus industry in bed with the
NSA – why do CIPAV, FinFisher and DaVinci still defeat AV? Since a large proportion of my income is derived from consultancy for the AV industry, you might think I ought to know the answer. (But let’s make it clear from the start that I’m not speaking on behalf of anyone but myself.) In fact, nobody is paying me specifically to talk to journalists, but I know very well that media consultants advise those who are to be circumspect in answering any question that resembles ‘are you still beating your wife?’ Kevin didn’t actually ask about anybody’s wife, but there is an assumption in the very question he did ask that begs another question about his readiness to assume failure – in this respect – on the part of the anti-malware industry. But I’ll come back to that.
Actually, I don’t know the answer to his question – the one about the NSA, that is. (Shock! Horror!) Or to be more precise, no AV researcher I’ve ever spoken to – and after nearly 25 years in security, I’ve got to know more AV researchers than most journalists – has admitted that a government agency might have asked them to turn a blind eye to government trojans. It’s possible, of course, that even security researchers don’t tell the whole truth the whole time. And it’s also possible, as a well-known European conspiracy theorist once assured me, that these deals are made at C-level, not with researchers and developers. I can’t say it’s impossible that once in a while some security company C*O rings down to someone on the shop-floor and says ‘please remove detection for a file with the following characteristics….’, but I have to wonder what communication channels exist for exerting that kind of pressure on a security company.
Is there someone at the NSA whose job is to monitor whether government trojans are currently detected by AV software? Do they have a lab, or are they just submitting the hashes to VirusTotal? Do they have a list of C*O phone numbers and use the threat of CIA black ops to enforce their requests? Do they have a mutual arrangement with the FBI so that CIPAV comes under the same umbrella? Do they also have a mutual agreement with foreign agencies and providers like the German police (Bundestrojaner) and Gamma International (FinFisher) so that they can apply the same pressure? If I want to avoid being monitored by the NSA, should I perhaps consider a Chinese AV product?
Well, there’s a serious point here. (Actually, I wasn’t being altogether flippant in the previous paragraph, either: it’s quite likely that AV detection of ‘government trojans’ are periodically if not continuously checked by those who write them.)
Speculation about this alleged alliance seems to be based on the assumption that the ‘AV Industry’ – we really ought to be talking about the ‘anti-malware’ industry at this point, though – is a single monolithic entity, or at least readily accessible via a single pressure point. On the basis of which of his own articles Kevin cites, I suspect that he might be thinking that AMTSO or the WildList Organization might be such a pressure point, but AMTSO is not WLO, and neither AMTSO nor WLO is the anti-malware industry incarnate. In fact, there are a great many security vendors whose products are focused on malware detection and/or blocking who aren’t represented in one or both, which in any case have a very specific focus, and neither is some generic mouthpiece for the security industry.
To get back to a question of my own that I hinted at earlier: why does he assume that these programs do defeat AV? Perhaps because Bruce Schneier says ‘…anti-virus software won’t detect them…’ Bruce Almighty is a very clever man, and the security landscape would be that much duller without him, but his understanding of anti-malware technology is not always perfect. He may well know more about the tools used by the NSA than I do, but at least two of the tools cited by Townsend are or have been detected by at least one anti-malware company: the Bundestrojaner (Win32/R2D2.A) and FinFisher (Win32/Belesak.D). (Of course, I very much doubt whether ESET was the only company to detect those two examples.) I can’t, of course, guarantee that there aren’t later samples of those or anything else that aren’t detected. And in fact, I’m pretty sure that Luis Corrons and Claudio Guarnieri, as quoted in Kevin’s article, are right in suggesting that the makers of such products (commercial or in-house) will adjust their products in order to avoid detection: that’s probably a continuous process for them just as it is for the gangs behind unequivocal malware.
I do think Schneier is pretty much on the money when he suggests that geopolitical differences between companies would make some products more susceptible to pressure than others. Political pressure, at any rate, though I suspect that non-US companies with a toe in the US market could be targeted for indirect commercial pressure. That’s an approach that could easily backfire, though. Kevin doesn’t seem convinced though. He cites Mikko Hypponen’s apologetic article for Wired, referring to the fact that the industry missed Stuxnet and Flame for so long, and hints that ‘two major government-sponsored malware samples known about and ignored by multiple AV companies for several years’ may not be coincidence.
Well, in PR terms it was certainly, as Mikko described it, a ‘spectacular failure’. In real life, though?
Nowadays, anti-malware labs process hundreds of thousands of samples a day: failure to realize the significance of a vanishingly small set of stealthy, low-prevalence samples is hardly describable as a success, but it’s not exactly a spectacular failure in statistical terms. Of course, if they were the only such failures over a period of years, that might be seen as a clear indication of a sinister conspiracy. But they aren’t. They are significant – and linked in Mikko’s article – because they’re conspicuous (and related) failures of automated preliminary analysis, but there are many such failures that no-one writes about. To argue malfeasance from two instances sounds pretty weak to me, or at best based on a misunderstanding of lab processes. No lab I know of has the resources to perform manual analysis on several hundred thousand samples per day, so they must to some extent rely on automated analysis to flag those samples as requiring further investigation. That automated analysis can’t be expected to be infallible. So, yes, once again security technology failed once again to provide 100% protection.
But Kevin’s article also asserts that if ‘the AV industry’ is not ‘in bed with the NSA’, it must be because ‘the AV industry is not as good as the “stops 100% of known malware” claims that it makes.” So the innuendo about complicity with the NSA is not really the point at all: the real target is an AV marketing claim. You may find it a little confusing if you’re not familiar with some of his other writing: surely anti-malware ought to detect everything it knows about? What he’s referring to, though, is – I imagine, seeing that he included a link to this article – the use of WildList testing as a measurement of AV effectiveness. I talked about the declining usefulness of WildList testing at some length here, but I’ll reproduce the relevant text here anyway. (I was referring to his blog here, by the way, not the blog about AMTSO.)
… Kevin summarizes the why-you-shouldn’t-use-VirusTotal-reports-as-a-detection-metric position succinctly and accurately, but he also makes other points that deserve further discussion.
But here’s the rub: the AV industry isn’t innocent of its own sleights of hand.
Hard to argue with that. AV marketing departments have made some pretty crass claims from time to time.
The one that gets me personally rather hot under the collar is the ‘destroys all known bacteria dead’. Well, that’s the clear message. The actual terminology is ‘stops 100% of viruses in the Wild’. What it is really saying is that Stoppem Anti Virus detects every virus in the Wild List. And the Wild List is very different to ‘in the wild’. In fact, the Wild List is effectively compiled by the AV industry; so in reality, any AV company that doesn’t score at least 99.99% success against viruses in the Wild is largely incompetent.
Well, sort of.
Once upon a time, the difference between the number of known viruses and the number of viruses literally in the wild (i.e. posing a real threat to the everyday user) was pretty small, so there was quite a lot of merit to the idea of the WildList, essentially a catalogue of virus names corresponding to a collection of verified samples known to be in the wild. (We tended to use the capitalization In the Wild or the abbreviation ItW to indicate that we meant malware qualified to be on the current WildList, rather than the entire population of known and unknown viruses that were out there and posing a threat.)
However, the general usefulness of the WildList to the world at large has declined as the number of samples on the WildList at any one time has become a tiny fraction of the total population of malicious programs that pose a potential threat to users, even though the range of malware that makes the WildList has widened. At the same time, many malicious samples have a lifetime of minutes, whereas old-time viruses could sometimes survive on the WildList for months, even years. As a result, the concept of ‘Wildness’ has become practically useless to people outside the AV industry, while the sheer volume of known malicious samples is unmanageable in terms of defining which samples are or are not ItW in some technical sense. So is the WildList (or the collection of samples it represents) of any use at all in testing?
On the negative side:
• It’s a pretty small sample set.
• It still doesn’t represent the whole range of malware that security software can or tries to detect.
• It doesn’t really represent the dynamic state of the threat landscape. You might say it lacks the element of surprise.
On the positive side:
• It represents a set of samples agreed to be truly malicious: that is, the verification process is better than that applied to most collections.
• It represents a baseline of samples that all products should make a fair fist of detecting: if you like, a minimum standard that all products should be capable of meeting. It’s not literally compiled by the AV industry, as Kevin suggests, but it is verified by vendors. So it has some value for certification purposes (e.g. VB100) but very little for comparative testing.
What about Kevin’s point about 100% of all bacteria? Well, he’s right. There are (at least) three levels of aspiration here.
• Detection of everything on the WildList. More difficult than you might think, and some mainstream companies (and testers) have simply decided not to bother with it any more in any case. When a product does achieve such certification, it’s certainly worth something, but it’s not a measure of absolute protection in the real world.
• Detection of everything known to be in the wild (note the capitalization). Theoretically achievable given enough resources, but I wouldn’t care to promise that any product can achieve it. Or that any testing organization could realistically assess its ability to achieve it. Blocking without specific detection is a little more achievable in principle (in the form of whitelisting, for example), but there’s a hidden cost there (false positives, convenience trade-off, and so on).
• Detection of all malware, known and unknown. Yeah, right. I believe that. I also believe in Santa Claus.
Ability to pass WildList-based certification is a good sign, but it’s not at all the same as catching everything that poses a threat.
Or, indeed, as catching everything known. A single company might at any point detect all the malware it knows about for which detection isn’t still in process. But it doesn’t mean that all companies detect it. (If it did, there wouldn’t be much point to comparative detection testing.)
Sorry, but I’m going to quote another of my own blogs (again, not an ESET blog).
Detection statistics and test performance as a promotional tool strikes me as a particularly contentious point, one that isn’t far removed from Kevin Townsend‘s point about WildList testing. I don’t happen to think it’s valueless, but if companies use marketing that suggests that everything in the wild (whatever you may understand by that) is on the WildList so a 100% detection of WildCore = 100% protection, that sets an expectation just as unrealistic as the 0-5% figures bandied by AV’s critics.
It’s important to keep improving products as they move further and further away from static detection, but if we’re to counter misinformation from other security sectors, we also need to make it clearer to our audiences and customers – not necessarily the same thing - what we really do and what they can realistically expect from us.
David Harley CITP FBCS CISSP
Small Blue-Green World
ESET Senior Research Fellow
