Collection Bias
This map does not show what FireEye claim it shows.
Israel, 1 0day? 8200 be slacking not hacking
Australia, 0 0days? ASD Aussie Slacker Directorate.
This is a map of 0day deaths where the death is attributed. Nothing more.
A map of attributed 0day deaths. It doesn’t even reflect operational methodology because Chinese and Russian operations face greater scrutiny than say, Australian operations. Indonesia isn’t dropping a lot of APT reports… Snugly Wombat is playing on safe ground.
Over the years covered by this dataset the detection capability of victims has changed. There is no consistency in the methodology of the collection, even with the same victims, year over year.
Collection Bias
A map of 0days caught in the wild and attributed back to threat actors seems like useful data. But it is not.
This data set is an example of a serious analytic error called “collection bias.” The data only reveals what was collected, not an accurate representation of the real world.
The distribution of 0days is a major indicator of how flawed this dataset is. Why does China have over three times more 0days than the US? The answer is immediately obvious: Chinese cyber operators are attacking environments that are monitored by multiple advanced threat detection companies. US cyber operators, typically, are not.
One way to think about collection bias is: a football game where statistics and goals are only kept for one side. Naturally the numbers won’t be useful for understanding the game. What they reveal is so distorted that only extremely careful analysis and cautious findings are possible. That is not the case with the FireEye report.
There is some merit to the theory that this map reveals operational methodology. Russia and China have different incentives regarding stealth than FVEY countries. But that is false.
The circumstances of the collection clearly reveal the problems with this interpretation. The Uzbekistan 0days were discovered only because of operator error. The Israeli 0day was from an attack that penetrated Kaspersky, only to later get detected by Kaspersky’s R&D next generation product.
The lack of analytic rigour is transparent from just a plain reading of the text. The Israeli company NSO is cited as a reason for Uzbekistan having their 0day exploits. Yet the authors don’t consider the question of why Israel has only 1 0day but Uzbekistan has 3.
Surely NSO has provided exploits to their own government? Even if not, the exploit developers for NSO must have come from somewhere, or gone to work somewhere. After all, it is common knowledge that Unit 8200 creates and uses exploits.
Errors from Analytic assumptions
Uzbekistan’s use of 0days wouldn’t be known except for an operator error. Clearly countries that don’t get caught by similar mistakes aren’t included. Therefore Uzbekistan is a sort of “self selection” bias.
The inclusion of such poorly sourced data just raises further questions such as: what other 0day does Uzbekistan have that they haven’t exposed via mistakes?
Detection Efficacy Bias
Over the years covered by the research study the capability to detect 0days has improved significantly. The number of 0days detected would be expected to increase over the duration of the studied period.
Some 0days, such as Uzbekistan’s, were discovered due to user error. There is no way to quantify accidents.
Conclusion
This dataset is too flawed to be of any value whatsoever.
The assumptions behind this report, that the data reflects an accurate approximation of 0day use globally, or by each threat actor over time, are baseless. The data is thoroughly contaminated by biases.
Casualty bias — the opposite of survivor bias. Only failures are counted. Combined with selection bias, only sampling a subset of threat actors’ 0day capabilities — those that were detected by antivirus or threat intelligence companies. The result is an over emphasis on countries that conduct cyber operations in the US and Europe.
This Map Shows the Global Spread of Zero-Day Hacking Techniques | WIRED
Leave a Reply