Today, I was scanning through various industry blogs when I stumbled upon an entry from a certain competitor.  What was interesting was that under the veil of improving testing quality, the blog openly admitted that the organization in question had been trying to play tricks on competing organizations just to position itself more favorably among the media.

The organization explained that it deliberately created clean files and added fake detections in order to “show” that other vendors copied it. This was a risky decision. Across the industry, research organizations share a level of trust and participate in sample-sharing programs in order to protect customers, which for Trend Micro, is what always comes first.

Aside from the organization’s prank, we were very pleased that the other resounding message that came from the blog post was that it finally understood and supported the message Trend Micro has been promoting for a long time now—the need for change in testing methodologies to include real-world testing such as those delivered by NSS Labs.

The need to change testing methodologies was also a primary reason for the foundation of the Anti-Malware Testing Standards Organization (AMTSO), which aims to come up with more realistic and useful benchmarks.

This story really shows just how influential the media is on the antivirus industry in that even a respected vendor should manipulate detection rates just so it can positively position itself with the press rather than focus on its customers.

But another more positive learning is also that the path that AMTSO is taking is the right way. Pure detection rates based on numbers or one-to-one comparisons are yesterday’s methods when verifying the value and performance of a security solution.

Customers need holistic reviews, giving them a real-world scenario-based feedback about what different solutions can offer them instead of pure “I detect more then you” headlines. I am glad to see that testing organizations like NSS Labs, AV-Comparatives, and AV-Test meanwhile understood and started to pick up these principles.

Post from: TrendLabs | Malware Blog – by Trend Micro

On the Trustworthiness of the AV Industry and AV Tests