Nearly 80 percent of security products fail to perform as intended when
first tested and generally require two or more cycles of testing before
achieving certification, according to a new ICSA Labs report. The
"ICSA Labs Product Assurance Report" – a first-of-its-kind study
co-authored by the Verizon Business Data Breach Investigations Report
research team – details lessons gleaned from testing thousands of
security products over 20 years.
The report found that the No. 1 reason why a product fails during
initial testing is that it does not adequately perform as intended.
Across seven product categories, core product functionality accounted
for 78 percent of initial test failures — for example, an anti-virus
product failing to prevent infection or an IPS (intrusion prevention
system) product failing to filter malicious traffic.
The failure of a product to completely and accurately log data was the
second most common reason security products do not perform as
intended. Incomplete or inaccurate logging of who did what and when
accounted for 58 percent of initial failures.
The report findings suggest that some vendors and enterprise users
consider logging a nuisance and merely a "box to check." According to
the report, logging is a particular challenge for firewalls. Almost
every network firewall (97 percent) or Web application firewall (80
percent) tested experienced at least one logging problem.
The third most significant reason for product failure is the finding
that 44 percent of security products had inherent security problems,
including vulnerabilities that compromise the confidentiality or
integrity of the system and random behavior that affects product
Even though it can be a demanding process, certification with a
trusted, established third party is critical to verifying product
quality, states the report. Product categories studied were:
anti-virus, network firewall, Web application firewall, network IPS,
IPSec VPN, SSL VPN and custom testing.
"Our goal is to help vendors develop more secure products," said George
Japak, managing director of ICSA Labs and a co-author of the report.
"When a product fails, we encourage vendors to view that as an
opportunity to improve the product before it goes to market. In
addition to benefiting the security industry, this open exchange of
information can greatly benefit enterprises by providing them more
reliable and available information to make educated product purchasing
and use decisions."
Only 4 percent of Products Pass First Round
The ICSA Labs testing and certification process is rigorous; only 4
percent of products tested attained certification during the first
testing cycle. However, 82 percent of products resubmitted for testing
eventually earn ICSA Labs certification. Once a vendor earns
certification, products are required to undergo ongoing testing to
Japak said, "The question I ask vendors is: ‘Who would you rather have
find an issue in your product — ICSA Labs in a safe testing
environment or a criminal in the real world?’"
The study also identified several other issues with the security
products tested including poor product documentation and problems
involving patching — a product’s ability to accept updates correctly.
Vendors are urged to read the "ICSA Labs Product Assurance Report" in
full, which contains aggregate data on problems and solutions
identified through rigorous testing. In addition, the report contains
common pitfalls to avoid, recurring deficiencies to be aware of, and,
factors that contribute to testing success.
Recommendations for Enterprises
The report recommends steps companies should take before purchasing and
using security products. Among the recommendations are:
* Demand quality. The market typically prefers feature-rich
products over performance. If end-users demand quality, vendors will
* Be suspicious of performance claims and numbers. Vet them. Question them. Be an educated, cautious buyer.
* Choose more established products over new. New products have
more problems; often times the kinks have not been worked out yet.
Choose more established products when possible. If a new product is
required, make sure the product is certified.
* Choose simplicity over complexity.
* Use certified products! This substantially reduces risk by
ensuring that products meet objective, publicly vetted criteria. When
shopping for a product, place certification at the top of the list.
When using a certified product, keep up with whether the certification
is current. Encourage vendors to maintain the certification after
end-of-life if the product is still in use.
* Prefer vendors that certify their products, and that participate
in industry and ICSA Labs consortia and other standards bodies.
Working with consortia and industry organizations is a positive
indicator for product quality. Technology held to an industry standard
improves both features and reliability.