Response to Recently Released 2014 NSS Next-Generation Firewall Comparative Analysis

5,153 people reacted 0 3 min. read


Category: Firewall


By: Lee Klarich, SVP Product Management at Palo Alto Networks

October 31, 2014 Update: We invite you to view an updated 2014 NSS Labs next-generation firewall test, in which Palo Alto Networks achieved 92.5% security effectiveness rating. Click here to see the updated report and read a letter of confirmation from NSS CEO Vikram Phatak.


As you may have seen, NSS recently released a report on Next-Generation Firewalls.  In their report, they purport to show Palo Alto Networks not scoring well.  There are a few important things to know about this report as Palo Alto Networks takes security and performance very seriously.

  • Palo Alto Networks intentionally did not participate in the 2014 NSS Next-Generation Firewall Comparative Analysis report that was recently published.  This means that unlike all of the other vendors in the report who configured and tuned their products specifically for this test, there was no input from us on the configuration of our device.
  • The reason we did not participate in this test is that over time we have come to believe that the NSS model of allowing vendor test tuning prior to public test is a “pay to play” approach and produces questionable objectivity and accuracy in results.
  • One year ago, we did participate and scored 96.4%. Since then, we have continued to invest even more in the sophisticated security capabilities of our products, as evidenced by our contributions to discovering Microsoft vulnerabilities – exceeding the findings of nearly every other network security vendor in the industry – and most recently our response to Heartbleed and Shellshock / Bash bug.
  • We take the efficacy of our Next-Generation Firewall very seriously.  We are trying to understand why they could have come to such a drastically different result compared to the same tests run against the same technology in 2013.  Importantly, the issues they’ve raised have never been observed in other tests conducted internally or with our install base of over 19,000 global enterprises.  It is also interesting to note that they say that we updated our OS in that time and broke the technology.  There is no basis for that claim as best evidenced by the fact that in the last year alone we added almost 6,000 new customers all of who have done their own stringent and detailed testing of our products in their mission critical environments.

We are committed to developing the most sound security technology that is designed to prevent sophisticated attacks. If you have any questions for me, I’d be very happy to talk to you directly, please email me here:

8 Reader Comments

  1. Thanks for posting this. As a Palo Alto Networks customer I was concerned by the NSS findings. However, I assumed that there was more to this story. I have deployed Palo Alto Networks firewalls in 5 offices around the world and have never had an issue with a threat making it past them (what a change from Juniper). The NSS CEO blames the results on software revisions which I don’t buy. All our firewalls are running the latest version of PAN-OS and each version offers more security, not less. The software is so good that during our last security audit our auditors were frustrated by the level of protection the firewalls provided. The auditors even asked me to disable some protection features so that their tests would run properly. Greg Young at Gartner said “Generally NSS stuff has been pretty good, but I need more information to help me understand this one.”. I agree and this posting is very helpful. It sounds like the firewall wasn’t configured correctly and the playing field wasn’t level.

  2. The days of bench testing in a lab and putting faith in the results appears to be over. As anyone who manages a NGFW knows there are many detection and protection mechanisms at your disposal. Only when you collectively look at all pieces together can you make an assessment as to the effectiveness of the product. Since all products in the marketplace differ in some manner, the effectiveness of component testing to rate a product against others in the marketplace has limited value. Since in the real world we use the entire suite of capabilities at our disposal it is not realistic to compare products by only looking at a subset of features. The real test is to put the product in your environment and test it either against what you currently have or with test conditions relevant or appropriate for your environment.

  3. NSS should not charge companies to bring in their Firewalls/IPS to test their features and functionality. Each Security Company should setup their Firewalls/IPS themselves based upon their best practices. Then NSS should run their tests and deliver a non-biased report based on the facts of the test.

  4. Reading the report the Palo Alto NGFW performed very well in protection against malware and other threats but did not protect against a few specific evasions, RPC Fragmentation and IP Fragmentation + TCP Segmentation. I understand this is likely an out of box configuration with security features turned on but not tuned and I believe these holes can be protected, but my concern is that a basically configured NGFW, which most customers will have, is not protected from these evasion attacks.

  5. Whether the numbers were skewed because Palo Alto didn’t get to “tune” their firewall, the NSS report was still correct in putting them on the left side for higher price per filtering bandwidth.

    They way I look at it, Palo Alto Firewalls are the Apple of Firewalls, they have a nice interface and are easy to use, and everyone pays a premium for it. When the real work needs to be done businesses go with Windows or Linux (aka Checkpoint or FortiNet).

  6. NSS Labs response to this article makes interesting reading?

  7. I have two points to make about the Palo Alto Networks – NSS Labs controversy. One, the NSS Labs Next Generation Firewall Comparative Analysis simply does not pass the smell test. Two, it’s not even clear to me that all of the firewalls tested are actually Next Generation Firewalls.

    Regarding my first point, I am a Principal at Cymbel, a Palo Alto Networks reseller since 2007. We work with some of the largest organizations in the United States who have put Palo Alto Networks firewalls through extremely rigorous evaluations for extended periods, and have then deployed Palo Alto firewalls for many years. NSS Labs seems to be saying that all of the people in these organizations are idiots. This does not make sense to me.

    In addition, NSS Labs seems to be saying that the Gartner people, who speak with far more firewall customers than we do, and place Palo Alto Networks in the Leader Quadrant and furthest to the right, are also morons. I’m not buying it.

    Regarding my second point, at a more basic level, what is NSS Labs’ definition of a Next Generation Firewall? Since I am not a paying customer of NSS Labs, I don’t know. Let me start with the definition of a firewall – the ability to establish a Positive Control Model. In other words, define what network traffic is allowed, and block everything else, i.e. default deny.

    In the 1990’s, this was relatively easy because all applications ran on well-defined port numbers. Therefore you could define policies based on port numbers, IP addresses, and protocols and be assured that you had full network visibility and control.

    Starting in the early 2000s, this well-behaved order began to break down. Applications were built to share already open ports in order to bypass traditional stateful inspection firewalls. By the mid-2000s, there were hundreds, if not thousands, of applications that share ports, automatically hop from port to port, and use encryption to evade traditional firewalls. Thus, these traditional firewalls were essentially rendered useless, and could no longer support a Positive Control Model.

    So a new type of firewall was needed. In order to re-establish a positive control model, this new type of firewall has to monitor all 65,535 TCP and UDP ports for all applications, all of the time. In other words, a firewall that enables you to define which applications are allowed, regardless of the ports on which they runs and block all of the others, known or unknown.

    Furthermore, a Next Generation Firewall must enable you to lock a specifically allowed application to specifically allowed port(s), and prevent any other application from running on the port(s) opened for that specific application.

    Palo Alto Networks, in 2007, was the first company to ship this new type of firewall that, in 2009, Gartner called a “Next Generation Firewall.” Since then, virtually every firewall vendor in the industry now uses the term. But in reality, which ones actually meet the real definition of a Next Generation Firewall?

    I would recommend that NSS Labs release the details of its testing methodology for all to review. By keeping their testing methodology behind a paywall, they are simply feeding into Palo Alto’s “pay to play” contention.

  8. “When the real work needs to be done businesses go with Windows or Linux (aka Checkpoint or FortiNet).”

    Man, I needed some humor to start the day, thanks for that. Nothing like saying the dinosaur and low cost knock-off are what gets real work done.

Got something to say?