One of my favorite stories is of a pen-test team who were brought in and situated next door to the SOC (Security Operations Centre); and after a week on-site they were invited for a tour of the SOC where they queried a series of alarms [that they had obviously caused] only to be told “oh that’s normal, we’ve been getting these continuously all week”.
People perform penetration tests (pen-tests) for a multitude of reasons; “I inherited a budget with an annual pen-test” or “it’s required by the audit committee” are the most common. Security teams use them to justify their budgets and hopefully show how good they are, but in the worst of cases people are paying consultancy rates for a simple vulnerability scan; but rarely are they used to find the “known unknowns” or even the “unknown unknowns”.
“because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns”
- United States Secretary of Defence Donald Rumsfeld, 2002
Aside from a specific pen-test to test, say, a new website before it goes live to the Internet, pen-tests will vary in scope and quality.
So, what does a bad pen-test look like? All pen-test companies will tell you of the phone calls that start “we need a pen-test” but when quizzed they have no idea what needs testing – they just need a “standard” pen-test because they’ve been told (by their management, auditors, audit committee that) they need to perform one – the “tick in the box”.
Conversely a good pen-test will be part of a series, each testing specific aspects of security – against a plan that is informed by a business-wide threat-assessment process that strives to understand who would want to harm your business and/or steal your data combined with the motivation behind your various adversaries; everyone from script-kiddies who do it for a laugh, organized crime who want to ransom your data, to the competitor funded by state intelligence who wants to steal your pre-patent intellectual property.
The best pen-tests are those that augment a security regime where the basics are already in place and part of everyday life. Why pay for a pen-test that conducts a one-off manual vulnerability assessment, when for probably less money you can implement continuous automated vulnerability assessment?
Going back to my initial story; every pen-tester will tell you how they’ve NEVER failed to get physical access to a building; and almost all companies run flat internal networks; therefore, simply assume they will get in, and simply give them a room with a network jack for the duration of the test.
The best pen-tests, in my opinion, use your threat assessment to identify “targets” or “flags to capture”, give the pen-testers access to the output from their automated vulnerability scanning system as well as physical access to their network, and then pay a bonus for every flag they capture, web-site they actually modify, or senior executive they manage to socially engineer. Because only this type of pen-test will help identify both the “known-unknowns” and “unknown-unknowns”.
Finally, before you start any pen-test you need to understand what you will do with the results – all pen-tests will expose a level of failing – such is the nature of modern IT systems. If you are able to pat yourself on the back and report to management “they did not manage to get in” then you either designed the pen-test wrong or hired the wrong company to perform the test.
