The Trouble with Pen Testing

Too Many Pros Rely on Tools, Not Strategy
The Trouble with Pen Testing
Penetration testers tend to rely too much on vulnerability assessment tools, rather than thinking strategically, says Scott Laliberte, managing director of Protiviti.

Often testers will analyze a system using one or two tools. While the tools are helpful, they have limitations, Laliberte says. Assessment tools can only go so far. Being able to adapt a strategy on the fly and go in a different direction if necessary requires smarts.

"There are still a lot of inexperienced people out there that are passing themselves off as experts," Laliberte says in an interview with BankInfoSecurity.com's Tom Field [transcript below].

Besides knowing how to use assessment tools, pen testers need to be able to articulate their findings to business executives. "It can be frustrating when you are trying to talk to a C-level executive, who is the person that has to understand what the core issue is and be able to decide whether or not they want to spend money ... to mitigate the risk associated with that vulnerability," Laliberte says.

The scope of a test is often overlooked also, says Laliberte. The operating system layer for example is often done very well, but layers like web applications, the underlying database and external entry points, such as through the Internet, need to be tested also. Often, unique vulnerabilities exist within the web application layer, and testers overlook those areas in their risk assessments.

Not being prepared and not knowing what you're doing can lead to overlooked gaps in security. "A poorly executed pen test in my mind is even worse than not doing one at all because you get that false sense of security," Laliberte says.

In an exclusive interview about penetration testing, Laliberte discusses:

  • What penetration testers are doing right and wrong today;
  • How organizations should approach penetration testing;
  • Advice for organizations looking to improve how they approach pen testing.

Laliberte is a managing director in the Philadelphia office of Protiviti Inc. providing clients with Information Systems Security Services. He leads operations for Protiviti's Global Information Security Practice.

Laliberte is a published author, accomplished speaker, and quoted subject-matter expert in the area information systems security. He co-authored a book about penetration testing and information security called Hack I.T., published in February 2002 by Addison-Wesley Publishing. Laliberte's second book, Defend I.T., is a collection of case studies in information security and was published in the spring of 2004. Prior to becoming a consultant, Laliberte was an information systems security officer for the United States Coast Guard.

TOM FIELD: To get us started, why don't you tell us a little bit about yourself and your work please?

SCOTT LALIBERTE: I'm managing director with Protiviti, which is a global business and risk consulting firm. I specialize in the areas of security and privacy, and really in helping organizations to identify technical and data-related risks, and identifying solutions to appropriately mitigate those risks. Those risks could be technical in nature. They could be regulatory-type risks, or just helping them manage their brand and making sure they don't incur brand reputational damage. The main areas that I work in include penetration tests, vulnerability assessments, risk assessments, reviews of third parties to understand the risk that those third parties may introduce to the organization, data privacy reviews and helping you to identify, classify and protect data and the access associated with it. Then I help to comply with different regulations such as PCI, the Payment Card Industry Standards, HIPAA for healthcare-related organizations, Gramm-Leach-Bliley for financial institutions and then various state data breach and data protection laws, as well as a lot of the international data protection regulations.

Hack I.T.

FIELD: Very good. Now you co-authored the book Hack I.T. What is the premise of the book?

LALIBERTE: Hack I.T. is a book about penetration testing, really written for technical folks who don't necessarily have a security background so that they could perform and understand how penetration testing works. We got the idea for the book, my co-authors and I, because we would go in and often conduct these penetration tests and the people we were working with didn't really understand what we were doing completely. A lot of times we would just find a lot of low-hanging fruit. We could teach folks the basics of penetration testing and they could take care of all of this low-hanging fruit. Then, when we come in we could spend more time really doing in-depth testing, looking for the more complicated, and more complex-type vulnerabilities. So what we did is we went and wrote Hack I.T., really trying to write it at a very basic level so if people didn't have a background in it they didn't need a ton of technical experience. They could get in there and perform that low-level testing. The other part of it is we felt that the more that our clients understood about how their adversaries, the hackers, work the better they were able to defend against it. By understanding how an exploit is done or how a hacker profiles you, you can better start to think about ways to defend against them. That was the premise of the book.

From there, it leads to the second book, Defend I.T., which was a collection of case studies. The feedback we got from people on the first book was that they loved the book and they especially loved the case studies. So we said why don't we write a whole other book just on case studies, and that's what Defend I.T. was.

What Pen Testers Do Right & Wrong

FIELD: So in your experience, what do you find that pen testers are doing right today?

LALIBERTE: It has become a pretty big industry and a lot more people are playing in it. And obviously there are those that do it well and those that don't do that great of a job. I think one of the big advantages going today are that there are a lot more tools that are out there, and the tools have gotten a lot better. They are a lot better in helping to identify the vulnerabilities. The training I think has gotten a lot better. There's more training that's available out there. That has helped bring up the skills of most of the people doing the test.

Then there's new certification in the field, such as the OSP, Offensive Security Professional. That test really is like a hands-on test. You get a log-in and you have to go in and successfully perform certain tests, penetration-testing activity. It really helps to ensure that the tester can actually execute and pass successfully, and they don't just understand the book sense, or the book parts, of pen testing. That has helped raise the skill level of the testers that are out there in providing a better service to our clients.

FIELD: So the flip side of it, what do you see pen testers doing wrong? In other words, where do you see some weaknesses that have shown up?

LALIBERTE: The double-edge sword of the tools getting better is that we tend to see a reliance by some people on those tools. We will have some folks come in and they run one or two tools, and although those tools are good they have limitations. All the tools have limitations. I've always said and always thought that the mind is the best penetration tool out there. If you can get a person that thinks strategically, typically somebody that's like a good chess player, those tend to be really good testers because they're thinking about different avenues and they're able to respond to the different types of avenues that you see to get into the company. The tools are able to identify many of those avenues, but then they're not able to actually take that down to the next step of compromising the next system down the line and then being able on the fly to adapt the strategy and go in a different direction if that's necessary.

The other part is there are still a lot inexperienced people out there that are passing themselves off as experts. You'll find somebody that maybe knows how to run a tool and they then say that they can conduct penetration tests, and really they are just running a vulnerability assessment tool. That's a big part of it. The other part I found, even with some of the really good technical testers that are out there, they can't always articulate the findings in business terms that business folks can understand and comprehend. It can be frustrating when you are trying to talk to a C-level executive, who is the person that has to understand what the core issue is and be able to decide whether or not they want to spend money, sometimes a significant amount of money, to mitigate the risk associated with that vulnerability. You really need to be able to communicate that in terms that a business person can understand.

I will give you a great example of one that we've done recently, where a person had found a vulnerability that could lead to the complete compromise of a system on this client's network. The person was happy about it. Our tester was happy about it and I patted him on the back. I said why is that a big issue? With the completely compromised system, what does that lead to? In reality it wasn't necessarily a big business issue because it was an inconsequential system that had no access to anything else and it was segmented away from everything. It didn't really have any production impact. Versus a different one that we found, it was a production site and we could change pricing of the customer's data on that site. Being able to articulate that is a really important trait that I don't find everybody in this industry is able to do well.

Approaching Penetration Testing

FIELD: That's a good point. In our organization we speak to financial institutions, healthcare organizations and government agencies. What are the ways that all types of organizations ought to be approaching penetration testing today?

LALIBERTE: The pen test really needs to be carefully planned and executed in a controlled manner. The approach should be changed from year to year. You don't want to be executing the same test with the same approach and attack vectors over and over because at some point they're going to close those avenues that you found. You really want to be looking for new avenues and new ways of getting in there. Make sure that the test is comprehensive, that it includes all layers of technology and systems that are out there. Many times we'll find some tests that are conducted really well at the operating system layer. That's a layer where probably the most mature tools exist, so there are a lot of people that can cover that layer well. But make sure things like the web application layer, where a lot of unique vulnerabilities lie, is properly tested. Make sure that the underlying database, which is an area that a lot of people don't usually understand how to secure very well, is included in the test. The database area is one that we really have seen people miss and it's because there's not a lot of good skill in that area. Also, DBAs [database administrators] typically are concerned with databases and making sure they operate effectively, not necessarily securing them.

The other one is also making sure that you are testing from an external perspective so you'll get that threat from an external attacker or the Internet. Have internal pen tests, where you are looking at somebody that might have penetrated the physical security of the organization, or an internal disgruntled employee all the way down to the social engineering aspect. You're looking at if somebody is able to trick or dupe your employees into doing something that they shouldn't that leads to a compromise. Making sure that you have all of those avenues covered and you've rotated the amount of emphasis that you put on those different avenues is really a way that you should go about doing it.

FIELD: Is penetration testing ever a bad idea for an organization?

LALIBERTE: If it's done improperly it can be a bad idea, or have bad consequences. There's always a risk when you perform penetration testing that you can disrupt production or you can bring systems down. With a poorly executed or poorly coordinated test, that risk tends to increase. That's one of the negative effects that you could have. The other one is if you don't have people that are really well trained and have good skills in this area. If you have a penetration test performed, and in actuality it wasn't done very well, but the organization gets a false sense of security that everything is great, there may be significant holes that exist out there. A poorly executed pen test in my mind is even somewhat worse than not doing one at all because you get that false sense of security.

The other one is going in with a competitive, or a contentious, attitude. A lot of times we'll find where an internal audit within an organization sponsors the test and they're out to get IT or there is a perception by IT that they're out to get them. Then that leads to people either not cooperating in the test or significantly objecting to the issues that are raised in the test, trying to downplay them rather than really going at it cooperatively with them. We're trying to make the organization stronger and we're trying to find the holes and prevent the bad guys from getting in and doing damage. We're going to work together to do that. That is the other one, going in with a poor attitude or a combative attitude. That can really lead to a poisonous, contentious relationship within the organization.

Improving Penetration Testing Efforts

FIELD: Final question for you. For organizations that are looking to improve how they approach penetration testing, what advice would you give to them?

LALIBERTE: Plan, plan, plan. Make sure that you plan the pen test well in aspects of what you're going to cover, how you are going to cover it, when you are going to conduct the test, who will know about the test and who will not. That's a pretty significant portion if you want to try what we call a stealth move, where you're trying to determine if the organization's detective capabilities would be able to identify the test and respond accordingly.

Make sure you've got the scope set properly, you know what systems you're going to include and which ones you're not going to include. Make sure that they understand and communicate the business risks so that they articulate the findings that come out of that test in a way that executives can understand. And hopefully they can gain the support that they need to fix those vulnerabilities. I always tell organizations, you really have to look at the cost benefit because if you go to your executives with the argument that you want to fix a risk that only has $100,000 risk associated with it, and you're going to spend a million dollars to fix it, you're going to loose credibility. Nothing else you say will be taken seriously, so make sure that they put it in that context.

Then, make sure the pen tests for follow-up activities are conducted on a regular basis. One area I find some organizations fall short is that they conduct the penetration test. They think they're fine. They fix the issues hopefully, and then they don't conduct any other follow-up activity for a year or two. Really what you want to do is kick forward that momentum you had from the penetration test and the fixes that you put in and re-perform a vulnerability assessment or some scanning activity. Scans alone do have some use. They're very good for following up on the issues that are identified in the test that you think you fixed, to make sure that they've been closed. Then supplement those scans periodically throughout the year, and then again either on an annual or bi-annual basis you are re-performing the test. Make sure it's a continuous life cycle rather than a specific test that's done at one point in time.


About the Author

Jeffrey Roman

Jeffrey Roman

News Writer, ISMG

Roman is the former News Writer for Information Security Media Group. Having worked for multiple publications at The College of New Jersey, including the College's newspaper "The Signal" and alumni magazine, Roman has experience in journalism, copy editing and communications.




Around the Network

Our website uses cookies. Cookies enable us to provide the best experience possible and help us understand how visitors use our website. By browsing govinfosecurity.com, you agree to our use of cookies.