The Good HackerWords of NSA's Tony Sager
Most civilian agencies have not conducted blue/red team analysis, but it's been a common practice for years within Defense and intelligence agencies.
Among the leading organizations conducting blue/red-team analysis for the Department of Defense, intelligence agencies and some units at the Department of Homeland Security is the three-year-old Vulnerability Analysis and Operations Groups at the National Security Agency.
Tony Sager serves as the group's chief, and he says such testing requires far more planning between his organization and client agencies than most people would expect. Sager spoke with Information Security Media Group's Eric Chabrow.
ERIC CHABROW: Legislation before Congress would require agencies to employ red-team attacks to determine how secure federal agency's IT systems are. How do these teams work, and how effective are they?
TONY SAGER: Well, you don't want to think of things like red and blue team testing as events in isolation. You want to see them as part of a process of continuous improvement. You don't get magic when the red team comes, you get a deep understanding of scenarios under which you might be vulnerable. For us, we have these standing organizations that are the red team and the blue team, and we compose a team by job, that is, to support a particular job that we do for a particular customer. And, there is a lot more work than you might think, negotiating up front what that job will be. It's not a free form, turn a bunch of people loose. There is a lot of consideration given to what is it the customer would like to learn, or what are their objectives, for the purpose of, say, bringing in the NSA red team? You know, is it to test parts of their defenses? Is it to test the readiness of their personnel? Is it part of a larger military exercise? Do we want to understand the impact that adversaries could have on our ability to execute a military operation? So, there is a lot of discussion up front about ground rules and expectation, but there is a lot of careful planning involved in this, before we would actually turn humans loose to do testing.
CHABROW: Please walk us through a scenario to show how the process works. Who meets initially with whom, and then where do you go from there?
SAGER: Our nonmilitary support is probably more frequent on the blue-team side than the red-team side. There is usually an initial inquiry, "We'd like some help, say, from the NSA blue team or red team." They'll have some series of discussions to kind of scope the job out. That often involves a site visit, or a pretty detailed technical discussion about the kind of technology to be found, and a discussion of the objectives. What is it you're trying to learn? What problem are you trying to get a handle on? There would be a discussion about timing, because our folks are in very high demand, so it's almost never the case, except in a crisis that we could be out there, physically, virtually overnight.
There is a lot of discussion about how do we synchronize the timing. When you test networks, all kinds of things can happen. There's also preparatory work on the customer's end, to be ready for a blue team or a red team. You don't want to be testing when you're in the middle of upgrading parts of your system, for example. You want to time the work, so that the right people are around and the system is in a state where it can be instrumented and stable, and you can use it as a launch point to solve these problems.
For us, this negotiation up front gets embodied in documents. We have a formal agreement on the ground rules, because of the nature of our work, and who we work for, there is very careful oversight of the work, and review by general counsel on both ends, to make official the agreement that we have worked out over the work that will be done.
For blue-team type work, it's typically something between a week to two weeks hands-on, onsite kind of work. Blue-team work, again, is fundamentally different because it is very open and cooperative and hands-on, typically onsite, working with system administrators, working together to scan and understand the environment and develop recommendations. It's very open, in that sense. We'd leave them with some recommendations right up front. There would probably be a fairly quick turnaround, kind of an initial report with a more formal report to follow, sometime later. Some customers are repeat customers, so there's maybe follow up from previous work that we've done for them. Again, the goal is to make this something that helps them focus on the important problems that they've got in their net, to make an improvement for them.
The red-team work is fundamentally different. It's much more focused on playing mock adversary. So, for the blue team, the goal is to have a very comprehensive wide-ranging understanding of problems. The red team, you might say, is much more precise, surgical, focused. Can I demonstrate an exploitable scenario, we would call it, or take advantage of a vulnerability, to demonstrate a particular thing? For example, could I effect the command and control of forces? Could I remove this data? Could I corrupt this data in this context? It wouldn't be near the breadth, but it would be more deeper and closer to it, and adversarial system on the red team.
CHABROW: What skills are needed to be a member of the red team or the blue team? Are these skills interchangeable?
SAGER: There are people that have done both. They're not typically interchangeable, I would guess.
Let me start with the red team. There is this sort of adversarial mindset that we are looking for; there are the basic technical skills. To do all this up-front work that I've talked about, I think most people would be surprised. We (don't) just turn a room full of hackers loose on somebody. There is a significant technical infrastructure that we have to have in place just to execute red-team work. We've got planners, we've got the managers that are involved, to handle all the negotiating and the working through the technical skills. In terms of technical skills, we've got people that understand the technology of the network and how it's vulnerable. They range from the most creative to folks that know the kind of attacks that are typical, and they can kind of take the attacks as we understand them and apply them. And then, there are people who can come up with attacks that have never been done before. They are often involved in operational jobs, but they have a back room job, too, like to figure out new ways to go after a system.
We have an understanding of all the mainline technologies we expect to see in our customer base, and how they can be exploited. Exploitation is more than understanding a flaw in a piece of software, for example, or a vulnerability in a piece of technology. The steps from "I know there's a flaw in a piece of software" to "How do I put together a series of steps to take advantage of that in the context of, for example, a DOD operation network," is quite a bit of work, and there is a lot of thinking, and you have to understand what is critical. How does the IT support the customer's operation, and how can I undo their security, is more than just understanding the flaw, to turn the flaw, for example, into a tool is often a highly complex technical step? To execute that tool in a stealthy manner, and in a way that evades detection and so forth, requires quite a bit of skill. There's a whole range of folks that it takes to pull that kind of work off.
We typically think of three bins. I think of skills for Windows and Unix and kind of infrastructures, which is kind of the network, itself, and all the things that don't fall into the other two bins. Those folks typically are computer-science degreed, many are specialized in one particular area or another. But, most of them have a pretty broad knowledge of technology, and how it implements operations in our customer setting, and, how to exploit it. They're very good with the main line tools, to scan and find problems. We use a variety of tools, commercial and in-house. We spend a lot of time sort of integrating across tools, to make sure we get the best possible view of what those problems are.
CHABROW: Do you perform blue-team assessments and red-team assaults together or are they usually performed separately?
SAGER: They work for the same line manager now, which was a fairly recent phenomenon for us. But, they're not closely synchronized.
The customer base is very wide for both, and the objectives are different. We are often dealing with the same commands, but the work isn't necessarily synchronized, like the blue team has to go in before the red team, because the objectives are dramatically different. There is a lot of repeat, but not necessarily synchronized (assignments).
The purposes are really different, too. Think of the blue team as trying to get a wide and deep understanding of vulnerability, to start the countermeasure process. Think of the red team as the dramatic demonstration of how we are vulnerable. You do that kind of as a test, or you might do that because decision makers are unsure of how bad their problems are, or you need to change someone's mind, perhaps to spend money, or to make big decisions about what their biggest problems are. We wouldn't claim that a red-team test, for example, gives you the same breadth of understanding that a blue-team job does. They have really different purposes.
CHABROW: How do you create the red teams? Where do you go to recruit team members? How many people are on the teams?
SAGER: It's bigger than you might think. Eight to 15. They are supported by a significant number of people around infrastructure and planning and so forth. But the technical people that work the operation is in the dozen range. Our workforce is a mix of military and civilian, all government employees; we do have some contract help in some of the infrastructure and pool development, I believe. And I think that makes us an anomaly in the DOD and the intelligence community, that we are entirely government employee and military, uniformed or civilian. Our teams are built on a per-job basis, so they all work in the same line organization. Once you sort of figure out what the objectives are, the team is composed, both based on availability, but also the skills that people have, to execute the objectives of the exercise for the red team.
CHABROW: Is this a full-time job for team members, or do they have other assignments?
SAGER: They're all full-time assigned to the red team or to the blue team. There are other things that they do on any given day, but this is their full-time occupation.
We recruit directly into there, or we get folks out of the military. Most require some technical training and development, once they wind up onboard with us. ... I think we are, technically, the certification authority for red teams across the DOD. We determine what constitutes an official red team for the DOD, and we develop that, based upon some years of experience.
The red-team model and blue teams are actually, by government standards, pretty mature business models. We've been doing this for a number of years. We have a pretty good notion of how we bring people in and train them and develop them to manage the turnover of military, uniformed military personnel, so we've always got fresh folks coming into that job who need to be trained and there is a constant refresh cycle around new technology, and so forth.
There is a part of this business that is kind of unique to us, and where the military really helps us is, we're not attacking networks in the abstract, we're looking to achieve an operational effect. It's just the fact that the military folks often come in with a pretty good understanding of what is it they're doing out there? What is the operational mission? How does information technology support that? And whether the critical vulnerabilities that would cause problems with the IT to disrupt the mission objectives? In the business sense, it's not just about going after technology. Right? Even in the business sense, you'd have to worry about the integrity of data, the availability of it, the confidentiality of it. You would want to have your red team understand enough about those things to know what is really important, what really matters, and how does IT synchronize with the important business issues. Otherwise, you're just going after the technology. You may not be hitting the really critical things.
CHABROW: On the red team, how many members are from the military and how many are civilian?
SAGER: On the red-team side, about two-thirds uniformed military, and about a third civilian. It's probably, percentage the other way on the blue-team side. We're mostly civilian on the blue-team side, with a small number of military on the blue-team side.