Employing Cybersecurity Index as a Metric

Gaining Insight from Understanding the Changing Threat Space

Information sharing has been trending upward, according to data from the monthly cybersecurity index. And while it's not surprising, it is "gratifying," says IT security practitioner Dan Geer.

Geer, who along with Mukul Pareek created the monthly cybersecurity index over a year ago, use the tool to gauge how security practitioners are observing the threat environment. And one recent takeaway has been the increase in information sharing.

"The state groups, consortiums, organizations, they're making a noticeable impact," says Pareek, a risk management expert, in an interview with Information Security Media Group's Eric Chabrow [transcript below].

"Generally, information sharing with industry participants is a low-key activity that does not get a great deal of publicity or notice in the press, so you would not know about it otherwise," he says.

Geer doesn't find the increase surprising, but says the information is reassuring. "Obviously enough, if you don't share information you will never know whether you're a target of choice or a target of opportunity," he says. "By sharing information you can learn such things as that, and we're hoping in fact that's one of the things that our respondents get out of this [survey]."

In the interview, Geer and co-publisher Pareek:

  • Explain how the index can be used by organizations to defend against cybersecurity threats;
  • Discuss changes they plan to make to the index to make it more relevant;
  • Interpret the steady rise in the index since its launch in April 2011 (see Measuring the Immeasurable: IT Security).

Geer is chief information security officer of In-Q-Tel, an independent strategic investment firm that identifies emerging technologies to support the missions of the U.S. intelligence community. In-Q-Tel was not involved in the creation or management of the cybersecurity index.

Pareek has worked as a vice president at a major investment bank and as a senior manager for a Big 4 accounting and consulting firm.

Cybersecurity Index

ERIC CHABROW: We first spoke just over a year ago, shortly after you launched the index of cybersecurity. You set the benchmark in March 2011 at 1,000. Since then, the index has increased by about 30 percent. What does such an increase mean?

DAN GEER: Strictly speaking, it means that the people who are the respondents, who are, as you say, professionals in the field, have viewed the overall risk as seen from their individual positions as having increased substantially over time, and to increase on a relatively regular basis as well. Unless you want to argue that the experts in the field are in some way misinformed, it's telling us that the risk in the cybersecurity arena is itself growing. By asking people who have the kinds of job descriptions that you mentioned, we think we're getting the best information available for something that's otherwise largely immeasurable.

Month-by-Month Increases

CHABROW: Looking at the chart of the index over the past year, I see a line representing the trajectory of the index angling upward at about 45 degrees, and that suggests as your results show a steady month-after-month increase of the index of about two percent, give or take a few tenths of a percentage point. Why are the month-to-month increases in the index so consistent?

GEER: That's a good question, one that both Mukul and I have talked about a fair bit. Let me just say - one thing that the widely published number doesn't reveal is that the contribution to that increase that you see there month-over-month is not always the same factors. We ask 20 questions or so and those 20 questions are then pooled on an even basis. They're not weighted. They're pooled on an even basis to form the overall index of cybersecurity, and in any given month what's the most influential issue - whether it's malware or hacktivism, counter parties or any of the other things we ask about - that has varied a great deal. What hasn't varied or what's as you see there in that picture, relatively steady, is the increase. On the other hand, the rate of increase does vary month-by-month and the constituents of what contribute that rate of increase vary even more month-to-month. I would say that this is sort of an open question yet as to what's going on here, other than of course methodologically speaking what's going on here is we're asking experts what they see.

CHABROW: And what's your takeaway when you see this kind of increase?

MUKUL PAREEK: I probably would not focus so much on the rate of increase, because it's what it is and that's what respondents are telling us. In fact, as Dan mentioned, both I and Dan continue to be surprised month after month to see the similar increase and really this isn't something that was planned as part of the design of the index and it wasn't something that you were expecting either. But at the same time, I must say that we didn't know what to expect when we started the index, so whatever we're seeing is just news to us.

But let me answer your question in a slightly different way. When I look back at the index over the past 12 months, what do I see? Is there anything that I know today from the trajectory and from the different components of the aggregate index that I did not know before? And I think from my perspective the answer is a resounding "yes," and this is, as Dan mentioned, between the different sub-indexes. There's considerable variation and things moving up and down between the different months. At the high level, when we started the index, we were very aware of where the risks lay, but the index made clear to us was what the priorities of security professionals were amongst the risks.

In the early days of the index, for example, malware and all its different flavors made the top risk for a number of months, but over time the risk actually shifted to something once small, a security professional may never see it go, given the traditional emphasis on botnets, wireless, APT and all of that stuff. All the practitioners actually pointed the risk from sharing data with others in their value chain as the most important risk, and of course that kind of makes sense when you look at it this way. Just like you have a physical supply chain for the bricks and mortar world, you have a similar evaluation chain in the information world, and people have expressed the fact that they have little control over the security controls that vendors, customers, agents and other people deal with, the controls that they apply to the information that they receive. And they have expressed this sort of helplessness - if I may call it that - in the extent to which they worry about it. That's an interesting revelation.

In the same way, there were other things that we found from the index as we carried it on over the past 12 months and an example was in the early days of the index, the nation state made it's place in the top three, and it's not there today. But the rise of the Anonymous stuff, Occupy Wall Street and all the other things that have been going on, we saw that those risks in certain cases were with other risks that were rated highly in the early days of the index.

I won't pay too much attention to the two percent or the 45-degree rise, and I think the 45-degree rise can be a bit misleading because that's a function of the scale, which as you know starts at 1000. It would have been a different angle had we started it at zero.

News vs. Analysis

CHABROW: It's interesting. The last couple of months, one of the high areas of concern you just said, are things such as the Anonymous attacks, the hacktivism that's out there. As you know, there are many IT security professionals who feel that maybe concentration on something like hacktivist threats sort of camouflages more serious threats that IT organizations face. Is the news somewhat dictating what's happening out there as a concern rather than maybe some other kind of analysis?

GEER: That's a question we get asked fairly often and let me just say, it's correlated but we don't know which way causality works, because to a degree news stories are derived from the very experts that we're pooling or people like them. In other words, which way is the causality running? Are there news reports because people who have important jobs in this space are telling what they see, or are the people who have important jobs in this space feeling various things because they've read it in the newspaper? We don't know which way causality goes, but we do know that press coverage and what's thought to be the top risk in a given month are not one-for-one at all. There tends to be some relation, but not enough to say that they are in some way causally related. It's just fascinating is all I can say.

Surprises from Index Results

CHABROW: The index is an avocation for both of you. You both have day jobs, as a chief information security officer for Dan, and, for Mukul, a risk management expert. When you see these kinds of results, are you somehow taken by surprise that this is all of a sudden what they're saying is a concern?

PAREEK: In the area that I practice in there's always some surprise that comes out from what we do on the index from a month-to-month basis. The one source, for example, of information which we find quite interesting is the comments that respondents provide as part of their responses. We do include those comments in our monthly reports and people actually make specific references to the kinds of threats that they're facing. In fact, there have been occasions where some very senior people have reported that they're personally being targeted, which is interesting because people like me think, "Hey, do we need to be doing more about key people in our own organization," and that's one example.

Another example I can give you, this is on the other end of the risk scale, which means we're not talking about risks increasing but the perception of actually improving. This one was a revelation at least to us, the increase in information sharing that our respondents have been reporting. Now generally, information sharing with industry participants is a low-key activity that does not get a great deal of publicity or notice in the press, so you would not know about it otherwise. But clearly in the state groups, consortiums, organizations, the ISACS and other forums that companies are increasingly participating in, they're making a noticeable impact. The sub-index for information sharing has consistently shown a decrease each and every month since the inception of the index. I think it's telling and in one of our communications to all of our members, we actually asked for examples of how you see this information sharing happening and these industry forums and sometimes government-sponsored industry partnerships was what people pointed to. These things are having an impact and this is not the kind of stuff that you would normally see in the media, which tends to focus more on the hype and the sensational.

Information Sharing

CHABROW: Would you consider that one of the big takeaways, one of the big findings of your survey, that organizations are sharing more information? And are they doing it mostly through industry groups or individually?

GEER: I won't say it's a surprise, but it's gratifying. Obviously enough, if you don't share information you will never know whether you're a target of choice or a target of opportunity. By sharing information you can learn such things as that, and we're hoping in fact that's one of the things that our respondents get out of this. I don't know if it was clear, but let me now attempt to make it clear.

In addition to the chart and number that you see on the public website, for people who are respondents we also give them a considerable more amount of detail for this. In a way, in a sense, you could argue that's information sharing as well. What we're doing with our respondents who obviously are contributing their time to something that we're working on is we give them back the detailed analysis of the components of the overall index on a question-by-question basis. And in doing so, we're giving them information that they couldn't get anywhere else, just like they're giving us information that we couldn't get anywhere else. It's a bit symbiotic in that regard and the fact that information sharing is thought to be an improving aspect of the overall cybersecurity arena, maybe we're doing our part in that regard.

Otherwise though, the respondents are nameless to everyone except us, and in our case, we can't even tell what they're answering. The thanks to their thankless work is all I know to say and in a way the thanks that we give them back or the way we express our thanks is to share details with them.

CHABROW: There are a lot of other organizations obviously who would be interested in the results of your surveys. Is there a way that they can get some additional information?

GEER: We're hoping that we can increase the number of respondents by a substantial number, by a substantial ratio, and part of that would be reaching organizations that are not part of our pool and once they're a part of our pool they do get the very thing you're speaking of. Now can someone else just subscribe to it or whatever? Well, so far we have decided not to do that. So far we've said that the people who get to see the details are the people who give us details. We could be argued out of that, but that's our current position and at least for the short term that's what we will continue to do.

PAREEK: That's the only thing we have in mind at this point. You can buy that information from us if you want, and the only currency you can pay us in is by your own participation. If you're a company or an organization that has an interest, surely you have someone who's directly involved in this kind of work and they should participate, and that's how they get more detailed information. And when we send out this more detailed information, we encourage them to share it with their professional colleagues. By no means is it a professional secret or a copyrighted thing. We allow them to share it freely with other people in their organization, another professional colleague.

GEER: Mukul put that very well. Remember, we're not preventing anybody from sharing it, but we're giving it directly only to people who are working with us.

Using Index as a Metric

CHABROW: With the limited public information that's available from your index each month, as well as those who get more detailed information, how can organizations use this information as a metric to sort of help them judge whether their approach to cybersecurity is effective?

GEER: We're not asking effectiveness measures per say. In other words, we don't ask things like what percentage of threats have you seen in the last month that you could dismiss out of hand. I mean, that's not the kind of question we're asking. On the other hand, as I said earlier, when you're doing information sharing, one of the things that happens is you get a view as to whether or not you're seeing the same trends that other people are seeing. And if everybody else seems to be seeing a trend that you don't see, then you have to ask yourself, "Am I a target of some specific entity that for whatever reason wants what I have?" In other words, if I have everybody's firewall logs, yours and mine and everybody else's, and they're all exactly alike, it tells me that largely attacks are opportunistic. If they're all alike except one of them is radically different, I would say that person has, shall we say, special friend, etc. It's that ability to say, not so much a metric of effectiveness, but a "how do we compare to our peers" characteristic in terms of the threat space that they see. I think it's actually valuable.

I have worked as a consultant for a long time. A company hires you and you come in and you look around and you tell them what you think, so forth and so forth. One of the first questions they'll ask is, "How do we compare to our peers?" And in a way, the consultant there is serving as an anonymization engine for information they couldn't otherwise get. In a way, we're copying that kind of phenomenon here. We're serving as an anonymization engine for what threat space do you see and how is it changing month over month that you can't otherwise easily get. We think that's one of the major values. "Do I have a trend working for me or against me that's somehow different than everybody else or not?" In a way, the right people now to have on your shoulder are people who receive this and can talk to you about that. What we have is anecdotal and as Mukul said the free text comments that people make. Several of them have been almost to the point of being love letters about how we now include this in our own strategic decision making.

Upcoming Changes

CHABROW: As you go forward, what changes can we expect in the index?

PAREEK: Going forward, a key change that we're looking to implement is to modify the questions that we ask. For the first year we kept the questions constant and even though we noticed opportunities for improvement, we decided not to make any changes on purpose, as Dan explained in the beginning. The part that underlies our thinking really is really four-fold. The first thing is the questions that show the smallest amount of month-to-month variability. They had little to the decision support even if they do accurately reflect the reality. Some of these questions that haven't shown too much change over the past 12 months, we're probably going to either withdraw them or we're going to combine it with something else.

For example, in the European Union, things have moved. They have been bobbing up and down a little bit, but they've stayed kind of constant over the last year and we don't believe that it adds a lot of support or provides any additional information to the respondents or to ourselves. That's something we're going to drop. In fact, there's another reason we would like to drop something like that, which is that if we believe that a question doesn't really relate the risk or it provides some sort of illusory sense of control of something that you really can't control such as regulatory cost, then it probably isn't adding a great deal of value and we would like to drop that.

What we also want to do is to make the questions easier to answer and visually - this is something that you would probably notice if you were participating - some data questions flag out on a single screen where you can get to the questions in a much quicker way. Yet another thing that we want to change is questions about attack methods, which are kind of temporary in so far as attack methods ... keep changing and therefore some of the newer things need to be included and some of the older ones need to be dropped.

Essentially, the key thing that we're looking to do is to focus on the kinds of things that our respondents have alerted us to over the past year. For example, people have said, "We're going mobile. Our corporation is going to utilize in the mobile infrastructure. Can you include a question on that?" Similarly, we have had questions about the cloud as more and more organizations want to include cloud as part of their infrastructure. So those are the kinds of things that are going to make it into the index going forward and then there are other things that we're going to drop.

Now what does that mean for the index? In it's totality, the aggregate index will continue in an unbroken way from the past, but what that also means is that the time for individual's sub-indexes is probably going to stop for some of the questions and begin anew for some other questions. Then there are some questions which will probably see an unbroken line from the beginning of the index all the way forward into next year as well. Essentially, our key focus for next year is making the questions more relevant, incorporating the feedback we have received from our participants, and the second thing we're looking to do, as Dan mentioned, is increase the number of participants and respondents that we have participating in the index.

GEER: Let me add two things to what he said. One is, the way in which we swap a new question in and swap an old question out, we're copying the way the financial world does that when say a company in the Dow Jones Industrial Average would be taken out and a different company would be put in. The Dow Jones average does not take a jump because of that. There's a correction factor you apply and you just proceed and we're going to do the same thing here. In other words, we're not breaking any new methods of logic ground there, which is absolutely on purpose. We're copying the way the financial world works.

And his point about variability and so forth, we're looking for things that are likely to be of decision-support quality. If you can predict the value of something quite reliably in the future, there's not a lot of decision support that comes from asking other people what they think. And the bit about pulling out the ones that have low variability is because they don't reflect an area where decision support is necessary. The ones where there's a large amount of variability, that's where the decision support becomes interesting and by pulling the information of a number of people who are sharing their views, perhaps we can help people on the decision-support basis.

We're not, again, trying to do anything that's novel with respect to the mechanisms we use. We're approaching a here-to-for novel subject matter and that's cybersecurity.

CHABROW: Anything else you would like to add?

GEER: The interest in this from the people who are in it, quite a few them now have volunteered to introduce colleagues that they know in other firms to the respondent pool, and I can't think of a better endorsement than, "I know some people who should be here." One of the things that Mukul and I did not want to do in fact is have this be a creature of our individual rolodexes, and so the voluntarism, if you will, of people who are part of the respondent base saying, "I know some people who should be there," is as good of an endorsement as I know how to get. We hope to continue. And as you said earlier - I don't know what you want to call this, an avocation, a hobby, an obsession, something - we're continuing to do this on our own. In the long run, we hope to find a permanent home for it, but that's not an immediate necessity. But if this turns out to have permanent value, well neither one of us are permanent so eventually it will have to have a home that's permanent. In the meantime, we'll continue to run it.

CHABROW: Are you guys having fun doing it?

GEER: Oh yeah; it's real work but it's fun. It's fun to be able to get at things that you can't otherwise get at, and we think we're onto something here and so far so good.




Around the Network

Our website uses cookies. Cookies enable us to provide the best experience possible and help us understand how visitors use our website. By browsing govinfosecurity.com, you agree to our use of cookies.