Security metrics: more is not better

I've been fascinated by the issue of measuring and improving information security management practices and controls for several years now and despite thinking long and hard about security metrics, I'm still not confident that I even fully understand the problem let alone have anything like a solution.  But, that said, it's hard not to poke holes in security metrics proposed by others, usually because most of those who gets into metrics seem to end up creating tedious lists of "Security Things That Can Be Measured" (STTCBM), as if that was what was needed.  It isn't.

Take for example a recently-updated NIST standard (Special Publication) on security metrics.  The previous version of SP800-55, "Security Metrics Guide for Information Technology Systems" [no longer online], was little more than a catalogue of STTCBM - a long long list, its true, so long in fact that anyone trying to actually use the standard would have been stuffed within the first few pages.  The effort and hence costs required to collect, measure and report all those STTCBM would have been a nightmare, in my opinion far far outstripping the value of the metrics.  As to how management were expected to interpret the STTBCM and adjust the organization's information security tiller acordingly, I have no idea.  It's a classic case of "more data than sense".

The shiny new version of SP800-55, renamed "Performance Measurement Guide for Information Security", takes a rather different tack but is still quite long (80 pages in total, half of which are appendices).  I suspect the primary reason for its existence is to suport FISMA (the US Federal Information Security Management Act, essentially a set of information security policies mandated in law for US Government agencies) by imposing a standardized set of metrics that can be used to benchmark agencies and force the laggards to pull their socks up.  It remains a highly beurocratic and costly response to a genuine management problem.

Another draft NIST standard, SP800-80 "Guide for Developing Performance Metrics for Information Security", emphasises the process of developing and implementing security metrics.  It includes a shorter list of STTCBM ('candidate metrics'), but again takes a database approach with forms in the appendices characterising the metrics by 'metric type', 'frequency of collection' etc., details which, by the way, are organization and implementation-specific and really not that hard for grown-up security managers to figure out for themselves. 

I've been fortunate enough to see and comment on the evolving draft standard ISO/IEC 27004.  Having made my opinions crystal clear to the authors and committee responsible for '27004, I won't lay into it again in this forum, except to say that infosec practitioners of my acquaintance generally don't need to be told that some metrics (measurements in ISO/IEC-speak) can be "derived" from others by a process commonly known as "arithmetic".  Call me a cynic but I honestly don't believe that sums are the primary issue in security metrics.   

Today I stumbled across The Metrics Center, a project "to connect people, information, and analytics for the purpose of transforming data into knowledge, action, and ultimately value" (in the context of information security).  Sounds great!  But a quick look at their Metrics Catalog reveals that it is yet another database of STTCBM.  Under the ISO/IEC 27002 section, for example, we find  a list of 98 metrics currently, each of which expands to a form with standardised information.  Here's a typical extract [sic]:

 

In relation to information security metrics and management systems, what to measure and why are far more important questions to me than how but while all the standards and initiatives mentioned above list the what and explain the how, none adequately cover the why.   Worse still, they don't really explain what to measure: they merely state what could be measured (STTCBM) and in so doing stuff the lists with trivia (simple counts and percentages are all the vogue) while missing out many more creative and valuable measures and information sources (such as employee and industry surveys and numerous excellent web sites detailing current infosec threats, vulnerabilities and incidents etc.). 

While the metrics standards  are promoting long lists of STTCBM, I'm left struggling to find "a few good measures" for information security, things that are simple for management and others to understand, things that clearly mean something useful and can be used to drive an organisation's information security management system to new heights.  As a rehabilitating IT auditor, I'm particularly annoyed by the attitude of some self-acclaimed security metrics experts who insist that everything has to be reduced to numbers.  Managers don't manage entirely by the numbers.  Informed commentaries in benchmarking reports and security surveys, for example, are every bit as valuable as pie charts and graphs.  I'm looking for the information security equivalent of the old "days since a lost time accident" health and safety boards outside the factory gates - something self-evident that immediately resonates with ordinary viewers and has value as security awareness material as well as for numbers-based management.  A "Days since the last information security incident" graphic on the corporate intranet, perhaps?  If you want to be really posh, click on the graphic to read all about the last incident (what happened, why, how it was discovered and what has been done to prevent recurrence), to explore a breakdown of security incidents by corporate business units/locations/types or whatever.  Never mind the  "percentage of individuals who are able to assign security privileges for systems and applications who are trained and authorized security administrators".  The hard questions are like "How secure are we today compared to yesterday, or compared to our peers?" and "What should we be doing to be more secure tomorrow?".  Most of all, "Are we secure enough?". 

Like I said, I don't even have all the questions yet, let alone the answers.

Kind regards,
Gary

Gary Hinson
Passionate about security awareness
www.NoticeBored.com  Creative awareness materials
www.ISO27001security.com  ISO/IEC 27000 standards

Comments

[Adrian]

Not sure if I agree in your arguments regarding security metrics. Metrics are good if you can obtain them in an automated way. I am thankful for projects like the security metrics project which gives me an idea of what metrics I *could* measure. But although there are projects and catalogues like those it's still my problem which metrics I choose and why I obtain them (and what I do with them afterwards).

Don't misunderstand me. I see no need in measuring the amount of viruses detected at my perimeter. Since those were filtered before reaching my network they never really were a problem. But I am interested in all those viruses which entered my systems and which successfully infected my systems. And most important, I want to obtain these numbers in an automated way. Just to use a simple example.

You asked the following question: "How secure are we today compared to yesterday, or compared to our peers?"

Now that is an interesting question. And you know what? That's one of those questions which made us start with SOMAP.org. We are constantly reinventing the wheel. And much worse, everybody is doing so on their own. How can we compare our own company with the companies of our peers when we do not share the same methodologies, the same assessment tools, the same metrics?

It is time for an open source risk and compliance assessment tool which can be used to assess our environments and to compare our findings with others.

But this text field is probably the wrong place to start such a discussion. Please drop me a note if you are interested to fully discuss this topic.

Cheers,
Adrian

[Gary Hinson]

Thanks Adrian. I will certainly look at SOMAP.org

A colleague from CERT has pointed me at the Center for Internet Security's "Security Metrics & Benchmarking Service". I must explore further. The CIS technical platform security guides at www.CIsecurity.org have been ideal to derive baseline technical security standards for various platforms so I'd like to know more.

Meanwhile, I've found an excellent paper in CSO Magazine re Andrew Jaquith's take on metrics:

http://www.csoonline.com/article/220462/A_Few_Good_Information_Security_Metrics

It pre-empts most of the points I made in my blog posting. Andrew's book "Security Metrics: replacing fear, uncertainty and doubt" is a good read and I agree with most of what he says.

Automating metrics collection/reporting is a separate issue to finding 'a few good security metrics' and has its own pitfalls, the main one being a bias towards metrics that can be collected and analysed in an automated fashion, which implies using data extracted from the IT systems. Unfortunately, some of the most interesting aspects of infosec are the people not the systems - for example, the effectiveness of policies and awareness activities can be measured by observing employee's changing behaviours. Only a few of those changes can be easily measured by the systems.

Thanks for the comments Adrian and good luck with SOMAP.

G.

[Betsy Nichols]

As the lead designer for the MetricsCenter site, I would like to say that I completely agree with your point that in security metrics, more is not necessarily better. Also, I fully acknowledge that there is a pervasive tendency to propose metrics based upon the not-particularly-selective criterion of STTCBM (“Security Things That Can Be Measured”).

If I understand correctly, you would like to see a small collection of quantitative values that are a good characterization of the current security state of an organization as well as a measure of its dynamic rate of change (hopefully improvement) over time. For executive audiences this is critical. It is my belief that the best candidates for such metrics should first focus on outcomes as opposed to the policies, people, processes, and technologies that have been put in place to achieve those outcomes.

As an example of an outcome, I would propose a couple of metrics that are very similar to those used now for general IT management (and SLA’s): Mean Time Between Security Incidents and Mean Time To Repair a Security Incident. These are just two examples. No doubt there are others of equal utility.

A project has been launched by the Center for Internet Security ( www.cisecurity.org ) to define a few practical consensus metrics with participation from a cross-section of companies, government, consultants, academics, and metric experts. Outcome metrics are one focus of the group. I am sure that Steven Piliero (spiliero@cisecurity.org), leader of that CIS effort, would welcome your input on the current initial set of nine (9) consensus metrics the consensus group is working on.

Having said this, however, once one turns to the problem of troubleshooting or tuning, one needs visibility of a more detailed sort. It is my belief that this use case generates a valid requirement for 10s of metrics, per target process, technology, or policy. For the TJX bad outcome (a huge Mean Time to Repair a Security Incident), troubleshooting would likely include metrics on the coverage of their vulnerability scanning process (it appears that perhaps 100s of wireless access points were not scanned), additional metrics on data leakage (for all the PII bleeding from their network), and yet more metrics to identify anomalous account activity. Managers and operations staff need these metrics to understand the effectiveness and efficiency of the myriads of contexts in which they must operate, as well as to drive improvement where needed.

So, recognizing that there are hundreds of metrics that an organization may want to consider implementing, the Metrics Catalog at www.metricscenter.org provides an open, free, structured repository along with the capability (soon) for individuals to provide commentary on any metric—why they used it, how effective it was for a specific audience, any unanticipated side effects, proposals for clarification/improvement, or whatever. The structured repository supports mapping metrics to a business context (e.g. standards, regulations, best practices, functional areas, etc) as well as key word searching. The plan is to allow experts to add new metrics and contexts to the catalog, as well as new revisions and comments.

Our hope is that MetricsCenter will become a living repository of metrics that represents the current and evolving science of measuring security—for all audiences. Like you, we are passionate about metrics and are actively investing in their advancement. IT security needs more science and less soothsaying.

Elizabeth A. Nichols, Ph.D.
CTO | PlexLogic
Box 557 | Great Falls, VA 22066

[Scott Wright]

Great discussion. I see the problem with using "proxy" metrics, but I think it depends how closely you are to the employee/user's "risk decision point". With some interpretation and adjustments (subjective, I agree), I believe they can still provide valuable indications.

However, I would like to get your takes on using Honey Sticks as "one" metric in the toolbox. I started The Honey Stick Project in early 2008 just as an experiment, to get some data to talk about. If you aren't familiar with it, I detect the opening of safe HTML files on randomly scattered USB drives (no software on the devices).

But the more I work on the project, the more I realize that being able to detect when somebody uses an unauthorized device in their computer (assuming it is against their organization's published policies), can be a good way to detect risky behavior.

The reason I say this is that the act of finding a device, picking it up and putting it into your computer is what I call a "forced risk decision". In other words, they have to consciously decide to use the device in light of all that they have been taught about the policies and good security practices.

Technically, it simply uses an external image file reference in an HTML file on the device. When the file is opened, the request is logged at a webserver. It is safe and easy to detect, as long as the user's computer is on the Internet, and they double-click on the file, which launches the default browser and makes the request.

I am now offering a free trial initiative, where an organization can request 10 filesets that I create with unique device ID numbers, and I email a report at the end of a month, with the raw data for the organization's devices.

I realize we are only measuring one aspect of awareness with this approach. But it is designed to simulate a real risk scenario, as opposed to using "proxy" metrics from IT systems. The novelty of the approach also seems to capture the imagination of CIOs who see it as something that can grab people's attention.