NIST Cybersecurity Framework

Now that the National Institute of Standards and Technology (NIST) has issued its Cybersecurity Framework, how will we know if it is working? There are two essential categories of measurement: adoption and effectiveness. These metrics provide essential information for guiding policy and for assessing the value of the framework.

The framework itself, not being directive, militates against easy measurement.  There is a degree of redundancy. The framework is more like a list of possible measures for better cybersecurity and a general structure for implementing those measures. Over time, it is likely that as companies implement the framework their experience will narrow it to a shorter list of measures, as they experiment with different approaches and find what works best. Each industrial sector may find that some parts of the framework are more important for their business than others. But an initial assessment of adoption and effectiveness would answer important questions about the Executive Order and the framework.

Since the framework is voluntary, and since the Federal government has no adequate means of assessing cybersecurity outside government and military networks, determining the rate of adoption will not be easy. The traditional approach of using a Request for Information (RFI) is inadequate because those companies that respond can “self-select,” which can provide a skewed picture.  However, the Commerce Department (of which NIST is a part) has the authority under the Defense Production Act (DPA) to require companies to respond to industrial surveys.  These DPA surveys are fairly routine. Commerce does not need to ask every company in America if they are implementing the framework. It needs to devise an adequate sample of companies that would allow it to estimate adoption rates by sector and company size.

Other agencies also can collect information for sector specific groups. This could be useful, but it is not a substitute for a broad national survey. It would be valuable, however, to have sector specific agencies involved in the framework contribute and refine a broad survey. 

There may be some resistance to conducting a survey.  This in itself will be a good indication of intent regarding the framework. If companies or their representatives do not want the government to conduct a survey, it suggests that they may not be taking framework implementation seriously. If the government is reluctant to conduct a survey of adoption rates, it suggests that it is not taking cybersecurity seriously. Experience shows that we cannot assume anything when it comes to cybersecurity, and that an approach to policy guided by anecdote and ideology is more likely to fail than succeed – no company would run its business in this kind of data-free manner.  

Assessing implementation for one set of companies is unavoidable for a serious cybersecurity effort. Section 9 of the Executive Order on Critical Infrastructure Cybersecurity orders the Secretary of Homeland Security to “to identify critical infrastructure where a cybersecurity incident could reasonably result in catastrophic regional or national effects on public health or safety, economic security, or national security.” The intent of this section was to identify the truly critical facilities where disruption would have an immediate strategic effect, and hold them to a higher standard. The somewhat timid language of the Executive Order says that the Secretary should develop “a process through which owners and operators of critical infrastructure may submit relevant information.” If the companies in question (most of whom the government has already notified of their status) are truly critical, “may” is not adequate for public safety or national security. Some of the companies selected as being at “greatest risk” objected to their designation. This does not bode well for adoption. All selected companies should identify how they are implementing the framework. Telling a company that it provides a service essential for public safety or national security and then not seeing what they are doing about it would be irresponsible. 

There are costs to complying with surveys. This is a normal consideration for the issuance of DPA orders. The Commerce Department and other involved agencies, however, should be cognizant of this and ensure that any survey is designed to minimize the burden on companies – the goal is to determine if companies are using the framework, not to conduct an exhaustive survey of their cybersecurity practices. Surveys should be minimally intrusive and not an excuse for any agency to assume a regulatory role.  

Since information sharing is an important part of the framework’s approach to cybersecurity, a survey could also simply ask companies if they had ever received information from the government and if so, how they used it (asking whether this information is useful is like asking to be kicked). A systematic approach to assessing the utility of information sharing as it is now conducted would let us measure where this often cited remedy works and where it needs to be changed.  

Even if companies fully implement the NIST Framework (whatever that might entail), it does not necessarily mean that there will be an improvement in cybersecurity. The measures listed by NIST are likely to improve security, if implemented correctly, but to what degree is unknown.  The only way to accurately measure effectiveness is to ask if the number of successful penetrations and the outflow of data decreased. If hackers still get in and data still flows out, the framework is not working. These are effects-based measures, fundamental for determining the return on investment in cybersecurity; many other things can be measured but they are useful only to the extent they can be correlated with effects.

If finding out how many companies are implementing the framework will be difficult, finding out how many have been successfully hacked will be even harder, but without this number we have no idea if what NIST is recommending makes any sense. In only a tiny number of instances (such as DHS’s continuous monitoring effort or the Australian Signals Directorate’s mitigation strategies) has there been an effort to show that implementing a measure produces an observable reduction in successful attacks. The difficulty in linking measure and effect strongly affects how we manage risk, and the lack of data hampers a range of initiatives, from creating a cyber insurance market to applying the Federal Information Security Management Act (FISMA). A compliance approach to security lists actions taken; a better approach is to list results. 

There are a number of valuable published surveys that give us an idea of the scope of the cybersecurity problem, including Verizon’s annual “Data Breach Investigations Report” and the work of the Poneman Institute, among others. What would be useful in addition to these surveys would be greater specificity on what measures a company has taken and whether they had succeeded in improving cybersecurity.

One approach would be to create test beds and apply different framework configurations, to see if they prevented infection. To test their mitigation strategies, the Australian Signals Directorate (ASD) ran an experiment to test the strategies performance against “real world malware.” ASD built 1,200 virtual machines and gathered together around 1,700 malware samples, some of which had been used against government agencies and others that had been collected “in the wild.” This experiment allowed them to try different configurations, determine which were successful and to what degree. A similar test bed approach for the framework could guide implementation, especially for smaller companies that may lack the resources to otherwise take full advantage of the framework.  

NIST could consider three approaches to assessing effectiveness: red team, “live” testing world, and ASD-style modeling. There could be other approaches. Red teaming would require a company’s permission to let someone attempt to hack the company's network after framework measures had been put in place. Companies can be reluctant to do this because of the chance of unexpected damage or out of liability concerns, but red teaming against a specific configuration can be very useful.

Another approach to assessing effectiveness would identify a baseline against which effectiveness could be measured. This baseline would quantify how many times the target network had been successfully penetrated before framework measures had been put in place and how much data was exfiltrated. This could then be compared to the number of successes after implementing the framework. A similar approach has been used to assess “continuous monitoring and mitigation” techniques with some success, and large companies that have implemented continuous monitoring of their networks might be able to collect the necessary data and share it without referring to specific incident. The reluctance to share information about failures might be mitigated is the data was aggregated and anonymized.  However, many companies do not have this data and for smaller companies, there might only be a single incident (which extracts all their IP and then leaves, sometimes in a few minutes). 

The government could use the DPA to obtain information, but this may require an expanded interpretation of Commerce's authorities. The DPA was used in 2011 to gather information from telecom companies on foreign equipment in their networks, including security-related incidents, so its use would not be without precedent. 

The problems with red teaming and baseline testing suggest that an approach similar to that used by the ASD, creating a large number of virtual machines and testing different configurations against already identified malware is the best approach.  NIST or some other agency would assemble a statistically significant number of machines and use them to test the effect of different framework configurations against known malware.  This approach is limited by its dependence on known malware, but known malware makes up the majority of successful attacks.  The advantage of this approach is that it does not require consent.  

We could argue that the NIST Framework does not identify any particular configuration of measures, leaving this up to the individual companies. Nor does it provide guidance on implementation. This was the right approach for developing the framework, but now that we are in the implementation phase, a different approach is required, where either NIST or companies will need to determine how best to implement the framework.   

What if we find asymmetry in the results of a survey, with wide adoption of the framework but little effect on penetration and exfiltration? This will be an important indicator for future policy.  We should, if the framework is effective, see changes in the attacker population, with the less skilled attackers dropping out and the more skilled (or better resourced) changing attack modalities. This means that the framework could lead to a decrease in the number of successful penetrations. 

The framework might not, however, produce a lasting decrease in the rate of data exfiltration, as skilled opponents adjusts to improved defenses. This is not an improbable outcome when the attacker seeing to exfiltrate data is an intelligence agency. Asymmetry suggests that the effect of the framework on risk might be different for critical infrastructure than for intellectual property.  Survey data on penetration and exfiltration success rates will show where individual defense are inadequate and where collective action is needed, perhaps through increased international engagement in diplomacy and law enforcement to reduce cyber risks. 

A survey is not a test. A survey is a snapshot of how something is actually working. The idea that we should delay framework implementation for some kind of testing does not make sense, since the framework was developed on existing industry best practices. We know, given the level of opponent success, that the implementation of best practices in an individual and ad hoc manner can be ineffective. What we are interested in is how the “package” of best practices organized under the NIST Framework actually works to reduce risk. Given the renewal of Iranian activity in probing U.S. critical infrastructure targets, now is not the time for further delays.    

Whether it is additional surveys, a test bed or the DPA, we cannot accurately assess if the framework is improving cybersecurity without data. The lack of data on incidents, losses, and effective defenses is a major impediment to effective policymaking.  Cybersecurity is a new priority that governments need to recognize in their collection of data if we are to make cybersecurity work at a national level.

James Andrew Lewis is a senior fellow and director of the Strategic Technologies Program at the Center for Strategic and International Studies in Washington, D.C.

Commentary is produced by the Center for Strategic and International Studies (CSIS), a private, tax-exempt institution focusing on international public policy issues. Its research is nonpartisan and nonproprietary. CSIS does not take specific policy positions. Accordingly, all views, positions, and conclusions expressed in this publication should be understood to be solely those of the author(s).

© 2014 by the Center for Strategic and International Studies. All rights reserved.


James Andrew Lewis
Senior Vice President; Pritzker Chair; and Director, Strategic Technologies Program