1

Malicious Software Prevention for NERC CIP-007 Compliance:Protective Controls for Operating Systems and Supporting Applications

Matthew E. Luallen, Co-Founder, Encari

Paul J. Feldman, Chairman of the Midwest ISO, Independent Director of Western Electricity Reliability Council (WECC)

Executive Summary

Utilities attempting to meet the North American Electric Reliability Corporation “Critical Infrastructure Pro-tection” (NERC CIP) requirements have encountered an unexpected, but very serious conundrum in the cyber-security realm: should they strive to meet the spirit or letter of the regulations?

The potential penalties are compelling, up to $1,000,000 per day of non-compliance per a requirement; however, “checking the box” and simply meeting the letter of the NERC CIP requirements should not be the primary goal. Increasing security for security’s sake should not be the goal either. All solutions must focus on meeting the true intention of the NERC CIP requirements — the same goal that has driven investments since the dawn of the electric infrastructure: protecting the reliability and availability of the Bulk Electric System (BES).

Utilities could “check the box” and meet the letter of the regulations by implementing and maintaining tra-ditional security solutions (e.g., blacklist-based antivirus, emergency security patches). However, security teams have discovered that these solutions may not only fail to protect reliability and availability, they may negatively impact the goals themselves. For example, on critical Process Control Systems (PCS) at the core of the electric infrastructure, blacklist-based applications may impose unacceptable performance bur-dens, while vulnerability patches may jeopardize stability, be delayed by the PCS vendor, or affect system availability during application and / or system reloading.

Fortunately, there is another way to truly meet the spirit of the NERC CIP requirements: application white-listing. Application whitelisting modifies the traditional antivirus and host security approach and turns it 180 degrees. Rather than maintaining an exponentially enlarging blacklist of detected malicious software, this newer and more powerful technology enforces a relatively small whitelist of the authorized applications for each system. Application whitelisting automatically eliminates all unauthorized applications by ensuring that only approved applications can execute, including the prevention of even unknown malware.

This paper will explain why application whitelisting may serve as a compensating control for NERC CIP-007, R3 (security patching) and solution for CIP-007, R4 (anti-malware). Application whitelisting also stops all unknown applications from executing; therefore, depending upon installation options, the same appli-cation whitelisting implementation may simultaneously aid utilities in meeting NERC CIP-003, R6 (change control and configuration management).

Cyber-security of the Electric Infrastructure

Almost every aspect of American life depends on the reliable delivery of electricity — from producing goods to saving lives, from defending the country to conducting electronic banking and commerce. Quite simply, the electric infrastructure is one of the United States’ most critical resources and needs to be protected as such.

2

The importance of the electric infrastructure is not news to utilities and regulators. Over the years, tech-nologies and regulations have been implemented with the singular goal of ensuring the reliability and availability of electric power through the high voltage electrical infrastructure (the Bulk Electric System or BES). The BES grid has been architected to prevent cascading power failures and to continue function-ing whenever one generator or transmission line fails (“N-1” failure protection). The industry has done a remarkable job lately in dealing with these N-1 situations that are often the result of natural disasters or some other rare event.

But what happens when there are multiple, simultaneous failures or system manipulations rather than just one? The grid is not currently equipped to handle this situation, referred to by many as the “N-x” failure situation. The cost of building a redundant, balanced ability to respond to this situation is prohibitive at best, and potentially even mathematically impossible. Fortunately, the odds of a natural event or a physical attack creating this situation have been so low that the investments weren’t warranted.

Today, nature is not the biggest concern when it comes to potential N-x situations. That distinction belongs to cyber-attacks. Whether for extortion or terrorism, cyber-attacks against the electric infrastructure are particularly ominous because they can easily be designed to create an N-x attack situation. Realizing this fact, the industry and regulators have been working to create standards and implement technologies to thwart these attempts.

The North American Electric Reliability Corporation (NERC), a self-regulatory organization that is subject to oversight by the U.S. Federal Energy Regulatory Commission (FERC) and governmental authorities in Canada, is a primary player in this effort. NERC’s mission is to ensure the reliability of the bulk power sys-tem in North America. Specific to cyber-attacks and their ability to create N-x situations, NERC has worked with the electric industry to create a set of requirements known as the NERC CIP (“Critical Infrastructure Protection”) Cyber Security Standards to protect the grid’s most critical assets. Utilities throughout North America are focused on the NERC CIP standards and reliability of the Bulk Electric System — but under a new and evolving regime of mandatory compliance and possible fines. Philosophies of “carrot” versus “stick” have not been fully worked out to establish a system whose focus is clearly reliability, and the dan-ger of a compliance focus (at the possible expense of reliability) is real. Every company needs to examine its own goals and motivations in this regard to ensure a continued focus on reliability with compliance as a byproduct rather than THE product. While reliability and compliance can go hand in hand, the industry still has work to do to ensure that is truly the case. In the meantime each company needs to ask the question — “Do our budget processes, organizational structures, reward systems, and senior management actions support a culture of reliability, or is reliability secondary to compliance. Companies must understand that the answers to this question will drive different actions throughout the organization with ultimately different consequences.

Security of Critical Cyber Assets: Introduction to NERC CIP-007

“Checking the box” and simply meeting the letter of the NERC CIP requirements should not be the goal. Increasing security for security’s sake should not be the goal either. All solutions must focus on meeting the true intention of the NERC CIP requirements and balance appropriately the delicate security model — the same goal that has driven investments since the dawn of the electric infrastructure: protecting the reliability and availability of electricity delivery. All stakeholders — the asset owners, customers and the government, must contemplate a thorough understanding of the risks. The owners and the shareholders need and deserve a fair return on their investments, the customers want to safely procure a valuable com-modity with high reliability and low cost, while the government needs to manage risk across all 18 Critical Infrastructures / Key Resources (CI/KR).

3

The NERC CIP standards provide a convenient roadmap for the assets that need controlling/protecting from cyber-attacks, “Critical Cyber Assets” (CCA) and Non-Critical Cyber Assets (NCCA) located within an Electronic Security Perimeter (ESP). While there are nine NERC CIP requirements, this paper focuses on the requirements that are directly related to securing the critical process control systems at the core of the electric infrastructure: Energy Management Systems (EMS), Distributed or Digital Control Systems (DCS), and Plant Control Systems (PCS). The primary anti-malware requirement is located within NERC CIP-007, and the most relevant sections associated with application whitelisting are as follows:

• CIP-007-R3: Security Patch Management: The Responsible Entity, either separately or as a compo-nent of the documented configuration management process specified in CIP-003 Requirement R6, shall establish and document a security patch management program for tracking, evaluating, testing, and installing applicable cyber security software patches for all Cyber Assets within the Electronic Security Perimeter(s).

R3.1: The Responsible Entity shall document the assessment of security patches and security upgrades for applicability within thirty calendar days of availability of the patches or upgrades.

R3.2: The Responsible Entity shall document the implementation of security patches. In any case where the patch is not installed, the Responsible Entity shall document compensating measure(s) applied to mitigate risk exposure or an acceptance of risk.

• CIP-007-R4: Malicious Software Prevention: The Responsible Entity shall use Antivirus software and other malicious software (“malware”) prevention tools, where technically feasible, to detect, prevent, deter, and mitigate the introduction, exposure, and propagation of malware on all Cyber Assets within the Electronic Security Perimeter(s).

R4.1: The Responsible Entity shall document and implement Antivirus and malware prevention tools. In the case where Antivirus software and malware prevention tools are not installed, the Responsible Entity shall document compensating measure(s) applied to mitigate risk exposure or an acceptance of risk.

R4.2: The Responsible Entity shall document and implement a process for the update of Anti-virus and malware prevention “signatures.” The process must address testing and installing the signatures.

The rest of this paper will outline how utilities can best meet these requirements — and simultaneously improve the infrastructure’s overall security, reliability and availability — in a manner that is consistent with the operational reality of control systems.

Unique Operational Realities of Critical Control Systems

Any discussion about the security of control systems must begin with an understanding of the realities of these critical implementations — realities that traditional security solutions simply cannot handle. While the list is long, there are four major challenges that deserve mention:

1. Many control systems are iso¬lated and not always connected to the Internet; therefore, the systems are unable to consistently download the latest antivirus signatures or patches, leaving them vulner-able even to known attacks.

4

2. Most control systems cannot be rebooted or can only be rebooted at specific times in very tight maintenance windows, making unplanned installations of operating system or application patches infeasible.

3. Control systems generally have limited memory and hardware resources available making them un-able to handle the performance impacts of resource-hungry security applications, including black-list-based antivirus.

4. Many security systems today are running on older operating systems that are no longer supported and for which patches are no longer created.

As a real world example, a global energy company headquartered in the northeast USA, was concerned about the use of blacklist-based antivirus solutions on the execution of real-time applications for its power generating plant control systems’ operations. The company’s Critical Cyber Assets include operator inter-faces (a.k.a. Human Machine Interfaces or HMI) in the DCS/PCS environment that are critical to reliable and safe operation of their assets and data historians. While a typical energy management (a.k.a. Super-visory Control and Data Acquisition or SCADA) system used by utilities for control of electricity production and delivery over large geographic areas can tolerate two second status and ten second analog updates, acceptable HMI operations in a generating plant environment are based on a one second maximum re-sponse time and refresh rate. Operational testing on a plant data historian demonstrated that blacklist-based applications imposed an unacceptable burden on response time and refresh rates through higher processor loading and additional network traffic between the control system cyber assets. Additionally, the generating company’s stringent uptime standards made unplanned patches — especially those that required rebooting — unfeasible. The company was also concerned that automatic delivery of blacklist sig-nature updates over the Internet or Intranet poses risk to application reliability and requires opening ports in firewalls that pose a threat to the security of the generating plant cyber assets; management of these risks requires additional resources.

In the end, traditional solutions can be implemented to simply meet the “check boxes” of the requirements, but utilities are forced to choose between various suboptimal outcomes when they do. These outcomes may include increased management costs, impacted performance and availability, and a false sense of security. Specifically:

• CIP-007, R3 Compliance: Patch management systems can meet the requirement if managed and implemented correctly on a consistent on-going basis using defined and approved procedures.

• CIP-007, R4 Compliance: Antivirus solutions can meet the requirement if managed and implement-ed correctly on a consistent on-going basis using defined and approved procedures.

• Performance Impacts: Blacklist scans impose an unacceptable performance burden on these criti-cal systems — this is especially the case on older systems with limited resources.

• System Availability: Control systems’ high availability is jeopardized by patches that require the rebooting of systems during normal operating hours.

• Complexity Risk: Even if patch management and traditional antivirus solutions are perfectly imple-mented, the fact that they introduce an element of continued implementation complexity versus al-ternatives is an added risk.

• False Sense of Security: Blacklisting solutions are ineffective against unknown, zero day and tar-geted malware, rootkits and most memory attacks. Systems without consistent connectivity may be exposed to even known threats if signatures are not updated or patches have not been applied. Out-

5

of-support legacy systems (for which patches will never be available) will remain unprotected against even known vulnerabilities.

Even in the face of this daunting list of operational inconsistencies, utilities would still implement traditional security solutions if they were highly effective at securing systems or if they were the only option to meet the NERC CIP requirements. The reality is that they are neither. Security professionals (and even the antivi-rus vendors themselves) agree that blacklisting is no longer sufficient to defeat today’s threats. Blacklisting cannot address whole classes of malware threats and attacks (e.g., zero-day exploits, targeted attacks, memory exploits, rootkits, etc.) and independent tests show blacklisting solutions’ detection rates con-tinue to drop. An alternative approach exists – Cyber Asset Application Whitelisting.

Why Cyber Asset Application Whitelisting is a Viable Solution

Application whitelisting takes the traditional antivirus approach and turns it 180 degrees. Rather than main-taining an exponentially enlarging blacklist of known malicious software, this new and powerful technology enforces a relatively small whitelist of the authorized applications for each computer. By ensuring that only approved applications can execute, application whitelisting automatically eliminates all unauthorized ap-plications — including even unknown malware. This approach meets the actual intention of the NERC CIP requirements: preventing all unauthorized applications from executing on Critical Cyber Assets.

While this paper is not intended to explain all of the technical intricacies of how application whitelisting solutions work, leading solutions are built on two fundamental principles. First and foremost, the solutions are designed to enforce a relatively small list of known and approved applications rather than chase a huge and exponentially growing list of detected malware. For instance, to protect your home would you rather issue house keys to all family members and friends that are allowed access (whitelist) or define restric-tions for each individual in the world that is not allowed access (blacklist)? This paradigm shift occurred in the mid 1990’s for firewall technology as entities began implementing “deny by default” firewall rules implemented for the past five years leveraging the whitelisting model. The solution must also be designed to easily handle the addition of new applications or updates without increasing management overhead or requiring any changes to the company’s existing operational approaches.

For application whitelisting to enforce a list of known and approved applications, each of the following must occur. The solution must have a way of building or acquiring the whitelist of applications for any given computer — preferably from the computer itself since no two computers are alike. Also, the solution must securely and efficiently enforce the whitelist on the computer. And finally, the solution must have the abil-ity to report any attempts to violate the security policies it is enforcing. These three capabilities together provide the security required to protect the computer, while at the same time reporting on system status.

The whitelist enforcement mechanism is best deployed in the form of a tamper-proof client installed on each computer or endpoint. It is crucial that local users or malicious programs cannot circumvent the en-forcement provided by this engine, so the client must function in the operating system itself. Through tight integration with the operating system, the solution is able to protect the system and have the greatest ef-ficiency — it essentially functions as part of the operating system rather than an add-on security feature. From within the operating system, the client reads in the whitelist, and ensures that only those applications on the whitelist are allowed to run. This process begins during boot time when the operating system is starting, and then checks all executables that load to ensure they are authorized. The client only performs checks when a new application or process attempts to start, so the ongoing performance impact is im-perceptibly low compared to blacklist antivirus scans. It is paramount that asset owners work with control system vendors to include this type of capability directly in to the control system software and/or hardware.

6

Control system vendors of IEDS, RTUs, Relays, Synchrophasors and other hardware and software need to ensure that their future systems are protected by default — application whitelisting is an excellent step in the correct direction.

Application whitelisting solutions also monitor activity to aid in NERC CIP-007, R6 compliance. For ex-ample, a solution can log attempts to overwrite protected applications on the computer or attempts to run unauthorized applications. The solution can also periodically remove all unauthorized applications that may have been copied to the Cyber Asset, ensuring the pristine condition of the Cyber Asset is main-tained. Compliance reports can show the system configuration has been maintained and any unauthorized executables that have been removed thereby providing supporting evidence that is necessary for NERC CIP-003, R6.

Addressing another one of application whitelisting’s fundamental principles, an application whitelisting solution must be able to automatically — without requiring real-time IT involvement — update the whitelist whenever new applications are added or existing ones are upgraded. Even in a controlled environment like energy systems, the Cyber Assets must eventually be updated with newer applications or patches. Some of these requirements are driven by compliance and company policies, while others are required to imple-ment new functionality.

Innovative whitelisting solutions allow authorized change while still maintaining security on the Cyber As-set. The term being applied to this process is “Trusted Change”. All trusted change is built on this simple concept: IT establishes multiple “sources of trust” from which users and Cyber Assets can install applica-tions or upgrades. As long as the users and Cyber Assets receive the applications or upgrades from these trusted sources, the applications or upgrades can be automatically added to the whitelist without any addi-tional IT involvement. The additions are transparent and friction-free. Examples of trusted sources include trusted applications, trusted digital signatures, trusted updaters, and even trusted users.

By preventing all unauthorized applications and malware from executing, application whitelisting simulta-neously serves as a compensating control for NERC CIP-007, R3 and a solution for CIP-007, R4 in a way that also increases security and protects the availability / reliability of electricity delivery. Specifically:

• CIP-007, R3 Compliance: By preventing the execution of malware — including those that are de-posited via vulnerabilities that haven’t been patched or via memory-based attacks like DLL injections — application whitelisting is a compensating control until the PCS vendor approved security patches are installed during regular maintenance windows.

• CIP-007, R4 Compliance: Application whitelisting may currently meet CIP-007, R4 (since it is clearly an anti-malware solution) or it may be considered a compensating control (since it eliminates all un-authorized applications from executing).

• Security: Application whitelisting, or deny by default, is far more effective than blacklisting because it prevents all malware, whether known or unknown. Leading versions stop rootkits and prevent memory attacks like DLL injections and attempts to write to kernel memory as well. Additionally, ap-plication whitelisting provides protection for out-of-support legacy Cyber Assets for which patches will never be available.

• Performance Impacts: Since the whitelists are relatively small and only run when an application at-tempts to execute, the performance impact is imperceptible.

• System Availability: Control systems’ high availability is protected because the Cyber Assets are not required to be rebooted during normal operating hours.

7

Application Whitelisting Aids in Complying with NERC CIP-003, R6 and CIP-007, R6

While the bulk of this paper has been focused on meeting the true intention of NERC CIP-007 R3 and R4, preventing the execution of any malware, the reality is that application whitelisting prevents the execution of all unauthorized applications — not just malicious ones. Unauthorized applications are a focus of NERC CIP-003, R6:

• CIP-003, R6: Change Control and Configuration Management: The Responsible Entity shall establish and document a process of change control and configuration management for adding, modifying, replacing, or removing Critical Cyber Asset hardware or software, and implement supporting configu-ration management activities to identify, control and document all entity or vendor related changes to hardware and software components of Critical Cyber Assets pursuant to the change control pro-cess.

Application whitelisting may aid in stopping a non-compliant, unapproved activity pursuant to CIP-003, R6 by providing attempted change detection and actual execution restriction of unauthorized applications. Application whitelisting solutions also typically include baselining and reporting to aid in the generation of evidence pursuant to CIP-007, R6 Security Status Monitoring:

• CIP-007, R6: Security Status Monitoring: The Responsible Entity shall ensure that Cyber Assets within the Electronic Security Perimeter, as technically feasible, implement automated tools or orga-nizational process controls to monitor system events that are related to cyber security.

Any identified security event may be a result of poorly defined procedures and executed practices of change control and configuration management or actual attempted Cyber Asset manipulation that should be man-aged and escalated according to an entities CIP-008 Cyber Security Incident Response Plan (CSIRP).

Other benefits of application whitelisting in lieu of blacklisting for control system host computers include:

• Eliminate the need to test and update blacklist signatures per CIP-007, R4.2.

• Control system hosts need not be continuously connected to an external network to maintain a high degree of protection against malware.

• The application environment for control system hosts is relatively static after initial Cyber Asset com-missioning. The use of application whitelisting requires minimal management resources compared to blacklisting.

Conclusion

In addition to superior protection against even zero-day attacks, application whitelisting is gaining a fol-lowing because it addresses the operational realities associated with control system implementations that blacklist-based solutions cannot. First, application whitelisting continues to provide protection without requiring signature or patch updates, so it can function in Cyber Assets that are not connected to the Internet. Second, whitelist-protected control systems remain online until regularly scheduled maintenance windows, instead of requiring downtime for emergency vulnerability patches. Third, application whitelisting solutions typically do not impact control system performance — a significant advantage over resource-hungry security applications like blacklist-based antivirus. Fourth, resource requirements for management of application whitelisting for control system hosts is minimal compared to blacklisting because of the

8

relatively static application environment in power plant control systems. And finally, leading application whitelisting solutions provide protection for control systems that are built on older, unsupported operating systems for which no patches are available.

For all of these reasons, application whitelisting is a solid option for utilities trying to secure control sys-tems, to meet NERC CIP requirements, and to protect the overall availability and reliability of the BES in North America.

Information Note: The views expressed in this paper are the authors’ own and not necessarily associated with any organization that the authors serve in Board, Client, or Advisory Board roles.