PATCH VALIDATION FOR ICS - foxguardsolutions.comfoxguardsolutions.com/wp-content/uploads/2017/02/... · the patch management process. ICS CERT and NIST both describe how impact plays

PATCH VALIDATIONFOR ICS

WHITEPAPER:WHAT IT IS, WHY DO IT AND HOW TO AVOID A CATASTROPHE

ROGER RADEMACHERFOXGUARD SOLUTIONS 2285 PROSPECT DRIVECHRISTIANSBURG, VA 24073

FOXGUARDSOLUTIONS.COM

RUSSELL DERNGE OIL & GAS DIGITAL SOLUTIONS 1800 NELSON ROADLONGMONT, CO 80501

GEOILANDGAS.COM


EXECUTIVE SUMMARYPatch Validation processes reveal unknown risks, drive management processes, and prevent unnecessary downtime. Systematic testing procedures enable quantitative operational risk analysis. Validation fuels change management and detection systems, contributes to mitigation plans, and jump starts distribution efforts. Read on to discover how the GE and FoxGuard experts handle Patch Validation.

INTRODUCTIONIn this whitepaper, GE Oil & Gas Digital Solutions and FoxGuard Solutions will examine common patch validation practices and requirements as identified by the National Institute of Standards and Technology (NIST), Department of Homeland Security (DHS) – Industrial Control Systems Cyber Emergency Response Team (ICS-CERT), and North American Electric Reliability Corporation (NERC). It is the intent of this whitepaper to provide insight into many challenges that organi-zations are faced with and suggest a path forward, while taking into account the specific challenges for industrial control system (ICS) environments.

Research from leading organizations in cyber security continues to indicate that updated systems improve overall cyber security posture. NIST echoes as much within SP.800-43r3 while, at the same time, drawing attention to the risk associated with automated update tools. Those in the control room see familiar notifications regarding software updates similar to updates users see with home and office systems. The decision to trust an update without proper valida-tion for a control system is more difficult than other systems. If an update is applied and something goes wrong, the operational implications can be quite costly. This paper will attempt to explore one key area of the patch management cycle - patch validation. This will include discussions around the what, why, and how relative to validated patch management.

REFERENCES NIST Framework for Improving Critical

Infrastructure Cybersecurity v1.0

NIST SP.800-40r3, Guide to Enterprise Patch Management Technologies

ICS CERT, Recommended Practice for Patch Management of Control Systems

NERC CIP-002-5.1, Cyber Security – BES Cyber System Categorization

NERC CIP-007-6, Cyber Security – System Security Management

NERC CIP-010-2, Cyber Security – Configuration Change Management and Vulnerability Assessments

GE I FoxGuard Solutions

DEFINING PATCH VALIDATIONThere is no official definition of “Patch Validation” or validation in general as it relates to the patch management process. Validation may be defined as “proof that something is based on truth or fact, or is acceptable.”

Most discussions around patch validation are limited to testing and ignore the larger picture. Patch Validation generates the information that drives Patch Management forward.

We define Patch Validation as “The management of available updates to an asset or system as part of an overall patch management process which includes the verification, testing, mitigation, documentation, and staging of approved updates.”

Like the Patch Management Lifecycle, Patch Validation may also be viewed as a cyclical effort which starts with a baseline; but ends just shy of production deployment. This indicates that Validation may be a contributing force to many patch management processes.

BACKGROUNDPatch Management is a maze of processes and procedures supported by a combination of automated and manual efforts. Validation has its own unique challenges and serves as a means to reduce risk by way of testing an update prior to deployment in production environments. Testing provides an opportunity to learn more about the update and answer critical questions:

What changes does it represent to the configuration and software baseline?

What files does it touch?

Does it create new user accounts or services?

Does it establish new network ports?

Does it impact the attack surface of the asset?

Does it impact other applications?

Unfavorable answers to these questions may lead to mitigation strategies that impact applicability and/or establish resolution efforts. Validation helps to mitigate issues before they are introduced into the process environment.

BaselinetheAsset

Mitigate

PatchValidation

3geoilandgas.com I foxguardsolutions.com


BASELINE THE ASSET VERIFY, DEPLOY, VERIFY TEST & VALIDATE UPDATE(S)

An Asset Baseline is a known, operationally sound, representation of an asset that may be used for testing. The Asset Baseline represents everything necessary to return an asset to a known state.

Updates must be source-ver-ified, deployed to a test asset, and install-verified. Source verification is the confirma-tion that the file(s) are indeed provided by and sourced from a trusted update repository. Deployment to a test asset is accomplished using produc-tion deployment tools. Install verification is the confir-mation that an update was successfully applied and its installation status is able to be discerned.

Testing and validating an up-date includes identifying and documenting changes to the asset or system, as well as, performing operational tests to ensure asset or system function. Testing may occur at individual asset levels (micro-validation), system levels (macro-validation), and anywhere between.

MITIGATE UPDATE BASELINE STAGE UPDATE

Mitigation includes all actions and associated documenta-tion which must take place in the stead of update applica-tion. Mitigating actions also may require additional testing and validation.

An updated baseline includes the approval, documentation and verification of the Asset Baseline for use in future vali-dation efforts.

The staging of an update serves as a final hurdle be-tween validation and pro-duction systems to ensure that only approved updates are available for production distribution.

EACH STEP IN THE VALIDATION PROCESS IS DISCUSSED BELOW:


PROBLEMSCriticality continues to influence security programs for industrial control systems. The nature of these systems demands caution as rogue or untested patches may lead to unsafe conditions and/or impacts to reliability and revenue.

We are going to discuss three topics that make Patch Validation for ICS difficult: operational impact, resources, and integration.

OPERATIONAL IMPACTOperational impact plays a large role when determining the path that an update may take through the patch management process. ICS CERT and NIST both describe how impact plays a role in security analysis and patch management with ICS CERT stating, “If there are strong business constraints or operational concerns related to implementing the patch at a specific time, then it may be necessary to hold off on patching the system until the scheduled maintenance window”(ICS CERT, Recommended Practice for Patch Management of Control Systems).

ICS CERT evaluates impact from the standpoint of a vulnerability footprint. What is the Operational Impact if I DON’T update? Impact is a driving factor behind the proposed patch urgency process where applicable updates are analyzed for operational risk to determine whether the update needs to be fast-tracked or scheduled. They list impact as one of four factors that influence an overall vulnerability footprint which is analyzed as part of determining urgency.

NIST discusses impact from the other end of the spectrum. What is the Operational Impact if I DO update? Vendors may try their best to test updates but they cannot test every scenario. More importantly, they cannot test the scenarios that have significance to each customer. All updates have a potential for unwanted behavior. Unlike vulnerability footprints where we may rate the operational risk of a security vulnerability, potential conflicts and unwanted behavior may only be discovered through observation and careful testing.

OT environments are likely to have a larger portion of assets and systems that are considered critical as an aggregate of functional responsibilities. This understanding leads most administrators to hold off updates for as long as possible and potentially place priority towards mitigation efforts versus solving for the unknown.

PROBLEM



RESOURCES“Because maintenance windows are small and approval to patch is difficult to get, the need for testing patches and verifying stability on a test bed/simulator before installation is significant”(ICS CERT, Recommended Practice for Patch Management of Control Systems). The scale of a representative test bed is difficult to determine.

A key factor in determining a representative test bed is the aggregate resource requirement over all assets and systems to be tested. Individual asset testing may need to scope out toward intra-system, inter-system and operational testing to ensure updates to not represent an operational risk. In other words, micro-validation may need to scale toward macro-validation to determine operational risk.

Resource costs increase as micro and macro factors are considered. How detailed is individual asset analysis? At what scale is testing carried to? How fast is testing conducted?

INTEGRATIONValidation is not an isolated process. The information gathered during patch validation efforts has an impact on almost all other aspects of patch management. This is an integration issue. Consider the following patch management process phases and familiar graphic:

Validation provides an opportunity to gather information for additional processes.

Configuration and Software Baselines that are managed per NERC CIP-010-03 are impacted by approved changes resulting from validation efforts. This influences the Asset ID & Baseline and Availability phases.

Mitigation decisions during validation efforts may impact the applicability of current and future updates.

Validation efforts absorb a verification element of the Acquisition phase to ensure that files are authentic.

Deployment methods are verified during validation efforts prior to staging updates for deployment to production systems.

Integration into Asset and Change Management Systems seems appropriate; but, without patch validation it becomes near to impossible to tie everything together.

Asset ID & Baseline Availability Applicability

Acquisition Validation Deployment

AssetID&Baseline

Acquisition

PatchManagement


WHY VALIDATEVendor update testing and validation may not consider application and system interactions present in an ICS environment. It is these interactions which give rise to unexpected issues. Our goal is to avoid undesired events. Let’s walk through a possible event dealing with an “Adverse Impact to a BES Cyber System (BCS)”.

Above, we discussed two possible threats, vulnerabilities and updates, which may lead to this event. It would be great if we were only trying to decide between the lesser of two evils. We could weigh the potentials and be done with it. This would, most likely, result in constant mitigation since weighting will always favor operational availability.

NERC CIP-002-5.1 attributes our example event to three general possibilities: loss, compromise, or misuse. If we are able to calculate the probability for each of these three factors we have the makings of a basic fault tree structure. Consider the following fault tree for each update being considered during NERC CIP-007-6 applicability analysis.

Any of events E1, E2 and E3 may take place (logical OR gate) establishing a sum of minimal probabilities or rare event approximation. Probability sums are often the focus of risk management and mitigation efforts as they quickly reach alarming levels of likelihood.

AdverseImpacttoaBESCyberSystem

(BCS)

E1=.01

LossE2=0to1

CompromiseE3=.8

Misuse

G1

SOLUTION


E 1 E2 E3


E1 represents the loss of the BCS which is loosely based on the sum and/or product of individual equipment failure ratings. The value listed here represents a 99% uptime rating. Suffice to say that Loss is not expected. Many factors such as redundancy would normally drive this negative probability lower but NERC CIP 002-5.1 provides specific guidance against considering redundancy when evaluating asset criticality. Given that equipment availability (E1) is high and remains constant, we may simplify our fault tree by removing it.

E2 represents compromise of the BCS for a number of reasons. For simplification we will consider only a functional compromise attributed to an adverse update. As a summa-tion of basic events, this event probability will only increase as additional basic events are detailed. Without validation, we do not know if there is any unwanted behavior as the result of an update. Without historical information, we may only assume that there may (1) or may not (0) be any unwanted behaviors.

E3 represents misuse. For sim-plification, we will consider only security vulnerabilities. Adjusted security ratings such as the Common Vulnerability Scoring System (CVSS) allow us to tailor a probability of mis-use for each vulnerability to organizational factors such as current mitigation strategies and system configurations.

When concentrating only on unwanted behavior stemming from untested updates and the potential vulnerability, the probability of the adverse event is the sum E2 + E3. This is almost always a losing battle as we cannot negligently assume 0 in the absence of validation. We must only assume 1 and thus decide to forgo the update as it stands.

We have the option of moving away from compounded mitigations and baseless assumptions. Validation allows us to quantitatively decide whether we should update or mitigate. With validation, safe updates to critical vulnerabilities work their way into production while ANY unsafe updates steer toward mitigation where they belong.


MULTI-STAGE VALIDATION

Consider establishing a multi-stage test environ-ment that spans micro- and macro-validation. A comprehensive testing program will account for both existing and new assets, and include many of the following test stages:

Inspection Cold Checks Equipment Tests Intra-system Tests Inter-system Tests Operational Tests Special Tests

Validation testing may be considered a progressive series of checks and verifications scaling from individual equipment functional tests to intra- and inter-system operational tests. Validation labs may not be appropriate for all testing scenarios.

The safest environment for testing is a dedicated validation lab. As an isolated collection of resources, there should be no chance for an update to impact production operations. Additional test equipment (hardware and/or software) may be used to verify update success. Operational or special tests may require assets or capabilities not found in a validation lab and are more appropriate for secondary or production systems.

Secondary systems provide an opportunity to place partially validated updates into a real-world environment to determine if intra- or inter-system communications are fully operational. Since most systems are not isolated, there is an expectation that systems will utilize data from and provide data to other systems. Secondary systems may not have the full capabilities afforded by a lab environment and, like production systems, may be limited to minimal signal analysis and observational tests.

Production systems provide the same testing opportunities as secondary systems with a further limitation of maintenance windows. It is plausible that an update may be partially tested in a lab to determine application conflicts and basic operational state and queued for additional testing during a maintenance window in the absence of a secondary system.

As testing moves away from dedicated lab resources toward production systems, the frequency and flexibility of testing also change. Organizations should evaluate the test procedures used to verify and validate updates and determine specific resource needs.

Validation approaches vary from minimalistic to comprehensive processes. The level of detail is determined by an organization’s need for information as part of operational risk management and integration into other management processes. Generally, we may deploy a combination of two validation methodologies depending on the level of detail we need to capture: Fail Fast or Interrogation.

VALIDATE



FAIL FASTThe term “fail fast” is used to describe a results-oriented testing approach focused on expedited evaluation. When applied to update validation, a fail fast approach includes aspects such as batch deployment and operational testing. When no issues arise, the mass application of updates limits the attribution of individual update informa-tion that might be captured for change management processes.

Batch update validation processes may discover operational issues quickly and lead to investigative efforts only for those updates that are suspect. Validation is streamlined in the event of non-issues and batches of stable updates are pushed through.

Fail fast validation will ultimately focus on failure event details and less on successful updates.

INTERROGATIONThe opposite of a fail fast method is one in which each update is individually evaluated from the start. Each update is analyzed and scrutinized to better understand impact to software baselines and configurations.

Success or fail, each update follows the same testing processes to collect information and determine operational risk. This is more time intensive and results in a larger body of information to be used within other processes.


VIRTUAL VS. PHYSICALTest procedures may dictate the use of physical and virtual assets for good reason. Physical assets provide the most accurate test results while virtual assets provide the most flexible testing platform.

Peripheral tests, in particular, may limit testing to physical assets since virtual platforms utilize different driver sets. The pass-through capabilities of some virtual platforms limits their use when testing serial communications as part of intra- and inter-system testing. Procedures that use test equipment such as oscilloscopes may require physical assets as virtual platforms cannot represent these signals accurately.

Multiple asset baselines may share common physical assets. Representative computing platforms may be re-imaged to assume the role of an HMI, historian, or other asset for the duration of test procedures. The application of an image and connections to the correct peripherals and test equipment may allow dedicated validation labs to maximize available resources.

Virtual assets may be quickly established, recovered, and managed throughout the testing process. Unlike physical assets, virtual assets are available on demand and provide an opportunity to utilize abstracted test tools such as offline analysis and snapshots. When driver compatibility is not a significant risk and peripherals are absent, virtual testing is a cost effective solution for validation.

The key to validation, whether physical or virtual, is establishing a representative asset to be tested. Test plans should clearly identify asset requirements and the need for physical vs. virtual equipment.

INTEGRATED ACTIVITYAs previously stated, validation may be considered an integrated activity with many other processes and systems.

BASELINE MANAGEMENTBaseline management processes rely heavily on the capability to establish a trackable software and configuration baseline for each asset. When an application is installed or updated, it may modify, create, or delete multiple files. A comprehensive baseline includes proper attribution of all files to parent applications. Shared files are identifiable and potential conflicts or incompatibilities may be identified. Individual update validation provides the information necessary to maintain comprehensive baseline lists and proper attribution.

CHANGE MANAGEMENTChange management processes protect authorized baselines against unintended changes. This may include change detection mechanisms. Validation processes should tie into approval gates for change management. Baseline profiles across multiple assets may be established as part of validation efforts to prepare production deployment and change detections systems.

ASSET MANAGEMENTA subset of the information tracked during validation may be considered valuable to asset management processes. Of particular importance are the attributes necessary to track the availability of updates for each application and device. As applications are validated for the first time, the procedures for update checking are developed and approved. In some cases, the information may impact what details need to be tracked for each asset as part of asset management and inventory efforts.

VULNERABILITY MANAGEMENTUpdates may identify a corresponding vulnerability. After applying the update, it may be required to test the vulnerability to verify its mitigation. Given the likelihood that an update may fail to install, causes a conflict, or may simply be mitigated, any associated vulnerability may need to be verified. The integration of validation into vulnerability management may be limited to the adoption of validation processes to verify the presence or lack thereof a vulnerability.

INTEGRATE


FOXGUARD & GEGE Oil & Gas Digital Solutions and FoxGuard Solutions are very familiar with patch validation. Both firms have been involved together in cyber security for critical infrastructure markets since 2009. In particular, both firms have worked together to establish a compre-hensive validation process and validation laboratory. The result of this partnership are deployments with 175 companies at 34 sites in 30 states and 62 countries. Additionally, the patch validation process from GE and FoxGuard has allowed for the identification of an average of two security patches per month that are not ready for deployment. During the validation process, these patches were found to cause problems under certain test cases that had been established by GE resulting in the patches being held from distribution to the end customers until mitigation could be performed.

LEARN MORE ABOUT OUR PATCH MANAGEMENT SOLUTIONSGE and FoxGuard provide a wide range of patch management solutions that help companies identify and mitigate gaps in the security of their systems and prepare for NERC CIP audits. More information on these solutions can be found on our website.

In addition, a weekly webinar is available to discuss ways to develop and implement a robust patch management program. To reserve a spot, please visit us at:

http://www.foxguardsolutions.com/resources/details/patch-management-webinar

If you would like our security experts to contact you, then kindly share your contact details and a brief summary of your challenges at:

http://info.geoilandgas.com/IndustrialControl Systems_CyberSecurity.html

WHAT ACCOMPLISHMENTS AWAIT US AT THE CENTER OF THE PATCH VALIDATION MAZE?

We have solved the unknown.

We have enabled management processes.

We have minimized downtime.

In short, the key benefit is identifying which updates are going to be problematic well before they are applied in a production environment. Doing anything less invites potential catastrophe.

While many vendors supply software for their equipment and offer patch management services, they may only be performing a limited scope of validation.

Are they validated against all of the possible permutations or test cases you are concerned with?

What happens when something does not work correctly?

Is the update not installed?

Is the update fixed?

During the process of mitigation, is there a Technical Information Notification sent out?

“To validate or not to validate?” that is the question. And the answer is, absolutely and unequivocally, YES! Do so with the most representative complement of systems as possible. If further information would be helpful, FoxGuard and GE would welcome the opportunity to share best practices and thought leadership regarding patch management and patch validation.

GE I FoxGuard Solutions geoilandgas.com I foxguardsolutions.com

Documents

PATCH VALIDATION FOR ICS - foxguardsolutions.comfoxguardsolutions.com/wp-content/uploads/2017/02/... · the patch management process. ICS CERT and NIST both describe how impact plays