Patching Has Its Place in SCADA and ICS Security

Actually, I think applying patches is a critical part of good security. According to US-CERT, about 95% of all network intrusions could have been avoided by keeping systems up to date with appropriate patches. If you never patch, you are leaving your system open to a decade of malware.

What I am against is patching as a knee-jerk reaction to security vulnerabilities. You can't expect your control system to operate reliably if you don't have a controlled process for patching.

In the words of Richard Brown, at Dow Chemical: “Patch management is about managing the risk of change.”

Patches are changes to your system. Changes to your system need to be managed. One cannot blindly deploy new patches into the process control environment without risking disruption of operations. Thus careful policy and practice is required to balance the need for system reliability with the need for system security.

A successful patching strategy balances system reliability with system security.

Prioritize Your Security Patches

There are a number of recommendations on patch management, but one of my favorites comes from the Edison Electric
Institute (EEI). Their suggestion is to push down patches to machines on a priority basis. The priorities are based on two factors: the criticality of the system being patched and the criticality of the patch.

The EEI process requires two sub-systems to be set up. The first sub-system involves an inventory in which all machines are prioritized and categorized into groups that define when and how they are to be patched.

Some examples are 'Early Adopters', who receive patches as soon as available and act as Test/Quality Assurance machines. Typically, these are lab or training computers. 'Business Critical' machines are those that are patched automatically when early adopters have been stable for a set period of time (depending on the patch's level of risk) and approval for the patch has been received from the control system vendor. This escalates up to 'No Touch' machines that require manual intervention and/or detailed vendor consultation before a patch is applied.

The figure below is an example from the pharmaceutical company Astra-Zeneca (A-Z), and shows the patch cycle for their systems. Note that none of the A-Z patching is performed in a rush - there is always a process to collect feedback from one stage before embarking on the next stage.

Astra-Zeneca Illustration from "SCADA and ICS Patching: On the Horns of a Dilemma" presentation. Source: Joakim Moby, Astra-Zeneca, ISA Expo 2006

The second sub-system is a procedure for keeping track of newly released patches and their level of importance to process operations. Whenever a new vulnerability is announced and/or a patch fix is available, it is tracked for its potential impact - good and bad - on the company control system. This patch is then evaluated and prioritized for adoption based on its risk evaluation.

For example, the risk evaluation could follow a set process of questions to decide on urgency of patching and level of testing required. These questions might include such concerns as "Are we currently being exploited?" or "What is the severity level designated by the vendor?" or "Has it been tested by the vendor?"

The risk evaluation would result in an overall implementation level being set. Considerable guidance for making this evaluation is available from vendor sites, such as Honeywell’s Microsoft Security Hot Fixes Qualification Matrix (you’ll need login credentials to access this), which reports on the testing status of newly released patches.

Implement a Patch Reaction Plan

The patch implementation levels are preset and tied to Reaction Plans. The table below shows an example of several different Reaction Plans. These vary according to the risk of not applying the patch.

For example, if the risk that a given patch addresses is low, then the adoption and testing cycle can be slower. If it is a very high risk, then the patch needs to be deployed more aggressively. More (or fewer) levels than the three examples below can be created as required.

Any patching plan requires close cooperation with all software and system vendors. Many vendors already have a system of prioritizing patches and approving their application that should be tied into the internal patch management system. As mentioned earlier, Honeywell maintains a patch approval process that usually approves patches for their newer versions of software within five days of a Microsoft patch release - and often within hours if the patch is critical.

Once the decision is made to patch a system, it is critical to have a secure method to distribute those patches. Distributing them directly from the business systems is not a good idea. For example, the Honeywell Security Guidelines state: “It is not best practice to distribute Microsoft hotfixes, patches, and updates to virus definition files directly from the business network to nodes on the process control network as this is contrary to the goal of minimizing direct communication between nodes on these networks.”

Most vendors recommend that a dedicated patch manager and an anti-virus server be located in the Demilitarized Zone (DMZ) between the control system and the IT network. Both roles can be often performed by a single server.

Finally, there are a number of automated tools and services available to assist companies in performing patch management. These typically include methods to inventory computers, identify relevant patches and workarounds, test patches, and report network status information to various levels of management.

Using this type of tool can significantly improve the response time for deploying critical patches, while at the same time reducing the work load on process control or security staff. For example, a major company involved in food processing reported that a single individual was able to successfully manage all PCN patching over six large company sites once the company deployed a patch management tool.

Repeat After Me - “Planned Patching is Good. Reactive Patching is Bad”

Don’t get me wrong, I’m not saying don’t patch! Far from it - patching for vulnerabilities is critical for good security. However, as I noted in my last blog, the IT strategy of rapid, reactive, continuous patching on a regular basis just isn’t feasible for SCADA and ICS systems.

For a patching strategy to be successful it must be planned properly and it must include testing and change management controls. Without these processes in place, you are putting the control system at serious risk.

In my next blog, I’ll explain how compensating controls can provide a workable alternative to patching in a hurry.