Principal Cyber Security Manager @bryansowen · •Incident response at OSIsoft –Limited to PI System domain of expertise •3 case studies supporting large user communities –No

© Copyright 2016 OSIsoft, LLC© Copyright 2016 OSIsoft, LLC

Bryan S Owen PEPrincipal Cyber Security Manager

@bryansowen

No Stone UnturnedStories from the Aftermath of Cyber Incidents

© Copyright 2016 OSIsoft, LLC 2

© Copyright 2016 OSIsoft, LLC

Who’s next?

3

Hollywood hospital hit with ransomware:

Hackers demand $3.6 million ransom


NIST CSF Core Function – “Recover”

4

• Recovery Planning

• Improvements

• Communications

…plans for resilience and to restore any

capabilities or services that were

impaired due to a cybersecurity event…


Long Tail Cost Example – SCADA

5

Dave Kennedy and Ryan Macfarlane (Oct 2010) – The long tail of information security

https://www.trustedsec.com/october-2010/the-long-tail-of-information-security


The Future of Security…

Professionals defend the tail too.

6


Disclaimer

• Incident response at OSIsoft

– Limited to PI System domain of expertise

• 3 case studies supporting large user communities

– No direct attacks on PI System, collateral damage

– Unofficially attributed to nation state actor

– IOC and forensic analysis by others

– Please don’t ask who, when, where, etc.

7


PI Interfaces

PI Connectors

PI Server

CollectManage

EnhanceDeliver

The PI System Layers

PI Data Access


Case 1

Enterprise

Zone

Cell/Area

Zone

Manufacturing

Zone

DMZ

Level 0

Level 1

Level 2

Level 3

Level 4

Level 5

Attack with Impact


Timeline

Respond

T0

“Disconnection”

Recover

T0+47 days

“Communication”

Recover

T0+140 days

“Improvement”

Recover

T0+320 days

“New Normal”

10


Respond

• Tool to verify installation files and registry entries

• Tool to verify meta data and configuration

• What to look for in logs to identify anomalies

• Review of known vulnerabilities

• Guidelines for anti-virus

11


Recover – Lessons Learned

• Post breach verification of big data sets is exhaustive

– Limit scope with read only PI archives

12


Restore

• Comprehensive architecture review

• Hardware platform and operating system concerns

• Application resiliency and residual risk

• Consideration of non-routable protocol

• Data flow diagrams and one-way enforcement

13

Webinar with Bryan Owen and Ralph Langner – Trends in OT Security

http://pages.osisoft.com/OTSecurityTrendsOD.html


Restore – Lessons Learned #1

• Communication

– Involve executives early and often

14



• Diminished Risk Appetite

– Unclear approval process to restore service

– Plan resiliency for long disconnects

15



• Network segmentation

– DMZ between IT/OT

proved very effective

– Add unidirectional flow

control as appropriate

16

NIST 800-82r2 Figure 5.4

Paired Firewalls between Corporation Network and Control Network


Case 2

Enterprise

Zone

Cell/Area

Zone

Manufacturing

Zone

DMZ

Level 0

Level 1

Level 2

Level 3

Level 4

Level 5

Attack with Impact


Timeline

Respond

T0

“Disconnection”

Recover

T0+18 days

“New Normal”

Respond

T1

“Disconnection”

Recover

T1+3 days

“New Normal”

18

1 year


Respond

• Could attacker use PI to actuate control?

• Advice on potentially sensitive data or configuration

19


Respond – Lessons Learned #1

• Planning during crisis isn’t ideal

– Have a plan

– 2nd time was ‘rote execution’

20


Respond – Lessons Learned #2

• Narrow the scope of response activities

– Monitoring only

– Contain control protocol and credential use to zone

21

Enterprise

Zone

Cell/Area

Zone

Manufacturing

Zone

DMZ

Level 0

Level 1

Level 2

Level 3

Level 4

Level 5

Dedicated Logon Authority


Restore

• Assistance with restore procedures

• Review all password persistence

• Advice on changing host name, IP address, TCP port

(security by obscurity)

22



• Time to restore key factors:

– IT professionals and vendor remote access

– Virtualization infrastructure

23



• Credential theft and reuse is a chronic problem

– Restrict direct access to database servers

– Shift majority of load to application servers

– MFA recommendation dismissed

24


Case 3

Enterprise

Zone

Cell/Area

Zone

Manufacturing

Zone

DMZ

Level 0

Level 1

Level 2

Level 3

Level 4

Level 5

Attack with Impact


Timeline

Recover

T0

“Communication”

Recover

T0+21 days

“Improvement”

Normal

T0+26 days

26


Response – Lessons Learned #1

• Plan for a rooted layer 3 computer

– Disconnect is not sufficient

27


Response – Lessons Learned #2

• Communication of forensic summary was helpful

– Theft of trade secrets

– Supply chain responders can optimize effort

28


Restore

• Deep dive on PI remote diagnostic service

– What is sent from diagnostic center to field agents

– Automatic update mechanism

– Crypto and minimum OS specifications

29



• ‘Repave’ the affected systems

– Migrate validated data and configuration

– Upgrades may not be deemed appropriate

30



• Remote diagnostic service

– Reduces overall response time

– Due diligence and strict platform prerequisites

are justified

31


Summary

• Stage 2 ICS attack observables are sparse

• Defending the ‘long tail’ of ICS risk is doable!

• Heed the sliding scale of security

32


Thank YouBryan S. Owen PE

OSIsoft – Principal Cyber Security Manager

[email protected]

Documents

Principal Cyber Security Manager @bryansowen · •Incident response at OSIsoft –Limited to PI System domain of expertise •3 case studies supporting large user communities –No