Atlantic First Nations Water AuthoSCADA Master Plan

Desired State Workshop

• Introductions – Consultant and Project• Review of Current State• Desired State Options• Break• Desired State Recommendations• SCADA Master Plan Next Steps• Questions

AGENDA

INTRODUCTIONS

Eramosa Engineering Inc.• Established in 1998, Guelph, ON• Currently 90 employees• Offices:

• Guelph, ON (Head Office)• Toronto, ON• London, ON• Calgary, AB• Kansas City, KS (Eramosa International Inc.)

Electrical SCADAIT/ Cyber & Other Cool

Stuff

What does Eramosa Do?

SCADA, Process Control & Communications Master Plan Introduction & Purpose• AFNWA incorporated July 18, 2018 to own, operate, and

maintain water and wastewater systems in participating communities to enhance service and build capacity within First Nation communities.

• AFNWA secured funding from Indigenous Services Canada (ISC) for asset management type initiatives.

• Intent of SCADA Master Plan project is to develop plans for the 5-year implementation of an integrated SCADA system.

Full Service - Decentralized• Reflects Hub & Spoke model• Tiered SCADA systems

• Shared visibility to communities within a Regional Operations Area

• Centralized monitoring of all systems

• Increased operational coverage

Central Headquarters

Regional Operations

Regional Operations

Regional Operations

Regional Operations

Community

Community

Community

Community

• Detailed Master Plan with costing for a fully coordinated and integrated system over 5 years.

• Recommendations for:• Technology• Resources• Training• Disaster Recovery• Cybersecurity

• Develop communication framework• Review current cybersecurity

practices & develop plans• Optimize for reliability and security• Assess SCADA needs• Recommend remote terminal unit

(RTU) standards and transition plans

• Improve historian use and alarming systems

Scope Objectives

Project Overview & TimelinesProject AwardSMP Awarded to Eramosa

Project Initiation & Scheduling

Q4 2020

Desired StateReview Industry Best PracticesEvaluate Technology Options

Develop Technology Recommendations

Q2 2020

Current StateSite InvestigationsPhone Interviews

Current State Evaluation & Summary Report

Q1 2021

Master PlanRecommendations Priority Scoring

Implementation RoadmapCapital & Operating Budgets

Q3 2021

REVIEW OF CURRENT STATE

Purpose• Gain an understanding of facilities and processes• Identify existing SCADA assets• Document current conditions• Assess maturity of existing systems• Identify what works well, what doesn’t and why• Understand where each community is with SCADA today• Focus on product obsolescence, level of available support and

alignment with industry best practices and standards

Approach• Review of background information• Site visits• Virtual workshops with communities constrained by pandemic

travel restrictions• Evaluate existing infrastructure against industry best practices• Preliminary recommendations for individual communities,

facilities, and systems

Summary• Little or no documentation• Mix of hardware platforms• Wide range of software versions in

use• Several operating systems no

longer supported• Minimal security measures in place• Majority of sewage lift stations not

integrated with SCADA

Component Level 0 Level I Level II Level III Level IVCommunity A W/WWCommunity B W/WWCommunity C WW WCommunity D W/WWCommunity E WW WCommunity F WW WCommunity G W/WWCommunity H W/WWCommunity I W/WWCommunity J W/WWCommunity K W/WWCommunity L W/WWCommunity M WW WCommunity N W/WWCommunity O WW WCommunity P WW W

DESIRED STATE OPTIONS

• Evaluate options for the AFNWA

• Provide information to make critical decisions about the evolution of the AFNWA’s SCADA system

1. Remote Terminal Units (RTU)2. Human Machine Interface (HMI)3. Communications4. Architecture5. Cybersecurity6. Data Management7. Alarm Management

Purpose

What Makes Up A SCADA System?

Field DevicesPumps, Valves

InstrumentationLevel, Pressure, Flow,

Chlorine Residual

Control Panels with RTUsAutomatic equipment

control based on instrumentation

SCADA Computers (HMI)Remote Monitoring

Remote ControlAlarming

Historical Data CollectionTrending

Electrical Wiring

Communication Cabling

Fibre OpticsDSL

Coaxial

CellularNetwork

CommunicationsOptions for media and

technology including wired and wireless

What is the driver for recommendations and upgrades for each component of the SCADA system – Speed, Reliability, Performance, Stability, Security? Can existing systems be leveraged?

ApproachBest Practices

What are other leading municipal water and wastewater owners doing that has been successful and can be applied to the future of AFNWA’s SCADA systems?

Industry Trends Where is technology headed for the industry? Has it been proven stable and beneficial and can it benefit AFNWA?

Technical/Business Objectives

Published Standards

User Needs

Are there document standards or guidelines which provide recommendations in a specific area that can steer the direction of the AFNWA’s SCADA systems?

Have the needs of operations and management staff been considered in addition to strict technology-based recommendations? Are the options proposed ‘right-sized’ for the user to operate and maintain?

Remote Terminal Units (RTU) – Current State Summary• Predominantly Allen-Bradley field controllers, various

generations• Mix of others including Modicon, Siemens, Motorola, Schneider

and Omron• Most hardware still current and can be supported

Desired State Options

Desired State Evaluation• Considerations for product standardization

• Investment to date• Local distributor support for spare parts• Local integrator support for programming, troubleshooting,

maintenance and upgrades• Programming• Consistency and standardization

Human-Machine Interface (HMI) – Current State Summary• 8/9 communities with 1 or more SCADA computers are using

VTScada by Trihedral• Majority not using latest version released• Generally low tag count licenses• Many workstation computers nearing end of life• Half of operating systems are Windows 7 which is no longer

supported and presents a significant risk to security

Desired State Evaluation & Options• Any vendor can be standardized on• Considerations for product standardization

• Integrator support• Product/company stability• Underlying technology• Some web-based

Desired State Evaluation• Installation Base Today (Ontario)

Desired State Evaluation• Leaders Looking Forward

Communications – Current State Summary• Mix of technologies connecting sites within each community

• Unlicensed/licensed radio, fibre optics, leased lines• Most SCADA computers connected to internet for remote

access• Exact connection details not all available i.e. fibre optics, DSL, etc.

Desired State Options• Wired Technologies:

• Digital Subscriber Line (DSL)• Coaxial• Fibre - 3rd Party

• Wireless Technologies: • Cellular (LTE)• Radio – Unlicensed Bands (Ex. 2.4GHz, 5.8 GHz, 900MHz)• Satellite• Radio – Licensed Bands (Ex. 450MHz)• LoRaWAN (Long Range Wide Area Network)• WiMAX (Worldwide Interoperability for Microwave Access )

Desired State Options• Wired Technologies:

• Digital Subscriber Line (DSL)• Coaxial• Fibre - 3rd Party

• Wireless Technologies: • Cellular (LTE)• Radio – Unlicensed Bands (Ex. 2.4GHz, 5.8 GHz, 900MHz)• Satellite• Radio – Licensed Bands (Ex. 450MHz)• LoRaWAN (Long Range Wide Area Network)• WiMAX (Worldwide Interoperability for Microwave Access )

Desired State Evaluation• Considerations

• Availability• Bandwidth• Security• Reliability• Capital Cost• Operational Cost• Redundancy Options

Architecture – Current State Summary• Those communities with SCADA systems each have dedicated

workstation computers• Some share a computer for Water/Wastewater while others

are dedicated• No redundancy in place with SCADA systems• No virtualization in place with SCADA systems• No backups in place with SCADA systems

Desired State Options1. Existing on-premise SCADA servers2. Hub site SCADA servers3. Cloud hosted SCADA servers4. Hybrid Cloud & On-Premise SCADA servers

Cybersecurity – Current State Summary• SCADA systems connected directly to the internet• Access credentials written down in plain sight• Common access credentials• Panels not locked• IP addresses written down in plain sight• Unnecessary application software installed on SCADA

computers• O/S out of date and unsupported

Desired State Options• Do nothing• Apply a right -sized

defense-in-depth approach

Data Security

Application Security

Endpoint Security

Network Security

Physical / Perimeter Security

Data Management – Current State Summary• Not all SCADA computers collecting historical data for trending

purposes• No dedicated historian computers• No data backup in place• None equipped with automated or electronic reporting• All paper based manual transfer of data for reporting

Desired State Options• Historical Data Collection

• Native (SCADA platform intended) time series or process historian• Relational database (Oracle, MySQL, MSSQL, etc.)• Whichever option is chosen, should be dedicated for SCADA data

Desired State Options• Reporting

• VTScada has built in reporting on SCADA data, including pre -defined SCADA Reports

• Variety of third -party products (e.RIS, XLReporter, Dream Reports)• Product selection may depend on AFNWA’s needs/desire to report on data from

non-SCADA sources• Can provide automated PDF, emailed, printed reports of compliance

and other data to eliminate manual data entry/transfer

Alarm Management – Current State Summary• Mix of software and hardware alarm dialers• Some communities not equipped with remote alarm

annunciation at all• Many communities have alarms sent to only a single individual

i.e. no escalation processes in place

Desired State Options• Software Dialers

• Native to SCADA (Ignition/VTScada)• VTScada - Alarm Notification System

• Third-Party (WIN-911, TopView)• Both do not connect with VTScada

• Hardware Dialers• Physical infrastructure, options for communicated or hardwired signals

• Combination of hardwired and software• May require data concentrator PLC to configure/architect in a

redundant fashion

DESIRED STATE RECOMMENDATIONS

RTU Desired State Options

RTU Evaluation

RTU Recommendations• Rockwell Automation – Allen-Bradley –

CompactLogix Series• AOI/UDT code use for modularization• Expandable I/O and memory• Interoperability• Native Ethernet CIP communications• Excellent distributor support geographically• 8/10 AFNWA communities with PLCs installed

have at least some AB hardware• Strong integrator support

Modular

ScalableOpen Architecture

Protocol Support

RTU Recommendations• Generally, low priority replacements/upgrades

• No known concerns with existing hardware/software functionality• Some outdated/unsupported can be replaced earlier• Most should communicate to any SCADA platform,

gateways/ converters can be used in rare situations• Recommend developing hardware/ software standards and

guidelines during 1st year with implementation following in later years

Desired State Evaluation• Leaders Looking Forward

HMI Recommendations• Standardize on VTScada

• Strong integrator knowledgebase and level of support• Company is local to the installation area• Historical upgrade path/approach is excellent• No reliance on underlying O/S components• Shortest time to release for security patching• License purchase and support cost is competitive with others

HMI Recommendations• Review options for licensing and architecture with Trihedral• Short-term focus on addressing HMI needs directly related to

cybersecurity, possibly temporary measures• Develop standards for tagging, graphics, colours, alarming,

historical data collection, etc.• In combination with Communications recommendations,

develop detailed SCADA architecture design• Rollout new SCADA platform in parallel with existing

community systems with staged transition

SCADA Server Communications RecommendationsRecommendation Priority Primary Connection Secondary Connection

1 Fibre Optics Coaxial2 Fibre Optics DSL3 Fibre Optics Cellular4 Coaxial DSL5 Coaxial Cellular6 DSL Cellular

Remote Site Communications RecommendationsRecommendation Priority Technology

1 (Existing Connection) 900MHz Unlicensed Wireless1 (New Connection) Cellular2 (New Connection) 5.8GHz Unlicensed Wireless3 (New Connection) 2.4GHz Unlicensed Wireless4 (New Connection) 900MHz Unlicensed Wireless5 (New Connection) Internet (DSL/Coaxial)

Architecture Evaluation1. Existing on-premise SCADA servers2. Hub site SCADA servers3. Cloud hosted SCADA servers4. Hybrid Cloud & On-Premise SCADA servers

Architecture Recommendation• Hybrid Cloud & On-

Premise SCADA servers

Cybersecurity Recommendations• Develop a cybersecurity governance framework• Apply user-based HMI security with role -based

functionality and auto -logout/timeout• Utilize firewalls on internet connected systems• Implement intrusion detection and prevention systems• Establish a centralized logging and monitoring solution• Implement endpoint protection• Implement a disaster recovery and backup solution

Identify

Protect

Detect

Response

Recover

Cybersecurity Recommendations• Cloud architecture for centralized

services• Firewalls at community plants• Resources• Training

Community Plant / Group Hub

FortiGate Cloud Firewall

Firewall Allowing Specific connectivity

IDS/IPS Guardian

Community Facility

Fortigate

Remote Client(s) with

Endpoint protection

VPN W\Strong Encryption

Remote Client(s) without

Endpoint protection

SSL Encryption

Offsite DR

Onsite Backup

Active DirectoryPrimary

AD Secondary

SCADA Systems

IDS Guardianremote controller

SIEM

Data Management Recommendations• Native SCADA time-series (process) historian• Location, backups, tiered collection, etc. all dependent on

SCADA and communications architecture• Automated reporting options• Potential integration of manually recorded data with

automatically recorded SCADA data

Alarm Management Recommendations• Software based alarm dialing system only

• Highly configurable for alarm schedules, escalation, etc.• Product tied to SCADA platform recommendation, recommend native

for VTScada instead of third -party• Hardware based dialers not recommended

• Typically requires on-site presence to modify• Requires hard connection to at least one PLC• Limitation of alarming functionality and density• To follow full service decentralized model, data concentrator(s) would

be required

Communities with MTAs• Option 1

• Data sharing and remote access to SCADA system owned by the authority operating

• Option 2• Install dedicated SCADA system owned by the community• Install PLC panel(s) and connect to existing instrumentation, or add

new for flow, pressure, residual at transfer points• Connect to existing PLC’s at facilities owned by community, but

operated and maintained by others

Sewage Lift Stations• Large number of stations with no remote monitoring or control• Many in poor condition, electrically• SMP team coordinating with Asset Management project team

to align recommendations• Options to tie new PLC control panels and SCADA into existing

systems• Potential benefits of combining replacement of electrical panels with

addition of SCADA panels

SCADA MASTER PLAN NEXT STEPS

Quick Wins• Recommendations made within the current state summary• Should be undertaken independent of SMP recommendations• Includes:

• Cybersecurity recommendations• Obsolete/unsupported hardware/software replacement• Implementation of simple policies and procedures• Software version updates

Projects• Summarizing all technical recommendations into manageable

chunks for detailed design and implementation• Evaluation of importance and urgency of each• Consideration for pre -requisites – which recommendations

require others be undertaken prior• Identification of projects which can be combined for efficiency,

time savings, or cost savings

Resources & Training• Identification of technical resources required to assist with

implementation and long-term support• Identification of training

• For technical personnel• For operational personnel

• Recommendations to be incorporated into SMP implementation roadmap and capital expenditure plans

Implementation Roadmap• Visual representation of recommendations

• By year• With prerequisites identified• Includes project groupings

• Can be used to track/manage SMP implementation phase over the next 5 years

• Will aid in capital expenditure (budget) development and tracking

Capital Expenditure Plans• Estimated budgets per project

• Design effort• Implementation effort• Materials costs• Markups

• Accounts for escalation

SCADA Master Plan• Executive Summary• Operational Alignment• Current State Summary• Desired State Options• Desired State Recommendations• Implementation Roadmap• Capital Expenditure Plans

QUESTIONS?