E. MatiasCanadian Light Source

University of Saskatchewan

CLS Control System Overview

Agenda

• Introduction (§1.0)– CID Staff– Goals– Background– Comparisons– Major Projects

• Functionality (§2.0)• Architecture (§3.0)• OPI Software (§7.0)• IOC Software (§8.0)• Device Level (§9 & 10)

• Other Interfaces (§11)• Timing System (§12)• Secondary Sys. (§13)• Tools (§14, 5, 6)• Servers (§15)• Networking (§16)• System Software (§17)• Anticipated Changes• Syn-apps• Conclusions

CID Department

Students – Intern (3)Students – Summer (1)

Controls & Instrumentation Development Department Manager

Elder Matias

Controls Systems Engineer/LeadRobby Tanner

RF TechnologistJonathan Stampe

Staff Scientist (Instrumentation)Johannes M. Vogt

Electronics TechnologistHeinz Buchmann

Control System Engineer/LeadMike Mckibben

Electronics TechnologistDenis Beauregard

Computer TechnologistCarl Finlay

Electrical Technologist/ForemanNeil Hovdestad

Systems Analyst/Control LeadsRuss Berg

Tony WilsonGlen Wright

Dioni Medrano (Term)

Control System Engineer/LeadHao Zhang

David Beauregard (Term)

Electrical Technologist Wayne McWillie

Doug Starns

RF TechnologistGlen PeggFuture (1)

Instrumentation Technologist Carl Jansen

Carmen Britton

Software Specialist (EFD)Ru Igarashi

Electrical Engineers/Controls Lead (ETS)Neil JohnsonWade Dolton

Goals

• Review our existing design.• Identify areas that we can improve.• Identify functionality that is needed.• Identify options that we should explore.• The architecture should be:

– Maintainable– Flexible– Reliable– Support our functional requirements– Low cost to maintain and support (labour)

• Once a system is deployed and commissioned we provide on-call support during scheduled beam time.

Background

• SAL (1964 to 1999)– Linear accelerator, EROS ring, Experimental Areas 1, 2

and 3– Control system based on CAMAC, Digital PDP, VAX and

Sun with some NeXT and Amiga computers.– LUCID data acquisition program

• CLS (1999 ….)– Highly distributed system (extensive use of RTEMS, single

board computers, and optical links to VME crates)– Based on EPICS– Diagnostics beamlines (2)– phase I beamlines (7)– phase II beamlines (7)– phase III beamlines (TBD)

Making Comparisons

• Scope– Initial build project $5 M (equip)

+ labour– In operations approximately $2 M /year

+ special projects

• Comparisons– Larger Staffing Level than MAXLab– Smaller Staffing Level than APS

• Good Comparisons– SPEAR 3? - similar parameters – BESSY? – similar size– other mid. size regional/national machines?

• SAC, and others compare us to APS; is this reasonable? ….

SafetyInterlocks

(17)

BeamlineControls

(11)

Controls(26)

RF(20)

PowerSystems

(13)

Diagnostics(17)

BuildingServices

(?)

Conv.Facilities

(?)

APS = 110CLS = 20

AccelSoftware

(??)

Leveraging Our Size

• Perhaps APS and CLS are not the best comparison.

• We need to take advantage of our size– Innovative, decisive, cross-disciplinary– Leverage technology and design choices across systems

• Technically evaluate our design choices• Adopt best-practice, from both within and outside of

the EPICS and synchrotron community• Move more towards a “software engineering”

paradigm– Provincial Legislation for “certification” of system analysts -

software engineers became law in 2005, as a first step towards licensing

– The “software engineering” methods are intended to reduce development costs and improve quality

Major Projects

• Phase I and Accelerator Operational Support• Phase II beamlines• CANARIE Remote Access• Infrastructure

– VME Monitor Program Upgrade (Russ, Neil J. + Intern)– CS-Studio Tree Explorer (Glen W. + Intern)– Data Acquisition Program Upgrade (Glen W, & Ru)– Next Generation Motor Control (Mike M. & Tony W.)

• Upgrades– Diagnostics Kicker, X-ray BPM and time resolve (Johannes)– Linac Gun &RF (Neil J. & Hao)– Linac ACIS System (Robby)– Replace remaining pre-1980 controls (Hao + summer student)

Agenda

• Introduction (§1.0)– CID Staff– Goals– Background– Comparisons– Major Projects

• Functionality (§2.0)• Architecture (§3.0)• OPI Software (§7.0)• IOC Software (§8.0)• Device Level (§9 & 10)

• Other Interfaces (§11)• Timing System (§12)• Secondary Sys. (§13)• Tools (§14, 5, 6)• Servers (§15)• Networking (§16)• System Software (§17)• Anticipated Changes• Syn-apps• Conclusions

VLANs for: each beamline, machine control, development, office, visitors

VME Crate

(Reflective Memory)

MicroStep

EROCIOC

RTEMS

FieldDev. RS-232

Devices

OPI

Linux

IOCStep Controller

RTEMS

Motors

MicroStep

OPI

Linux

OPI

Linux

Touch PanelOPI Linux

NetworkServer

(bootp, dhcp,auto restore)

Linux

DataArchiveServer

Linux

AlarmServer

MS-Win

MS-SQLServer

MS-Win

PowerEdgeIOC

Linux

PS BoardsIOC

RTEMS

PowerSupplies

EROCIOC

RTEMS

FieldDev.

EthernetDevices

PLC & GPIB

FieldDev.

MagnetsMotors

1GigBridge

IOC

Linux

FieldDev.

ProfibusPLC

System Architecture (3.0)

Agenda

Turning to the design document….• Introduction (§1.0)

– CID Staff– Goals– Background– Comparisons– Major Projects

• Functionality (§2.0)• Architecture (§3.0)• OPI Software (§7.0)• IOC Software (§8.0)• Device Level (§9 & 10)

• Other Interfaces (§11)• Timing System (§12)• Secondary Sys. (§13)• Tools (§14, 5, 6)• Servers (§15)• Networking (§16)• System Software (§17)• Anticipated Changes• Syn-apps• Conclusions

Tools

• PV Crawler

• Cables Database

• Sequencer

• MKS Source Integrity

• CS Studio

• Rational Software

Network

Office

Gateway

MS-SQL

50

BAOPI

AlarmPrinter

OPI

IOC

IOC

IOC

PLC

OPI

IOC

IOC

Server Subnet

Office Subnet

Beamline BBeamline A

Main Control Subnet

Syn-apps

• Considered in 2001– reviewed as part of EPICS fact finding – part of the EPICS versus Altersys ECR

• Considered (partially) in 2004– an option for Mono-control

• Considered in Feb. 2005– see memo from E. Matias.

• Considered again in Nov. 2005– see ECR

The Next Step…• Deploy MKS Problem Tracking and the New MKS

– This should provide more responsive and predictable operational support– More advanced and configurable access control

• Participate in Building and Deploying CS Studio– This tool will provide the ability to permit users and beamline scientist more direct

access to configure PV variables – Provide automation for the generation of configuration files

(db files, alarm handler, gateway, data archiver, auto-save-restore, edm screens)– Better support for re-factoring

• Exploit Lessons Learned from CANARIE Project– First major project based strongly based on UML– First major project to make extensive use of CASE tools– First major project to use Java and Web Services paradigm

• Enhance VME Monitor Program– Improved performance– Improved data collection

• Improve Data Acquisition Program

• Move to Next Generation Motor Control (perhaps based on ESRF design?)

• Improve capability for time resolve and diagnostics

Development Process

E. Matias

Inception Elaboration Construction Transition

Requirements Analysis Design Implementation Test

Unified Process

Core Workflows In Each Phase

- Establish Feasibility- Establish Business/Scientific Case- Capture Essential Requirements- Identify Critical Risks- Establish initial budget & schedule

Deliverables (as required):- Project Plan- Risk Assessment- Initial Requirements (10-20%)

Requirements: - Refine System ScopeAnalysis: - Establish what to buildDesign:- Create an ArchitectureImplementation- Build an architecture baseline- Build any prototypesTest- Test the architecture baseline- Test any prototypes

Deliverables (as required):- PID Drawings- Wiring Diagrams- Updated Requirements Document- System breakdown

Requirements: - Uncover missing requirementsAnalysis: - Finish the analyis Design:- Finish detailed designImplementation- Build and install the systemTest:- Test and ring-out the system

Deliverables (as required):- Running System

- Correct any defected- Provide support for commissioning - Prepare final documentation

Deliverables (as required):- Final documentation- Working system

CLSI System (Beamline) Engineering Process

ProposalConceptual

DesignPreliminary

DesignDetailedDesign

Build Commissioning

Some recommendations?

• Should we get rid of requirement gathering?– If anything the reverse is true, we need to improve our requirements capture

process…• Should we get rid of P&ID drawings?

– They have become critical to our design process. We are moving to add more not less information to P&IDs….

• Should we get rid of Wiring Diagrams?– Without them (or something equivalent) we can’t maintain a system this size with

the staff we have. • Should we get rid of design standards?

– They have been critical to being able to maintain a system this size with the staffing levels we have.

• Should we get rid of ECRs & ECOs?– As part of our license renewal applications the CNSC was concerned we did not

give them sufficient detail and asked for more.• Should we remove access controls?

– For the most part, the beamline administrator accounts provide a great deal of control.

– Difficult to justify after trying to undo changes in the middle of the night.

Winter at the CLS

Moving Forward

• Current CLS Control System– Built on a common design (circa 2000 and 2002)– Homogonous – Common structure and design across the facility– Built on 6 years of EPICS experience – Built on SAL (30 years) of accelerator and nuclear physics science

• Critical Questions– Does it represent best industry practice?– How can we improve quality (scientific capability and the user experience)?– How can we improve efficiency?– Is it safe and ethical?

• How Do We Answer These Questions?– Collectively (cross-disciplinary)– Open to new ideas and methods– Invite people from inside/outside of EPICS to help us….– Invite people from inside/outside synchrotrons to help us…– Built on our in-house experts…– Decisions to change are technically driven with backup (ECR)– Exploit automation to reduce costs and increase reliability

• Shift in approach, from just-in-time make-it-workbuilding well designed, structured reusable applicationssupport highly configurable applications configured just-in-time