Upload
timothy-mccoy
View
218
Download
0
Embed Size (px)
Citation preview
ICS – Software Engineering Group
1
SNS Project Status
Ernest L. Williams Jr.
27 April 2005
EPICS Collaboration Meeting @ SLAC
ICS – Software Engineering Group
2
Outline
Installation Status» Global Systems
– Network– Timing– MPS– CCR
» Accelerator Systems» Cryogenic Controls » PPS» Target Controls
Operational Improvements Issues Summary
ICS – Software Engineering Group
3
The Spallation Neutron Source
The SNS construction project will conclude in 2006
At 1.4 MW it will be ~8x ISIS, the world’s leading pulsed spallation source
An upgrade to 3MW has been approved for CD0 planning.
ICS – Software Engineering Group
4
Communication Infrastructure is nearly complete
CD4 Fiber Backbone is complete
Tubes installed for future addition of “blown” fiber
“Global” cables nearing completion
» Ethernet» Timing» MPS
All accelerator cable SROs entered
CISCO Switch firmware bug found (and fixed)
Fiber Backbone
Cable Infrastructure
ICS – Software Engineering Group
5
Timing System Installation is Keeping Up
All IOCs (including NADS)
Multiple Fixed Rep Rates for RF Systems
CLO Test Timing Network» Infrastructure for Diagnostics,
LLRF, XFD, and Power supply groups
Line Synch monitor» Delivered from BNL» Setting up test network for XFD
Chopper testing
“General Time” added to EPICS» Provides best time from multiple
sources» Assures monotonicity» Accepted into “EPICS Core”
SNS Event Link Monitor» Event statistics» Actual Events vs.
Scheduled Events
ICS – Software Engineering Group
6
Number of MPS Inputs has nearly Doubled(170 → 313)
Linac Only
MPS Inputs MEBT_BS CCL_BS L_Dmp MEBT_BS CCL_BS L_Dmp
IBCM 1
BLM / ND 66 129 66 129
Beam Stop / Dump 2 4 6 2 4 6Intercepting diagnostics 11 27 6 11 27 6
Power Supplies 11 15 52 11 61 90
Vacuum 3 8 10 3 8 10
RF Systems 11 26 104 11 26 104
MPS inter connects 9 4 6 9 4 6DBCM 6
Totals 47 150 313 48 202 351
Commissioning - no correctors After CD4 - correctors added
All linac MPS hardware installed – includes 27 new chassis
Copper phased out, using fiber everywhere
After CD4, corrector PS status will be added
170
ICS – Software Engineering Group
7
Operations have moved to the CCR(Central Control Room in CLO)
Move took place in February as scheduled
Two console arcs are complete (ahead of schedule)
PPS and ODH systems moved (That was the worrisome part)
Software Development Area in use All work planned for FY05 is
complete Servers are moved to “CER”
(Central Equipment Room) TPS also now in the CCR Prototype Console moved to
Target Control Room Upgraded all multi-head OPIs
» Faster processor» More RAM» Faster graphics adapter (Matrox
G450) X-windows and Java Apps have a
much better performance
PPS Equipment
ICS – Software Engineering Group
8
Linac Controls Installation isKeeping Up with the Project Schedule
We have kept up (barely!!) with RF testing, RF Conditioning, PS testing, cryomodule cool-down, etc.
Medium-Beta SCL controls mostly complete» Cryomodule MB1 controls testing remains» Some “warm section” vacuum controls testing remains» Some MPS hardware installation remains
High-Beta SCL controls approaching completion» RF, Cryo, & Vacuum controls checkout are well along» Magnet power supply interface checkout has been
integrated with overall power supply checkout
HEBT controls installation in progress Automated procedures have helped keep schedule
ICS – Software Engineering Group
9
Linac LLRF Installation is Keeping Up(This is the biggest task in terms of numbers and complexity)
Number of installed LLRF IOCs: 46 (up from 12) for 80 systems (out of a total of 96 systems)
Configuration automated via scripts, avoiding inconsistencies between SCL systems
From last ASAC: "Add Adaptive feed forward to firmware if required"
» Was required for higher beam current» Not foreseen in current FCM, so successfully added to IOC software
– recently refined to eliminate parameters that were hard to tune
Automation of RF ramp-up, warmup, loop closure» Allowed hands-off recovery from trips on the NC Linac» Needs adaptation for SCL
Waveform updates synchronized to timing system» Allows physics apps to compare data from several systems for the
same beam pulse Refining overview screens, giving better notice of trip reasons,
active overrides, etc.
ICS – Software Engineering Group
10
Cryogenic Control System nearly complete!!
All CHL Controls equipment installed and
All IOC and PLC programming complete» CHL Gas Management, Purifier and
Warm Compressor» CHL 2.1K and 4.5K Cold Boxes» Medium and High Beta Cryomodules» HEBT Service Building IOC and PLC
2K Cold Box Turn-on was automated Cryomodules will complete in April
Cool DownFirst Liquid Full RF Power
Warm Up
tested
ICS – Software Engineering Group
11
The Cryogenic Control System layout
11 IOCs
17 PLCs
2 Servers
4 OPIs
ICS – Software Engineering Group
12
Ring Controls is ahead of schedule:BNL effort completed in March
Handover Plan - Complete and in use (cooling water and ramping software scope moved to ORNL)
FSDs - Complete, in use and being updated as needed to keep them as-built
Design - Essentially complete. All drawings and specs needed are in use for fabrication and installation and being updated as needed to keep them as-built
Hardware Fabrication - WBS 1.9 work complete except for installing 1 PLC for cooling water skid. WBS 1.5 work complete except for 2 motion control racks, 2 PLCs for RF, and Vacuum racks
Procurement - All hardware procured except 1 PLC for water skid and some miscellaneous parts
Cabling - Schedule for Ring and RTBT cables depends on budget allocations» HEBT Communications cables pulled, being terminated
» HEBT Process control cabling in process
Software - Vac, BLM, Mag PSC/PSI, HPRF, Beam Dump TCs ready for handover. All software ready for field testing by Jun 05
Test Plans - Drafted for all systems except RF - Will be updated as needed during installation
ICS – Software Engineering Group
13
PPS System Installation is keeping up(in spite of many configuration changes)
75% of the PPS controlled devices in the Linac have been connected
The temporary Linac shielding has been removed and the Linac has been returned to one segment for phase 1.3
PPS remote I/O racks for the HEBT are installed
Conduit & cable for PPS devices installed in HEBT tunnel
Installation of conduit & cable for PPS devices in Ring tunnel is 100% complete
The PPS control room racks have been relocated to the CLO CCR as a part of phase 1.3 (SCL commissioning)
ICS – Software Engineering Group
14
The Target & Instrument PPS are alsothe responsibility of the Controls Team
Cable & conduit design for target building complete, installation underway using AIMSI crafts (beamline 2 and transfer service bay) and fixed price subcontractor (high bay)
Target PPS equipment design is 85% complete
Target PPS equipment fabrication is 70% complete
Equipment installation ramping up as electricians become available
Instrument PPS design for beamline 2 underway. Prototype user panel being fabricated
ICS – Software Engineering Group
15
223 IOCs currently on Accelerator Network:» 109 VME / VXI --- vxWorks-based » 2 Linux-based “Soft IOC” Servers
– Running 24 Soft IOCs (18 + 6)
» 90 Intel-Pentium --- Win32-based IOCs (the “NADS”)– Wirescanners, Faraday Cups, BPMs, BCMs
Upgraded control system to EPICS R3.14.7» Improved reliability in channel access.
– Results in fewer white screens in the control room.
Upgraded OS for VME/VXI-based IOCs» vxWorks 5.5.1 (SNS CVS Release SNS07)
– Better network driver and network stack configuration– Auto-retry on boot-up if boot server is unavailable.– Better memory management
The number of IOCs has risen dramatically
ICS – Software Engineering Group
16
Power Supply Group is Routinely UsingAutomated Testing and Calibration
Method uses EPICS and so assures “end-to-end” testing including control system
Resulted in large schedule reduction – one test instead of two
PS Calibration, current/field mapping and cycling profiles come from Oracle RDB
9 SCL IOCs installed, 8 operational
1 HEBT IOC installed and operational
ICS – Software Engineering Group
17
Turn-on and Turn-off of HPRF Systemshas been automated
State Diagram monitors the processfor each HVCM
Overall summary is displayed
ICS – Software Engineering Group
18
The New Archive Engine is stable and amany-featured Java Archive Viewer is now in use
Archives now well organized with central configuration file and scripts for» generation of directory structure and skeleton files» start/stop/restart/monitoring of sampling engines» update of status web page» email notification of engine status and disk usage
Currently 11 distinct archives each running one or more sampling engines (total: 25).
» (Conventional Facilities, Magnets, RF, Cryo, ...) Data rate kept to ~2GB/day with occasional peaks at 4GB/day. Limiting performance factor is currently neither CPU nor network but disk I/O.
New Java Viewer is stable and many features are being exploited » Uses XAL Framework» Web interface available» Plots actual data in various formats» Offers absolute and relative request times» Performs computations, exports data to spreadsheets» Investigating new uses via command line and web» Scroll, zoom, legends, titles, multiple plot styles, arithmetic and boolean operations
ICS – Software Engineering Group
19
The new Archive Viewer (Chevtsov)has many useful features
•Relative start/end times "-5 d(ays)" →
"now"
•Multiple value axes
•Tool-tip info for individual data points
•XAL Framework
Support for calculations based on archived data
Export to spreadsheets, progress bar
Green squares in plot launch tool-tips for meta data, in this case daily restart of sampling engine.
ICS – Software Engineering Group
20
Database Collaboration with APS has grown(SNS, APS, SLAC, TRIUMF, BESSY, FNAL, Diamond, PSI)
• IRMIS: Integrated Relational Model for Installed Systems (v1.0 release in April)
• Agreed “Core Schema” describes control system with three hierarchies:
• Signal• Housing• Power
• Uses the SNS Physics XAL “Framework”• Pelaia et al
• RDB synchronized using “crawlers”
• 106 SNS IOCs successfully “crawled”• EPICS versions 3.13.9, 3.14.6, 3.14.7• 228,270 PVs found
– SCL 112141– DTL 33607 – CCL 27443 – MEBT 10918 – HEBT 8074
• “Crawlers” in development for all EPICS clients, IOCs, edm, Archiver, Alarms, etc.
ICS – Software Engineering Group
21
Issues
Increased installation pace.
RF Processing and installation checkout means Control Systems Group must provide operational support.
Hardware/Software Configuration Management is highly correlated with reliability/availability. Work is in progress using a RDB, CVS, and good release control scheme.
ICS – Software Engineering Group
22
Issues: How to manage increasing number of IOCS?
223 IOCs currently on Accelerator Network:» 109 VME / VXI --- vxWorks-based » 2 Linux-based “Soft IOC” Servers
– Running 24 Soft IOCs (18 + 6)
» 90 Intel-Pentium --- Win32-based IOCs (the “NADS”)– Wirescanners, Faraday Cups, BPMs, BCMs
Upgraded control system to EPICS R3.14.7» Improved reliability in channel access.
– Results in fewer white screens in the control room.
Upgraded OS for VME/VXI-based IOCs» vxWorks 5.5.1 (SNS CVS Release SNS07)
– Better network driver and network stack configuration– Auto-retry on boot-up if boot server is unavailable.– Better memory management
ICS – Software Engineering Group
23
Issues: How to manage increasing number of IOCS?
ICS – Software Engineering Group
24
Issues: How to manage increasing number of IOCS?
ICS – Software Engineering Group
25
Summary
Installation is keeping pace» Required use of more automated configuration and
testing, database use» PPS keeps pace notwithstanding changes» Cryo controls nearing completion» Will be ready for SCL commissioning» Ready to pick up on Ring controls
Operational Improvements as well» Automated HPRF turn-on» Automated 2K Cold Box turn-on» Improved Archive viewing tools