LHC CMS Detector Upgrade Project 401-04-04 Calorimeter Trigger Sridhara Dasu, University of Wisconsin DOE CD1 Review 26 August 2013 Sridhara Dasu, 26 August

LHC CMSDetectorUpgrade

Project

DOE CD1 Review - US CMS Upgrade - Trigger 1

401-04-04 Calorimeter Trigger

Sridhara Dasu, University of Wisconsin

DOE CD1 Review

26 August 2013

Sridhara Dasu, 26 August 2013


Project


Calorimeter Trigger WBS Detail


←MuonTrigger

M&S paid byCMS-France


Project

Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 3

Calorimeter Trigger ComponentsType Name Purpose

TCC Trigger Concentrator Card Existing ECAL trigger primitives (TP) (France)

oSLB Optical Serial Link Board Transmit ECAL TPs to RCT & CTP (Portugal / France)

mHTR HCAL Trigger & DAQ Board Create and transmit HCAL TPs

RCT Regional Calorimeter Trigger Existing trigger system

oRM Optical Receiver Modules Receive ECAL TPs on RCT(Portugal / France for Stage-1)

oRSC Optical Regional Summary Card Convert & transmit RCT output (US DOE NP for Stage-1)

CTP Compact Trigger Processor Large FPGA w/ optical & backplane

CIOX Compact IO cross-point switch Intra-crate sharing switch

MP7 Multipurpose Processor Large FPGA w/ only optical links (UK)


Project


Upgraded HCAL Readout & Trigger Electronics (μHTR) is after split that sends data to old electronics (HTR) μHTR supplies TP to upgrade trigger, HTR continues to send to present trigger

ECAL Trigger Concentrator Cards (TCC) are not upgraded, instead their mezzanines with trigger serial links are replaced (Lisbon): Optical Serial Link Board oSLB replaces single copper link to current Regional

Calorimeter Trigger (RCT – U. Wisc.) with two optical links (WBS 401.04.04.06) One optical link to optical Receiver Module (oRM) on RCT (WBS 401.04.04.05)

o Replaces copper Receiver Module One optical link to input of Upgraded Calorimeter Trigger: CTP7 Card

574 oSLB’s & oRM’s with Fibers installed in 2014 French Contribution to Calorimeter Trigger (no US Upgrade Funds)

Presently oSLB & oRM in final prototype stage oSLB:

oRM:


HCAL & ECAL Trigger Primitives

OSLB Output / oRM Input: 4x channels @ 1.2 Gbps 4.8 Gbps


Project


Main processor card Virtex-7 690T

for processing ZYNQ

for TCP/IP + linux 60 10G optical

Input links 36 10G optical

Output links Function: Find ET

clusters & transmit to Layer-2

• Card Count = 46• 36 total +

8 spares + 2 test setups


401.04.04.02: CTP Card Concept

Virtex-7 VX690T FPGA

ZYNQXC7Z03

0EPP

1.5V

S

up

ply

2.5V

S

up

ply

3.3V

S

up

ply

1V

30A

S

up

ply CXP Module

12Tx + 12 Rx

CXP Module12Tx + 12Rx

CXP Module12Tx + 12 Rx

12XRx

12XRx

(CTP-6 BG View)


Project


CTP: Virtex-6 Prototype Exists


Back End FPGAXC6VHX250T/XC6VHX380T

Front End FPGAXC6VHX250T/XC6VHX380T

Avago AFBR-810B Tx Module

4X Avago AFBR-820B Rx Module

MMC Circuitry

JTAG/USB ConsoleInterface Mezzanine

Power Modules

Dual SDRAM for dedicated DAQ and TCP/IP buffering

12x Multi Gig Backplane Connections


Project

Validation of signal integrity on CTP6


CTP-6 Intraboard link eyepattern


Project


12-links tested at 6.4 Gbps – no errors


HCAL TPG to CTP Integration Test


Project


Running Linux on CTP6 FPGA – PetaLinux

Startup screen via USB console

Root login, pinging controller PC at 192.168.1.2

Running on CTP-6 back end FPGA (`VHX250T)

Important step towards CTP-7 ZYNQ-based Linux



Project


Physicists can contribute to CTP6 software Familiar, reliable environment Faster development cycles Use of IPBus and uHAL over TCP/IP (CMS standards) Coupled with AMC13 (Boston), DAQ can be implemented more

easily No need

for customfirmware orkernel softwarefor monitoring and control

Postdocs arealready usingPetalinux


Advantage of Linux System on Chip


Project


CTP7 is simpler than CTP6 prototype Engineering effort for in-house design based on CTP6 work Procurement of PCBs is based on recent CTP6 purchase Good estimate of cost and schedule from prototype manufacture

Quotes available for all parts Virtex-7 690T is the major cost driver – with recent quote

o 3 690Ts donated by Xilinxo 5 690Ts purchased from Xilinxo Total of 8 FPGAs in-hand for prototypes

Optical modules based on recent purchase Miscellaneous parts also have quotes or estimated from

prototype purchases (power modules)


CTP: Basis of Estimate - Hardware


Project


Firmware and software efforts estimated from prototype testing + prior RCT experience Firmware algorithms for testing provide the basis for the core

firmware, which is the most complicatedo Tom Gorski and Mathias Blake

Trigger algorithm firmware work is based on prior experience managing GCT project (UK)o Wesley Smith was L1 project manager overseeing GCT

Software effort is based on current RCTo Sophistication of embedded Linux / TCP and shorter timeline for

implementation requires professional software effort– Jes Tikalsky

o Bulk of testing software from post-doc and graduate student– Pam Klabbers, Evan Friis and graduate students will develop software

Board and Firmware/Software done by the same group (Wisconsin) results in efficiencies


CTP: Basis of Estimate - Effort


Project


Function: For inter-board connections

Input/Output: 48x 4.8-6.4 Gbps

Prototype exists

Fully tested

Firmware is minimal and exists

CIOX cost well known

Ready for production


401.04.04.03: Crosspoint IO Card– U. Wisconsin

Controller (MMC and link mgmt)

4X Avago AFBR-79EQDZ QSFP+ ModulePositions

4x4

Lan

e B

idir

ecti

on

al M

ult

i G

ig

Bac

kpla

ne

Co

nn

ecti

on

s

Backplane Rx/Tx

Redriver ICs (top and

bottom sides)

(2/crate x 3 crates = 6 + spares)


Project

DOE CD1 Review - US CMS Upgrade - Trigger 14Sridhara Dasu, 26 August 2013

401.04.04.04: CTP Infrastructure:Vadatech VT894 Crate Test Setup

BU

AM

C13

Vad

atec

h

MC

H UW

CT

P-6

UW

CT

P-6

TTC Downlink

UW

Au

x

(Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6)

U. Wisconsin designed backplane with dense card interconnectsmanufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog


Project


The dominant risks are: Availability of ALL trigger primitive inputs from ECAL,

HCAL in optical format Validation of large optical link plant in limited time Fully validated trigger algorithms in firmware Control and operations software

Risk mitigation strategy: Continue to provide fully operational current trigger

system in parallel with upgrade commissioning Partial operation of the upgrade systems provide

tangible benefits from 2015 onwards


CTP: Risk and Mitigation Strategy


Project


Goal (Original): Provide readout of original RCT descoped

during construction project Present readout through GCT input buffer

not workable with trigger evolution Uses connection to a single CTP6 prototype

or CTP7 card for DAQ readout

optical Receiver Summary Card (oRSC) paid by DOE Nuclear

VME Slave Interface Card Fits in current RCT Crates (1 per crate) 18 Cards in 2015 System Receives RCT Jet Sum Card Output

to GCT on Copper “SCSI” Data Cables Provides direct optical input to GCT

o Bypassing old optical conversion cards Planned use for Heavy Ion Triggers Prototype under test

Multiple optical outputs provide: Inputs for upgrade calorimeter trigger &

parallel operation of old & new trigger wherever ECAL & HCAL electronics notavailable in 2015


Calorimeter Trigger Risk MitigationoRSC Function:

Conversion to opticaloRSC Input:

RCT egt and jet clusters

80 MHz Parallel ECLoRSC Output:

6x copies at 6.4 Gbps optical


Project


4 Boards manufactured and tested

12 layer board, with Isola iSpeed Laminate for high-speed SERDES

Kintex-7 355T-2 as the RCT parallel-to-optical-Tx interface device

Spartan-6 for VME Slave Interface and I2C/general device controller

Two Avago miniPOD 12-lane transmitters for optical outputs per card

One card per RCT crate, capturing the output from the Jet Summary Card, and receiving clock/control signals from the RCT Crate Clock Card

Card is functional: Tested oRSC to CTP6 optical rx/tx at 6.4 Gbps (23 links) oRSC integration test with MP7 (UK) planned for July 2 oRSCs can be used to fully emulate the 18-crate RCT output

o Will be used for developing RCT readout validation in 2013


oRSC Card Status


Project


oRSC-CTP6 Monitoring and Validation

Zero bit errors10^-14 BER!



Project


CTP6 Readout Validation

Two Virtex6 FPGAs on CTP6 Both have Microblaze embedded processor Enables straight forward development in C++

Read out captured transmitter data via IPBus protocol from remote server Back End FPGA embedded processor running Linux

o serves IPBus over TCP/IPo Forward IPBus packets over serial protocol to FPGA2o Able to readout arbitrary memory locations from both FPGAso Receive upgrade commands for writing FPGA configuration images and additional

Linux executables to onboard flash memory Front End FPGA embedded processor running standalone app

o Able to accept and decode IPBus Packetso Control or report status of all 48 fiber linkso Read capture RAMs for all or selected fibers



Project


CTP Readout Configuration


CTP6


Project

DOE CD1 Review - US CMS Upgrade - Trigger 21Sridhara Dasu, 26 August 2013

CTP Monitoring and Validation Software

• Software status and control• Real Time monitor link status• Real time fiber link receiver control• Read out from selected link Capture RAMs


Project


CTP6 / Linux Monitoring & Validation


• Six links connected from oRSC to CTP6 during integration test

• Displays real time fiber connection status for each of 48 CTP6 fiber inputs• Overflow, Underflow, Loss of Sync, Data Error Detect, and PLL Lock

• All six fiber links locked and error free


Project


CTP / Linux Monitoring & Validation


• Fiber Capture RAM readout

• SW controllable capture• Sync character to start capture• Length of Capture• Links to readout

• Capture RAM auto clears on each capture

• All six links capture expected pattern from oRSC• Captured data automatically checked by host


Project


oRSC to MP7 Link Validation

Bathtub Curve Eye Diagram

Extraordinary margin available in bathtub curve

Superb eye opening measured at MP7 receiver

Zero Bit errors with overnight iBERT PRBS testing

Data validation: oRSC pattern RAM to MP7 capture RAM



Project


mHTR to CTP6 integration test at Madison (done) Wisconsin/Minnesota : 12 links operated successfully

oRSC to CTP6 demonstrator at Madison (done) Test 2-3 oRSCs and interface before shipment to CERN

oRSC to CTP6 platform at Prevessin (done) 2 oRSCs emulate RCT + 1 CTP6 for RCT readout and algorithm tests Full platform for system operations and control software development

oRSC - MP7 Integration test at Prevessin (done) Wisconsin/Imperial July 2013 Milestone

CTP6 - oRSC-oRM/RCT - oSLB/TCC Integration Test at Prevessin Wisconsin/Lisbon/LLR October 2013 Milestone

CTP6 - oRSC-oRM/RCT - oSLB/TCC -uHTR Integration Test at 904 Wisconsin/Lisbon/LLR/Minnesota November 2013 Milestone

CTP7 - MP7 Integration test at Prevessin Wisconsin/Imperial March 2014 Milestone


Demonstrators & Critical Integration Tests


Project


2013 M&O + R&D Activity : oRSC construction, installation Firmware development for RCT readout and testing using CTP6 Lisbon / France : oSLB and oRM construction, installation Minnesota / Wisconsin : mHTR to CTP6 testing

2014 Ensure fully working legacy RCT system with oRSC + CTP6 RO CTP7, CIOX, … construction, validation Firmware development for CTP7 testing and 2015 operation

2015 Commission and operate Stage-1 system Firmware development and integration of Stage-2 system

2016 Commission and operate Stage-2 system


Adiabatic Upgrade Timeline


Project


CT Milestones


Activity ID Activity Name Target Completion

Milestone

T1020 L2 – Deliver Stage-1 Layer 1 to CERN 15-Nov-2014 15-May-2015

T1030 L2 – Deliver Stage-2 Layer 1 to CERN 10-Mar-2015 10-Sep-2015

T1040 L2 – Test operation at CERN 15-Aug-2015 15-Feb-2016

T1070 L2 – Test High-Luminosity Capability 28-Sep-2016 28-Mar-2017

T3310 L3 – Commission CTP7 for Initial Ops 15-Apr-2015 15-Jul-2015

T4000 L3 – Commission CIOX for Final Ops 04-Sep-2016 04-Dec-2016

T3320 L3 – Commission CTP7 for Final Ops 29-Sep-2016 29-Dec-2016

T4740 L4 – oSLB-oRM Optical Fibers Complete 15-Sep-2014

T4090 L4 – CIOX Software and Firmware Complete 10-Dec-2014

T4210 L4 – CIOX Production Complete 08-Jan-2015

T3710 L4 – CTP7 Production Complete 08-Jan-2015

T4510 L4 – CTP Infrastructure Production Complete 29-Jul-2015

T4300 L4 – CIOX Commissioning Complete 09-Sep-2015

T3980 L4 – CTP7 Test, Install and Commissioning Complete

30-Sep-2016

T3520 L4 – CTP7 Software and Firmware Complete 30-Sep-2016


Project


Backup Slides for Q&A



Project

DOE CD1 Review - US CMS Upgrade - Trigger

Evolution: Stage-1 Upgrade in 2015

Sridhara Dasu, 26 August 2013 29

Layer 2 Calo Trigger

HCALenergy

Regional Calo Trigger

Global Calo Trigger

HFenergy

oRSC

Layer 2Calo Trigger

oRM

ECALenergy

Layer 1 Processors

oSLB ECAL & HF ClustersHalf-tower position

4x4 E+H Clusters2x1 E+H Clusters

2-bit region ID

Layer 2 Processors

MuonJets & Sums

eGammaTau

Heavy IonSpare

RCT RO + Testing

GCT fallback remains

With just a fraction of final cards or even prototypes, derive many benefits

of full upgrade, incl. muon isolation

Uses reprogrammed RCT clusters with improved algorithms in Layer-2 Also, brings in all the power of finer grain HF using “Slice Test” of Layer-1 & 2

Improved PU subtraction, Isolation Calculation (e/γ/τ/μ) & Half-tower Position Resolution

US Items: oRSCs, oSLB/oRM commissioning CTP Layer 1 Processors RCT RO+Testing CTP

UK Items: MP7 Layer 2 Processors

36 mHTRs

2 CTP7s 9 CTP7s

Add to bring in finer grain EM clusters

2 CTP7s

4 MP7s


Project


Calorimeter Trigger Upgrade Options

Two modes of connectivity required

Layer 1: Pipelined: forms

various cal. clustersto send to L2 nodesfor different triggers

TMT: distributes allcal. Info. in eventto one multiplexedL2 node for all triggers

Layer 2: Finds different trigger

objects using clustersor scanning (TMT)all cal. info. & thendemultiplexing

Keep new trigger flexible to adapt to needs of evolving CMS physics program Both architectures have two processing layers

o Layer 1 optimized for backplane connectivity, Layer 2 for optical TMT architecture chosen as initial baseline

o Stage-1 must use pipelined traditional architecture

Fully Pipelined Calorimeter Trigger

Time Multiplexed Calorimeter Trigger

US: Layer 1:CTP7 Cards

UK: Layer 2: MP7 Cards

Demux

Layer 1

Layer 2



Project


Crate GUp to 12TriggerPath

Modules

Trig Path 1(MP7)

Fiber patch panel

Crate C

() () CT

P

() () ()() () () CT

P

() () ()

Trig Path 2(MP7)

Trig Path 3(MP7)

Trig Path 4(MP7)

Trig Path 5(MP7)

Trig Path 6(MP7)

ECAL

Each 1st-Level CTP7 Drives 2 fibers out to 6-12 Trigger Paths @ 9.6 Gbps

Each Trigger Path MP7 rcvs 48 total 1st-Level fibers @ 9.6 Gbps (2/region)

To GT To GT

()()() () ()() ()

Crate A

Crate G

Pipelined Calorimeter Trigger (Stage-1)

Calorimeter Trigger Processor (CTP) Card


Each Layer 2 processor calculates suite of trigger paths

Layer 1 and RCT send dedicated clusters to Layer 2

(Default plan till all ECAL & HCAL TPs are available in optical format.)

HBHE

Current RCT

HF HF

oRS

C


Project



Modules

Trig Path 1(MP7)

Fiber patch panel

Crate B Crate C

() CT

P

() CT

P

() CT

P

CT

P

() CT

P

() CT

P

()CT

P

() CT

P

() CT

P

() CIO

-U

CIO

-L

6 x 8-fiber ribbons for Lateral Network(@ 4.8 Gbps)

Trig Path 2(MP7)

Trig Path 3(MP7)

Trig Path 4(MP7)

Trig Path 5(MP7)

Trig Path 6(MP7)

ECALECAL ECAL



To GT To GT

()CT

P

CT

P

()CIO

-L

CT

P

() CIO

-U

CIO

-L

CIO

-U

Crate A

Crate G

Pipelined Calorimeter Trigger (Stage-1)

Calorimeter Trigger Processor (CTP) CardCrosspoint I/O (CIO) Card



Layer 1 sends dedicated clusters to Layer 2


HBHE

Current RCT

HF HFHF


Project



Modules

Trig Path 1(MP7)

2x288-fiber patch panel (2x24×12-fiber optical ribbons in & out)

Crate B Crate C

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CT

P

CIO

-U

CIO

-L

6 x 8-fiber ribbons for Lateral Network(@ 4.8 Gbps)

Trig Path 2(MP7)

Trig Path 3(MP7)

Trig Path 4(MP7)

Trig Path 5(MP7)

Trig Path 6(MP7)

HCALECALHCALECAL HCALECAL



To GT To GT

CT

P

CT

P

CT

P

CT

P

CIO

-L

CT

P

CT

P

CIO

-U

CIO

-L

CIO

-U

Crate A

Crate G

Pipelined Calorimeter Trigger Upgrade

Calorimeter Trigger Processor (CTP) CardCrosspoint I/O (CIO) Card



Layer 1 sends dedicated clusters to Layer 2



ProjectTMT Calorimeter Trigger Upgrade


Time-multiplexed architecture, calo data from an entire event processed by a single processor


Projecte / g / t Position Resolution


Significantly improved position resolution Further enhanced with optional Stage-1


Project


Electron / Photon Trigger

Improved control on isolation even at Stage-1 Factor of 2-3 reduction in rate with small loss in efficiency



ProjectTau Trigger

Big improvement in efficiency with ~10X rate reduction!


Current tau trigger has large and negative PU dependence


ProjectJet Triggers


Single jet

Single jet thresholds similar to current Multi-jet trigger thresholds better in upgrade (PU sub.) Small improvement in efficiency turn-on

Quad jetHT = S PTJets


Project


Process(x2 improvement highlighted)

1.1 x 1034 cm–2 s–1 2.2 x 1034 cm–2 s–1

Current Upgrade Current Upgrade

W(en),H(bb) 57.7% 87.0% 37.5% 71.5%

W(mn),H(bb) 95.9% 100% 69.6% 97.9%

VBF H( ( )tt mt ) 42.6% 51.3% 19.4% 48.4%

VBF H( ( )tt et ) 24.4% 44.3% 14.0% 39.0%

VBF H( ( )tt tt ) 17.2% 53.7% 14.9% 50.1%

H(WW(eenn)) 91.4% 97.8% 74.2% 95.3%

H(WW(mmnn)) 99.9% 99.9% 89.3% 99.9%

H(WW(emnn)) 97.6% 99.4% 86.9% 99.3%

H(WW(menn)) 99.6% 99.5% 90.7% 99.7%

StopbWce, jets (600 – 450 GeV) 55.8% 68.2% 50.3% 64.8%

StopbWc ,m jets (600 – 450 GeV) 78.1% 81.6% 76.4% 84.5%

RPV Stopjets (200 GeV) 70.1% 99.9% 43.6% 99.9%

RPV Stopjets (300 GeV) 93.7% 99.9% 79.7% 99.9%

Physics Performance Summary

Ave

rage

Im

prov

emen

t: 1

7% (

Low

Lum

i) &

40%

(H

igh

Lum

i)



Project


Shown examples of improved object performance Taken in isolation these do not show we can deliver the CMS physics programme: how do triggers

fit together? what about overlaps? Can we fit everything in 100 kHz limit?

Develop simplified L1 trigger menus to illustrate thresholds attainable within an overall fixed rate 1.1x1034 cm-2 s-1 with 50ns BX and 50 pile-up 2.2x1034 cm-2 s-1 with 25ns BX and 50 pile-up

Menus contain: Single lepton triggers Isolated single lepton triggers Dilepton triggers Lepton cross-triggers (lepton and jets or MET) Hadronic triggers

Captures about 80% of the rate in 2012 L1 trigger menu Does not include calibration triggers, prescaled trigger for efficiencies etc.


Expected performance


Project


L=1.1x1034 cm-2s-1 with 50ns BX and 50 pile-up


⎬Single e/μ

Multijet


Project


L=2.2x1034 cm-2s-1 with 25ns BX and 50 pile-up


⎬Single e/μ

Multijet


Project


Physics priorities: Measure all Higgs BR as precisely as possible to confirm Standard

Model or not, so retaining or improving current trigger capability is critical

Want to be able to answer the question of naturalness - whether or not there is new physics stabilising the Higgs mass or noto SUSY remains a leading candidate, but if it is so, must have light stopso Also must be able to trigger on and search for all variants (e.g. RPV with

all hadronic final states) to draw a firm conclusion

Consider a set of benchmark physics channels Look at the performance of these channels in 2012 analyses with and

without L1 trigger upgrade at different luminosities using toy menus Benchmark the signal efficiencies in each case


Physics studies


Project


Benchmark channels considered to highlight improvements

Higgs WH: H bb (➔ single lepton triggers) H ττ (➔ new tau algorithm, single lepton triggers) H WW (➔ single and dilepton triggers)

SUSY Direct stop squark production (single lepton, jets and MET triggers) RPV SUSY decays (hadronic triggers e.g. multijet)

Heavy Ion pT asymmetry in b jets (underlying event subtraction in jet trigger) Ratio of 3 to 2 jets (underlying event subtraction in jet trigger)


Physics performance


Project


Higgs summary



Project


SUSY summary



Project


Higgs summary


Single e/μ ⎬⎬Tau ID &

Single e/μ


Project


SUSY summary


Multijet


Project


Heavy Ion performance


Readout rate is limited by data volume to 3 kHz Need reduction to 5% of current jet trigger

rate to reduce bandwidth Pile-up subtraction in upgraded trigger

achieves goal

Investigation of flavour dependence in jet quenching jet asymmetry in ➔doubly b-tagged events

Documents

LHC CMS Detector Upgrade Project 401-04-04 Calorimeter Trigger Sridhara Dasu, University of Wisconsin DOE CD1 Review 26 August 2013 Sridhara Dasu, 26 August