Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
High Performance Computing for fast tracking trigger at LHC/CMS
Naman Priyadarshi
Fermi National Accelerator Laboratory
QuarkNet
Challenges at High Luminosity Large Hadron Collider (HL-LHC)
1032 cm-2s-1 1033 cm-2s-1
1034 cm-2s-1 1035 cm-2s-1
Raw information from silicon detector/tracker reaches ~ Pb/s ! at Phase II luminosity 7/23/2013 2 Naman Priyadarshi
Will need to reconstruct
charged particle trajectories
“on-the-fly” for every beam
crossing (25 ns, or 40 Million
beam crossings per second),
from an ocean of input data
at up to ~ 100Tb/s.
This requires extremely fast
high bandwidth data
communication as well as
massive pattern recognition
power, with billion of known
patterns to be compared
against the multiple input
data streams simultaneously
with near zero latency.
and Fermilab has been
developing this special
system…
7/23/2013 3 From "Trigger Challenges at High Luminosity LHC" by Ted Liu
High Performance Computing from “Report to the President and Congress” by President’s
Council of Advisors on Science and Technology, Dec. 2010 (page 65)
• Compute-intensive
– massively parallel computation involving very large number of processing elements;
• Communication-intensive
– high-speed transfer of data among processing elements;
• Data-intensive
– high-speed manipulation of very large quantities of data
HL-LHC L1 Tracking Trigger is High Performance Computing (Non-von Neumann approach) Low Latency and Real Time
7/23/2013 4 Naman Priyadarshi
7/23/2013 6 Naman Priyadarshi
H U B
H U B I
N A M 1 2 3 4
Dividing Detector region into smaller sectors in which each crate will correspond to one of the sectors.
Two high schoolers having fun
7/23/2013 7 Naman Priyadarshi
Testing Prototype boards
7/23/2013 8 Naman Priyadarshi
7/23/2013 Naman Priyadarshi 9
f r ont panel
DPM2
RCE5
RCE6DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
f ron
t pan
el
DPM2
RCE 5
RCE 6
DPM2
RCE 3
RCE 4
DPM3
RCE 7
RCE 8
DPM 1
RCE 1
RCE 2 DTM
RTM
f ron
t pan
el
DPM
2RC
E 5
RCE 6
DPM
2RC
E 3
RCE 4
DPM
3RC
E 7
RCE 8
DPM
1RC
E 1
RCE 2
DTM
RTM
f ront panel
DPM2
RCE5 RCE
6
DPM2
RCE3 RCE
4
DPM3
RCE7 RCE
8
DPM1
RCE1 RCE
2
DTM
RTM
f ront panel
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
f ront panel
DPM
2
RCE5
RCE6
DPM
2
RCE3
RCE4
DPM
3
RCE7
RCE8
DPM
1
RCE1
RCE2
DTM
RTM
front panel
DPM
2
RCE5
RCE6
DPM
2
RCE3
RCE4
DPM
3
RCE7
RCE8
DPM
1
RCE1
RCE2
DT
M
RT
M
front panel
DPM2
RCE5
RCE6 DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
front panel
DPM
2
RCE
5RC
E6
DP M
2
RCE
3RC
E4
DPM
3
RCE
7RC
E8
DPM
1
RCE
1RC
E2
DT
M
RT
M
fron
t pan
el
DPM
2
RCE
5
RCE
6
DPM
2
RCE
3
RCE
4
DPM
3
RCE
7
RCE
8
DPM
1
RCE
1
RCE
2
DTM
RTM
fron
t pan
el
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
front panel
DPM2
RCE5RCE6
DPM2
RCE3RCE4
DPM3
RCE7RCE8
DPM1
RCE1RCE2
DTM
RTM
front panel
DPM
2RCE
5
RCE6
DPM
2RCE
3
RCE4
DPM
3RCE
7
RCE8
DPM
1RCE
1
RCE2
DTM
RTM
front panel
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2DTM
RTM
1-40 GEx 28 40 GE
Rear Transition Module (RTM)
Full Mesh Backplane
7/23/2013 10 Naman Priyadarshi
7/23/2013 11 Naman Priyadarshi
Prototype Board in Crate
Board has been inserted and powered on
7/23/2013 12 Naman Priyadarshi
Test for local bus
Communication between top and bottom FPGAs
7/23/2013 13 Naman Priyadarshi
7/23/2013 14 Naman Priyadarshi
7/23/2013 Naman Priyadarshi 15
2D Eye Scan V
olt
age
(mV
)
Sound (dB)
7/23/2013 16 Naman Priyadarshi
Rear Transition Module (RTM)
7/23/2013 17 Naman Priyadarshi
Systematic Problem Solving
7/23/2013 Naman Priyadarshi 18
1 2 3 4 5 6 7 8 9 10
Slots
2 1
FAILED LINK
Candidates for causing problem: •Pins in ATCA Crate (Slot 1 and 8) •Problems in Board 2 and 1 in the lines required for communication
Boards
Changing Slot 8 to Slot 7
7/23/2013 Naman Priyadarshi 19
1 2 3 4 5 6 7 8 9 10
Slots
2 1
FAILED LINK
Changing board 1 to 7
7/23/2013 Naman Priyadarshi 20
1 2 3 4 5 6 7 8 9 10
Slots
2 7
FAILED LINK
Changing Slot 1 to 2
7/23/2013 Naman Priyadarshi 21
1 2 3 4 5 6 7 8 9 10
Slots
2 7
FAILED LINK
Hypothesis: Damaged or Missing pins exist in Board 2 in channel 8
Testing Completed
• The 8 boards have been checked for
– Local bus (communication between FPGAs) successful for all boards
– RTM (6 of 8 boards – no errors, 2 boards had problematic channels)
– ATCA Backplane (3 board tested, 1 problematic channel discovered)
7/23/2013 22 Naman Priyadarshi
Crate/board level
H U B
H U B I
N A M 1 2 3 4
Input board x 8
AM board x 4
From silicon detector
7/23/2013 23 Naman Priyadarshi
f r ont panel
DPM2
RCE5
RCE6DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
f ron
t pan
el
DPM2
RCE 5
RCE 6
DPM2
RCE 3
RCE 4
DPM3
RCE 7
RCE 8
DPM 1
RCE 1
RCE 2 DTM
RTM
f ron
t pan
el
DPM
2RC
E 5
RCE 6
DPM
2RC
E 3
RCE 4
DPM
3RC
E 7
RCE 8
DPM
1RC
E 1
RCE 2
DTM
RTM
f ront panel
DPM2
RCE5 RCE
6
DPM2
RCE3 RCE
4
DPM3
RCE7 RCE
8
DPM1
RCE1 RCE
2
DTM
RTM
f ront panel
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
f ront panel
DPM
2
RCE5
RCE6
DPM
2
RCE3
RCE4
DPM
3
RCE7
RCE8
DPM
1
RCE1
RCE2
DTM
RTM
front panel
DPM
2
RCE5
RCE6
DPM
2
RCE3
RCE4
DPM
3
RCE7
RCE8
DPM
1
RCE1
RCE2
DT
M
RT
M
front panel
DPM2
RCE5
RCE6 DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
front panel
DPM
2
RCE
5RC
E6
DP M
2
RCE
3RC
E4
DPM
3
RCE
7RC
E8
DPM
1
RCE
1RC
E2
DT
M
RT
M
fron
t pan
el
DPM
2
RCE
5
RCE
6
DPM
2
RCE
3
RCE
4
DPM
3
RCE
7
RCE
8
DPM
1
RCE
1
RCE
2
DTM
RTM
fron
t pan
el
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
front panel
DPM2
RCE5RCE6
DPM2
RCE3RCE4
DPM3
RCE7RCE8
DPM1
RCE1RCE2
DTM
RTM
front panel
DPM
2RCE
5
RCE6
DPM
2RCE
3
RCE4
DPM
3RCE
7
RCE8
DPM
1RCE
1
RCE2
DTM
RTM
front panel
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2DTM
RTM
1-40 GEx 28 40 GE
7/23/2013 24 Naman Priyadarshi
f r ont panel
DPM2
RCE5
RCE6DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
f ron
t pan
el
DPM2
RCE 5
RCE 6
DPM2
RCE 3
RCE 4
DPM3
RCE 7
RCE 8
DPM 1
RCE 1
RCE 2 DTM
RTM
f ron
t pan
el
DPM
2RC
E 5
RCE 6
DPM
2RC
E 3
RCE 4
DPM
3RC
E 7
RCE 8
DPM
1RC
E 1
RCE 2
DTM
RTM
f ront panel
DPM2
RCE5 RCE
6
DPM2
RCE3 RCE
4
DPM3
RCE7 RCE
8
DPM1
RCE1 RCE
2
DTM
RTM
f ront panel
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
f ront panel
DPM
2
RCE5
RCE6
DPM
2
RCE3
RCE4
DPM
3
RCE7
RCE8
DPM
1
RCE1
RCE2
DTM
RTM
front panel
DPM
2
RCE5
RCE6
DPM
2
RCE3
RCE4
DPM
3
RCE7
RCE8
DPM
1
RCE1
RCE2
DT
M
RT
M
front panel
DPM2
RCE5
RCE6 DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
front panel
DPM
2
RCE
5RC
E6
DP M
2
RCE
3RC
E4
DPM
3
RCE
7RC
E8
DPM
1
RCE
1RC
E2
DT
M
RT
M
fron
t pan
el
DPM
2
RCE
5
RCE
6
DPM
2
RCE
3
RCE
4
DPM
3
RCE
7
RCE
8
DPM
1
RCE
1
RCE
2
DTM
RTM
fron
t pan
el
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
front panel
DPM2
RCE5RCE6
DPM2
RCE3RCE4
DPM3
RCE7RCE8
DPM1
RCE1RCE2
DTM
RTM
front panel
DPM
2RCE
5
RCE6
DPM
2RCE
3
RCE4
DPM
3RCE
7
RCE8
DPM
1RCE
1
RCE2
DTM
RTM
front panel
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2DTM
RTM
1-40 GEx 28 40 GE
7/23/2013 25 Naman Priyadarshi
f r ont panel
DPM2
RCE5
RCE6DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
f ron
t pan
el
DPM2
RCE 5
RCE 6
DPM2
RCE 3
RCE 4
DPM3
RCE 7
RCE 8
DPM 1
RCE 1
RCE 2 DTM
RTM
f ron
t pan
el
DPM
2RC
E 5
RCE 6
DPM
2RC
E 3
RCE 4
DPM
3RC
E 7
RCE 8
DPM
1RC
E 1
RCE 2
DTM
RTM
f ront panel
DPM2
RCE5 RCE
6
DPM2
RCE3 RCE
4
DPM3
RCE7 RCE
8
DPM1
RCE1 RCE
2
DTM
RTM
f ront panel
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
f ront panel
DPM
2
RCE5
RCE6
DPM
2
RCE3
RCE4
DPM
3
RCE7
RCE8
DPM
1
RCE1
RCE2
DTM
RTM
front panel
DPM
2
RCE5
RCE6
DPM
2
RCE3
RCE4
DPM
3
RCE7
RCE8
DPM
1
RCE1
RCE2
DT
M
RT
M
front panel
DPM2
RCE5
RCE6 DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
front panel
DPM
2
RCE
5RC
E6
DP M
2
RCE
3RC
E4
DPM
3
RCE
7RC
E8
DPM
1
RCE
1RC
E2
DT
M
RT
M
fron
t pan
el
DPM
2
RCE
5
RCE
6
DPM
2
RCE
3
RCE
4
DPM
3
RCE
7
RCE
8
DPM
1
RCE
1
RCE
2
DTM
RTM
fron
t pan
el
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2
DTM
RTM
front panel
DPM2
RCE5RCE6
DPM2
RCE3RCE4
DPM3
RCE7RCE8
DPM1
RCE1RCE2
DTM
RTM
front panel
DPM
2RCE
5
RCE6
DPM
2RCE
3
RCE4
DPM
3RCE
7
RCE8
DPM
1RCE
1
RCE2
DTM
RTM
front panel
DPM2
RCE5
RCE6
DPM2
RCE3
RCE4
DPM3
RCE7
RCE8
DPM1
RCE1
RCE2DTM
RTM
1-40 GEx 28 40 GE
7/23/2013 26 Naman Priyadarshi
Input board Close-up
7/23/2013 27 Naman Priyadarshi
AM board Close-up
7/23/2013 28 Naman Priyadarshi
“A New Concept of Vertically Integrated Pattern Recognition Associative Memory” TIPP 2011 Proceedings
http://www.sciencedirect.com/science/article/pii/S1875389212019165
Pattern recognition for tracking
is naturally a task in 3D
track
fired road
7/23/2013 29 Naman Priyadarshi
High Performance Computing from “Report to the President and Congress” by President’s
Council of Advisors on Science and Technology, Dec. 2010 (page 65)
• Compute-intensive
– massively parallel computation involving very large number of processing elements;
• Communication-intensive
– high-speed transfer of data among processing elements;
• Data-intensive
– high-speed manipulation of very large quantities of data
HL-LHC L1 Tracking Trigger is High Performance Computing (Non-von Neumann approach) Low Latency and Real Time
7/23/2013 31 Naman Priyadarshi
Level 1 tracking Trigger will be necessary for future LHC
experiments during higher luminosity
Goal of the new system is to go far beyond current high
performance computing with near zero latency
Opportunities to use 3D IC in future
Smart-phones
High-performance Digital Video
Wireless Connectivity
7/23/2013 Naman Priyadarshi 32
A newfound understanding and
appreciation of hardware by
understanding this system
Opportunity to work with scientists on
cutting edge parallel processing
technology
Chance to learn and practice a
systematic way to solving problems
7/23/2013 Naman Priyadarshi 33