High Performance Computing for fast tracking trigger at ... · Naman Priyadarshi Fermi National...

Preview:

Citation preview

High Performance Computing for fast tracking trigger at LHC/CMS

Naman Priyadarshi

Fermi National Accelerator Laboratory

QuarkNet

Challenges at High Luminosity Large Hadron Collider (HL-LHC)

1032 cm-2s-1 1033 cm-2s-1

1034 cm-2s-1 1035 cm-2s-1

Raw information from silicon detector/tracker reaches ~ Pb/s ! at Phase II luminosity 7/23/2013 2 Naman Priyadarshi

Will need to reconstruct

charged particle trajectories

“on-the-fly” for every beam

crossing (25 ns, or 40 Million

beam crossings per second),

from an ocean of input data

at up to ~ 100Tb/s.

This requires extremely fast

high bandwidth data

communication as well as

massive pattern recognition

power, with billion of known

patterns to be compared

against the multiple input

data streams simultaneously

with near zero latency.

and Fermilab has been

developing this special

system…

7/23/2013 3 From "Trigger Challenges at High Luminosity LHC" by Ted Liu

High Performance Computing from “Report to the President and Congress” by President’s

Council of Advisors on Science and Technology, Dec. 2010 (page 65)

• Compute-intensive

– massively parallel computation involving very large number of processing elements;

• Communication-intensive

– high-speed transfer of data among processing elements;

• Data-intensive

– high-speed manipulation of very large quantities of data

HL-LHC L1 Tracking Trigger is High Performance Computing (Non-von Neumann approach) Low Latency and Real Time

7/23/2013 4 Naman Priyadarshi

7/23/2013 6 Naman Priyadarshi

H U B

H U B I

N A M 1 2 3 4

Dividing Detector region into smaller sectors in which each crate will correspond to one of the sectors.

Two high schoolers having fun

7/23/2013 7 Naman Priyadarshi

Testing Prototype boards

7/23/2013 8 Naman Priyadarshi

7/23/2013 Naman Priyadarshi 9

f r ont panel

DPM2

RCE5

RCE6DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

f ron

t pan

el

DPM2

RCE 5

RCE 6

DPM2

RCE 3

RCE 4

DPM3

RCE 7

RCE 8

DPM 1

RCE 1

RCE 2 DTM

RTM

f ron

t pan

el

DPM

2RC

E 5

RCE 6

DPM

2RC

E 3

RCE 4

DPM

3RC

E 7

RCE 8

DPM

1RC

E 1

RCE 2

DTM

RTM

f ront panel

DPM2

RCE5 RCE

6

DPM2

RCE3 RCE

4

DPM3

RCE7 RCE

8

DPM1

RCE1 RCE

2

DTM

RTM

f ront panel

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

f ront panel

DPM

2

RCE5

RCE6

DPM

2

RCE3

RCE4

DPM

3

RCE7

RCE8

DPM

1

RCE1

RCE2

DTM

RTM

front panel

DPM

2

RCE5

RCE6

DPM

2

RCE3

RCE4

DPM

3

RCE7

RCE8

DPM

1

RCE1

RCE2

DT

M

RT

M

front panel

DPM2

RCE5

RCE6 DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

front panel

DPM

2

RCE

5RC

E6

DP M

2

RCE

3RC

E4

DPM

3

RCE

7RC

E8

DPM

1

RCE

1RC

E2

DT

M

RT

M

fron

t pan

el

DPM

2

RCE

5

RCE

6

DPM

2

RCE

3

RCE

4

DPM

3

RCE

7

RCE

8

DPM

1

RCE

1

RCE

2

DTM

RTM

fron

t pan

el

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

front panel

DPM2

RCE5RCE6

DPM2

RCE3RCE4

DPM3

RCE7RCE8

DPM1

RCE1RCE2

DTM

RTM

front panel

DPM

2RCE

5

RCE6

DPM

2RCE

3

RCE4

DPM

3RCE

7

RCE8

DPM

1RCE

1

RCE2

DTM

RTM

front panel

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2DTM

RTM

1-40 GEx 28 40 GE

Rear Transition Module (RTM)

Full Mesh Backplane

7/23/2013 10 Naman Priyadarshi

7/23/2013 11 Naman Priyadarshi

Prototype Board in Crate

Board has been inserted and powered on

7/23/2013 12 Naman Priyadarshi

Test for local bus

Communication between top and bottom FPGAs

7/23/2013 13 Naman Priyadarshi

7/23/2013 14 Naman Priyadarshi

7/23/2013 Naman Priyadarshi 15

2D Eye Scan V

olt

age

(mV

)

Sound (dB)

7/23/2013 16 Naman Priyadarshi

Rear Transition Module (RTM)

7/23/2013 17 Naman Priyadarshi

Systematic Problem Solving

7/23/2013 Naman Priyadarshi 18

1 2 3 4 5 6 7 8 9 10

Slots

2 1

FAILED LINK

Candidates for causing problem: •Pins in ATCA Crate (Slot 1 and 8) •Problems in Board 2 and 1 in the lines required for communication

Boards

Changing Slot 8 to Slot 7

7/23/2013 Naman Priyadarshi 19

1 2 3 4 5 6 7 8 9 10

Slots

2 1

FAILED LINK

Changing board 1 to 7

7/23/2013 Naman Priyadarshi 20

1 2 3 4 5 6 7 8 9 10

Slots

2 7

FAILED LINK

Changing Slot 1 to 2

7/23/2013 Naman Priyadarshi 21

1 2 3 4 5 6 7 8 9 10

Slots

2 7

FAILED LINK

Hypothesis: Damaged or Missing pins exist in Board 2 in channel 8

Testing Completed

• The 8 boards have been checked for

– Local bus (communication between FPGAs) successful for all boards

– RTM (6 of 8 boards – no errors, 2 boards had problematic channels)

– ATCA Backplane (3 board tested, 1 problematic channel discovered)

7/23/2013 22 Naman Priyadarshi

Crate/board level

H U B

H U B I

N A M 1 2 3 4

Input board x 8

AM board x 4

From silicon detector

7/23/2013 23 Naman Priyadarshi

f r ont panel

DPM2

RCE5

RCE6DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

f ron

t pan

el

DPM2

RCE 5

RCE 6

DPM2

RCE 3

RCE 4

DPM3

RCE 7

RCE 8

DPM 1

RCE 1

RCE 2 DTM

RTM

f ron

t pan

el

DPM

2RC

E 5

RCE 6

DPM

2RC

E 3

RCE 4

DPM

3RC

E 7

RCE 8

DPM

1RC

E 1

RCE 2

DTM

RTM

f ront panel

DPM2

RCE5 RCE

6

DPM2

RCE3 RCE

4

DPM3

RCE7 RCE

8

DPM1

RCE1 RCE

2

DTM

RTM

f ront panel

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

f ront panel

DPM

2

RCE5

RCE6

DPM

2

RCE3

RCE4

DPM

3

RCE7

RCE8

DPM

1

RCE1

RCE2

DTM

RTM

front panel

DPM

2

RCE5

RCE6

DPM

2

RCE3

RCE4

DPM

3

RCE7

RCE8

DPM

1

RCE1

RCE2

DT

M

RT

M

front panel

DPM2

RCE5

RCE6 DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

front panel

DPM

2

RCE

5RC

E6

DP M

2

RCE

3RC

E4

DPM

3

RCE

7RC

E8

DPM

1

RCE

1RC

E2

DT

M

RT

M

fron

t pan

el

DPM

2

RCE

5

RCE

6

DPM

2

RCE

3

RCE

4

DPM

3

RCE

7

RCE

8

DPM

1

RCE

1

RCE

2

DTM

RTM

fron

t pan

el

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

front panel

DPM2

RCE5RCE6

DPM2

RCE3RCE4

DPM3

RCE7RCE8

DPM1

RCE1RCE2

DTM

RTM

front panel

DPM

2RCE

5

RCE6

DPM

2RCE

3

RCE4

DPM

3RCE

7

RCE8

DPM

1RCE

1

RCE2

DTM

RTM

front panel

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2DTM

RTM

1-40 GEx 28 40 GE

7/23/2013 24 Naman Priyadarshi

f r ont panel

DPM2

RCE5

RCE6DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

f ron

t pan

el

DPM2

RCE 5

RCE 6

DPM2

RCE 3

RCE 4

DPM3

RCE 7

RCE 8

DPM 1

RCE 1

RCE 2 DTM

RTM

f ron

t pan

el

DPM

2RC

E 5

RCE 6

DPM

2RC

E 3

RCE 4

DPM

3RC

E 7

RCE 8

DPM

1RC

E 1

RCE 2

DTM

RTM

f ront panel

DPM2

RCE5 RCE

6

DPM2

RCE3 RCE

4

DPM3

RCE7 RCE

8

DPM1

RCE1 RCE

2

DTM

RTM

f ront panel

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

f ront panel

DPM

2

RCE5

RCE6

DPM

2

RCE3

RCE4

DPM

3

RCE7

RCE8

DPM

1

RCE1

RCE2

DTM

RTM

front panel

DPM

2

RCE5

RCE6

DPM

2

RCE3

RCE4

DPM

3

RCE7

RCE8

DPM

1

RCE1

RCE2

DT

M

RT

M

front panel

DPM2

RCE5

RCE6 DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

front panel

DPM

2

RCE

5RC

E6

DP M

2

RCE

3RC

E4

DPM

3

RCE

7RC

E8

DPM

1

RCE

1RC

E2

DT

M

RT

M

fron

t pan

el

DPM

2

RCE

5

RCE

6

DPM

2

RCE

3

RCE

4

DPM

3

RCE

7

RCE

8

DPM

1

RCE

1

RCE

2

DTM

RTM

fron

t pan

el

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

front panel

DPM2

RCE5RCE6

DPM2

RCE3RCE4

DPM3

RCE7RCE8

DPM1

RCE1RCE2

DTM

RTM

front panel

DPM

2RCE

5

RCE6

DPM

2RCE

3

RCE4

DPM

3RCE

7

RCE8

DPM

1RCE

1

RCE2

DTM

RTM

front panel

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2DTM

RTM

1-40 GEx 28 40 GE

7/23/2013 25 Naman Priyadarshi

f r ont panel

DPM2

RCE5

RCE6DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

f ron

t pan

el

DPM2

RCE 5

RCE 6

DPM2

RCE 3

RCE 4

DPM3

RCE 7

RCE 8

DPM 1

RCE 1

RCE 2 DTM

RTM

f ron

t pan

el

DPM

2RC

E 5

RCE 6

DPM

2RC

E 3

RCE 4

DPM

3RC

E 7

RCE 8

DPM

1RC

E 1

RCE 2

DTM

RTM

f ront panel

DPM2

RCE5 RCE

6

DPM2

RCE3 RCE

4

DPM3

RCE7 RCE

8

DPM1

RCE1 RCE

2

DTM

RTM

f ront panel

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

f ront panel

DPM

2

RCE5

RCE6

DPM

2

RCE3

RCE4

DPM

3

RCE7

RCE8

DPM

1

RCE1

RCE2

DTM

RTM

front panel

DPM

2

RCE5

RCE6

DPM

2

RCE3

RCE4

DPM

3

RCE7

RCE8

DPM

1

RCE1

RCE2

DT

M

RT

M

front panel

DPM2

RCE5

RCE6 DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

front panel

DPM

2

RCE

5RC

E6

DP M

2

RCE

3RC

E4

DPM

3

RCE

7RC

E8

DPM

1

RCE

1RC

E2

DT

M

RT

M

fron

t pan

el

DPM

2

RCE

5

RCE

6

DPM

2

RCE

3

RCE

4

DPM

3

RCE

7

RCE

8

DPM

1

RCE

1

RCE

2

DTM

RTM

fron

t pan

el

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2

DTM

RTM

front panel

DPM2

RCE5RCE6

DPM2

RCE3RCE4

DPM3

RCE7RCE8

DPM1

RCE1RCE2

DTM

RTM

front panel

DPM

2RCE

5

RCE6

DPM

2RCE

3

RCE4

DPM

3RCE

7

RCE8

DPM

1RCE

1

RCE2

DTM

RTM

front panel

DPM2

RCE5

RCE6

DPM2

RCE3

RCE4

DPM3

RCE7

RCE8

DPM1

RCE1

RCE2DTM

RTM

1-40 GEx 28 40 GE

7/23/2013 26 Naman Priyadarshi

Input board Close-up

7/23/2013 27 Naman Priyadarshi

AM board Close-up

7/23/2013 28 Naman Priyadarshi

“A New Concept of Vertically Integrated Pattern Recognition Associative Memory” TIPP 2011 Proceedings

http://www.sciencedirect.com/science/article/pii/S1875389212019165

Pattern recognition for tracking

is naturally a task in 3D

track

fired road

7/23/2013 29 Naman Priyadarshi

High Performance Computing from “Report to the President and Congress” by President’s

Council of Advisors on Science and Technology, Dec. 2010 (page 65)

• Compute-intensive

– massively parallel computation involving very large number of processing elements;

• Communication-intensive

– high-speed transfer of data among processing elements;

• Data-intensive

– high-speed manipulation of very large quantities of data

HL-LHC L1 Tracking Trigger is High Performance Computing (Non-von Neumann approach) Low Latency and Real Time

7/23/2013 31 Naman Priyadarshi

Level 1 tracking Trigger will be necessary for future LHC

experiments during higher luminosity

Goal of the new system is to go far beyond current high

performance computing with near zero latency

Opportunities to use 3D IC in future

Smart-phones

High-performance Digital Video

Wireless Connectivity

7/23/2013 Naman Priyadarshi 32

A newfound understanding and

appreciation of hardware by

understanding this system

Opportunity to work with scientists on

cutting edge parallel processing

technology

Chance to learn and practice a

systematic way to solving problems

7/23/2013 Naman Priyadarshi 33

Recommended