BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois

BTeV Availability Issues

We don’t have problems any more,we have issues...

21 Aug 2001

M. Haney, University of Illinois

[email protected] 21aug01

2

Early on, high energy physicists found too many particles...

’

a0

f0

a1

h0

h1

f’2

f1

a2

f2

fj

a4

f4

K*

K

K1

K2

K*2

K3

K*3

K*4

Ds

D1

D*2

D*

D

D2B*

B

Bs

Bc

c

c

“the secret is to bang the rocks together…” Douglas Adams


3

which led to the Standard Model:

quarks & leptons

quark charge

up

down

charm

strange

top

bottom

+ 2/3

- 1/3

-1

0

e

e

lepton charge

(truth)

(beauty)


4

quarks & anti-quarks

proton neutron

+ D0 Bs

u

d

c

s

t

b

+ 2/3

- 1/3

- 2/3

+ 1/3

u

d

c

s

t

b

Now we can easily build any ofof the particles we have discovered(and predict some we haven't)

(baryons)

(mesons)

anti-proton

Rule: Quarks cannot exist alone,but only in 2’s and 3’s


5

How to build a Particle Accelerator

Protons are also quite easy: Ionize hydrogen

+

e +

e

+

e

+

e

+e+

e

+

e

+

e

ENERGIZER

-+

Start with some particles (usually electrons or protons)

Electrons are easy: Just heat up a filament ee

eeeee eeeeeee ee

ENERGIZER

+-


6

• if you match the frequency correctly,

• the particle will “surf” the RF wave...

Linear Accelerator (concept)

+

-+

+

-

+

-


7

Linear Accelerator (practice)

• LINAC at Fermilab(0.4 GeV)


8

Vacuum tube (beam pipe)

Synchrotrons

bending magnet

Accelerating section

Focussingmagnets


9

Fermi National Accelerator Lab

• The ring is about 4 miles around...


10

Two for the price of one

1000 GeV-1000 GeV +

2000 GeV

• accelerate protons andantiprotons– same ring– opposite

charge,+ direction

• bang themtogether...


11

4 Detector• The typical arrangement

is to wrap the detectoraround the interactionpoint:

CDF at Fermivertex(later…)


12

BTeV is rather different...

protons antiprotons


13

Too much physics!

• Let’s talk about computational requirements:– every 132ns, two “clouds of quarks”

cross paths...– on the average, there will be 2 “events”

(collisions)– 200 KBytes of data are produced per event


14

BTeV Data Path

About 1 TByte of (distributed) buffer memory...


15

Why have a Trigger?• Compression is not enough

– To get from 1.5 TBytes/sec from the detector,– to 200 MBytes/sec to tape, you need to:

• identify/reject “boring” events• identify/accept/save-to-tape “interesting” events• keep extremely good records on every action!

– All of the analysis is statistical– Bad statistics = Mission failure

– Losing an event is survivable• Not knowing that you lost it, or why, is death


16

The BTeV Pixel Trigger• (there is also a Muon Trigger;

there may be others...)– 2 KBytes/crossing (from Pixel Detector)

• every 132ns...– Fast, pipelined FPGA(s) to sort,

match up pairs and triplets between planes– DSP(s) to find tracks, compute vertex(s)

• time available– <1 ms, from crossing to trigger “opinion”


17

Muon DSP Farm

L1Buf (basis)

L1Buf (basis)

Muon Preprocessor

Segment Preprocessor L1Buf (cooked)

MTSM

Detector

Front End Board (DCB)

Crossing Switch

Pixel Preprocessor(FPGAs)

Segment Preprocessor(FPGAs)

Pixel DSP Farm(joint track+vertex)

PTSM

RegionalControl

andMonitoring

BTeV Run Control

GL1

L2/L3(Linux Farm)

Opinions

Requested/AssignedCrossing data

1 Highway

GLSM

L1Buf

L1Buf (raw)

L1Buf (basis)

L1Buf (details)

L1Buf (details)

L1Buf (basis)

Accept/Reject Decisions

Resource Mgr

/c/spot

Others...


18

Trigger Path Issues• direct (optical) link, front-end to FPGA-part

– geographic specificity• Many-to-many interconnect, from FPGAs to DSPs

– work assignments handed out, one crossing to each DSP

– all FPGAs contribute to the “one crossing”• hence many-to-(one-of)-many

– of)-many ~ 2500 DSPs (!)

– no inter-DSP connection (needed…)


19

Trigger Path Issues (page 2)

• DSPs to GL1– each DSP offers one “opinion” per crossing

• multiple triggers = multiple opinions– every 132ns

• GL1 decides– accept/reject

• based on the opinions received for that crossing– 7.6 million decisions per second


20

Trigger Path Issues (page 3)

• Resource Manager– conveys GL1 decisions to L1 Buffers,

• “push” (as opposed to “pull”) architecture

• L1 Buffers deliver data to appointed L2 processor– Pentium-class Linux machine(s)

• again, many-to-(one-of)-many– of)-many ~ 2000 Linux machines


21

Trigger Path Issues (page last)

• L2 Processor is L3 Processor– sufficiently interesting events

receive additional (L3) processing– events that satisfy L3 are recorded to “tape”

• or saved on disk, or burned to CD, or ...


22

Muon DSP Farm

L1Buf (basis)

L1Buf (basis)

Muon Preprocessor

Segment Preprocessor L1Buf (cooked)

MTSM

Detector

Front End Board (DCB)

Crossing Switch

Pixel Preprocessor(FPGAs)

Segment Preprocessor(FPGAs)

Pixel DSP Farm(joint track+vertex)

PTSM

RegionalControl

andMonitoring

BTeV Run Control

GL1

L2/L3(Linux Farm)

Opinions

Requested/AssignedCrossing data

1 Highway

GLSM

L1Buf

L1Buf (raw)

L1Buf (basis)

L1Buf (details)

L1Buf (details)

L1Buf (basis)

Accept/Reject Decisions

Resource Mgr

/c/spot

Others...


23

Pixel (Muon) Trigger Supervisor Monitor (P/M)TSM

• Control Functions:– Initialization

• sets the maximum bandwidth necessary– can not take all day...

– Command Parsing and Distribution• to subordinates, from RunControl

– Error Response • autonomous, “local” (regional)• not to be confused with higher-level driven Error

Handling, which appear as “Commands”


24

(P/M)TSM (continued)

• Monitor Functions– Error Message Collection and Reporting

• organized, formatted, sent higher up– Hardware and Software Status

• not unlike Error Messages...– Status and Data Histogramming

• utilizes remaining (P/M)TSM system bandwidth


25

EtherNet,for example

ARCNet,for example

Close-up: FPGATrigger Supervisor and Monitor

Preprocessorfor example...

_TSM

Regional Controland

Monitoring

BTeV Run Controland database(s), etc.

_TSM

Local Controland

Monitoring

FPGA

JTAG,programming

and debug/c/spot run

Local Config(Flash)

Regional copyof config data,

and history

FPGA

StandaloneOperationalCapability

ControlInfluence

MonitoringAwareness

Fire DetectPower, Cooling Monitoringand Control


26

EtherNet,for example

ARCNet,for example

Close-up: DSPTrigger Supervisor and Monitor

DSP

_TSM

Regional Controland

Monitoring

BTeV Run Controland database(s), etc.

_TSM

Local Controland

Monitoring

JTAG,programming,

debug, and monitoringrun(spot);

run;

Local Config(Flash)

Regional copyof config data,

and history

DSP

StandaloneOperationalCapability

ControlInfluence

MonitoringAwareness

Fire DetectPower, Cooling Monitoringand Control

Host PortInterface

DMAin

BSPout

FPGA


27

Factoids• TMS320C67x DSP

– ~70 Kbyte internal RAM, ~1200 MIP– fixed/floating point

• TMS320C64x DSP– ~1 Mbyte internal RAM, ~3x faster… – fixed point only

• times 2500 devices (!)– Sony Playstation 2.5K ?


28

More Factoids• Host Port Interface (TI DSP only)

– almost-direct access into the DSP• peek, poke• uses DMA (like) resources…

– (concept not unique to TI)• DMA

– crossing data in• Buffered Serial Port(s)

– opinions out; dual, ~75Mbps (C6x)


29

DSP/BIOS (Texas Instruments)• based on SPOX

– “the oldest DSP RTOS…”• scalable real-time kernel

– small footprint (< 2Kw)– preemptive multithreading

• scheduling (tasks, soft/hard interrupts)• synchronization (between tasks, interrupts)

– hardware abstraction– extensible


30

DSP/BIOS (continued)

– real-time analysis• real-time instrumentation

– explicit API calls– (controllably) implicit, at context changes

• host-target communications– statistics gathering

• same as above– host data channels

• binds kernel I/O objects to host files


31

RTDX - Real Time Data Exchange

• utilizes JTAG chain (and emulation bits)• target (kernel) component

– moves messages to/from DSP/BIOS queues from/to JTAG-accessible buffer

– “real time” target to host (?)• host component

– data visualization and analysis tools– data to target “not” real-time… (?)


32

RTDX (continued)• fundamental (TI) model

– one PC running CCS (or app with CCS API calls)

– small number (4’ish) of DSPs on one JTAG chain

– JTAG/emulator “pod” connects PC to chain• (2 “extra” emulation bits, in addition to

TDI/TDO/TMS/TCLK)

• BTeV challenge– not buying 500 PC’s to support 2000 DSPs…


33

BTeV Conclusion(s)• fierce data rates• massive parallelism• heterogeneous control-in-depth

• limited loss-of-data (or performance)– tolerable

• any loss of understanding– unacceptable


34

BTeV TimeTable• detailed technical design report

– required Feb 2002• R&D for TDR, now

• to be operational in 2006

• http://www-btev.fnal.gov/btev.html

Documents

BTeV Availability Issues We don’t have problems any more, we have issues... 21 Aug 2001 M. Haney, University of Illinois