Fysikalisk Systemteknik Christian Bohm Overview: About the group Overview of projects

FysikaliskSystemteknik

1Instrumentation seminar- 2003-03-20

Fysikalisk Systemteknik

Christian Bohm

Overview:About the group

Overview of projectsWhat is an FPGAOur major projects



Fysikalisk Systemteknik

Personell:Professor Christian BohmLecturer Sam Silverstein

Part time lecturer Magnus EngströmAdjunkt Eddie Ahlestedt

Forskningsingenjör (emeritus) Hans Eriksson

Forskningsstuderande:Jonas Klereborn

Abdelkader BousselhamAttila Hidvegi

Florian Bauer (external)New

We collaborate with experimental physics groups, focusing on thedevelopment of new instruments.

Make it easier to develop and maintain useful engineering skillswhile retaining an active grasp of the relevant physics.

Use experience from projects to solve new problems.

Look for general solutions and methodologies which are easierto carry over to new problems.

Experimental Physics Technology

Instrumentation Physics



Different instrumentation projects:

In collaboration with particle physics, SU• RD-16 FERMI and RD27 first level trigger• Digitizing electronics for ATLAS TileCal• The Jet/Energy-sum processor for the

ATLAS first level trigger

In collaboration with physicists at KI• Development of a SPECT camera• Development of a PET-Camera

In collaboration with molecular physics, SU• Frequency stabilisation of semiconductor

lasers for laser trapping and coolingof atoms

In collaboration with astroparticle physics, SU• Participation in the development of IceCube



What is a Field Programmable Gate Array?

Configurationmemory

Logic withData path switches

Firmware



Configurable Logic Blocks

FPGAP

rogramm

able interconnect

a

d

f g clk h

e

cb

Programmable IO-blocks

111111111-------1--1---11111111-------1---1-111-11-1

Configuration memory



Configuration memory pattern defines circuit

001011100-------1--0---01000000-------1---0-110-1010

a

d

f g clk h

e

cb

a

d

clk

e

cb



Modifying the memory content changes the circuit

001101100-------0--0---11111000-------0---1-000-1010

a

d

f g clk h

e

cb

a

d

clk

e

b



FPGAs have been around since mid-1980s

Early components were programmed ata bit-level using graphic editors

Increased complexity required better methods:

High level languages (VHDL), or

Schematic specifications



When designing complex circuits with FPGAsone has to consider:

Does the design fit?

Is it fast enough?

Is it too expensive?



FPGA design process

High level description (VHDL)

Functional simulation (no timing)

Select FPGA type

Synthesize – translate to simpleprimitives (compilation)

Simple timing simulation

Place and route

Full timing simulation



State-of-the-art FPGAs

•Very complex (Xilinx, Alterra)many gates ~ 8 million gatesmany i/o pins ~ 800 (400 diff)flexible interconnects ~ 7 metal layershigh costs ~ 50 kkr

•Multiple clocks - 8

•Embedded memories – 4 MB

•Embedded multipliers - 200

•Embedded processors – 4 PowerPC

•High speed IO – 16 x 3 GB/s



• Re-use of previously developed code blocksIntellectual Property blocks

• IP-blocks can be:In-house developedCommercially availableFreely available – open-core

• Part of the design can be accomplished byassembling compatible IP-block

Processors (embedded or IP)MemoriesBussesInterfacesEtc.

Efficient tools required



• What to implement in logic

• What to implement in processor software

Hardware software co-design

VHDL System-C or Handle C

When designing complex FPGA modulesone must decide:



.

SELECT

ROI

1kHz

2s

40kHz

40MHz

10Hz

ROIselect

*

Muondetector

HadronCalorimeter

ElectromagneticCalorimeter

Tracker

MERGE

DATA

SECONDLEVELTRIGGERPROCESSOR

+200ms

+2ms

L1-accept

L2-accept

L3-accept

7TeVproton

7TeVproton

FIRSTLEVELTRIGGERPROCESSOR

FIRSTLEVEL

BUFFER

SECONDLEVEL

BUFFER

THIRDLEVEL

BUFFERTHIRD

LEVELTRIGGERPROCESSOR

MASSSTORAGEDEVICE

ATLAS data flow

LHC physics looks for rareevents – 1 in 1014

High event rates andHigh selectivity

1 event in 10000

new data every 25 ns

Since all data must be stored while waiting for the L1 decision the L1 processing must be quick – 1ns

Data from entire detektor but with low spatial resolution and reduced dynamic range from calorimeters and muon detector

10 H

z1

kHz

75 -

100

kHz

40M

Hz

About 100 million channels

1 event in 100

1 event in 100

Data from ROIs with high spatial resolution and full dynamic range from all subdetector

Entire detector with high spatial resolution and full dynamic range from all subdetector



MuonTrigger (Italy)

Calorimetertrigger

Cen

tral

Tri

gger

(C

ER

N) L1 accept

ROI info

The calorimeter trigger is a Birmingham-London-Rutherford-Stockholm-Heidelberg-Mainzcollaboration

Looks for typical features for event selection

Pre

proc

esso

r(H

eiel

berg

)

CalorimetertriggerElectron/Tau

Processor (GBR)

Jet (Sthlm) andMissing energy (Mainz)

processor

AnalogInput

signals

64x64x2Analogsignals

Digitizesdeterminesamplitudes

and pulse starts

Looks for isolated clusters resembling singleElectrons/hadrons in the ECAL and HCAL

64x64x28-bit

32x32x28-bit

Looks for energyclusters

ATLAS first level triggercollaboration with particle physics SU

Looks for energybalance



Cluster Processor(e/ and /had)Cluster Finding

Cluster Processor(e/ and /had)Cluster Finding

CalorimeterLAr, Tile

CalorimeterLAr, Tile

Tower 0.1 x 0.1

Pre-ProcessorTiming alignment

10-bit FADCFIFOBCID

Lookup TableBC-MUXSum 2x2

Pre-ProcessorTiming alignment

10-bit FADCFIFOBCID

Lookup TableBC-MUXSum 2x2

Analog

CountCount

Jet/Energy-SumProcessor

JetsET EX EY

CountCountET, ETET, ET

Pre-ProcessorRODs (DAQ)

Pre-ProcessorRODs (DAQ)

CP/JEPRODs (DAQ)

CP/JEPRODs (DAQ)

Region OfInterest

Builder (L2)

Region OfInterest

Builder (L2).1 x .1

.2 x .2

Level-1CTP

Level-1CTP

Realtimedata path



The JET/energy sum trigger

Look for .4x.4, .6x.6 and .8x.8 energy clusterscentered around a local .4x.4 maximum

Form global sums of total Et and missing Et

Process ~1024 .2x.2 jet elements in parallelAll requiring neighborhood information

32 processor boards with large FPGA for Jetand missing energy processing, sharing overlapping environment data

Latency (processing time) 200 ns

Many differentmodule types

Standardizedmodules



The JET trigger

We have built a 18 layer backplane with>20 000 pins for the Jet and the E/ processors

VME - -Communicationwith neighborsReport results



The JET trigger

We have participated in the design of the JEMProcessor board.

And developed firmware for the algorithmsand control functions



Experiences from the trigger project:

Large scale system design

Massive pipelined parallel processing

Reliability

Large FPGA design (> 1 Mgates)

Draw on experience from earlier bit-serialtrigger project to do pipelined processingof multiplexed data -> more efficient useof logic and interconnects!



Many prototypes – test beam tests – earliestATLAS subsystem – lots of firsts – productionexperience – 2000 boards this year

Tile Calorimeter

Module

Drawercross-section

PMTs

Fiber

Particle

3-in-1

Digitizer

HV-control

PMT L ig h t m ixe r

3-in-1 mother board

Drawers Superdrawer

interface board s-link lsc

Tile_DMU Tile_DMUTTC-rx




Tile_DMUTTC-rx




Tile_DMU

optical fiber

The ATLAS TileCal Digitizercollaboration with particle physics SU



Task: to digitize pre-amplified PMT-pulsesand to transfer data selected by the L1-triggerto the higher level triggers.

The ATLAS TileCal Digitizer

• 16-bits dynamic range with limited precision• L1 buffer memory – 2.5 us• Storage of selected data• Format data• send to level 2

• Physical layout• Noise control• Radiation tolerance• Reliability (physical chain – electrical star)

• We also made a optical link with matchingreliability



Analog part

Digital part

• Noise control

• Reliability

• 16-bits dynamic range with limited precision

10

• L1 buffer memory – 2.5 us

L1 buffer

• Physical layout

AD

C

AD

C

AD

C

AD

C

AD

C

AD

C

AD

C

AD

C

AD

C

AD

C

L1a L1 buffer

Event storage

Format and send

• Storage of selected data

Event storage

L1aTrigger and

TimingCircuit

System clock – level 1 accept

• Format data

Format and send

• Send to level 2

Data till second level trigger

• Radiation tolerance

Radiation hard ASIC

Radiation tolerantcustom ASIC- no FPGA

Components OfThe Shelf - COTS

10

AD

C

AD

C

Higain

Logain



Experiences from the digitizer project:

Large scale system design

System aspects – timing and grounding

Reliability

Radiation tolerant design

Production

Even if did not use FPGAs in the Digitizerwe used them extensively when buildingprototypes and testbenches.



SU – SPECTCollaboration with Karolinska hosptal

72 PMTs around crystal – position determinationvia light sharing

Earlier design based on transputers discontinued

Pulse detection + sampling ADCsDigital pulse processing + digital triggerFirewire networkXilinx FPGAsTexas Instrument DSP – TMS 320 6000 family

The design of a SPECT camera with an innovativecylindrical crystal

1 2 . 5 mmQ ua rt z g l as s

1 2 . 5 mmN a I ( T l )

L e ad s hi e ld

7 2 6 0 mm h ex P M - t ub e s

Co l l imat or b l oc k



ICE-CUBECollaboration with the astroparticle physics group at SU

80 strings

Volume 1 km3

Photomultiplier

Self-triggers on each pulse Captures waveforms Time-stamps each pulse Digitizes waveforms Performs feature extraction Buffers data Responds to Surface DAQ Set PMT HV, threshold, etc Noise rate in situ: ≤500 Hz

1400 m1000 m

60 m

odul

es/s

trin

g

Digital Optical Module (designed by D. Nygren)



ICE-CUBE

The DOM circuit board

2 DOMs share 1 twisted pair for power supply and communication2 ATWD - 4 channel transient waveform recorder 300 MHz 256 samples

2 channels – hi and lo gain from PMTSymmetric timing pulses between hub and DOM sampled at

20 MHz 10bitsSupports a higly stable local clock 3.3 ns rmsFPGA and CPU combined in new Altera FPGA

Our part feature extraction



ICE-CUBEExperimental Requirements IceCubeExperimental Requirements IceCube

Time resolution: <5 ns rms Waveform capture:

>250 MHz - for first 500 ns~40 MHz - for 5000 ns

Dynamic Range:>200 PE / 15 ns>2000 PE / 5000 ns

Dead-time: < 1% OM noise rate: < 500 Hz (40K in glass sphere)

--

DOM Pair

20 kB/sec

StringProcessor

N x 20 kB/sec

All Hits -0.6 MB/sec

80 Strings

String Subsystem:60 DOMs

N pairs

100 BaseTTotal traffic:1.6 MB/sec

StringCoincidenceMessages

GlobalTrigger Event Triggers /

Lookback Requests forall Strings - 0.8 MB/sec

EventBuilder

Built events ~ 1 MB/sec(all event builders)

SAN(Network

Disk Storage)

"DOMHUB"

Lookback RequestsString CoincidenceMessages - 170 kB/sec

Fulfill Lookback Messages 0.6 MB/sec

FulfillLookbackMessages

Online LAN100 BaseT

Total traffic:1MB/sec

Proposed IceCube DAQNetwork Architecture

String LAN100 BaseTTotal traffic:0.6 MB/sec

OfflineData

HandlingTape

Satellite

Event LAN



Digital Laser ControlCollaboration with Anders Kastbergs group

Frexghi Habte

Aim: to design a simple laser control that canmanage a large number of units

Our solution: use an FPGA based lock-in module

Cordic algorithm to produce sine and cosinewaveforms + second order Butterworth low passFilter (4Hz)

Hardware design based on SPECT module

laserAbsorption

cell

Modulation

detector

Lock-inamplifier

Lock on lowfrequency component 0 lock on maximum

xAsin(t+) Asin(2t+)-Asin Asin

cos(t)/2



Future

Our involvement in the ATLAS projectswill eventually decrease – the digitizerduring 2004 and the trigger during 2006.Now they are quite intense

The SPECT camera project should terminateIn its present form this year.

We are participating in a EU applicationCoordinated by Anders Brahme at KI. Ourpart here would be in the development of ahigh resolution whole body positron camera.Lars Eriksson from KI and CPS would bepartner in this project.

There will surely be other exciting newprojects coming up.

Documents

Fysikalisk Systemteknik Christian Bohm Overview: About the group Overview of projects