Infieri 2014 1

Data transmission needs and challenges for Frontier Particle Physics: Part 1

• Why HEP needs special?• Radiation-hard high speed optical links• Lasers and p-i-n diodes Suen’s talk• Radiation tolerance of fibres• Single Event Upsets

• Solution for LHC phase 1 upgrades– Versatile Link: optoelectronics– GBT: rad-hard chipset

Tony Weidberg

Infieri 2014 2

HEP Requirements• Need high speed links

– ~ 100 M detector channels– Typical read out rates for LHC detectors ~ 100 kHz

• Usual requirements– High speed links (5 to 10 Gbps)– Low costs

• Unique requirements– Radiation tolerant up to ~ 500 kGy (Si)– Single Event Upsets– Very low mass (material degrades resolution) and small– Very low power (cooling more material)– Avoid magnetic materials (e.g. ferrite cores)– Access very difficult very high reliability

Tony Weidberg

Infieri 2014 3

Example: CMS silicon tracker (LHC)

Tony Weidberg

Silicon detectors

Readout and optoelectronics(space limitations)

Fibre ribbons

Access to optoelectronics clearly very difficult very high reliability

Tony Weidberg Infieri 2014 4

Step Index Multi Mode (SIMM) Fibres

• Simplest fibre: Step Index Multi-mode fibre.

• Light trapped by total internal reflection.

• Maximum angle

• Problem:– Larger angles q longer optical

path length longer flight time. – Many modes with different q

propagate (multi-mode)– large modal dispersion very

low bandwidth.

2 2 1/ 21 2sin( ) ( )MAX n n

Typical diameters (mm)

Core 50

Cladding 125

Buffer (protection) 250

OBSOLETE

Tony Weidberg Infieri 2014 5

Graded Index (GRIN) fibres

Adjust refractive index profile n(r) vs r to minimise modal dispersion.

RIA Im(n) must change Re (n) must also change small change in a could increase dispersion (degrade bandwidth).

a

})(21{)( 1

ar

nrn

Dispersion vs a

Very deep minimum ~ 2.a

Singlemode fibres• Use small diameter core ~ 5 to 10 mm.

– Must use wave optics (not geometrical)– Fibre is dielectric waveguide: solve Maxwell’s equations subject

to boundary conditions.• Continuity at core/cladding boundary: Dperp , E// , Bperp ,H//.• Propagating wave in core • Evanescent wave in cladding (exponentially decreasing with

radius)• Allowed solutions propagating modes in fibre.

– If diameter is small enough only one mode allowed

Tony Weidberg Infieri 2014 6

Infieri 2014 7

Pros & Cons of Singlemode fibres

• Pros: no modal dispersion higher bandwidth• Cons: only one mode and smaller core diameter more

precise alignment, i.e. harder to couple light into singlemode fibre

• Multi-mode fibres benefit from cheap low power VCSELs (850 nm)

• Use Singlemode fibres for long-haul and Multimode fibres for short distances (up to ~ 1 km).

Tony Weidberg

Infieri 2014 8

Dispersion in Singlemode Fibres

• Two sources of dispersion in SM fibres:– Chromatic dispersion, variation of dn/dl with l– Intra-modal dispersion: effect of waveguide vg(k) is a

function of k, “different path length for different k” (very naïve picture).

• 1310 nm better for chromatic dispersion but 1550 better for absorption (also for Er doped fibre amplifiers).– Telecoms fibres use dispersion shifted fibres• Waveguide dispersion designed to cancel material

dispersion at 1550 nm.

Tony Weidberg

Tony Weidberg Infieri 2014 9

Fibre Dispersion and Attenuation

Dispersion is a minimum ~ 1.3 mm

Attenuation is minimum ~1.5 mm

Long distance uses ~ 1.3 or 1.5 umShort distance use 850 nm because of availability of cheap VCSELs

l (mm)

l (mm)

Infieri 2014 10

Radiation Tolerance Fibres

• Radiation creates “colour centres”, states which allow increased absorption Irradiated fibre goes yellow!

• Radiation Induced Attenuation (RIA)– Important for other applications, e.g. nuclear

power stations, fusion reactors etc.• Fibre bandwidth• Fibre mechanical reliability after radiation

Tony Weidberg

Infieri 2014 11

Radiation Induced Attenuation

• Most fibres use P dopants slows down annealing of radiation damage and results in very poor radiation tolerance.

• Normal dopant to create refractive index profile is GeO2 but F increases annealing rate better radiation tolerance with F doping (n.b., F decreases refractive index but can still design correct doping profile).

• Other tricks commercially sensitive …Tony Weidberg

Infieri 2014 12

Radiation Source

• 60Co g sources at RITA (SCK SEN) Belgium &BNL.

• Annealing of damage is very important– Temperature and dose rate

important• At HL-LHC Si detectors will

operate ~ -25oC– Warm and cold measurements

of fibres• Dose rates : 0.0265 to 22.5

kGy(Si)/hourTony Weidberg

Infieri 2014 13

Radiation Induced Attenuation• Measure returned optical power for– Reference fibre (not in radiation zone)– Fibre in radiation zone– Difference gives Radiation Induced Attenuation (RIA).

Tony Weidberg

Infieri 2014 14

Dose Rate Effects• Damage for same dose lower for lower dose rates because

of annealing.• Spike at very high dose rate not seen at lower dose rates.

Tony Weidberg

No cooling

Infieri 2014 15Tony Weidberg

Infieri 2014 16Tony Weidberg

Note “spurious annealing” when fibre removed from source

Infieri 2014 17

Cold Fibre Irradiation (1)• Some fibre types survived high dose rate

(HL-LHC in 24 hours) at T=-25C

Tony Weidberg

Singlemode Fibre X

CO2 cooling

Infieri 2014 18

Cold Fibre Irradiation (2)• Other fibre types did not survive high dose

rate (HL-LHC in 24 hours) at T=-25oC (but was good in warm irradiation).

Tony Weidberg

Corning Infinicor SX+

Damage so large it saturated our measurement system!

Inconclusive!

Infieri 2014 19

Next Tests

• We have fibres that are – radiation tolerant at high dose rate and warm

temperature.– Not radiation tolerant at high dose rate and low

temperature (T~ -25oC).• Need long exposure at low dose rate at T=-25oC.– Not trivial … used evaporative CO2.

– Blow-off system changed CO2 bottles every day for 10 days.

Tony Weidberg

Infieri 2014 20

Radiation Induced Attenuation

• Measure RIA vs dose.

• T=-25oC • Low dose rate– 0.7 kGy(Si)/hour

• Combine this with expected dose at HL-LHC

• 5 fibres qualified for use at HL-LHC.

Tony Weidberg

Cold irradiation

Infieri 2014 21

Fibre Bandwidth

• Need high bandwidth – speed * distance (units MHz. km)– Distance for detectors ~ 0.1 km, readout rate ~

10Gbps bandwidth > ~ 1000 MHz . km• Commercial fibres can meet these

requirements– OM3: 1500 - 2000 MHz . km – OM4: 3500 - 4700 MHz · km

• Will bandwidth be affected by radiation?Tony Weidberg

Infieri 2014 22

Fibre Bandwidth & Radiation

• RIA imaginary part of n(w) changes with irradiation – can affect profile of n(w) which is precisely

optimised for GRIN fibre– Change in is frequency dependent, Kramers-

Kronig relation expect to change and hence change chromatic dispersion.

Tony Weidberg

dP0

220

0)(

/)(21/)(

)()(

Infieri 2014 23

Differential Mode Delay (DMD)Principle

Tony Weidberg

Scan injected pulse over fibre core and measure arrival timeHigh power laser coupled to SM fibreSensitive to modal dispersion.

Infieri 2014 24

DMD Results

• Mode delay varying across fibre as expected

• Negligible change with irradiation.

Tony Weidberg

Infieri 2014 25

Chromatic Dispersion

• Measured by Time of Flight (TOF) for different wavelengths.

• Units are ns/(km mm).• Negligible change after

500 kGy(Si).• Fibre qualified as

radiation tolerant from RIA perspective.

Tony Weidberg

Draka Elite SRH-MMF

Infieri 2014 26

Fibre Mechanical Reliability

• Extensive studies by fibre manufacturer’s predicted excellent long-term reliability.

• Confirmed by reliability of installed fibres.• Does radiation damage change reliability?– Some evidence that it can happen, e.g. F in

cladding can produce aggressive radicals that damage the fibre.

– Perform tests on qualified radiation tolerant fibres.

Tony Weidberg

Infieri 2014 27

Fibre Mechanical Reliability

• Long term reliability methodology based on statistical analysis of fibre breaks in destructive tests.

• Measure median breaking strain at a given stress rate.

• Repeat at different stress rates

• Stress corrosion parameter n long term reliability.

Tony Weidberg

2 point bend testerJaws move together at constant speed until fibre breaks.Detected by microphone.Record breaking strain.

Infieri 2014 28

Micro-bending• Mechanical strength

dominated by glass.• Buffer coating protects fibre

and minimises loss from micro-bending

• Assess quality by fibre winding machine with sandpaper to introduce micro-bending.

• Compare fibre before and after irradiation.

• Small improvement with radiation!

Tony Weidberg

Infieri 2014 29

Summary Fibre Radiation Tolerance

• Have identified suitable multi-mode and singlemode fibres– Small increase in attenuation with radiation, can

absorb loss into optical power budget.– No significant change in fibre bandwidth with

radiation.– Small improvement in mechanical reliability with

radiation!

Tony Weidberg

Infieri 2014 30

Single Event Upsets in ATLAS SCT

• Single Event Upsets (SEU) studied for ATLAS & CMS in test beams – Study of SEU in ASICs in LHC operation.

• Expectations for (SEU) from test beam data.• SEU in SCT operation and comparisons with test beam– p-i-n diodes in TTC link.

• Mitigation for ATLAS operation.• Mitigation strategy for SEUs at HL-LHC.

Tony Weidberg

Infieri 2014 31

SEUs in SCT, how and where?

• Particles deposit sufficient charge in small region of silicon bit error (SEU)– Typically needs nuclear interaction to deposit sufficient

energy, i.e. MIPs are harmless.• In p-i-n diode that receives optical TTC signal– Single bit error loss of synchronisation of a FE module.

• Also In static registers in ABCD– Don’t care about dynamic memory (pipeline) but static

registers will stay wrong after an SEU until reset. – Look at effects in DAC threshold register.

Tony Weidberg

Infieri 2014 32

SEU Studies

• Measure SEU rates for prototype in test beams: – Low energy p/p beams (mainly 200 – 500 MeV/c)– Extrapolate to LHC spectrum?– No synchronisation with beam bunches.– Angle of incidence.

• Measure actual SEU rates in ATLAS operation and compare with test beam based predictions.– Results shown for barrel SCT only.

Tony Weidberg

Infieri 2014 33

SEU In SCT Optical Links

• On-detector p-i-n diode is Sensitive to SEU – Small electrical signal before

amplifier stage.

• Measure BER with loopback– With beam – Without beam– Difference SEU

Tony Weidberg

TTC

Infieri 2014 34

SEU in p-i-n diode – Test Beam • Measured SEU vs current in p-i-n

diode IPIN (simple loopback test) .– No errors with beam off.– No errors for MIPs.– Measured Bit Error Rate vs IPIN

with beam on.– ac coupled charge required

to cause bit flip is proportional to IPIN .

• s higher for 300 MeV/c pbecause of D resonance large variation of s with energy difficult to predict rates for LHC operation.Tony Weidberg

s(SEU)=# bit errors/fluenceJ.D. Dowell et al., Single event upset studies with the optical links of the ATLAS semiconductor tracker, Nucl. Instr. Meth. A 481 (2002) 575.

Infieri 2014 35

SEU in ATLAS Operation • p-i-n diode receives optical TTC signal.• Indirect measurement BER• Signature for SEU in p-i-n diode is loss of synchronisation for L1A

trigger:– TTC sends

• full L1A number to ROD: L1A(full)• L1A signal to detector FE via optical links.

– On-detector 4 bit counter counts L1A and returns 4 LSBs in data stream: L1A(4)

– SEU causes 01 can cause loss of L1A on-detector.– Compare L1A(full) with L1A(4). Persistent discrepancy is SEU.

• No errors seen in “physics mode” running with no beam suspect that these errors during beam are due to SEU.

Tony Weidberg

Infieri 2014 36

Are errors really SEU ?• SEU rate should scale

with module occupancy (proxy for particle flux).

• Occupancy changes from luminosity variations and decreases as radius of barrels increase

• Shows expected linear behaviour

• Total number SEU– Predicted: 2504– Observed: 1949Tony Weidberg

Infieri 2014 37

Mitigation Strategies for ATLAS Operation• SEU in TTC links

– Use large values of IPIN (> 100 mA) to reduce s(SEU)

– Reset pipeline in FE chips and all counters if this de-synchronisation detected by DAQ (20 to 50s).

• Mitigation strategies reduce effects of SEU to negligible level.

Tony Weidberg

Infieri 2014 38

SEUs @ HL-LHC

• Expect SEUs to be more important @ HL-LHC because of higher Luminosity.

• What can we do to mitigate SEU?– Triple event redundancy in gates – Error correction on TTC link. – Propose to correct for sequence of error bursts

up to 16 bits long slide.

Tony Weidberg

Infieri 2014 39

Versatile Link TTC SEU

• Measured BER vs optical power, Optical Modulation Amplitude (OMA).

• SEU killed by error correction (FEC)Error correction required for TTC links

• Tests to determine if it is also required for data

Tony Weidberg

A. Jimenez Pacheco et al., Single-Event Upsets in Photoreceivers for Multi-Gb/s Data Transmission, IEEE Trans. Nucl. Sci., Vol. 56, Iss. 4, Pt. 2 (2009), pp. 1978 – 1986.

Infieri 2014 40

Versatile Link

• Optical links for data read out from detector and Timing, Trigger and Control data needed by the detector.

• Generic system for LHC detector upgrades

Tony Weidberg

Infieri 2014 41

Versatile Link• Need radiation-tolerant optoelectronics

for on-detector components.– Select VCSELs and p-i-n diodes from

radiation studies (see Suen’s talk).– Want very reliable and low mas/non-

magnetic package.– Commercial optical transceivers can’t be

used– Need radiation-tolerant ASICs.– Too much material, magnetic and use

ferrite cores as inductors in laser drivers.• Keep optical sub-assembly and

commercial connectors for reliability but use plastic package.

Tony Weidberg

Infieri 2014 42

Performance• Measure Bit Error Rate

(BER) over 400m fibre (pessimistic)– Designed for 4.8

Gbits/s but works up to ~ 10 Gbps.

– Small penalty in minimum power required.

Tony Weidberg

dBm is optical power in dB relative to 1 mW.dB=10*log10(p2/p1)

Infieri 2014 43

The GBTx System

Tony Weidberg

FEModule

FEModule

Phase – Aligners + Ser/Des for E – Ports

FEModule

E – PortE – Port

E – Port

GBT – SCA

E – Port

Phase - Shifter

E – PortE – Port

E – PortE – Port

CDR

DEC/D

SCR

SER

SCR/ENC

I2C MasterI2C Slave

Control Logic Configuration(e-Fuses + reg-Bank)

Clock[7:0]

CLK Manager

CLK Reference/xPLL

External clock reference

controldata

One 80 Mb/s port

I2CPort

I2C (light)

JTAG

JTAGPort

80, 160 and 320 Mb/s ports

GBTIA

GBLD

GBTXe-Link

clock

data-up

data-down

ePLLTxePLLRx

clocks

Infieri 2014 44

Other Radiation Tolerant ASICS

• GBLD– Laser driver

• GBTIA– Receives optical signal

from p-i-n diode, transimpedance amplifier/discriminator

• Produced in 130 nm IBM process. Radiation tolerance qualified.

Tony Weidberg

4.8 Gb/s, pre-emphasis on

Total jitter: ≈ 25 ps

Infieri 2014 45

Optical Power Budget• Is there enough power

for receiver to work allowing for all losses?

• In particular consider effects of radiation damage.

• Margin represents additional safety margin with worst case assumptions for all other components, so 1.8 dB is fine!

Tony Weidberg

Infieri 2014 46

Summary

• Optoelectronics used for high speed data transfer in HEP.

• Some special requirements, particularly radiation tolerance.– Fibre radiation tolerance demonstrated– Laser and p-i-n diodes in Suen’s talk

• Versatile Link for HL-LHC detectors.

Tony Weidberg

Infieri 2014 47

Backup

Tony Weidberg

Infieri 2014 48

Absolute Rates (2)

• Naïve prediction:– N(SEU) = s(SEU) * Fluence– Ignore variation in s(SEU) with LHC spectrum.– Corrected for variation of s(SEU) with IPIN.– Fluence: use <module occupancy>– Reject long SEU bursts (>60s) 13% uncertainty– Reject modules with multiple errors in one run: 5 to 6% bias.

• Number SEU in data set– Luminosity 7.81 fb-1

– Measured: 2504– Predicted : 1949– Good agreement within large uncertainties.Tony Weidberg

Infieri 2014 49Tony Weidberg

Infieri 2014 50

SEU in ATLAS Operation (2)

• L1A signal is 110 • Short code vulnerable to single bit error (minimize

latency).• Assume 01 transitions more probable than 10 because

of high value of IPIN.• Most probable error “110” “111” • In ATLAS energy deposition synchronised to bunch

crossing, unlike test beam• Creates large uncertainties in extrapolating test beam

cross section to ATLAS operation.Tony Weidberg

Infieri 2014 51

Some References• The radiation induced attenuation of optical fibres below −20°C exposed to

lifetime HL-LHC doses at a dose rate of 700 Gy(Si)/hr, 20012 JINST 7 C01047 doi:10.1088/1748-0221/7/01/C01047

• The Versatile Link common project: feasibility report, 2012 JINST 7 C01075 doi:10.1088/1748-0221/7/01/C01075.

• A Study of the effect of a 500 kGy(Si) radiation does on the bandwidth of a radiation hard multi-mode fibre, 2012_JINST_7_P10021, http://dx.doi.org/10.1088/1748-0221/7/10/P10021

• A study of the effect of radiation on the mechanical strength of optical fibres, 2013_JINST_8_P05011, http://dx.doi.org/10.1088/1748-0221/8/05/P05011

• Further studies of the effect of radiation on the mechanical strength of optical fibres, 2013 JINST_8_P12002, http://dx.doi.org/10.1088/1748-0221/8/12/P12002

• The Optical Links of the ATLAS SemiConductor Tracker, 2007_JINST_2_P09003, http://www.iop.org/EJ/abstract/1748-0221/2/09/P09003

• Single Event Upset Studies Using the ATLAS SCT, 2014_JINST_9_C01050.

Tony Weidberg