APA integration test for the ProtoDUNE-SP LAr TPC at CERN · 2018. 11. 22. · 6/12/2018 • Xilinx Kintex UltraScale • 16-lane PCIe Gen3.0, throughput tested ~101Gb/s • 48 optical

APA integration test for the ProtoDUNE-SP LAr TPC at CERN

Junbin ZhangOn behalf of the cold electronics team

Brookhaven National Laboratory

21st IEEE Real Time Conference June 12,2018

Deep Underground Neutrino Experiment (DUNE)

6/12/2018

over 1000 collaborators from 175 institutions in 30 countries

4x10 kton LAr TPCsFar Detector at 1.5km underground:

4x10 kton Liquid Argon Time Projection Chambers (TPCs)

Two TPC concepts:• Single Phase (LAr)• Dual Phase (Ar gas + LAr)

J. Zhang - 21st IEEE RT

NP04 (ProtoDUNE-SP) ~ 1% of DUNE

6/12/2018

CPA modules

Field cage modules

Field cage modules with ground plane

APAsEnd wall field cage modules

– ProtoDUNE-SP TPC: ▪ 6 full-size (6x2.4 m2) Anode Plane

Assemblies (APAs) ▪ A total of 15,360 TPC channels

▪ 3 Cathode Plane Assemblies (CPA)

• 2 x 3.6 𝑚2 drift regions

– Goals:▪ Prototyping production and installation

procedures for DUNE▪ Validate design from perspective of basic

detector performance▪ Accumulate test-beam data to

understand/calibrate response of the detector

▪ Demonstrate long term operational stability of the detector.


Readout electronics (cold side)

6/12/2018

Cold side

7m cold cables

20 FEMBs / APA

8 FE ASICs + 8ADC ASICs / FEMBJ. Zhang - 21st IEEE RT

Readout electronics (warm side)

6/12/2018

Warm side

1 WIEC / APA1 flange board / APA1 PTC / APA1 PTB / APA5 WIBs / APA


FEMB production

6/12/2018

FE + ADC ASIC Screening FEMB Assembly FEMB Screening (Preliminary)

FEMB Final Screening

(FEMBs assembled in CE boxes)

Packaging

CERN

RT(300K)LN2(77K)

RT(300K)LN2(77K)

RT(300K)LN2(77K)


APA Integration Test

CE box checkout (in barrack) Install CE boxes on the top of APA CE box checkout ( on APA)

Move APA into the cold boxExperimental area--- extension EHN1 hallMove APA into Cryostat

Warm test

Cooling down

Cold test(150 K)

Warming up

Warm test

6/12/2018 J. Zhang - 21st IEEE RT

ENC Performance vs Temperature

6/12/2018

APA2 (2018-01) Lowest temperature reached ~ 159K

1. Uniform gain (77 e-/bin) is applied for calculating noise of all channels2. HV Bias voltages were off3. Data are read out chip by chip over local diagnostic GbE port.J. Zhang - 21st IEEE RT

Cooling down

ENC at 159K:U-plane: 481 e-

V-plane: 481 e-

X-plane: 398 e-

Cryogenic temperature Warming up

Data flow (FEMB in cold side)

6/12/2018

FE_ASIC1-4

ADC_ASIC1-4

con

nec

tor

FE_ASIC5-8

ADC_ASIC5-8

con

nec

tor

FPGAEP4CGX50DF27C7N

TX0

TX1

TX2

TX3

Det

ecto

r (1

28

ch

ns)

16 × 4 = 64 𝑐ℎ𝑛𝑠

16 × 4 = 64 𝑐ℎ𝑛𝑠

29 × 16𝑏𝑖𝑡 × 2𝑀𝐻𝑧 × 1.25(8𝐵/10𝐵) = 1.16𝐺𝑏𝑝𝑠

32 × 16𝑏𝑖𝑡 × 2𝑀𝐻𝑧 × 1.25(8𝐵/10𝐵) = 1.28𝐺𝑏𝑝𝑠

Data rate of physical link

Data rate of payload

For each Tx link:

@2𝑀𝑠𝑝𝑠/ADC

@2𝑀𝑠𝑝𝑠/ADC


Data flow (WIB in warm side)

6/12/2018

FEMB1

TX0

TX1

TX2TX3

FEMB2

TX0

TX1

TX2TX3

FEMB3

TX0

TX1

TX2

FEMB4

TX0

TX1

TX2

TX3

TX3

Equ

aliz

erEq

ual

izer

Equ

aliz

erEq

ual

izer

CO

LDA

TA_R

X(T

ran

scei

ver)

gro

up

1

CO

LDA

TA_R

X(T

ran

scei

ver)

gro

up

2

CO

LDA

TA_R

X(T

ran

scei

ver)

gro

up

3

CO

LDA

TA_R

X(T

ran

scei

ver)

gro

up

4

LIN

K F

IFO

sLI

NK

FIF

Os

LIN

K F

IFO

sLI

NK

FIF

Os

FEM

B d

ata

pro

cess

or

RCEPCS

FELIXPCS

EventBuilder

QSFP+

RCE

SFPGbEUDP_IO

Slow control

Timingprocessor

FELIX

link speed: 9.6 𝐺𝑏𝑝𝑠

Link speed: 1.0 𝐺𝑏𝑝𝑠

PC

link speed: 5 𝐺𝑏𝑝𝑠

Link speed: 1.28 𝐺𝑏𝑝𝑠

PTC

Warm Interface Board (WIB)

PTB

Timing (fiber)


ProtoDUNE-SP WIB-FELIX readout

6/12/2018

• Xilinx Kintex UltraScale• 16-lane PCIe Gen3.0, throughput tested ~101Gb/s• 48 optical transceivers (MiniPOD)• On-board timing interface

BNL-712 (FLX-712) PCIe card

APA

WIB1

WIB2

WIB3

WIB4

WIB5

FELIX

FULL mode: @9.6 Gbps per link

One FLX-711/712 card has enough bandwidth to read out one APA with 10 FULL mode links.


WIB-FELIX test stand @ BNL


FEMB1 FEMB2 FEMB2 FEMB3

WIB

QSFP+ GbE

FLX-711

Gb

EFELIX server

⚫ FPGA DAC (1,2,3,4,5,8)⚫ Gain = 25mV/fC⚫ Peaking time = 1us

Pedestal from one WIB

6/12/2018

Collection wires: X

Induction wires: V

Induction wires: U


Calibration Waveform

6/12/2018

dac = 1,2,3,4,5,8

Collection wire

DAC = 1,2,3,4,5,8


Calibration: ADC (Peak & Area) vs. Charge

6/12/2018

Peak vs. Charge

Area vs. Charge

DAC = 1,2,3,4,5,8 INL = 0.212%

DAC = 1,2,3,4,5,8INL = 0.184%


ProtoDUNE-SP –Present and Future

6/12/2018

All 6 APAs and 3 CPAs have been installed into the ProtoDUNE-SP cryostat

• Cold Electronics (CE) and photodetectors (PD) have been tested before and after the installation on APAs

• All CE and PD cables are routed though the chimney out of the

cryostat and connected to the corresponding crate.

• Integration and readout tests are ongoing at CERN and BNL

• Beam run is scheduled from August 29th to November 11th


Summary

6/12/2018

• BNL developed cold electronics towards low temperature (77K-89K) is an enabling technology for liquid detectors. All CE boxes were assembled at BNL and shipped to CERN after a comprehensive set of QA/QC tests.

• All APAs are fully characterized before moving into the cryostat.

• Good ENC performance reached at 159K (398 e- for X, 481e- for U,V)

• BNL will advance the design of DUNE cold electronics and continue to contribute efforts at FELIX readout for ProtoDUNE and DUNE in future.



Thank You!

https://jobs.bnl.gov/job/upton/postdoctoral-research-associate/3437/8059129

•This job post is for multiple openings

•The successful candidate is expected to play an essential role in FELIX and

Global Trigger development in the ATLAS upgrades and other high-energy

physics, nuclear physics and astronomy experiments. • Design, develop, prototype, and produce hardware and firmware for the ATLAS experiment

• Evaluate hardware and firmware and characterize system performance

• Participate in system integration of multiple combined systems

Job opening at BNL for Trigger & DAQ

https://jobs.bnl.gov/job/upton/postdoctoral-research-associate/3437/8059129

Backup slides


• FE ASIC

– Built-in 6-bit DAC for calibration pulse generation

– Built-in analog monitoring output for debug

– Address pole-zero cancellation and drive capability in buffer-off mode

– Add higher bias current (1nA and 5nA) options and smart reset

– Revise BGR start-up circuit and increase ESD protection on I/O

– Will be used to instrument SBND and ProtoDUNE-SP

• ADC ASIC

– Implement COLDATA (DUNE baseline design by FNAL, prototype expected in FY19) compatible interface and FE ASIC compatible configuration

– Address the early saturation and roll-back

– Implement power-on default configuration and extend soft-control functions

– Revise BGR start-up circuit and increase ESD protection on I/O

– Improve ADC INL/DNL → not completely resolved

– Will be used to instrument ProtoDUNE-SP

– SBND is exploring COTS ADC option

– Cold ADC ASIC development is very challenging given the amplified mismatch error and inaccurate simulation model in cryogenic temperature

CMOS Cold ASICs Upgrades Implemented

4.5 mm

6.1

mm


Front End Electronics production at BNL

• Based on DUNE design, 20CE boxes have been installed on each APA (120 CE boxes, 15360 electronic channela in total).

• CE boxes were assembled at Brookhaven National Laboratory (BNL) and shipped to Cernafter a comprehensive set of QA/QC tests.

• Integration tests in LN2 were performed at BNL by a 40% of DUNE APA (2.8 m x 1.0 m) in a smaller cold box.


Inside Cold BoxFilled with LN2

WIEC

Ch

imn

ey

40% APA

6/12/2018

Final: CERN Grounding Scheme (with SHV cable connected)

20 CE boxes(FEMBs)

ProtoDUNE APA (6m x 2.5 m)

Cold Box

5WIB

Faraday cage

CR + PSL

WIENER PS

+48 V

0V(GND)230V

Detector ground

WIEC

Flange boardFT

Optical SW

Ethernet cable

Optical fibers

7 meter samtec power cables & data cables

48V PS cable (20m)

LV PS (-)detector ground

LV PS (+)

Fans PS cable (20m)

+18 V230V0V(GND)

DCFans

PS RACK

open

~35ft (10m)25 mm2

Building ground

PTB

PTC

Panel

Note: 1. Heater cable is connected but not power applied. The shielding of heater cable is connected to Heater PS but open on WIEC crate (same as Fans PS cable)2. Two SHV cables (-650V for G plane and 800V for X plane) connect to HV power supply3. Currently, U plane is shorted to Detector ground somewhere (investigate after cold test). The SHV cable for U plane is disconnected4. The SHV cable for ED outer is disconnected

HV

230V 0V(GND)

10 kohm

7 meterHV cables

HV

box

SHV cables (20m)

Note:Power supply for DC fans are separated from supply for CE, the shielding of fans PS cable disconnects to flange. (DC fans are isolated from WIEC)


Equivalent Noise Charge (ENC) – Integrals

Noise Coefficients A1, A2, A3

2 2 2 2 2 2 211 11 2 0 3 2 0 32 2n in in f n in in f

AENC e C I C A I qI I e C C A A qI A

= + + = + +

( ) ( )

( ) ( )

( ) ( )

2 2 2 11

221 2

2 2

22

3 3

1

2

1

2

1

2

AI w t dt H j d

I w dt H j d A

I w t dt H j d A

− −

− −

− −

= = =

= = =

= = =

( ) ( ) ( )2 22 2 21

2

0

1 1

2 2neq neqENC i H d i H d

−

= =

Series

white

1/f

Parallel

white

Time domain

(weighting function)Frequency domain

(transfer function)6/12/2018 J. Zhang - 21st IEEE RT

Noise (ENC) vs TPC Sense Wire and Signal Cable Length for CMOS at 300K and 89K

MIP Signal for 3x3 and 5x5 mm Sense Wire Spacing


CMOS at 77K: ENC< 103 e rms

MicroBooNEENC~400 e rms

DUNE with warm electronics (300K)ENC~6x10^3 e rms

6/12/2018

At 77-89K, charge carrier mobility in silicon increases, thermal fluctuations decreasewith kT/e, resulting in a higher gain, higher gm /I, higher speed and lower noise.

Cold vs. Warm CMOS: static characteristics vs. T

0.0 0.3 0.6 0.9 1.2 1.5 1.810

-5

10-4

10-3

10-2

10-1

100

101

~18m

V/d

ec

~72m

V/d

ec

(ln(

10)n

VT)

CMOS018

MEASURED

ID

gm

LN

RT

I D [m

A],

gm

[mS

]

VGS

[V]

NMOS, L=0.18µm, W=10µm

~ 30 300

~ 116 77

m

D B

at T Kg q

at T KI nk T

=→ =

=

10-6

10-5

10-4

10-3

10-2

10-1

100

101

102

0

20

40

60

80

100

120 MEASURED

NMOS PMOS T=300K

L=360nm

L=270nm

L=180nm

NMOS PMOS T=77K

L=360nm

L=270nm

L=180nm

gm

/ID [

V-1

]

Drain Current Density [mA/mm]

CMOS018

Transconductance / drain

current

Documents

APA integration test for the ProtoDUNE-SP LAr TPC at CERN · 2018. 11. 22. · 6/12/2018 • Xilinx Kintex UltraScale • 16-lane PCIe Gen3.0, throughput tested ~101Gb/s • 48 optical