Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
Journal of Instrumentation
A PCI Express optical link based on low-costtransceivers qualified for radiation hardnessTo cite this article A Triossi et al 2013 JINST 8 C02011
View the article online for updates and enhancements
You may also likeAssessing the safety of express bus frompassengersrsquo perspectiveA H Azman S Abdullah S S K Singh et al
-
Research on optimization of trafficmanagement and control measures forexpressway emergency based on VISSIMsimulationLi Jiahui Wu Keman Li Yudian et al
-
(Invited) the Progress and Outlook of GaNLaser DevicesJung Han
-
Recent citationsDigital Front-End Electronics for theNeutron Detector NEDAF J Egea Canet et al
-
This content was downloaded from IP address 21910037239 on 29112021 at 1051
2013 JINST 8 C02011
PUBLISHED BY IOP PUBLISHING FOR SISSA MEDIALAB
RECEIVED November 15 2012ACCEPTED December 26 2012
PUBLISHED February 5 2013
TOPICAL WORKSHOP ON ELECTRONICS FOR PARTICLE PHYSICS 201217ndash21 SEPTEMBER 2012OXFORD UK
A PCI Express optical link based on low-costtransceivers qualified for radiation hardness
A Triossia1 D Barrientosbcd MBellatob D Bortolatoa R Isocrateb G Rampazzob
and S Venturab
aIstituto Nazionale di Fisica Nucleare ndash Legnaro National LaboratoryViale dellrsquoUniversita 2 35020 Legnaro (PD) Italy
bIstituto Nazionale di Fisica Nucleare ndash PadovaVia Marzolo 8 35131 Padova Italy
cIFIC CSIC-Universitat de Valenciac Catedratico Jose Beltran 2 46980 Paterna Spain
dDepartamento de Ingenierıa Electronica Universitat de ValenciaAvinguda de la Universitat sn 46100 Burjassot Valencia Spain
E-mail atriossipdinfnit
ABSTRACT In this paper we want to demonstrate that an optical physical medium is compatiblewith the second generation of PCI Express The benefit introduced by the optical decoupling ofa PCI Express endpoint is twofold it allows for a geographical detachment of the device and itremains compliant with the usual PCI accesses to the legacy IO and memory spaces We proposetwo boards that can bridge the PCI Express protocol over optical fiber The first is a simple opticaltranslator while the second is a more robust switch developed for connecting up to four devices toa single host Such adapters are already working in the control and data acquisition system of aparticle detector at CERN and hence they had been qualified for radiation hardness The positiveoutcomes of the radiation tests of four types of off-the-shelf transceivers are finally reported
KEYWORDS Data acquisition circuits Radiation-hard electronics Detector control systems (de-tector and experiment monitoring and slow-control systems architecture hardware algorithmsdatabases)
1Corresponding author
ccopy 2013 IOP Publishing Ltd and Sissa Medialab srl doi1010881748-0221802C02011
2013 JINST 8 C02011
Contents
1 Introduction 1
2 LINCO boards 221 Optical adapter 322 One-to-four optical switch 4
3 Radiation test 531 Test setup 532 Test results 6
321 Infineon V23848-N305-C56 6322 Intel TXN31115D2 7323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 7
33 PCI Express compatibility 8
4 Conclusion 8
1 Introduction
Over the past decade we have seen a definite stand out of PCI Express [1] as the de-facto standardprotocol for host to peripheral interfaces It allows devices to achieve very high data transfer rateimproving the bandwidth steadily every time a new generation appears on the market Clearlyusing copper cabling the maximum reachable physical distance of a remote IO device decreaseswith the increase of the data rate Thus industry leaders in PCI Express solutions and in fiber opticproducts are moving towards a new off the record standard for the PCI Express physical layer [2]We want to contribute to such research topic carrying on with our preliminary results accomplishedin the context of the LINCO project [3] We developed a bus adapter able to bridge remote buses(gt 100 m) to a single-host computer giving the legacy PCI compatibility to the endpoint device andwithout even the need of a specialized driver Furthermore the adapter was made tolerant to harshenvironmental conditions like strong magnetic fields or radiation fluxes requirements that the dataacquisition of high-energy physics experiments often face
The adoption of the LINCO adapter in several physics experiments like AGATA [4] WArP [5]or CMS at LHC [6] demanded an improvement in the hardware design in order to exploit thefull bandwidth of the high speed PCI Express protocol and in order to achieve full decouplingbetween remote devices and the local hosts keeping the transparency of the link With the aimof facing these two specific requirements we developed two new models of LINCO board thatwill be described in section 2 They were designed around the PCI Express features that highlightthe communication protocol nature of the PCI Express IO bus Indeed the protocol provide forload-store operations between two nodes performed by exchanging framed packets in accordance
ndash 1 ndash
2013 JINST 8 C02011
to a suite of stacked protocol layers taking care of the physical link and transaction issues of thechannel By only replacing the physical layer with an optical one we want to keep the model offield bus control in which a host and a networked peer node exchange software arranged packets toaccess memory and registers of the field bus for IO operation
All the LINCO boards are used in the Detector Control System (DCS) and local Data Ac-quisition System (DAQ) of the muon drift tube chambers employed in the trigger of the CMSexperiment Hence a major consideration in the design of the link is the performance in presenceof radiation The remote part of the link resides in electronic crates that will be exposed to an in-tegrated proton fluence of 5 middot1011 pcm2 (corresponding to a total ionizing dose of about 66 Krad)on the components surface equivalent to over 10 years of LHC running Early performance testsof the candidate link and irradiation studies [3] have been accomplished dividing the board in twodifferent areas one involving the PCI-PCI Express bridge and the clock generation and filteringsystem and the other only the optical transceivers No single event upset (SEU) was observed buta series of micro latch up events in the transceivers electronics became destructive at a total radia-tion dose of 1 krad although the link was based around VCSELs (Vertical Cavity Surface EmittingLasers) that were chosen for their radiation tolerance [7] During the whole life span of CMS ex-periment devices are generally biased during the irradiation periods therefore it is important toexamine the transmission life parameter keeping the circuits biased and working during the fulltest A detailed study of the transceivers performance in presence of proton radiation is thereforesubject of this paper In section 3 four kind of commercial low-cost Small Form Factor Pluggable(SFP) transceivers will be compared Finally an accurate jitter analysis to check the PCI Expressrequirements will validate the effectiveness of the optical link at the Bit Error Ratio (BER) levelsettled by specifications
More precisely in this paper we provide the following novel contributions
bull Manufacturing of a PCI Express optical translator able to exploit the full bandwidth theprotocol can offer
bull Manufacturing of a general purpose one-to-four PCI Express Gen2 optical switch that canbe used as simple fan-out generic data mover or for distributed computing and
bull Improved testing on the proton radiation tolerance of the laser part in order to ensure PCIExpress compatibility up to 5 middot1011 pcm2
2 LINCO boards
The LINCO board comes in three flavors a PMC mezzanine a native PCI Express version and aone-to-four PCI Express switch The first adapter (described in [3]) was centered around a PCI-PCIExpress bridge and it exploited a couple of optical transceivers to transmit the data and the clocksignals To use up the full bandwidth of the link and to be up-to-date with native PCI Expressmotherboards a bridge-less version of the board was manufactured In the next section the boarddescription and results from a bandwidth test are reported In section 22 the last version of LINCOis presented it embeds a PCI Express switch that allows the collection of up to four data lanes andgrants more flexibility to the reference clock configuration
ndash 2 ndash
2013 JINST 8 C02011
(a)
Pc
iEx
1x
Fin
ger
CPLD
Filter
SFP Transceiver
SFP Transceiver
Buffer
PCI Ex x125GTs
Perst
REF CLK
(b)
Figure 1 (a) PCI Express optical adapter (b) block diagram
21 Optical adapter
The PCI Express optical adapter shown in figure 1(a) is a x1 lane PCI Express optical trans-lator with an edge connector that allows the insertion in a standard PC motherboard Two SFPtransceivers are assigned to optically convert the data and clock differential pairs (figure 1(b)) APhase-Locked Loop (PLL) with a small loop bandwidth (ICS8741004AIL) is used to filter the jit-ter of the reference clock and to provide a signal suitable for feeding the related transceiver Thedifferential signal received by such transceiver is routed to an Altera programmable logic device(CPLD) with the purpose of detecting the presence of a remote device if it loopbacks the clocksignal The CPLD is also in charge of taking care of the PCI Express receiver detect mechanismpower cycling the Pericom PI2EQX4401 buffer when it receives a PCI Express reset signal andswitching off only the receiving channel when the remote device is not sending valid data
Special attention was paid to the power supply system as potential power integrity problemsrelated to the high-speed signals (signal rise time is approximately 70 ps) may appear An increaseof jitter whose budget is strictly limited could rise from an excessive power related noise SiWavesoftware by Ansoft Corporation was first used to spot potential power or ground bounce and toanalyze DC voltage distributions and finally to simulate the electromagnetic field on the high-speed traces and to obtain the characteristic S-parameters to be used in a Hspice program
To test the bandwidth limit of the PCI Express protocol transmitted on optical medium wedesigned a DMA engine that feeds a PCI Express core instantiated on an Field ProgrammableGate Array (FPGA) This commercial IP directly drives the Multi-Gigabit Transceivers (MGTs)required for PCI Express communication and implements all the protocol staked layers up to thetransactional one To carry out the test we chose a Xilinx Virtex4 evaluation board with SFPoptical transceivers in order to establish the optical link with our adapter which was plugged in aPCI express slot of a personal computer We used a simple driver charged to initially allocate a200 MByte of non-contiguous main memory and afterwards to request the DMA transfer and tomanage the interrupts notification In this way an user program can ask the driver for a big chunkof the PCI device memory recovering back after the DMA transfer completed a virtual memoryvector pointing on the required data The maximum payload available by the core (128 Byte) wasquite far from the 4 KBytes foreseen by the PCI Express specifications however we could achievea top transfer rate of 200 MBytes per second
ndash 3 ndash
2013 JINST 8 C02011
(a)
MuxDemux PI2PCIE2412
PLX Switch PEX 8609
P
ciEx
4x
Fin
ger
MuxDemux PI2PCIE2412
SFP Transceiver
SFP Transceiver
SFP Transceiver
SFP Transceiver
CLK Fan-out PI6C20400S
CLK Fan-out PI6C20400S
100 MHz Oscillator
FPGA Spartan3AN
EEPROM
Power Manager
POWR1014A
MOSFET
DC-DC DC-DC
DC-DC
PCI Ex x45GTs
REF
CLK
PERST
I2C
Differential
SEL1
SEL0
MOSFET
DC-DC
(b)
Figure 2 (a) One-to-four optical switch (b) block diagram
22 One-to-four optical switch
The card is based on a PLX switch (PEX8609) that has the upstream port (x4 lanes) routed to thePCI Express finger connector (figure 2(a)) The downstream ports can be configured by a smallon board FPGA (Xilinx Spartan 3AN) in two possible modes of operation four one-lane ports orone four-lane port The FPGA is also charged to configure the clock domain and hence the clockpath in one of the following options
bull One clock domain the reference clock of the local PCI Express bus is broadcast to the switchand to two transceivers
bull Two clock domains the reference clock is used only by the upstream port of the switch anda plane 100 MHz clock is broadcast to the downstream ports
The same physical routes towards the optical transceivers can be used by clocks or data lanesdepending on the selected path by the configuration of two multiplexerdemultiplexer (PericomPI2PCIE2412) In figure 2(b) the block diagram of the board is shown
It is worth noting that we can avoid to send the reference clock to the remote devices if theSpread Spectrum Clock (SSC) is not active on the upstream port since the clock can be recoveredfrom the datastream In such case we should not be worried about Electromagnetic Interference(EMI) because there is a reduced need of emissions suppression since the links are optical
Due to the embedded features of the PLX switch the board can be used in several applicationsThe four Gen2 PCI-Express lanes offer an aggregated maximum bandwidth of 20 GTs suitable forhigh throughput DAQ systems Furthermore the integrated DMA engine removes the burden re-sulting from moving data between devices away from the processor The board can fit in distributedcomputing DAQ systems as well a downstream port can be configured as non-transparent in orderto isolate host memory domains by presenting any processor subsytems as a simple endpoint ratherthan a complete memory system
We have performed extensive jitter tests on the receiver side as specified by the second revisionof the PCI Express base specification [9] in order to verify that even with the introduced jitter in
ndash 4 ndash
2013 JINST 8 C02011
(a) (b)
Figure 3 Eye diagram after the optical receiver (a) and after the equalizer (b) at 5 Gbps Time scale 40 ps
the optical conversion the transmission still has a Bit Error Ratio lower than 10minus12 For the testsetups we used two boards plugged into two PCs The first was configured to be connected throughthe upstream port to the root complex of one PC and with the downstream port optically linkedto the upstream port of the second board that was connected to the root complex of the secondPC through a non-transparent port in order to avoid a conflict during the PCI bus enumerationThe optical link was successfully established over 30 meters of multimode optical fibers using two85Gbps SFP transceivers (Finisar FTLF8528P2BNV) qualified for 4x Fibre Channel Figure 3(a)shows the eye diagram measured at the optical receiver (eye opening 042 Unit Interval UI) Atthe end of the lane a redriver (PI2EQX5864C) reshapes the signal removing part of the jitter byapplying a proper equalization The good signal integrity due to the contribution of such device isshown in figure 3(b) The total jitter of 038 UI at a BER level of 10minus12 guarantees that error-freedata is recovered
3 Radiation test
In this section the qualification of the candidate transceivers to a harsh radiation environment isreported All the tested transceivers have a maximum bitrate of 25 Gbps and hence are compatibleonly with PCI Express Gen1 We assume that the radiation exposure affects only the randompart of the jitter while the deterministic part (dominated by the InterSymbol Interference ISI) wasmeasured in laboratory using a PCI Express compliance pattern that ensures a worst-case ISI
We describe the setup of the test its outcomes for different models of transceiver and finallywe perform a specific analysis to understand whether the PCI Express compatibility is guaranteed
31 Test setup
The devices under test were commercial SFP optical transceivers high-performance integratedmodules for bi-directional communication over optical fiber During the beam tests four differentSFP transceivers provided by different vendors were tested Intel TXN31115D2 Infineon V23848-N305-C56 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 All these modelsused a 850 nm VCSEL technology laser emitter except for the JDS Uniphase one which operatesat 1310 nm The irradiation exposure was carried out at the Paul Scherrer Institut in Zurich with63 MeV protons to a maximum total fluence of 5 middot1011 pcm2 corresponding to a total ionizing dose
ndash 5 ndash
2013 JINST 8 C02011
(a) (b)
Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising
of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device
32 Test results
In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below
321 Infineon V23848-N305-C56
The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region
ndash 6 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
PUBLISHED BY IOP PUBLISHING FOR SISSA MEDIALAB
RECEIVED November 15 2012ACCEPTED December 26 2012
PUBLISHED February 5 2013
TOPICAL WORKSHOP ON ELECTRONICS FOR PARTICLE PHYSICS 201217ndash21 SEPTEMBER 2012OXFORD UK
A PCI Express optical link based on low-costtransceivers qualified for radiation hardness
A Triossia1 D Barrientosbcd MBellatob D Bortolatoa R Isocrateb G Rampazzob
and S Venturab
aIstituto Nazionale di Fisica Nucleare ndash Legnaro National LaboratoryViale dellrsquoUniversita 2 35020 Legnaro (PD) Italy
bIstituto Nazionale di Fisica Nucleare ndash PadovaVia Marzolo 8 35131 Padova Italy
cIFIC CSIC-Universitat de Valenciac Catedratico Jose Beltran 2 46980 Paterna Spain
dDepartamento de Ingenierıa Electronica Universitat de ValenciaAvinguda de la Universitat sn 46100 Burjassot Valencia Spain
E-mail atriossipdinfnit
ABSTRACT In this paper we want to demonstrate that an optical physical medium is compatiblewith the second generation of PCI Express The benefit introduced by the optical decoupling ofa PCI Express endpoint is twofold it allows for a geographical detachment of the device and itremains compliant with the usual PCI accesses to the legacy IO and memory spaces We proposetwo boards that can bridge the PCI Express protocol over optical fiber The first is a simple opticaltranslator while the second is a more robust switch developed for connecting up to four devices toa single host Such adapters are already working in the control and data acquisition system of aparticle detector at CERN and hence they had been qualified for radiation hardness The positiveoutcomes of the radiation tests of four types of off-the-shelf transceivers are finally reported
KEYWORDS Data acquisition circuits Radiation-hard electronics Detector control systems (de-tector and experiment monitoring and slow-control systems architecture hardware algorithmsdatabases)
1Corresponding author
ccopy 2013 IOP Publishing Ltd and Sissa Medialab srl doi1010881748-0221802C02011
2013 JINST 8 C02011
Contents
1 Introduction 1
2 LINCO boards 221 Optical adapter 322 One-to-four optical switch 4
3 Radiation test 531 Test setup 532 Test results 6
321 Infineon V23848-N305-C56 6322 Intel TXN31115D2 7323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 7
33 PCI Express compatibility 8
4 Conclusion 8
1 Introduction
Over the past decade we have seen a definite stand out of PCI Express [1] as the de-facto standardprotocol for host to peripheral interfaces It allows devices to achieve very high data transfer rateimproving the bandwidth steadily every time a new generation appears on the market Clearlyusing copper cabling the maximum reachable physical distance of a remote IO device decreaseswith the increase of the data rate Thus industry leaders in PCI Express solutions and in fiber opticproducts are moving towards a new off the record standard for the PCI Express physical layer [2]We want to contribute to such research topic carrying on with our preliminary results accomplishedin the context of the LINCO project [3] We developed a bus adapter able to bridge remote buses(gt 100 m) to a single-host computer giving the legacy PCI compatibility to the endpoint device andwithout even the need of a specialized driver Furthermore the adapter was made tolerant to harshenvironmental conditions like strong magnetic fields or radiation fluxes requirements that the dataacquisition of high-energy physics experiments often face
The adoption of the LINCO adapter in several physics experiments like AGATA [4] WArP [5]or CMS at LHC [6] demanded an improvement in the hardware design in order to exploit thefull bandwidth of the high speed PCI Express protocol and in order to achieve full decouplingbetween remote devices and the local hosts keeping the transparency of the link With the aimof facing these two specific requirements we developed two new models of LINCO board thatwill be described in section 2 They were designed around the PCI Express features that highlightthe communication protocol nature of the PCI Express IO bus Indeed the protocol provide forload-store operations between two nodes performed by exchanging framed packets in accordance
ndash 1 ndash
2013 JINST 8 C02011
to a suite of stacked protocol layers taking care of the physical link and transaction issues of thechannel By only replacing the physical layer with an optical one we want to keep the model offield bus control in which a host and a networked peer node exchange software arranged packets toaccess memory and registers of the field bus for IO operation
All the LINCO boards are used in the Detector Control System (DCS) and local Data Ac-quisition System (DAQ) of the muon drift tube chambers employed in the trigger of the CMSexperiment Hence a major consideration in the design of the link is the performance in presenceof radiation The remote part of the link resides in electronic crates that will be exposed to an in-tegrated proton fluence of 5 middot1011 pcm2 (corresponding to a total ionizing dose of about 66 Krad)on the components surface equivalent to over 10 years of LHC running Early performance testsof the candidate link and irradiation studies [3] have been accomplished dividing the board in twodifferent areas one involving the PCI-PCI Express bridge and the clock generation and filteringsystem and the other only the optical transceivers No single event upset (SEU) was observed buta series of micro latch up events in the transceivers electronics became destructive at a total radia-tion dose of 1 krad although the link was based around VCSELs (Vertical Cavity Surface EmittingLasers) that were chosen for their radiation tolerance [7] During the whole life span of CMS ex-periment devices are generally biased during the irradiation periods therefore it is important toexamine the transmission life parameter keeping the circuits biased and working during the fulltest A detailed study of the transceivers performance in presence of proton radiation is thereforesubject of this paper In section 3 four kind of commercial low-cost Small Form Factor Pluggable(SFP) transceivers will be compared Finally an accurate jitter analysis to check the PCI Expressrequirements will validate the effectiveness of the optical link at the Bit Error Ratio (BER) levelsettled by specifications
More precisely in this paper we provide the following novel contributions
bull Manufacturing of a PCI Express optical translator able to exploit the full bandwidth theprotocol can offer
bull Manufacturing of a general purpose one-to-four PCI Express Gen2 optical switch that canbe used as simple fan-out generic data mover or for distributed computing and
bull Improved testing on the proton radiation tolerance of the laser part in order to ensure PCIExpress compatibility up to 5 middot1011 pcm2
2 LINCO boards
The LINCO board comes in three flavors a PMC mezzanine a native PCI Express version and aone-to-four PCI Express switch The first adapter (described in [3]) was centered around a PCI-PCIExpress bridge and it exploited a couple of optical transceivers to transmit the data and the clocksignals To use up the full bandwidth of the link and to be up-to-date with native PCI Expressmotherboards a bridge-less version of the board was manufactured In the next section the boarddescription and results from a bandwidth test are reported In section 22 the last version of LINCOis presented it embeds a PCI Express switch that allows the collection of up to four data lanes andgrants more flexibility to the reference clock configuration
ndash 2 ndash
2013 JINST 8 C02011
(a)
Pc
iEx
1x
Fin
ger
CPLD
Filter
SFP Transceiver
SFP Transceiver
Buffer
PCI Ex x125GTs
Perst
REF CLK
(b)
Figure 1 (a) PCI Express optical adapter (b) block diagram
21 Optical adapter
The PCI Express optical adapter shown in figure 1(a) is a x1 lane PCI Express optical trans-lator with an edge connector that allows the insertion in a standard PC motherboard Two SFPtransceivers are assigned to optically convert the data and clock differential pairs (figure 1(b)) APhase-Locked Loop (PLL) with a small loop bandwidth (ICS8741004AIL) is used to filter the jit-ter of the reference clock and to provide a signal suitable for feeding the related transceiver Thedifferential signal received by such transceiver is routed to an Altera programmable logic device(CPLD) with the purpose of detecting the presence of a remote device if it loopbacks the clocksignal The CPLD is also in charge of taking care of the PCI Express receiver detect mechanismpower cycling the Pericom PI2EQX4401 buffer when it receives a PCI Express reset signal andswitching off only the receiving channel when the remote device is not sending valid data
Special attention was paid to the power supply system as potential power integrity problemsrelated to the high-speed signals (signal rise time is approximately 70 ps) may appear An increaseof jitter whose budget is strictly limited could rise from an excessive power related noise SiWavesoftware by Ansoft Corporation was first used to spot potential power or ground bounce and toanalyze DC voltage distributions and finally to simulate the electromagnetic field on the high-speed traces and to obtain the characteristic S-parameters to be used in a Hspice program
To test the bandwidth limit of the PCI Express protocol transmitted on optical medium wedesigned a DMA engine that feeds a PCI Express core instantiated on an Field ProgrammableGate Array (FPGA) This commercial IP directly drives the Multi-Gigabit Transceivers (MGTs)required for PCI Express communication and implements all the protocol staked layers up to thetransactional one To carry out the test we chose a Xilinx Virtex4 evaluation board with SFPoptical transceivers in order to establish the optical link with our adapter which was plugged in aPCI express slot of a personal computer We used a simple driver charged to initially allocate a200 MByte of non-contiguous main memory and afterwards to request the DMA transfer and tomanage the interrupts notification In this way an user program can ask the driver for a big chunkof the PCI device memory recovering back after the DMA transfer completed a virtual memoryvector pointing on the required data The maximum payload available by the core (128 Byte) wasquite far from the 4 KBytes foreseen by the PCI Express specifications however we could achievea top transfer rate of 200 MBytes per second
ndash 3 ndash
2013 JINST 8 C02011
(a)
MuxDemux PI2PCIE2412
PLX Switch PEX 8609
P
ciEx
4x
Fin
ger
MuxDemux PI2PCIE2412
SFP Transceiver
SFP Transceiver
SFP Transceiver
SFP Transceiver
CLK Fan-out PI6C20400S
CLK Fan-out PI6C20400S
100 MHz Oscillator
FPGA Spartan3AN
EEPROM
Power Manager
POWR1014A
MOSFET
DC-DC DC-DC
DC-DC
PCI Ex x45GTs
REF
CLK
PERST
I2C
Differential
SEL1
SEL0
MOSFET
DC-DC
(b)
Figure 2 (a) One-to-four optical switch (b) block diagram
22 One-to-four optical switch
The card is based on a PLX switch (PEX8609) that has the upstream port (x4 lanes) routed to thePCI Express finger connector (figure 2(a)) The downstream ports can be configured by a smallon board FPGA (Xilinx Spartan 3AN) in two possible modes of operation four one-lane ports orone four-lane port The FPGA is also charged to configure the clock domain and hence the clockpath in one of the following options
bull One clock domain the reference clock of the local PCI Express bus is broadcast to the switchand to two transceivers
bull Two clock domains the reference clock is used only by the upstream port of the switch anda plane 100 MHz clock is broadcast to the downstream ports
The same physical routes towards the optical transceivers can be used by clocks or data lanesdepending on the selected path by the configuration of two multiplexerdemultiplexer (PericomPI2PCIE2412) In figure 2(b) the block diagram of the board is shown
It is worth noting that we can avoid to send the reference clock to the remote devices if theSpread Spectrum Clock (SSC) is not active on the upstream port since the clock can be recoveredfrom the datastream In such case we should not be worried about Electromagnetic Interference(EMI) because there is a reduced need of emissions suppression since the links are optical
Due to the embedded features of the PLX switch the board can be used in several applicationsThe four Gen2 PCI-Express lanes offer an aggregated maximum bandwidth of 20 GTs suitable forhigh throughput DAQ systems Furthermore the integrated DMA engine removes the burden re-sulting from moving data between devices away from the processor The board can fit in distributedcomputing DAQ systems as well a downstream port can be configured as non-transparent in orderto isolate host memory domains by presenting any processor subsytems as a simple endpoint ratherthan a complete memory system
We have performed extensive jitter tests on the receiver side as specified by the second revisionof the PCI Express base specification [9] in order to verify that even with the introduced jitter in
ndash 4 ndash
2013 JINST 8 C02011
(a) (b)
Figure 3 Eye diagram after the optical receiver (a) and after the equalizer (b) at 5 Gbps Time scale 40 ps
the optical conversion the transmission still has a Bit Error Ratio lower than 10minus12 For the testsetups we used two boards plugged into two PCs The first was configured to be connected throughthe upstream port to the root complex of one PC and with the downstream port optically linkedto the upstream port of the second board that was connected to the root complex of the secondPC through a non-transparent port in order to avoid a conflict during the PCI bus enumerationThe optical link was successfully established over 30 meters of multimode optical fibers using two85Gbps SFP transceivers (Finisar FTLF8528P2BNV) qualified for 4x Fibre Channel Figure 3(a)shows the eye diagram measured at the optical receiver (eye opening 042 Unit Interval UI) Atthe end of the lane a redriver (PI2EQX5864C) reshapes the signal removing part of the jitter byapplying a proper equalization The good signal integrity due to the contribution of such device isshown in figure 3(b) The total jitter of 038 UI at a BER level of 10minus12 guarantees that error-freedata is recovered
3 Radiation test
In this section the qualification of the candidate transceivers to a harsh radiation environment isreported All the tested transceivers have a maximum bitrate of 25 Gbps and hence are compatibleonly with PCI Express Gen1 We assume that the radiation exposure affects only the randompart of the jitter while the deterministic part (dominated by the InterSymbol Interference ISI) wasmeasured in laboratory using a PCI Express compliance pattern that ensures a worst-case ISI
We describe the setup of the test its outcomes for different models of transceiver and finallywe perform a specific analysis to understand whether the PCI Express compatibility is guaranteed
31 Test setup
The devices under test were commercial SFP optical transceivers high-performance integratedmodules for bi-directional communication over optical fiber During the beam tests four differentSFP transceivers provided by different vendors were tested Intel TXN31115D2 Infineon V23848-N305-C56 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 All these modelsused a 850 nm VCSEL technology laser emitter except for the JDS Uniphase one which operatesat 1310 nm The irradiation exposure was carried out at the Paul Scherrer Institut in Zurich with63 MeV protons to a maximum total fluence of 5 middot1011 pcm2 corresponding to a total ionizing dose
ndash 5 ndash
2013 JINST 8 C02011
(a) (b)
Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising
of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device
32 Test results
In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below
321 Infineon V23848-N305-C56
The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region
ndash 6 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
Contents
1 Introduction 1
2 LINCO boards 221 Optical adapter 322 One-to-four optical switch 4
3 Radiation test 531 Test setup 532 Test results 6
321 Infineon V23848-N305-C56 6322 Intel TXN31115D2 7323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 7
33 PCI Express compatibility 8
4 Conclusion 8
1 Introduction
Over the past decade we have seen a definite stand out of PCI Express [1] as the de-facto standardprotocol for host to peripheral interfaces It allows devices to achieve very high data transfer rateimproving the bandwidth steadily every time a new generation appears on the market Clearlyusing copper cabling the maximum reachable physical distance of a remote IO device decreaseswith the increase of the data rate Thus industry leaders in PCI Express solutions and in fiber opticproducts are moving towards a new off the record standard for the PCI Express physical layer [2]We want to contribute to such research topic carrying on with our preliminary results accomplishedin the context of the LINCO project [3] We developed a bus adapter able to bridge remote buses(gt 100 m) to a single-host computer giving the legacy PCI compatibility to the endpoint device andwithout even the need of a specialized driver Furthermore the adapter was made tolerant to harshenvironmental conditions like strong magnetic fields or radiation fluxes requirements that the dataacquisition of high-energy physics experiments often face
The adoption of the LINCO adapter in several physics experiments like AGATA [4] WArP [5]or CMS at LHC [6] demanded an improvement in the hardware design in order to exploit thefull bandwidth of the high speed PCI Express protocol and in order to achieve full decouplingbetween remote devices and the local hosts keeping the transparency of the link With the aimof facing these two specific requirements we developed two new models of LINCO board thatwill be described in section 2 They were designed around the PCI Express features that highlightthe communication protocol nature of the PCI Express IO bus Indeed the protocol provide forload-store operations between two nodes performed by exchanging framed packets in accordance
ndash 1 ndash
2013 JINST 8 C02011
to a suite of stacked protocol layers taking care of the physical link and transaction issues of thechannel By only replacing the physical layer with an optical one we want to keep the model offield bus control in which a host and a networked peer node exchange software arranged packets toaccess memory and registers of the field bus for IO operation
All the LINCO boards are used in the Detector Control System (DCS) and local Data Ac-quisition System (DAQ) of the muon drift tube chambers employed in the trigger of the CMSexperiment Hence a major consideration in the design of the link is the performance in presenceof radiation The remote part of the link resides in electronic crates that will be exposed to an in-tegrated proton fluence of 5 middot1011 pcm2 (corresponding to a total ionizing dose of about 66 Krad)on the components surface equivalent to over 10 years of LHC running Early performance testsof the candidate link and irradiation studies [3] have been accomplished dividing the board in twodifferent areas one involving the PCI-PCI Express bridge and the clock generation and filteringsystem and the other only the optical transceivers No single event upset (SEU) was observed buta series of micro latch up events in the transceivers electronics became destructive at a total radia-tion dose of 1 krad although the link was based around VCSELs (Vertical Cavity Surface EmittingLasers) that were chosen for their radiation tolerance [7] During the whole life span of CMS ex-periment devices are generally biased during the irradiation periods therefore it is important toexamine the transmission life parameter keeping the circuits biased and working during the fulltest A detailed study of the transceivers performance in presence of proton radiation is thereforesubject of this paper In section 3 four kind of commercial low-cost Small Form Factor Pluggable(SFP) transceivers will be compared Finally an accurate jitter analysis to check the PCI Expressrequirements will validate the effectiveness of the optical link at the Bit Error Ratio (BER) levelsettled by specifications
More precisely in this paper we provide the following novel contributions
bull Manufacturing of a PCI Express optical translator able to exploit the full bandwidth theprotocol can offer
bull Manufacturing of a general purpose one-to-four PCI Express Gen2 optical switch that canbe used as simple fan-out generic data mover or for distributed computing and
bull Improved testing on the proton radiation tolerance of the laser part in order to ensure PCIExpress compatibility up to 5 middot1011 pcm2
2 LINCO boards
The LINCO board comes in three flavors a PMC mezzanine a native PCI Express version and aone-to-four PCI Express switch The first adapter (described in [3]) was centered around a PCI-PCIExpress bridge and it exploited a couple of optical transceivers to transmit the data and the clocksignals To use up the full bandwidth of the link and to be up-to-date with native PCI Expressmotherboards a bridge-less version of the board was manufactured In the next section the boarddescription and results from a bandwidth test are reported In section 22 the last version of LINCOis presented it embeds a PCI Express switch that allows the collection of up to four data lanes andgrants more flexibility to the reference clock configuration
ndash 2 ndash
2013 JINST 8 C02011
(a)
Pc
iEx
1x
Fin
ger
CPLD
Filter
SFP Transceiver
SFP Transceiver
Buffer
PCI Ex x125GTs
Perst
REF CLK
(b)
Figure 1 (a) PCI Express optical adapter (b) block diagram
21 Optical adapter
The PCI Express optical adapter shown in figure 1(a) is a x1 lane PCI Express optical trans-lator with an edge connector that allows the insertion in a standard PC motherboard Two SFPtransceivers are assigned to optically convert the data and clock differential pairs (figure 1(b)) APhase-Locked Loop (PLL) with a small loop bandwidth (ICS8741004AIL) is used to filter the jit-ter of the reference clock and to provide a signal suitable for feeding the related transceiver Thedifferential signal received by such transceiver is routed to an Altera programmable logic device(CPLD) with the purpose of detecting the presence of a remote device if it loopbacks the clocksignal The CPLD is also in charge of taking care of the PCI Express receiver detect mechanismpower cycling the Pericom PI2EQX4401 buffer when it receives a PCI Express reset signal andswitching off only the receiving channel when the remote device is not sending valid data
Special attention was paid to the power supply system as potential power integrity problemsrelated to the high-speed signals (signal rise time is approximately 70 ps) may appear An increaseof jitter whose budget is strictly limited could rise from an excessive power related noise SiWavesoftware by Ansoft Corporation was first used to spot potential power or ground bounce and toanalyze DC voltage distributions and finally to simulate the electromagnetic field on the high-speed traces and to obtain the characteristic S-parameters to be used in a Hspice program
To test the bandwidth limit of the PCI Express protocol transmitted on optical medium wedesigned a DMA engine that feeds a PCI Express core instantiated on an Field ProgrammableGate Array (FPGA) This commercial IP directly drives the Multi-Gigabit Transceivers (MGTs)required for PCI Express communication and implements all the protocol staked layers up to thetransactional one To carry out the test we chose a Xilinx Virtex4 evaluation board with SFPoptical transceivers in order to establish the optical link with our adapter which was plugged in aPCI express slot of a personal computer We used a simple driver charged to initially allocate a200 MByte of non-contiguous main memory and afterwards to request the DMA transfer and tomanage the interrupts notification In this way an user program can ask the driver for a big chunkof the PCI device memory recovering back after the DMA transfer completed a virtual memoryvector pointing on the required data The maximum payload available by the core (128 Byte) wasquite far from the 4 KBytes foreseen by the PCI Express specifications however we could achievea top transfer rate of 200 MBytes per second
ndash 3 ndash
2013 JINST 8 C02011
(a)
MuxDemux PI2PCIE2412
PLX Switch PEX 8609
P
ciEx
4x
Fin
ger
MuxDemux PI2PCIE2412
SFP Transceiver
SFP Transceiver
SFP Transceiver
SFP Transceiver
CLK Fan-out PI6C20400S
CLK Fan-out PI6C20400S
100 MHz Oscillator
FPGA Spartan3AN
EEPROM
Power Manager
POWR1014A
MOSFET
DC-DC DC-DC
DC-DC
PCI Ex x45GTs
REF
CLK
PERST
I2C
Differential
SEL1
SEL0
MOSFET
DC-DC
(b)
Figure 2 (a) One-to-four optical switch (b) block diagram
22 One-to-four optical switch
The card is based on a PLX switch (PEX8609) that has the upstream port (x4 lanes) routed to thePCI Express finger connector (figure 2(a)) The downstream ports can be configured by a smallon board FPGA (Xilinx Spartan 3AN) in two possible modes of operation four one-lane ports orone four-lane port The FPGA is also charged to configure the clock domain and hence the clockpath in one of the following options
bull One clock domain the reference clock of the local PCI Express bus is broadcast to the switchand to two transceivers
bull Two clock domains the reference clock is used only by the upstream port of the switch anda plane 100 MHz clock is broadcast to the downstream ports
The same physical routes towards the optical transceivers can be used by clocks or data lanesdepending on the selected path by the configuration of two multiplexerdemultiplexer (PericomPI2PCIE2412) In figure 2(b) the block diagram of the board is shown
It is worth noting that we can avoid to send the reference clock to the remote devices if theSpread Spectrum Clock (SSC) is not active on the upstream port since the clock can be recoveredfrom the datastream In such case we should not be worried about Electromagnetic Interference(EMI) because there is a reduced need of emissions suppression since the links are optical
Due to the embedded features of the PLX switch the board can be used in several applicationsThe four Gen2 PCI-Express lanes offer an aggregated maximum bandwidth of 20 GTs suitable forhigh throughput DAQ systems Furthermore the integrated DMA engine removes the burden re-sulting from moving data between devices away from the processor The board can fit in distributedcomputing DAQ systems as well a downstream port can be configured as non-transparent in orderto isolate host memory domains by presenting any processor subsytems as a simple endpoint ratherthan a complete memory system
We have performed extensive jitter tests on the receiver side as specified by the second revisionof the PCI Express base specification [9] in order to verify that even with the introduced jitter in
ndash 4 ndash
2013 JINST 8 C02011
(a) (b)
Figure 3 Eye diagram after the optical receiver (a) and after the equalizer (b) at 5 Gbps Time scale 40 ps
the optical conversion the transmission still has a Bit Error Ratio lower than 10minus12 For the testsetups we used two boards plugged into two PCs The first was configured to be connected throughthe upstream port to the root complex of one PC and with the downstream port optically linkedto the upstream port of the second board that was connected to the root complex of the secondPC through a non-transparent port in order to avoid a conflict during the PCI bus enumerationThe optical link was successfully established over 30 meters of multimode optical fibers using two85Gbps SFP transceivers (Finisar FTLF8528P2BNV) qualified for 4x Fibre Channel Figure 3(a)shows the eye diagram measured at the optical receiver (eye opening 042 Unit Interval UI) Atthe end of the lane a redriver (PI2EQX5864C) reshapes the signal removing part of the jitter byapplying a proper equalization The good signal integrity due to the contribution of such device isshown in figure 3(b) The total jitter of 038 UI at a BER level of 10minus12 guarantees that error-freedata is recovered
3 Radiation test
In this section the qualification of the candidate transceivers to a harsh radiation environment isreported All the tested transceivers have a maximum bitrate of 25 Gbps and hence are compatibleonly with PCI Express Gen1 We assume that the radiation exposure affects only the randompart of the jitter while the deterministic part (dominated by the InterSymbol Interference ISI) wasmeasured in laboratory using a PCI Express compliance pattern that ensures a worst-case ISI
We describe the setup of the test its outcomes for different models of transceiver and finallywe perform a specific analysis to understand whether the PCI Express compatibility is guaranteed
31 Test setup
The devices under test were commercial SFP optical transceivers high-performance integratedmodules for bi-directional communication over optical fiber During the beam tests four differentSFP transceivers provided by different vendors were tested Intel TXN31115D2 Infineon V23848-N305-C56 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 All these modelsused a 850 nm VCSEL technology laser emitter except for the JDS Uniphase one which operatesat 1310 nm The irradiation exposure was carried out at the Paul Scherrer Institut in Zurich with63 MeV protons to a maximum total fluence of 5 middot1011 pcm2 corresponding to a total ionizing dose
ndash 5 ndash
2013 JINST 8 C02011
(a) (b)
Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising
of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device
32 Test results
In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below
321 Infineon V23848-N305-C56
The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region
ndash 6 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
to a suite of stacked protocol layers taking care of the physical link and transaction issues of thechannel By only replacing the physical layer with an optical one we want to keep the model offield bus control in which a host and a networked peer node exchange software arranged packets toaccess memory and registers of the field bus for IO operation
All the LINCO boards are used in the Detector Control System (DCS) and local Data Ac-quisition System (DAQ) of the muon drift tube chambers employed in the trigger of the CMSexperiment Hence a major consideration in the design of the link is the performance in presenceof radiation The remote part of the link resides in electronic crates that will be exposed to an in-tegrated proton fluence of 5 middot1011 pcm2 (corresponding to a total ionizing dose of about 66 Krad)on the components surface equivalent to over 10 years of LHC running Early performance testsof the candidate link and irradiation studies [3] have been accomplished dividing the board in twodifferent areas one involving the PCI-PCI Express bridge and the clock generation and filteringsystem and the other only the optical transceivers No single event upset (SEU) was observed buta series of micro latch up events in the transceivers electronics became destructive at a total radia-tion dose of 1 krad although the link was based around VCSELs (Vertical Cavity Surface EmittingLasers) that were chosen for their radiation tolerance [7] During the whole life span of CMS ex-periment devices are generally biased during the irradiation periods therefore it is important toexamine the transmission life parameter keeping the circuits biased and working during the fulltest A detailed study of the transceivers performance in presence of proton radiation is thereforesubject of this paper In section 3 four kind of commercial low-cost Small Form Factor Pluggable(SFP) transceivers will be compared Finally an accurate jitter analysis to check the PCI Expressrequirements will validate the effectiveness of the optical link at the Bit Error Ratio (BER) levelsettled by specifications
More precisely in this paper we provide the following novel contributions
bull Manufacturing of a PCI Express optical translator able to exploit the full bandwidth theprotocol can offer
bull Manufacturing of a general purpose one-to-four PCI Express Gen2 optical switch that canbe used as simple fan-out generic data mover or for distributed computing and
bull Improved testing on the proton radiation tolerance of the laser part in order to ensure PCIExpress compatibility up to 5 middot1011 pcm2
2 LINCO boards
The LINCO board comes in three flavors a PMC mezzanine a native PCI Express version and aone-to-four PCI Express switch The first adapter (described in [3]) was centered around a PCI-PCIExpress bridge and it exploited a couple of optical transceivers to transmit the data and the clocksignals To use up the full bandwidth of the link and to be up-to-date with native PCI Expressmotherboards a bridge-less version of the board was manufactured In the next section the boarddescription and results from a bandwidth test are reported In section 22 the last version of LINCOis presented it embeds a PCI Express switch that allows the collection of up to four data lanes andgrants more flexibility to the reference clock configuration
ndash 2 ndash
2013 JINST 8 C02011
(a)
Pc
iEx
1x
Fin
ger
CPLD
Filter
SFP Transceiver
SFP Transceiver
Buffer
PCI Ex x125GTs
Perst
REF CLK
(b)
Figure 1 (a) PCI Express optical adapter (b) block diagram
21 Optical adapter
The PCI Express optical adapter shown in figure 1(a) is a x1 lane PCI Express optical trans-lator with an edge connector that allows the insertion in a standard PC motherboard Two SFPtransceivers are assigned to optically convert the data and clock differential pairs (figure 1(b)) APhase-Locked Loop (PLL) with a small loop bandwidth (ICS8741004AIL) is used to filter the jit-ter of the reference clock and to provide a signal suitable for feeding the related transceiver Thedifferential signal received by such transceiver is routed to an Altera programmable logic device(CPLD) with the purpose of detecting the presence of a remote device if it loopbacks the clocksignal The CPLD is also in charge of taking care of the PCI Express receiver detect mechanismpower cycling the Pericom PI2EQX4401 buffer when it receives a PCI Express reset signal andswitching off only the receiving channel when the remote device is not sending valid data
Special attention was paid to the power supply system as potential power integrity problemsrelated to the high-speed signals (signal rise time is approximately 70 ps) may appear An increaseof jitter whose budget is strictly limited could rise from an excessive power related noise SiWavesoftware by Ansoft Corporation was first used to spot potential power or ground bounce and toanalyze DC voltage distributions and finally to simulate the electromagnetic field on the high-speed traces and to obtain the characteristic S-parameters to be used in a Hspice program
To test the bandwidth limit of the PCI Express protocol transmitted on optical medium wedesigned a DMA engine that feeds a PCI Express core instantiated on an Field ProgrammableGate Array (FPGA) This commercial IP directly drives the Multi-Gigabit Transceivers (MGTs)required for PCI Express communication and implements all the protocol staked layers up to thetransactional one To carry out the test we chose a Xilinx Virtex4 evaluation board with SFPoptical transceivers in order to establish the optical link with our adapter which was plugged in aPCI express slot of a personal computer We used a simple driver charged to initially allocate a200 MByte of non-contiguous main memory and afterwards to request the DMA transfer and tomanage the interrupts notification In this way an user program can ask the driver for a big chunkof the PCI device memory recovering back after the DMA transfer completed a virtual memoryvector pointing on the required data The maximum payload available by the core (128 Byte) wasquite far from the 4 KBytes foreseen by the PCI Express specifications however we could achievea top transfer rate of 200 MBytes per second
ndash 3 ndash
2013 JINST 8 C02011
(a)
MuxDemux PI2PCIE2412
PLX Switch PEX 8609
P
ciEx
4x
Fin
ger
MuxDemux PI2PCIE2412
SFP Transceiver
SFP Transceiver
SFP Transceiver
SFP Transceiver
CLK Fan-out PI6C20400S
CLK Fan-out PI6C20400S
100 MHz Oscillator
FPGA Spartan3AN
EEPROM
Power Manager
POWR1014A
MOSFET
DC-DC DC-DC
DC-DC
PCI Ex x45GTs
REF
CLK
PERST
I2C
Differential
SEL1
SEL0
MOSFET
DC-DC
(b)
Figure 2 (a) One-to-four optical switch (b) block diagram
22 One-to-four optical switch
The card is based on a PLX switch (PEX8609) that has the upstream port (x4 lanes) routed to thePCI Express finger connector (figure 2(a)) The downstream ports can be configured by a smallon board FPGA (Xilinx Spartan 3AN) in two possible modes of operation four one-lane ports orone four-lane port The FPGA is also charged to configure the clock domain and hence the clockpath in one of the following options
bull One clock domain the reference clock of the local PCI Express bus is broadcast to the switchand to two transceivers
bull Two clock domains the reference clock is used only by the upstream port of the switch anda plane 100 MHz clock is broadcast to the downstream ports
The same physical routes towards the optical transceivers can be used by clocks or data lanesdepending on the selected path by the configuration of two multiplexerdemultiplexer (PericomPI2PCIE2412) In figure 2(b) the block diagram of the board is shown
It is worth noting that we can avoid to send the reference clock to the remote devices if theSpread Spectrum Clock (SSC) is not active on the upstream port since the clock can be recoveredfrom the datastream In such case we should not be worried about Electromagnetic Interference(EMI) because there is a reduced need of emissions suppression since the links are optical
Due to the embedded features of the PLX switch the board can be used in several applicationsThe four Gen2 PCI-Express lanes offer an aggregated maximum bandwidth of 20 GTs suitable forhigh throughput DAQ systems Furthermore the integrated DMA engine removes the burden re-sulting from moving data between devices away from the processor The board can fit in distributedcomputing DAQ systems as well a downstream port can be configured as non-transparent in orderto isolate host memory domains by presenting any processor subsytems as a simple endpoint ratherthan a complete memory system
We have performed extensive jitter tests on the receiver side as specified by the second revisionof the PCI Express base specification [9] in order to verify that even with the introduced jitter in
ndash 4 ndash
2013 JINST 8 C02011
(a) (b)
Figure 3 Eye diagram after the optical receiver (a) and after the equalizer (b) at 5 Gbps Time scale 40 ps
the optical conversion the transmission still has a Bit Error Ratio lower than 10minus12 For the testsetups we used two boards plugged into two PCs The first was configured to be connected throughthe upstream port to the root complex of one PC and with the downstream port optically linkedto the upstream port of the second board that was connected to the root complex of the secondPC through a non-transparent port in order to avoid a conflict during the PCI bus enumerationThe optical link was successfully established over 30 meters of multimode optical fibers using two85Gbps SFP transceivers (Finisar FTLF8528P2BNV) qualified for 4x Fibre Channel Figure 3(a)shows the eye diagram measured at the optical receiver (eye opening 042 Unit Interval UI) Atthe end of the lane a redriver (PI2EQX5864C) reshapes the signal removing part of the jitter byapplying a proper equalization The good signal integrity due to the contribution of such device isshown in figure 3(b) The total jitter of 038 UI at a BER level of 10minus12 guarantees that error-freedata is recovered
3 Radiation test
In this section the qualification of the candidate transceivers to a harsh radiation environment isreported All the tested transceivers have a maximum bitrate of 25 Gbps and hence are compatibleonly with PCI Express Gen1 We assume that the radiation exposure affects only the randompart of the jitter while the deterministic part (dominated by the InterSymbol Interference ISI) wasmeasured in laboratory using a PCI Express compliance pattern that ensures a worst-case ISI
We describe the setup of the test its outcomes for different models of transceiver and finallywe perform a specific analysis to understand whether the PCI Express compatibility is guaranteed
31 Test setup
The devices under test were commercial SFP optical transceivers high-performance integratedmodules for bi-directional communication over optical fiber During the beam tests four differentSFP transceivers provided by different vendors were tested Intel TXN31115D2 Infineon V23848-N305-C56 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 All these modelsused a 850 nm VCSEL technology laser emitter except for the JDS Uniphase one which operatesat 1310 nm The irradiation exposure was carried out at the Paul Scherrer Institut in Zurich with63 MeV protons to a maximum total fluence of 5 middot1011 pcm2 corresponding to a total ionizing dose
ndash 5 ndash
2013 JINST 8 C02011
(a) (b)
Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising
of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device
32 Test results
In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below
321 Infineon V23848-N305-C56
The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region
ndash 6 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
(a)
Pc
iEx
1x
Fin
ger
CPLD
Filter
SFP Transceiver
SFP Transceiver
Buffer
PCI Ex x125GTs
Perst
REF CLK
(b)
Figure 1 (a) PCI Express optical adapter (b) block diagram
21 Optical adapter
The PCI Express optical adapter shown in figure 1(a) is a x1 lane PCI Express optical trans-lator with an edge connector that allows the insertion in a standard PC motherboard Two SFPtransceivers are assigned to optically convert the data and clock differential pairs (figure 1(b)) APhase-Locked Loop (PLL) with a small loop bandwidth (ICS8741004AIL) is used to filter the jit-ter of the reference clock and to provide a signal suitable for feeding the related transceiver Thedifferential signal received by such transceiver is routed to an Altera programmable logic device(CPLD) with the purpose of detecting the presence of a remote device if it loopbacks the clocksignal The CPLD is also in charge of taking care of the PCI Express receiver detect mechanismpower cycling the Pericom PI2EQX4401 buffer when it receives a PCI Express reset signal andswitching off only the receiving channel when the remote device is not sending valid data
Special attention was paid to the power supply system as potential power integrity problemsrelated to the high-speed signals (signal rise time is approximately 70 ps) may appear An increaseof jitter whose budget is strictly limited could rise from an excessive power related noise SiWavesoftware by Ansoft Corporation was first used to spot potential power or ground bounce and toanalyze DC voltage distributions and finally to simulate the electromagnetic field on the high-speed traces and to obtain the characteristic S-parameters to be used in a Hspice program
To test the bandwidth limit of the PCI Express protocol transmitted on optical medium wedesigned a DMA engine that feeds a PCI Express core instantiated on an Field ProgrammableGate Array (FPGA) This commercial IP directly drives the Multi-Gigabit Transceivers (MGTs)required for PCI Express communication and implements all the protocol staked layers up to thetransactional one To carry out the test we chose a Xilinx Virtex4 evaluation board with SFPoptical transceivers in order to establish the optical link with our adapter which was plugged in aPCI express slot of a personal computer We used a simple driver charged to initially allocate a200 MByte of non-contiguous main memory and afterwards to request the DMA transfer and tomanage the interrupts notification In this way an user program can ask the driver for a big chunkof the PCI device memory recovering back after the DMA transfer completed a virtual memoryvector pointing on the required data The maximum payload available by the core (128 Byte) wasquite far from the 4 KBytes foreseen by the PCI Express specifications however we could achievea top transfer rate of 200 MBytes per second
ndash 3 ndash
2013 JINST 8 C02011
(a)
MuxDemux PI2PCIE2412
PLX Switch PEX 8609
P
ciEx
4x
Fin
ger
MuxDemux PI2PCIE2412
SFP Transceiver
SFP Transceiver
SFP Transceiver
SFP Transceiver
CLK Fan-out PI6C20400S
CLK Fan-out PI6C20400S
100 MHz Oscillator
FPGA Spartan3AN
EEPROM
Power Manager
POWR1014A
MOSFET
DC-DC DC-DC
DC-DC
PCI Ex x45GTs
REF
CLK
PERST
I2C
Differential
SEL1
SEL0
MOSFET
DC-DC
(b)
Figure 2 (a) One-to-four optical switch (b) block diagram
22 One-to-four optical switch
The card is based on a PLX switch (PEX8609) that has the upstream port (x4 lanes) routed to thePCI Express finger connector (figure 2(a)) The downstream ports can be configured by a smallon board FPGA (Xilinx Spartan 3AN) in two possible modes of operation four one-lane ports orone four-lane port The FPGA is also charged to configure the clock domain and hence the clockpath in one of the following options
bull One clock domain the reference clock of the local PCI Express bus is broadcast to the switchand to two transceivers
bull Two clock domains the reference clock is used only by the upstream port of the switch anda plane 100 MHz clock is broadcast to the downstream ports
The same physical routes towards the optical transceivers can be used by clocks or data lanesdepending on the selected path by the configuration of two multiplexerdemultiplexer (PericomPI2PCIE2412) In figure 2(b) the block diagram of the board is shown
It is worth noting that we can avoid to send the reference clock to the remote devices if theSpread Spectrum Clock (SSC) is not active on the upstream port since the clock can be recoveredfrom the datastream In such case we should not be worried about Electromagnetic Interference(EMI) because there is a reduced need of emissions suppression since the links are optical
Due to the embedded features of the PLX switch the board can be used in several applicationsThe four Gen2 PCI-Express lanes offer an aggregated maximum bandwidth of 20 GTs suitable forhigh throughput DAQ systems Furthermore the integrated DMA engine removes the burden re-sulting from moving data between devices away from the processor The board can fit in distributedcomputing DAQ systems as well a downstream port can be configured as non-transparent in orderto isolate host memory domains by presenting any processor subsytems as a simple endpoint ratherthan a complete memory system
We have performed extensive jitter tests on the receiver side as specified by the second revisionof the PCI Express base specification [9] in order to verify that even with the introduced jitter in
ndash 4 ndash
2013 JINST 8 C02011
(a) (b)
Figure 3 Eye diagram after the optical receiver (a) and after the equalizer (b) at 5 Gbps Time scale 40 ps
the optical conversion the transmission still has a Bit Error Ratio lower than 10minus12 For the testsetups we used two boards plugged into two PCs The first was configured to be connected throughthe upstream port to the root complex of one PC and with the downstream port optically linkedto the upstream port of the second board that was connected to the root complex of the secondPC through a non-transparent port in order to avoid a conflict during the PCI bus enumerationThe optical link was successfully established over 30 meters of multimode optical fibers using two85Gbps SFP transceivers (Finisar FTLF8528P2BNV) qualified for 4x Fibre Channel Figure 3(a)shows the eye diagram measured at the optical receiver (eye opening 042 Unit Interval UI) Atthe end of the lane a redriver (PI2EQX5864C) reshapes the signal removing part of the jitter byapplying a proper equalization The good signal integrity due to the contribution of such device isshown in figure 3(b) The total jitter of 038 UI at a BER level of 10minus12 guarantees that error-freedata is recovered
3 Radiation test
In this section the qualification of the candidate transceivers to a harsh radiation environment isreported All the tested transceivers have a maximum bitrate of 25 Gbps and hence are compatibleonly with PCI Express Gen1 We assume that the radiation exposure affects only the randompart of the jitter while the deterministic part (dominated by the InterSymbol Interference ISI) wasmeasured in laboratory using a PCI Express compliance pattern that ensures a worst-case ISI
We describe the setup of the test its outcomes for different models of transceiver and finallywe perform a specific analysis to understand whether the PCI Express compatibility is guaranteed
31 Test setup
The devices under test were commercial SFP optical transceivers high-performance integratedmodules for bi-directional communication over optical fiber During the beam tests four differentSFP transceivers provided by different vendors were tested Intel TXN31115D2 Infineon V23848-N305-C56 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 All these modelsused a 850 nm VCSEL technology laser emitter except for the JDS Uniphase one which operatesat 1310 nm The irradiation exposure was carried out at the Paul Scherrer Institut in Zurich with63 MeV protons to a maximum total fluence of 5 middot1011 pcm2 corresponding to a total ionizing dose
ndash 5 ndash
2013 JINST 8 C02011
(a) (b)
Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising
of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device
32 Test results
In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below
321 Infineon V23848-N305-C56
The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region
ndash 6 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
(a)
MuxDemux PI2PCIE2412
PLX Switch PEX 8609
P
ciEx
4x
Fin
ger
MuxDemux PI2PCIE2412
SFP Transceiver
SFP Transceiver
SFP Transceiver
SFP Transceiver
CLK Fan-out PI6C20400S
CLK Fan-out PI6C20400S
100 MHz Oscillator
FPGA Spartan3AN
EEPROM
Power Manager
POWR1014A
MOSFET
DC-DC DC-DC
DC-DC
PCI Ex x45GTs
REF
CLK
PERST
I2C
Differential
SEL1
SEL0
MOSFET
DC-DC
(b)
Figure 2 (a) One-to-four optical switch (b) block diagram
22 One-to-four optical switch
The card is based on a PLX switch (PEX8609) that has the upstream port (x4 lanes) routed to thePCI Express finger connector (figure 2(a)) The downstream ports can be configured by a smallon board FPGA (Xilinx Spartan 3AN) in two possible modes of operation four one-lane ports orone four-lane port The FPGA is also charged to configure the clock domain and hence the clockpath in one of the following options
bull One clock domain the reference clock of the local PCI Express bus is broadcast to the switchand to two transceivers
bull Two clock domains the reference clock is used only by the upstream port of the switch anda plane 100 MHz clock is broadcast to the downstream ports
The same physical routes towards the optical transceivers can be used by clocks or data lanesdepending on the selected path by the configuration of two multiplexerdemultiplexer (PericomPI2PCIE2412) In figure 2(b) the block diagram of the board is shown
It is worth noting that we can avoid to send the reference clock to the remote devices if theSpread Spectrum Clock (SSC) is not active on the upstream port since the clock can be recoveredfrom the datastream In such case we should not be worried about Electromagnetic Interference(EMI) because there is a reduced need of emissions suppression since the links are optical
Due to the embedded features of the PLX switch the board can be used in several applicationsThe four Gen2 PCI-Express lanes offer an aggregated maximum bandwidth of 20 GTs suitable forhigh throughput DAQ systems Furthermore the integrated DMA engine removes the burden re-sulting from moving data between devices away from the processor The board can fit in distributedcomputing DAQ systems as well a downstream port can be configured as non-transparent in orderto isolate host memory domains by presenting any processor subsytems as a simple endpoint ratherthan a complete memory system
We have performed extensive jitter tests on the receiver side as specified by the second revisionof the PCI Express base specification [9] in order to verify that even with the introduced jitter in
ndash 4 ndash
2013 JINST 8 C02011
(a) (b)
Figure 3 Eye diagram after the optical receiver (a) and after the equalizer (b) at 5 Gbps Time scale 40 ps
the optical conversion the transmission still has a Bit Error Ratio lower than 10minus12 For the testsetups we used two boards plugged into two PCs The first was configured to be connected throughthe upstream port to the root complex of one PC and with the downstream port optically linkedto the upstream port of the second board that was connected to the root complex of the secondPC through a non-transparent port in order to avoid a conflict during the PCI bus enumerationThe optical link was successfully established over 30 meters of multimode optical fibers using two85Gbps SFP transceivers (Finisar FTLF8528P2BNV) qualified for 4x Fibre Channel Figure 3(a)shows the eye diagram measured at the optical receiver (eye opening 042 Unit Interval UI) Atthe end of the lane a redriver (PI2EQX5864C) reshapes the signal removing part of the jitter byapplying a proper equalization The good signal integrity due to the contribution of such device isshown in figure 3(b) The total jitter of 038 UI at a BER level of 10minus12 guarantees that error-freedata is recovered
3 Radiation test
In this section the qualification of the candidate transceivers to a harsh radiation environment isreported All the tested transceivers have a maximum bitrate of 25 Gbps and hence are compatibleonly with PCI Express Gen1 We assume that the radiation exposure affects only the randompart of the jitter while the deterministic part (dominated by the InterSymbol Interference ISI) wasmeasured in laboratory using a PCI Express compliance pattern that ensures a worst-case ISI
We describe the setup of the test its outcomes for different models of transceiver and finallywe perform a specific analysis to understand whether the PCI Express compatibility is guaranteed
31 Test setup
The devices under test were commercial SFP optical transceivers high-performance integratedmodules for bi-directional communication over optical fiber During the beam tests four differentSFP transceivers provided by different vendors were tested Intel TXN31115D2 Infineon V23848-N305-C56 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 All these modelsused a 850 nm VCSEL technology laser emitter except for the JDS Uniphase one which operatesat 1310 nm The irradiation exposure was carried out at the Paul Scherrer Institut in Zurich with63 MeV protons to a maximum total fluence of 5 middot1011 pcm2 corresponding to a total ionizing dose
ndash 5 ndash
2013 JINST 8 C02011
(a) (b)
Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising
of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device
32 Test results
In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below
321 Infineon V23848-N305-C56
The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region
ndash 6 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
(a) (b)
Figure 3 Eye diagram after the optical receiver (a) and after the equalizer (b) at 5 Gbps Time scale 40 ps
the optical conversion the transmission still has a Bit Error Ratio lower than 10minus12 For the testsetups we used two boards plugged into two PCs The first was configured to be connected throughthe upstream port to the root complex of one PC and with the downstream port optically linkedto the upstream port of the second board that was connected to the root complex of the secondPC through a non-transparent port in order to avoid a conflict during the PCI bus enumerationThe optical link was successfully established over 30 meters of multimode optical fibers using two85Gbps SFP transceivers (Finisar FTLF8528P2BNV) qualified for 4x Fibre Channel Figure 3(a)shows the eye diagram measured at the optical receiver (eye opening 042 Unit Interval UI) Atthe end of the lane a redriver (PI2EQX5864C) reshapes the signal removing part of the jitter byapplying a proper equalization The good signal integrity due to the contribution of such device isshown in figure 3(b) The total jitter of 038 UI at a BER level of 10minus12 guarantees that error-freedata is recovered
3 Radiation test
In this section the qualification of the candidate transceivers to a harsh radiation environment isreported All the tested transceivers have a maximum bitrate of 25 Gbps and hence are compatibleonly with PCI Express Gen1 We assume that the radiation exposure affects only the randompart of the jitter while the deterministic part (dominated by the InterSymbol Interference ISI) wasmeasured in laboratory using a PCI Express compliance pattern that ensures a worst-case ISI
We describe the setup of the test its outcomes for different models of transceiver and finallywe perform a specific analysis to understand whether the PCI Express compatibility is guaranteed
31 Test setup
The devices under test were commercial SFP optical transceivers high-performance integratedmodules for bi-directional communication over optical fiber During the beam tests four differentSFP transceivers provided by different vendors were tested Intel TXN31115D2 Infineon V23848-N305-C56 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2 All these modelsused a 850 nm VCSEL technology laser emitter except for the JDS Uniphase one which operatesat 1310 nm The irradiation exposure was carried out at the Paul Scherrer Institut in Zurich with63 MeV protons to a maximum total fluence of 5 middot1011 pcm2 corresponding to a total ionizing dose
ndash 5 ndash
2013 JINST 8 C02011
(a) (b)
Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising
of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device
32 Test results
In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below
321 Infineon V23848-N305-C56
The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region
ndash 6 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
(a) (b)
Figure 4 (a) Infineon transmitter A) Eye diagram before irradiation B) after 32 Krad C) after 43 KradD) after 66 Krad (b) V23848-N305-C56 TIE and σ trend during total dose rising
of about 66 Krad The average flux during irradiation varied between 08 middot108 and 28 middot108 pcm2sAll irradiations and measurements were performed at room temperature The transceivers werelodged in a dual optical module by Memec Design in order to irradiate the transmitter or the receiverof each transceiver separately We used a 4 cm deep plate of aluminium to shield only one of thetwo transceivers A dual output power supply was used to monitor the absorbed current fluctuationof the two transceivers independently During the tests a 0101 pattern at 25 Gbps was sent toone of the optical transceivers by a very low jitter data pattern generator and sent back by the othertransceiver to a 6 GHz bandwidth scope To cover the distance between the experimental area andthe control room we used two 30 m multimode optical fibers The electrical to optical and opticalto electrical conversions of the data signal were performed by another couple of SFP transceiversplaced nearby the data generator and the scope in the control room The dual optical module wasset on an acrylic glass frame aligned to the beam line so the beam was perpendicular to the planeof the SFP device
32 Test results
In this section we summarize the effects of irradiation on the different components During alltests two main parameters were observed the absorbed current and the total jitter The former waslogged every second while the latter was recorded twice a minute by direct measure of time andvoltage margins of the data signal (eye diagram) and by a histogram of the jitter distribution Theoutcomes are reported below
321 Infineon V23848-N305-C56
The current absorbed by both transmitter and receiver during their respective radiation sectionsfollowed the same trend it was constant until some micro latch up events occurred and then thecurrent increased linearly More interesting is the output jitter behavior presented in figure 4(a)It shows the eye diagram before the beam exposure and its closure observed during the Infineontransmitter irradiation tests at three different steps of total dose absorbed In figure 4(b) is plottedthe peak-to-peak amplitude and standard deviation of the Time Interval Error (TIE) which is thetime difference between the recovered clock from the data stream and the data signal in terms ofthe total radiation dose before micro latch up events occur An exponential fit on the rising region
ndash 6 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
(a) (b)
Figure 5 (a) TXN31115D2 receiver eye diagram at 25 Gbps (b) BER functions at different radiationdose
shows a good correlation (the correlation coefficient is R=099) for both parameters Systematicerrors did not affect the jitter during the whole radiation exposition the process kept always a zeromean normal distribution The rise of the standard deviation reflects the increase of the randomjitter until it saturates Micro latch up events in the modulation control unit of the laser driver maybe the cause of such a jitter trend
The receiver radiation session reveals a different jitter behavior instead of a progressivegrowth we can find some bits with a crossing point greatly translated respect of their expectedposition like in figure 5(a) These isolated events (which diverge more than 7σ from the mean ofthe distribution) could be considered as failures Knowing the hadrons flux in the CMS environ-ment [8] we can estimate the Mean Time Before Failure (MTBF) as 106 years The maximumjitter collected is however very low reaching only a peak of 27 ps
322 Intel TXN31115D2
We did not observe any effect during the whole transmitter radiation exposition A fluence of 5 middot1011 pcm2 was achieved corresponding to more than 40 years of operation in the CMS experimentThe receiver MTBF is 184 years with a maximum gap of 17 ps (figure 5(a)) The better resultto proton radiation is justified by the different laser driver present on the device It is the onlytransceiver within the four we tested without a sophisticated modulation current logic used forcompensating the temperature characteristic of laser diode slope efficiency
323 Finisar FTLF8524P2WNL and JDS Uniphase CT2-MS1LBTD32C2
These two transceivers are related because of their similar behaviors Differently from the otherdevices mentioned above one can see a sudden and complete eye closure due to the shutdownof the laser The fast decrease of the absorbed current observed at the same time agrees withthe hypothesis of the disabled laser This phenomenon belongs to the class of Single Event Effects(SEE) not depending from the total dose absorbed but from an error introduced by charged particles(protons) losing energy by ionizing the medium through which they pass The non-destructive
ndash 7 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
nature of this effect is clear when the power cycle of the transceiver makes the laser begin to workagain The observed mean fluence between failures (on three consecutive measurements) is 6 middot1010
and 125 middot1011 pcm2 for the Finisar and the JDS Uniphase transceivers respectivelyThe transceiver feature that can be responsible of the turning off of the laser has to be charged
to their laser driver enabling mechanism In fact all the tested transceivers have got a microcon-troller which is very sensitive to SEE since it embeds breakable components like RAM but onlythe Finisar and JDS Uniphase ones use it to enable the laser driver In such devices a Single EventUpset (SEU) in the microcontroller could forbid the laser emission
33 PCI Express compatibility
Since we need a strict PCI express compatibility of the hardware some compliance tests were car-ried out in order to measure the jitter introduced by the transceivers into the opto-electrical channelThe measure of the eye opening at a Bit Error Ratio (BER) of 10minus12 reveals that all the transceiversproposed above are compliant with the PCI Express Base Specification [9] A minimum eye open-ing at the receiver of 04 UI is allowed and taking into account that a maximum jitter of 00575UI could be piled up between the transceiver and the PCI Express device the total eye opening atBER level of 10minus12 cannot be lower than 04575 UI corresponding to 183 ps A measurement ofthis parameter for the Infineon V23848-N305-C56 transceiver during proton radiation was neededand it was performed by means of the dual-Dirac model and the tail fit method [10] For evaluatingthe deterministic part of the jitter Probability Density Function (PDF) that corresponds to the dis-tance between the two Dirac distributions we measure the average of the time interval error trend(378 ps) during the transmission of the PCI Express compliance pattern The BER function is ob-tained as the cumulative distribution function of the two timing total jitter PDFs (PDFT ) related tothe two crossing point of an eye diagram
(BER(t))i = ρT
(int +infin
t(PDFT (τ))i dτ
prime+int t
minusinfin
(PDFT (τ))i dτprime)
(31)
Where the index i denote the radiation process progress and the parameter ρT is the transitiondensity the ratio between the number of transitional bits and the total number of transmitted bits
In figure 5(b) a bunch of BER bathtub functions are plotted at different radiation dose val-ues We have deduced the real eye opening as the time difference between the curves branchescorresponding to a 10minus12 BER level (blue surface) From this extrapolation it turns out that theInfineon V23848-N305-C56 transceiver is not suitable for PCI Express protocol transmission aftera radiation dose of more than 32 middot1011 pcm2 (27 years of operation at LHC)
4 Conclusion
The compatibility of an optical medium as physical layer for the second generation of the PCIExpress protocol gave us the possibility of developing two boards that act as optical translatorsExtensive tests were carried out in order to deploy them in the DAQ system of nuclear and particlephysics experiments The flexibility in lane distribution among remote devices and in the clocktransmission scheme makes such cards suitable for a wide set of domestic and industrial applica-tions Indeed they can allow users to employ the optical detachment of a PCI Express endpoints
ndash 8 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash
2013 JINST 8 C02011
for instance as a memorydisk system interconnection a high-end audiovideo application a highperformance computing or multi-chassis system interconnections
A 25 Gbps demonstrator link based on PCI Express over optical medium has been success-fully achieved in a harsh proton environment We found very different radiation characteristics forthe commercial optical transceivers tested both in the transmitter and receiver channel Only oneout of four tested devices can be used in the context of the CMS experiment The radiation tol-erance of the Intel transceiver has been demonstrated under proton radiation up to 5 middot 1011 pcm2Although transient errors not critical for the protocol were observed during radiation exposure inthe receiver channel
In order to guarantee maintainability of the optical PCI Express links present in the CMS ex-periment future radiation tests should address the radiation hardness of faster transceivers nowa-days more easily available on the market
References
[1] PCI-SIG Pci Express Base Specification 10 (2003) wwwpcisigcom
[2] PLX Technology and Avago Technologies A Demonstration of PCI Express Generation 3 over aFiber Optical Link wwwavagotechcomdocsAV02-3245EN white paper
[3] M Bellato et al Remoting Field Bus Control by Means of a PCI Express-based Optical Serial LinkNucl Instrum Meth A 570 (2007) 518
[4] S Akkoyun et al AGATA mdash Advanced Gamma Tracking Array Nucl Instrum Meth A 668 (2012)26 [arXiv11115731]
[5] R Brunetti et al WARP liquid argon detector for dark matter survey Nucl Instrum Meth A 49(2005) 265 [astro-ph0405342]
[6] CMS collaboration The CMS experiment at the CERN LHC 2008 JINST 3 S08004
[7] ML Chu Radiation hardness studies of VCSELs and PINs for the opto-links of the AtlasSemiConductor Tracker Nucl Instrum Meth A 579 (2007) 795
[8] M Huhtinen Radiation Environment Simulations for the CMS Detector technical report (1995)
[9] PCI-SIG Pci Express Base Specification 20 (2007) wwwpcisigcom
[10] Agilent Technology Jitter Analysis The Dual-Dirac Model RJDJ and Q-Scalehttpcpliteratureagilentcomlitwebpdf5989-3206ENpdf white paper
ndash 9 ndash