Optical PCB Overview - IBM WWW Page · Optical PCB Overview Frank Libsch IBM T.J. Watson Research Center Yorktown Heights, NY. ... Optical printed circuit board technology based on

IBM Research

IBM Internal November 16, 2011 © 2009 IBM Corporation

Optical PCB Overview

Frank LibschIBM T.J. Watson Research CenterYorktown Heights, NY

2 November 2011IBM INTERNAL ONLY © 2009 IBM Corporation

Outline

?System view of why optics is needed

?Potential OPCB Technologies for Next Generation HPCs

– OSA Designs

– OSA Assemblies

– Packaging

– Optical Component Characterization


Optimized solutions will require detailed analysis of trade-offs

Power

CostDensity

Margin, Packaging Integration, Data-rate…

Packaging Integration, Channel Integration, Margin, Cooling, Data-rate…

Base Manufacturing Cost, Yield, Channel Integration, Data-rate, Reliability…


Image courtesy of the National Center for Computational Sciences, Oak Ridge National Laboratory

Why High Performance Computing?

• Materials Science• Geophysical Data Processing• Environment and Climate Modeling• Life Sciences / Drug Discovery• Fluid Dynamics / Energy• Industrial Modeling• Financial Modeling• Transportation

Larger scale, more complex, higher resolution, multiscale PhysicsShorter time to solution ? real time response

More than 50% of Top500 systems are for industry*Growing number of flops are industry (~15% in 1990s to ~30% today)*

*www.top500.orgCourtesy L. Treinish, IBM


Chip ~50% ? 30% CAGR*

Maintaining the HPC Performance Trend

1.E-01

1.E+00

1.E+01

1.E+02

1.E+03

1.E+04

1.E+05

1.E+06

1.E+07

1.E+08

1.E+09

1.E+10

1.E+11N

ov-9

2

Nov

-94

Nov

-96

Nov

-98

Nov

-00

Nov

-02

Nov

-04

Nov

-06

Nov

-08

Nov

-10

Nov

-12

Nov

-14

Nov

-16

Nov

-18

Nov

-20

Time

Per

form

ance

(Gig

aflo

ps)

Semiconductors & Pkg:~15-20% CAGR, slowing

Systems:85-90% CAGR,continuing**

Accelerators

Increasing Parallelism: ? System BW at all levels of assembly must scale exponentially

– ~(0.1-1 Byte/Flop)? AND/OR new architectures, topologies and algorithms required

* http://www.bigncomputing.org**chart data from www.top500.org

1st500th

Total

10x /3.5-4yrs = 85-90% CAGR

Increasing Parallelism1PF

1EF


Cost and power of a supercomputer

YearPeak

PerformanceMachine

CostTotal Power

Consumption

2008 1PF $150M 2.5MW

2012 10PF $225M 5MW

2016 100PF $340M 10MW

2020 1000PF(1EF)

$500M 20MW

?Assumptions: Based on typical industry trends –(See, e.g., top500.org and green500.org)

– 10X performance / 4yrs (from top500 chart)– 10X performance costs 1.5X more– 10X performance consumes 2X more power

J. Kash Photonics Society Annual Meeting Nov 2010 and OWQ1 OFC 2011


Evolution of Optical interconnects

WAN, MANmetro,long-haul

LANcampus, enterprise

Systemintra/inter-rack

Boardmodule-module

Modulechip-chip

Chipon-chip buses

1980’s 1990’s 2000’s

Time of Commercial Deployment (Copper Displacement):

Distance Multi-km 100’s m 10’s m < 1 m < 10 cm < 20 mm

> 2012

Increasing integration of Optics with decreasing cost, decreasing power, increasing density

TelecomDatacom

Computer -com

cards Card edge Card edge/on card

Module Si C or chipIntegration

BW * Distance: Optics >> Copper

On chip


Why Optics?

LGA Socket

• Electrical Buses become increasingly difficult at high data rates (physics):• Increasing losses & cross-talk • Frequency resonant affects

• Optical data transmission is easier: • Much lower loss, esp. at higher data rates• Additional advantages include:

• Cable bulk, connector size, EMI…• Potential power savings

• KEY ADVANTAGE: BW * Distance > electrical• Optics trends for large servers and data centers are

following same trends as telecom network:• Longer links first• Short link optics requires tighter package integration:

• Higher BW closer to signal source• Changes the supplier/manufacturing paradigm

Card

Card

CPU

Bac

kpla

ne

Cross talk Freq dependent losses

Resonance effects

Copper

Optics

9 November 2011IBM INTERNAL ONLY © 2009 IBM Corporation9

Bandwidth: the Bane of the Multicore Paradigm:

?Logic flops continue to scale faster than interconnect BW

• Constant Byte/Flop ratio with N cores (constant ?) means:Bandwidth(N-core) = N x Bandwidth(single core)

• 3Di (3D integration) will only exacerbate bottlenecks

Assumptions:• 3 GHz clock• ~ 3 IPC• 10 Gb/s I/O

• 1 B/Flop mem• 0.1 B/Flop data• 0.05 B/Flop I/O

Pins per chip

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

1 2 4 8 16 32 64 128

Number of Cores

Sig

nal +

Ref

eren

ce P

ins

Sig

nal +

Ref

eren

ce P

ins

M. Ritter Topical Workshop on Electronics for Particle Physics, Sept 2010

10 November 2011IBM INTERNAL ONLY © 2009 IBM Corporation10

0

2000

4000

6000

8000

1000012000

14000

16000

18000

1 2 4 8 16 32 64 128

Number of Cores

Sig

nal +

Ref

eren

ce P

ins

Implications of BW Scaling:

Module Escape Bottleneck

Card Escape Bottleneck

Chip escape limit, 200?m pitch

Module escape, 1mm pitch

Card escape, 8 pair/mm(QCM w/8 Cores…)

Pins per chip

Sig

nal +

Ref

eren

ce P

ins

M. Ritter Topical Workshop on Electronics for Particle Physics, Sept 2010


Total bandwidth, cost and power for optics in a machine

YearPeak

Performance(Bidi) Optical

BandwidthOptics Power Consumption

Optics Cost

2008 1PF0.012PB/s

(1.2x105Gb/s)0.012MW $2.4M

2012 10PF1PB/s

(107Gb/s)0.5MW $22M

2016 100PF20PB/sec

(2x108Gb/s)2MW $68M

20201000PF(1EF)

400PB/sec(4x109Gb/s)

8MW $200M

?Require >0.2Byte/FLOP I/O bandwidth, >0.2Byte/FLOP memory bandwidth– 2008 optics replaces electrical cables (0.012Byte/FLOP, 40mW/Gb/s)– 2012 optics replaces electrical backplane (0.1Byte/FLOP, 10% of power/cost)– 2016 optics replaces electrical PCB (0.2Byte/FLOP, 20% of power/cost)– 2020 optics on-chip (or to memory) (0.4Byte/FLOP, 40% of power/cost)

J. Kash Photonics Society Annual Meeting Nov 2010 and OWQ1 OFC 2011


1000

10000

100000

1000000

10000000

100000000

2004 2006 2008 2010 2012 2014 2016 2018 2020

Year

Num

ber

of O

ptic

al C

hann

els

HPC driving volume optics ? Computercom market

MareNostrum

ASCI Purple

Blue Waters*

WW volume in 2008

Single machine volumes similar to today’s WW parallel optics

Roadrunner2.5Gbps

10Gbps

5Gbps

?

* Expected Summer 2011 T. Dunning, NCSA, https://hub.vscse.org/resources/86/download/VSCSE_FutureofHPC_Jul10-2.ppt#265,1,Future of High Performance Computing


Electronic Packet Switching

? Typical architecture (electronic switch

chips, interconnected by electrical or

optical links, in multi-stage networks) works well now---

– Scalable BW & application-optimized cost

• Multiple switches in parallel

– Modular building blocks • many identical switch chips & links)

? -- but challenging in the future– Switch chip throughput stresses the

hardest aspects of chip design• I/O & packaging

– Multi-stage networks will require multiple E-O-E conversions

• N-stage Exabyte/s network = N*Exabytes/s of costN*Exabytes/s of power

Central switch racks

J. Kash OFC tutorial 2008

By courtesy of Barcelona Supercomputing Center - www.bsc.es


Next Gen Fiber Optics

Switch hub , optics on MCM

• ~Known (vendor) Technologies• Higher cost• Moderate Density

Potential Optics Technologies for Next Gen HPC – transition to OPCB?

Optical PCB Polymer waveguides

201820172016201520142013201220112010

300 PF20 PF

TSV Si carrier Optochips assembled

6.4mm x 10.4mm

2x12 PD

array

2x12 RX IC

2x12 LDD IC

2x12 VCSEL array

RXTXLDD RX

Si CarrierVCSEL

Lens Arrays

PD

Organic Carrier

PCBPolymer Waveguides

or Flex

To optical connector

TSV Si carrier Optochips assembled

6.4mm x 10.4mm

2x12 PD

array

2x12 RX IC

2x12 LDD IC

2x12 VCSEL array

RXTXLDD RX

Si CarrierVCSEL

Lens Arrays

PD

Organic Carrier

PCBPolymer Waveguides

or Flex

To optical connector

• Needs commercial ecosystem• Lower cost• Higher Density

20Gbps10Gbps 25-40Gbps

1 EFHPC Peak

Performance?

Optics Bitrate?

Volume commercial use lags HPC by ~4-5 years

IBM has technology expertise both in WGs and high bitrates


Advantages of Polymer Waveguide Technologycompared to Parallel Fiber Optics

? Integrated mass manufacturing– Board, sheet, film level processing of optical interconnects– Lower assembly, waveguide jumper costs

– Costs for wide busses should scale better

? Simple assembly– Avoid fiber handling (integrated approach)

– Similar assembly procedures as for electrical components and boards (pick and place, etc.)

– Establish electrical and optical connections simultaneously (pick, place, reflow, etc.)– Avoid separate optical layer (if integrated with board)

? New or compact functionality supporting new architectures– Shuffles, Crossings, splitters, …– Enabler for multi-drop splitting, & complex re-routing that is expensive in fiber

? Higher density, waveguide pitch < 125 um (best future fiber pitch)– Higher bandwidth density, less signal layers…demonstrated 62.5um

? Cost– Reduced Optics module cost AND jumper/connector costs both important

– Possible Lower Maintenance costs


Integrated packaging is more complex, requires close relationship with suppliers (IBM PERCS) Ideal candidate for PWG Packaging.

Heat Spreader for Optical DevicesCooling / Load Saddle for Optical Devices

Optical Transmitter/Receiver Devices 12 channel x 10 Gb/s28 pairs per Hub - (2,800+2,800) Gb/s of optical I/O BW

Heat Spreader over HUB ASIC

Strain Relief for Optical RibbonsTotal of 672 Fiber I/Os per Hub, 10 Gb/s each

Hub ASIC (Under Heat Spreader)

A.Benner, Future Directions in Packaging (FDIP) Workshop, EPEP Oct 2010


Density of optical links

Courtesy of Avago Technologies

Optics Module and electrical connectorCard edge

~8X8mm, ~0.75mm pitch

~18x41mm 1.27mm pitch

Active cable, electrical at card edge

Courtesy of AvagoTechnologies

4x4 VCSELArray

4x4 PD

Array

4x4 VCSELArray

4x4 PD

Array

Optical at card edge

~60K fibers/ RACK

ROADRUNNER ~1PF)

~40K fibers / SYSTEM

PERCS >10PF

8@5Gbps/cable

48@10Gbps/cable

~5x3mm 0.2mm pitch

~10x25mm

~5X15mm

> 40x denser


Optical waveguide interconnects:The Terabus project and related work

Dense Hybrid Integration: demonstrate a low-cost packaging approach compatible with conventional PCB manufacturing and surface-mount board assembly

Waveguide Lens ArrayWaveguide Lens ArraySLCSLC

CMOS ICCMOS ICCMOS IC

OE’sOE’sOE’sVCSELSLCSLCSLCSLC

OE’sVCSEL

CMOS ICCMOS IC

OE’sOE’sOE’s

Waveguide Lens ArrayWaveguide Lens Array

SLCSLC



OE’sVCSEL

CMOS ICCMOS IC

OE’sOE’sOE’s

CMOS ICVCSEL PD

Optochip

Optomodule

Optocard

Optomodule 2

Polymer waveguides

Waveguide Lens ArrayWaveguide Lens ArrayWaveguide Lens ArrayWaveguide Lens ArraySLCSLC



OE’sVCSEL

CMOS ICCMOS IC

OE’sOE’sOE’s

SLCSLC



OE’sVCSEL

CMOS ICCMOS IC

OE’sOE’sOE’s

Waveguide Lens ArrayWaveguide Lens ArrayWaveguide Lens ArrayWaveguide Lens Array

SLCSLC



OE’sVCSEL

CMOS ICCMOS IC

OE’sOE’sOE’s

SLCSLC



OE’sVCSEL

CMOS ICCMOS IC

OE’sOE’sOE’s

SLCSLC



OE’sVCSEL

CMOS ICCMOS IC

OE’sOE’sOE’s

CMOS ICVCSEL PD

Optochip

Optomodule

Optocard

Optomodule 2

Polymer waveguides

• Low-density, conventional electrical interface for power & control • High-density, wide and fast optical interfaces for data I/O? Much higher off-module bandwidth at low cost in $$ and power

Circuit Board w/ both electrical traces & optical waveguides

CPU OE XCVR

TransceiverOptochip

Other Chips

organic chip carrier

Future Vision: optically-enabled MCM’s Circa 2014-2016

< 10 FIT per channel

Reliability

2 Tbps/cm2

(on module)Density

25Gbps/channelDatarate

<$0.25/Gbps (TRx+ on-card optics)

Cost

<10mw/Gbps (EOE)

Power


Outline

?System view of why optics is needed

?Potential OPCB Technologies for Next Generation HPCs

– OSA Designs

– OSA Assemblies

– Packaging

– Optical Component Characterization


Optical Printed Circuit Boards

?IBM Research has invested heavily in the past 7+ years in Optical printed circuit board technology based on multi-mode polymer waveguides – Partially funded by the US Government (Terabus program)

?We believe this technology will be needed to provide the needed BW for future server generations, allow highly integrated electrical-optical links and provide a path to much lower cost optical links.

?We are highly interested in establishing a market eco-system that will provide a set of suppliers, standards and specifications and users for this technology.


y Offset (?m)

x O

ffse

t (?m

)

-80 -60 -40 -20 0 20 40 60 80

-80

-60

-40

-20

0

20

40

60

80

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

x 10-4

-80 -60 -40 -20 0 20 40 60 80

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

Offset (?m)

Cou

plin

g E

ffic

ienc

y (d

B)

BGA site for Optomodule

Waveguide Lens Array

35µm x 35µm

62.5µm pitch Module Attachment? BGA attachment process

? Alignment of OE lens array to waveguide lens array

Optocard Technology: Waveguides, Module Attachment and Tolerances

-50 -40 -30 -20 -10 0 10 20 30 40 50

-4.5

-4

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

Offset (? m)

Cou

plin

g E

ffic

ienc

y (d

B)

Lens Array

TRX IC

OE

SLC Carrier

FR4

Lens Array

TRX IC

OE

SLC Carrier

FR4

Tx: ±35 µm

Rx > ±65 µm

Optical coupling efficiency


Lens Array

TRX IC

OESLC Carrier

FR4

Lens Array

TRX IC

OESLC Carrier

FR4

Optocard waveguides: Turning mirrors and Lens Array

BGA site for Optomodule


?Integrated turning mirrors– TIR – laser ablation– Dense waveguide pitch

?Integrated 48-channel collimating lens array

– Provides 40mm alignment tolerance for Optomodule

channel 35, 40 not shown, in-coupling scattering

0.00

0.50

1.00

1.50

2.00

2.50

0 10 20 30 40 50

Channel Number

Bo

ard

Lo

ss (d

B)

Median : 1.6Stdev: 0.1

1.6 dB average loss?0.9dB for 7.5cm WG @980nm?~0.7dB for mirror/lens assembly

Waveguide/Mirror Uniformity

48-channel Waveguide mirror array

waveguide cores on 62.5um pitch

48-channel Waveguide mirror array

waveguide cores on 62.5um pitchTIR mirrors


Waveguide Connectors: Passive alignment with optical precision

MT pins

Alignmentstuds

Coppermarkers

waveguides

Alignmentslots

PCB

MT ferrule aligned by copper markers Positioned MT ferrule to polymer waveguides

Connector interface 12 waveguides Alignment marker

Top FR4 stack (with electrical lines)

Polymer waveguide layer

Bottom FR4 stack


Terabus Transceivers – 985nm and 850nm built

?Optochip: Single-chip CMOS transceiver IC with flip-chip attached optoelectronic arrays

?985nm, substrate emitting OEs

Optomodule


Optocard

SLCSLC

Transceiver Optochip


OE’sOE’sOE’sVCSEL OE’sOE’sOE’sPDOptomodule


Optocard

Waveguide Lens ArrayWaveguide Lens Array

Optocard

SLCSLCSLCSLC

Transceiver Optochip


OE’sOE’sOE’sVCSEL OE’sOE’sOE’sPD





?Optochip: Si carrier platform for heterogeneous integration of OEs and ICs

?Electrical and Optical vias in Si carrier

Chip-to-chip optical interconnect on a PCB using 985nm transceivers (non-standard wavelength)

Chip-to-chip optical interconnect on a PCB using 850nm wavelength transceivers (industry-standard wavelength)

Optochip

?Optomodule: High-speed, high-density organic carrier?Extendible to optically enabled MCM (OE-MCM)

?Optocard: PCB with integrated polymer waveguides?Replace complex electrical PCB with simpler electrical PCB (power and control), plus

waveguides

SLC Substrate

SiCarrier

LDD VCSEL PD TIA

SLC

SiCarrier

LDD VCSEL PD TIA SC

LDD VCSEL PD TIA

Lens ArrayLens ArrayLens Array

FR4 Substrate

SLC Substrate

SiCarrier

LDD VCSEL PD TIA

CoreEZ

SiCarrier

LDD VCSEL PD TIA SC

LDD VCSEL PD TIA


FR4 Substrate


FR4 SubstrateFR4 Substrate

SLC Substrate

SiCarrier

LDD VCSEL PD TIA

SLC

SiCarrier

LDD VCSEL PD TIA SC

LDD VCSEL PD TIA

SLC Substrate

SiCarrier

LDD VCSEL PD TIA

SLC

SiCarrier

LDD VCSEL PD TIA SC

LDD VCSEL PD TIA


FR4 Substrate

SLC Substrate

SiCarrier

LDD VCSEL PD TIA

CoreEZ

SiCarrier

LDD VCSEL PD


FR4 Substrate

SLC Substrate

SiCarrier

LDD VCSEL PD TIA

CoreEZ

SiCarrier

LDD VCSEL PD TIA SC

LDD VCSEL PD TIA


FR4 Substrate

Lens ArrayLens Array

TIA SC

LDD VCSEL PD TIA


FR4 Substrate


FR4 SubstrateFR4 Substrate


Full Terabus Link (985-nm): 2 Transceiver Optomodules on Optocard

16 Channels TRX1 ? TRX2 at 10Gb/s + 16 Channels TRX1 ? TRX2 at 10Gb/s

TRX1:

16TX + 16RX

TRX2:

16TX + 16RX

4x4 VCSELArray

4x4 PD

Array


Terabus 850 nm Transceiver Optochip Assembly

? Silicon Carrier provides dense high-speed wiring, mechanical support, and optical I/O through holes etched in the carrier

? Heterogeneous integration of ICs and OEs through dense arrays of solder “microbumps”

– IC pads: ?= 35?m, pitch = 50?m– OE pads: ?= 35?m, pitch = 100?m– ~1um Au-plating on pads

? Optochip assembly: four sequential flip-chip soldering processes– Suss FC-150 flip-chip bonder– AuSn solder pre-deposited on ICs, VCSEL & PD arrays

– Reflow solder at about 320 °C

Silicon Carrier

RX ICLDD IC

High-speed probe pads

VCSEL arrayPD array

LDD IC RX IC

VCSEL array PD array

TSV Si carrier


Assembled Terabus 850-nm Optomodule(Optochip on CoreEZ)

First row of solder joins visible beneath the Optochip

850-nm Optochip demonstrated? 300Gb/s bidirectional aggregate data rate

TX:15Gb/sRX:12.5Gb/s


VCSEL Transmitters and Receivers can achieve 2015 metrics for density, bitrate and power ? 90-nm IBM CMOS-Driven VCSEL Transmitters and Compatible Receivers

– Power and Speed Records

28

2.8 pJ/bit0.7 pJ/bit

20Gb/s

1.9 pJ/bit

29Gb/s 32Gb/s20Gb/s

1.75 pJ/bit

VCSEL Transmitters

Compatible Receivers

2.9 pJ/bit 3.5 pJ/bit 6.7 pJ/bit

15Gb/s 17.5Gb/s 20Gb/s

IBM technology expertise in high bitrate transceivers


Development of VCSELs for >25 Gb/s links –collaborations with OE vendors

26Gb/s 30Gb/s

Joint work with Finisar AOC

Joint work with Emcore Corp

20Gb/s 25Gb/s

8mA700mVpp

*R. Johnson and D. M. Kuchta, “30Gb/s directly modulated VCSELs,” CLEO 2008

*N. Y. Li et al., “High-Performance 850 nm VCSELs and Photodetector Arrays for 25 Gb/s Parallel Optical Interconnects,” OFC 2010.*N. Y. Li et al., “Development of High-Speed VCSELs Beyond 10 Gb/s at Emcore,” Photonics West 2010.

6mA375mVpp


Embedded Waveguide Technology: 120 Gb/s Board-to-Board Optical Link Demonstrator

? Embedded polymer waveguides (12 channels)

? Passive alignment of MT standard based connectors

? MT interface as standard interface for WG, fiber bundles/optical flexes and transceivers

? Pluggable TX/RX module (butt coupling)

Optical TX/RX board as building block Complete 12x10 Gb/s link demonstrator

12 embeddedwaveguides

EO modulMT

interface

10 Gb/s 10 Gb/s

Eye diagrams for 2 channels at 10 Gb/s

IBM technology expertise in high bitrate optical PWG Link Prototype builds


Field Replacable Optics – OE Carrier/LGA , Dual Layer PWG

?OE Carrier: Wafer based micro-optic assemblies, organic substrate for electrical interconnect (low cost). Removable carrier, passive alignment.?PCB / Waveguide Board: PCB with dual layer buried polymer waveguides,

integrated waveguide lens arrays.?LGA Assembly: Low force LGA, course & fine alignment means.?Optical Components: 1x12 arrays : 850nm VCSEL and photodiode Arrays (10Gb/s)

VCSEL arrayPD array

LDDTIA

WG arrays with mirrorsLens arrays

Backplane Connector (TRL)

Organic CarrierLGA

Heat SpreaderCarrier

PCB

LGA

IBM technology expertise in field replaceable optical TRx packaging

Libsch, F.R. et. al., “ MCM LGA packaging for Optical I/O Passively Aligned to Dual Layer Polymer Waveguides”,IEEE Electronic Components and Technology Conference (ECTC), 2006


Waveguide-on-flex cables:192 Channel flexible waveguide optical backplane

8 waveguide flex sheets, 192 waveguides, 8 connectors

4 connectors, 48 waveguides each

?Multi-layer waveguide optical connector?Based on passive alignment?Basic building block for optical pcb backplane technology

Optical L-Links

Optical D-Links


48-Channel-to-4x12-Channel Waveguide Flex Fan-Out

12 WGswith 250 ?m pitch

48 WGswith 62.5 ?m pitch

Light incoupling

Waveguide cross-section


Summary and Questions for Discussion?As shown, optical polymer waveguide interconnect technology

are a candidate for high performance computer optical interconnects.

?IBM is working with other system houses to define commonality, drive volumes up, and provide for lower cost to both system and component vendors.

?IBM is not likely to build this technology for itself, just as we currently do not manufacture PCBs and organic build-up technologies or optical transceivers

• IBM has developed research-level technologies for all aspects of the technology

• Commercialization of waveguide-on-card manufacturing would require working with a commercial PCB vendors and a waveguide materials supplier, as well as a manufacturer of the optical transceivers

?What role do vendors (yourself) envision for yourself in that ecosystem? Comments welcomed.

Documents

Optical PCB Overview - IBM WWW Page · Optical PCB Overview Frank Libsch IBM T.J. Watson Research Center Yorktown Heights, NY. ... Optical printed circuit board technology based on