AJF

Aperture ArrayAperture Array

Station ProcessingStation Processing

Andrew Faulkner

SKADS Project Engineer

September 2009 Station Processing – Cape Town

AJF

SKA Overall StructureSKA Overall Structure

September 2009 Station Processing – Cape Town

..

Sparse AA

Dense AA

..

Mass Storage

TimeTimeStandardStandard

Central Processing Facility - CPF

User interfacevia Internet

...

To 250 AA Stations

...

DSP

...

DSP

To 2400 Dishes

...

12-15m Dishes

Correlator – A

A &

Dish

16 Tb/s

80 Gb/s

Control Processors

& User interface

Post Processor

Data

Time

Control

70-450 MHzWide FoV

0.3-1.2 GHzWide FoV

0.8-10 GHzSingle

Pixel or Focal plane

array

Beam Data

Tile &Station

Processing

AJF

Mid-Frequency Aperture ArrayMid-Frequency Aperture Array

September 2009 Station Processing – Cape Town

Georgina HarrisGeorgina Harris

• Freq range 0.3 1.2 GHz

• Dense array (Nyquist sampled)

• ~75,000 Receiver chains

~60m

Tile

Support

Bunker

AJF

EMBRACE - SKADSEMBRACE - SKADS

August 2009 Station Processor

AJF

The AA StationThe AA Station

September 2009 Station Processing – Cape Town

StationProcessor 1

0.3-1.0GHz Analog links

StationProcessor 2

StationProcessor X

…Mid P1

Mid P2

Mid Py

.

InternalDigital links

n x Optical fibres per 2nd

stage processor

To

Co

rrelato

r

AA-hi

Phase Standard &Distribution

500MHz Analog links

. .

AA-lo

Control processors

To CentralControl

10Gb Digital fibre links

Phase transferover fibre

1st Stage Processors

Next Proc. Bunker

Prev. Proc. Bunker

O-E?Low P1

. .

O-E?Low Pz

.....

Box

>1016 MACs

AJF

1-D Beamforming1-D Beamforming

September 2009 Station Processing – Cape Town

Incoming signal

Geo

met

ricD

elay

, t

+

Electronic Delay

+ + + + + + + + + + + + + + + + + + + + + + + Beam

Del

ay

Element #

Elements

t

0

AJF

1-D Beamforming1-D Beamforming

September 2009 Station Processing – Cape Town

Incoming signal

Geo

met

ricD

elay

, t

+

Spectral Separation + Electronic Delay

+ + + + + + + + + + + + + + + + + + + + + + + BeamC0I0+ C1I1+ C2I2+ C3I3+ C4I4+ C5I5 + C6I6+ C7I7+ C8I8+ C9I9+ C10I10 + C11I11+ C12I12+ C13I13+ C14I14+ C15I15 + C16I16+ C17I17+ C18I18+ C19I19+ C20I20+ C21I21+ C22I22+ C23I23

Elements

C0I0+ C1I1+ C2I2+ C3I3+ C4I4+ C5I5+......Using Phase delay

approximation in narrow

frequency bands

AJF

Hierarchical structureHierarchical structure

September 2009 Station Processing – Cape Town

Incoming signal

Geo

met

ricD

elay

, t

Electronic Delay

+ + + + + + + + + + + + + + + + + + + + + + +

Beam

Elements

+ + + + +

Tiles

Station processor

AJFSeptember 2009 Station Processing – Cape Town

AA Station ImplementationAA Station Implementation

Ae

Ae

Ae

…..

…..

…..

TileProcessor

- hi

TH_0

TH_1

TH_n

StationProcessor

…..

……

.

To

Co

rrelato

r

Tile beam data

Station beam data

AJF

Survey Speed – AA’s forte!Survey Speed – AA’s forte!

• AA station is big: say 60m dia. small beams

~0.28º width

• So, many, many station beams: >3000 !>3000 !

• But, Tile beams are larger, ~5º width

• So, # of Tile beams required:1010

August 2009 Station Processor

At 1 GHz with 250 deg2 FoV:

Easy....?

AJF

Visually..Visually..

September 2009 Station Processing – Cape Town

Tile beam

Station beams

Great!

Constrains data rate of Tile Station processor

AJF

Ah, but...Ah, but...

September 2009 Station Processing – Cape Town

Tile beam

Station beams

Only perfect for the central beam on Tile beam

This is what we’ve done.....

AJFSeptember 2009 Station Processing – Cape Town

Incoming signal

Electronic Delay

+ + + + + + + + + + + + + + + + + + + + + + +

Beam

Elements

+ + + + +

TilesStation processor

t

Del

ay

Element #

Tile BeamformingTile Beamforming

Tile Beam Discontinuities will give high sidelobes and variable forward gain:

Dynamic range badly affected0

AJF

Some ConsequencesSome Consequences

• Need ‘extra’ Tile beams over minimum calculated.– Probably also interpolate between Tile beams for more precision

• Performance can still be arbitrarily good: dynamic range

etc. Maybe on limited FoV.

• Bandwidth from Tiles to Station processor determined on

quality of beams as well as FoV: still programmable.

• Concentrate beamforming centrally as much as possible

• Calculating “allowable error”, hence Tile Station Proc.

Communications requirment is “work-in-progress”

September 2009 Station Processing – Cape Town

Hierarchical Beamforming looks best option

AJF

AA Station Processor Reqts.AA Station Processor Reqts.

August 2009 Station Processor

1. Beamform the output of all the Tile processors into Station beams

2. Send “correlator ready” data over fibre to the central processor

3. Part of the station calibration scheme

4. Flexibly handle the data from the AA-hi and AA-lo arrays

5. Maybe provide the ‘F’ part of the FX correlator

6. Maybe provide station data for local processing

7. Possibly hold ‘observation history’ data to post process

8. Be readily scaleable for:

a. Number of tiles c. Internal station data rates

b. Data bandwidth to correlator d. Data length to correlator

AJF

Outline spec. of Processing ChipOutline spec. of Processing Chip

September 2009 Station Processing – Cape Town

Processing Device:

PChipPChip

1013 MACor 10 TMAC

Inputs Outputs

0

1

2

3

4

5

Each Stream:12 x 10Gb/s

0

1

2

3

4

5

AJF

PChip B’forming requirementsPChip B’forming requirements

September 2009 Station Processing – Cape Town

Assuming: All 6 outputs at full speed using all 6 inputs 2 reals per complex sample and 4 real MACs per complex

MAC.

1. 6 inputs, I0 – I5 and 6 outputs O0 – O5.

2. Beamforming for one output stream:

a. On = C0I0+ C1I1+ C2I2+ C3I3+ C4I4+ C5I5

b. The input data rate per stream is 120Gb/s raw: 96Gb/s actual

c. Each complex sample is 2 x 4 bit reals so: ~12.5GS/s per stream.

d. Processing per output stream is: 4*6*12.5 = 300 GMACs

e. Total processing for 6 output streams = 1,800 GMACs.

3. This is well within the PChip capability of 10TMACs

4. Maybe some pre-processing required on each sample.

AJF

Station CommsStation Comms

• 120 Gb/s total

• 50m range

• Power:– Tx: 2.4 watts– Rx: 2.0 watts

• 12-channel fibre– 10Gb/s channels

• VCSEL technology

• Pluggable

• 19mm pitch

August 2009 Station Processor

AJFSeptember 2009 Station Processing – Cape Town

AA Station Data ratesAA Station Data rates

Ae

Ae

Ae

…..

…..

…..

TileProcessor

- hi

TH_0

TH_1

TH_n

…..

…..

….. Tile

Processor- lo

TL_0

TL_1

TL_m

StationProcessor

0e/o

e/o

e/o

e/o

…..

…..

o/e

o/e

o/e

o/e

o/e

o/e

o/e

o/e

……

.

e/o

e/o

e/o

e/o

Station Processor n…

….

Long distance drivers

…..

o/e

o/e

o/e

o/e

o/e

o/e

e/o

e/o

e/o

e/o

e/o

e/o

e/o

e/o

Long distance drivers

…..

Long distance drivers

…..

....

…..

1.0-1.2GHzanalogue

1.0 GHzanalogue

12 fibre lanes @10Gb/s each

……

…...

12 fibre lanes @10Gb/s each

10Gb/s fibre

…..

Max 4 Station Processors

Local Processinge.g. Cal; pulsars

To C

orrelator

Inputs #: 1296Channel rate: 120Gb/s

(raw)Total i/p rate: 1.5 Pb/s

Typical:AA-hi tiles: 300AA-lo tiles: 45Total: 345I/p data rate: 42Tb/s

Notes:1. No control network shown2. Up to 4 station processor systems can

be implemented in parallel3. Data shown are raw, typ. get 80% data

…..

AJFAugust 2009 Station ProcessorAugust 2009 Station ProcessorStation Processor BoardStation Processor Board

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

o/eo/eo/eo/eo/eo/e

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

e/oe/oe/oe/oe/oe/o

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

PChip

0

1

2

3

4

5

30

31

32

33

34

35

Each link is 12 fibre lanes@10Gb/s

To 1st stage Processorsor Primary

Station Processors

To Secondary Station Processors or long distance fibre drivers

Each link is 12 fibre lanes@10Gb/s

Each link is 12 diff. copper lanes@10Gb/s

12-channel Rx module. e.g Avago

AFBR-820BXXZ

12-channel Tx module. e.g Avago AFBR-810BXXZ

Total Raw input data rate: 4.32Tb/s

Total Raw output data rate: 4.32Tb/s max

“All to All”Connections

ControlProcessor

Lin

eT

x/Rx

Station Control

AJFSeptember 2009 Station Processing – Cape Town

Station Processor system (120Gb/s per Tile)Station Processor system (120Gb/s per Tile)

PrimaryStation

ProcessorBoard

0

…..

PrimaryStation

ProcessorBoard

1

…..

PrimaryStation

ProcessorBoard

(max 35)

…..

…..

SecondaryStation

ProcessorBoard

0

SecondaryStation

ProcessorBoard

1

SecondaryStation

ProcessorBoard

(max 35)

…..

…..

…..

…..

…..

…..

…..

012

35

To each1st stage

Processor

AA

-hi

AA

-lo

Long distance drivers

…..

0

1

2

Long distance drivers

…..

Long distance drivers

…..

To Correlator

Each link is 12 fibre lanes@10Gb/s

Each correlator channel is10Gb/s (maybe colour multiplexed together)

“All to All”Connections

Each link is 12 fibre lanes@10Gb/s

Max no. of Tiles

AA-hi+AA-lo

is

1296…

..…

..…

..

Station Processor

AJFAugust 2009 Station Processor

Long distance drivesLong distance drivese/oe/oe/oe/oe/oe/oe/oe/oe/oe/oe/oe/o

Each link is fibre10Gb/s raw

Long distance 10km Tx module.

PChip

o/eo/eo/eo/eo/eo/e

0

1

2

3

4

5

e/oe/oe/oe/oe/oe/oe/oe/oe/oe/oe/oe/o

12-channel Rx module. e.g Avago

AFBR-820BXXZ

Each link is 12 fibre lanes@10Gb/s

To Secondary Station

Processors

To Correlator

Total Raw output data rate: 720Gb/s max

…..

Block 0

Block 5

Total Raw input data rate: 720Gb/s

ControlProcessor

Lin

eT

x/Rx

Station Control

AJF

Estimated Costs Estimated Costs (120Gb/s per Tile)(120Gb/s per Tile)

August 2009 Station Processor

Total: €480k

AJF

ConclusionsConclusions

1. Beamform to 1 chip depth at the Tile

2. Station Processor data rate is much higher than output

3. Processing performance is unlikely to be an issue

4. Comms costs and performance critical

5. Focus on making wide-area comms cheaper!

August 2009 Station Processor

AA Station AA Station Beamforming can be Beamforming can be

done!done!