Danie Ludick MScEngDanie Ludick MScEngStudy Leader: Prof. D.B. DavidsonStudy Leader: Prof. D.B. Davidson

Computational Electromagnetics Group Computational Electromagnetics Group Stellenbosch UniversityStellenbosch University

Extended Studies of Focal Plane Arrays for the SKA and the

MeerKAT

Focal Plane Array Feed-structures Focal Plane Array Feed-structures for the SKAfor the SKA

Parabolic Dish Reflector Antennas for the SKAParabolic Dish Reflector Antennas for the SKA Typical Radio Astronomy ReceiversTypical Radio Astronomy Receivers MeerKAT – already built a prototypeMeerKAT – already built a prototype

- the XDM (15 m dish)- the XDM (15 m dish) Requires a feed-structure at the focal point Requires a feed-structure at the focal point

to collect the focused energyto collect the focused energy

Focal Plane ArraysFocal Plane Arrays Some Applications of SKA requires whole sky Some Applications of SKA requires whole sky

imaging – very time consuming with conventionalimaging – very time consuming with conventional

single pixel feeds, such as horn antennas single pixel feeds, such as horn antennas SolutionSolution: Reduce survey time by using focal plane: Reduce survey time by using focal plane

arrays arrays →→ multiple beams can be formed that can multiple beams can be formed that can

be scanned independently of each otherbe scanned independently of each other

Focal Point

XDM

Focal Plane Array DesignFocal Plane Array Design

Difficulty … Simulating the FPA StructureDifficulty … Simulating the FPA Structure Large FPA designs, such as an array Large FPA designs, such as an array

of Vivaldi Antennas of Vivaldi Antennas →→ memory intensive, memory intensive,

simulations can take several hours even days !simulations can take several hours even days !

Need to investigate effective simulation techniques based on the Need to investigate effective simulation techniques based on the Method of MomentsMethod of Moments

Using FEKO in a High Performance Parallel

Computing Environment

Developing efficient solution Developing efficient solution techniques using the techniques using the

Characteristic Basis Function Characteristic Basis Function Method (CBFM)Method (CBFM)

The Method of MomentsThe Method of Moments

The Method of Moments (MoM) …The Method of Moments (MoM) … Computational EM method for determining unknown surface current distribution Computational EM method for determining unknown surface current distribution

on an electromagnetic scatterer, when illuminated by various excitationson an electromagnetic scatterer, when illuminated by various excitations

Computational Cost of ~ O(NComputational Cost of ~ O(N22) memory storage and ~ O(N) memory storage and ~ O(N33) for ) for solving linear equationssolving linear equations

[A]{x} = {y}(size N x N)

N = Unknowns, i.e. current associated with non-boundary edges of triangles

Discritise the problem

Solve for {x}

Using FEKO in a Using FEKO in a High Performance High Performance Computing (HPC) Computing (HPC) environmentenvironment

The Benchmarking Study …The Benchmarking Study … Simulating various sized Vivaldi Arrays on a Cluster at the CHPC in Cape TownSimulating various sized Vivaldi Arrays on a Cluster at the CHPC in Cape Town

CHPC Infrastructure … IBM e1350 Linux Cluster … the iQuduCHPC Infrastructure … IBM e1350 Linux Cluster … the iQudu 160 Compute Nodes 160 Compute Nodes →→ 2 dual-core AMD Opteron 2.6 GHz Processors + 2 dual-core AMD Opteron 2.6 GHz Processors +

16 GByte of RAM per Node16 GByte of RAM per Node In Total 640 ProcessorsIn Total 640 Processors

~ 2.5 Teraflops Processing~ 2.5 Teraflops Processing Power Power 2 Available Interconnects … 2 Available Interconnects …

1 GByte Ethernet and 10 GBit Infiniband1 GByte Ethernet and 10 GBit Infiniband

Number of elements in arrayNumber of elements in array 11 88 1616 3232 6464 8181 128128

Number of Unknowns (N)Number of Unknowns (N) 550550 4,3684,368 8,7368,736 17,47217,472 33,62433,624 38,34038,340 63,40063,400

the iQudu

Example of HPC FPA SimulationExample of HPC FPA Simulation

64 Element Vivaldi Array64 Element Vivaldi Array 32 Nodes32 Nodes 33,624 Unknowns33,624 Unknowns 25 Frequency Points Analyzed 25 Frequency Points Analyzed

(400 MHz – 2 GHz)(400 MHz – 2 GHz)

ResultsResults

On 1 node, estimated runtime is ~ 40 hours (1.6 days) !On 1 node, estimated runtime is ~ 40 hours (1.6 days) !

SimulationSimulation PredictionPrediction

Total Runtime Total Runtime ~ 4 hours~ 4 hours ~ 4.5 hours~ 4.5 hours

Memory UsageMemory Usage 9.756 GByte9.756 GByte 8.4 GByte8.4 GByte

|S11

| [dB

]

0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2-25

-20

-15

-10

-5

0

Frequency [GHz]

What we have in terms of What we have in terms of simulation powersimulation power

Fast … but not fast enoughFast … but not fast enough

What we want …What we want …

To achieve this … improve algorithmsTo achieve this … improve algorithms

The Characteristic Basis Function Method The Characteristic Basis Function Method (CBFM)(CBFM)

Conventional Solvers …Conventional Solvers …

Direct Methods Direct Methods

(Fast, N is limited)(Fast, N is limited) Iterative SolversIterative Solvers

(Big problems, very long (Big problems, very long runtimes)runtimes)

The CBFM …The CBFM …

Reduces the size of the matrix Reduces the size of the matrix equationsequations

Solve using direct methodsSolve using direct methods

[A]{x} = {y}(size N x N)

MoM

MoM

Solve

The Characteristic Basis Function Method The Characteristic Basis Function Method (CBFM) … introductory example(CBFM) … introductory example

2

1

3

4

Z

X

Y

3λ3λ

Ni unknowns

([A]{x} = {y}) NxN

N ~ 1000’s

2

H E

Z(1)(1) Sub-domains (1 … 4)Sub-domains (1 … 4)

(2)(2) Primary CBFs, Primary CBFs, JJii(i)(i) i = 1 .. 4 i = 1 .. 4

(3)(3) Secondary CBFs, Secondary CBFs,

JJkk(i)(i) i = 1..4; k = 1 .. i-1, i+1, .. 4 i = 1..4; k = 1 .. i-1, i+1, .. 4

(4)(4) Entire solution to the problem …Entire solution to the problem …

(5) Reduced [A]{x} = {y}

(6)(6) Solve this equation for the complex Solve this equation for the complex αα coefficients with direct methods coefficients with direct methods

)(

1

)(

1

)2(

)2(

1

)1(

)1(1

:

0

0

...

0

:

0

0

:

0

M

k

M

k

M

k

M

k

k

k

M

k

k

kN

J

J

J

x

The Characteristic Basis Function Method The Characteristic Basis Function Method (CBFM) – Numerical Results(CBFM) – Numerical Results

Results show Current Distribution on 4 x 1.5Results show Current Distribution on 4 x 1.5λλ plates ( plates (λλ/12 discritization)/12 discritization)

Number of Unknowns (N) = 3,744Number of Unknowns (N) = 3,744

Number of CBFM Sub-domains (M) = 4Number of CBFM Sub-domains (M) = 4

Number of CBFs (Number of CBFs (MM22) = 16) = 16

Runtime reduces by factor of ~ 13Runtime reduces by factor of ~ 13

Iterative MethodsIterative Methods CBFM CBFM

MethodMethod CGSCGS GMRESGMRES CBFMCBFM

Solution Solution Time (s)Time (s) 256.34256.34 258.84258.84 19.2819.28

00.5

11.5

22.5

33.5

00.5

11.5

22.5

33.5

-1

-0.5

0

0.5

1

Z

XY

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1I [A/m]

00.5

11.5

22.5

33.5

00.5

11.5

22.5

33.5

-1

-0.5

0

0.5

1

Z

XY0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

I [A/m]

Iterative Solutions include Preconditioning

The Way ForwardThe Way Forward

HPC resources (e.g. the iQudu) and methods such as CBFM provides basis HPC resources (e.g. the iQudu) and methods such as CBFM provides basis for simulating very large Focal Plane Arraysfor simulating very large Focal Plane Arrays

CBFM is highly parallelizable … implement on cluster such as iQudu … CBFM is highly parallelizable … implement on cluster such as iQudu … dramatically reduce simulation timedramatically reduce simulation time

Apply efficient numerical techniques to various FPA’s (Vivaldi, Apply efficient numerical techniques to various FPA’s (Vivaldi, Checkerboard, New Ideas …)Checkerboard, New Ideas …)

Goal:Goal:

Provide SKA Engineers with a means of analyzing

large FPA feed-structures efficiently

Questions ?Questions ?