Beam Monitoring and Control with FPGA Based Electronics Nathan Eddy Instrumentation Department

Beam Monitoring and Control with FPGA Based Electronics

Nathan Eddy

Instrumentation Department

2/1/2007 Nathan Eddy Instrumentation

Outline

FPGA Overview Example Applications at Fermilab Next Generation Wireless Observations on FPGA Systems Summary


Field Programmable Gate Arrays or…System on a Programmable Chip

Provide reconfigurable hardware implementations in a single chip solution Combine the speed of hardware

with flexibility usually associated with software

Features vary from basic logic and I/O to complete systems with dedicated RAM, DSP blocks, Clock Management, and advanced digital I/O capability

Used in a wide variety of applications Cell Phones, Wireless, Radar,

Image Processing, etc

Doubles Every18 Months

pcsupport


The Bells and Whistles or…What’s in there? Up to 180k Logic Elements Up to 10MBits of RAM

Able to implement true dual port ram & FIFOs

Dedicated DSP blocks running at up to 500MHz Able to implement multiplies,

multiply-accumulate, and FIR Filters

Sophisticated Clock Management Circuitry Internal Phase Lock Loops for

multiply, divide, and phase shifting clocks

Dedicated Clock networks throughout the chip

Single-end and Differential I/O for all common standards Up to 1100 user defined pins Very Flexible & Configurable


Embedded Systems

Hard Core Embedded processor is a dedicated physical component of the chip, separate from the programmable logic 2-4 times faster than Soft Core

Soft Core Embedded processor is built out of the programmable logic on the chip A 32 bit RISC processor uses

about few percent of total resources

Implementation of existing code directly in the FPGA without having to develop HDL code

Hard Core Resource Allocation

Soft Core Resource Allocation


Altera FPGA Families

Stratix I, II, & III High end FPGA with all the features

Latest interfaces and fastest speed grades Dedicated Adaptive Logic and DSP blocks More logic, ram, variety of PLLs Packages from 484 to 1760 pin FBGA

$200-$8000 depending on size & speed Cyclone I & II

Low cost FPGA with scaled down features Much the same feature set – No Adaptive Logic or DSP

blocks Less logic, less ram, less PLLs, slower Packages from 144 (TQFP) to 672 (FBGA) pin

$10-$250 depending on size & speed


Benefits of FPGAs and Digital Design Extreme flexibility inherent in FPGA

Algorithms and functionality can be changed and updated as needed

Code base which can be used for multiple projects Intellectual Property (IP) cores provide off the shelf solutions

for many interfaces and DSP applications The speed of parallel processing

Can perform up to 512 multiplies simultaneously The Pipeline nature of FPGA logic is able to satisfy

rigorous and well defined timing requirements Digital Design provides a straightforward simulation

path for design development and verification A variety of commercial tools available – MatLab, SimuLink,

AccelDSP, etc


FPGA Based Instrumentation at Fermilab Foster Digital Damper Board

Main Injector Dampers Recycler Damper

VME Digital Damper Board Recycler Adaptive RF Correction Recycler ACBEAM Intensity Monitor Replace Foster Boards in MI and Recycler

PBar Downconverter Digitizers Developed as BPM electronics for AP3 Used to “fix” MI SBD Timing problems


The Original Foster Board

Developed by B. Foster and A. Seminov

Required a number of blue wires and patches to get working 4 of 5 boards made operational Operated as the MI Damper

system since 2004 Numerous talks by B. Foster &

P. Adamson Used to implement the Recycler

Transverse Damper System in 2005


Recycler Transverse Instability Damper

Provide negative feedback to damp out transverse instabilities Independent Horizontal and Vertical systems


Digital Filter Implemented in FPGA

Implemented a digital filter in the FPGA Four sample boxcar average on the input remove 53MHz harmonics One turn notch filter to remove revolution harmonics Able to re-use ADC, DAC, and Clock interfaces from MI Damper

Uses Pipeline ability of the FPGA to control delays All gains and delays are user controlled by registers implemented in

the FPGA and tied to ACNET devices


Able to implement the FPGA firmware for the system in a couple weeks

Commissioning took another 4-6 weeks Pseudo Density

With no damper, observed instability induced beam loss when D > 0.8

With the damper, routinely able to run with D as high as 1.5 with no beam loss

After commissioning, had one occurrence where the damper caused beam loss when the input RF clock went away during a Low Level RF system reset Implemented RF clock verification

algorithm in the FPGA which disables the damper and reports and error back to the control system

Recycler Damper Commissioning and Performance


VME Digital Damper Hardware Next Generation Board

M. Larwill & N. Moibenko of PPD did schematics & layout 3 Prototype boards 15 Production boards

Can synchronize ADCs & DACs with external clock

All clocks controlled by Phase Lock Loops in FPGA

Traded Sharc for VME Take advantage of existing

controls software Replace FIFOs with DDR

RAM Bigger FPGA and upgraded

DACs All interfaces in FPGA

Designed as Swiss Army Knife of Instrumentation

HugeFPGA

VME

DDR Sodimm1 GByte4 ch

DAC636MHz

4 chADC

212MHz

RF Clock In

DigitalOutputs

DigitalInputs

Extras Available–Ethernet

USBLVDS serial

Sharc LinkPort

MaxCPLD


Recycler Low Level RF and Beam Distribution

Beam Profile can be predicted by looking at the Cavity Fanback Martin Hu showed the profile could be

improved by correcting the LLRF based upon the fanback

Need to correct distortions of Recycler Low Level RF barrier buckets Amplifier response, cables, cavity response Need a better than 1% correction on 2kV

barriers

Cavity Fanback

Beam Distribution


Adaptive Correction for Recycler LLRF

Implement closed loop correction for LLRF drive signal based upon Cavity Fanback

Build a correction which is constantly adapting to system changes The sampling is synchronous with the machine RF which allows the

building of a one turn waveform of 588 53MHz samples The averaging and the correction are done on the one turn waveform

588 SamplesPer Turn

588 SamplesPer Turn


WCM

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.00E+00 5.00E-06 1.00E-05 1.50E-05

WCM

Adaptive LLRF Correction Performance The board has been integrated

into the Recycler LLRF crate and operational since October See about 10% variation on

anti-proton bunch intensity delivered to TeV

The variation was more than 50% without the correction

Studies are still underway to optimize the system and further flatten the cold beam profile Develop an analytical model for

the system Remove effect on transformers

on inputs

Correction Off

Correction On


Recycler Intenstiy Monitor

The Recycler DCCT used to monitor the beam intensity is no longer working reliably and needs to be replaced Likely to occur during next long shutdown

Need a reliable alternative to measure the beam intensity ASAP

Determine the intensity of the bunched beam by integrating the signal from a toroid Problem is determining the baseline

Depends upon the intensity and distribution Needs to work during injection/extraction and RF

manipulations A “smart” digitizer can help


ACBEAM Custom Digitizer Design

Construct 588 sample 1 turn waveform Use RRBeamSync to sync with turn marker

Keep a running sum of the last N turns Currently N = 2 to 100, plan to increase it to 1000 Able to re-use design from RF correction

The N Turn waveform and resulting integration can be latched upon request and then readout at leisure

Toroid Data212MHz

RF Clock53MHz

RRBeamSync

VME

ΣSum to53Mhz

N Turn Running

Sum

588 SampleWaveform

RAM

1 Turn Integral


ACBEAM is Now Operational

Able to put together the hardware (FPGA) in a couple of days System was designed, tested and commissioned in a few

weeks (J. Crisp, A. Ibrahim, D. Voy) Dependence on bucket length understood Still studying the slow time drift (1% over a few hours) Suspect dependency on the momentum spread of the beam

RMS < 0.1e10

Time Drift


High Order Mode BPM from Tesla RF Cavity

Look at a dipole mode which couples strongly to the beam Cavity imperfections cause

polarization frequency split Can determine the 4D (X, X’,

Y, Y’) beam position from the amplitude and phase of the signals

Coupler 1 Coupler 2

Single Bunch


HOM BPM Calculations

knk

n

jnj

n

kjk

j

AA

AA

XX

XX

VV

VV

1

111

1

111

1

111

n

n

n

n

knk

n

k

k

yy

yy

xx

xx

AA

AA

MM

MM

1

1

1

1

1

111

1,441

1,111

11111

Mode VectorsRaw Data

Amplitudes

Calibration Matrix

4D Position

k ~ 6, j ~ 100 to 4k


Custom FPGA Digitizer Implementation

Need to read out raw data for mod*cav*coupler channels at 4k to 10k data points per for multibunch then perform dot products to determine mode amplitudes

The current system is unable to report a position for every pulse at 5Hz for single bunch even with only a few cavities per module enabled

Alleviate the I/O bottleneck and greatly reduce the processor load by calculating the mode amplitudes in the hardware

Store mode vectors in FPGA RAM and perform the dot product on the data as it comes in

This can be done in the digitizer hardware very efficiently by an FPGA This can improve the processing time by several orders of magnitude for

online measurements Proto-type design being tested at DESY using VME Digital Damper

FPGA

xn*vn,jCoupler Data Amplitudes


PBar Downconverter Digitizer

Developed by S. Hansen, B. Ashmanskas, D. Peterson, T. Kiper


Downconverter Digitizer as BPM Designed as used as a BPM in

the transfer line downstream of the Anti-proton production target

The original system was unable to see the anti-proton signals due to small amplitude signals and out of band kicker noise

Use analog downconversion to select 53MHz component of the signal

New system allows position measurements on anti-proton beam in AP3 line with ~100m of resolution

Kicker NoiseBeam SignalBeam Envelope


PBar Downconverter as a Programmable Trigger Module The Sampled Bunch Display (SBD)

system is a scope based system used to record longitudinal bunch data for analysis Was down for several weeks due

to issues with old CAMAC timing system

Implemented a programmable delay trigger state machine to provide scope triggers The triggers are synchronized with

the turn marker from the RF system and completely user controlled

As an extra, we also implemented an MDAT decoder for readback of machine parameters like momentum on each trigger

The basic system was developed and commissioned in about a week

FPGATriggerModule

FastScope

MDATSerial Link

Turn Marker

Beam Signal

Trigger

Ethernet

Ethernet


NimBin FPGA Development Board Developed from initial Downconverter Digitizer as cost effective general

purpose FPGA instrumentation (S. Hansen, B. Ashmanskas, D. Peterson) Similar to commercial development boards available but targeted for use

in Fermilab

NimBin provides low overhead

Basic hardware for many applications Monitoring and

Feedback Sophisticated Test

Signal Source Interface Testing

Currently 4 proto-type boards exist


Next Generation Wireless

Will use Multiple Input Multiple Output (MIMO) Technology A key element is multipath which refers to reflections of RF

waves in a physical environment Spatially multiplexed orthogonal frequency multiplexing MIMO Uses reflections to tune system performance and minimize errors Increase transfer rate based upon multiple antennas

More data and better spectral density Requires very sophisticated algorithms to run realtime

Scattering, diffraction, absorption are all considerations Need a channel model to account for all of these effects

Software based mathematical models Need a real world environment the MIMO system will operate in

Requires abiltiy to rapidly tune transmitter and receiver


MIMO System

Use 2 to 4 antennas Data Rate = BW * Ess * Num Attennas BW is 20MHz/40MHz, Ess is spectral efficiency

2.7 to 3.6bps/Hz Use Linear Algebra to decouple the channel

matrix in spatial domain and recover the transmitted data Matrix inversion, maximum likelihood detection

(MLD), and SVD used to resolve signal Has been demonstrated on FPGA based

hardware at speeds up to 1Gbps Flexibility of FPGA hardware allows fast testing

and of new algorithms

Transmitted Data

Received Signal


A Couple Observations on FPGA Electronics

FPGAs are digital objects and the effectiveness of any system is dependent upon it’s Analog interfaces

Timing is everything


The Analog Components – ADCs & DACs Need to work with analog input signals

Beam pickups, Schottky detectors, Torroids, etc Requires Analog to Digital Converters (ADCs) Current ADC performance

Up to 2-3 GSPS with 8 bit precision Up to 500 MSPS with 12 bit precision Up to 100 MSPS with 14 bit precision

Need to produce analog output signals To act on the beam – RF, kick signals, etc Require Digital to Analog Converters (DACs) Current DAC performance

Up to 1 GSPS with 14 bit precision The effectiveness of FPGA solutions is largely dominated by the

performance of the analog components and converters Mixers can be used to downconvert or upconvert the signals

Speed needed is dictated by the bandwidth requirements


Parallel and Pipelined - Timing

Developing the algorithms and logic is often straightforward With modern HDL languages very similar to writing C code

The details are in making sure timing requirements are met Need to keep pipeline structure in mind when designing Possible for a small change in a design to produce timing errors


Summary

FPGA are now the acknowledged leader of cutting edge fast DSP applications where speed and flexibility are needed

FPGA based electronics are being used to implement algorithms for beam control and monitoring systems The flexibility of general purpose designs based upon FPGAs

allows fast development & deployment of new systems Provide convenient proto-types for future projects

The size, speed, and feature sets of FPGAs are growing by leaps and bounds while the cost is decreasing

How can we take advantage of the increased power of FPGAs for future accelerator control and monitoring?


Backup Slides


Adaptive Logic Block


From Design to Implementation FPGA tool is used to generate

RTL for configuring FPGA Provides simulation but can be

tedious Write HDL code and feed it to

the FPGA tools Can use IP like subroutines

MatLab/Simulink to FPGA tools with 3rd Party synthesis tool

C to FPGA tools with 3rd Party tools

Working towards plug and play orientated design Easier to learn Faster development

FPGA Tool EDA Apps


Recycler Transverse Damper Response

-80.00

-60.00

-40.00

-20.00

0.00

20.00

0.00 2.00 4.00 6.00 8.00 10.00

f/frot

db

-180.00

-90.00

0.00

90.00

180.00

270.00

deg

f irst 10 rotation lines

-60.0

-40.0

-20.0

0.0

20.0

0.00 50.00 100.00 150.00 200.00

MHz

db

sinX/X

avg 4 samples of 212MHz

-3db at 19MHz


Recycler Damper Commissioning

Commissioning and checkout done by open loop VSA measurements of upper and lower tune lines

Make sure phase is correct at different rotation harmonics


Dipole Mode Response

Documents

Beam Monitoring and Control with FPGA Based Electronics Nathan Eddy Instrumentation Department