The Atacama Large Millimeter Array

(ALMA) Test Facility in Retrospect

Debra Shepherd National Radio Astronomy

Observatory

The ALMA Test Facility

(ATF)

URSI National Radio Science Meeting Boulder, ColoradoJanuary 5, 2009

ContentsALMA

• A brief introduction to the ALMA Test Facility (ATF)

• Evolution of Accomplishments at the ATF• September 1, 2007 to December 20, 2008

• ATF: Lessons Learned• Operations decisions - what was critical and

what did not work

• Where do we go from here? – Looking to Chile

Why build the ATF?ALMA

• Built initially at the Very Large Array (VLA) in New Mexico to determine if prototype antennas met specifications

• Antenna Evaluation Group (AEG) built the facility, sharing infrastructure with the VLA, to evaluate the antennas using non-production Back-End (BE) and Front-End (FE) equipment.

• Systems Engineering (SE) took over the facility to test hardware components and signal flow

• Evaluated hardware in a semi-integrated environment with software that gave reasonable hardware status but could not run as an integrated system.

• Computing took over when SE focus moved to Chile. • Primary goals: Develop an operational facility (total power &

interferometry) that provided a stable platform to develop and test software, develop operational procedures, test production BE hardware, support ALMA assembly & integration testing.

Prototype Antennas at ATF (2006)

Mitsubishi antenna (in Chile now…) Vertex prototype

antenna AEC prototype antenna

ALMA Test Facility (2008)

ObservationPreparation

Scheduling

Data ReductionPipeline

Archive

Executive

ALMA Common Software

PrincipalInvestigator

1. Create observing project

2. Store observingproject

3. Get projectdefinition

4. Dispatch scheduling block id

6. Start data reduction

8. Notify PI

7.1. Get raw data & meta-data

7.2. Store science results

9. Get projectdata

ArchiveResearcher

TelescopeOperator

f. Get science data

d. Notifyof

SpecialCondition

e. StartStop

Configure

c. Alter Schedule / Override action

Control System

Correlator

Calibration Pipeline

Quick Look Pipeline

5. Execute scheduling block

5.2 Setup correlator

5.3. Storeraw data

5.4. Storemeta-data

5.6. Store calibration results

5.7. Store quick-look results

Primary functional paths Additional functions ALMA software subsystem external agent

Real-time

a. Monitorpoints

b. Monitorpoints

5.5b. Access raw data & meta-data

h. Store admin data

g. breakpointresponse

5.5a. Access raw data & meta-data

5.1. Get SB

ALMA Software

Architecture

Software development at the ATF:

• Control antenna mount software• Access & display monitor points through out the system • Control/interface to correlator, backend, frontend hardware• Operator control & system status screens• Develop underlying software to support both interferometric

and total power observations• Schedule projects manually using scripts appropriate for

commissioning (python)• Schedule projects automatically with scheduling blocks (SB)• Archive & extract data• Test Observe Tool to create projects & control scripts • Fill, display & calibrate data (CASA)• Improve system build & integration procedures, robustness

& efficiency

November 17-21, Santiago 8ALMA Computing Review

The Evolution of Accomplishments at the ATF

Array Operator Interface

Antenna Mount Control

Interface

Early Accomplishments: December 2007

Correlator GUI dynamic Fringes on Venus. Top: autocorrelations for each antenna. Bottom left: amplitude versus channel. Bottom right: phase versus channel. (Using Darrell Emerson’s polynominal delay model, not Computing on-line (CALC) delay model.)

Dynamic Fringes on 3c454.4: Residual fringe rate as a function of time on a single channel (ch 125) at the center of the correlator bandpass.

Early Accomplishments: January 2008

First interferometric spectrum at the ATF using the evaluation front-end receivers and production back end equipment.

The Orion Hot Core centered on the CS(2-1) line at 98 GHz.

Credit: T. Hunter, R. Laing, and ALMA operators H. Alarcon & R. Aviles.

Early Accomplishments: January 2008

First ASDM generated at the ALMA Test Facility observing the ATF beacon - beacon signals clearly detected (highlighted by dashed lines)

Mid-term Accomplishments:

April 2008

Un-calibrated, raw spectrum from one ASDM dataset (including known “birdies”)

Edited & calibrated (gain, bandpass & flux) spectrum of 4 ASDM datasets.

Zoom in on amplitude and phase of 2 lines - good comparison between the 4 datasets.

Spectrum toward Orion (OMC 1) using tunable filter banks of the ALMA prototype correlator

Data filled, edited, calibrated, displayed in CASA.

Observations & reduction courtesy of Al Wootten & Remy Indebetouw.

Mid-term Accomplishments: June 2008

Vertex Sky Dip showing linear fit

Calibrated raster images of moon with Alcatel & Vertex antennas:

Raster map of the sun - measuring the spillover sidelobe response of the antenna.

(Response not centered on the sun because the feed is off-center).

Later Accomplishments: November 2008

Total power rasters, CASA reduction & imaging

Lessons learned

Organization

• The ATF was run as an Observatory with a strict schedule (not open-ended laboratory testing and development)

• Managed as a joint project (evolved over the course of about 4 months):

• Scheduling using a Google Calendar (set up by SE, embraced by Computing IPT)

• Roster of support from Chilean, Socorro & European developers, scientists and engineers (ATF site, BE, Systems Engineering).

• ATF journal (absolutely required!)

• Management reports with more than just Computing-centric information:• Management info - efficiency and effectiveness measures• Hardware/firmware issues relating to production:

• Many BE & antenna issues that affect production identified, summarized and now tracked by project.

• CIPT accomplishments, risks, development issues

Keep All Management Reports & Status on a Central Twiki

Key people contribute to twiki.

Only a few individuals manage content.

ATF schedule well defined, updated monthly

Scheduling must be dynamic and flexible:

Google calendar of activities for Computing development, ATF support roster, Operator support, Science activities

ATF Support roster – updated weekly

• Manager on duty every day (sometimes remote)• Extensive CIPT support (including now European developers at the ATF)

ATF Journal (twiki) to track activities &

issues

ATF weekly coordination

meetings track progress, priorities and solve cross-

disciplinary issues.

Organization not enough:A perspective shift was needed

All parties put in major effort to get this to work:

• CIPT:

• Early ATF: Hardware development was not complete. CIPT developed tools and threw them over the ‘wall’ to users to await feedback (often made in anger)

• End life: Hardware is a stable platform. CIPT is part of an integrated commissioning team that delivers needed functionality to our users and values their feedback which is constructive.

• Science IPT & SE:

• Early ATF: Software should be complete so the hardware could be tested.

• End life: all work together. The software is part of an integrated system, Science IPT and SE commission the software as well as the hardware and provide detailed feedback to developers as soon as possible.

• BE:

• BE took advantage early on to test out new equipment with software. Very pro-active and helpful.

Operations: team work

• Single points of contact responsible for the ATF: • Debra Shepherd for ATF management & final goals • Jeff Kern for Software deployment and ATF support• Alison Peck/Robert Laing for Science

• Communication between Computing IPT and deployment team in Chile (Computing: AIV, Ops) well-developed.

• A direct result of the fact that they have all been through the ATF & have worked with Socorro team - they are not “them” - they are “us”

• Testing and accepting software seen as a process rather than a single Milestone on a calendar.

• The project knew what to expect, rather than getting surprised at acceptance time.

Simulators are not enough

• Best tool to test interferometry is an interferometer.

• Simulation is useful to prepare for deployment but we consistently discovered a class of bugs at the ATF that we have been unable to replicate in simulation.

• These bugs are wide ranging involving timing issues, synchronization, problems only detectable with hardware (residual delays) and actual science data.

• Regression tests run every Saturday & Sunday to verify development and periodic execution at the ATF.

• ATF success shows Computing & Commissioning access to an interferometer is a must– On site software support shifts available seven days a

week. No extra effort to support OSF interferometer– Organization “Code” reused from the ATF:

• OSF Journal already in place• Chilean Computing Group and AIV/Science

participating in final release integration• Google calendar implemented to schedule

activities• Regular progress reports written with hardware &

software status

What’s Next?

Operations Support Facility (OSF) Interferometer

Sunset on the Last Day of the ATF

Questions?

Photo courtesy of Leonardo Testi