ISTA 520/420Midterm

AJ JichaRyan JichaIan KauferAJ StanglRoy Zacharias

“Astrocats” - Group 3

AgendaAstroCats Staff

Project Summary

Project Goals

Results

Data Processing Runtime

Obstacles

Future Goals

AstroCats StaffProject Managers Ian Kaufer, Roy Zacharias

Chief Architect AJ Jicha

System Admins Ian Kaufer, AJ Jicha

Development/Test Manager AJ Stangl

Release Manager AJ Stangl

Chief Programmer Ryan Jicha

Technical Writer Ian Kaufer, Roy Zacharias

Project Summary• Carl Herzenrother - requesting images processed using HPC for efficiency• Input: FITS files• Process:

• Python and Bash scripts• Generate modified FITS file• Generate Astrometrica config file• Linux, HPC, makeflow, Astrometrica, Astrometry.net

• Output:• Modified FITS image file• Astrometrica .cfg file for associated FITS image

Project Goals

• Sort files with varying extensions (i.e. fit, fitz, .zip) into a common directory for processing

• Process each .fits file with solve field in a parallelized manner on HPC• Modify each new .fits file with calculated parameters from solve field (Field

center values)• Create Astrometrica configuration file with modified field rotation (PA) and

calculated focal length to match each new .fits file • Streamline process for easy replication by client on future datasets

Project Work Flow

Results• Scripts to process images and metadata functional• .makeflow script to process FITS images functional• Leveraging HPC/HTC and workers to process many tasks functional• Output files located in the following directories:

• /gsfs1/xdisk/ikaufer/midterm/midterm-Carl-Hergenrother/output/• Generate new FITS images functional• Generate .cfg files for FITS images functional• Generate list of unsolved/error files functional

Data Processing Runtime• Benchmark:• 40 workers, 222 jobs: 2 hours 51 minutes

• Desired time:• 1,500 workers, 9,570 jobs: 2 hours 29 minutes

• Actual time:• 524 workers, 9,570 jobs: X hours Y minutes

Obstacles• Issues running makeflow• .fits metadata in a single huge line• Three of the five group members did not have any standard HPC hours• Lack of uniform file extension naming scheme • Random .jpeg and .mp4 files• Spaces within file names

Example Obstacle Findings

Future Goals• Implement procedure to handle multiple pixel scale values

• Unify all parts of scripts into a single script for ease of use

Questions?