The Universe. As it Happens. Fragmentation: Astronomys Data
Problem Dexter Jagula
Slide 2
2 Overview Introduction Fragmentation
Slide 3
3 Fin Designing a Solution Our Vision Our Roadmap
Slide 4
4 A little about me
Slide 5
5 Perceived Problems FRAGMENTATION Heterogeneity Data is
available from a multitude of sources Siloed Data No reuse of data
for other science objectives Storage Inconsistent methods of how
data is stored Archival Data preservation is usually an
afterthought Sharing Inefficient methods are used to share data
Collaboration Third-party tools dont cater to astronomers
8 NASA Space Apps Win Created a platform for public access to
astronomy data. SaaS Solve single challenges to create a suite of
services to be offered to the community. New Surveys Projects like
LSST and SKA will provide a firehose of data for the community to
manage. Data Aggregator Collecting catalogued data and providing a
single-point access. How did we get here?
Slide 9
9 Built-in collaboration tools throughout all services
One-click sharing capability on any datasets, files, or feeds Full
and complete control of data SkyWatch Services Intelligent
Filtering Aggregating data from multiple sources Filtering false
positives Curating data for specific science objectives Utilizing
machine learning for real-time classification of incoming data Raw
File Archival Data Processing Tools Collaboration Tools Utilizing
Hadoop for distributed file storage Ability to tag data Query on
metadata extracted from FITS files Utilizing Google Cloud
Uploading/importing datasets Leveraging Apache Spark for ad-hoc
analysis and data reduction Implementing machine learning
techniques on data Utilizing Google Cloud
Slide 10
10 Principal Investigator Unprecedented access to curated data
Collaborator Annotate and share data easily and efficiently
Follow-up Observatory Direct alerts for easy prioritization Author
Perform analysis using modern technologies Data Ensure preservation
Survey Effective brokering of data Advances for
Slide 11
11 Our Vision ? ? ? ?
Slide 12
12 01 Build-up the R/T Feed Add search capability
Mobile-friendly APR 2015 - JUN 2015 02 Observatories can publish to
the R/T Feed Socket connection to the Feed Roles and permissions
Image processing JUL 2015 - SEP 2015 03 Collaboration tools
built-into the R/T Feed Machine learning classifier used for image
data OCT 2015 - DEC 2015 04 Improvements to the R/T Feed and image
processing Enhanced roles and permissions Tabular data storage and
processing JAN 2016 - MAR 2016 05 Continuous improvement to all
facets Increased workbench capabilities Ability to integrate
workbench and intelligent filtering Tools for theorists APR 2016
-
Slide 13
13
Slide 14
14
Slide 15
15 Acknowledgements Iair Arcavi Scott Barthelmy Guillaume
Belanger Eric Bellm Federica Bianco Todd Boroson Peter Brown Yi Cao
Hsin-Yu Chen Eric Christensen Diana Dragomir Simon Hodgkin Andy
Howell Tim Jenness Kyler Kuehn Andrej Prsa Sumner Starrfield Rachel
Street Jonathan Swift John Swinbank Brad Tucker Stefano Valenti
Giacomo Vianello LCOGT NASA-GSFC ESA-ESAC Caltech NYU LCOGT
Mitchell Institute / Texas A&M Caltech University of Chicago
University of Arizona LCOGT Institute for Astronomy, Cambridge
University LCOGT LSST Australian Astronomical Society Villanova
University ASU/Earth and Space Exploration LCOGT The Thacher School
Princeton Mt. Stromlo Observatory, ANU / UC Berkeley LCOGT Stanford
university