"Recent Advances in Data Mining and Applications for Heliophysics"

Recent Advances in Data Mining and Applications for Heliophysics

Kirk D. BorneGeorge Mason University and QSS Group Inc., [email protected] or [email protected]://rings.gsfc.nasa.gov/nvo_datamining.html

6/9/2006 http://rings.gsfc.nasa.gov/~borne/nvo_datamining.html 3

OUTLINE• The New Face of Science• Heliophysics (Data) Environment• Knowledge Discovery• Data Mining Examples and Techniques• Discovery Informatics for Large Database

Science– Heliophysics Example





The New Face of Science – 1

• Big Data (usually geographically distributed)– High-Energy Particle Physics– Astronomy and Space Physics– Earth Observing System (Remote Sensing)– Human Genome and Bioinformatics– Numerical Simulations of any kind– Digital Libraries (electronic publication

repositories)

• e-Science– Built on Web Services (e-Gov, e-Biz) paradigm– Distributed heterogeneous data are the norm– Data integration across projects & institutions– One-stop shopping: “The right data, right now.”


• Databases enable scientific discovery– Data Handling and Archiving (management of

massive data resources)– Data Discovery (finding data wherever they exist)– Data Access (WWW-Database interfaces)– Data/Metadata Browsing (serendipity)– Data Sharing and Reuse (within project teams; and

by other scientists – scientific validation)– Data Integration (from multiple sources)– Data Fusion (across multiple modalities & domains)– Data Mining (KDD = Knowledge Discovery in

Databases)

The New Face of Science – 2


The Promise of e-Science• The best of Google and Amazon.com

– Go to one place to shop for all your data needs

– Use scientific indexing (through scientific metadata)

– Find the data that you need

– Ignore data that are not relevant

– Recommend “also relevant” data sets

– Access distributed data seamlessly (transparently)

– Integrate multiple data sets

– Integrate data sets into analysis/visualization software packages

– Provide value-added services

– Provide intelligence within the archive

– Provide intelligence at the point of service





Sun-Earth Space Environment – Rich Source of Heliophysical

Phenomena


Multi-point Observations and Models of Space Plasmas Deliver a Deluge of Physical Measurements



Space Science data volumes aregrowing and growing and…

a few terabytes "yesterday” (10,000 CDROMs)

tens of terabytes "today” (100,000 CDROMs)

100’s of petabytes "tomorrow" (within 10-20 years) (1,000,000,000 CDROMs)


Technological Advances: the cause and the solution?


Data Access and Analysis Tools are Essential, but do not scale well with Exponential Data

Growth


The Data Flood is Everywhere!

• Huge quantities of data are being generated in all business, government, and research domains:– Banking, retail, marketing,

telecommunications, homeland security, computer networks, other business transactions ...

– Scientific data: genomics, space science, physics, etc.

– Web, text, and e-commerce

http://www.cultindustries.com/new/html/frame.html

6/9/2006 http://rings.gsfc.nasa.gov/~borne/nvo_datamining.html 16(Credit: Tim Eastman)





How do we learn about our Universe and the World around us?

WE GATHER INFORMATION,FROM WHICH WE DERIVE KNOWLEDGE,

FROM WHICH WE LEARN WHAT IT ALL MEANS

Data Information Knowledge Understanding / Wisdom!


Data-Information-Knowledge-Wisdom

• T.S. Eliot (1934):

“Where is the wisdom we have lost in knowledge?

Where is the knowledge we have lost in information?”


Understanding: the Universe is expanding!!

Astronomy Example

Data:

Information (catalogs / databases):– Measure brightness of galaxies from image (e.g., 14.2 or 21.7)– Measure redshift of galaxies from spectrum (e.g., 0.0167 or 0.346)

Knowledge:Hubble Diagram Redshift-Brightness

Correlation Redshift = Distance

(a) Imaging data (ones & zeroes) (b) Spectral data (ones & zeroes)


So what is Data Mining?

• Data Mining is Knowledge Discovery in Databases (KDD)

• Data mining is defined as “an information extraction activity whose goal is to discover hidden facts contained in (large) databases."





Data Mining• Data Mining is the Killer App for Scientific Databases.• Scientific Data Mining References:

– http://rings.gsfc.nasa.gov/nvo_datamining.html

– http://www.itsc.uah.edu/f-mass/

• Framework for Mining and Analysis of Space Science data (F-MASS)

• Data mining is used to find patterns and relationships in data. (EDA = Exploratory Data Analysis)

• Patterns can be analyzed via 2 types of models:– Descriptive : Describe patterns and to create meaningful subgroups

or clusters. (Unsupervised Learning, Clustering)– Predictive : Forecast explicit values, based upon patterns in known

results. (Supervised Learning, Classification)

• How does this apply to Scientific Research? … – through KNOWLEDGE DISCOVERY

Data Information Knowledge Understanding / Wisdom!


Data Mining is a core database function

• Data Mining has many names / aliases :– Knowledge Discovery in Databases (KDD)– Machine Learning (ML)– Exploratory Data Analysis (EDA)– Intelligent Data Analysis (IDA)– On-Line Analytical Processing (OLAP)– Business Intelligence (BI)– Customer Relationship Management (CRM)– Business Analytics– Target Marketing– Cross-Selling– Market Basket Analysis– Credit Scoring– Case-Based Reasoning (CBR)– Connecting the Dots– Intrusion Detection Systems (IDS)– Recommendation / Personalization Systems!


Data Mining is Ready for Prime Time• Why are Data Mining & Knowledge Discovery such hot topics? -- because of the

enormous interest in existing huge databases and their potential for new discoveries.

• Data mining is ready for general application because it engages three technologies that are now sufficiently mature:

1. Massive data collection & delivery

2. Powerful multiprocessor computers

3. Sophisticated data mining algorithms

• 5 Reasons to use Data Mining:

– Most agencies collect and refine massive quantities of data.

– Data mining moves beyond the analysis of past events … to predicting future trends and behaviors that may be missed because they lie outside the experts’ expectations.

– Data mining tools can answer complex questions that traditionally were too time- consuming to resolve.

– Data mining tools can explore the intricate interdependencies within databases in order to discover hidden patterns and relationships.

– Data mining allows decision-makers to make proactive, knowledge-driven decisions.


Examples of real Data Mining in Action• Classic Textbook Example of Data Mining (Legend?): Data mining

of grocery store logs indicated that men who buy diapers also tend to buy beer at the same time.

• Blockbuster Entertainment mines its video rental history database to recommend rentals to individual customers.

• Astronomers examined objects with extreme colors in a huge database to discover the most distant Quasars ever seen.

• Credit card companies recommend products to cardholders based on analysis of their monthly expenditures.

• Airline purchase transaction logs revealed that 9-11 hijackers bought one-way airline tickets with the same credit card.

• Wal-Mart studied product sales in their Florida stores in 2004 when several hurricanes passed through Florida. Wal-Mart found that, before the hurricanes arrived, people purchased 7 times as many strawberry pop tarts compared to normal shopping days.


Strawberry pop tarts???


Mega-Flares on normalSun-like stars = a star like

our Sun increased in brightness 300X one night!

… say what??

Exploringthe Time Domain

Astronomy Data Mining in Action


ClusteringClassificationAssociationsNeural NetsDecision TreesPattern RecognitionCorrelation/Trend AnalysisPrincipal Component AnalysisIndependent Component

AnalysisRegression AnalysisOutlier/Glitch IdentificationVisualizationAutonomous AgentsSelf-Organizing Maps (SOM)Link (Affinity Analysis)

Data Mining Methods and Some Examples

Classify new data items usingthe known classes & groups

Find unusual co-occurring associationsof attribute values among DB items

Organize information in the database based on relationships among key data descriptors

Identify linkages between data items

based on features shared in common

Group together similar items andseparate dissimilar items in DB

Predict a numeric attribute value


Some Data Mining Techniques Graphically Represented

Self-Organizing Map (SOM)

Outlier (Anomaly) Detection

Clustering

Link Analysis Decision Tree

Neural Network


Data Mining Application: Outlier Detection

Figure: The clustering of data clouds (dc#) within a multidimensional parameter space (p#).

Such a mapping can be used to search for and identify clusters, voids, outliers, one-of-kinds, relationships, and associations among arbitrary parameters in a database (or among various parameters in geographically distributed databases).

• statistical analysis of “typical” events• automated search for “rare” events


Outlier Detection:Serendipitous Discovery of

Rare or New Objects & Events


Learning From Legacy Temporal Data (Time Series):Classify New Data (Bayes Analysis or Markov Modeling)


Principal Components Analysis &Independent Components Analysis

Cepheid Variables:Cosmic Yardsticks-- One Correlation-- Two Classes!


Classification Methods:Decision Trees, Neural Networks,

SVM (Support Vector Machines)

There are 2 Classes!

How do you ...-Separate them?-Distinguish them?-Learn the rules?-Classify them?

ApplyKernel

(SVM)


Data Mining: For Exploration, Discovery,

and Decision Support (in science, government, homeland security, and

business)


Sample Space Science Data Mining Use Cases•Discover data stored in geographically distributed heterogeneous systems.•Search huge databases for trends and correlations in high-dimensional parameter spaces: identify new properties or new classes of scientific objects.

•Discover new linkages & associations among data parameters.•Search for rare, one-of-a-kind, and exotic objects in huge databases.•Identify repeating patterns of temporal variations from millions or billions of observations.

•Identify parameter glitches / anomalies / deviations either in static databases (e.g., archives) or in dynamic data (e.g., science / instrumental / engineering data streams).

•Find clusters, nearest neighbors, outliers, and/or zones of avoidance in the distribution of objects or other observables in arbitrary parameter spaces.

•Serendipitously explore huge scientific databases through access to distributed, autonomous, federated, heterogeneous, multi-experiment, multi-institutional scientific data archives.





Existing Space Science Data Infrastructure• The Recent Past: many independent distributed heterogeneous data archives

• Today: VxO’s = Virtual Observatories– Web Services-enabled: e-Science paradigm (middleware, standards, protocols)**– Provides seamless uniform access to distributed heterogenous data sources

• “Find the right data, right now”• “One-stop shopping for all of your data needs”

– Emerging environment consists of many VxO’s – for example:• NVO = National Virtual Observatory (precursor to VAO = Virtual Astro Obs)• VSO = Virtual Solar Observatory• VSPO = Virtual Space Physics Observatory• NVAO = National Virtual Aeronomy Observatory• VITMO = Virtual Ionospheric, Thermospheric, Magnetospheric Observatory• VHO = Virtual Heliospheric Observatory• VMO = Virtual Magnetospheric Observatory

• ** Standards for data formats, data/metadata exchange, data models, registries, Web Services, VO queries, query results, semantics

• ** And of course: The Grid, Web Services, Semantic Web, etc. ...


Our science data systems should enable distributed multi-mission database access, discovery, mining, and analysis.

DISCOVERY INFORMATICS


What is Informatics?• Informatics is the discipline of structuring,

storing, accessing, and distributing information describing complex systems.

• Examples:1. Bioinformatics2. Geographic Information Systems (= Geoinformatics)3. New! Discovery Informatics for Space Science

• Common features of X-informatics:– Basic object granule is defined– Common community tools operate on object granules– Data-centric and Information-centric approaches– Data-driven science– X-informatics is key enabler of scientific discovery in the era of

large data science


X-Informatics Compared

Discipline X• Bioinformatics

• Geoinformatics

• Space Science Informatics

Common Tools• BLAST, FASTA

• GIS

• Classification, Clustering, Bayes Inference, Cross Correlations, Principal Components, ???

Object Granules• Gene Sequence

• Points, Vectors, Polygons

• Time Series, Event List, Catalog


Discovery Informatics

• Key enabler for new science discovery in large databases

• Essential tool (Large data science is here to stay)

• Common data integration, browse, and discovery tools will enable exponential knowledge discovery within exponentially growing data collections

• X-informatics represents the 3rd leg of scientific research: experiment, theory, and data-driven exploration (Reference: Jim Gray, KDD-2003)

• Discovery Informatics should parallel Bioinformatics and Geoinformatics: become a stand-alone research sub-discipline


Key Role of Data Mining

• Data Mining (KDD) is the killer app for scientific databases

• Space and Earth Science Examples:– Neural Network for Pixel Classification: Event Detection and

Prediction (e.g., Wildfires)

– Bayesian Network for Object Classification

– PCA for finding Fundamental Planes of Galaxy Parameters

– PCA (weakest component) for Outlier Detection: anomalies, novel discoveries, new objects

– Link Analysis (Association Mining) for Causal Event Detection (e.g., linking Solar Surface, CME, and Space Weather events)

– Clustering analysis: Spatial, Temporal, or any scientific database parameters

– Markov models: Temporal mining of time series data


Space Science Knowledge Discovery


This is the Informatics Layer


This is the Informatics Layer

Informatics Layer:• Provides standardized

representations of the “information extracted” – for use in the KDD (data mining) layer.

• Standardization is not required (nor feasible) at the “data source” layer.

• The informatics is discipline-specific.

• Informatics enables KDD across large distributed heterogeneous scientific data repositories.


Space Weather Example

CME = Coronal Mass EjectionSEP = Solar Energetic Particle


Key Role of Discovery Informatics

• The key role of Discovery Informatics is :– ... data integration and fusion ...

– ... across multiple heterogeneous data collections ...

– ... to enable scientific knowledge discovery ...

– ... and decision support.


Future Work: Discovery Informatics Applications• Query-By-Example (QBE) science data systems:

1. “Find more data entries similar to this one”2. “Find the data entry most dissimilar to this one”

• Automated Recommendation (Filtering) Systems:1. “Other users who examined these data also retrieved the following...”2. “Other data that are relevant to these data include...”

• Information Retrieval Metrics for Scientific Databases:1. Precision = “How much of the retrieved data is relevant to my query?”2. Recall = “How much of the relevant data did my query retrieve?”

• Semantic Annotation (Tagging) Services:– Report discoveries back to the science database for community reuse

• Science / Technical / Math (STEM) Education:– Transparent reuse and analysis of scientific data in inquiry-based

classroom learning (http://serc.carleton.edu/usingdata/ , DLESE.org )

• Key concepts that need defining (by community consensus): Similarity, Relevance, Semantics (dictionaries, ontologies)



(science knowledge sharing & re-use)

(*** Repositories of information,knowledge, and scientific results.)

(***)

(***)


Informatics: Synergy between Scientific Measurement, Mining, and Modeling


Data Mining and Discovery Informatics:It is more than just connecting the dots

Reference: http://homepage.interaccess.com/~purcellm/lcas/Cartoons/cartoons.htm

Documents

"Recent Advances in Data Mining and Applications for Heliophysics"