CCMC Operations Support, Metrics and V&V Report

http://ccmc.gsfc.nasa.gov

M. Kuznetsova, A. Taktakishvili, P. MacNeiceM. Hesse, L. Rastaetter, A. Chulaki

CCMC Workshop,

Arecibo 2007

CCMC Function

• Provide tool by which science progress at NASA, NSF feeds into Space Weather operations

• Perform independent and unbiased testing and validation of science models with operational benefits

Outline

• Operations Support Activities• Role of Runs on Request System Users in V&V• Examples of Metrics and V&V

• Other Metrics Opportunities and Future Plans

• Outlook

Model Testing and Validation Components

• Science-based validation• Test model validity• Detailed analysis for selected events, broad range of parameters

• Broad feedback to code developers

• Essential for further model improvement

• Metrics • Measure model usefulness for operations in comparison with some

simple standard model.

• Focus on parameters useful for operations

• Repeatable comparison between model output and measurements.

• Blind studies

• Test model robustness

L. Rastaetter, P. MacNeice

• Experimental real-time simulations 24/7- http://ccmc.gsfc.nasa.gov• BATSRUS/SWMF (2001)• Fok ring current (2003)• Chain of solar and heliospheric models (WSA, PFSS, ENLIL) (2006)

Operations Support Activities:Model Testing for Operational Environment

• Quasi-operational real-time tools (tailored to NOAA SWPC specifications)- http://ccmc.gsfc.nasa.gov/

cgi-bin/display/RT_t.cgi

Operations Support Activities:R2O Workshop support

A. Chulaki, M. KuznetsivaL. Rastaetter, P. MacNeice

• Releases of historic event simulations in “real-time”- http://ccmc.gsfc.nasa.gov/R2O

• Expose models to operators• Help to tailor model output to operator

needs- Help to define what visualization is useful- Help develop templates for future real-time

pages- Tool for feedback

2001 – 2003: ~ 200 requests, 2004 – 2005: ~ 400 requests,

2006 – 2007: ~ 1100 requests, 60 publication/presentationsinformal feedback from users

Role of RoR Users in V&V

Development/utilizing of “ready-for-validation” tools:- output along trajectories- time series- vertical profiles

Simulation setup according to user specifications

8

• Comparison of the ENLIL model simulation results with the ACE satellite data. • Both simulation and observation data are smoothed over 6 hour period for entire CR time interval. • Mean model as reference

• Skill score calculation for - different input coronal models, - different observatory magnetograms.

ENLIL Background SW Model Metrics

mean

mod

DD=M −1

S. Taktakishvili

9

Comparison of the ENLIL results to the ACE data for the randomly selected CR-s

CR 2012

CR 2017 CR 2018

CR 2014

10

Skill Score statistics (Skill Score vs CR number)

• Model performs much better than mean model for all parameters.• Velocity has the highest skill score, followed by magnetic field magnitude, then temperature and density at the end.

Coronal model:MAS

Input magnetogram: NSO

S. Taktakishvili

11

Skill Score for different inputsolar corona models - MAS and WSA

Model performs slightly better for the WSA than for MAS.

S. Taktakishvili

Input magnetogram: NSO

12

Skill Score for different input magnetograms -MWO and NSO

Model performs slightly better for the Kitt Peak Observatory (NSO) than for the Mount Wilson Observatory (MWO) input magnetograms.

Input coronal model: WSA

S. Taktakishvili

Skill Score with respect to Persistence Model

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

0 50 100 150 200

Interval (hours)

Skill

sco

re

S. Francis, S. Taktakishvili

Persistence interval 48 hours Skill score as a function of persistence interval

Enlil Cone model input is generated using a set of consecutive SOHO LASCO C3 Running Difference images (minimum two)

Enlil Cone Model (D. Odstrcil)

S. Taktakishvili

• CME arrival time prediction• Strength of impact• Duration of activity

Enlil Cone Model Prediction Validation

• Future plans: Event prediction - % of time magnitude exceeds a threshold- % of time magnetopause inside geosynch. orbit- % of time Bz < 0

ENLIL Cone Model CME arrival time prediction

Reference Model:ballistic propagation with the average velocity(average of Halo CME speed from the CME catalogue).

Vavg=850 km/s

Average propagation time to the ACE satellite:

Tavg (prop) = 48 hours

Shown here: arrobs

arrenlil tt −,tt arr

obsarrrefer −

Arr

ival

tim

e er

ror (

abs.

val

ue in

hou

rs)

4/24/99 6/01/99 11/8/00 3/29/01 10/9/01 4/21/02 8/1602 8/24/02 10/29/03 12/13/06 --0

2

4

6

8

10

12

14

16

18

20

22

24

reference modelENLIL

EVENT

S. Taktakishvili,S. Krog,P. MacNeice

Future plans:

- Error bars due to uncertainty in cone model parameters- Utilize STEREO SWAVES data to better estimate CME speed

Strength of Impact

0

50

100

150

200

250

300

VACE(μ0nACEmp)

1/2

12/16/0612/15/06

Time12/14/06

Venlil(μ0nenlilmp)

1/2

S. Taktakishvili

• Future plans: Time intervals of southward (Bz < 0)

Magnetic field strength required to stop the SW

mp nsw Vsw2 = (1/μ0) B*2B*

[nT]

SW ram pressure

Estimated magnetic field at the magnetopause

Estimated Maximum Magnetic Field at the Magnetopause

4/24/996/01/9911/8/003/29/0110/9/014/21/028/1602 8/24/0210/29/0312/13/060

50

100

150

200

250

300

350

400

450

ENLIL simulationACE observation

EVENT

nT

Estimated Minimum Magnetopause Standoff

ENLIL simulationACE observation

4/24/99 6/01/99 11/8/00 3/29/01 10/9/01 4/21/02 8/1602 8/24/02 10/29/03 12/13/060

2

4

6

8

10

EVENT

S. Taktakishvili

ENLIL Cone Model CME duration time prediction metrics

Reference duration:

Mean duration of the ram pressure pulse at ACE observed for 10 studied CME-s:

ΔTrefer = 24 hours

Shown here: ,TT obsrefer

TobsΔ

Δ−Δ

obs

obsenlil

TTT

Δ

Δ−Δ

0

10

20

30

40

50

60

70

80

90

100

110

120

130

140

150

reference ΔTrefer

=24 hENLIL

4/24/99 6/01/99 11/8/00 3/29/01 10/9/01 4/21/02 8/1602 8/24/02 10/29/03 12/13/06

Event date

% Duration time error w/respect to TobsΔ

S. Taktakishvili

21

Coupled BATSRUS + Fok RCModels: 1-st event – directly SW driven RC

January 21-22, 2005

- 3 0

- 2 0

- 1 0

0

1 0

2 0

3 0

Bz (

nT

)

U T2 2 /1 2 : 0 0

2 1 / 2 4 : 0 02 1 /1 2 : 0 0

- 1 0 0 0

- 8 0 0

- 6 0 0

- 4 0 0

Vx (

km

/s)

0

2 x 1 0 5

4 x 1 0 5

6 x 1 0 5

8 x 1 0 5

Te

mp

era

ture

(K

)

0

1 0

2 0

3 0

4 0

5 0

6 0

De

ns

ity

(

cm-3

)

100

101

102

103

104

105

106

SW input to BATSRUS model January 21-22, 2005 “CAWSES” event

Flu

x(pr

oton

s /s

m2 s s

r keV

)

LA N L-97A , Janua ry 21 -22 , 2005 P 75-113k eV P 113-170keV P 170-250keV P 250-400keV

100

101

102

103

104

105

106

321

92.06 keV 138.6 keV 206.16 keV 300.0 keV

Flu

x(pr

oton

s /s

m2 s s

r keV

)

U T01/22 1 2:0 001 /21 24:0001/21 1 2:00

FR C to LA NL -9 7A , Jan .21 -2 2, 2 005

LANL-97A observations &FRC model output for proton fluxes for 5 energy channels

Model performance isgood (at least qualitatively)

S. Taktakishvili

22

SW input to BATSRUS model August 10-11, 2000 event

100

101

102

103

104

105

106

107

LANL-1994-084 observations &FRC model output for proton fluxes for 5 energy channels

2-nd event – quasi steady Bz<0, “sawtooth” RC oscillations

Flux

(pro

tons

/sm2 s

sr k

eV)

LANL-1994-084, Aug.10-11, 2000 P50-75keV P75-113keV P113-170keV P170-250keV P250-400keV

100

101

102

103

104

105

106

107

61.24 keV 92.06 keV 138.60 keV 206.16 keV 300.00 keV

Flux

(pro

tons

/sm2 s

sr k

eV)

UT08.11/06:0008.10/24:0008.10/18:00

FRC to LANL-1994-084 , Aug.10-11, 2000

- 3 0

- 2 0

- 1 0

0

1 0

2 0

3 0

B z ( n

T )

1 1 /0 6 : 0 0U T

1 0 / 2 4 :0 01 0 /1 8 : 0 0

- 1 0 0 0

- 8 0 0

- 6 0 0

- 4 0 0

Vx (

km

/s)

0

2 x 1 0 5

4 x 1 0 5

6 x 1 0 5

8 x 1 0 5

Te

mp

era

ture

(K

)

August 10-11, 2000

0

1 0

2 0

3 0

4 0

5 0

6 0

De

ns

ity

( c

m-3 )

Model responseis poor evenqualitatively

S. Taktakishvili

Some internal dynamics is missing

Oct 24-29, 2003 event - snapshot

A. Pulkkinen

Dashed line: new UT day

Highly fluctuating IMF

Several moderately negative “sweeps”

A. Pulkkinen

Oct 24-29, 2003 event - the driver

Oct 24-29, 2003 event - GICG

ree n

l an d

IMA

GE

Comparable magnitudes

However, in general, the qualitative spatiotemporal morphology perhaps surprisingly similar.

Some (internal?) dynamics missing A. Pulkkinen

Dynamic Solar Wind Input

Observations at geostationary orbit Simulations

Simulations

QuickTime™ and aGIF decompressor

are needed to see this picture.

QuickTime™ and aGIF decompressor

are needed to see this picture.

Ideal MHD with Numerical Dissipation for Steady Southward IMF Driving

Observations at geostationary orbit(GOES): quasi-periodic oscillations: 2 hours

Input: quasi-steadysouthward IMF driving

Steady Solar Wind Input

Signature at Geostationary Orbit: X = - 6.6 RE , Z = 0

Bz[ nT ]

85.0

90.0

95.0

100.0

105.0

120 160 200 240 280Time

M. Kuznetsova

Non-MHD effects are important

Global MHD simulations with kinetic corrections at magnetotail reconnection sites

SW Input: Average SW parameters during August 10, 2000 sawtooth event

• Continue to follow National Space Weather Program Implementation Plan guidelines.

• Validation of other models (GAIM, AbbyNormal,…). Use Arecibo data.

• Focus on parameters most useful to operations.• Work with operators to identify suitable visualization.• Work with operators to identify suitable metrics. • Explore event prediction metrics.• Development of reusable V&V and metrics software.• Expand RoR System to benefit V&V studies.• Continue working with model developers to improve model

performance.

We welcome suggestions!

Future Plans

0 20 40 60 80 100 120 1404

8

12

16

2002/04/14 12:00

Nm

F2 [1

011 m

-3]

time [hours]

Arecibo [18.3, 293.2] Jicamarca [-11.96, 283.13]

SAMI2CTIP

Electron Density Parameters: Vertical Profiles, NmF2, TEC

Observations: Incoherent Scatter Radars, GPS, Ionosonds

Models: CTIP, SAMI2, GITM2, GAIM

Other Metrics Opportunities.Ionospheric Forecasting.

Supplementary Slides

Solar magnetogram + magnetic connection (WSA, ENLIL)

• The sun’s surface magnetic field and an estimate of the footpoints of the fieldlines connecting the 4 inner planets to the sun at a specific time. (Earth is *)

• Future plans: Footpoints on visible disk. Testing & validation.

P. MacNeice, B. Elliott

34

Metrics: Skill Score for modelling with Lowand High spatial resolution

Model performs better for Low spatial resolution than for High spatial resolution simulation. This is true for all the parameters and almost for all simulated CR-s.

Input model: WSAInput observ: NSO

35

Event Prediction Metrics: Forecast Ratio

2x102 4x102 6x102 8x1020.01

0.1

1

10

100

30 min 15 min

For

ecas

t Rat

io

Flux threshold (protons/sm2 s sr keV)

250 keV - 400 keV Channel

60 min

2x104 4x104 6x104 8x1040.01

0.1

1

10

100

60 min 30 min 15 min

For

ecas

t Rat

io

Flux threshold (protons/sm2 s sr keV)

75 keV - 113 keV Channel

15,30 & 60 min windows, 2 energy channels

• Event: within a forecast window of length ΔΤ flux exceeds a threshold Fthr

• Correct forecast: both model and observation exceed Fthr at least once within a window

• False alarm if model predicts F>Fthe while observed flux never exceeds Fthr

• Forecast ratio: Ncorrect/Nfalse

ENLIL CME arrival time prediction metrics:Skill Score

arrobs

arrrefer

arrobs

arrenlil

tt

tt

ScoreSkill

−

−−

=

1

4/24/99 6/01/99 11/8/00 3/29/01 10/9/01 4/21/02 8/1602 8/24/02 10/29/0312/13/06 ---5

-4

-3

-2

-1

0

1

Skill Score=1-|Δt|enlilerr /|Δt|ref

err

Ski

ll S

core

EVENT

37

Skill Score w/respect to the Mean and Persistence Models

1 2 3 4 5-4

-3

-2

-1

0

1

mean pers. 25 min pers. 50 min pers. 100 minSk

ill S

core

for L

og(F

lux)

- 2-

nd in

terv

al

Energy Channel1 2 3 4 5

-4

-3

-2

-1

0

1

mean pers. 25 min pers. 50 min pers. 100 min

Skill

Sco

re fo

r Flu

x -2

-nd

inte

rval

Energy Channel

-4

-3

-2

-1

0

1

mean pers. 25 min pers. 50 min pers. 100 minSk

ill S

core

for L

og(F

lux)

- 1-

st in

terv

al

-4

-3

-2

-1

0

1

mean pers. 25 min pers. 50 min pers. 100 min

Skill

Sco

re fo

r Flu

x - 1

-st i

nter

val

-80

-60

-40

-20

0

meanpers. 25 min pers. 50 minpers. 100 min

1 2 3 4 5-12

-10

-8

-6

-4

-2

0

Ski

ll S

core

for L

og(F

lux)

-3-

rd in

terv

al

meanpers. 25 min pers. 50 minpers. 100 min

Energy Channel

Ski

ll S

core

for L

og(F

lux)

-w

hole

inte

rval

Skill Score for:1 - pre storm, quiet interval2 - active interval

Skill Score for:3 - post active interval &

the whole interval

S. Taktakishvili

open open FLFL Polar cap boundary observed by POLARVIS Earth Camera

Polar Cap BoundaryBoundary between open and closed field lines

open open FLFL

closedclosed FLFL

SEP have free access along the open field lines

Energetic particles precipitation from the tail

L. Rastaetter

Geomagnetically Induced Total Electric Field (GIE)Geomagnetically Induced Currents (GIC) Proxy

Quiet Storm

A. Pulkkinen

BATSRUS FL tracing: 0.226 BATSRUS FAC: 0.149 OpenGGCM FL tracing: - 0.137 OpenGGCM FAC: - 0.252Weimer-2K FAC: - 0.473

BATSRUS

Mean Skill Score

SEP access to the magnetosphere

L. Rastaetter

Observations in tail lobes (Geotail):quasi-periodic TCR (multiple plasmoids)

Input: quasi-steadysouthward IMF driving

J. Slavin

Global Ionosphere Models

Global Assimilation of Ionospheric Measurements (USU-GAIM, Schunk et al.)

USU-GAIM uses a physics-based Ionosphere Forecast Model (IFM).USU-GAIM assimilates electron density (Ne) profiles from a diverse set of real-time (or near real-time) measurements:

- slant TEC from GPS ground stations via RINEX files, - bias information for GPS satellites and ground-stations,- electron density profiles from DISS ionosondes via SAO files

AbbyNormal Model (Eccles et al., Space Environment Corporation) calculates absorption values for HF signals

Ionosphere Forecast Model (IFM) needs - F10.7, daily Ap, 3-hour Kp indices

` - neutral wind, electric field, auroral precipitation, solar EUV (empirical inputs)

USU-GAIM Model: Total Electron Content (TEC)

Normal TEC

Increased TEC

AbbyNormal Model: Absorption Values for HF Signals

Normal Absoption

Vertically Integrated Signal Loss in Decibels for 5 and 10 MHz HF Signals

Increased Absoption