Data Collection Strategies, Methods and

Standardization

July 24, 2006

Jeff Mossey

Assistant Director, Kentucky NSF EPSCoR

Brief Review of Typical Activities

• Annual Reporting (Fastlane)• State Reporting (likely different)• Brochure Annual Report• Nuggets/Success Stories• Newsletters• Online Reporting Systems • Diversity and Evaluation Reports/Activities• Reverse Site Visits

Agenda• Update on Evaluation Activities• Identify Some (Evaluation) Problems and

Possible Solutions• Demonstrate the “EPSCoR Connections” Site

– Strengths/Weaknesses/Improvements

– Is this something that we can do as a program?

• Open Discussion of Issues/Concerns/Ideas• Present it to NSF; Nat. Conf. Breakout

Status of Evaluation Activities

• Report from Norman Webb coming soon• “Three Phases” of activities:

Phase I – (Conceptual) Framework• Conduct Workshops• Establish Indicators• Develop Report

Phase II – (Detail) Development • Establish metrics and standards• Find out what data is available

Phase III – Implementation and Testing

Six Major Program Indicators

1. Human Resource Development

2. Research Production

3. Research Portfolio Quality

4. Research Investments and Materials

5. Research Collaboration and Networking

6. Research Climate, Culture, and Communications

Norman’s Comments

• Data availability? Both nationally and at the state-level (e.g. funding/publications)

• Fastlane/EPSCoR Database Relationship– Common System or Common File Format

• Strength of the Connection ???– Attribution to the project (Direct/Indirect)

• Timing of the Reporting (Fed. Fiscal Year)

Problems We Must Overcome

• LSAMP Program = – (Goal) Increasing the graduation rate of underrepresented

students in the sciences.– (Metric) Did they go to grad school?

• EPSCoR Program = – (Goal) Research Infrastructure Development– (Goal) Increased Grant Funding (Competitiveness)– (Goal) Diversity– (Goal) Economic Development– (Goal) Outreach and Education (K-12)– (Goal) Networking and Strategic Development with Industrial

Governmental, and Academic Partners

Problems We Must Overcome

EPSCoR Investment/ Activities

Meaningful, Aggregated Success Stories

1.) Use common metrics, stories, and techniques for gathering success indicators

2.) Use common systems for collecting the data

3.) Gather the right metrics/stories

Two Layers of Success

• Aggregated, Impressive Numbers

“Annually over 200,000 people participate in this program” -LSAMP Brochure

• Nugget Short Stories

http://www.kynsfepscor.org/test/EpscorTest/test.php

Weaknesses

• Won’t accommodate “other” successes (e.g. NIH awards, Nobel Prizes)

• Progress toward milestones and other “narrative” updates will still be needed

• Programs newer to EPSCoR will have fewer stories

Strengths• Success stories are program-wide and

standardized• Tracing backward, or “backing into” a story is

easier than constant monitoring “outward”• Attribution to EPSCoR can vary• Minimal time commitment• More dated stories aren’t neglected• By definition, connects EPSCoR to successful

outcomes … follow-on awards = increased competitiveness

• Aggregate connections to show an impressive percentage of involvement.

• Stories (i.e., history of EPSCoR investments) is archived

Next Steps … Be Proactive!

• Is this something we want to do?– Standardizing the story gathering process?

Develop “users manual”

• Document other evaluation comments

• Share with Norm/Sherry