Assessing Research Impact

Teresa Clarke PhD

Director of Research Solutions

Serkan Bilgi MBA

Senior Business Analyst

Objectives

• Ways you can use your SmartSimple system to measure the impact of your research projects.

• Learn how to capture research metrics and KPIs to maintain stakeholder engagement.

Agenda

• Motivation of Research Funding Agencies• Gathering Key Performance Indicators (KPIs)

– Classes and Demo

• Utilizing Information– Portfolio analysis for Governance– Streamlining reporting for Researcher– Identifying success for Public

• Conclusions

Motivation of Funding Agencies

Funding Agency

Governance• ROI and

Outcomes• Governance• Accountability

Researcher• Simple

reporting• Award renewal

Public• Media• Education• Entertainment

Differences between RMS360⁰ and GMS360 ⁰

• Research funding agencies award large amounts of funds over multiple years ($1-5M over 4 years).

• Increased frequency of progress reports (annual or more frequently).

• Increased detail of monitoring.

Performance Measurement

• Reporting requirements are included in the award contract.

• Progress reports are required to document grantee accomplishments and compliance with terms of award.– Describe scientific progress– Identify significant changes to project plan– Describe plans for the next budget period– Report on personnel– Number of accomplishments

Measurable Impact

• Driven by the program deliverables.• Short term objectives:

– Job creation– Retaining international caliber talent– Building research infrastructure

• Long term goals:– Public policy and services– Economic development– Developing new therapies

Program Deliverables• Fits within the strategic plan of the agency.• Private: Industry or Charity

– Specific research area or to generate knowledge for the public good

• Public: Government– Reflect national research priorities

Captured in SmartSimple on the Program

Level.

Project Milestones • Relate to the program deliverables.• SMART

– Specific, Measurable, Attainable, Realistic and Timely

Captured in the project application form and progress reports.

Key Performance Indicators (KPIs)

• Job creation• Publications

– High impact journals– Other products to government, communities of interest and the public

• Additional investment or partnerships– Leveraged Funding– Partner funding

• International caliber performance– Awards, collaborations, conferences, committees

• Commercialization• Protection of Intellectual Property (IP)

Captured as individual records on progress report.

Measuring Impact using KPIs

• Live demo of Research Metrics in SmartSimple.

Portfolio AnalyticsType Indicator Comments

Level of Funding Summary Overview containing the amount of funding and distribution across CanadaDistribution across burden of disease

Breakdown of awarded amount of funding by disease area in each provinceSum of funding by provinceBreakdown of number of projects and amount of funding by province only

Competition Summary Number and range of targeted funding competitionsNumber of applications in each competition 

Breakdown of number of funded projects by disease area in each competitionSum of funding per competition

Team count per competition

Number of teams funded in each priority area 

Breakdown of count of teams in each disease area in each competitionSummary count of number of teams in each competition 

Performance Metrics Number of publications, presentations and awards, etc.

Count of performance metrics (number of publications, presentations and awards, spin-offs etc.) per projectSummary count of each performance metric

Publication impact Number of significant research advancements

Count of high impact journal articles per project(list of high impact journals to be provided by Brain Canada)Sum of articles per journal

FTE count Number of supported personnel List of FTEs and HQPs per projectSum of FTE and HQP count per position

Streamlined Reporting

• Information Standards• CASRAI - Consortia Advancing Standards

in Research Administration Information http://casrai.org/

• euroCRIS http://www.eurocris.org/ establish metadata standards (CERIF)

Governance and Audits• Accountability• Return on Investments (ROI) in key areas

– Geographical location– Distribution in disease area

• Identify trends in research• Project future needs

Generate informative reports and dashboards

Dissemination of Information

• Promote scientific breakthroughs• Build collaborative environments

– Networks of researchers

• Knowledge search tools– Researcher Databases– National Surveys and Census

Promote research investments using SmartSimple tools or connectors.

Predicting Research Success

• Most funders use a mixture of quantitative and qualitative criteria during the peer review process.

• A “success” story is qualitative mix of public interest and funder priorities.

• Identify attributes in a research proposal that could contribute to desired outcomes.

Using Janus to connect research outcomes to potential projects.

Future of Research Funding

• Align with strategic plan and priorities of stakeholders.

• Predicting “success” of funded research by anticipating outcomes.– Definition of specific metrics important at beginning of

funding cycle

• Tailor RFA, select projects with criteria indicative of research performance.

• Ensure continued funding to research.

Questions?