Embed Size (px)
Citation preview
An Applicant and a Couple of Reviewers Walk into a Webinar: Lessons Learned from PCORI Merit Review
December 11, 2014
Welcome and Introductions
We welcome your questions and comments via the chat function on the right side of your screen
We welcome your comments via Twitter to @PCORI and #PCORI
An archive of this webinar will be posted to http://www.pcori.org/get-involved/pcori-in-practice/ following this event.
If we are unable to address your question during this time, please e-mail your question to us at [email protected]
2
Introduction: PCORI Moderators
James HulbertPre-Award Manager,Contracts Management and Administration
Tsahai Tafari, PhDAssociate Director for Merit Review, Research Integration and Evaluation Program
Kirstin Margosian, MAProgram Associate, Engagement
3
Introduction: PCORI Moderators
Laura Forsythe, PhD, MPHSenior Program Officer, Research Integration and Evaluation Program
Lori Frank, PhDProgram Director, Research Integration and Evaluation Program
4
Introductions: Panelists
Bryan Amick, PharmD, MBAPCORI Stakeholder Reviewer Pharmacy Director/Director of Clinical Quality& Population Health, SC Department of Health and Human Services
Kevin Fiscella, MD, MPHPCORI AwardeeProfessor, Family Medicine and Public Health Sciences, University of Rochester and Associate Director of Center for Communication and Disparities Research,University of Rochester
Kori JonesPCORI Patient ReviewerEmbedded PFCC Coach, Operations and Support ServicesUniversity of Michigan Health System
5
Agenda
Describe PCORI’s unique Merit Review Process & changes over time
Learning from past reviewers and applicants: PCORI Merit Review surveys
Panel Discussion with past applicant and reviewers
6
Have More Questions?
General [email protected] | (202) 827-7200
Research/Programmatic [email protected] | (202) 627-1884
Administrative/Financial/Technical Questions
[email protected] | (202) 627-1885
Find us Online at:www.pcori.org
7
About PCORIpcori.org/about
About PCORI
An independent research institute authorized by Congress through the Patient Protection and Affordable Care Act.Funds comparative clinical effectiveness research (CER) that engages patients and other stakeholders throughout the research process.Seeks answers to real-world questions about what works best for patients based on their circumstances and concerns.
PCORI helps people make informed health care decisions, and improves health care delivery and outcomes, by producing and promoting high integrity, evidence-based information that comes from research guided by patients, caregivers and the broader health care community.
Our Mission
Snapshot of Funded Projects
Number of projects: 360
Amount awarded: $671 million
Number of states where we are funding research: 39 (plus the District of Columbia and Quebec)
As of September 30, 2014
Evaluation
ENGAGEMENT
Engagement as a Path To Useful, High-Quality Research
Proposal Review;Design and Conduct of Research
Topic Selection and Research Prioritization
Dissemination and Implementation of Results
Who Are Our Stakeholders?
PCORI Community
Patient/ Consumer
Caregiver/Family
Member of Patient
Clinician
Patient/ Caregiver Advocacy
Org
Hospital/ Health System
Training Institution
Policy Maker
Industry
Payer
Purchaser
PCORI Merit Review
PCORI Merit Review
The goal of PCORI Merit Review is to identify applications that have the strongest potential to improve patient outcomes.
2
16
Composition of Merit Review Panels
PCORI has three reviewer types that bring a variety of perspectives to the merit review process.
Patients Stakeholders
Scientists
Merit Review Criteria
Criterion #1: Impact of the condition on the health of individuals and population
Criterion #2: Potential for the study to improve healthcare and outcomes
Criterion #3: Technical merit
Criterion #4: Patient-centeredness
Criterion #5: Patient and stakeholder engagement
18
Patient and Stakeholder Reviewers
Scientist Reviewers
15
Scoring Range
Range Score Descriptor Characteristics
High
1 Exceptional Exceptionally strong with essentially no weaknesses
2 Outstanding Extremely strong with negligible weaknesses
3 Excellent Very strong with only some minor weaknesses
Medium
4 Very Good Strong but with numerous minor weaknesses
5 Good Strong but with at least one moderate weakness
6 Satisfactory Some strengths but also some moderate weakness
Low
7 Fair Some strengths but with at least one major weakness
8 Marginal A few strengths and a few major weaknesses
9 Poor Very few strengths and numerous major weaknesses
The scoring range consists of a nine point scale. A score of 1 indicates an exceptionally strong application.
A score of 9 indicates an application with serious and substantive weaknesses.
16
Competitive Letter of Intent (LOI) process
Reduced number of applications assigned to reviewers
Improved application assignments through expertise matching
Role of Merit Review Officers (MROs)
PCORI Merit Review Process Improvements
20
PCORI Merit Review Process Improvements
PCORI Funding Announcement (PFA) guidance for reviewers in the offline critique template
First written critique webinar for scientist reviewers
Updates to reviewer training and reviewer town halls, and additional training for chairs
21
Improvements to PCORI Application Process
Greater clarity on the types of studies we aim to fund
Telephone support by the Helpdesk and programmatic staff
New resources for developing engagement plans
Revised application template and patient & stakeholder biosketch
22
Wish List for PCORI Merit Review
Make the online system more user friendly
Communicate large amounts of complex information concisely and clearly
Continue process improvements while keeping changes manageable for applicants and reviewers
23
Mentor Program
Comprised of veteran patient and other stakeholder reviewers
Help effectively implement feedback
Provide advice in written critique phase and the in-person panel
Help keep progress on track
Rotated back to serve as reviewers
62 trained mentors
24
PCORI Merit Review Panelists by Role
Stakeholder, 74
Patient, 75
Scientist, 157
Breakdown of Participants from Spring 2014 Merit Review
25
PCORI Stakeholder Merit Review Panelists by Community (N = 74)
Clinician23%
Hospital or Health System Representative
16%
Industry10%
Payer1% Policy Maker
6%
Purchaser1%
Researcher30%
Training Institution5%
Patient, 6%
ScientistPatient
Stakeholder
26
Incorporating Multiple Perspectives in PCORI Merit Review
Industry
What research is currently being conducted in the market on various interventions? What post-market considerations should be considered?
Payers
Which projects can help patients and providers better understand their treatment options?
Purchasers
Will the research findings help employees make better health decisions? Improve employee health and improve absenteeism rates? Keep employees actively involved in their work?
Policy Makers
Which projects address relevant questions about patient-centered health policy?
27
How to Apply to be a PCORI Reviewer
Visit: www.pcori.org/content/become-reviewer
QualificationsReviewer ExpectationsCompensation Standing PanelistsTraining Materials Frequently Asked Questions
9
Learning About PCORI Merit Review
30
Applicant and Reviewer Surveys
Cycle II
(Nov 2012)
Cycle III
(Mar 2013)
August 2013
(Aug 2013)
Winter 2014
(Jan 2014)
Spring 2014
(May 2014)
Applicant surveys
--Total N = 613
Response rates = 27 – 61%
Reviewer Surveys
Total N = 1054
Response rates = 70 - 83%
Applicant Survey Results
32
Helpfulness of PCORI Funding Announcements (PFAs)
7% 7%4%
7%2% 3%
86%91% 92%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
August 2013 Winter 2014 Spring 2014
Unhelpful Neutral Helpful
33
Understanding Merit Review Criteria Criterion 3: Technical Merit
7%1% 2%
10%5%
8%
83%
94%91%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
August 2013 Winter 2014 Spring 2014
Disagree Neutral Agree
34
Understanding Merit Review Criteria Criterion 4: Patient Centeredness
7%3%
6%
13%
4% 6%
80%
92%88%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
August 2013 Winter 2014 Spring 2014
Disagree Neutral Agree
35
Understanding Merit Review Criteria Criterion 5: Patient and Stakeholder Engagement
11%5% 5%
8%4%
8%
81%
91%87%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
August 2013 Winter 2014 Spring 2014
Disagree Neutral Agree
36
19%15% 17%
22% 20% 21%
59%65%
62%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
August 2013 Winter 2014 Spring 2014
Disagree Neutral Agree
Review Panels Comprised of Scientists, Patients, and Other Stakeholders Help Ensure Funded Research Proposals are Methodologically Rigorous
37
Review Panels Comprised of Scientists, Patients, and Other Stakeholders Help Ensure Funded Research Proposals are Important to Patients and Stakeholders
8% 6%10%
20% 18% 16%
73%77%
74%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
August 2013 Winter 2014 Spring 2014
Disagree Neutral Agree
Reviewer Survey Results
39
Understanding Merit Review Criteria Criterion 3: Technical Merit
2%
11%
0%
16%
7%0%
82% 82%
100%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
40
Understanding Merit Review Criteria Criterion 4: Patient Centeredness
0% 2% 2%0%5% 5%
100%93% 93%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
41
Understanding Merit Review Criteria Criterion 5: Patient and Stakeholder Engagement
0% 2% 3%0% 0% 2%
100% 98% 95%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
42
The Scientific Reviewers Provided Valuable Input During the Discussion
0% 3% 3%0% 3% 0%
100%94% 97%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
43
The Patient/Stakeholder Reviewers Provided Valuable Input During the Discussion
0% 2% 5%2% 5% 6%
98%93%
89%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
44
Scientific Reviewers were Receptive to Input from Patient and Stakeholder Reviewers
7% 10%
1%5% 5% 6%
88% 85%
93%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
45
Patient/Stakeholder Reviewers were Receptive to Input from Scientific Reviewers
0% 0%7%
3% 6%10%
97% 94%
83%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
46
My Final Scores were Influenced by Input from Other Reviewers
3% 3% 1%0% 2%7%
97% 95% 92%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
47
Changes in Review Scores after Discussion
No change, 38%
Change of 1 point, 31%
Change of 2+ points, 31%
75% of score changes are towards poorer
(lower) scores
48
Review Panels Comprised of Scientists, Patients, and Other Stakeholders Help Ensure Funded Research Proposals are Methodologically Rigorous
1% 3%
12%5% 8% 6%
94%89%
82%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
49
Review Panels Comprised of Scientists, Patients, and Other Stakeholders Help Ensure Funded Research Proposals are Important to Patients and Stakeholders
3% 2% 5%3% 3% 5%
94% 95%90%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Patients Stakeholders Scientists
Spring 2014
Disagree Neutral Agree
Panel Discussion
Introductions: Panelists
Bryan Amick, PharmD, MBAPCORI Stakeholder Reviewer Pharmacy Director/Director of Clinical Quality& Population Health, SC Department of Health and Human Services
Kevin Fiscella, MD, MPHPCORI AwardeeProfessor, Family Medicine and Public Health Sciences, University of Rochester and Associate Director of Center for Communication and Disparities Research,University of Rochester
Kori JonesPCORI Patient ReviewerEmbedded PFCC Coach, Operations and Support ServicesUniversity of Michigan Health System
51
52
Thank you!
• Acknowledgements– Scientist, Patient, and Stakeholder reviewers– Webinar panelists
• Stay current with email alerts at http://www.pcori.org/home/signup and follow us on Twitter @PCORI
• Please send questions or comments to:Lori Frank, PhDDirector of Research Integration and [email protected]
PCORI Merit Review: Learning from Patients, Scientists and other Stakeholders 52
