View
4
Download
0
Category
Preview:
Citation preview
Social Sciences and HumanitiesResearch Council of Canada
Conseil de recherches ensciences humaines du Canada
Evaluation Follow-up:Monitoring management responses and action plans and capturing evaluation evidence
Mom Yem, Social Sciences and Humanities Research CouncilShannon Clark Larkin, Social Sciences and Humanities Research Council
Canadian Evaluation Society Conference May 2 – May 5, 2010
2
Presentation Outline
Context
Developing the Management Response Action Plan (MRAP) database
Overview of Tool & Process
Lessons Learned
Forward Looking
3
Context
Government prioritiesAccountability for resultsEvidence-based decision-making
SSHRC’s corporate prioritiesImprove reporting on the results and impacts of SSHRC funded research and trainingIntegrate all performance, evaluation and audit activities into program delivery and decision-making
4
Context
SSHRC’s evaluation governance
“SSHRC’s Performance and Evaluation Committee (PEC) ensures the integration of evidence-based decision-making into SSHRC’s management structure and practices through guidance and oversight of the organization’s performance and evaluation functions.”
PEC is responsible for reviewing management responses to evaluations and related action plans, and tracking their implementation
Head of Evaluation (Director, Corporate Performance and Evaluation) is responsible for providing PEC with the information required to effectively carry out this mandate
5
Developing the MRAP database
6
Objectives
Track management responses and action plans in response to evaluation recommendations
Monitor the status of implementation of actions
Provide evidence of evaluation use and impact on decision-making
7
Key Considerations
Management buy-in
Explore existing tools/IT supportSimple tracking database; or Web-based tracking database
Learn from other organizations Environment Canada (adapted internal Audit tool)National Research Council (word document template)
8
Key Considerations
Data Requirements
Identify data requirements and develop a blueprint for the database structure
Map the requirements on the basis of a relational database and develop a data dictionary
9
Mapping of data requirements
10
Key Considerations
Reporting Requirements
Design and customize sample reports e.g., number of action items implemented, number of high priority action items pending, etc.
Reporting accurately relies on completeness of information and quality of data
11
Overall Considerations
Development phaseProper planning ensures the efficiency of the database Ensure flexibility in development to accommodate changes in table structureDevelop and test the database
Implementation phaseFlat database provided to programs requesting for status updates on the implementation of actionsPrograms completed the spreadsheets, with guidance and support
12
Overview of Tool & Process
13
Management Response Matrix - Basis of Development
14
Management Response Matrix - ExampleSummary Table of Management Response to the Joint Initiative Evaluation
Summary Table of Management Response to the Joint Initiative Evaluation – modified from its original version for the purpose of providing an example, implementing the suggested management response guidelines, particularly for the Responsibility and Timeline fields.
15
Guidance on the MRAP
Process for management responses and actions resulting from evaluation recommendations
Guide on “Developing, approving and tracking management responses to evaluation recommendations”
Process for following-up on the status of implementation of actions
Guide on evaluation follow-up process
16
Management Response Matrix (Follow-up Criteria)
Implementation TrackingStatus: Identify the status of implementation of action itemsComment: Provide comments and explanations relating to the implementation status (e.g. deferred/cancelled action items)Time-frame: Annual basis
17
Management Response Matrix - Implementation Tracking
18
Lessons Learned
19
Lessons Learned - Completing the response matrix
Management buy-in and support Draft guidance document circulated to programs/directors and senior staff for feedback Draft guidance document endorsed by PEC, 2008
Simplified guidance documents for better communicationHigh-level summary (2pager)Detailed guidelines and proceduresCommon and standardized templates
Encourage proper completion of the Management Response Matrix
Potential non-response and non-compliancemissing information for ‘Responsibility’ and ‘Priority/Timeline’emphasize actionable responses to recommendationsinconsistencies in responses and structure
Foundation for building a database
20
Lessons Learned - Status Update
1st round of updates on the implementation statusEstablish clearly defined roles and responsibilities Availability of support to programs
e.g., demo and training of the MRAP database
Limited resources and time constraintsReasons for requesting status updates
What is the purpose? Why are we doing this?How will it be used?
Reports submitted to PEC and TBS (e.g. MAF Assessment)
21
Forward Looking
Refine the MRAP guidelines and procedures for completion and follow-up
Continue to work with programs to successfully achieve full implementation of the MRAP processes over the long-term
Develop an integrated database toolModule 1: MRAP (developed/implemented)Module 2: Evaluation Evidence (under-development)
22
Forward Looking
Considerations for developing andtesting Module 2:
Evaluation evidence matrixcreated according to TBS’core evaluation issues (2009)Not all core issues are addressed in an evaluationLink to the MRAP fields: program name, recommendation, etc.Adapted for summativeevaluation more thanformative evaluation
23
Thank you.
Questions? Comments?
Contact Information:Mom Yem: mom.yem@sshrc-crsh.gc.caShannon Clark Larkin: shannon.clark-larkin@sshrc-crsh.gc.ca
Recommended