The World of

Monitoring & Evaluation

Objective

•Learn the why, what and how-to approach to monitoring•Review monitoring techniques and define the roles monitoring plays in our lives both personally and professionally•Understanding the types of monitoring and their impact on program operations•Provide insight on the critical elements of monitoring•Open Discussion

What Is Monitoring?Terminology Minefield

Monitoring is a very widely used and ambiguous termMerriam Webster

Monitoring is to watch, keep track of, or check for a special purpose

General Definition

Used to describe the process of checking or regulating some defined activity

Why Monitor?

Why Do We Have To Monitor?

•IT’S REQUIRED•WE HAVE TO!

Why Monitor?

Importance of Monitoring•Through assessment, we know how well we are doing•Through assessment, we know how effective and efficient we are•Through assessment, we are able to monitor our resources•You may go on using resources to no useful end

Why Monitor?

Monitoring can be an invaluable management tool that enables you to assess the quality and impact of your work against your action plans

Why Monitor?

Like riding a bicycle, you can adjust as you go along and ensure that you are on the right track

Why Monitor?

Reasons Monitoring Is Not Popular

•It’s intrusive•If it’s not broke, no need to fix it•Other competing priorities•No complaints, no problems•Not necessary •Like pulling teeth

Why Monitor?

What If We Didn’t Monitor•Our Health•What We Eat•Restaurants-Food•Wardrobe•Check-book

• Investments• Pollution• Homes• Government• Water Quality

Role of Monitoring

Monitoring Is Used In…•Project planning and implementation•Analyzing the situation•Determining whether the inputs in the project are well utilized•Identifying problems facing the project or program and finding solutions

Role of MonitoringMonitoring Is Used To…•Ensure all activities are carried out properly by the right people and in the allotted amount of time

•Apply lessons from one project experience on to other projects

•Determine whether project planning and problem solving is as appropriate and efficient as possible

Role Of Monitoring

4 Principles of Monitoring1. Monitoring is the regular observation and

recording of activities taking place in a project or program

2. Monitoring is a systematic and purposeful observation based on defined inputs

3. Monitoring involves giving feedback about the progress of the project or program

4. Monitoring enables the gathered information to be used in making decisions for improving project performance – Lessons learned

Role of Monitoring

Monitoring Is A Feedback LoopFor Example

• Checking to ensure that we did what we said we would do• Checking the assumptions and estimates used to make decisions• Checking on program quality and efficiency• Feedback and continuous improvement

Role Of Monitoring

Monitoring Can Be Carried Out•To meet one or more of a wide variety of objectives•By measuring one or more of a wide range of variables•At different levels of resolution

What To Monitor

What Are We Monitoring

•Program Requirements •Sensitivity and Risk Analysis Can determine key variables in developing a monitoring plan and monitoring requirements

World of Monitoring

What?

• Training • Performance• Implementation plans• Certifications

Why?

EffectivenessEfficiencyValidationImpact

How?

Onsite or remotelyRecordsInterviews

World of Monitoring

Strategic Plan

Implement

Monitor

ReflectLearn DecideAdjust

Feedback

Types Of MonitoringThe phrase “monitoring is used in a wide variety of situations”

Examples Include:•Fiscal Monitoring •Validation Monitoring•Compliance Monitoring•Effectiveness Monitoring•Administrative Monitoring •Programmatic Monitoring

Program Monitoring

Programmatic Compliance Monitoring Consists of 2 Components

•Administrative Compliance Asks the question, “how well does RWB administrative documentation reflect actual conditions on the ground before and after program activities took place?”

•Field ComplianceAsks the question, “how well are the rules and the terms of the program being applied on the ground?”

Program MonitoringProgrammatic Compliance Monitoring The primary focus of programmatic monitoring is:•To assess program operational practices, procedures and processes for compliance with applicable federal and State laws, policies

and guidance

•To ensure that policies and practices relative to program quality and outcomes meet the program’s objectives

•To assist the RWB in identifying problem areas and recommending alternatives and practices that will promote positive practices and outcomes

Program Monitoring

Critical Elements•Define population of interest and take a representative sample•Compare estimates from the representative sample with specific benchmarks•Various levels of resolution depends on objectives and target population

Sampling Techniques•Statistical Sampling - probability sampling where every item in the population under review has a known chance of selection •Judgmental Sampling – sample size or composition determined by professional judgment rather than the laws of probability•Sliding Scale - approximates the sample size determined using a semi-statistical approach•Bootstrap Method?

Sampling Techniques

DEO’s program monitoring section uses a statistical, random sampling method to select program files to be reviewed for each program

Sampling Techniques

NO MATTERWhat methodology is used, make sure that it is a valid representative of the population being reviewed

Tools &Guides

Types Of Tools/Guides Used:

Administrative

•Process Management Review Tool

Compliance

•Process Specific Review Tool

Tools & Guides

Submission To Regions

Monitoring tools are normally submitted to RWB staff at the beginning of the annual monitoring cycle for review and feedback

Tools & Guides Process Management Review Tool•Asks questions about program processes, design elements, and organization•Allows you to become familiar with program operations•Evaluates program design in the context of whether it is demand-driven•Allows you to draw objective conclusions about program operations, including promising “Best Practices”

Tools & Guides

Program Specific Tool•Used to determine compliance with specific program requirements•Serves as an indicator of whether objectives or requirements are being met•Identifies issues that may be contrary to law or other requirements

Tools & Guides

•A separate tool is developed for each program monitored•Areas covered by tools include questions related to:•Training Services•Support Services•Credential Attainment•Participant Eligibility Criteria•Assessment and Plan Development•Grievance/Complaint/EEO requirements

Local InitiativesQuality Assurance•Establish goals and objectives on what you want accomplished

•Develop an action/implementation plan based on the goals, activity levels, targets, etc

•Develop local operating policies/procedures/guidelines based on core management functions and program operations

•Develop tools based on your objectives, rules, tenets of law and other guidance (one or more indicators to assist in determining if objectives have been met)

•Train staff

Local InitiativesCase Managers

•Ensure all files have required documentation•Develop a checklist of what should be included in the files•Make sure detailed case notes are entered in system•Conduct peer reviews/evaluations•Exchange case files for review by different case managers•Check accuracy of source documents in files against system data

Summary•Monitoring is a very ambiguous term that is used for continuous improvement

•Don’t worry about what a monitoring program is called •Focus on •The pre-defined objectives•What is being done•Why it’s being done•What impact is it having

•Remember that monitoring is the feedback loop that can provide useful information in planning and making changes

References

•Phil Bartle, PhD•Civicus•J.S Thrower & Associates Ltd.

Questions?

For More Information On

Monitoring and Evaluations Please Contact Ken

Williams at Kenneth.williams@deo.

myflorida.com