Embed Size (px)
Citation preview
Welcome2013 HIV/AIDS Community Information Outreach Program Evaluation Webinar
Online HIV/AIDS Resourceshttp://sis.nlm.nih.gov/outreach/training_outreach.html
Presenters
LaFrancis McMurray, MPHHealth Education SpecialistOak Ridge Associated [email protected]
Arletha Williams-Livingston, Ph.D., MPHEvaluation and Organizational Capacity-building ConsultantTransforming Communities, [email protected]
PurposeThe purpose of this webinar is to build organizational evaluation capacity, skills and knowledge, to discuss program evaluation requirements and to connect participants to evaluation technical assistance.
ObjectivesUnderstand the importance of program evaluation
Effectively evaluate ACIO projects
Access evaluation technical assistance as needed
Today’s TopicsWhat is Evaluation?Why Evaluate?Building Organizational Evaluation CapacityACIO Evaluation Requirements, Resources and Technical Assistance
What is Program Evaluation?
Program– Your campaign, action, training, intervention
Evaluation – Is the systemic investigation of the merit, worth, or
significance of your workProgram Evaluation
– Is a way to evaluate the specific projects and activities of community groups and organizations
In other words…Program Evaluation is a combination of:
– Good Reporting– Good Detective Work– Good Historical Documentation and Storytelling
Program Evaluation helps organizations see how and in what ways they are really helping the community, OR how and if an organization is achieving its goals and objectives.
Three Main Types of Program Evaluation
Process EvaluationOutcome Evaluation Impact Evaluation
Process EvaluationDescribing and documenting the course of a program For Example:
– Who are the Members or Participants?– What activities were conducted?– How many people were in attendance?– When and Where are events and meetings being held?– What staff was there?– Who was involved in the planning?– Who was involved in the implementation?
Outcome Evaluation
What were the immediate effects of the program?Changes in skills, knowledge, attitudes, and behavior For example:
– Did the participants learn anything?
– Do the participants have a different attitude?
– Do the participants do something differently or do something new?
– Was there a campaign victory?
Impact Evaluation What were the ultimate effects on the community?For example:
– How was the community affected? What were the effects?
Three types of Impact evaluation– Immediate (more like an outcome)
• Knowledge of housing shortage issue
– Intermediate • Advocacy work and heightened policy making and community awareness of
housing issue
– Ultimate • Decrease in percentage of the target population who are in need of housing assistance
Why Evaluate?
Evaluation Is Used for:
Program Development Mass Campaigns Policy FormationsIssue AdvocacyOrganizing Communities Funding Applications
Program Evaluation is used to…
Determine if goals were met
Show effectiveness and impact of a program
Provide evidence of the process and outcomes
Questions to Ask When Developing an Evaluation Plan
Building Evaluation Capacity
Capacity Building Values
Interdependence
Cultural diversity
Contextualized interpretation
Empowerment
ACIO Information Access Categories Evaluation InformationMeasuresQuestionsToolsResources
Information Access Category I Information Retrieval
Information Retrieval
To provide or improve access to electronic AIDS- and other health-related information resources by the organization or by the clients they serve and to implement access to state-of-the-art AIDS-related information tools via the Internet and/or mobile phone technologies. This may require the purchase of hardware (computer equipment), software, or contractual relationships necessary to obtain a connection to the Internet for the purpose of accessing AIDS-related information or sharing AIDS-related information.
Project Examples 1Online living library databaseInformation retrieval skills development training for health outreach workers and local community based organizationsIncrease access to critical information for at risk youth using a multimedia hub-spoke model of online, social media and mobile platformsMobile-Internet-based HIV/AIDS Education: Linking the Public to SciencePurchase of computer equipment/software for cyber lab
Measures 1
AwarenessMarketing/awareness of informationCommunity partnerships/community engagement
AccessRetrieval skills Access to computers/mobile/Internet
Proof of access/number of visits/types of resources
Evaluation Questions 1Awareness
In what ways is information advertised to potential consumers/clients?How are community partnerships engaged to help provide information access to residents?
AccessHow often are clients retrieving information?How are clients accessing information via computers, Internet and mobile?How many times & which types of resources clients are accessing?
Sample Evaluation Tools 1AwarenessMarketing plan objectives, outcomesNumber of MOAs/MOUsNumber & quality of community partnership/community advisory board meetingsObservations of community partner engagement
AccessSocial Media monitoring tools i.e. Tweetdeck, Website analytics i.e. Google Analytics
Resources 1Social Media/ Health communications planning with evaluation componenthttps://www.healthcommworks.org/Account/LogOn?signInArea=SocialMediaWorks
US Department of Education Evaluating Online Learningwww.ed.gov/admins/lead/academic/evalonline/evalonline.pdf
Evaluation strategy for mobile device learninghttp://www.salt.org/salt.asp?pn=procabs&ss=l&key=153320&pid=2406
Information Access Category IISkills Development
Skills Development
Training to develop skills in accessing or using HIV/AIDS-related information including the use of NLM online databases and mobile resources as well as general AIDS information resources in order to improve services and benefit the community. This may include training for the staff of the agency, development of training programs for clients and patrons, training programs for other agencies, etc.
Project Examples 2Capacity-building training for youth serving professionals; and peer-to-peer educationInformation retrieval skills development training for health outreach workers and local community based organizationsTraining classes and workshops on NLMs online resources and social media sitesTraining for new HIV positive peer educators/counselors as online instructors and facilitate new training classes featuring health literacy
Measures 2TrainingDevelopment of trainings/types of: access/utilization/ general computer skills/database trainingTraining target populations: staff/clients/ patrons/agencies/special populationsNumber of trainings/trained usersTraining satisfactionTraining outcomes: knowledge/skills
Measures 2 (cont’d)UtilizationTypes/amount of resources/information utilized: Internet/social media/email/mobile/brochures/factsheets/face-to-faceUtilization outcomes: Knowledge/skills/empowerment/efficacy/service improvement/community outcomes
Evaluation Questions 2TrainingWhat types of trainings are being developed?Who is the target training population?How many people are being trained? (Attendance/completion)Are consumers satisfied with trainings?What are the training outcomes/skills developed?UtilizationWhat type of information consumers are utilizing?How is that information being utilized?What are the outcomes associated with consumer utilization of information resources?
Sample Evaluation Tools 2TrainingTraining Development TemplateSign-In sheetsPre/Post assessmentsSatisfaction surveysUtilizationSocial Media monitoring tools i.e. Tweetdeck, Website analytics i.e. Google AnalyticsQuestionnaires, Surveys, Qualitative analysis of responses
Resources 2
Training development templateSample pre/post skills assessmentsWriting measurable learning outcomes
http://www.gavilan.edu/research/spd/Writing-Measurable-Learning-Outcomes.pdf
Evaluating Training Programs-Kirkpatrick’s 4 levelshttp://www.wa.gov/esd/training/toolbox/tg_kirkpatrick.htm
Information Access Categories III and IVResource DevelopmentEquipment Acquisition
Resource Development
To support an organization to meet the needs of their clients or communities by developing specific educational or informational materials such as culturally or language specific fact sheets, bibliographies, guides or other products.
Equipment Acquisition
To improve access to electronic AIDS and other health-related information resources which may require the purchase of hardware (computer equipment), software, or contractual relationships necessary to obtain a connection to the Internet for the purpose of accessing AIDS-related information or sharing AIDS-related information.
Project Examples 3
Produce and syndicate culturally and linguistically appropriate messagesReady Reference Guides to person with HIV/AIDS or at high risk for HIVCulturally competent and age appropriate information in English and Spanish emphasizing prevention, risk reduction and skills buildingDevelop and facilitate access to factsheets on the treatment of HIV/AIDS and related information written in nontechnical language and translated into several languages
Measures 3Resource needs Target population needs assessment/types of resources neededResource developmentResource materials created: type such as cultural specific, bibliography Resource materials utilized/sustained/maintained/updatedPurchases of hardware/software to develop resourcesUtilization of consultants/contractual relationships to develop resources AccessPurchases of hardware/software to access and share informationUtilization of consultants/contractual relationships to access and share information
Evaluation Questions 3Resource needs Has a resource needs assessment been conducted for the target population?If so, what were the outcomes of that needs assessment?Resource developmentWhat types of resource materials are being developed?How are the resource materials being utilized/sustained/maintained/ updated?Was hardware/soft purchased to develop resource material? If so, what kind and how was it used?Were consultants/contractual relationships used to develop resource material? If so, who and what work did they complete?
Evaluation Questions 3 (cont’d)AccessWas hardware/soft purchased to help agencies provide access/share materials? If so, what kind and how was it used?Were consultants/contractual relationships used to help agencies provide access/share materials? If so, who and what work did they complete?
Sample Evaluation Tools 3Resource needs Outcomes of Community/resource needs assessmentResource developmentResource development templateResource material tracking (How utilized/sustained/maintained/updated)Comparison checklist for equipment/software purchasesConsultant/contractual reportsAccessEquipment/software inventoryUtilization logConsultant/contractual reports
Resources 3TeamLab-University of Southern California
Designing materials for particular audiences/understanding culture http://teamlab.usc.edu/learn/roadmap/audience.html
CenterLink-The Community of LBT CentersConducting Needs assessment
http://www.lgbtcenters.org/how-to-do-a-community-needs-assessment.aspx
Resources 3 (cont’d)Harvard Family Research Project Evaluation Exchange, Understanding Cost-Benefit and Cost-Effectiveness Evaluation
http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/methodology-15/at-what-price-benefit-cost-analysis-and-cost-effectiveness-analysis-in-program-evaluation
TechSoup.orgWebsite for Non-Profits and technology. Planning and how to articles
http://www.techsoup.org/support/articles-and-how-tos/articles-by-subject?subjectid=14
ACIOP Evaluation Requirements
Quarterly reportsFinal reportsACIOP Resources Blog http://nlmacioawardeeblog.wordpress.comIndividual agency evaluation conference callsEvaluation technical assistance
What’s next?Visit ACIOP Blog http://nlmacioawardeeblog.wordpress.comIndividual agency calls to discuss:
– Agency Evaluation capacity– Project evaluation plans– Technical assistance
Mid Year Webinar – Milestones– Evaluation needs of agencies– Final reports
Q & A
