View
26
Download
3
Category
Preview:
DESCRIPTION
Evaluating. Dr Andrea Baldwin Creative Industries Faculty QUT. a Train the Trainer program in Papua New Guinea. WHAT’S THE PROJECT?. Sexual health – focus on STIs and HIV Behaviour change communication using applied theatre - PowerPoint PPT Presentation
Citation preview
Dr Andrea Baldwin
Creative Industries Faculty
QUT
Evaluating
a Train the Trainer program
in Papua New Guinea
• Sexual health – focus on STIs and HIV• Behaviour change communication
using applied theatre• Experiential learning, not one-way
dissemination of health information• Train the trainer model• Ultimate beneficiaries are young people
(15-25)• Developing hybrid forms of applied theatre
for education/ health promotion
WHAT’S THE PROJECT?
• Professor Brad HasemanCreative Industries, QUT
• Associate Professor Anne Hickling HudsonEducation, QUT
• Dr Andrea BaldwinCreative Industries, QUT
• Ms Hayley Linthwaite
Creative Industries, QUT
WHO ARE THE TEAM?
• Ms Jane Awi Creative Industries, QUT
• Mr Martin Tonny Research and Administrative Assistant
• Ms Jackie Kauli Creative Industries, QUT
• Research Advisory Groups and Participants in Field Sites
• Australian Research Council – Linkage Grant
• Queensland University of Technology• Griffith University • National AIDS Council Secretariat PNG• University of Goroka• University of Papua New Guinea
WHO ARE THE PARTNERS?
Evaluation Framework
Quantitative Qualitative Performative
Context
Inputs
Processes
Outcomes
Haseman, 2006
Stufflebeam, 2003
THE MAP IS NOT THE TERRITORY
Spiral In:
- what’s the bottom line?
- did implementation go to plan?
- did the project create the desired change?
- what are the take-home learnings?
- yes/no, definitive answers
- quantitative
Spiral Out:
- who learnt what from this project?
- what impacts did the project have – positive and negative, expected and unexpected?
- what does this mean for the future?
- descriptive, expansive answers
- qualitative, performative
RESEARCH/EVALUATION METHODOLOGIES MUST:
• Be culturally appropriate, meaningful and relevant to the participants and other stakeholders
• Be sensitive to issues of advantage/ disadvantage for individuals and groups of stakeholders
• Complement and advance the aims and approach of the overall project
A CONCEPTUAL METAPHOR
PROGRAM
Is the seed viable?
PROJECT
What have we learnt about propagation?
EVALUATION RESEARCH
• Involves determining the worth, merit, or quality of an evaluation object, such as an educational program
• Formative evaluation – helps in the design, implementation and improvement of the program
• Summative evaluation – helps make decisions about whether a program should be supported into the future
Johnson & Christensen, 2008
MAGIC PILLS?
• Drug trial model
• Laboratory
• Quantitative, use of inferential statistics
• Clear distinctions between the agent (drug) and conditions of administration (dosage, interactions, etc.)
PARTICIPATORY ACTION RESEARCH
• Much more organic – can’t make such clear distinctions between the agent (program, action, education materials) and the context (community or organisation)
• Requires more holistic approach to collecting and analysing data, interpreting findings, making recommendations for future
IMPORTANT TO:
• Capture negative/undesirable impacts of the project as well as positive/desirable ones
• Capture and assess unexpected impacts of the project as well as intended/expected ones
NEW PHASE, NEW APPROACH
The Life Drama Program
• Development highly participatory – Tari, Madang, Karkar Island
• Quantitative, qualitative and performative data collected
• Triangulating and synthesising vast quantity of data from participants and other stakeholders
The Train the Trainer Program
• Life Drama program content established
• Train the Trainer program content and structure established – Handbook and Bilum
• Collect qualitative and performative data from two sites
WHAT MORE HAVE WE LEARNT ABOUT THE SEED?
1. Have the educators themselves acquired new knowledge and attitudes in relation to sexual health and HIV?
• Interview/focus group data
• Performative data
2. Do the educators believe their students will acquire/have acquired new knowledge and attitudes through the training?
• Post and follow-up interview data
WHAT MORE HAVE WE LEARNT ABOUT PROPAGATION?
1. What factors enhance or impede
a) the delivery and effectiveness of training Life Drama trainers?
b) the formation and functioning of a network of trainers?
2. What are the similarities and differences between training community educators and teacher educators, which are likely to impact differently on the effectiveness of these two groups as trainers?
3. What are the similarities and differences that are likely to impact on their ability to support one another as a functioning network?
GENERALISABILITY?
• These questions being investigated in PNG – two specific sites/groups – capture realistic picture of this specific scenario
• Interpretation of data will include the attempt to draw out general principles that might apply in other settings
“BASIC”
EVALUATION
• PROGRESS REPORTS – is the project being implemented as designed? Is the research team doing what it said it would do? Are budget milestones and timeframes being met? If not, why not? (emphasis on Context, Inputs and Processes)
• FINAL REPORT– did the project achieve intended outcomes hoped for? What factors enhanced or hindered the achievement of these outcomes? (Emphasis on Context and Products)
OBJECTIVE 1: TRAINEE GROUP
1. To train between 20 and 40 Life Drama trainers, with an equal balance of male and female trainers, through two training hubs:
• University of Papua New Guinea, Port Moresby, National Capital District
• University of Goroka, Goroka, Eastern Highlands
EVALUATION
Quantitative data – how many males and how many females completed training?
Performative data – how many trainees passed requirements for certification?
REQUIREMENTS FOR CERTIFICATION
By completion of the training, trainees will demonstrate -
•A practical knowledge of contextual, communicative practices and their place in society;•A deeper awareness of community issues, in particular sexually transmitted illnesses and HIV;•Specific skills in using drama techniques confidently to help community groups explore community issues, in particular sexually transmitted illnesses and HIV;•Leadership skills which help community groups find their own solutions to those problems and barriers which are preventing widespread understanding of, and positive response to, the challenges of sexually transmitted illnesses and HIV;•Critical, analytical and creative approaches about connections between purposes, audiences and diverse ways of communication;•Individual and group interaction leading to brain-storming, understanding and action;•Attitudes appropriate to inclusive communication practices for cultural, linguistic, gender differences and collaborative contexts.
OBJECTIVE 2: TRAINER NETWORK
2. To support the trainee group to function as a “trainer network” – sharing practice, exchanging ideas, assisting one another’s professional development, and expanding knowledge of the Life Drama program beyond the original group of trainees
EVALUATION
Qualitative data – post-training and follow-up interview and focus group data on factors that may enhance or hinder/have enhanced or hindered the functioning of the trainer network
OBJECTIVE 3: TRAINER EFFECTIVENESS
3. To assist participants to integrate Life Drama techniques with their existing educational skills and strategies, to enhance their effectiveness as trainers of HIV-related material
EVALUATION
Qualitative data – post-training and follow-up interview and focus group data on how trainees intend to use/ are using their Life Drama skills for HIV education purposes
Performative data – review in-training video footage of trainers’ practice, and follow-up practice
OBJECTIVE 4: RESEARCH CAPACITY
To build research capacity in Papua New Guinea:
•Participatory Action Research project involving senior staff in arts health areas of University of Papua New Guinea and University of Goroka•Develop research skills of PNG national members of the research team•Encourage research participants to gather data as “co-researchers”
EVALUATIONQuantitative data – how many PNG national staff from the two universities are actively involved in the project?
Qualitative data – what is the level of engagement/mutual benefit for the PNG national staff and the QUT research team?
Qualitative data – Quality of reflective journalling and reporting by participants as co-researchers
GROUP 1
UNIVERSITY OF
PAPUA NEW GUINEA
PORT MORESBY
COMMUNITY EDUCATORSPort Moresby organisations:
University of Papua New Guinea•Lecturing staff•HIV peer educators
Anglicare StopAIDS
National Capital District Commission
Centre for Democratic Institutions PNG
Family Health International PNG
Motukoita Community Youth Development
Department of Education
Hela Province Organisations:
Porgera Joint Venture
PNG Police Service
Hela Rural Women’s Foundation
OilSearch
GROUP 2
UNIVERSITY OF GOROKA
GOROKA, EASTERN HIGHLANDS
TEACHER EDUCATORS
University of Goroka:
•U100 (mandatory HIV course)•Language and Literature Department•Social Sciences Department•Expressive Arts Department•Science Faculty•Education Faculty•Early Childhood Department•Student Services
Madang Teachers College
Oxfam PNG
Independent Community Theatre Practitioners
OBJECTIVE 1: TRAINEE GROUP
Port Moresby
•6 females
•14 males
•20 TOTAL
Goroka
•15 females
•11 males
•26 TOTAL
TOTAL = 46 TRAINEES, 25 MALE AND 21 FEMALE
OBJECTIVE 2: TRAINER NETWORK
• Only post-training focus group data available so far (follow-up focus group data to be collected in November 2011)
• Only in-training video performative data available so far (follow-up video of trainees’ practice to be collected in November 2011)
Factors Enhancing
• Membership of same organisation (eg. university staff)
• Ability to access communication technology (internet, email, phone)
• Organisational support (esp. U of G)
• Ongoing relationship with QUT
• Ongoing resourcing
Factors Impeding
• Membership of different organisations
• Working with different client groups – different needs
• Communication barriers (few trainees have internet access; internet access often unreliable in PNG; phone access ditto; phone communication expensive)
• Lack of resourcing
OBJECTIVE 3: TRAINER EFFECTIVENESS
Qualitative Data:
•Only post-training focus group data available to date. Follow-up focus group/ interview data will be collected in November 2011
Performative Data:
•Only in-training video available for review to date. Follow-up video will be collected in November 2011
•Video to be analysed by members of team expert in:
– Drama-in-Education/ Theatre for Development (Creative Industries Faculty)
– Pedagogy (Education Faculty)
OBJECTIVE 4: RESEARCH CAPACITY
• # PNG senior university staff actively engaged in project, not as trainees: 7 (5 in Goroka, 2 in Moresby)
• “Active engagement” evidenced by: participating in training sessions, observing training sessions, providing additional supports and assistance to the program, contributing to focus groups
• Level of mutual benefit for PNG national research staff:
- 2 PNG national research staff undertaking PhDs with QUT as a result of Life Drama project (assisting with data collection and analysis for Train the Trainer program)
- 1 PNG national research staff member now undertaking Masters with another university, partly as a result of involvement in Life Drama
- QUT and University of Goroka now exploring future partnership opportunities
• Quality of reflective journalling and report provision by participants – yet to be assessed (November 2011)
TWO GROUPS: COMPARE AND
CONTRAST
CONTEXT
Port Moresby
• Primarily community educators
• Significant percentage from outside Moresby or away from the university (eg. settlement areas)
• Large range of organisations represented, sometimes by 1 or 2 people
Goroka
• Primarily teacher educators
• Primarily local (living and working in Goroka)
• Most trainees affiliated with University of Goroka at some level
INPUTSPort Moresby
• Less organisational commitment – less resourcing – more difficult learning environment
• Indoor venue in poor repair
• Several trainees self-funding attendance – resentments over resourcing
Goroka
• Two indoor venues in good repair but not always available; using outdoor spaces without disturbing exams
• More organisational commitment – more resourcing – more condusive learning environment
• Issues over resourcing for non-local attendees
PROCESSES
Port Moresby
• Handbook not available until last day
• Trainees generally less comfortable with English
Goroka
• Handbook available throughout training, used for daily review of lessons
• Trainees all comfortable with English
PRODUCTS
Port Moresby
• Harder to follow up and evaluate
• We expect less effectiveness of trainer network because fewer participants have good access to communication mechanisms and infrastructure
Goroka
• Easier to follow up and evaluate
• We expect more effectiveness of trainer network because most participants have access to communication mechanisms and infrastructure, and/or are geographically co-located
WHAT MORE HAVE WE LEARNT ABOUT THE SEED?
1. Have the educators themselves acquired new knowledge and attitudes in relation to sexual health and HIV?
• Interview/focus group data – yes, eg. difference between HIV and AIDS, how to live healthy life with HIV
• Performative data – yes, eg. condom demonstrations
QUALITATIVE DATA
“I thought AIDS was an instant disease. But I realise now there’s a harmony in the body that tries to fight against this disease. If I’m thinking that, what about the students? We need to teach them all the things you can do to stay healthy”
Female participant, early 50s, lecturer in U100 HIV program at Goroka University. Post-training focus group.
PERFORMATIVE DATA
2. Do the educators believe their students will acquire/have acquired new knowledge and attitudes through the training?•Post and follow-up interview data – yes, particularly Dancing Diseases
WHAT MORE HAVE WE LEARNT ABOUT PROPAGATION?
1. What factors enhance or impede
a) the delivery and effectiveness of training Life Drama trainers?
Shared language, resources, communication, venues, equipment.
Regular debriefing/consolidation periods.
Handbooks.
b) the formation and functioning of a network of trainers?
Shared language, organisational commitment to trainees and their professional development/ongoing peer support, access to communication, resourcing.
2. What are the similarities and differences between training community educators and teacher educators, which are likely to impact differently on the effectiveness of these two groups as trainers?
•Similarities include degree of commitment to education, identification with the educator role
•Differences include foundational skills (in teaching, in drama), organisational support, resourcing, communication
3. What are the similarities and differences that are likely to impact on their ability to support one another as a functioning network?
•Similarities include degree of commitment to education, identification with the educator role.
•Differences include organisational support, resourcing, communication
METHODS
Quantitative
-Individual interviews with rating scale responses
-Observer ratings of skills displayed in workshops (live and on video)
-Recordings of numbers eg. number of participants, sessions attended
-Trainers rating their own participants’ understanding of HIV issues
Qualitative
-Focus groups
-Interviews
-Stakeholder meetings
-Case studies
-Participant journals
-Lead trainer reflective journals and field notes
Performative
-Participants observed and videoed both as participants in training workshops, and conducting training of others
-Participants performing “stories” illustrating “best” and “worst” moments delivering training (these stories videoed)
UNEXPECTED OUTCOMES?
• Teacher educators excited by new approach to pedagogy – keen to apply it in areas apart from HIV (including early childhood teaching)
• One director keen to re-engineer her “office job” and re-engage with teaching
• Trainers identify range of applications for Life Drama in other life roles eg. Sunday School teacher, police officer, pastor
CHALLENGES
PARTICIPANTS
•Communication
•Literacy
•Numeracy
•Languages
•Cultural expectations
•Culturally-appropriate data collection “tools” and methods
PARTNER ORGANISATIONS
•Power relations between organisations
•Organisational and national politics
•Perceived benefits for individuals and groups
•Infrastructure including venues, equipment, communications
•Travel
STUDENTS OF TRAINEES
•Religious and cultural resistance to discussing sex, body parts
•Resistance to condom use
•Gender issues
•Sensitive topics such as sexual activity with children
RESEARCH OUTPUTS
• Life Drama Train the Trainer booklets: Life Drama Handbook and Life Drama Bilum
• Interim Report and Final Report for participating organisations and participants
• Journal articles• Conference presentations• Further applications of Life Drama
• Life Drama website – evaluate separately as a tool for communicating performative outcomes
PUBLICATIONS
Baldwin, A. (2010). Dancing Diseases: An Applied Theatre Response to the Challenge of Conveying EmotionallyContradictory Messages in HIV Education. Applied Theatre Researcher, 11.
Baldwin, A. (2010). Life Drama Papua New Guinea: Contextualising Practice. Applied Theatre Researcher, 11.
Jennings, M. and Baldwin, A. (2010). “Filling out the Forms was a Nightmare”: Project evaluation and the reflective practitioner in community theatre in contemporary Northern Ireland. Music and Arts in Action, 2(2), pp. 72-89.
QUESTIONS?
www.lifedrama.net
Recommended