• An overview of evidence of effectiveness vs evidence of need
• An overview of the range of evidence that can be used to
demonstrate the effectiveness of your initiative’s approach
• Examples of how organizations have successfully used evidence to
support their applications
• Tips on where to search for external arts, culture and heritage
evidence
Strong Application
Activitiestied to
evidence
Good evidence
(quality)
Relevant evidence
(i.e. Grant Result)The best applications use good quality evidence that is relevant to their project, and aligns with
the chosen Grant Result
Example:
Applicants must show evidence that the
model/approach you wish to use is proven
to be effective in achieving its objective
1. Program efficacy
2. Evidence-based approach
Applicants should articulate why the
program is needed in the community they
are working in
1. Evidence of need
2. Evidence of demand for program or approach in
the community
Don’t confuse evidence of need with evidence of effectiveness:
Solid evidence of
effectiveness
Solid evidence of need
Both have their own value, but are distinct in an application!
Example: A community-based performing arts program for racialized youth aims to help them foster a career in the arts
Data demonstrating program effectiveness
• Survey data demonstrating specific positive change
• Interview or focus group data showing positive effects
• Observed changes by staff, using the Observation Method
Data that does not demonstrate effectiveness
• Anecdotes from participants
• Comments about satisfaction
• # people that participated
• Non-specific or vague survey data
Data demonstrates need
• Lack of access to programming locally
• Low rates of associated skills in youth
• Community consultations on need for programming
Building on outside evidence
Documenting evidence of your program’s
effectiveness
Leveraging relevant evidence from academic or community research or evaluations from other
organizations/programs
Using your own evidence that has been generated through evaluation
8
• Evaluation is the systematic gathering and interpretation of information about a program, activity, policy or needo Results from evaluations can help us learn about what's working, and what isn’t, and can
help us make decisions based on our own evidence. Evaluation is one way of generating evidence!
• Research is the systematic investigation and study of materials and sources in order to establish facts and reach new conclusionso Research is also a way of exploring or investigating something. It’s about establishing
facts and drawing conclusions from those facts. Research is one way of generating evidence!
There are many kinds of evidence, but the way we collect it and the purpose behind it are distinct.
Systematic reviews or meta-analyses• Reviews an entire
body of research on a particular program or method
• Summarizesevidence both for and against
• Provides an estimate of the size of impact
Program Evaluations
Vary greatly in terms of method, approach, rigour, context
Grey Literature
Share similar methodologies with research studies, mostly not academically reviewed
Example – Literature Review produced by United Way on youth engagement
Individual research studies• Experiments and
quasi-experiments• Longitudinal studies
with pre- and post-measures
• Typically peer-reviewed, scrutinized by other topic experts; may be less biased
• May have issues of validity and objectivity
Traditional Knowledge
• Orally transmitted knowledge, innovations and practices of Indigenous communities
• From experience gained over the centuries and adapted to the local culture and environment
INSPIRED PEOPLE
Better quality programming and infrastructure to
experience culture, heritage and the arts
Skills and knowledge are transferred to the next generation of artistic leaders
More people connect with culture, heritage and the arts
Access to arts-based learning opportunities and compelling artistic,
cultural and heritage experiences
People are engaged in community-based arts
creation
Preservation and animation of
cultural heritage
12
Grant Result: Skills and knowledge are transferred to the next generation of artistic leaders
The Project:
• Building on the success of a proven model, they are establishing low-barrier digital creator spaces for youth in six communities across Northern Ontario
• 3 year grant, $700,000• Population: General• Age Group: Youth (adolescents 15 to18)• Community Size: Rural or Small Communities (20,000 or less)
13
What type of evidence is it?
• Grey Literature• A study using developmental evaluation produced the University of Chicago
What was the research question?
What was the motivation of the research?
What were the findings?
Used external evidence to demonstrate a proven model “Teens, Digital Media, and the Chicago Public Library”, The University of Chicago Consortium on Chicago School Research, 2013.
• Answer key questions about the YouMedia program at Chicago Public Library• Help inform the design of similar initiatives to engage youth in learning
through technology and youth-centered activities
• How does the design of YOUmedia shape youth participation?• What benefits do youth perceive from their participation? • How do youth engage with media? What role to adults play in engaging teens?
• The model provides teens with a sense of belonging, supports teens’ interests, self-reported improvement in digital media skills
• Successful replication will depend on both local and universal conditions, such as strong partnerships, the right space and adequate resources for equipment
14
• Relevant to their project (promising model)
• Aligned with the Grant Result (skills & knowledge transfer to the next generation of artistic leaders)
• Leverages external evidence (via developmental evaluation)
What makes this a good demonstration of evidence in an application?
15
Grant Result: Access to arts-based learning opportunities and compelling artistic, cultural and heritage experiences
The Project:
• A creative writing workshop expansion project to provide under-served children with the inspiration and support to become confident and creative storymakers through writing, visual arts and digital media
• 3 year grant, $150,000• Population: Diverse cultural communities• Age Group: Children up to 12 years• Community Size: Urban Centres and Metropolitan Suburbs (100,000+)
16
Developed its own preliminary results through evaluation
Collected quantitative & qualitative data from students & teachers via a created survey
• Increased skill & knowledge development (87% students)• Increased interest in writing (92% teachers)• Self-confidence in writing and art• Enthusiasm for future story making (93% students)
How was evidence collected?
What did they wish to discover?
Wanted to understand extent to which students felt inspired and supported to become confident and creative story-makers through writing, visual arts and digital media
What did their preliminary results state?
17
Used external evidence to demonstrate promising practice Something to Say: Success Principles for Afterschool Arts Programs From Urban Youth and Other Experts, Wallace Foundation, 2013.
What type of evidence is it?
• Grey Literature• Research commissioned by the Wallace Foundation• Literature review
What was the research question?
“How do nonprofit organizations serving urban, low-income communities develop or refine effective, engaging and accessible arts programs?”
What was the motivation of the research?
To provide a model of practices that can provide a blueprint for community-based organizations to emulate, so proven approaches can be deployed and more often
What were the findings?
“10 Principles for Effective, High-Quality Out of School Time Arts Programs”Example: Principle #4: There is a culture of high expectations, respect for creative
expression and an affirmation of youth participants as artists.
18
• Relevant to the project (promising model)
• Aligned with the Grant Result (access to arts-based learning)
• Promising preliminary results
• Leveraged both internal and external evidence
What makes this a good demonstration of evidence in an application?
19
Grant Result: Preservation and animation of cultural heritage
The Project:
• Growing and enhancing cultural programming at an Indigenous community cultural centre (cultural counselling, facilitating sweats and other ceremonies, collecting oral histories, language classes)
• 3 year grant, $750,000• Population: Indigenous• Age Group: General• Community Size: Urban Centres and Metropolitan Suburbs (100,000+)
20
Used external evidence to demonstrate promising practice (1) “Creative Arts, Culture, and Healing: Building An Evidence Base”, Pimatisiwin: A Journal of Aboriginal and Indigenous Community Health 8(3) 2010
What type of evidence is it?
• Grey Literature• Research commissioned by Aboriginal Health Foundation of 137 FNMI
projects they funded• surveys & key informant interviews
What was the research question?
“What happens when art, music, dance, storytelling and other creative arts become a part of healing programs?”
What was the motivation of the research?
Addressing a gap in program evaluation data on the role the arts play in healing programs
What were the findings?
Evidence that creative arts, culture and healing are linked:• Eg. Reconnecting to cultural traditions through practice of traditional creative
arts can contribute significantly to healing (drumming for male youth)
21
Used external evidence to demonstrate promising practice (2) “Closing the Gap: Supporting healthy communities through arts programs”, Resource sheet no.28 produced for the Closing the Gap Clearinghouse, Australian Institute of Health and Welfare 2014
What type of evidence is it?
• Grey Literature• Research paper produced by Australian government • Literature Review
What was the research question?
• Why are arts activities important in Indigenous communities?• What are the benefits of participating in arts programs?• How do arts programs support healthy Indigenous communities?
What was the motivation of the research?
Review the available evidence of a range of arts programs in relation to their effects on supporting and building healthy communities
What were the findings?
Evidence of beneficial impacts and outcomes from arts programs in Indigenous communities + principles for implementing effective arts programs in Indigenous communities
22
• Relevant to their initiative (effectiveness of approach)
• Aligned with the Grant Result (preservation & animation of culture)
• Leveraged two external pieces of grey literature
What makes this a good demonstration of evidence in an application?
23
Testimonials/Letters of Support
Program Satisfaction Surveys
Attendance lists
Reports that describe the need
Posters and festival guides
Emails regarding revenue levels
Lesson Plans/Workshop agendas
Strategic Plans
Quotes
Examples of evidence that do not demonstrate effectiveness:
Relevance
• How similar is the context of the study to your community? Can findings be generalized?
• Are the study or evaluation’s goals aligned with your project’s intended outcomes?
• If looking at an older study, is it still relevant in today’s context?
Quality
• Were good research/evaluation methods used? Consider how data was collected and analyzed, what the period of time was, and if a comparison group was used.
• How big was the research sample, and was it representative?
• What was the extent of desired change?
• Consider if other changes/policies/programs are responsible for changes?
• Who produced the study/evaluation, and why?
26
27
Additional Search Tips:
• Don’t stick to one study; try to find multiple pieces of research• Look at guiding documents from your sector • Search for pieces of research that have done a systematic review of available
research (e.g. literature reviews, meta analyses)
Additional Application Tips:
Don’t make it a challenge for the people assessing your application!
• Properly cite the research you are using • Make sure to attach it to your application, highlight important areas/key
findings that relate to your project and the Grant Result
o Be critical!
o Evidence should inform your expertise, not replace it
o Think about the value evidence can bring to informing your work generally, not just in a grant application
o The onus is on you to demonstrate that your project is informed by relevant and good quality evidence
o If don’t have internal evidence, consider applying up to 10% of your application budget toward doing your own evaluation.
30
OTF’s online Evaluation E-Learning coursehttps://otf.ca/knowledge/resources/evaluation-e-learning-course
Arts Evaluation Webinars on the Knowledge Centre:
What makes ARTS FOR CHANGE work EXCELLENT with Animating Democracy
How do ARTS & CULTURE make a DIFFERENCE with Animating Democracy
Evaluating Arts for Social Change & Community-Engaged Arts Projects with the International Centre for Arts for Social Change