Childrens Issues Evaluation Brief Spokesperson Training Evaluation Conducted for the Alberta Family

1 © FrameWorks Institute – Evaluation Brief

Spokesperson Training Evaluation Conducted for the Alberta Family Wellness Initiative

Prepared By:

Tiffany Manuel, PhD, Director of Impact and Evaluation Suzanne Lo, MPH, Senior Associate, FrameWorks Institute

In March 2011, the FrameWorks Institute conducted a spokesperson training with a set of influential leaders in the child development field in Alberta, Canada. The training was sponsored by the Alberta Family Wellness Initiative (AFWI) and facilitated by FrameWorks staff. Anecdotal feedback from the training has been overwhelmingly positive and, as a result, there is some interest in extending this training to a wider group of leaders in the province. To better understand the value-added of this type of training, both AFWI and the FrameWorks Institute wanted to obtain more structured feedback from the participants about the effectiveness, utility and impacts they attribute to the training. This evaluation brief summarizes the feedback we received from a post-engagement survey and follow-up interviews with participants. THE LEARNING ENGAGEMENT The FrameWorks Institute conducts spokesperson trainings with influential leaders, policy experts, advocates, communications professionals and scholars each year. The purpose of the training is to provide participants with a firm grounding in frame analysis, to

introduce the relevant FrameWorks evidence-base on topics of interest to the participants, and to subsequently give each participant the opportunity to build their capacity to reframe a typical conversation of the type they experience as spokespersons on child development. More specifically, FrameWorks offers participants a training that consists of three major components:

• preparatory readings/videos to introduce participants to Strategic Frame Analysis™ (a method of frame analysis pioneered by the FrameWorks Institute which roots communications practice in the cognitive and social sciences);

• a workshop covering FrameWorksʼ

relevant topical research alongside the extant communications recommendations; and

• mock one-on-one interviews with each

participant and FrameWorks staff, which are videotaped for participant observation and reflection.

The FrameWorks Institute is an independent nonprofit research organization founded in 1999 to advance the nonprofit sectorʼs communications capacity by identifying, translating and modeling relevant scholarly research for framing the public discourse about social problems. FrameWorks designs, commissions, manages and publishes multi-method, multi-disciplinary communications research to prepare nonprofit organizations to expand their constituency base, to build public will, and to further public understanding of specific social issues.

EVALUATION BRIEF


Finally, participants are given the opportunity to observe themselves in the videos as part of the learning process, and are given suggestions on using the reframing recommendations to advance the science that undergirds child health and well-being. THE EVALUATION METHODS This evaluation assesses the effectiveness of the training conducted in March 2011 and its impacts on the practices of the participants. FrameWorks evaluation staff used two methods to collect substantive feedback about the training from participants. First, we asked participants to respond anonymously to a short follow-up survey sent via email. Second, we followed up those surveys with telephone interviews in which participants were asked to provide deeper insight into the effectiveness of the training, what they felt were the main takeaways, the value-added to their professional work, the areas where they could have used more resources (or where they felt the training could have been improved), and to what end they have used the information/skills they acquired. Out of a total of 12 participants, five answered the survey (anonymously) and 10 of the 12 were available to participate in the 15-20 minute follow-up interviews. In terms of the demographics of the participants, all participants reported that they have direct responsibilities as spokespersons on child development in their current professional roles, 83 percent had graduate-level educational backgrounds, and 95 percent had been in their current professional roles for three years or more. The data from this evaluation were mostly qualitative in nature and that data was then coded across thematic categories, analyzed and organized to highlight the major issues identified by respondents. The few quantitative measures used in this evaluation were tabulated, integrated within the text, and occasionally appear in text boxes through the findings section.

THE FINDINGS The first set of findings to emerge from this data involves the effectiveness of the training itself. In both the survey and the interviews, we asked questions of participants about the concepts, ideas or information they retained from the training, and which of those they thought were particularly useful. In addition, several questions in both the survey and the follow-up telephone interviews asked participants to think about the value of the information they received in terms of their own professional roles and responsibilities.

• Structure of the Engagement In thinking about the structure of the engagement, participants felt that the overall structure, sequencing of the activities, and interactive pedagogical model used were effective. As one participant commented, “there was a nice logical sequence to the training.” Participants ranked the amount of information presented, the groupʼs size and the diversity of participants as “just right” when asked whether the balance of these resources in the training was “too much, too little, or just right.” In addition, most participants reported that they felt the materials and handouts (especially the pre-workshop readings) were helpful and informative. Participants reported that the nonpartisan perspective of the information presented, the practical exercises provided, and the ability to talk among themselves and with trainers, were all particular features of the training that enhanced the learning experience.

100% of the participants surveyed and interviewed

said that the amount of information presented was

“just right”.


• Credibility of the Evidence-Base Although we did not ask a question specifically about the credibility of the evidence-base presented in the training, six of the interviewees mentioned this issue as being particularly important to them. As a result, this issue emerged organically as part of the qualitative elaboration during the interviews. “The surety that I have, to be able to say with conviction, that these are really well researched ideas to explain complex and important issues to motivate people for call to action, is critical.” As such, the feedback in the interviews made it clear that participant understanding of the science behind the communications research helped to enhance the credibility of the evidence-base, and also helped those who would otherwise be more cynical to confidently adjust their practice. “I was really impressed. I was a little cynical, being a journalist, but that disappeared really quickly as the day went on — I was really energized and excited about what was offered.” Moreover, participants clearly appreciated the “literature references” and the alignment of the communications recommendations with “the science of the early brain research.” “There were quite a few slides that were presented in the workshop as the background research, that had me scrambling to write down the references, even though I knew I was going to get the slides.” The one challenge that participants mentioned in this area was in believing that one set of frames would work to effectively educate all audiences. As such, they wanted more evidence that a common message could have common impacts across distinct audiences and in various media environments. Addressing public assumptions about audience segmentation, as opposed to broad cultural constructs, remains a challenge.

• Assessing the Value of the Workshop

Participants felt the workshop was useful to them and provided sufficient depth of information to inform their communications practices. When asked what concepts seemed most germane and memorable, a strong majority (eight of 10 interviewees) mentioned the heuristic of the “swamp” of cultural models. FrameWorks uses the “swamp” metaphor to describe the ideas that are top-of-mind for people when they think about an issue, as well as those ideas that shape the public discourse. More than anything else, participants remembered the importance of the swamp as a symbol of what they needed to address in their messaging. Secondarily, participants remembered the metaphors FrameWorks presented from its research and talked quite generatively about the utility they saw in using them. That is, the participants found the metaphors to be especially helpful in their attempts at knowledge translation around the early brain science. The two most commonly cited metaphors were “serve and return” and “brain architecture.” Although participants could not point to systematic data, they reported that their usage of the metaphors improved their message delivery and audience understanding. Third, seven of the 10 interviewees found the bridging and pivoting concepts to be important, valuable and useful. They appreciated understanding how to answer questions in ways that support investments in child development and how to bring the conversation back when they felt it had veered into unproductive territory. “I was using it [bridging] a little beforehand. But the day really drove home how important it is. I definitely use [this technique] much more now.” Finally, several of the participants (five of the 10) found the general framing lessons particularly memorable. In their elaborations, they spoke of the need to “set the context,”


“segue into metaphors,” “build up the story,” “build a bridge of public understanding” and “draw the link” between their expert knowledge and broader public understanding on the ground.

• Videotaped Interviews More than any other aspect of the training, participants cited the video interviews as the most valuable part. They were articulate in explaining how informative and instructive this part of the training was to them. “The actual media practice simulation was very good. This is an incredible opportunity. One can never get too much of that.” “There is nothing like experience. It is one thing to see people on camera but quite another to really know what youʼre doing when youʼre on camera.” When asked to recount the most important ideas and concepts recalled from the video training, participants generated a long list of principles that had “stuck” from the training. These included substantive framing advice such as “use the brain architecture metaphor to help introduce the idea of child development” and “set up the value before you explain your main points,” as well as more general advice such as “slow down when you speak.” Participants found that the training gave them a kind of spokesperson “checklist” to facilitate their future public engagements. This included, for example “to think about the question in terms of the frame and respond to that message,” “to fine-tune the key portions of the message and especially how I use the metaphors for the science” and “to keep it simple and to remember to stay on message through bridging and pivoting.” In the survey and in the interviews, participants spoke extensively about the value of the skills they had gained around pivoting, bridging and “taking charge of the direction of the interview”

by effectively framing the content. In fact, six of the 10 interviewees mentioned the opportunity to practice pivoting or bridging techniques as one of the things they found particularly helpful about the video training. “When people ask me questions (and I get a lot of questions), I get away from just answering the questions, but I reframe the questions that I want to answer instead. I feel much more confident doing that now.” “I really make sure that my comments are reframed in such a way that the question I answer is something I wanted to communicate and I donʼt try to bullshit on parts of the question I do not know about.” “I learned to answer the questions you want to answer and not necessarily answer them how they are being asked by the interviewer. I also learned how to keep the message simple and easy to understand.” Several of the participants noted that, although they had participated previously in spokesperson trainings, they particularly appreciated training with a specific focus in the science of early brain development. That is, they found specific value in the fact that this training was mapped onto specific translation techniques for talking about brain science, rather than a more generic approach. The interview data gave participants a chance to make it clear that they saw the efficacy of the training as being about capacity building and skill development rather than about a conceptual shift in how they see their own roles as spokespersons. As such, this reifies FrameWorksʼ goal of the training as refinement of participantsʼ communication skills, not merely

Participants overwhelmingly reported that the value of

the training for them was in the concrete and practical

skill-building that were immediately applicable to

their work.


a major shift in their assessment of their spokesperson roles. In talking about how the training had affected the development of those skills, almost half said that the spokesperson training had pretty “dramatically” changed the way they communicate about early child issues, while another quarter said that it had “moderately” changed the way that they communicate. More generally, the consensus was that, more than anything else, the training helped them stay more focused and “on message.” “The framing of this issue helped me to have a better sense of what I wanted to talk about and how to do it in ways that would be more effective.” “The training helped me focus on what is really important. It is certainly about having an impact on people. You tell a story, begin with a value, donʼt overwhelm people with facts — focus on key ideas.” Of the remaining few who did not feel like the spokesperson training changed the way they communicate about early child issues, almost all reported that it had helped “reconfirm” things they were already doing well in their messaging, and helped them to sharpen their existing skills, giving them a bit of an edge.

• Usefulness of the Training to Participantsʼ Professional Roles

Participants found both the workshop and the

video training useful to their professional duties, as many of them develop messages for a variety of audiences and stakeholders,

and train others, especially researchers, who talk to the media and are spokespersons on a daily basis. The important finding here is that participants overwhelmingly reported that they have been using what they learned from the training in their professional roles as spokespersons on child development issues. “Often people come to me to ask advice on how to tell the story pretty frequently. My new comfort with that translates into having people want to engage with me to tell a story. I have become a resource for the organization and I am always looking for new ways to bring the information more squarely to the organization.” “I was in the process of preparing a major report on early child development, which has now been released. The release has required me to do a lot of presentations on the topic and in different settings and forums. So the spokesperson training was really timely and it was almost like a dress rehearsal in a way because the key concepts that I use in talking about brain development are really good now. I enjoyed it and the team was great.” “When I meet with people [in my organization] to work on PowerPoint presentations, I often reflect back to what FrameWorks teaches as a kind of general guide.”

• The Affective Experience of the Training

One of the more residual impacts of the training is the affect it has on the participantsʼ own sense of their effectiveness and their confidence in presenting the science. The unanimous conclusion from

Nine out of the 10 interviewees reported that they are still actively using what they learned as part of the spokesperson training more than six months after the training.

100% of the participants surveyed and

interviewed said that the training raised their

confidence in presenting the science of early brain development.


participants is that they felt more confident in framing and communicating about early child development as a result of the training. “The more I am exposed to it, the more confidant I am. Thatʼs the really important part. That is the successful part. We have to be able to do it ourselves and be able to do it.” Many participants welcomed the opportunity of the training, as they indicated they would be doing a lot of media work in the coming months. As one participant stated, “the experience paid off for me.” “I remember coming away from it being really excited about what I heard, and energized in terms of things I learned, and looking forward to taking that stuff away and using it.” “I found it very enjoyable and a good learning experience. I was very appreciative of the opportunity to come.”

• Outcomes Surely the most important finding in terms of the outcomes of the training relates to the extent to which the Core Story (the overall combination of values, metaphors and principles that make up FrameWorksʼ reframes for early child development) is becoming the basis for a new language of early child development. As a result of these trainings, the field now has a common language, an asset that is highly prized in the framing research literature as an indicator of effective social movements. “I see elements of the core story being picked up in work that the ministry is doing and the language is being reflected in some of the materials the ministry is doing or producing. Everyone is starting to speak the same language.” Participants readily reported that another outcome of the training and their resulting improvements in practice was that they were

increasingly called upon to serve as spokespersons in a wide variety of policymaking and other forums. “The number of requests for the workshops by policymakers is increasing exponentially. And, at least in Alberta, there is more and more interest initiated by government officials and those that work with policymakers.” “We are getting more and more requests to give these workshops to not only the school districts, but the teachers and the civil servants. It [the spokesperson training] is working. FrameWorks has helped a lot in helping us get through to people more than we were able to do before.” Finally, participants also reported more confidence that the media would interpret and report their messages with greater fidelity. “I feel more comfortable with how the media has interpreted my message.” “The message I was trying to get across — the media got it! Based on what they wrote and what they sent back to me, they got it. This is mainly in print media, which is huge for me. It got the points across and included the points that I wanted to get across and never veered off in some odd direction that I wasnʼt comfortable with.”

• Sharing the Knowledge Participants overwhelmingly said that they would recommend this training to their colleagues, and all said that they expected to share this information with colleagues in some way. In fact, about half the

100% of the participants surveyed and interviewed said

that they would recommend the training to their colleagues

and would share the information from the training

with them. About 50% said that they had already shared

what they learned.


participants reported that they had already shared some portion or all of what they learned at the training with their colleagues, supervisors, faculty and trainees. “I have talked to our communications folks about this since I deal with media on a weekly basis.” “Actually, I already shared it with the communications team here. They seemed interested in it and we had some discussion around some of the techniques I found to be really useful.” “We talk about brain development all the time. ʻServe and Returnʼ is the one that my team got right away.” Participants generally felt that management was supportive of their investments in greater framing fluency and appreciated the training. Seven of the 10 interviewees said they had noticed more support for strategic framing as more opportunities to speak on child development issues have emerged. “They [management] realize the importance of having strong messages and the importance of having someone be able to walk them through the swamp when they are dealing with media.” Even so, about half of those interviewed said that they were not sure that their organizational leadership really understood or wanted to

expand the focus on framing. The reasons they gave to explain this dissonance were diverse. Two participants felt it was difficult to share what they learned with people in their organizations who

were used to doing things “the same old way,” were resistant to change, and had been “in the

system for a while.” Three other participants felt that not enough people in their organizations had fully embraced framing yet. “Most of them think I am nuts to be doing it at all. Most of my colleagues donʼt see the point. They donʼt understand the concept of knowledge translation in this way.” One participant talked about her attempts to train junior researchers, a practice she found ultimately unsatisfactory, however, because the younger scholars appeared to be more focused on their basic research and less on how that research could be translated for broader audiences. These comments suggest the need for a broader conversation among leading science institutions and at the university level of the importance of knowledge translation. When asked about the arguments they had tried to make to their organizational leadership about why a focus on strategic framing was important, the answers varied here as well. “I just say ʻIf itʼs not about your work and getting people to use the knowledge you are generating from your work, then whatʼs the point.ʼ” “The public pays our salary, so we should give back. In order to affect public policy regarding funding of research, you need people in the public to understand why the research is important. This not just ivory tower stuff.” “If we donʼt do it, it is never going to happen. At some point you are going to have to pay back what the public has given you.” “a) We need to do this and b) this is the time to get this done.” Participants unanimously said they would recommend this kind of training to their colleagues. “I would definitely recommend they (my colleagues) do it, without hesitation!”

More than half of the participants said that sharing the information they received was made somewhat more difficult because their organizational leaders were just beginning to embrace the concept of framing.


“I think for public health officials to know how to work with the media is a critical tool. If you canʼt work well with the media, itʼs like a surgeon that doesnʼt know how to use a scalpel. You really need that skill set, and here is a really good group of trainers. Your team is providing a top-notch experience.” “Yes, I think it is extremely valuable, even if you never get in front of a camera or do any media interviewing at all. It is important to think about your messaging and how you deliver messages — I would recommend it to all managers within my organization, the CEOs and senior management. Anyone that carries messages for the organization needs to do this training.”

• Other Barriers to Using or Sharing the Information from the Training

Most participants reported that they saw no barriers to using or sharing the information from the training. The common response to this question in the interviews was that, even after the training, it would take some time for them to fully integrate, synthesize and internalize the reframes in their work. Most said that they could use additional practice to really “get good” at framing and they found the challenge of thinking on their feet during the video interviews especially difficult. “It was fairly interactive and had the simulation. But unless you do it every day, doing it on the spot is harder to implement. To be able to do it fluently is difficult. I need more practice because I donʼt do this every day.”

• Additional Topics to Address in the Spokesperson Training

Given their skepticism that one set of messages could reach all audiences, a good portion of the participants thought the training should include more examples of talking to diverse audiences (i.e., talking to public audiences versus policymakers, deputy ministers and the like).

This is easily accommodated within the FrameWorks practice. In addition, two participants explicitly wanted more information about the “call to action” and wondered whether the frames would serve to move individual behaviors as well as they do policy support. This represents a need for FrameWorks to incorporate a more robust discussion in the workshop portion of the training about the difference in framing for behavioral versus broader social understanding.

• Areas for Improvement and Development

We also asked participants what kinds of additional resources would have improved their experience. Participants were generative on this issue. First, two participants thought it would have been useful at the beginning of the training for people to talk about their own communications challenges. While the training was useful for the group to get to know about FrameWorksʼ research, some felt it didnʼt allow sufficient room for participants to really use each other as resources for learning. Second, three participants said that they would like an online resource containing access to materials, video clips and sample interviews with the participants, as well as the before-and-after examples and tips. In particular, at least six of the participants wanted access to the video clips of their individual sessions to share with their colleagues. Third, two participants asked for a “cheat sheet” or a quick refresher handout that summarized some of the main concepts. Participants talked about the training itself as being a kind of “cheat sheet,” but a couple wanted this idea realized in a print document. One of the two participants who made this suggestion thought a fridge magnet that said, “the real question is…” could serve as a way to remember to pivot, as well as to internalize the lessons learned.


The FrameWorks Institute reserves all rights associated with this publication. Please follow standard APA rules for citation, with FrameWorks Institute as publisher. Manuel, T., & Lo, S. (2011). Spokesperson training evaluation conducted for the Alberta Family Wellness Initiative. Washington, DC: FrameWorks Institute.

FRAMEWORKS ONLINE RESOURCES The FrameWorks Institute publishes its research and recommendations on our website at www.frameworksinstitute.org. To learn more about this issue, our learning engagements, or our assessment techniques, please visit the website.

.me s Some Other Helpful Resources: § Cultural Models Report on How Albertans

Think about Early Child Development § Experimental Research Report on Values § Alberta ECD Messaging Toolkit Some Resources Related to Strategic Frame Analysis™: § Strategic Frame Analysis eWorkshop

§ An Introduction to FrameWorks Perspective

and Strategic Frame Analysis™

Lastly, although most people felt there was enough practice built into the training, four of the 10 interviewees said they would have liked another opportunity to do the video training session again after some time for reflection and practice. This was primarily because many felt the concepts learned needed time to be fully integrated, synthesized and internalized. So, a shorter (2-3 hour post session) refresher course was recommended as a way to help participants revisit and reinforce the concepts again. SUMMARY Taking into account the totality of findings in this evaluation brief, several suggestions for subsequent rounds of the spokesperson training seem especially germane.

• Build in more upfront peer-to-peer learning opportunities so that participants can get to know each otherʼs communications challenges going into the training.

• Support effective integration, synthesis

and internalization of concepts after the training ends.

o Provide a quick refresher

handout that participants can take with them.

o Generate creative ways to help

participants practice and internalize concepts.

o Conduct a second video training

session to allow participants to incorporate the feedback from the initial session.

• Introduce an online resource that

provides access to all materials, including the suite of video sessions, so participants can effectively share this information with their colleagues. This might also help participants to more

readily and effectively seed framing into their organizational priorities, a challenge they identified to the evaluators.

• Incorporate discussion topics into the

workshop presentation to address two issues of concern to participants: (1) that frames can be nuanced to adapt them to different audiences, and (2) the difference in framing for behavioral change at the individual level versus reframing for broader social understanding

Documents

Childrens Issues Evaluation Brief Spokesperson Training Evaluation Conducted for the Alberta Family