Usability Test White Paper

8/2/2019 Usability Test White Paper

1/12

InDesign Team 1

Tianna Drew, Danie Merriman, Justin Shearer, Aerin Truskey

ENGL317

April 13, 2012

Relative Effectiveness of Positive and Negative Framing

in Increasing Public Support for Wildfire Mitigation Efforts

Abstract.

Using differently designed and framed information in the form of losses and gains frame

infographics, we assessed the effects of each of these frames on participants views on

wildfire mitigation efforts in the Rocky Mountain west. With a pre-test to gather data on these

participants existing opinions and knowledge, followed by one of the infographic frames, and

finally a post-test, we measured whether and by how much each infographic frame changed

participants ideas about wildfire management.

The results of our surveys, interviews, and a focus group were admittedly mixed. Certain survey

and interview statements (regardless of frame) produced dramatic changes in participants

responses, while others proved less subject to this change. Furthermore, some of these

statements showed that subjects changed their answers in response to one infographic frame

but not the other. Some of our results may be very helpful to further studies of framing and its

effectiveness in communicating land management agencies desired message; however, many

of our participants reported that certain survey and interview statements were confusing, vague,

irrelevant, or even all of the above. In light of this we would suggest that any future researchers

amend their surveys and interviews to better reflect the information featured in the infographics,

and take caution to write these tailored materials clearly and concisely.

Introduction.

In this pilot study we tested the effects of differently presented information, specifically in the

form of two differently designed infographics, on participants opinions on and knowledge of

fire mitigation efforts (particularly prescribed burns and mechanical thinning). We administered

these different presentations of information to each participant in one of two forms, called

frames. The gains frame, also called the positive frame, presented data and arguments

in terms of what landowners have to gain from taking the suggested action. Its counterpart

the losses frame, also known as the negative frame, focused on potential detrimental

consequences to these same landowners if they did not take the suggested action.

Using a numerical Likert scale to gauge subjects agreement with certain statements or levels

of knowledge on the topic, we collected their responses both before and after administering oneof the two frames. We have based our conclusions on whether subjects changed their views

in response to the information we presented to them, and if so, by how much, along with the

reasoning underlying their change in response (when available). We are assuming that changes

in numerical response are positively correlated with an infographics effectiveness in educating

and persuading the subject.

The results of this initial study will inform future research into whether positive or negative


2/12

InDesign Team 1


ENGL317

April 13, 2012

framing of information is more effective in persuading the public to support their local land

management agencies wildfire mitigation efforts. Given the results of that later research, said

management agencies might distribute the most effective information to the landowners in their

area, garnering the public support that is so important to achieving their land management

goals.

Methods.

In the weeks of March 19th and 26th, researchers conducted a number of surveys, interviews,

and focus groups to collect data. Using interview guides and survey sheets, (Appendices B and

C) we recorded subjects views on and knowledge of wildfire mitigation on a Likert scale from -3

to 3; -3 indicated that subjects strongly disagreed with a given statement, and 3 indicated strong

agreement. 0 represented neutral sentiment or no opinion. These guides were used at during all

methods of data collection and then were recorded on a group document so as to compare the

data.

Surveys

Our team selected participants from existing contacts, and then individual team memberscontacted and questioned each participant. We divided our efforts between each infographicframe; two team members administered to subjects only the positive frame while the other twoused only the negative frame (Figure 1).

Positive Negative

Danie 2X

Justin 2X

Aerin 2X

Tianna 2X

Figure 1. Assignment of infographic frames administered to subjects.Although team members were present while participants completed their surveys, this was only

for the purposes of answering questions that subjects might have concerning the wording or

meaning of a survey statement. We did not read the statements aloud or inquire after subjects

reasoning in responding to any of the survey statements.InterviewsParallel to the survey portion of data collection, our team selected interview candidates from

a pool of existing contacts before members met with individual subjects. Again, the team split

into two groups: those administering the positive frame to our participants and those using the

negative frame (Figure 1).

The interview presented to subjects the same set of statements as did the survey, but rather


3/12

InDesign Team 1


ENGL317

April 13, 2012

than letting participants read and respond to the statements alone, we conducted the survey

verbally (see Appendices B and C). We then recorded the participants responses on the

interview guides, and asked participants to articulate their reasoning when a statement gave

him or her pause, or in the case of the post-test, when the participant had changed his or her

response..

Focus group

The final method of data collection for this study was a focus group. Each team member

recruited one person to attend the meeting wherein we verbally asked all the participants

to respond to, as a group, the statements contained in the survey and interview guide (from

Appendix B/B). That is, we asked the group to come to a consensus on each statement.

We recorded the groups responses, and took note of comments on what could improve the

effectiveness of the infographic or survey.

Results.

In four interviews and four surveys using the positively framed infographic, most statementshad a change in opinion from the pretest to the post-test. Nine out of 24 statements showedan average increase in participants agreement with the statement in question. 12 out of 24statements showed an average decrease in participants agreement, and three statements didnot show any average change in responses from pre- to post-test.

Figure 2a.

Using the same process for the negatively framed infographic, participants changed their post-

test responses to almost every statement on their survey or interview. 16 out of 24 statements

showed an average increase in statement agreement. Five out of 24 statements showed


4/12

InDesign Team 1


ENGL317

April 13, 2012

an average decrease statement agreement, and three statements did not show an average

response change.

Figure 2b.

Figure 2. Raw values of the participants responses for the positively (2a.) and negatively (2b)

framed infographic were averaged, and the trends of these responses entered into a bar graph.

We then compiled all of our subjects raw response-change values, from both infographic

frames, into absolute values so as to compare the magnitude (rather than the direction) of the

change in their Likert scale responses. This allows us to pinpoint the statements that showed

the most variation between pre- and post-test responses, and to analyze these statements

further.

Figure 3a.


5/12

InDesign Team 1


ENGL317

April 13, 2012

Figure 3b.

Figure 3c.

Figure 3.

a.Absolute values for all of the positively framed infographic responses.

b. Absolute values for all of the negatively-framed infographic responses.c. The 11 statements with the most change in response were placed side by side to highlight

patterns in both infographics.

Out of these 11 most variable survey/interview statements, we selected the five statements with

the greatest change in Likert scale responses from pre- to post-test. These greatest changes

could be evident in either the positive or negative infographic, and change does not necessarily


6/12

InDesign Team 1


ENGL317

April 13, 2012

have to be present for both frames. That is to say that some of the selected statements that

elicited a very pronounced response change for one frame did not necessarily show the same

amount of response change for the other frame. This shows that both frames of the infographic

had certain information they demonstrated with more importance than did the other frame.The

following five graphs indicate the statement from the survey/interview as well as our participants

mean responses.

Figure 4. Mean response change for Statement 3. Note: subjects that received the positive

infographic treatment did not show any mean response change.

Figure 5. Mean response change for Statement 6.


7/12

InDesign Team 1


ENGL317

April 13, 2012




8/12

InDesign Team 1


ENGL317

April 13, 2012


Statements 3, 6, 7, 8, and 11 showed the most mean change in response for the positive frame.

These statements cover the use of prescribed burning techniques, fire intensity, fire frequency,

family fire planning, and the role of forests today; all of which are covered in the infographic.

Translates into the infographic having a direct influence on the participants values in these

subject matters.

The focus group was somewhat of a special case, collectively unaffected by administration of

the infographic (in this case the negative frame); on their post-test, they came to consensus on

the decision to not change any of their responses from the pre-test. The details of their reactions

and comments appear below (see Appendix D).

Analysis.

Among all the tests administered, across both the positive and negative frames, there werefive statements that displayed the greatest mean change in participants responses from pre- topost-test. In some instances the data shows that one frame produces greater average changein responses. In others the framing appears to be insignificant, since both frames show similarlylarge change values.

For example, Statement 3 (Fig.4), concerning whether prescribed burning and thinningtechniques make the forest look better, the negatively-framed infographic elicited significantchange towards increased agreement with the statement in question, while the positively-framed infographic did not, on average, change the participants opinion of the matter..The way the infographics were framed sometimes made the participant disagree more stronglywith values and ideas presented in the infographic itself. Two such examples are Statements6 and 7(Fig. 5 & 6), wherein the positively framed infographic appears to make participantsdisagree more with the suggestion that fire frequency and intensity are increasing in the RockyMountain Northwest region. That is, rather than garnering increased concern from viewers, thepositively-framed infographic tends to assuage any pre-existing fears of intensifying wildfires


9/12

InDesign Team 1


ENGL317

April 13, 2012

these viewers might have had. This effect could be advantageous or problematic for landmanagers distributing this particular frame, depending on the managers desired effect. Incontrast, the same positive framing encouraged more agreement from participants with theStatement 8s proposition of making wildfire plans for their families, so participants concern

about wildfire has not been totally nullified after administration of the positive frame.Statement 11(Fig. 8) showed that, regardless of infographic framing, the participants responses

showed marked change in the same direction (specifically increased agreement with the

statement). Both infographics managed to make participants agree more with the somewhat

oversimplified concept that the primary role of forests today is to produce jobs and income.

As mentioned above, the focus group portion of the study did not show any change in the

participants views after administration of the (negative) infographic.This may have had a great

deal to do with the group members backgrounds, all of which included some involvement or

education in wildfire, forestry, or a related field. Since these users were already experienced in

fire management, they did not feel that the information presented in our infographics was novel,and so did not change their responses to the survey/interview statements.

Discussion.

First, it is important to note that, as discussed above, our sample did not necessarily encompassthe target audience that future researchers will be seeking: landowners in the Rocky MountainWest, who live close to public lands which federal or other government agencies manage.Given this discrepancy between our participants backgrounds and the backgrounds of futureresearchers participants, our results may or may not predict future studies outcomes..Concerning said results, in addition to the Likert Scale measurements of each infographics

effects on participants views, we collected viewers comments and observations concerning

the design of each of the frames (detailed below). The negative frame produced the most

feedback, and some comments appeared repeatedly. These common observations included a

lack of context for the figures and statistics displayed in the infographic, misplacement of proper

focus and message, and reports of confusion over vague wording of certain survey or interview

statements.

The most pressing suggestions, that is the ones that participants most readily reported,

concerned design elements specific to the negative frame. (While the above observations were

collected from participants who had viewed the negative frame, they could in fact apply to either

frame since the two infographics share those elements.) Participants from the focus group,interviews, and surveys all posited that the red background of the negative frame was off-

putting, and that it communicated the air of a scare tactic, seeming like propaganda rather

than education.


10/12

InDesign Team 1


ENGL317

April 13, 2012

So while the negative and positive frames contained the same information, it appears that the

negative frame worked against its own purposes by engendering skepticism in its viewers,

making them doubt the veracity of its message. One interviewee went so far as to say that a

gains frame would be more effective, since a positive frame would better connote the message

that wildfire mitigation is a positive action.

Finally, many participants complained of confusing or unclear wording of several survey and

interview statements (namely numbers 2, 5, 7, 10, and 13). These instances of confusion

and misunderstanding were easy to remedy in a focus-group or interview setting, in which

participants can ask questions of the study team, but they had more problematic potential in

a survey situation. For instance, a participant might respond 0 or neutral simply because

the statement was difficult to understand, a response which could have contaminated a

considerable portion of our data set. Similarly, a few participants felt that some survey and

interview statements were deliberately extreme or inflammatory (e.g. Statements 4, 11). In

addition, some of the statements on the surveys and interview guides (which a graduatestudent, involved in the later study of this data, submitted to us) did not appear to coalesce

with the content of the infographics we used. This led to certain statements in the surveys and

interviews that addressed topics absent from our infographics, causing further confusion.

For the sake of more efficient results in future, larger studies of these infographics effects

on viewers perceptions of wildfire mitigation, we suggest that those conducting similar tests

tailor their surveys and interview guides to better match the information presented in their

infographics, and that they direct more focus onto clear, concise writing. Without ambiguities to

confound results, land managers might better ascertain which framing style (if any) is best for

communicating their message to their constituencies.

Appendix D.

Focus group comments.

1. Some of the statements are not worded clearly (#s 7,13); #3 of section 2 is irrelevant; #5 of

section 2 is not specific enough (it depends on the fire).

2. Seems like a scare tactic, induces skepticism, seems like propaganda rather than education

3. What do the numbers (acres) mean? Theres no scale (is that a lot?).

4. Is the poster blaming USFS? If so, thats a distraction from the main message.

5. Prescribed burns are not the only mitigation tool.

6. Dollar amounts are good.7. But where are these burns? Do these (acre) numbers include prescribed burns?

8. Not enough focus on the importance of prescribed burns (tiny box)

9. Why the dollar signs? Why that many of them?

10. It would be better to put together a list ofallmitigation strategies: too much importance given

to bucket, shovel...

Other subjects comments on the negative frame.


11/12

InDesign Team 1


ENGL317

April 13, 2012

1. #s 1, 2, & 10 were not well represented in the infographic (not covered).

2. #s 2 & 10 are not specific/detailed enough.

3. Financial statistic (money lost, in timber, to fire) suggests that the primary role of forests

today... is to produce jobs & income.

4. #2 is poorly worded.

5. All wildfires near my community is not specific enough-- is this community near people/

housing?

6. The fire frequency box needs more info/detail.

7. Red background is off-putting.

8. Seems sensationalist; a positive image would be more convincing, b/c it would better

represent the positive action that is wildfire mitigation.

9. Past years statistics, not just last years, would be helpful (in establishing a trend/pattern).

Subjects comments on the positive frame.1. Informative on acreage burned.

2. Informative on estimated value of forests.

3. For the most part did not really influence a change in responses.

4. Statistics on acreage and the estimated income from forests is questionable.


12/12

InDesign Team 1


ENGL317

April 13, 2012

Appendix A: Pre- and post-test survey/ interview statements and interview guide

Documents

Usability Test White Paper