1

ClickerPlus Pilot Study:Applying Advertising Research Methods to the

Measurement of Student Learning Goals

Teaching and Technology Conference

Baruch College

26 March 2010

Prepared by: Brian Moore

Agenda

• Background• Advertising research methods • Students’ preconceptions• ClickerPlus Method• “Survey” the audience• Results of the pilot study• Results of the audience survey• Discussion

3

What is ClickerPlus?

• Quantitative method – What college students know– What they learn

• Pre/post measurement technique • Borrowed and adapted from advertising

research• Student response systems (a.k.a. “Clickers”)

Hypothesis

• ClickerPlus made me a better teacher:– Students learned (more)– Students appreciated the class (more)

Advertising Research Methods

How do we know that advertising is effective?

Case History:

Dove Deodorant

Tracking Study

(New)Dove Deodorant Moisturizing cream

Soothes and beautifies the underarms.

8

Dove Deodorant Communication Goals

• Be recognized as the best brand at caring for the look and feel of underarm skin

• Become the beauty icon in the deodorant market through focusing on improving the look of underarm skin

• Ensure that perceived efficacy remains at parity with the market average

– Source: Euro Effie Awards Case History, 2009

To achieve the goals, people have to “learn” :

Benchmark Survey

(pre-wave)

Wave 1 Survey

(post-wave)

“leaves underarm skin soft and smooth”

X% X+%

“helps me cope with the problems of shaving”

Y% Y+%

…

10

Advertising is designed to achieve these goals.

Results

12

Dove deodorant “owns” underarm beauty.

Papers & Case Studies – World Advertising Research Council

How do I know that I am an effective teacher?

Establish concrete learning goals

Measure what students know

Measure what they learn

What do students know?Are they Blank Slates?

Intrinsic aptitudes

Today, the sciences of human nature have threatened the Blank Slate by trying to delineate what has to be

present in the mind in order for learning to occur in the first place.

Stephen Pinker

The Blank Slate

2002

17

ClickerPlus Pilot Study Results Summer 2009

Prepared byBrian Moore

Baruch CollegeJuly 2009

18

Method

• Construct a series of true-false statements • Have the students respond to the statements

– Using classroom response systems (clickers)– TurningPoint

• Analyze the responses.• Adjust ensuing lectures and tests accordingly.• Clarify and reinforce (circa mid-semester). • Re-administer on the last day of class.• Compare the “pre-wave” to the “post-wave.”• Adjust future class lectures and tests accordingly.

19

Learning

End of the semesterJuly 2, 2009

Beginning of the semesterJune 1, 2009

True/False questions True/False questions

Benchmarks Changes over time

20

Sample Description and Timing

• Place: Baruch College • Course: Advertising & Promotion (MKT3520)• Sample Composition: Baruch undergraduates• Sample Sizes:

– Pre-Wave, n=38; – Post-Wave, n= 25

• Timing: – Pre-Wave: June 1, 2009– Interim Review: June 18, 2009– Post-Wave: July 2, 2009

21

Limitations

• Results are merely directional.• Sample is insufficient, i.e.

– Small– Unequal from wave to wave– Unrepresentative

• Need to know if the statements accurately measure the actual grasp of the underlying concept.

22

Instructions to the students in the classroom

This is not a test. You won’t be graded on this. We just want to find out what you know.

23

First Slide

Based on everything you know or may have

heard about advertising, please indicate

whether you believe that the following

statements are true or false.

Pick up your Clickers

25

The 10 Statements

1. The most important thing good advertising does is make an emotional connection with consumers.

2. The production of advertising ideas follows a clearly defined process.3. Good advertising can keep a bad product alive.4. American Idol would be a good place to advertise the new version of the iPhone.5. What you say in advertising is more important than how you say it.6. Emotional advertising appeals are more effective than rational advertising appeals.7. Advertising increases the value of the advertised product.8. If your advertising grabs people’s attention, it will probably work.9. If you like a particular product, you will be better prepared to create great advertising

for that product.10. It is more effective to have your advertising seen or heard by more people than it is

to have fewer people see it more often.

How did the students do?

27

Summary of before/after results

For all 10 Statements

28

On some questions, students improved dramatically over the course of the semester. On

other questions, they did not improve at all.

Correct Answers

22

63

13 12

74

85

21

3639

73

65

35

67

57

75

57

36

9

7574

0

10

20

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10

Statements

Percentage

Jun-09 Jul-09

29

Results for each statement, before and after the semester

30

The most important thing good advertising does is make an emotional connection with consumers.

2239

7861

0102030405060708090

100

Jun-09 Jul-09

Percentage

True

False

False: “If it doesn’t sell, it is not creative.,” David Ogilvy (Belch & Belch 285)

31

The production of advertising ideas follows a clearly defined process.

3826

6374

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

True: “The production of ideas is just as definite a process as the productionof Fords.,” James Webb Young (Belch & Belch 260).

32

Good advertising can keep a bad product alive.

13

73

88

27

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

False: “Nothing will put a bad product out of business faster than a goodadvertising campaign.,” David Ogilvy

33

American Idol would be a good place to advertise the new version of the iPhone.

9

65

91

35

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

False: It would be a waste of mass media on an exclusive product, i.e. a poor return

on investment (Belch & Belch 354).

34

What you say in advertising is more

important than how you say it.

88

65

12

35

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

True: “What really decides consumers to buy or not to buy is the content of your advertising, not its form.,” David Ogilvy (Belch & Belch 291).

35

Emotional advertising appeals are more

effective than rational advertising appeals.

26 25

74 75

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

True: You should always seek emotional benefits. They are more compelling,and they are easier to defend (Belch & Belch 285).

36

Advertising increases the

value of the advertised product.

1533

8567

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

True: The simple act of advertising a product makes it appear more valuable. Good advertising can increase value perceptions and direct pricing strategies (Moore).

37

If your advertising grabs people’s

attention, it will probably work.

21

57

79

43

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

False: “Advertising succeeds or fails depending on how well it communicatesthe desired information and attitudes to the right people at the right time and at the right cost,” Russell Colley (Belch & Belch 220).

38

If you like a particular product, you will be better prepared to create great advertising

for that product.

36

75

64

25

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

False: You may not be representative of the target consumer, and you won’t be objective (Moore).

39

It is more effective to have your advertising seen or heard by more people than it is to have fewer

people see it more often.

36

57

64

43

010203040

5060708090

100

Jun-09 Jul-09

Percentage

TrueFalse

False: It depends on your objectives, e.g.; if your message is complicated, or If your product is unfamiliar to people, you probably need more frequency (Belch & Belch 336).

Am I an effective teacher?

Did the students learn?

Did they appreciate the class?

41

Did the students learn?: Yes and No

1. The most important thing good advertising does is make an emotional connection with consumers.

2. The production of advertising ideas follows a clearly defined process.3. Good advertising can keep a bad product alive.4. American Idol would be a good place to advertise the new version of the iPhone.5. What you say in advertising is more important than how you say it.6. Emotional advertising appeals are more effective than rational advertising appeals.7. Advertising increases the value of the advertised product.8. If your advertising grabs people’s attention, it will probably work.9. If you like a particular product, you will be better prepared to create great advertising

for that product.10. It is more effective to have your advertising seen or heard by more people than it is

to have fewer people see it more often.

42

Did the ClickerPlus students appreciate the class?:(Yes, but I made a lot of other changes as well).

Student Course and Faculty Evaluation

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Course Content Course Delivery Learning Environment Impact on you

Mean Ratings on a 5 Point Scale

Fall 2008 Summer 2009

How did the audience do?

44

Potential ClickerPlus Benefits

• Objective • Flexible• Infinitely scalable• Easy to construct and administer• Several types and levels of learning • Yields real-time results• Measures the attainment of learning goals• Inspires new goals • Provides benchmarks for:

– Overall improvement– Developing individual goals for students

• Preserves and protects academic freedom

Conclusions

• ClickerPlus made me a better teacher– Learned– Appreciated

• Need more data• Incorporate student test scores• Construct better and larger samples• Validate the survey

– Qualitative research– Search phrase algorithm

• Automate the survey

Next Steps

• Awaiting response to our grant proposal from the U.S. Department of Education– More instructors– Different subjects– Different levels– $$$

• …• Brian.Moore@baruch.cuny.edu• 646-312-3332

47

Works Cited

• Belch, George, and Michael Belch. Advertising and Promotion. New York: McGraw-Hill, 2009

• Georgia Tech School of Civil and Environmental Engineering. [2000] Instructional objective writing assistant

• Moore, Brian. Lectures on Advertising and Promotion, Baruch College, New York, June 2009.

• Ogilvy, David. Various lectures, speeches and books.• Pinker, Steven. The Blank Slate. New York: Penguin

Group, 2002

THE END

48

49

Appendix

Scatter diagram

LEARNING GOALSAt the end of this course, you will understand how global advertising agencies propose integrated marketing communication programs to global manufacturers of consumer package goods. You will learn how to conduct research to define the target audience and its media habits, brainstorm to generate messaging ideas, develop a creative strategy, create executions derived from the strategy, produce the executions, test whether or not the executions communicate their intended messages, plan and buy the media to deliver the messages to the target audience, and incorporate all of your findings into a persuasive, visually-intensive, oral presentation. You will learn how to work as part of a cross-functional team to achieve the goals stated above.

51

The technique discriminates, and more students answered the questions correctly at the end of the semester than they

did at the beginning of the semester.

0

20

40

60

80

100

0 1 2 3 4 5 6 7 8 9 10

Statements

Percent Correct

Jun-09

Jul-09

52

53

54

Formatting Learning Goals

• Student centered (e.g., “by the end of the course, students will be able to…”),

• Outcome oriented (e.g. “write a scholarly research paper…”), and

• Detailed (e.g., “using MLA-style referencing and formatting…”).

Georgia Tech [2000]

COM3150 Course Review

55

My Learning Goals1. Defend the primacy of strategic messaging.2. Describe and apply the advertising planning process.3. Recognize why good advertising cannot keep a bad product alive.4. Analyze the efficiency of different media vehicles, using syndicated

research.5. Explain why the content of your advertising is more persuasive than

its form.6. Discuss why emotional benefits are more persuasive than rational

benefits.7. Describe how and why advertising increases the value of the

advertised product.8. Interpret and explain the hierarchy of effects model.9. Identify target audiences by analyzing syndicated research.10.Demonstrate the trade-offs between advertising reach and

frequency.

56

Less Effective Example Effective Example

This course in Engineering Graphics will introduce students to the use of computer-aided-design software

At the end of this course in Engineering Graphics, you, the student, will be able to draw a multi-view representation of a solid object using a computer-aided-design software

Georgia Tech [2000]