Upload
keanu
View
42
Download
0
Embed Size (px)
DESCRIPTION
Measurement and Evaluation. Jim Macnamara PhD, FPRIA, FAMI, CPM, FAMEC Professor of Public Communication University of Technology Sydney. Where are we? Where to next?. This discussion. A quick update on international standards Continuing gaps and inconsistencies - PowerPoint PPT Presentation
Citation preview
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
Jim Macnamara PhD, FPRIA, FAMI, CPM, FAMEC
Professor of Public CommunicationUniversity of Technology Sydney
Where are we? Where to next?
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
This discussionThis discussion A quick update on international standards Continuing gaps and inconsistencies Some further barriers to overcome
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
Barcelona Declaration 2010 Barcelona Declaration 2010 Goal setting and measurement are fundamental aspects of any
PR program Measuring outcomes is preferred to measuring outputs The effect on business [organisation] results should be
measured where possible Media measurement requires quantity and quality Advertising value equivalents (AVEs) are not the value of PR Social media can and should be measured Transparency and replicability are paramount
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
International standards pushInternational standards push Coalition for Public Relations Research Standards
• Established 2011 by AMEC, IPR and CPRF Social Media Standards Conclave (#SMMStandards)
• Established 2012 by Coalition + 8 other PR/communication organizations– Global Alliance, PRSA, CIPR, IABC, SNCR, FIBEP, WOMMA, DAA
• Working with eight companies– Dell, Ford, General Motors, McDonalds, Proctor & Gamble, Thompson
Reuters, South-West Airlines, SAS
• Also consulting with – Media Ratings Council, AAAA, ANA, Web Analytics Association
Measurement Summit discussions 2011–2013 Academic input?
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
M&E outputs and guidesM&E outputs and guides Proposed Interim Standards for Metrics in Traditional Media
Analysis Social Media Standard Definitions for Reach and Impressions Sources and Methods Transparency Table
Institute for Public Relations (IPR)
Association for Measurement and Evaluation of Communication (AMEC)
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
• Message recall/ retention• Consideration• Responses• Feedback• Engagement
Evaluated against objectivesCommunication
Objectives
PR Activity Undertaken
PR Outputs Analysed
PR Outcomes Analysed
For example:
• Media relations• Publications sent• Events held• Sponsorships undertaken• Communication engaged in
Organisation/Business Results
Organisation/Business Objectives
Did activities generate outputs that reached publics
effectively and efficiently?
• Audience reach• Tone/sentiment of communication• Key messages• Share of voice• Competitor bench- marking
What outtakes and outcomes were achieved among key publics and to what extent?
• Awareness• Understanding• Consideration • Attitudes/perceptions• Action/behaviour• Relationships• Reputation
• Brand positioning• Sales/profits• Market share• Share price• Donations/grants• Customer/staff retention
PRIA Evaluation Model
MonitoringMonitoring MeasurementMeasurement ReportingReporting
Formative ResearchFormative Research
Did organisation/business results shift as a consequence?
Evaluative ResearchEvaluative Research
Q:Q:
PR Outtakes Analysed
Evaluative ResearchEvaluative Research
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
How far we’ve come …How far we’ve come …
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
“… REACH represents the total number of unique people who had an opportunity to see an item”
#SMMStandards, 2012, para. 15.
IMPRESSIONS– the number of people having the opportunity for exposure to a media story; also known as ‘opportunity to see’ (OTS) … usually refers to audited circulation”
Proposed Interim Standards for Metrics in Traditional Media Analysis (Eisenmann et al., 2012, p. 3)
Impressions – “the number of people who might have had the opportunity to be exposed to a story”
Dictionary of Public Relations Measurement and Research (Stacks & Bowen, 2013, p. 14)
“Impressions represent the number of times an item was displayed”
“Impressions represent the gross number of items that could have been seen by all people, including repeats”
Social Media Standards Definitions: Reach and Impressions (Digital Analytics Association, 2013)
Impressions represents “the number of possible exposures of a media item to a defined set of stakeholders” …
Dictionary of Public Relations Measurement and Research (Stacks & Bowen, 2013, p. 14)
Definitions – e.g., ‘reach’ and ‘impressions’Definitions – e.g., ‘reach’ and ‘impressions’
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
Return on Investment (ROI) PR ROI Return on Impressions (ROI) Return on Media Investment (ROMI) Return on Target Influence (ROTI) Return on Earned Media (ROEM) Return on Expectations (ROE) Return on Communication Investment (ROCI) Return on Brand Investment (ROBI) Social media ROI (SROI)
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
MetricsMetricsBasic outputs Outputs Outtakes Outtakes Outcomes
Counting press clippings Unique visitors Engagement
Audience Views Influence
Reach Likes Impact
Target audience reach Followers Awareness
Impressions Fans Attitudes
Opportunities to see (OTS) Clickthroughs Trust
Share of voice Downloads Loyalty
Cost per thousand (CPM) Comments Reputation
Hits Tone Relationships
Visits Sentiment ROI?
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
• Message recall/ retention• Consideration• Responses• Feedback• Engagement
Evaluated against objectivesCommunication
Objectives
PR Activity Undertaken
PR Outputs Analysed
PR Outcomes Analysed
For example:
• Media relations• Publications sent• Events held• Sponsorships undertaken• Communication engaged in
Organisation/Business Results
Organisation/Business Objectives
Did activities generate outputs that reached publics
effectively and efficiently?
• Audience reach• Tone/sentiment of communication• Key messages• Share of voice• Competitor bench- marking
What outtakes and outcomes were achieved among key publics and to what extent?
• Awareness• Understanding• Consideration • Attitudes/perceptions• Action/behaviour• Relationships• Reputation
• Brand positioning• Sales/profits• Market share• Share price• Donations/grants• Customer/staff retention
PRIA Evaluation Model
MonitoringMonitoring MeasurementMeasurement ReportingReporting
Formative ResearchFormative Research
Did organisation/business results shift as a consequence?
Evaluative ResearchEvaluative Research
Q:Q:
PR Outtakes Analysed
Evaluative ResearchEvaluative Research
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
The so-called barriersThe so-called barriers Cost Time Too hard Management don’t want it
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
Major barriers to addressMajor barriers to address Conflation of measurement and evaluation
Measurement is taking measures – metrics Evaluation is determining VALUE Value = a perception from a perspective
Measurement Evaluation
Early findings used to fine-tune campaign
Data collection Data analysis
Findings primarily used for reporting
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
Major barriers to addressMajor barriers to address Most measurement and evaluation look backwards
• “Tells us what we know intuitively”• “What’s done is done … can’t change the past”• “Post-rationalisation and self-justification”
Look forwards• What insights can improve PR/communication/relationships in future?• What insights can contribute to the organisation’s strategy and
outcomes?– Gaps– Trends– Opportunities– Competitor strategy– Emergent issues
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
Obsession with numbersObsession with numbers
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
ConclusionsConclusions Recognise measurement and evaluation as different and
separate• One is about metrics and analytics, the other is about VALUE
Collect qualitative research data as well as quantitative• Noting that many outtakes, outcomes and outflows are perceptions and
attitudes (e.g., satisfaction, trust, reputation) and relationships Look forward, not simply reporting retrospectively
• Contribute to organisational strategy and outcomes
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
Professor Jim Macnamara PhD, FAMI, CPM, FPRIA, FAMEC
University of Technology Sydney
References/resourcesReferences/resources Interim Standards for Metrics in Traditional Media Analysis
http://www.instituteforpr.org/topics/proposed-interim-standards-for-metrics-in-traditional-media-analysis
Social Media Standards ‘Sources and Methods Transparency Table’http://www.smmstandards.com/category/content-sourcing-methods/ ‘Social Media Standard Definitions for Reach and Impressions’,
http://www.smmstandards.com/2013/03/proposed-social-media-standard-definitions-for-reach-and-impressions-from-the-digital-analytics-association/. Also at http://www.smmstandards.com/wp-content/uploads/2013/03/SMM-Standard-Definitions_DAA_v4.pdf
Jim Macnamara, (2014), ‘The development of international standards for measurement and evaluation of public relations and corporate communication: A review’, available at http://www.uts.edu.au/research-and-teaching/our-research/australian-centre-public-communication/research/research-reports
Tom Watson & Anzgar Zerfass, (2011), ‘Return on investment in public relations: A critique of concepts used by practitioners from communication and management sciences perspectives’, PRism, vol. 8, no. 1, available at http://www.prismjournal.org/vol8_1.html