30
RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist New York City Department of Health and Mental Hygiene

RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH AND

FOCUS OF NEW MEDIA OUTREACH

Anthony Freeman, MASenior Evaluation Specialist

New York City Department of Health and Mental Hygiene

Page 2: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

CONFLICT OF INTEREST DISCLOSUREANTHONY FREEMAN, MA

none.

Page 3: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

YAHOO

FACEBOOK

TWITTER

HOOTSUITE

JACK’D

GRINDR

CRAIGSLIST

SCRUFF

MEETUP

PINTEREST

INSTAGRAM ADAM 4 ADAM

WHATSAPP

TUMBLR

PLENTY OF FISHHORNET

SNAPCHAT

YOUTUBE

TINDER

“NEW MEDIA” = SOCIAL MEDIA + DATING APPS

Page 4: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

CONTINUOUS QUALITY IMPROVEMENT

“The purpose of continuous quality improvement (CQI)

programs is to improve health care by identifying

problems, implementing and monitoring corrective action

and studying its effectiveness.”- National Commission on Correctional Health Care

Page 5: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

LEARNING OBJECTIVES

1. Explain how to evaluate a social media plan in the context of HIV

prevention

2. Understand how to measure your agency’s social media efforts

and return on investment

3. Describe how to use data to help inform programming and

strategic changes

Page 6: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

OVERVIEW

• Background: PlaySure Network and Social Media/New Media

• Developing Social Media/New Media Evaluation

– Creating Engaging Content

– Internal Workgroup

• Documenting Social Media efforts

– Content Calendar

– Social Media Reporting Log

• Preliminary Data

• Discuss Future Plans: Evaluation and Technical Assistance (TA)

Page 7: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

PLAYSURE NETWORK: NEW NETWORK, NEW MEDIA

1. Promote approaches to sexual healthand combination HIV prevention thatare appropriate for each client

2. Increase access to PrEP and PEP forpeople who may be exposed to HIV –especially priority populations who areuninsured

3. Link people who test positive for HIV,or those who are HIV-positive and outof care, to HIV care services so they canget treated and become virallysuppressed<40 agencies total

17 agencies with new media deliverables

Page 8: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

GOALS OF SOCIAL MEDIA

Page 9: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

EXAMPLE: FACEBOOK EVENT PROMOTION VOCES LATINA

Page 10: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

EXAMPLE:

AWARENESS OF SERVICES (PAID ADS)

MT. SINAI

Page 11: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

EXAMPLE:DATING APP OUTREACH (GRINDR)CAMBA

Page 12: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SOCIAL MEDIA EVALUATION

• Strategies to evaluate the success ofsocial media/new media have laggedbehind scale-up

• We developed strategies to monitorand evaluate use of new media forHIV prevention outreach with the goalof increasing capacity through data-driven technical assistance (TA) to HIVprevention programs

Page 13: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

METHODS

• Workgroup and stakeholder engagement

• Content Calendar and Post Projections

• Log/Indicators

– Quality Improvement

– Examination of results

• Comparing log data to referral source from service data

Page 14: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SOCIAL MEDIA WORK GROUP

Page 15: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

CONTENT CALENDAR

Date Platform Message Staff Responsible

Page 16: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

CONTENT PROJECTIONS

Page 17: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SOCIAL MEDIA WORKFLOW – ADDING EVALUATION STRATEGIES AT EVERY STEP

PEOPLE

REACHED

ENGAGEMENT/

INTERACTION

PEOPLE

REFERRED

NEW

CLIENTSPOSTS

Page 18: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SOCIAL MEDIA CORE INDICATORS

• Platforms utilized(list of sites and dating apps)

• # of posts

– # of banner ads(social media & dating apps)

• # of interactions/engagement

per post

– # of views/impressions per banner ad

• # of direct messages

• # of referrals to agency(documented through social media interaction)

• # of new clients(documented as referral source at enrollment)

process

outcome

Page 19: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SOCIAL MEDIA MONTHLY LOG

Page 20: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SOCIAL MEDIA WORKFLOW

PEOPLE

REACHED

ENGAGEMENT/

INTERACTION

PEOPLE

REFERRED

NEW

CLIENTSPOSTS

Page 21: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

AGGREGATE NEW MEDIA DATA(AUGUST 2017 – JULY 2018)

*PRELIMINARY DATA

PEOPLE

REACHED

ENGAGEMENT/

INTERACTION

PEOPLE

REFERRED

NEW

CLIENTSPOSTS

7118 totalRange: 0 – 125 per month

Page 22: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

AGGREGATE NEW MEDIA DATA(AUGUST 2017 – JULY 2018)

*PRELIMINARY DATA

PEOPLE

REACHED

ENGAGEMENT/

INTERACTION

PEOPLE

REFERRED

NEW

CLIENTSPOSTS

15,293,010 totalRange: 0 – 1,631,680 per post

161,559 totalRange: 0 – 6069 per post

829 totalRange: 0 – 50 per month

Page 23: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SERVICE DATA: 243 INTAKES FROM SOCIAL MEDIA(AUGUST 2017 – JULY 2018)

*PRELIMINARY DATA

2

164

243

300

333

717

975

998

1077

1173

0 200 400 600 800 1000 1200 1400

PEP Call Center

Flyer/advertisement

Social media

Sexual Health Clinic

Word of mouth

Internet search

Other

Outreach Event

PlaySure Network Site

Existing client at the agency (in-reach)

number of PlaySure Network intakesAugust 2017 - July 2018

Ref

err

al S

ou

rce

Page 24: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

232/243 CLIENTS (95%) FROM SOCIAL MEDIA FROM A PRIORITY POPULATION(AUGUST 2017 – JULY 2018)

*PRELIMINARY DATA

social media

243

Priority

population*

232

All other

referral sources

5739

Non-priority

population

11

*MSM (including YMSM), TGNC, BLWSM, etc.

Page 25: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

65%Of the total clients who

entered the PlaySure Network

through social media and were

eligible for a PrEP prescription

received a PrEP prescription.**(clinical sites only)

(August 2017 – July 2018)

Page 26: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SUMMARY

• Collecting and analyzing aggregate new media monitoring data was

critical in supporting agencies.

• Evaluation strategies:

– Helped contracted agencies complete their new media

activities

– Helped HD staff gauge whether agencies were meeting their

specific goals

– Helped centralize new media best practices

Page 27: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

SUCCESSES & CHALLENGES

Successes

• Standardized new media indicators for entire PlaySure Network

• Helped agencies do more with less

• Increased capacity of both DOHMH and agency staff

Challenges

• Diversity of capacity to execute social media among agencies

• Diversity of DOHMH staff knowledge/experience with social media

• Potential for error at every step (e.g. log data entry mistakes)

Page 28: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

FUTURE PLANS

• Compare performance by agency and service category

• Social Media 101 Training

• Evaluate new clients referred through social media

• Move log submission to web form to reduce submission burden

Page 29: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist

ACKNOWLEDGEMENTS

• Contracted Agencies

• NYC DOHMH Prevention ProgramStaff

– Testing TA Unit

– Prevention TA Unit

– Research & Evaluation Unit

• Julie Myers

• Oni Blackstock

• Co-authors:

– Nancy Morisseau

– Stephen Jean III

– Lucila Wood

– Carolina Alcala

• Zoe Edelstein

• Divya Bisht

• Emily Appel

• Sarah Ramteke

• Fardina Miah