Evaluating media messages on cancer studies

Project LEAD workshop

Gary SchwitzerPublisher, HealthNewsReview.org

Adjunct Assoc. Prof., School of Public Health, University of Minnesota

Our criteria: Does the story explain…• What’s the total cost?

• How often do benefits occur?

• How often do harms occur?

• How strong is the evidence?

• Are there alternative choices?

• Is the condition exaggerated?

• Is this really a new approach?

• Is it available?

• Who’s promoting this?

• Do they have a financial conflict of interest?

69%

66%

65%

61%

57%

Percent unsatisfactoryafter 1,889 story reviews –7 years

Most common flaws

• Conveying a certainty that doesn’t exist

– Exaggerating effect size

– Using causal language to describe observational studies

– Failing to explain limitations of surrogate markers/endpoints

– Single source stories with no independent perspective

– Failing to independently analyze quality of evidence

Exaggerating effect size –Absolute vs. Relative Risk

• Two ways of saying the same thing

• One way – relative risk reduction – makes effect size seem larger

• Other way – absolute risk reduction – makes effect size seem smaller.

• We absolutely think you should use the absolute figures.

??

Relative vs. absolute risk reductions

30330%030%%% 10%

Chance of death at 1 yearRisk reduction

67%

67%

67%

??20%

Placebo DRUG Relative Absolute

2%

0.002%

30% 10% 67% 20%

3% 1% 67% 2%

0.003% 0.001% 67% 0.002%

Nolvadex (tamoxifen)Reducing breast cancer risk by 50 .

You must ask: 50% of what?

• The “of what?” is the absolute risk.

• When you’re only told the relative risk – 50% -it could be risk reduction from 90 in 100 down to 45 in 100…so 45 benefit.

• Or it could be from 2 in 100 to 1 in 100….so only 1 benefits.

• You want to know the absolute size of the effect, not just one relative to another.

What if we used only relative terms on harms?

• In that same Nolvadex/tamoxifen study example, the tamoxifen users had 210% more uterine cancers and potentially life-threatening blood clots in the lungs and legs.

Over 6 years, what happened …

Benefits: Nolvadex lowered chance

Getting breast cancer

Harms: Nolvadex increased chance

Having a serious blood clot

3.3% 1.7%

0.5% 1.0%

Net effect of Nolvadex for every 1000 women:16 fewer women get breast cancer

5 more women get serious blood clots6 more get uterine cancer

Getting uterine cancer 0.5% 1.1%

NOLVADEXPLACEBO

USING CAUSAL LANGUAGE TO DESCRIBE OBSERVATIONAL STUDY RESULTS…FAILING TO DESCRIBE LIMITATIONS OF SUCH DATA

Association ≠ Causation

Limits of observational studies

• HealthDay News: “Going up a skirt size every 10 years raised chances of developing breast cancer by 33%.”

Reported by many news organizations.

What is the impact of this on a woman who goes from size 8 at age 40 to size 10 at age 50?

Sometimes even journals get it wrong

BMJ news release in June, 2014

Many news stories used the “raises breast cancer risk” language.

But The Guardian in the UK used just 6 little words:

“association does not necessarily imply causation.”

I wrote about a half dozen such BMJ news releases in recent years, until finally, they started using boilerplate language:

“This is an observational study so no definitive conclusions can be drawn about cause and effect.”

FAILING TO EXPLAIN LIMITATIONS OF SURROGATE MARKERS/ENDPOINTS

An example of journalism excellence

Many cancer drugs approved on the basis of progression-free survival -which means how long patients survived before doctors detected a tumor worsening. It does not necessarily mean an improvement in overall survival. And many of these drugs have failed to show improvements in overall survival after they were approved by the FDA.

RELYING ON A SINGLE SOURCE WITH NO INDEPENDENT PERSPECTIVES

(SOMETIMES RELYING SOLELY OR LARGELY ON NEWS RELEASES OR PUBLIC RELATIONS ANNOUNCEMENTS)

• Was urged by co-workers to have 1st mammogram on the air. • Didn’t disclose details of results, but announced she’d get bilateral

mastectomies. • She wrote:

“I can only hope my story will… inspire every woman who hears it to get a mammogram, to take a self-exam. No excuses. It is the difference between life and death.”

• No excuses? Mammography should be informed choice, not mandate. • Women can have rational reasons for declining mammograms. • Mammogram will NOT be the difference between life and death for

most women.

The Tyranny

Of The

Anecdote

Just a few out of 1000s of stories

• Boston Globe: Futuristic sports bra could detect early signs of breast cancer

• ABC: Is a new heat sensing bra the breast medicine?

• CBS: Bra aims to detect breast cancer before mammogram.

• WCBS, NY: Cancer-detecting bra may save women’s lives

• CBC: High tech ‘smart bra’ could replace mammograms

• CNN: A bra that could detect cancer

Take notes, then we’ll discuss

We blogged & raised ?s

• When news was released, NBCC staffer couldn’t find clinical trial data on company website.

• Details about what was actually detected were lacking

• Existing thermography unreliable for differentiating cancer from normal tissue

• Ted Gansler, MD, editor of CA: A Cancer Journal for Clinicians said:

“At this time, based on evidence currently available, I think the vast majority of doctors will agree that a woman who chooses any breast cancer screening test based on temperature measurements, instead of mammography, would be making a serious mistake that could have fatal consequences.”

Dr. Russell Harris, Univ. of North Carolina Cancer Center:

• The problems lie not with their numbers, but with what they left out.• They are talking about a one-time screen. But for multiple screens over 10

years, say, the number of women with at least one positive mammogram is more like 500 out of the 1,000.

• And the biopsy rate (at least one biopsy) is also higher.• And they don’t discuss the anxiety caused by having a false positive or from

having to return for a follow-up surveillance mammogram. • And they don’t say how many of the 5 women with breast cancer diagnosed

not by screening but by symptoms would have also had a “high cure rate” (only 1 additional 40 year old women in 1,000 who are screened for 10 years have their lives extended by screening – 2 in 1,000 for 50 year olds, 3 in 1,000 for 60 year olds – and these are optimistic figures – it may be less than this).

• And they don’t say how many of those 5 women diagnosed with breast cancer are overdiagnosed – i.e., would never have been diagnosed at all if they had not been screened.

• But they do emphasize that “Mammography Saves Lives: One of them may be yours.”

• Breakthrough?• Exclusive?• Developing Story?• Cancer Cure within

reach?

It was all based on a medical center’s public relations announcement…not any new research or any new finding.

What about all the other research at all the other cancer centers aiming to do the same thing?

Remember 2 of our 10 story criteria…

• Did the story go beyond a PR effort or news release? NO

• Did the story disclose conflicts of interests in its sources?

NO. CNN didn’t report on….

• Resignations, questions about conflicts of interest and fraud probes of the program in question.

• Not likely to be the kinds of angles and issues one pursues when there are “exclusive” reporting arrangements between a news organization and a medical center.

How did this…..

….become this?

And all of this…. 327,000 results on a Google search!

A university news release played a role

FAILURE TO INDEPENDENTLY ANALYZE THE QUALITY OF THE EVIDENCE…..OR, STENOGRAPHY

“Good journalism has a subtle feature of reticence. We don’t publish everything we hear. We filter. We curate. The goal of the traditional journalist is to create a reputation for accuracy, fairness, relevance and timeliness, and this requires the willingness to not publish things that are unlikely to be true. … There’s nothing at stake here except the survival of credible journalism.”

- Joel Achenbach

Washington Post

Look for the extra dimension in health care news

June 25: Faithful to the latest journal article, many news organizations dutifully reported what they were told by authors of a study published in the Journal of the American Medical Association, “Breast Cancer Screening Using Tomosynthesis in Combination with Digital Mammography.“

Many stories used sensational language – “breakthrough, game-changer, best way of detection, any woman should have this, lifesaver.”

Some stories delivered more…

• USA Today: “3D mammograms find more cancers, but do they save lives?”– “Other breast cancer advocates caution that doctors

still don’t know whether finding these extra cancers will actually save lives, however, or simply lead more women to treatment.” Later, this:“The absolute differences between (3-D and standard mammograms), in terms of the number of cancers found, is very small, says Fran Visco, president of the National Breast Cancer Coalition. And other benefits are uncertain.

The study didn’t follow women over time. So doctors don’t know whether undergoing tomosynthesis actually saves lives, Visco says.”

Some stories delivered more…

• NY Times: “The verdict is still out on the long-term worth of this new technology. The new results are promising but not definitive…Even so, more and more mammography centers are buying the equipment.”

• Boston Globe: “Finding more cancers isn’t our goal,” said Dr. Lisa Schwartz, professor of medicine at the Dartmouth Institute for Health Policy and Clinical Practice. “It’s about reducing breast cancer deaths. We don’t know whether this new technology finds more cancers destined to kill people.”

Some stories delivered more…

• MedPage Today: “If we step back and ask whether these papers show that better breast imaging will actually translate into lives saved, it’s unclear. The differences are small enough that you want to analyze very carefully the trade-offs in terms of extra biopsies, extra imaging effort, and the extra cost involved, before thinking that it should be a routine piece of our public health screening for breast cancer,” said Harold Burstein, MD, PhD, of Dana-Farber Cancer Institute in Boston.

That’s quite a bit different than “breakthrough...game-changer…best way of detection…any woman should have this…lifesaver” of ABC, NBC, Fox, Chicago Sun-Times, Washington Post and others.

This gives you an idea of what a difference independent perspectives, asking tough questions, and independently vetting claims –not just practicing journal stenography – can make for ensuring accuracy, balance and completeness in health care news.

Pharma says that for every 5,000 compounds in pre-clinical testing, only 5 make it to human trials. And of those, only 1 will get FDA approval.

But we keep seeing stories like:

– The Toronto Star: “New breast cancer drug heralded as breakthrough” (Phase 2 study)

– CNN: “Stem cell medical breakthrough?” (Phase 1 study)

– Medscape: ‘Truly Remarkable’ Response with Combination for Melanoma (Phase I study – 37 people followed 12 weeks)

Another estimate: only 8% of new drugs in Phase I studies ever get approved

“BREAKTHROUGH” on all 3 main TV networks on a Phase I study

Note that all 3 practice “question mark journalism” – which allows you to say anything, followed by ?

Cover story “breakthrough”

16 words: “moving into the testing phase”

Stenography, not journalism

Reported on Phase 2 breast cancer drug study – directly from a Pfizer news release –which didn’t include any data.

But the story lifted a glowing quote from the company news release:

“The study suggests ‘the potential to transform the standard of care,’ said Mace Rothenberg, Pfizer Oncology's chief medical officer. ‘This is encouraging information for these women.’ “

No data. No independent perspective. Free publicity on results no one will see for two more months when they are presented at a meeting.

But Reuters and AP also reported on the Pfizer news release. Those stories were picked up by Huffington Post, ABC, Charlotte Observer, Boston Herald, Idaho Statesman, Washington Times, Bradenton Herald, Fort Worth Star Telegram, Myrtle Beach Sun News, Philly.com, others

2013: The year we “cured” cancer – three times !

“Breaking Through Cancer’s Shield” - Oct 2013

JOSH BILLINGS (PEN NAME OF HUMORIST HENRY WHEELER SHAW, 1818 – 1885)

“The trouble with people

is not that they don't know, but

that they know so much that ain't so.

…

I honestly believe it is better to know

nothing that to know what ain’t so.”

Journal of Incidentalomas

Journal of Experimental EfficacyJournal of Preclinical Self-Importance

Journal of Surrogate Markers Only

September 2012

Positive “spin” was identified in about half of press releases and news stories. The main factor associated with “spin” in press releases was the presence of “spin” in the journal article abstract conclusion.

In other words, a direct link from published study news release news story.

Who thinks about the reader, the consumer, the patient…at the end of this food chain?

Mayo’s Dr. Victor Montori to a journal club:

“Beware spin: composite endpoints, surrogate markers, subgroup analyses, inadequate comparators (too much or too little of an effective alternative or placebo when an effective alternative exists). Above all, avoid the intro/discussion sections which is where most of the interpretational spin is introduced.”

But many journalists – who feed off a steady diet of journal studies in order to meet their story quotas and click rates – have no idea what any of these terms mean.

Dr. Richard Lehman has published reviews of journal articles for 8 years on BMJ website

“I too was once a conclusion-of-the-abstract reader, and was quite smug that I had even got that far. It took me some years to become aware of perhaps the most important principle of critical reading: never believe the stated bottom line without confirming it from the data. And beware of the limitations of the data.”

Dr. Lehman reminds us:

350 years ago the Royal Society of London for Improving Natural Knowledge used as its motto,

Nullius in verba

Rough translation:

“Don’t take anyone’s word for it.”

We still have much to learn from those notable 17th

century skeptics.

“Bad science is no excuse for bad journalism.”

Unprecedented peaks of excellence in health care journalism

Daily Drumbeat of Dreck

But the valleys between those peaks are becoming wider and deeper – what we call…..

Journalists could help people understand and deal with the clash between:

• Science

• Evidence

• Data

• Recommendations for entire population

• What we can prove

• Grasping uncertainty and helping people apply critical thinking to decision-making issues

Intuition

Emotion

Anecdote

Decision-making by an individual

What we believe, wish, or hope

Promoting false certainty and non-evidence-based, cheerleading advocacy

Gary@HealthNewsReview.org

Thank you