Picking the NYT Picks:Editorial Criteria and Automation

in the Curation of Online News Comments

Nicholas DiakopoulosUniversity of Maryland, College Park – College of Journalism

@ndiakopoulos | nickdiakopoulos.com | nad@umd.edu

“NYT Picks is the most popular comment queue. We spend a lot of time tweaking that and

getting that right.”

What are criteria for selection?How can we augment moderator capability to consider more

comments?

Criteria from Literature

Negative / ExclusionPersonal attacks, profanity, abusive behavior

Positive / Inclusion Internal CoherenceThoughtfulnessBrevity / Length

RelevanceFairness / Diversity

Novelty

Argument QualityCriticalityEmotionalityEntertainment ValueReadabilityPersonal Experience

Crowdsourcing

Argument QualityCriticalityEmotionalityEntertainment ValueReadabilityPersonal Experience

Internal CoherenceThoughtfulnessBrevity / Length

RelevanceFairness / Diversity

Novelty

RQ1: Do “NYT Picks” comments reflect positive editorial criteria identified in literature?

Automation

Argument QualityCriticalityEmotionalityEntertainment ValueReadabilityPersonal Experience

Internal CoherenceThoughtfulness

Brevity / LengthRelevance

Fairness / DiversityNovelty

RQ2: Can algorithmic approaches to assessing criteria be developed?

Automated scores point towards scalable opportunities for moderation and UX…

But automation also raises questions about over-generalization across contexts, and algorithmic transparency

Questions?

ContactNick DiakopoulosUniversity of Maryland, College of Journalism

Twitter: @ndiakopoulosEmail: nad@umd.eduWeb: http://www.nickdiakopoulos.com

More InfoN. Diakopoulos. The Editor’s Eye: Curation and Comment Relevance on the New York Times. Proc. CSCW. March, 2015.