View
466
Download
0
Category
Preview:
Citation preview
Sharing and growing the world's knowledge with machine learning
Lei Yang (leiyang@quora.com)
April 2016
Our mission
“To share and grow the world’s
knowledge”
● Millions of questions & answers
● Millions of users
● Thousands of topics
● ...
Demand
What we care about
Quality
Relevance
Data@Quora
Topic Question
User
Answer
Actions
Lots of data relations
Complex network propagation effects
Importance of topics & semantics
Machine Learning@Quora
Ranking - Answer ranking
What is a good Quora answer?
● Truthful
● Reusable
● Provides explanation
● well formatted
...
Ranking - Answer ranking
How are those criteria translated
into features?
● Features that relate to the text quality
itself
● Interaction features (upvotes/downvotes,
clicks, comments…)
● User features (e.g. expertise in topic)
Ranking - Feed
Present most interesting stories for a user at
a given time
● Interesting = topical relevance +
social relevance + timeliness
● Stories = questions + answers
● Personalized learning-to-rank approach
● Relevance-ordered vs time-ordered = big
gains in engagement
● Challenges
○ Potentially many candidate stories
○ Real-time ranking
○ Objective function
Ranking - Feed
● Personalized LTR model
● Features
○ Quality of question/answer
○ Topics the user is interested in
or knows about
○ Users the user is following
○ What is trending/popular
○ ...
● Different temporal windows
● Multi-stage solution with different
“streams”
Recommendations - Topics
Recommend new topics for the user
to follow, based on
● Topics you already follow
● Users you already follow
● Interactions with questions/answers
● Topic-related features
● ...
Recommendations - Users
Recommend new users for the user
to follow, based on:
● Users you already follow
● Topics you already follow
● Interactions with users
● User-related features
● ...
Related questions
Given interest in a question, what other questions
are interesting?
● Not only about similarity, but also “interestingness”
● Features such as:
○ Textual
○ Co-visit
○ Topics
○ …
● Important for logged-out use case
Duplicate questions
● Important issue for Quora
○ Want to make sure we don’t disperse
knowledge to the same question
● Binary classifier trained with labelled data
● Features
○ Textual vector space models
○ Usage-based features
○ ...
User expertise inference
Infer user’s trustworthiness in relation
to a given topic
● We take into account:
○ Answers written on topic
○ Upvotes/downvotes received
○ Endorsements
○ ...
● Trust/expertise propagates through the network
● Useful as input/features in other models
Spam detection and moderation
● Very important for Quora to keep quality of
content
● Pure manual approaches do not scale
● Hard to get algorithms 100% right
● ML algorithms detect content/user issues
○ Output of the algorithms feed manually
curated moderation queues
Content creation prediction
● Quora’s algorithms not only optimize for
probability of reading
● Important to predict probability of a user
answering a question
● Some product features completely rely
on that prediction
○ E.g. A2A (ask to answer) suggestions
Trending topics
Highlight current events that are interesting
to the user
● We take into account:
○ Global “Trendiness”
○ Social “Trendiness”
○ User’s interest
○ ...
● Trending topics are a great discovery mechanism
Models &Experimentation
Models
● Logistic Regression
● Elastic Nets
● Gradient Boosted Decision Trees
● Random Forests
● (Deep) Neural Networks
● LambdaMART
● Matrix Factorization
● LDA
● ...
Open source project -- QMF
Quora Matrix Factorization
https://github.com/quora/qmf
● Currently BPR and WALS
● Multithreaded implementation
in C++14
ML platform
● Allow ML Engineers and Data
Scientists to collaborate within
the same ML framework
● Easy integration with well known
tools and open source libraries
● Offline evaluation and debugging
● User friendly Python frontend
● High performance and scalable
C++/CUDA backend
Redshift MySQL
S3 PythonUser Interface
Trainer Box
Session
CPU GPU
Disk
...WALS BPR
● Extensive A/B testing, data-driven
decision-making
● Separate, orthogonal “layers” for
different parts of the system
● Experiment framework showing
comparisons for various metrics
Experimentation
Conclusions
Conclusions
● At Quora we have not only Big, but also “rich” data
● Our algorithms need to understand and optimize complex aspects such
as quality, interestingness, relevance, or user expertise
● We believe ML will be one of the keys to our success
● We have many interesting problems, and many unsolved challenges
We are hiring! www.quora.com/careers
Recommended