Upload
keith-rose
View
13
Download
0
Embed Size (px)
Citation preview
DQM Update
17 Feb 2017Twitchcrement DemonstrationTwitchcrement
Making Spam fun for everyoneKeith Rose
17 Feb 2017
Twitchcrement Demo
Data Engineering
1
Twitch.tv17 Feb 2017Twitchcrement Demonstration
Online streaming service focused on video games
81st in worldwide web traffic
4th peak North American traffic
2Driven by subscriptions and advertisements
Strength: User InteractionConsequence: SPAM!
2
Characterize spam/spammers17 Feb 2017Twitchcrement Demonstration3Does spam show up in multiple channels? (Related networks)
Is it the same user spamming over and over again? (Probably bot)
Are otherwise inactive users spamming? (Might be more responsive to user appeals)
3
Data Flow17 Feb 2017Twitchcrement Demonstration
Scrape top 500 channels
Chatmessage (user, channel, message)
Count chat messages in 10s span, aggregate with spam messages, count unique usersSpammessage (channel, message, spam count, user list)
stream to consumerschatmessagespammessage4
4
Design challenges17 Feb 2017Twitchcrement DemonstrationDesign an algorithm that works as a dynamic trigger
Dont want to store list of spam terms in memory
Would like to avoid continual database reads even to a lightweight system (i.e. redis)
Solution: Flink and stateful streaming
Flink maintains state of its messages and updates as new messages come in on a topic
5
5
Design challenges17 Feb 2017Twitchcrement DemonstrationSolution: Propogate spam messages as parallel stream to chat messages, loop around and perform union6
Spam messages keyed similarly to chat messages
Comparisons only occur when and where relevant
Message counts maintain state (SPAM/NOT SPAM)Incoming spam topics maintain state (NEW/ACTIVE)
6
Demonstration link17 Feb 2017Twitchcrement Demonstration
http://twitchcrement.us:8000/
Django application takes as argument a channel, streams spam messages and unique participants live7
7
About Me17 Feb 2017Twitchcrement Demonstration
Keith Rose
Ph.D. Experimental Physics (2012)
Built data pipeline for part of a major accelerator experiment
Extensive C++/python experience
Longtime gamer for 30+ years
Swiss National Dance Dance Revolution champion
8
8