Making the most of social historic data

Aleksander KolczTwitter, Inc.

Usual “Big Data” questions

• How to scale storage?• How to scale processing?• What type of processing is important?• What problems represent useful benchmarks?

• What questions are asked less often?

The “first” web revolution

• Large quantities of unstructured data (html)• Open access (mostly)• Graph structure of recovered while crawling• A few down-sides– A lot of data is in fact noise– Running web-scale crawlers affordable only to few

The new “personalized” web

• Lots and lots of log data– Who searches for what– Who clicks on what– Who comments (blogs, tweets, posts) on what– When do the above happen

• Lot’s of companies mining their users’ data

Problems

• Only some of the data is open• Sharing is limited• Nobody gets the full picture

How to remove the silos?

• Data is often ‘gold’• Expectations of privacy– The most interesting data is often personal– Users have expectations that their data will not be

widely shared– Can aggregation and anonymization insure privacy

while retaining enough useful information?

Enron email corpus

• The Enron email trial gave an unexpected boost to the research community

• The corpus of emails used in court evidence became public domain

• It still remains one of the key public email datasets

• The research community probably cannot count on too many such lucky accidents

Query log data

• Attempts to release anonymized search query logs ended badly

• An adversary who knows the target may be able to identify the target by context even if the data is anonymized

Revocations

• Even for data that is effectively public there may be cases where users may want to “take it back”– Even more so for no-so-public ones– Email unsend– Deleting of user accounts– Removal of web-pages, blog comments, etc

• Once the data gets out of the original content management system it is not clear how such requests can be handled

Suppose we have the data …

Live A/B tests

• Users provide feedback based what they have a chance to interact with

• Alternative algorithms of content representation/recommendation can be evaluate via A/B testing– Split the user population into disjoint sets– Present each one of algorithm-specific version of X

• Usually this works quite well in determining which approach is more preferable

Historic A/B tests

• The log data capture the response of users (clicks, engagement) to the information they were presented with

• One typical objective of mining large datasets is to determine what would work better

• Can we reliably assess how a new algorithm would perform based on the historical information alone?

Adversarial data

• Many datasets contain patterns of abuse– Email spam– IM spam– Web search spam– Twitter spam

• How useful are they in preventing future abuse

Evidence of hedging the bets

• Abusers have an imperfect knowledge of the system they attack

• Abusers may be attacking several different systems and combining the techniques used against each one

• ML based techniques trained with large quantities of historical data capture some of the diversification in the methods of attack and will be effective to an extent

But the are no guarantees

• Any new defense measure elicits a direct response and this is seen in the historical data

• It is hard to anticipate what the response would be if the measure is truly new

• What is the limit to the amount of information we can learn from “old” data?

Conclusions

• Efforts to create large datasets of “real world” data need to address the privacy concerns

• We need a better understanding how past data can be used to tune algorithms of the future

• This is particularly so in the ever-important anti-adversarial domain