VP  Developer  Evangelism    Samisa  Abeysinghe  

Big  Brother  for  Enterprises:    Social  Media  Stream  Use  Cases  

26  Feb  2014  

2  

About  the  Presenter  

๏  Samisa  Abeysinghe  VP  Developer  Evangelism  samisa@wso2.com    

๏  Samisa  Abeysinghe,  Vice  President  of  Developer  Evangelism  joined  the  company  in  September  2005.    Prior  to  the  current  role,  Samisa  used  to  be  VP  of  Engineering  and  managed  the  development  of  WSO2  Carbon  based  product  plaOorm.    

3  

About  WSO2  ๏  Global  enterprise,  founded  in  2005  

by  acknowledged  leaders  in  XML,  web  services    technologies,  standards    and  open  source  

๏  Provides  only  open  source  plaOorm-­‐as-­‐a-­‐service  for  private,  public  and  hybrid  cloud  deployments  

๏  All  WSO2  products  are  100%  open  source  and  released  under  the  Apache  License  Version  2.0.  

๏  Is  an  AcXve  Member  of  OASIS,  Cloud  Security  Alliance,  OSGi  Alliance,  AMQP  Working  Group,  OpenID  FoundaXon  and  W3C.  

๏  Driven  by  InnovaXon  

๏  Launched  first  open  source  API  Management  soluXon  in  2012  

๏  Launched  App  Factory  in  2Q  2013  

๏  Launched  Enterprise  Store  and  first  open  source  Mobile  soluXon  in  4Q  2013  

4  

What  WSO2  Deliver  

NSA like Monitoring for Your Enterprise ๏ Analyze volumes of data

๏ Address correlation complexities in analytics

๏ Off line vs Real time operations implications

๏ Some operations got to be in real time, else the value is lost

๏ Summarized data over time (and other) dimensions for analytics

A “Big Brother” that keeps an eye on the whole enterprise

5  

Why should I bother? ๏ Deal with high volume (terra bites) of information

๏ In order to make decisions

๏ Real time & Offline

๏ Take action

6  

WSO2 :ONLY COMPLETE & INTEGRATED Platform ๏ Complete and integrated for

๏ Data capture

๏ Analysis: both real-time and batch

๏ Visualization

๏ Action taking business process execution

Making data driven intelligence for your enterprise easy

7  

Big Data: Strategy

8  

Responsive and Effective

9  

WSO2  Big  Data  Analy0cs  Pla4orm  for  Your  Enterprise  

WSO2 Big Data Analytics Platform for Your Enterprise

10  

Key  Elements  

Data  CollecXon  

Data  Analysis  

Data  VisualizaXon  

Taking  AcXon  

11  

WSO2 Big Data Analytics Platform for Your Enterprise

12  

Data Collection Data Analysis Data Visualization & Taking Action

Use  Case  Scenario:  Analyzing  Twiaer  Streams  

13  

Social  media  …  !  

o  Test  

o  Test  

   

Processing  Social  Media  !!!  

   

Why  Monitor  Social  Media  ?  

●  To  get  useful  insight  about  the  market    

○  CompeXtor  Analysis  

○  Customer  Trend  and  Interest  

●  IdenXfy  and  Compare  Trends    

○  Monitoring  the  brand  

○  Comparing  the  trends  of  the  public  interest  

●  Alerts  when  interesXng  thing  happens    

○  Hot  news  about  the  compaXor    

○  To  idenXfy  opportuniXes  

   

16  

WSO2  Business  AcXvity  Monitor  (BAM)  

●  Can  capture  data  from  any  applicaXon/service    ●  APIs,  Services  or  source  including  file-­‐based  data  

○  WSO2  has  lightweight  agents  to  provide    ■  raw  data  from  numerous  sources    ■  as  well  as  support  for    

●  gateways  (lightweight  or  heavyweight)    ●  Service  and  API  proxies  ●  Data  imports  from  various  sources  (including  file  based  data)  ●  Any  other  source  (extended  agents)  

17  

WSO2  Business  AcXvity  Monitor  (BAM)  

18  

WSO2  Complex  Event  Processor  (CEP)  ●  CEP  process  mulXple  event  streams  generated  within  the  

enterprise  to  construct  data  abstracXon  and  idenXfy  meaningful  paaerns  among  those  streams  ○  Powered  by  WSO2  Siddhi  

■  Processes  more  than  2.5M  events/sec  on  single  server  (commodity  hardware)  ■  Has  a  powerful  and  extensible  SQL  Like  Siddhi  query  language    

○  Integrates  with  any  enterprise  system  for  event  capture  ■  Support  mulXple  transports  &  event  formats:  HTTP,  JMS  &  SMTP  on  XML,  JSON,  Text  &  Map  

○  Efficient  large  volume  event  publishing  to  enterprise  systems  and  custom  noXficaXons    ■  XML,  JSON,  Text  over  HTTP,  JMS  &  SMTP      ■  SMS  Support    ■  MySQL  DB  and  Cassandra  Support  

19  

WSO2  Complex  Event  Processor  (CEP)  

20  

E.g.  Usecases  

●  Finding  out  the  trend  of  #Cloud  Twiaer  tags  over  Xme    

●  IdenXfy  the  public  interest  on  cloud  market  leaders  based  on  tweets  on  #Cloud  Twiaer  tags      

●  Comparing  interest  trends  on  cloud  market    

●  NoXfying  via  Email  when  an  interesXng  tweet  was  posted.  

21  

SoluXon  Architecture  

22  

Twiaer  Agent                //Configuring  Agent                  AgentConfiguraXon  agentConfiguraXon  =  new  AgentConfiguraXon();                  agentConfiguraXon.setTrustStorePassword("/Users/suho/wso2/dev/twiaer-­‐sample/wso2cep-­‐3.0.0/repository/"  +                                  "resources/security/client-­‐truststore.jks");                  agentConfiguraXon.setTrustStore("wso2carbon”);                  Agent  agent  =  new  Agent(agentConfiguraXon);                    //New  Data  publisher  to  BAM                  final  AsyncDataPublisher  asyncDataPublisher  =  new  AsyncDataPublisher("tcp://localhost:7611",                                  "admin",  "admin",  agent);                    //Defining  the  Stream                  String  STREAM_NAME  =  "twiaer_data_stream";                  String  VERSION  =  "1.0.0";                  String  streamDefiniXon  =  "{"  +                                  "  'name':'"  +  STREAM_NAME  +  "',"  +                                  "  'version':'"  +  VERSION  +  "',"  +                                  "  'nickName':  'Twiaer_Data_Stream',"  +                                  "  'descripXon':  'Stream  of  twiaer  data',"  +                                  "  'payloadData':["  +                                  "  {'name':'createdAt','type':'LONG'},"  +                                  "  {'name':'user','type':'STRING'},"  +                                  "  {'name':'tweet','type':'STRING'}"  +                                  "  ]"  +                                  "}";                  asyncDataPublisher.addStreamDefiniXon(streamDefiniXon,  STREAM_NAME,  VERSION);    

23  

Visualizing  #Cloud  Twiaer  Social  Stream  

24  

25  

Finding  out  the  trend  of  #Cloud  Twiaer  tags  over  Xme  ๏  insert  overwrite  table  Twi@erDataStreamStatsPerMinute  select  count(1)  as  

total_tweet_count,  year(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  ))  as  year,  month(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  ))  as  month,day(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  ))  as  day,hour(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  ))  as  hour,  minute(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  ))  as  minute,concat(substring(from_unixXme(cast(created_at/1000  as  BIGINT),  'yyyy-­‐MM-­‐dd  HH:mm:ss'),0,16),':00')  as  Xme  from  TwiaerDataStream  group  by  year(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  )),  month(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  )),day(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  )),hour(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  )),minute(from_unixXme(cast(created_at/1000  as  BIGINT),'yyyy-­‐MM-­‐dd  HH:mm:ss.SSS'  )),substring(from_unixXme(cast(created_at/1000  as  BIGINT),  'yyyy-­‐MM-­‐dd  HH:mm:ss'),0,16);  

๏     26  

27  

IdenXfy  the  public  interest  on  cloud  market  leaders    insert  overwrite  table  CountOnCompanyTwi@erDataStream    select  'IBM'  as  company,  count(1)  as  cnt  from  TwiaerDataStream  where  instr(lower(tweet),  'ibm')  >0  ;    insert  overwrite  table  CountOnCompanyTwi@erDataStream    select  'Google'  as  company,  count(1)  as  cnt  from  TwiaerDataStream  where  instr(lower(tweet),  'google')  >0  ;    insert  overwrite  table  CountOnCompanyTwi@erDataStream    select  'VMware'  as  company,  count(1)  as  cnt  from  TwiaerDataStream  where  instr(lower(tweet),  'vmware')  >0  ;    insert  overwrite  table  CountOnCompanyTwi@erDataStream    select  'Azure'  as  company,  count(1)  as  cnt  from  TwiaerDataStream  where  instr(lower(tweet),  'azure')  >0  ;      

28  

29  

Comparing  interest  trends  on  cloud  market  

30  

31  

NoXfying  via  Email  when  an  interesXng  tweet  was  posted!  

&  Take  AcXon  

32  

CEP:  Retweet  NoXficaXon    from  twiaerStream[tweet  contains  'RT  ']    

insert  into  reTweetStream;  

     

33  

Demo  …..  http://wso2.com/library/demonstrations/2014/02/screencast-bigbrother-for-the-enterprise-social-stream-analysis/

34  

Log  Analysis  Demos  …..  http://wso2.com/library/demonstrations/2014/02/screencast-monitoring-system-logs-with-wso2-business-activity-monitor/

35  

http://wso2.com/library/demonstrations/2014/02/screencast-http-log-monitoring-with-wso2-business-activity-monitor/

36  

Business  Model  

37  

Call  to  acXon  page  ๏  Be  your  own  NSA:  hap://wso2.com/landing/nsa/    

๏  Big  Data  Webinar  Series:  hap://wso2.com/landing/wso2-­‐bigbrother-­‐webinar-­‐series/    

๏  WSO2  Business  AcXvity  Monitor:  hap://wso2.com/products/business-­‐acXvity-­‐monitor/    

๏  WSO2  BAM  Docs:  hap://docs.wso2.org/display/BAM240/WSO2+Business+AcXvity+Monitor+DocumentaXon    

๏  WSO2  Complex  Event  Processor:  hap://wso2.com/products/complex-­‐event-­‐processor/    

๏  WSO2  CEP  Docs:  hap://docs.wso2.org/display/CEP300/WSO2+Complex+Event+Processor+DocumentaXon    

Contact  us  !