Team ProjectTwitter Trend Analyzer

Group 20Ikwhan Chang / Quan Trung Nguyen

Prof. Hungwen Li

11-May, 2017

1

Final Presentation

INDEX• Overview

• Twitter Developer API

• Search API

• Streaming API

• Web Socket

• Chart & Map

• Heroku for the PaaS

• Architecture

• Data Flows

• User Interface & Demo

2

Overview• Project Title

• The trend analysis and predict engine for a specific keyword of Twitter by using cloud platform

• Description

• Twitter is one of popular social networking tools to share our think or news, and it will rapidly spread across the worlds.

• We will make a web application to collect the Twitter's feeds related to the specific word, and then provide the user a daily trend by using a realtime linear chart and geographical chart

• Team Members

3

Name Ikwhan Chang Role Frontend / API / Cloud / UX

Name Quan Nguyen Trung Role Backend / Data Handling

Twitter Developer API• Register the application

• To use the Twitter’s API, we need to register our application at dev.twitter.com

• OAuth

• Twitter data are collected by using Twitter APIs, which require OAuth to identify Twitter applications and users.

• Allow users to access to a protected database. (e.g. twits, mentions..)

• The user authentication will use a signed request to identify an application‘s identity along with the user’s access token. There are four secret keys

4

dev.twitter.com

Consumer Key/Secret

Access Token/Secret

Twitter Search API• It generates queries to collect data of recent and popular Tweets with certain criteria

• We can also use a set of parameters to have a better control of the search results.

• Result Type: indicates that the search results will be recent or/and popular Tweets

• Geolocalization: restricts the location of Tweets

• Language: restricts the languages of Tweets

• Iterating in a result set: controls size of search results

• When the user enters the keyword, the application will create an URL-encoded search query and will be sent to the server via the HTTP request.

• After receiving the HTTP request, the server will pass keyword value to parameter q {q: keyword} to perform the search.

• When the server receives the search result from Twitter, it will send the result back to client via HTTP response in form of JSON files.

5

Search API - backend

Search - frontend

Twitter Streaming API

• We used Streaming APIs to obtain high volume, real-time data from Twitter for our charts

• Twitter provides several basic streaming types:

• Public streams: the streaming data comes from the public data flowing through Twitter

• User streams: the single-user streaming data comes from a single user’s view of Twitter

• Site streams: this is the multi-user version of user streams

• The streaming connection process will get the collected data, perform the tasks parsing, filtering and store the result into a data store.

• Then the HTTP server will get the processed result to send back to the user by using Web Socket

6

Streaming API overview

Web Socket

• WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection.

• WebSockets are based on http, but are completely different protocols than the http protocol.

• Since WebSockets do not send header information except for the initial connection, they are much more cost effective in terms of network cost and can be implemented quickly and without waiting because of full-duplex communication.

• Our application uses WebSockets to send the streamed Twits into web browser by using socket.io

7

Web Socket Flows

socket.io

Chart and Map• HighCharts (www.highchart.com)

• For the real-time chart, we used the javascript library of HighCharts’s linear chart.

• In order to draw the graph like above, you need to use jquery called Highcharts

• Google Map API (https://developers.google.com/maps

• We used the Google Map API in order to the geographical chart and the geocoding.

• The geocoding means we can convert from actual user’s address to coordinates like latitude and longitude.

• Since twitter API provide us just user’s location character, we need to convert from it to the actual geo coordinates in order to show into our geographic chart.

8

Heroku for PaaS

• Heroku is a new way of deploying web applications, where developers are providing services based on the idea that they can concentrate on the code of the web application without losing time for tasks such as server configuration, deployment environment, and server expansion.

• Developers are only deployed and restarted by writing source code, committing the code to the git repository for deployment, and pushing it to the remote Heroku repository.

• If the success of your web application requires more servers, increasing the number of processors called Dyno will have the same effect as adding multiple servers.

9

Type of Cloud Computing

Heroku Deployment Architecture

Architecture10

Back-end Front-end

Integrated Framework

UI/CSS Framework

Dependencies Management

Javascript Framework

Testing

Task Management

Test Runner

Pre-processor

Compressor

UglifyJS, UglifyCSS

Code Qualifier

Documentation

JSDoc3

Lazy Loader

Integrated Framework

Cloud

Streaming

Twitter Stream API

Data Flows11

Frontend Backend OAuth

Search Search

/search?q=iphone&start_date=20160227&end_Date=20170327

Trend Chart

Streaming

Geographic Chart

{twits:[{ ‘no’:’1’, ‘twit’:’#iphone is gorgeous!!’ ‘date’:’2017-01-18’, ‘location’:’San Jose, CA’},….. ]}

{[{‘date’:’2017-03-26’,’expected’:45’},{‘date’:’2017-03-27’,’expected’:45’},

….]

Date Range

http://localhost:8080

http://localhost:8080

Stream API{keyword:’iphone’}

Key Exchange (Secret key, token)

User Interface & DemoSearch Box

Realtime Linear Chart

Geographical Chart

Actual Twit Data

# of streamed Tweets

http://twitter-trend-analyzer2.herokuapp.com

Thank You13

https://github.com/IkwhanChang/twitter-trend-analyzer