60-440 1

03-60-440 Mashups

60-440 2

Mashups

• From Wikipedia– Mashup (music), a musical genre of songs that consist entirely of

parts of other songs – Mashup (video), a video that is edited from more than one source

to appear as one – Mashup (web application hybrid), a web application that

combines data and/or functionality from more than one source

• Combine data from multiple sources. – Mostly the data sources lay outside of the organizational

boundaries• Create a new and distinct web service that was not

originally provided by either source.• Content is typically sourced via a web API or a RSS Feed

Mashup

60-440 3

Mashup example• The ChicagoCrime.org Web site

– A mapping mashup– One of the first popular mashups– Mashes crime data from the Chicago Police Department’s online

database with cartography from Google Maps– The concept and the presentation are simple and the composition of

crime and map data is visually powerful

The following slides are borrowed from Umut Orhan

Mashup

60-440 4

Why mashup

• Everybody needs customized applications, tailored to his/her own requirements, taste, style.

• DIY in software/web application construction– There are many building blocks (web APIs, web services)– there are also some tools to assemble the building blocks

• It is like end-user programming

60-440 5

Mashup flavours

• Presentation mashups: e.g. Google Maps apps, with emphasis on presentation• This is the shallowest form of mashup in the sense that

underlying data and functionality don’t meet.  Information and layout is retrieved and either remix or just placed next to each other.  Many of the Ajax desktops today fall into this category and so do portals and other presentation mashup techniques.

• Data mashups: merging data from different sources• Business mashup: combination of above. combines data

integration, presentation plus addition functionality, such as collaboration features

Mashup

60-440 6

Genres of mashups

• Mapping mashups• Video and photo mashups• Search and Shopping mashups• News mashups

• Mashup top categories in Programmableweb.com (2009 vs. 2008):

Mashup G

enres

60-440 7

Top APIs in programmableweb in 2008

Top API Description Category Mashups

Google Maps Mapping services Mapping 1291     

Flickr Photo sharing service Photos 296     

Amazon eCommerce Online retailer Shopping 211     

YouTube Video sharing and search Video 204     

411Sync SMS, WAP, and email messaging Messaging 118     

eBay Online auction marketplace Shopping 116     

Microsoft Virtual Earth Mapping services Mapping 115     

Yahoo Maps Mapping services Mapping 108     

Google Search Search services Search 95     

del.icio.us Social bookmarking Bookmarks 94     

Yahoo Search Search services Search 93     

Yahoo Geocoding Geocoding services Mapping 68     

Technorati Blog search services Blog Search 50     

Google Homepage Portal gadgets Widgets 41     

Facebook Social networking service Community 39     

Last.fm Music playlist management Music 37     

Yahoo Local Search Local search service Search 37     

Google Ajax Search Web search components Search 36     

Yahoo Image Search Image search services Search 36  

60-440 8

Top APIs in Oct 2009

60-440 9

Map mashups

• Locational information presented graphically using maps in a specified context

• Google Maps API opened the floodgates– mash all sort of data from nuclear disasters to Boston’s

CowParade cows onto maps

• Other APIs– Microsoft (Virtual Earth)– Yahoo (Yahoo Maps)– AOL (MapQuest)

Mashup G

enres

60-440 10

Photo and Video

• Photo/video hosting and social networking sites resulted interesting mashups.– Flickr– YouTube– Facebook

• Metadata associated with the hosted images and videos– Who took the picture– What it is a picture of– Where and when it was taken– So on…

Mashup G

enres

60-440 11

Search and shopping

• Exist long before the term Mashup was coined– Combinations of B2B technologies or screen-scraping to

aggregate comparative price data– eBay– Amazon– …

Mashup G

enres

60-440 12

News

• News sources such as BBC and Reuters have used syndication technologies like RSS and Atom since 2002.

• Personalized newspaper by Syndication feed mashups• Doggdot.us, combines feeds from the techie-oriented

news sources Digg.com, Slashdot.org and Del.icio.us

Mashup G

enres

60-440 13

Content provider

• Web Protocols– REST Services– SOAP Web Services– RSS/Atom

• Screen Scraping– Scraping is the process of using software tools to parse and

analyze content that was originally written for human consumption in order to extract semantic data structures representative of that information that can be used and manipulated programmatically.

60-440 14

Mashup platforms

• IGoogle• Yahoo Pipes• Openkapow• IBM QedWiki• Microsoft popfly (discontinued on august 24, 2009)• Google Mashup Editor (in january 2009, migrated to Google App Engine)

• BEA’s Aqualogic Pages • Apatar • Dapper • Applibase’s DataMashups • Denodo’s data mashup product suite. • Extensio • JackBe’s Presto • Proto • RSSBus • SnapLogic • SOA Express • Teqlo • WSO2 Mashup Server

Mashup platform

s

The following slides are based on a presentation by SAP research

60-440 15

IGoogle

• IGoogle is a mash-up platform that aims to centralize all personal information in a portal (or web page)

• IGoogle is based on:– Free wheel composition of gadgets in a web page, considered as

a personal desktop from where a web users’ experience starts every day

– Users can choose gadgets from:• A catalogue of ready-made components

• Gadgets created by end-users

Mashup platform

s

60-440 16

My igoogle

Mashup platform

s

60-440 17

Add other gadgets

Mashup platform

s

60-440 18

Advantages of IGoogle

• Catalogue has a lot of existing gadgets to choose from – 28,000 + gadgets in Jan 2008.

• You have the option of creating new gadgets to use in the mashup– Google Mashup Editor (GME).– Here is a good tutorial

• IGoogle users can accumulate a lot of useful information on their home page

• End-users can fix the mashup, replacing/resizing gadgets on the page

• Users can create complex applications, located on their home page

Mashup platform

s

The following slides are from SAP Research

60-440 19

Disadvantages of IGoogle

• End-users will find it difficult to create complex mashups, because:– Heavy programmatic skills are required– There is a poor variety of data inputs– There is high coupling between mashups and the platform (a

middleware gadget is required to allow gadgets to communicate)– Created mashups are not compatible with any other platform– There is no inherent communication between gadgets

Mashup platform

s

60-440 20

Yahoo! pipes

• Visual development environment for generating data-oriented mashups

• Development is based on:– Dragging gadgets (pipes)

from a toolbox and dropping them in work zone

– Specifying data input– Interconnecting gadgets

through pipes– Specifying data output

format

Mashup platform

s

60-440 21

Advantages of Yahoo Pipe

• Entirely graphic and intuitive IDE; not one line of code is required

• Inherent internal communication between gadgets• RSS, CSV and JSON support• Portability of mashups to other platforms (i.e. QedWiki)

Mashup platform

s

60-440 22

Disadvantages of Yahoo Pipe

• Yahoo pipes does not generate XML from the mashup (limiting hacking and remixing)

• Data inputs are limited to a prefixed set• It is very difficult to create complex mashups with really

powerful functionalities early on• Other platforms are usually required to create enterprise

mashups

Mashup platform

s

60-440 23

Openkapow

• http://openkapow.com/ – to be Shut Down on December 11th, 2009

• Mashup platform for creating and remixing gadgets called “robots”. Robots have the following features:

• Each robot is like a little components mashup• Robot inputs:

– XML– HTML through screen scraping

• Robots can manage a wide variety of data outputs• Robots can communicate with each other• These robots are created and managed with a desktop

IDE, RoboMaker

Mashup platform

s

60-440 24

Advantages of Openkapow

• This platform has a large catalogue of prefabricated resources

• There is inherent communication between gadgets• RoboMaker is a powerful and visual IDE• It has a wide range of data inputs (XML, HTML, …) and

outputs (XML, HTML, CSV, JSON, XHTML)• It includes several ways to interact with end-users in final

mashups (forms, REST calls, …)• Platform users can have their own robots server

Mashup platform

s

60-440 25

OpenKapow - Disadvantages

• The platform is focused entirely on development, instead of exploitation

• desktop IDE, not browser based• targets heavy-tech users, with previous programming

skills• Robots can’t be easily hacked or remixed• The OpenKapow mashup server is not free• The platform is overly centered on HTML and screen

scraping

Mashup platform

s

60-440 26

IBM QedWiki

• Mashup platform that combines two perspectives (data-oriented mashups and presentation mashups)

• Features:– Data inputs:

• Databases• Feed RSS• Yahoo Pipes mashups• Spread sheets• XML• …

• Mashup process based on drag & drop operations, using gadgets from a toolbox, or further searching from a wiki of resources– Internal communication between gadgets is supported

– Mashup results can be entered in a wiki, tagged, and refined by other users

Mashup platform

s

60-440 27

IBM QedWiki - Advantages

• QedWiki combines several tendencies; it supports data mashups, like Yahoo Pipes,and creates complex presentation mashups, like OpenKapow

• It supports internal communication between gadgets• It supports a wide range of data inputs, managing

databases directly• Web-based IDE, no additional software has to be

downloaded• It can manage the internal code of each gadget and even

of each complete mashup (according to its own XML-based standards)

• Simple documentation support oriented to inexperienced users

Mashup platform

s

60-440 28

IBM QedWiki - Disadvantages

• You need to know code standards and the gadget programming language to build complex solutions because of limitations in the supplied visual IDE

• Screen scraping is not supported• There is little, poorly detailed and slowly updated

documentation. The QedWiki Project is not hot today.

Mashup platform

s

60-440 29

Ajax• AJAX (Asynchronous JavaScript and XML) • Ajax and AJAX• Characteristic: increased responsiveness and

interactiveness of web pages – exchanging small amounts of data with the server – entire web page does not have to be reloaded each time the user

performs an action.

• Not a technology itself, but a term refer to the use of a group of technologies

• The "core" and defining element of Ajax is the XMLHttpRequest object, which gives browsers the ability to make dynamic and asynchronous data requests without having to reload a page, eliminating the need for page refreshes.

Web 2.0

The following slides are from Jimmy LinThe iSchoolUniversity of Maryland

60-440 30

“Old-School” Web Applications

server-side systems

backend database

browser

Inte

rfac

e

Webserver

HTTP request

HTTP response

1

user does something

2

browser sends request to server

3

server generates Web page as a response to the request

4

data is returned in response to the request

5

browser replaces view with data sent from server

60-440 31

Characteristics

• User-driven: Things only happen when the user does something (e.g., clicks on a link or button)

• Views defined by URLs: You can bookmark something and come back to it; use the forward/backward button

• Simple user interaction model: Not that many things you can do in browser

• Synchronous Interaction: System responses are synchronized with user-driven events

60-440 32

Synchronous Interactions

browser

server-side

Time

user activity user activity user activity

server processing server processing

Re

qu

est

Re

qu

est

Re

spo

ns

e

Re

spo

ns

e

browser server-side systemsHTTP request

HTTP response

12

45 3

1

2

3

4

5

60-440 33

LAMP

Linux

Apache

MySQL

PHP/Python/Perl

So what do you run on the server side?

60-440 34

From “Old-School” to Ajax

browser

Inte

rfac

e

request

response

Ajax intermediates between the interface and the server.

server-side systems

backend database

Webserver

data management

Ajax“engine”

interaction management

60-440 35

Inside the Browser

browser

HTML / CSSdata

HTML / CSSdata

other data(e.g. images)

other data(e.g. images)

Inte

rfac

e

Ren

der

ing

En

gin

e

HTTP request

HTTP response

60-440 36

Enter JavaScript

browser

HTML / CSSdata

HTML / CSSdata

other data(e.g. images)

other data(e.g. images)

Inte

rfac

e

Ren

der

ing

En

gin

e

HTTP request

HTTP response

JavaScript Engine

JavaScriptcode

JavaScriptcode

60-440 37

Enter Ajax

browser

HTML / CSSdata

HTML / CSSdata

other data(e.g. images)

other data(e.g. images)

Inte

rfac

e

Ren

der

ing

En

gin

e

HTTP request

HTTP response

JavaScript Engine

XMLdata

XMLdata

JavaScriptcode

JavaScriptcode

XMLHttpRequest

HTTP request

60-440 38

From Synchronous Interactions…

browser

server-side

Time

user activity user activity user activity

server processing server processing

Re

qu

est

Re

qu

est

Re

spo

ns

e

Re

spo

ns

e

60-440 39

To asynchronous Interactions

browser

server-side

Time

user activity

server processing server processing

Re

qu

est

Re

qu

est

Re

spo

ns

e

Re

spo

ns

e

client-side processing

60-440 40

Components of an Ajax Interaction

1. A client event occurs (captured by JavaScript event handlers)

2. An XMLHttpRequest object is created and configured

3. An asynchronous request is made to the server via the XMLHttpRequest object

4. Server processes request and returns data, executing a callback in the XMLHttpRequest object

5. The HTML DOM is updated based on response data

60-440 41

DOM

• Document Object Model: platform- and language-independent way to represent XML– Adopts a tree-based representation– W3C standard, supported by modern browsers

• JavaScript uses DOM to manipulate content– To process user events– To process server responses (via XMLHttpRequest)

60-440 42

Ajax: Things to watch out for!

• Hype– Best thing since sliced bread?

• Application development/maintenance cost– Brower incompatibilities– Many different approaches and tools– For many things, lack of agreed-on best practices

• Behavior is not ‘Web-like’ – Standard things often don’t work correctly (e.g., browser ‘back’

button, bookmarks) – Usability issues for users with disabilities

• Security issues– Whole new class of cross-site scripting (XSS) exploits

60-440 43

Learning Ajax

• Bewildering options:– PHP vs. Python vs. Perl vs. ASP vs. JSP …– XML vs. JSON– Countless toolkits, frameworks, libraries, etc.

• Amazing amount of information online:– Numerous Web tutorials– Learn by example– Learn by building