select * from internetYahoo Query Language:

Derek Gathright

var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…

…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…

http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2

http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c

var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…

…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…

http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2

http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c

http://weather.yahooapis.com/forecastrss?p=FRXX0076&u=c

…if (flickcurl_prepare(fc, "flickr.photos.geo.correctLocation", parameters, count)) { … }…

http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&query=Corvette&results=2

var map = new YMap(document.getElementById('map'));…var currentGeoPoint = new YGeoPoint( _c.Lat, _c.Lon ); map.addMarker(currentGeoPoint);…

YQLYQL stmt

Why YQL?

Thousands of Web Services and sources that provide valuable data

Require Developers to read documentation and form URLs/queries.

Data is isolated

Needs filtering, combining, tweaking, shaping even after it gets to the developer.

What is YQL?

Cloud web service and SQL-Like Language • Familiar to developers• Synonymous with Data access• Expressive enough to get the right data.

Self describing - show, desc table

Allows you to query, filter, join and update data across any structured data on the web / web services

• And Yahoo’s sherpa cloud storage

Real time engine

YQL Statement Basics

show: lists the supported tables

desc: describes the structure of a table

select: fetches data

insert/update/delete: modify data

use: use an Open Data Table

set: define key-values across Open Data Tables

Insert / Update / Delete

With INSERT / UPDATE / DELETE SQL verbs you can modify data on the internet or cloud

• create tweets, add blog entries, update DBs etc• store tables or pieces of code in sherpa

Some tables with I/U/D• Twitter, Bit.ly shortener• Yahoo profile status, application update stream• Amazon SimpleDB

Demo: developer.yahoo.com/yql/

Get any data:

Open Data Tables

Open Data Tables

• http://github.com/yql/yql-tables/tree/master

• datatables.org mirrors the master and builds a single environment with all tables (faster serving)

• Console with community tables

• Brings the power of the table model to new APIs

• Schema defines mapping between YQL and Endpoint

• Keys can either be query, path or matrix parameters

• Keys can be marked as required

• YQL Compiler validates existence of required keys

YQL

Open Data Table example

Open Data Table example

Open Data Table example

ODT <execute> element

Extends Open Data Tables with server-side code

Javascript server-side scripting• No DOM, events etc, pure JS scripting• E4X support - XML literals, filtering ...

YQL no longer performs a GET request to the table binding URL

YQL provides additional useful global objects• request, response, y.rest, y.include, y.query...

Execute tag must return the data that will be the output of the YQL select

Move business logic of your application to the cloud

What you can do with <execute>

access APIs that require authentication• Netflix OAuth, FlickrAuth, Google AuthSub

join data across services• grab New York Times article tags and generate find associated flickr photos• combine multiple searches into a single result (twitter, web, news and images)

augment data:• city/state to APIs that just return zip code

create APIs from web page• celebrity birthdays scraped from imdb, caltrain

data transformation• convert the result from xml to Google's kml format

hello world

<table xmlns="http://query.yahooapis.com/v1/schema/table.xsd"> <meta> <sampleQuery>select * from {table} where a=’hello' and b=’world';</sampleQuery> </meta> <bindings> <select itemPath="" produces="XML"> <urls><url>http://fake.url/{a}</url></urls> <inputs> <key id='a' type='xs:string' paramType='path' required="true" /> <key id='b' type='xs:string' paramType='variable' required="true" /> </inputs> <execute><![CDATA[ // Your javascript goes here. We will run it on our servers response.object = <item> <url>{request.url}</url> <a>{a}</a> <b>{b}</b> </item>; ]]></execute> </select> </bindings></table>

Scraping a page

<select produces="XML" itemPath="deals.deal"> <urls><url>http://deals.yahoo.com</url></urls> <execute><![CDATA[var dealhtml = y.query("select * from html where url='http://deals.yahoo.com' and xpath='//div[contains(@class, \"product-display\")]'").results;var deals = <deals/>;function compress(object) { if (!object) return ""; return object.toString().replace(/\s+/, " ")}for each (var div in dealhtml.div) { var deal = <deal/>; deal.name += <name>{compress(div.h3.a.text())}</name>; deal.link += <link>{div.div[0].a.@href}</link>;… deal.description += <description>{compress(div.p[0].*)}</description>; deal.logo += <logo>{compress(div.div[1].*)}</logo>; deals.deal += deal;}response.object = deals; ]]></execute> </select>

<select itemPath="" produces="XML"><urls><url>http://twitter.com/statuses/show/{id}.xml</url></urls><inputs>

<key id="username" type="xs:string" required="false" paramType="variable"/><key id="password" type="xs:string" required="false" paramType="variable"/><key id="id" type="xs:integer" required="true" paramType="path"/>

</inputs><execute><![CDATA[

var r = null; if (username && password) { y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); r = request.header("Authorization",authheader).get().response; } else { r = request.get().response; } response.object = r;

]]></execute></select>

select * from twitter.status where id="2108869549" and username="xxx" and password="yyy”

Twitter Select

<insert itemPath="" produces="XML"><urls>

<url>http://twitter.com/statuses/update.xml</url></urls><inputs>

<value id="username" type="xs:string" required="true" paramType="variable"/><value id="password" type="xs:string" required="true" paramType="variable"/><value id="status" type="xs:string" required="true" paramType="variable"/>

</inputs><execute><![CDATA[

y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); var content = "status="+status; response.object = request.header("Authorization",authheader).post(content).response;

]]></execute></insert>

insert into twitter.status (status,username,password) values ("new tweet from YQL",”xxx”,”yyy”);

Twitter Update

<delete itemPath="" produces="XML"><urls>

<url>http://twitter.com/statuses/destroy/{id}.xml</url></urls><inputs>

<key id="username" type="xs:string" required="true" paramType="variable"/><key id="password" type="xs:string" required="true" paramType="variable"/><key id="id" type="xs:string" required="true" paramType="path"/>

</inputs><execute><![CDATA[

y.include("http://yqlblog.net/samples/base64.js"); var authheader = "Basic "+Base64.encode(username+":"+password); response.object = request.header("Authorization",authheader).del().response; ]]></execute></delete>

delete from twitter.status where id="2108869549" and username="xxx" and password="yyy”

Twitter Delete

Conclusion

More easily build applications• fewer lines of code• data processing done away from app• consistent and familiar syntax for all data access

Conclusion

Build applications that run faster• service in the cloud does the work

• store tables, environments, js, text files…

• conversion, filtering, parallel requests...

With smaller network footprint• fewer (client) calls

• smaller data amounts

• closer to the data, fatter pipes

Conclusion

Flexible• Supports XML, JSON, JSONP

• Transform documents between formats

• Easy to integrate into any development environment w/ HTTP

Questions?

http://developer.yahoo.com/yql

Derek Gathright

drg@yahoo-inc.com

twitter.com/derek

meetup.com/socaljs