budapest.py

Google Appengine as a perfectmashup platform

Janos Hardijanos@liligo.com

http://github.com/yaanno

What is a mashup?

Some original idea implemented on the top of various 3rd party web

services by mixing them.

What do you need for a good mashup?

A smashing idea

Technical background

3rd party services

An example

A good idea (me thinks):

A travel search bot based on a 3rd party engine plus Twitter (+ Jabber) messaging.

Search types: flight, hotel, inspiration, etc.

Sideslide

Twitter bots: http://twitter.pbworks.com/Bots

Twitter mashups: http://twitter.pbworks.com/Mashups

Google AppEngine

Technical background:

Database (datastore)

Cache (memcache)

3rd party apis via HTTP (urlfetch)

... and many many more ...

3rd party web services

Good starting points:

programmableweb.com (apis)

mashable.com (news)

Mashup ingredients

Twitter API

Jabber API

Yahoo Trip Planner API

Travel Search APIs

Travel Search APIk

Kayak

Liligo

Cleartrip

Kayak Travel Search API

- Kayak infrastructure

- Slightly complicated data exchange

- XML based

- Poor documentation

- Discontinued! Now you have “RSS APIs” ...

Liligo Travel Search API

- Liligo infrastructure

- Complicated setup procedure

- XML based

- Good documentation

Cleartrip Travel Search API

- Cleartrip infrastructure

- Dozens of services

- Simple data exchange

- XML based

- Excellent documentation

- From India with love

Yahoo! API

- Stone simple

- XML based

- Good documentation

Twitter API

- Simple as stone

- XML/JSON based

- Good documentation

Google (Data) APIs

Loads of stuff, amazing.

http://code.google.com/p/gdata-python-client/

Now what?

What we have is ether nothing or almost useless.

Demo

http://tripideas-bot.appspot.com

Dumped (partially) :(

Web based bot?

Cron service calls /cron resource

/cron sets up default tasks

Task manager processes these tasks (queue api)

Default tasks

- Do we have new messages?

- Do we have new searches stored?

- Do we have unreplied messages?

... and a lot more ...

How?

Apparently we don't have parallel / async events, processes and so on but we have a nice Task Queue api.

Task Queue

class MessageHandler(webapp.RequestHandler): def get(self): task_type = self.request.get('type') queue = Queue(name='messaging') if task_type == 'collect': task = Task( url='/worker/collect_messages', method='GET', ) queue.add(task)

Task Queueclass SearchHandler(webapp.RequestHandler): def get(self): task = Task( url = '/worker/search', params = { 'engine' : 'yahoo', 'searchtype' : 'inspire', 'query' : query['query'], 'maxresults' : '30', 'request_id' : query['request_id'], }, method = 'GET', ) queue.add(task)

Message Workerimport loggingfrom worker import Workerfrom .modules.messaging import Messaging

class MessageWorker(Worker): def __init__(self): logging.debug('new messaging worker spawned') def get(self): messaging = Messaging() messaging.collect() messaging.process() messaging.save()

Search Worker

import loggingfrom worker import Workerfrom .modules.search import Search

class SearchWorker(Worker): def __init__(self): logging.debug('new search worker spawned') def get(self): request = self.request.query_string query = self.normalize(request) search = Search(query) raw_result = search.search()

Processing #1

def collect(self): self.messages = t.status_replies()

def process(self): self.messages = self._serialize(self.messages)

def _serialize(self,serializable): return json.loads(serializable)

Processing #2

import xml.etree.ElementTree as ET

xml = ET.fromstring(self.result)

trips = xml.findall("trips/trip")

for trip in trips:id = trip.get("id")priceelem = trip.find("price")price = priceelem.texturl = priceelem.get("url")

Page render & Cache

class UserPage(Page):

def get(self,*args):

userpageid = str('userpage_' + args[0]) page = memcache.get(userpageid)

if page is None: userid = long(args[0]) author = Author.all().filter('author_id', userid).get() messages = Message.all().filter('author_id',

author.key().id()).fetch(20) results = { 'author': author, 'results':[] } page = self.prerender('user_page', {'results':results}) memcache.put(page) self.response.out.write(page)

Natively http://googleappengine.blogspot.com/2009/02/roadmap-update.html

External service http://morethanseven.net/2009/02/21/example-using-xmpp-app-engine-imified/

PyGtalkRobot basedhttp://code.google.com/p/aiassistant/

Jabber on Appengine

Thanks for your time!