How The Guardian Embraced the Internet using Content , Search, and Open Source

From publisher to platform: How the Guardian embraced the internet using content, search, and Open Source

Stephen Dunn, Guardian News and [email protected], 25th May, 2011

Twitter: @cuica, @openplatform

Thursday, 26 May 2011

1

From publisher to platformHow the Guardian embraced the Internet using content, search, and Open SourceStephen Dunn, Guardian News and Media

2


The publishing era

3


We started a long time ago:


Swine flu

Keyword page

Twitter updates

Content partnerships

Audio

Video Open platform API

Live blogs

Comment

Mobile siteApps

Newspapers


To secure the financial and editorial independence of the Guardian in perpetuity. To promote freedom in the press and liberal journalism globally.

To become the world's leading liberal voice.

To secure the financial and editorial independence of the Guardian in perpetuity

To promote freedom in the press and liberal journalism globally


7

Open Web Principles


8

2009


• “A cool URI is one that does not change” Tim Berners-Lee 1998• 1.5 million resources redirected to new scheme

9

1. Permanent

http://www.flickr.com/photos/fstorr/




10

2. Addressable★ Resources are “about” something - ready for the

social web.

★ We live in “the age of point-at-things” (Coates 2005)


11

★ Multiple routes to content

★ Tagging drives discovery

3. Discoverable


12

4. Open


/technology/internet

/technology/all

/environment/climatechange

Example: The Hackable Guardian

http://www.guardian.co.uk/....

/rss

/rss

+business/globaleconomy/rss


http://www.guardian.co.uk/technology/all











http://www.guardian.co.uk




Results...

14


15

3,750,000

7,500,000

11,250,000

15,000,000

18,750,000

22,500,000

26,250,000

30,000,000

Sep 2005 Oct 2006 Nov 2007 Dec 2008

Uni

que

Use

rs Pre - project

First release

Final ReleaseSite traffic growthUnique Users

40M


However...

16


17

1 Billion+Internet Users!


18


19


20


21

...“How I stopped worrying about my website and learned to love the whole internet.”

Matt McAlister


22

The Open Strategy

OPEN IN

Bring in data and apps from the Internet

OPEN OUT

Enable partners to build applications using Guardian content and services for other platforms


23


24

"Our most interesting experiments lie in combining what we know with the experience, opinions and expertise of the people who want to participate rather than passively receive.”


25


26


27


28


29


30


31


32


33


34

“The Guardian alongside Al Jazeera was the one news source that everybody on the streets in Tahrir - not just in Cairo but in surrounding cities and major centers of revolutionary activity - that people were talking about.”

Jack Shenker


3522

The Open Strategy

OPEN IN


OPEN OUT



36

The Open Platform


37

The suite of services enabling partners to build applications with

the Guardian


3822

OPEN IN


OPEN OUT



CONTENT APIA service for selecting and

collecting content from the Guardian

for re-use

DATA STOREA directory of

useful data curated by Guardian editors

POLITICS APIOpen database of candidates, voting records, constituencies, election results,

live data on election day


Mutualised news!

40


Mutualised news!

41


Mutualised news!

42


43


44


45


46


DATA STOREA directory of

useful data curated by Guardian

editors


POLITICS APIOpen database of candidates, voting

records, constituencies, election results, live data on election day


POLITICS APIOpen database of candidates, voting

records, constituencies, election results, live data on election day

49


<OBLIGATORY DOGFOOD SLIDE >

50


51






56

Open for Business


57

3 Tiers of access3 Revenue models

Keyless: Take our headlines. You keep associated revenues.

Approved: Take our full article content, but with an advert. Guardian keeps ad revenue, you keep rest-of-page revenue.

Bespoke: Take, reformat, augment our content Revenue model to be negotiated. Combination of Media, Fees, Downloads.


58


59

What this means

Open Out: Developers can now access full content APIs on demand with keys post-approved

Platform is positioned as a place to do business

So rapid scalability, reliability and performance are now core requirements


OPEN INBring in data and

apps from the internet

OPEN OUTAllow partners to build applications using Guardian

content and services for other

platforms


61

A framework for integrating 3rd party

applications into guardian.co.uk

MICROAPPS Simple REST/HTTP framework allows lightweight development

Applications proxied for performance

Apps generally hosted in the cloud, allows hot deployment into production


62

A framework for integrating 3rd party

applications into guardian.co.uk

MICROAPPS


• What could I cook?


64

Bringing it together


65


App showcase

66


67

From publisher to platform

Seeking massive growth, but no longer only broadcasting content on the website

User/partner engagement & contribution onJournalismdatasoftwareapplicationsrevenue and ads

Support developers and partners with data and APIs,need scalability, reliability, speed


68

Evolving the architecture


App server App server App server

Web server Web server Web server

CMS

Oracle

Memcached (added later)


App server App server App server

Web server Web server Web server

CMS

Oracle

Memcached

Why RDBMS?

5 years ago, fewer alternatives

Understand operations procedures

Can easily recruit DBAs / devs

Developer/ops tools

Business critical system: a safe choice


71

3,750,000

7,500,000

11,250,000

15,000,000

18,750,000

22,500,000

26,250,000

30,000,000

Sep 2005 Sep 2006 Sep 2007 Sep 2008

Uni

que

Use

rs

Scaling trafficUnique Users


72


73


74


75


76


77


We chose Solr/Lucene

78

Can perform complex queries, including full-text search

We can change the schema with no downtime

Most queries are of similar cost

Scales very well horizontally

“Just worked” in the cloud

No strange control processes/engines

Developers just loved working with it!


79


App server

Web servers

CMS

Memcached

RDBMS

80

Solr

Solr

Solr

Solr

Solr

Solr

Cloud, EC2

Api


8122

OPEN IN


OPEN OUT


What about Open In?


App server

Web servers

CMS

Memcached

RDBMS

82

App

App

App

App

App

App

Apps

Proxy

external hostingapp engine etc


App server

Web servers

CMS

Memcached

Solr

Core

Solr

Solr

Solr

Solr

Solr

Cloud, EC2

Out

App

App

App

App

App

App

In

Proxy

external hostingapp engine etc

rdbms

83


84


85


Documents

How The Guardian Embraced the Internet using Content , Search, and Open Source