THE WEB OBSERVATORY DATA GATHERING WITH

EUGENE SIOW & XIN WANG

29 JANUARY 2016

WHAT IS THE WEB OBSERVATORY?

SEARCH + ACCESS

DATA | WO | APP OPEN

PRIVATE

NoSQL

SQL STREAMS

LINKED DATA

JS

PYTHON NODE

GATHERING DATA WITH SCRAPING

DATA ON WEBPAGES DATA I CAN USE transform ( )

THE PROCESS OF SCRAPING

INVESTIGATE THE STRUCTURE OF THE PAGE

CHECK IF THERE IS AN API {APPLICATION PROGRAMMING INTERFACE}

USE CHROME’S INSPECTOR OR FIREBUG

EXTRACT, TRANSFORM, LOAD WHAT IS YOUR DESIRED END FORMAT?

HANDS-ON RESOURCES

codepen.io/xgfd/pen/wMyQWb

github.com/eugenesiow/datathon2016/wiki

DATA-DRIVEN APPS USING THE WO

DATA GATHERING

webobservatory.soton.ac.uk THE SOTON WEB OBSERVATORY

BACKGROUNDS FROM THE HUBBLE SPACE TELESCOPE