The Benefits of Publicly-Accessible Data

An Open Source Interface and Reimplementation of NOHRSC SNODAS Products

Thomas Horner – November 2015

Context• Web GIS is redefining a lot of things

• I think I have sat through this slideshow 500 times• Webmaps have become the preferred data visualization method

for the public – Google Maps changed everything in 2005• 2005: Car navigation• 2015: “Navteq/Yelp/Yellowpages/Fodor’s/Michelin Guide/Zagat/AAA all in one.”

• Popular casual apps: Waze, Strava, Weather Underground, Google Earth

• Great map-based web tools here in Colorado:• Colorado Avalanche Information Center (CAIC)• COTrip• CalTopo• 14ers.com• NWS

Context• Not just maps but data too – web services

• Data is always up to date• Source has control over distribution and quality of data

• Govt agencies using open web services:• USGS – Dozens of vector layers and tilelayers• NOAA – Tons of layers• Dept of State – Again, tons of layers• Dept of Transportation• US EPA• …State of North Dakota

User

ServerWMSData

DataWeb

Browser

ServerWMSData

Data

So WhatFrom just a webmap developer perspective…

1. Copy this code2. Paste into

webmap code3. Move on with your life

Adding a data layer…

Hundreds of layers

So WhatCalTopo – Popular free tool – lots of web services consumed and distributed

So WhatFreshyMap

Publicly-Accessible Web Services

• Developers don’t waste their time• Layers can be added immediately• Working with the data is no mystery - standards

• Cartographers don’t waste their time• MapBox, OSM base data, so many tilelayer

choices• Choice between pre-styled layers or DIY

• Scientists, academics, professionals don’t waste their time• Real-time inputs to complicated models• Easy to procure dozens of layers and datasets• Easy to determine if dataset is useful without

downloading all the data• Work with only what you need• Always having to hunt down data and metadata

SNODAS• SNODAS: National snowpack dataset (NWS)

• Several data layers• Updated every 24 hours• Great for water planners, but accurate enough for appropriate

use in other applications• No web service – archives uploaded to FTP

• ‘Official’ webmap interface is painful to use• Data format is ‘primitive’ – cannot natively be used• Has any of the tech been upgraded since 2003?

• So how do we use this layer in a modern webmap? (let’s see what we’re currently working with)

SNODASOfficialSNODASInterface

To zoomand panyou haveto refreshthe map

SNODAS“Uhhh, where exactly is Castle Creek?”

Dealing With It• I want to use this real-time data in

webmaps• Time to write a wee bit of code…• Automate these steps:

1. Find and download latest dataset from FTP server

2. Convert to standardGeoTIFF format for GIS use

3. Stylize and/or push outas web service

1. Automated Retrieval

Have to provide the exact, correct URL based on the current date, ex:ftp://sidads.colorado.edu/DATASETS/NOAA/G02158/masked/2015/11_Nov/SNODAS_20151117.tar

1. Automated Retrieval• Now decompress the archive….

WHAT HAVE I DONE???

1. Automated Retrieval• Decompress THOSE archives• Now have .hdrs and .dats• Prep for conversion• Run a regex to strip “illegal strings” from hdrs

• Rename files to match layer names”us_ssmv11034tS__T0001TTNATS2015111705HP001” -> “snow_water_equivalent”

2. Conversion• GDAL – The ultimate open source GIS data tool• Supports SNODAS conversion in recent

versions(the files could still be read in previous versions,with a little trickery)

• Script executes ‘gdal-warp’ after data is prepared

• Data layers are converted to GeoTIFF andweb Mercator projection

• Old archives and raw data files removed

3. Distribution• Overwrite the previous GeoTIFF outputs• MapServer consumes these files as layers for

WMS, WCS, etc.• WMS is styled to match the colors of the official

webmap• Anyone consuming the WMS is always getting the

latest uploaded SNODAS data• Can be added to any webmap with one or two

lines of code – or into GIS apps like QGIS or ArcGIS

The Web

Slippy-SNODAS• Demo interface that consumes this

SNODAS web service• Leaflet.js – runs in browser or on mobile

phone• Demo hosted on my personal website

• Automated scripts look for new data on each page load• Will we get to see a demo?

http://alptruth-geo.com/SlippySNODAS/

Slippy-SNODAS

Open Source• MIT License• Hosted on GitHub – Collaborative coding,

issue tracking, documentation• Let’s get some live layers going!

• Maybe NOHRSC will upgrade their technology in the next decade

• github.com/thurs/slippy-snodas

Conclusion• SNODAS data is provided reliably to the public

– but archaically• Difficult to immediately jump into a project that uses

SNODAS• Difficult to use the web interface to perform analysis

• But all that is required is open access and documentation

• Public is empowered to consume the data and upgrade/maintain it with newer technology

• Many web services (of public data) were started/hosted by the public -- (especially universities) – govt agencies are coming around to providing it themselves for better control

Thanks• SNODAS Data:

• National Operational Hydrologic Remote Sensing Center. 2004. Snow Data Assimilation System (SNODAS) data products at NSIDC. Boulder, CO: National Snow and Ice Data Center. Digital media.

• Official webmap:• http://

www.nohrsc.noaa.gov/interactive/html/map.html• Slippy-SNODAS demo:

• http://alptruth-geo.com/SlippySNODAS/• Code repository:

• github.com/thurs/slippy-snodas