[Socrata] Open Innovation - Volume 2

8/13/2019 [Socrata] Open Innovation - Volume 2

1/21

Designing the Future:

Code for America

Makes an Impact

Data-Driven

Government

in Action

Codelescence:

Engineering Comes

of Age

Socrata

2013 Highlights

WINTER 2014

Fixingthe Federal ITProcurement Process


2/21


3/21

OPEN DATA

INFOCUS

OPEN INNOVATION WINTOPEN INNOVATION WIN TER 2014

OPEN DATAI

In todays world o data standards, we

are operating in pre-mainstream chaos,

similar to that which existed with

electricity in Edison and eslas time.

Te potential or data-as-a-uel exists as

a natural resource in much the same

way as magnets and copper did in the

industrial revolution. As this next wave

o economic opportunity takes root, the

same kind o arguments are being made.

GREAT THINGS ARE POSSIBLE

WITH OPEN DATA

We benefit rom technical standardsdaily. Tey allow drivers to purchase gas

rom any station without ear o using

incompatible uels. Tey allow a text

message rom a Verizon phone to be

transmitted to an A& phone. Data

standards operate in a similar manner,

but their use within modern applications

is relatively new. Te most popular one

in use today is the General ransit Feed

Specification (GFS). GFS allows users o

Google Maps to know when t he next bus is

going to arrive within participating cities.

Looking to evolving data standards that

deserve greater adoption, a great example

at the civic level is HouseFacts. Tere

are many businesses that help people

buy and sell homes (e.g. rulia, Zillow,

most commercial banks, etc.), but the

inormation on the saety and health o

those homes is messy and reported in

Why We NeedOpen DataStandards

Right NowBy Ian Kalin

Socrata Director of Open Data

As the open data movement gains

momentum, more organizations, businesses,

and citizens are looking to share data and

collaborate on projects. What is essential

to them doing so? Data standards. Data

standards are a topic that deserves our

communitys full attention right now. The

government innovation movement must

address their importance across industries,

across borders, and even between

departments.

THE DIFFERENCE

STANDARDS MAKE

First, lets discuss the importance o

standards in amiliar industries. Imagine

i the electrified world we live in today

had no standards; you might have to

replace your toaster every time you move

to a new house. When electricity first

made its way into the households o the

world, standards did not yet exist. o

that point, two o the worlds greatest

inventors, Nikola esla and Tomas

Edison, became engaged in an epic debate

between alternating-current versus direct-

current. Each argument had strengths and

weaknesses but esla made the innovative

choice to back his argument up with

household appliances, guaranteeing his

victory in the debate and establishing our

universal outlets today.

very different ormats. Te solution is

a uniorm ormat or reporting things

like asbestos, pest inestations, and

even abusive landlords. As standards

like HouseFacts are adopted by city

governments, businesses can aggregate the

data that most cities are already collecting

and integrate that inormation into the

websites people are using to make housing

decisions. Te map or standard adoption

becomes a map or business growth.

Te challenge lies in developing t

standards. Development can be d

because o the complexity o the

reflects the world in which it is cr

Te maintenance alone o engine

standards is hard, as demonstrate

huge network o organizations th

with standards, like the Internati

Organization or Standardization

the American National Standard

(ANSI), and UL.

As open data moves into our mai

world, making it as available as p

a consistent, efficient manner, is

encourage you to engage in the d

o defined data standards so that

open data truly serves t he world

who own it.

As open data moves into our mainstream

world, making it as available as possible, in a

consistent, efficient manner, is essential.
https://developers.google.com/transit/gtfs/http://housefacts.me/http://www.ansi.org/http://www.ul.com/global/eng/pages/http://www.ul.com/global/eng/pages/http://www.ansi.org/http://housefacts.me/https://developers.google.com/transit/gtfs/


4/21

(polynomial trend line)

Data Served by Socrata 2011-2013

2011 2012 2013


OPEN DATAIPEN DATAIN FOCUS

Bigger, Faster,Broader: HowOpen Data UseIs Changing

y Ben McInnis

ocrata Director of Product

magine a state that tracks K-8 student consumption

f free and reduced lunch and puts that data

ongside test scores and attendance information.

magine this data is then summarized in an internal

ashboard and sent to teachers via a mobile app.

this scenario, millions of data points updated

egularly could help with resource allocation on a

onthly, weekly, or even daily basis.

sites, and decision support systems or

government. says Socrata VP o Product

Sa Rabah.

Businesses, governments, and citizens

now use open data on a daily basis, and

they want to look at as much o it at

once as possible. For example, the City

o Chicago offers a traffic dataset t hat is

more than one million rows long. And,

San Franciscos app showing parking

availability is updated in near-real-time.

Having hosted millions and millions o

rows o public data since 20 07, Socratahas seen data use change dramatically,

particularly over the last two years. Here

are three o the strongest trends weve

noticed.

BIGGER DATA

wo years ago, the average dataset stored

on Socrata was under 10,000 rows. odays

common datasets all somewhere between

10 - 40 million rows. Socrata stores

datasets as large as 100,000,000 rows and,

by end o 2014, we expect to store datasets

o 1 billion rows or more.

As data scales in size, two interesting

things happen. First, data becomes

more useul or indirect and unoreseen

purposes because, within larger datasets,

common keys are more likely to exist.

Second, with more and more correlated

records across datasets, the need to

precisely clean and ormat data decreases.

Because o these characteristics,

analysis o large datasets can ofen yield

surprising and proound results. For

example, the City o Chicagos analytics

team discovered a relationship between

streetlight outages and petty thef--

correlations that were always latent within

their massive 311 and crime datasets.

While the connection might have seemed

obvious, having the data to clearly

understand this relationship helped the

city strategically address the problem.

FASTER DATA

High-requency data (data thats updated

very ofen) has seen similar growth. In

the last two years, as data rom sensor

networks and operational systems has

been published as open data, the number

o real-time datasets Socrata hosts has

grown more than 2,000 percent. (See

chart above.)

While static data is useul primarily or

archival and analytical purposes, real-

time or streaming data can be leveraged

as an input into other systems. Snowplow

locations can be ed in real-time to

citizens and the media, bus locations and

traffic signal state inormation can be

synchronized to optimize traffic flows,

and less obvious connections can drive

business efficiencies. A coffee vendor

might synchronize the preparation o resh

coffee and warm pastries to correspond

with the anticipated arrival o a commuter

train - not the scheduled arrival, but the

arrival time according to data coming

rom the tra ins computer system.

By transorming data rom a historical

record to a dynamic eed, it can be uti lized

on an on-going basis to make programs

and decision-making more efficient.

MACHINE-TO-MACHINE DATA

Data is also increasingly accessed, not by

humans, but by systems and applications

via application programming interaces

(API). Tey make data assets useul by

integrating them as an input or other

systems. For example, a new generation o

connected and sensor-enabled municipal

water systems provide data on water usage.

Tis inormation lets citizens understand

and manage their consumption, p

officials accurately plan or capac

needs and seasonal variance, and

personnel pinpoint leaks and ser

interruptions.

A study by technology market re

firm Hurwitz & Associates titled

Benefits o APIs in the App Econ

ound that organizations offering

as opposed to those that didnt, in

customer reach by 70 percent, nu

apps created by 50 percent, and n

mobile platorms supported by 58

oday, nearly every dataset is acc

via API and many thousands o t

per day. Socrata is the largest pro

government data APIs in the wor

more than 77,000 datasets availab

via API.

And, the total volume o data ser

Socrata via API has grown more

1,500 percent in the last two year

explosion in API usage across the

data movement comes rom inno

citizen apps like WasMyCarowe

app uses public data APIs to tell m

who return to their parking spot

no car, i their car wa s towed and

where.

FROM ARCHIVE TO FUEL

As the size o datasets grows, req

o dataset updates increases, and

machine-to-machine interaction

datas useulness is being tapped

novel uses and reuses are coming

light. Publishing data or the sak

transparency is a noble, yet short

goal. In the era o big, ast, useul

we are challenged to do more.

This way o tracking and

reporting on school programs

via web and mobile apps is just

one example o the many that

will emerge as government organizations

offer more public data as open data.

Since its ounding in 2007, Socrata has

seen open data move in larger quantities,

aster, and to more places than ever beore.

Were evolving past the cat alog phase o

open data and into the data-as-a-platorm

era, where value is measured by the real-

time flow and distribution o data th rough

application ecosystems, consumer internet
http://hurwitz.com/recent-research/item/web-api-study-the-benefits-of-apis-in-the-app-economyhttp://hurwitz.com/recent-research/item/web-api-study-the-benefits-of-apis-in-the-app-economy


5/21



Te Importanceof Engaged OpenData Advocates

y David Eaves

pen Innovation Expert

uring my 2011 keynote at Open Government Data

ampI talked about how the open data movement was

t an inflection point. For years we have been on the

utside, yelling that open data matters. Now we are

eing invited inside and we have a great responsibility

o be of service. Once you have world leaders talking

bout things like a G8 Open Data Charteryou are no

nger on the fringes - not even remotely.

initially. From the cell phone to t he car to

the printing press to open source sofware,

all these inventions have helped billions

o people, but they did not distribute

themselves evenly, especially at first.

So the question cannot be reduced to

will open data empower the empowered,

but to what degree, and where, and with

whom? Ive seen plenty o evidence where

data has enabled small g roups o people to

protect their communities or make more

transparent the impact (or lack there o)

o a government regulation. Open data

expands the number o people who canuse government inormation or their

own ends this, I believe is a good thing

but that does not mean we shouldnt be

constantly looking or ways to ensure t hat

it does not reinorce structural inequity.

PROMOTING THE IDEA OF DATA

AS A PLATFORM

Some o the issues around usability Ive

addressed above in the accessibility piece

or portals that genuinely want users,

the axis o evolution is pointed in the

right direction with governments and

companies like Socrata trying to embed

more tools on the website to make the data

more usable.

I also agree wit h a point by Proessor

Kitchin that, rather than creating a

virtuous circle, poorly thought out and

launched open data portals will create

negative doomloops in which poor

quality data begets little interest which

begets less data. However, the problem is

bigger than that.

One o the main reasons I have been an

advocate o open data is a desire to help

citizens, nonprofits, and companies gain

access to inormation that could help

them with their missions. I also wanted

to help change the way governments deal

with their data, so that they can share it

internally more effectively. I ofen cite a

public servant I know who had a sum mer

intern spend three weeks surfing the

national statistical agency website to find

data they knew ex isted but could not find

because o terrible design and search. A

poor open data site is not just a sign that

the public cant access or effectively use

government data; it usual ly suggests that

the governments employees cant access oreffectively use their own data. Tis is ofen

deeply rustrating to many public servants.

Tus, the most important outcome created

by the open data movement may be that

government organizations, save or those

in the intelligence community, realize that

they are not comortable with using data

to drive decisions. Getting governments

to think about data as a platorm (yes,

Im a an o government as a platorm or

external use, but above al l or internal use)

is, in my mind, one way we can enable

public servants to gain better access to

inormation. Adoption o this principle

will also, in many cases, obviate the need

or costly solutions rom huge vendors

(like SAP and Oracle), whose $100 million

dollar implementations ofen silo off data,

rarely produce the results promised and

are so obnoxiously expensive it boggles

the mind.

Te key to all this is that open data cannot

be something you slap on top o a big I

stack. I try to explain this in my blog post

Its the Icing Not the Cake about how

Washington, DC was able to effectively

launch an open data program so quickly

(which was, apparently, so effecti

bringing transparency to procure

data the subsequent mayor rolled

Te point is that governments ne

thinking in terms o platorms i

the long term open data is goin

And it needs to start thinking o

the primary consumer o the dat

being served on that platorm.

My main point is thi s: lets not pl

edges and merely define this chal

as one o usability. It is a much bi

problem than that. I we get it wr

then the big government vendors

inertia o bureaucracy win. I we

right, we could potentially save ta

millionswhile enabling a more

effective, and responsive governm

I try hard to be critica l advocate

data one who engages the risks

challenges posed by open data. I

perect and balancing these two g

advocacy and a critical v iew is

but I hope it is how we in the ope

movement see our role.

WHY IMPROVING DATA LITERACY

IS IMPORTANT

We must address the reality that data, even

while open to all, can be used to by the

most powerul to gain more power. Tere

are definitely cases where data can serve to

urther marginalize at-risk communities.

For example, we should never publish

publicly the locations o womens shelters

or worse, the list o amilies taking reuge

in them.

Tere are two things that g ive me some

hope in this space. Te first is that,

when it comes to open data, the a xis ocompetition among providers usually

centers on accessibility. For example,

the Socrataplatorm (a provider o open

data portals to government) invests heavily

in creating tools that make government

data accessible and usable to the broadest

possible audience. Tis is not a claim that

all communities are being engaged and

that a great deal more work cannot be

done, but there is a desire to show greater

use, which drives some data providers to

find ways to engage new communities.

Te second is that i we want to create a

data literate society and I think we do,

or reasons o good citizenship, social

justice, and economic competitiveness

we need the data first or people to learn

and play with. One o the best ways to help

people become data literate is to give them

more interesting data to play with. We did

not build libraries afer everyone knew

how to read, we built them beorehand

with the goal o having them as a place that

could acilitate learning and education.

Tere are also things that ofen depress

me. I struggle to think o technologies that

did not empower the empowered at least

Rob Kitchin, Proessor o

Geography at the National

University o Ireland at

Maynooth, recently wrote

a list o open data critiques or his blog

Programmable City. His post inspired

me to remind the open data community

particularly the advocates - o our

responsibility to take part in the debates

around open data, right now. We need to

engage in the discussions on a number o

topics i we want open data to reach its ull

potential or effecting positive change in

the world. Specifically, I will address two

critiques that Proessor Kitchin raised:

using data to empower the less powerul

and how to improve utility and usabilit y o

that data.

One of the main

reasons I have bee

an advocate of op

data is a desire to

citizens, nonprofit

and companies gaaccess to informat

that could help th

with their mission
http://eaves.ca/2011/10/21/the-state-of-open-data-2011/http://eaves.ca/2011/10/21/the-state-of-open-data-2011/http://eaves.ca/2011/10/21/the-state-of-open-data-2011/http://www.international.gc.ca/g8/open_data-donnees_ouvertes.aspx?lang=enghttp://eaves.ca/2011/07/12/its-the-icing-not-the-cake-key-lesson-on-open-data-for-governments/http://www.socrata.com/http://www.socrata.com/http://eaves.ca/2011/07/12/its-the-icing-not-the-cake-key-lesson-on-open-data-for-governments/http://www.international.gc.ca/g8/open_data-donnees_ouvertes.aspx?lang=enghttp://eaves.ca/2011/10/21/the-state-of-open-data-2011/http://eaves.ca/2011/10/21/the-state-of-open-data-2011/


6/21

OPEN INNOVATION WINTEOPEN INNOVATION WI NTER 2014


Open Datan Europe

Leaps Beyondransparency

y Erika Smith

crata VP of Worldwide Markets

he open data movement is thriving in many parts

f Europe, including well-established programs

the Netherlands, the United Kingdom, and

candinavian countries like Sweden and Norway.

ut, in response to recent questions like, What

ave we achieved and what can we do with all

f this data? the conversation has turned from

pen data for transparencys sake to open data to

rengthen the economy.

In act, today, the United Kingdom

hosts one o the most mature open data

programs in the world. From this vantage

point, UKs thought leaders are talking

more and more about the t remendous

economic and social potential o open

data, especially in machine-readable,

standardized ormats.

In late October 2013, Open Data Institute

(ODI) ounders Sir im Berners-Lee

and Sir Nigel Shadbolt announced the

creation o a global network o nodes

where programs aligned with ODIs

principles o openness and economic

innovation would be established. As

part o the announcement, they were

quoted as saying, Te best way that open

becomes the new deault is demand:

rom businesses and organizations, both

public and private, rom individuals and

corporations.

Berners-Lee and Shadbolt assume that

open data will be put to use supporting

new businesses. Te ODI, a not-or-profit

organization that is just over one year old,

has so ar helped set up more than a dozen

open data-based startup companies in

the UK.

Socrata became an official partner o

ODI in the all o 2013. ODI lead Gavin

Starks says, We love Socratas view o

open data as uel or new businesses. We

delivery within Europe, Socrata r

partnered with Microsof to mak

Socrata Open Data Portal availab

Microsof Azure platorm. Azure

number o data centers on the co

Socrata Senior Site Reliability En

Paul Paradise noted, As a growi

company, we scale internationall

better by partnering with an esta

trusted, international data-hostin

platorm like Microsof Azure.

MORE BUSINESS IN THE FU

While transparency is still a centgoal or the open data movement

traditionally transparency-ocuse

organizations, like the Open Gov

Partnership (OGP), acknowledge

economic power o open data. Be

its Open Government Partnershi

Summit in London in October, th

announcedthat 37 countries had

commitments to, among many e

radically open up government d

boost entrepreneurship, growth a

accountability.

Open data is uel or innovation

world will never be the same, now

we can so easily take public data

it to work to make peoples lives b

says Merritt. Businesses will ine

put the public data shared to wor

economy. Socratas job is to make

it is easily accessed and in useul

want to collaborate with them to promote

the creation o new products and services

based on open data.

Socrata CEO Kevin Merritt presented

on a panel at the Open Data Summit

with the title, Open or Business Te

Commercial Impact o Open Data.

Merritt commented on the growing

interest in economic growth through

open data, saying, In the UK, and all o

Europe, people are asking, Open data, so

what? And, the ODI and other incubators

o new businesses help demonstrate that

innovation, new jobs, and better services

are possible when non-private data is easy

to access.

FASTER MOVEMENT OF DATA TO

SUPPORT INNOVATION

With the growing list o businesses relying

on government open data, the speed

and requency with which governments

deliver that data has become more

crucial. For example, Spend Networkis a

startup supported by ODI that uses open

spending data to create new insights or

Government and its suppliers. Spend

Networks services are only as useul and

accurate as the data it gathers.

Anticipating the greater demands on

datacenters across Europe and, in an effort

to improve the speed o its existing data

UK AIMS FOR ECONOMIC

GROWTH

In the all o 2010, when the UK launched

a searchable database o business plans

or government departments, Prime

Minister David Cameron was quoted as

saying, We want to be the most open

and transparent government in the

world. Since then, getting non-sensitive,

non-personal data online has become

something British citizens expect.

With the growing list of businesses relying ongovernment open data, the speed and frequency

with which governments deliver that data has

become more crucial.
http://www.theguardian.com/technology/2013/oct/29/web-inventor-open-data-institute-new-global-networkhttp://www.theguardian.com/technology/2013/oct/29/web-inventor-open-data-institute-new-global-networkhttp://www.theguardian.com/technology/2013/oct/29/web-inventor-open-data-institute-new-global-networkhttp://www.opengovpartnership.org/ogp-summit-statementhttp://www.spendnetwork.com/index/http://www.spendnetwork.com/index/http://www.opengovpartnership.org/ogp-summit-statementhttp://www.theguardian.com/technology/2013/oct/29/web-inventor-open-data-institute-new-global-networkhttp://www.theguardian.com/technology/2013/oct/29/web-inventor-open-data-institute-new-global-network


7/21

OPEN INNOVATION WINTEOPEN INNOVATION WINT ER 2014


Making the wo-Way Street of OpenData a Reality

y Chris Whong

ocrata Data Solutions Architect

Department o Inormation echnology

and elecommunications (provider o

the ootprints data) and are launching

a community project to systematically

import the citys treasure trove o

building data into OSM. Te project

was announcedin late September.

LEGIONS OF GEOSPATIAL

ANALYSTS

So whats in it or the city? Te potential

or updates that only a system like OSM

can provide. I OSM users see something

wrong, they can fix it. Maybe a buildingootprint is misaligned, or maybe the

building doesnt exist anymore. Te city

receives a daily update o changes to

the building data, and can review those

changes. I they are legitimate, DoI

can use these changes to gu ide updates to

their own master database, making it more

accurate and up to date. Its as i the cit y

has legions o geospatial analysts quality-

checking their data and sending updates!

Alex Barth, Data Lead at the web mapping

company MapBox and OSM Advocate, has

been a key organizer o the N YC OSM

collaboration, and has been working on

the idea since early 2012. Te data was

already publicly available back then, but

carried a license that was incompatible

with OSM. NYCs Open Data Law, passed

in March 2012, cleared up the licensing

issue and provided the way orward.

o Barth, the project is not simply about

buildings, but is an experiment and

learning experience about the impact o

community-driven projects like OSM. Its

a data improvement effort that has positive

side effects and really lets us grow t he

community. Te longer-term vision goes

beyond OSM or even geodata, and hopes

to redefine open data publishing: Tis is

about an open data commons, a single space

in which government and citizens interact.

Te first gathering o volunteer mappers

to work through the monumental task o

importing the citys data met in October.

Liz Barry, another leader in the NYC-OSM

collaboration, hosted the meeting at the

offices o the Public Lab in Brooklyn, and 22

community members showed up to help. Te

data was broken down into election districts,

and the team set out validating ootprints

against aerial imagery, checking geometries,

and correcting overlapping polygons.Existing attribute data in OSM could also

be merged with better polygons rom the

city data. Barry said the workflow is still

being vetted, and is not quite ready or ul l-

scale deployment. Te idea is that once the

workflow is perected, updates wont require a

physical meetup. Volunteer OSM users will be

able to import a chunk o t he citys building

ootprints whenever and wherever they can.

Te real un will begin when large amounts

o the data have been successully imported

and the city can report back about the volume

and utility o OSM-contributed changes. In

many cases, there may be more inormation

about a building in OSM than the city

maintains on its own, meaning the two-

way street o open data may not flow evenly

in both directions. Te OSM community

has ound a partner in DoI, and this

experiment will serve as an early model o

the power o citizens and activists to improve

government data.

OpenStreepMap allows users to view multiple regions,

from a birds-eye view of Europe to a zoomed in view of the

streets of Manhattan.

and can be just as easily undone. Users

may choose to update the map or many

reasons, rom just knowing more about

conditions on the ground than anyone

else, to improving the map or a specific

project such as an app.

What i a user needed some building

outlines that OSM didnt have yet? Tat

user could manually trace over the satell ite

imagery, pointing and clicking lots o

custom polygons into existence. But what

i they needed a whole town? What i

they needed New York City? Tey do, and

NYC has an open dataset or t hat. Te

citys detailed GIS database o building

outlines and point data is reely available

or download at data.cityonewyork.us.

While it will still take human effort to

import and veriy data or over a million

buildings, creating them manually would

be an unathomable and time-consuming

process. Leaders in the mapping tech

community have partnered with N YCs

Open data has by and

large been a one-way

conversation. Governments

produce public data and

ake it reely available, while citizens,

urnalists, researchers, and hackers

nsume it in whatever ways suit them.

ut, having more eyes on the data once it

released may be able to provide value

ck to the government, turning users

the data into a source o new data and

ality control. Tis is the experiment in

o-way open data that New York City is

oneering with OpenStreetMap.

penStreetMap (OSM) is the Wikipedia

Maps, where anyone can contribute

anges. (Yes, i theres a ootpath or bike

ail near your house that doesnt show up

mainstream web maps, you can literally

raw it into OSM, name it, and connect

to existing roads.) Like Wikipedia,

anges to the map are subject to quality

ntrol by the rest o the community,
https://www.mapbox.com/blog/nyc-and-openstreetmap-cooperating-through-open-data/http://www.nyc.gov/html/doitt/html/open/local_law_11_2012.shtmlhttp://www.nyc.gov/html/doitt/html/open/local_law_11_2012.shtmlhttp://www.nyc.gov/html/doitt/html/open/local_law_11_2012.shtmlhttps://data.cityofnewyork.us/http://www.openstreetmap.org/#map=5/51.500/-0.100http://www.openstreetmap.org/#map=5/51.500/-0.100https://data.cityofnewyork.us/http://www.nyc.gov/html/doitt/html/open/local_law_11_2012.shtmlhttp://www.nyc.gov/html/doitt/html/open/local_law_11_2012.shtmlhttps://www.mapbox.com/blog/nyc-and-openstreetmap-cooperating-through-open-data/


8/21

OPEN INNOVATION WINTEOPEN INNOVATION WIN TER 2014


Open DataHad Better BeData-Driven

y Thomas Levine

Dada Artist

Weve been opening government data for some

me now. Without realizing it, weve amassed

ome rather rich data about how people publish

nd consume open data. With this data about

he use of open data, its possible to use data-

acked benchmarks, projections, and decisions in

ur open data strategies, and this can make our

pproaches to open data more systematic, logical,

nd obtainable.

urious about what the data behind open data can

each us? Read on.

Every dataset has basic, easy-to-extract

properties like the number o records

it contains, and the size it takes up on a

hard drive. We can come up with more

complicated inormation too, like the

number o missing values and the date o

the oldest record.

In addition, when a dataset is published

online, it creates metadata about t he

dataset and its use. Metadata provides

details like when it was first published,

who uploaded it, and how many times

people have downloaded it since then.

By collecting some o these simple

properties and metadata rom each

dataset, its possible to create a dataset

about the publication and use o other

datasets what I call a super-dataset.

DATASETS AS DATA POINTS

A super-dataset compiles inormation

about each dataset when it was

published, what kind o inormation it

contains, etc. into a single row, creating a

record (i.e. a data point) about that dataset.

With this setup, you can perorm all

kinds o unctions and analyses about the

publication and use o datasets: you can

look at how many datasets are on different

catalogs, how data is queried and reported,

what licensesdatasets have, and how many

there are o a certain category. Here are

some examples o analyses I have done.

Number of Datasets:I compared the

number o datasets on some various

government data portals that r un Socratas

open data platorm sofware. I ound that

New York City, Chicago, and the state o

Oregon offer the most datasets.

Licensing:I also used my super-dataset o

properties and metadata to look into this

question: What licenses do people apply

to their open data? I discovered that many

portals avor public domain or some orm

o open license, but most list no license at all.

Groupings of Datasets:By looking

at similarities in the titles, schemas,

and other metadata o datasets,

I determined what sorts o data

government organizations were putting on

their open data portal s and how different

datasets were related to each other.

METADATA AS DATA

Even though the word metadata contains

the word data, people dont typically

think o metadata as something to analyze.

Metadata is ofen invisible. I we could

see it, it might just look like background

inormation about web pages and their

contents. For example, Socratas open

data platorm displays metadata about

each dataset, such as community rating,

number o visits, number o downloads, etc.

When you use metadata to populate

records in a super-dataset, that metadata

becomes data that we can see, analyze, and

learn rom.

WHY THIS MATTERS

Tere are numerous case studies, by

organizations such as Code or America,

CKAN, Open Data Institute, about how

to open up government data, but these are

based strongly on personal experiences,

not precise, quantitative statistics. Tere

is comparatively littlework that uses data

to produce guidelines or best pract ices or

the opening o data.

Open data experts ofen talk about making

use o open data and building products

rom it, but a super-dataset accomplishes

something different. It reveals inormation

A visualization of the

datasets available on

data.cityofnewyork.us by

category, showing both

raw datasets, as well as

derivative charts, maps, and

filtered views.

about how people publish and us

data. It paints t he bigger picture

going on, whats working, and wh

Imagine what we can do with thi

backed understanding o our ope

We can find out what has been do

and what has worked, allowing o

publishers to plan their releases m

strategically. We can measure rel

strategies against solid, quantitat

statistics to make sure that they a

helping us achieve our goals. We

even use these findings to build p

that help people interact with op

Tese are just some o the possib

emerge when we think o data ca

datasets o datasets.

DATA, DATA, EVERYWHERE

We have data about open data, but it

always starts out in ormats that are not

convenient or this sort o study o open

data. So, we first need to turn this raw data

into a dataset. In my mind, a dataset is a

collection o things, with some consistent

properties describing each o the things.

(We ofen represent datasets as tables.)

Were going to treat each dataset a s a thing

inside our collection o many datasets.
http://thomaslevine.com/!/socrata-deduplicate/http://thomaslevine.com/!/socrata-deduplicate/http://thomaslevine.com/!/socrata-deduplicate/http://www.chriswhong.com/nycopendata/http://thomaslevine.com/!/open-data-licensing/http://thomaslevine.com/!/missouri-data-licensing/http://beyondtransparency.org/http://ckan.org/case-studies/http://theodi.org/case-studieshttp://thomaslevine.com/open-data/http://thomaslevine.com/open-data/http://theodi.org/case-studieshttp://ckan.org/case-studies/http://beyondtransparency.org/http://thomaslevine.com/!/missouri-data-licensing/http://thomaslevine.com/!/open-data-licensing/http://www.chriswhong.com/nycopendata/http://thomaslevine.com/!/socrata-deduplicate/http://thomaslevine.com/!/socrata-deduplicate/


9/21


FixingtheFederal IT

ProcurementProcess

By Kevin Merritt

Socrata Founder and CEO

Sometimes it takes a disasteror people to recognize the

need or change. Te botched

rollout o HealthCare.gov

something I would actually describe as a

systemic catastropheprovides a historic

opportunity to overhaul the ederal

government technology procurement

system, which is so clearly broken.

DIAGNOSING THE PROBLEM

Government technology projects have long

been characterized by significant delays,

mammoth cost overruns, and sofware

products that routinely underperorm.

But, beore the HealthCare.gov fiasco,

relatively ew people were aware o the

magnitude o the problems plaguing

the ederal I purchasing system. Now

that the story has made ront-page news,

everyone wants to know what went wrong.

Just as any good sofware engineer looks

to uncover the root causes o bugs in a

computer program, it is important to

meticulously diagnose the underlying

causes that continue to produce I

disasters at the ederal level.

SELLING TO THE FEDERAL

GOVERNMENT IS A BYZAN

PROCESS

Federal I procurement is overly

and opaque. Te process is ull o

rules and oppressive costs that di

many technology providers rom

the market. Te Federal Inorma

Security Management Act (FISM

a prime example. Tis legislation

enacted at a time when applicatio

almost always hosted on premise

waterall methodology (not ag

the predominant approach to sof

development. imes have change

have learned a lot since then.

Yet, because FISMA is still in pla

vendors looking to bid on ederal

technology projects must comply

with outmoded inormation secu

requirements, like using tape bac

replicate files offsite. Tis is comp

incongruent with modern I bes

practices. More importantly, the

data loss in a single environment

tape backups is statistically much

than in a geo-redundant environ

where data is replicated across m

servers.


10/21


INNOVATORS NEED NOT APPLY

Te net result is that many nimble,

innovative companies simply orego doing

business with the ederal government,

concluding that it is just too costly. Over

the years, this has created an environment

that nurtures mediocrity. oday, there

are a whole class o enterprise companies

that specialize in navigating the bloated

and broken procurement system. Most

o these businesses make their money by

selling consulting hours, hiring dozens

o subcontractors to do the work, and

overseeing the development o expensive,

customized technology solutions that

ofen do not perorm as advertised.

We admire startups because their core

competency is innovation. Tey establish

traction and momentum because they

invent novel solutions to old problems,

develop new technologies, or come up with

creative new ways o doing business. Te

irony in the ederal I space, however, is

that these behemoth systems integrators

(SIs) are wi nning business not because

THE GSA JUGGERNAUT AND

OTHER BURDENS

o bid on ederal government projects,

businesses must first be listed on the

General Services Administration (GSA)

schedule. Maintaining this listing is

practically a ull-time job, requiring

continuous updates, renewals, and, o

course, piles o paperwork. o keep up,

many smaller businesses are compelled

to hire consultants just to clear all o the

administrative hurdles.

In a very real sense, GSA requirements

operate like an extra tax on businesses,

siphoning profits rom sales to ederal

government agencies. In addition to this

tax, GSA vendors must offer their lowest

commercial price to the government, even

i the market va lue or their products is

significantly higher. Tis squeezes profits

urther, disproportionately affecting the

ability o smaller businesses to compete.

As i this were not burdensome enough,

any vendor that wants to sell to the ederal

government must be able to decipher the

Federal Acquisition Regulation (FAR),

the principal set o rules that govern the

ederal procurement process. Te FAR

is every bit as complex and unwieldy as

the ederal tax code, containing rules as

picayune as the width o the margins or

request or proposal (RFP) responses.

And these rules change constantly. Few

emerging technology firms have the

wherewithal or the patience to wade into

that kind o logistical labyrinth.

they have innovated. Rather, it is just the

opposite. Teir sole core competency

has nothing to do with technology or

innovation at all; their only real skill is

that they have perected the art o selling

to the government!

CARVING A PATH FORWARD

Government leaders need to introduce

a serious-minded reorm agenda i any

real progress is to be made in revamping

the ederal I procurement process. Tis

means directly applying the open data

ideals that the Obama administration has

generally espoused to the specifics o the

I purchasing process.

First and oremost, there needs to be ull

transparency into ederal contracts. All o

the details or each contractincluding

the amounts paid and the names o the

people who authorized each payment

need to be online in machine readable,

ully sortable, and searchable ormats.

Te ederal government also needs to

make it simple or the public to ollow

the flow o money and influence between

ederal government agencies requesting

vendor support and the contractors

bidding on projects. As a country, we

have made great strides in recent years

in increasing visibility into campaign

contributions. o weed out corruption

and improve efficiency, senior government

leaders need to bring that same level o

effort and scrutiny to the procurement

process.

Further, policy makers should en

providers o cloud and SaaS-base

technology solutions to discuss w

to modernize the ederal governm

inormation security requiremen

the wake o the a ilure o HealthC

there is an exciting opportun ity t

the government officials who cer

security mandates about modern

best practices.

WHO IS READY TO TAKE U

MANTLE OF PROCUREMEN

REFORM?

Te once-stodgy subject o eder

procurement has gone viral. In tu

this awareness has produced broa

consensus around the need or m

changea precious commodity t

should not be wasted. Right here

now, I am putting orth a clarion

civic leaders on both sides o the

the time is ripe to take up the ma

real procurement reorm. By unl

the ingenuity o sofware compan

that have previously been pushed

sidelines, you can help launch a n

technology innovation that impr

lives o Americans or generation

to come.

Government

leaders need

to introduce a

serious-minded

reform agenda if

any real progress

is to be made

in revamping

the federal IT

procurement

process.

We admire

tartups because

heir core

ompetency is

nnovation. They

stablish traction

nd momentum

ecause they

nvent novel

olutions to old

roblems, develop

new technologies,

r come up with

reative new ways

f doing business.


11/21

I was very impressed to see the level of

civic innovation and ownership from the

Code Michigan hackathon participants.

Many coders I spoke to had never been to

a hackathon before and were very proud

to share how their products were going to

improve their city and many others.

Ian Kalin

Director of Open Data for Socrata

at Code Michigan

It is so amazing to see in a

city that has such a diverse

set of problems that we can

unite to create a diverse set

of solutions.

Beth Blauer

Director of GovStat for Socrata

at Hack for Change Baltimore

This is an amazing display of

the power of civic hacking and

what we can all do together.

Hannah Young

Program Coordinator of the National

CfA Brigade on the National Day of

Civic Hacking

For those of us who

work in government,

this feels like our own

national holiday.

Shannon Spanhake

City of San Francisco

Deputy Innovation Officer

on the National Day of

Civic Hacking

AWARDS

EVENTS


SOCRATA2013 HIGHLIGHTS

13 was a big year for the open data movement. It was filled with

ckathons, datapaloozas, and hack nights. Leaders and innovators in the

ovement won awards and recognition. We here at Socrata were thrilled

be a part of it all. Here is a look back at some of th e top U.S. events

attended, awards our customers won, and favorite things we heard

ng the way.

Awards

Bloomberg Philanthropies

Mayors Challenge (Chicago)

Data Liberation Award

(NYC Dept of Health)

2013 Digital Counties Survey

Award (Montgomery County,

Snohomish County, King County)

State Program Innovation Award

(Oregon)

2013 Web 2.0 Award (Raleigh, NC)

2013 Best of the Web Winner

(Alameda County)

2013 Digital Government

Achievement Award Winner

(Alameda County)

2013 CSAC Merit Award

(Alameda County)

2013 Achievement of Excellencein Procurement Award

(Alameda County)

2013 Digital Cities Winners

(Boston, Seattle, Austin, Chicago,

Baltimore, Raleigh)

Events

ATX Civic Hackathon II

(Austin, TX)

Code Michigan

Code Across America

(Philadelphia)

Code Across America (

Vegas Hack

San Francisco Housing

Open Science (Mountai

EcoHackSF (San Francisc

Urban Data Challenge

Hackathon (San Francis

Data P2PU Course Spri

(Mountain View)

Code for Seattle

NC Data Jam (Raleigh)

Hack 4 Change (SeattleAlameda County Apps

NYC Data Week

SpoCode (Spokane)

Apps4Halifax

API Craft Conference(

API Strategies and Pra

Conference (San Francis

Code for America Sum

(San Francisco)

Code for L.A.

NC Datapaolooza (Rale

Hawaii Digital Governm

Summit

SXSW Eco Hackathon 2

(Austin)

Atlanta Govathon

Code for Oakland

International Open Da

(Washington, D.C., New

Seattle, Atlanta)


12/21

OPEN INNOVATION WINTEOPEN INNOVATION W INTER 2014

DESIGNINGTHE FUTURE:CODE FORAMERICA

MAKES ANIMPACTBy Alida Moore

Socrata Content Strategist

Before 2011, parents in Boston, MA struggled to figure out

which schools their children were eligible to attend. The

confusing process, involving a 28-page manual and a lottery

system, was a source of contention and, at times, violent

conflict among parents across the city. A turning point came

when Code for America (CfA), a non-profit organization

dedicated to using data and technology to improve lives,

turned their team on the problem.


13/21


Lauren Reid, Senior Public

Affairs Manager at CA,

explains how they built a

solution. Te CA ellows in

Boston developed an app where parents

could enter a ew simple data points, such

as address and age o child, and find out

quickly which schools were available as

options or their child, Reid says. Te app

was hugely successul. Te City o Boston

told us that this app, developed in just

three short months, would have taken the

city more than two years and two million

dollars to create -- had it gone through the

standard procurement process. ogether

with the City, were resolving a decades-

old problem using modern technology and

open data, and changing the conversation

between parents and the schools system

rom one o contention to one that is

positive and productive.

Code or Americawas ounded in 2009

by Jennier Pahlka, the current Deputy

Chie echnology Officer or the United

States. Te organization seeks to find

out what can happen when smart, savvy

researchers, developers, and designers

are deployed to cities across the nation to

help local governments unlock their data

to create solutions and deliver services to

their citizens.

THE FORMULA

Te Code or America ormula is simple.

Developers and designers commit a year o

their careers to helping a city government

in need o problem solving. In exchange

or a modest stipend, these participants,

called ellows, live in their assigned

cities, and use their skills to help move

government orward to meet the needs

o 21st century citizens.

CFA PROGRAMS

Code or America has developed our

programs to help urther their mission.

Te Fellowship:Code or Americas

flagship program, in which developers

and designers are matched with local

governments to transorm data into usable

orms or public improvement.

Te Brigade:Civic-minded volunteers

come together to orm brigades, bringing

grassroots efforts to data use and

transparency.

Te Accelerator:Provides financial and

logistical support to civic tech startups,

rom a $25K grant to mentorship and

networking opportunities.

Te Peer Network:A learning network or

government innovators who want to work

with other local governments to harness

the power o open data in their cities.

Ben Berkowitz, CEO and

Co-Founder of SeeClickFix,

presents at the Code for

America Summit 2011.

Top: Andrew McLaughlin,

self-proclaimed nerd,

Sunlight Foundation board

member, and CEO of Digg

and Instapaper, presents

at the Code for America

Summit 2011. Bottom: Laura

Meixell, 2013 Code for

America Fellow in

San Francisco.

FELLOWS IN ACTION

Matthew Hampel, 2012 CA Fellow

(Detroit), spoke to Socrata about his

experience as a ellow. One o his projects

or the city o Detroit was to create a

web and mobile app that would update

commuters about bus schedules. Hampel

told us about the app, called extMyBus.

Originally, bus data was tracked

manually through an ancient interace

and it wasnt available to public, Hampel

explained. So the city provided us with

access to their servers and we exposed the

data. With that, we built a text messaging

app that helps people figure out when their

buses are arriving. extMyBus has proven

quite popular. As o December 2013, the

app has served over 1.1 million users.

CA ellows have contributed 62 apps

across America so ar, rom helping

citizens navigate the public school system

to receiving text alerts when services, like

ood stamps, are about to expire.

We are on

the cuttin

edge of G

2.0 and c

hacking a

geeking.
http://codeforamerica.org/http://codeforamerica.org/


14/21

OPEN INNOVATION WINTEOPEN INNOVATION WINTER 2014

Foley agrees. Your civic technical

ecosystem should overlap with

entrepreneurship. As part o that

ecosystem, our brigade influences the civic

I department by promoting open data.

We are on the cutting edge o Gov 2.0 and

civic hacking and geeking , Foley says.

You can expect to see more brigades pop

up in 2014, allowing or crossover and

partnership. As Foley explains, Your city

boundaries shouldnt limit innovation.

I, or example, you have an app t hat

catalogues the greenways in your city,

it should continue beyond city limits,

says Foley.

A LOOK FORWARD

In 2014, Code or America plans to add 31

new ellows to the ellowship program.

For people like Matt Hampel, who is

passionate about harnessing open data or

civic improvement, Code or America has

been a vehicle or positive change. Data

helps you design the uture you want to

see, Hampel explains. Data gives you the

power to make inormed decisions, and

inormed decisions are better decisions.

Code for Americaby Numbers

2011

19fellows deployed to Boston, Philadelphia, and Seattle

2012

26fellows sent to eight cities: Austin, Chicago, Detroit, HonMacon, New Orleans, Philadelphia, and Santa Cruz2013

27fellows embedded within nine communities: Kansas City

Kansas City, Mo.; Las Vegas; Louisville, Ken.; New York;

Calif.; San Francisco; San Mateo County, Calif.; South Be

Summit County, Ohio

2014

31fellows will join the Code for America program. Cities toannounced.

62apps created by CfA fellows over the years

p: Volunteers gather at

rigade meeting in San

ancisco. Bottom left: Former

y of Seattle Chief Technology

ficer Bill Schrier presents at

e Code for America Summit

11. Bottom Right: CfA will send

ore fellow to more cities than

er before in 2014.

BRIGADES IN ACTION

Some o Code or Americas most

passionate, committed volunteers live in

Raleigh, NC. Jason Hibbets and Chad

Foley are two o the Raleigh brigades our

co-captains. Each man volunteers hours o

time each month, outside o their day jobs,

to the brigade.

Being part o the Raleigh brigade allows

us to have an impact on loca l government

and our community, says Hibbets. He

believes that government transparency

in data is paramount to innovation.

Open data is the oundation o civic

entrepreneurship, Hibbets explain s.

Open data belongs to the people. Put

data in the right hands and apps that can

improve the daily lie o citizens can

be created.

Data gives you

the power to

make informed

decisions,

and informed

decisions are

better decisions.


15/21


DATADRIVENGOVERNMENTIN ACTIONBy Beth Blauer

In the previous issue of Open Innovation, I

talked about the importance of fact-based

decision-making and how gut decisions

can be expensive and dangerous. For

governments to play a leading role in the

data revolution, performance measurement

and successful delivery are mandatory. This

truth is something I am passionate about.

My passion resonates from my experience

managing StateStat and the Delivery Unit in

Maryland and is affirmed in conversations I

have had with government leaders since.

Beth Blauer is a

leading expert

in implementing

stat programs in

government and is

Director of GovStat

at Socrata.

Fortunately, the open data

movement has resulted in

tools or data-driven decision-

making, perormance, and

delivery. Socratas product, GovStat,

helps local governments become more

transparent, engage citizens, and measure

progress against initiatives and goals. It

also allows stakeholders to collaborate

throughout the entire process on one

common platorm.

In Beyond Transparency: Open Data and

the Future of Civic Innovation1, I stressed

the idea that its not a matter o i data-

driven government can create the best

solutions to societys problems; its a matter

o how soon governments will embrace the

idea and reap the benefits. Using a tool li ke

GovStat enables governments to collect

and update data across departments,

build beautiul data visualizations, and

create both internal and citizen-acing

dashboards to track progress.

I would like to highlight three governmentorganizations that have chosen to use

GovStat as their primary perormance

measurement solution. I will share their

specific challenges in moving toward data

1 Code for America Press (2013)

transparency, why they chose GovStat as

their solution, and how each organization

plans to use GovStat to i ncrease efficiency

and accountability to their constituents.

TELLING THE STORIES

THAT MATTER:

COOK COUNTY, IL

In 2011, Cook Countybegan a program

to create a strategic plan around the use

o open data or decision-making and

communications. Andrew Schwarm, Chie

Perormance Officer o Cook County, was

the project head tasked with finding a

perormance measurement tool.

Te county was publishing our quarterly

reports in a PDF on our website, which

was ar rom best practice, Schwarm says.

Using a tool lik

GovStat enable

governments

to collect and

update data ac

departments, b

beautiful data

visualizations, create both int

and citizen-fac

dashboards to

progress.
http://beyondtransparency.org/http://beyondtransparency.org/http://beyondtransparency.org/https://performance.cookcountyil.gov/https://performance.cookcountyil.gov/http://beyondtransparency.org/http://beyondtransparency.org/


16/21


We wanted to transorm our external

rormance reporting rom a static

DF to a dynamic open data web portal.

ovStat jumped out as the product that

ade the most sense or us.

n early adopter o the GovStat platorm,

ook County decided on a two-phase

plementation process. First, the County

plicated the PDF data and created

ports or each department. GovStat

owed us to take the data already

thered and put it on a more flexible,

er-riendly, open, and transparent

atorm, Schwarm says.

nce the data was updated and made

ailable to the entire organization,

ook County entered phase two o

plementation: using data to drive

cisions. [Data-driven decision-making]

now part o our culture a nd the way we

business, explains Schwarm. Further,

accurate data allows Cook County to tell

accurate stories. One goal the county

tracks is lowering sick leave or county

staff; Cook County hopes to reduce

employee sick leave to 5.2 hours per month

beore December 2013. On October 25,

2013, Cook County published a GovStat

report on the progress toward this goal,

declaring it on track.

Schwarm is excited to track other county

initiatives, including public saety and

healthcare goals. We plan to roll out one

goal per month and continue to report on

its progress, Schwarm says. He continues,

Te use o good, timely, accurate data,

especially or a government acing tough

fiscal situations, allows us to make

decisions and prioritize at a high level.

Citizens of Cook County can view progress on goals, like reducing

the amount of sick leave taken by government employees, using

GovStats citizen dashboard.

This chart, updated daily, shows citizens in Kansas City the percentage

of customers who have been satisfied with the citys response to water

main break service requests.

DYNAMIC REPORTING

AND COLLABORATION:

KANSAS CITY, MO

Kansas City, MOhas been a leader in

government transparency and data-

driven government. Since taking office,

Mayor James has convened regular

KCStat meetings with his senior team a nd

holds them accountable to the goals and

strategies that are most important or the

citizens o Kansas City. Te technology

behind this program began as their

homebuilt perormance measurement

platorm, KCStat. A year later, the City

added Socratas open data portal.

As the program grew, the team ound they

were limited in their reporting abilities. Sowhen Senior Perormance Analysts Kate

Bender and Julie Steenson began looking

or a perormance management solution to

enhance KCStat, they knew GovStat would

meet those needs.

O all GovStats eatures, the KCStat team

is particularly excited about the tools

drag-and-drop reporting capabilities.

We already were on board with the

concept o telling our story through data

and improving government perormance,

Bender says. Te only thing missing was

the ability to create dynamic reports and

data visualizations, she continues.

Kansas City hopes to improve how they

communicate their results to the larger

community. Te Kansas City council has

always been dedicated to civic engagement.Each monthly council meeting is filmed,

televised, and shared online, which

attracts serious stakeholders. Still, the

city wanted a way to communica

every citizen, quickly and easily.

ound the GovStat public-acing,

citizen, dashboard most useul

GovStat dashboard is a way to en

every stakeholder, says Steenson

storytelling device that makes th

more accessible.

Kansas City launched its citizen d

in early October and looks orwa

using the tool to make progress to

the citys initiatives. One big step

was having the city council adop

set o strategic priorities, which

the backbone o our da shboard,

Steenson. Te next step was assi

measurements to those prioritiescontinues. Te council made a p

statement about their priorities a

then adopted specific measures t

The use of good,

timely, accurate

data, especially

for a government

facing tough fiscal

situations, allows us

to make decisions

and prioritize at a

high level.
https://kcstat.kcmo.org/https://kcstat.kcmo.org/


17/21


progress. Tese metrics or tracking

progress will be the advantage o theGovStat dashboard as we build it out.

Kansas City plans to roll out a new goal

every month over the next six months,

into 2014. Residents can ollow each goal s

progress on the citizen dashboard. In the

meantime, Bender is excited to see how the

tool helps improve efficiency. Dynamic

reporting saves so much time, she says.

Its great to work in a system designed

around government use.

TAKING PERFORMANCE

MANAGEMENT TO THE NEXT

LEVEL: SAN MATEO COUNTY, CA

Te San Mateo Countygovernment

is known or its deep commitment to

transparency. Te Countys Shared

Vision 2025, a comprehensive community

planning process designed to get direct

input rom citizens, is a sterling ex ample o

this open, collaborative governing style. So

when county residents approved a hal-

cent sales tax increase in 2012, the Board

o Supervisors promised that residents

would be able to see how their tax dollars

were being spent.

SMC Perormance eatures a dashboard

with a series o interactive tiles, each tile

acting as a hub or the goals associated

with a specific Measure A initiative. By

clicking on the tile, users can explore

perormance measures, review raw data,

read the actual proposal and, in some

cases, look at charts, graphs, and maps

created with the data. One o my avorite

things about GovStat is how easy it is to

create maps and other visualizations o

the data we collect, says Shanna Collins,

a Budget Analyst in San Mateo Countys

Office o Budget and Perormance. Tese

visualizations help us identiy where the

greatest needs are throughout the county,

so we can make budget choices that are

based on objective data.

In the coming months, San Mateo County

will use GovStat to track the perormance

o the original programs unded by

Measure A to monitor progress in key

areas. In addition, the County is planning

to roll out a new dashboard to ollow the

nine community impact goals that make

up its Shared Vision 2025 in early 2014.

GovStat has been at the heart o our move

toward true data-driven management,

says Farrales. And it gives us a platorm

or involving the community and our

employees in decision-making, which is

central to our mission.

Executives in the County Managers Office

viewed the approval o the tax known

locally as Measure A as an opportunity

to take the countys transparent, data-

driven approach to budgeting to the

next level. Te Measure A sales tax will

generate about $65 million in revenue

each year over the next 10 years, says

Reyna Farrales, Deputy County Manager

or San Mateo County. We have a duty

to show how the services unded by the

tax contribute to specific, measurable

goals and how those results ultimately fit

together with the priorities in our Shared

Vision 2025.

When voters approved the sales tax

increase, county leaders began theirsearch or a technology solution to keep

the community inormed about progress

in the coming years. We were aware o

the success o the StateStat program in

Maryland and were really impressed with

the perormance dashboard they were

using, says Farrales. We discovered that

Maryland, along with a number o other

cities and states, were all using the Socrata

platorm. Our newly appointed Chie

Innovation Officer (CIO) Jon Walton came

rom San Francisco and had experience

with Socrata, so he a nd his staff were able

to help us ramp up our Socrata-powered

open data portal and move into GovStat

right away.

Afer implementing GovStat, the County

Managers Office used the system to help

departments define goals and metrics

or their respective Measure A unding

proposals. In September 2013, San

Mateo County became the first county

government in the U.S. to deploy a public-

acing GovStat site. Te launch o SMC

Perormance coincided with the Board

o Supervisors approval o 22 projects

totaling more than $50 million.

vStats mapping

pabilities help

tablish patterns.

e map above shows

water leak, water

eter, and hydrant

pair requests open in

nsas City.

This pie chart shows the number of emergency response vehicles by

category for San Mateo.

GOALS IN ACTION

Each o these organizations offer

compelling story o what is possib

by putting the principles o data-

driven decision-making, perorm

measurement, and delivery into a

Greater adoption o these practic

a trend; its a undamental shif i

way governments around the wor

embracing their mission.

n September 2013,

an Mateo County

ecame the first

ounty government

n the U.S. to deploy

public-facing

ovStat site.
https://performance.smcgov.org/https://performance.smcgov.org/


18/21


CODELESCENCE:ENGINEERINGCOMES OF AGE

y Patrick Hasseries

aff Writer

ocrata is maturing, and it is most evident

hrough the evolution of its engineeringpast,

resent, and future.

Whats really exciting about Socrata right now

that were transitioning from a startup in

he true sense of the word to a company,

ays Steve Greenberg, one of Socratas newest

ngineering Program Managers.

When Socrata was

originally ounded,

it began with a small

team o scrappy

sofware engineers. eam structure

was simple, and early team members

quickly developed an office culture that

paralleled the products they were building:

open, engaging, and collaborative. Tey

encouraged ree discussion, shared a love

or challenges, and held events planned

or spontaneous that built cama raderie.

Since the earliest days o the company,

dozens o engineers have joined

Socrata, rom part-time high school

interns to veteran programmers with

decades o education and experience.Te engineering team has grown into

an engineering department, and the

original inrastructure must evolve to

accommodate more members and

bigger projects.

DIVIDING TEAMS WITHOUT

CAUSING DIVISION

When companies expand, they must be

careul not to let growth impede progress

or dull internal culture. So how will

Socratas engineering team restructure

itsel without sacrificing its liveliness?

Were emulat ing Spotiys ribes, Guilds,

Squads model, says Jerome Gagner,

Socrata Director o Engineering. Trough

it well keep the culture that we have as

well as develop engineers careers and

individual disciplines.

Based on Spotiys model or structuring

internal teams, Socratas engineers are

reorganizing into three types o groups.

All engineers dedicated to a specific

service or product orm large teams

called ribes. All engineers who share a

discipline (ront-end development, back-

end development, user interace design,

etc.) collaborate through inter-tribal

groups called Guilds. Within each tribe

are also Squads, small teams dedicated

to specific disciplines within the tribe.

Squads act like miniature startups in their

own right, maintaining their own cultures

and core values.

A CULTURE OF CODE

AND CHARACTER

Anyone who has visited Socrata can attest

to the dedication and personality that

each employee brings to the company,

particularly the engineers.

One thing I requently talk abou

riends outside o work is that So

engineering team is filled with a b

o unique, unny people. We hav

o characters here, but at the end

day, everyone is deeply committe

customers and to the companys m

says Greenberg.

Socratas engineers are inventive

rarely bored. When they arent b

running code, they run maratho

hot sauce, craf micro-brews, and

printers to produce models o Iro

helmet. In between tasks, they als

riendly euds, rom office Ner-g

to internet pranks.

radition is another important p

Socratas culture, with the engine

among its most avid ollowers. Si

companys ounding, engineers h

lunch every Friday at the same te

place in Seattles Pioneer Square.

enjoy an office happy hour on Fr

afernoons, bringing in beer or w

Data is Socratas

business, and it handles

datasets of all sizes

from those with only

a dozen records to

others with records

ranging in the millions.A search request on

a small dataset takes

only a few seconds, but

advanced searches on

large datasets naturally

take much more time

to process. Socratas

engineers have recently

enjoyed the challenge

of developing methods

to make even advanced

searches in large

datasets complete in less

than four seconds.

Challenging

QueriesSocratas engineering team

members have attended

dozens of hackathons

around the world, serving

as instructors, presenters,

and judges.

Anyone who has visited Socrata can attest to the

dedication and personality that each employee

brings to the company, particularly the engineers.
http://www.scribd.com/doc/113617905/Scaling-Agile-Spotifyhttp://www.scribd.com/doc/113617905/Scaling-Agile-Spotifyhttp://www.scribd.com/doc/113617905/Scaling-Agile-Spotifyhttp://www.scribd.com/doc/113617905/Scaling-Agile-Spotify


19/21


share. On their anniversary o employment, team

embers also share one pound o chocolate or every

ar theyve worked at Socrata.

hile these perks and traditions are part o what

akes Socrata great, ask the engineers what they

ve most about their job and they will likely reply

arning, coding, and problem-solving.

We ace a lot o hard engineering challenges, and

ose challenges are very exciting to work on, says

nthony Nowell, a Socrata Sofware Developer. I

ve no doubt that Im surrounded by ridiculously

ight people who can overcome those challenges, and

m really encouraged knowing that were helping each

her to grow technically.

XPONENTIAL GROWTH

oneering tech companies like Google, Spotiy, and

cebook have demonstrated that well-defined culture

d open organizational models are valuable growth

ols. Tey help attract new tea m members, develop

mployee competency, and oster loyalty, which

turn inspires employees to produce innovative

ojects that put the company on the leading edge

entrepreneurship.

crata adds members to its team on a near-weekly

sis, many o them engineers. Part o this growth

ccess comes rom the attract iveness o Socratas

lture, which the company has nurtured since its

unding. Moving orward, Socrata will continue

emphasize organizational models that maintain

at culture.

s amazing seeing Socrata grow as big as it has, says

hris Metcal, Director o Developer Platorm and six-

ar Socrata employee. I dont have kids but, to me,

is is like seeing my child go off to college.

CHRIS CHARMS ARMSTRONG

Achievements unlocked:Helped

develop a number of startups

Geeks out on:Baking, beer, current

events, dev-ops, live music, tabletop

games, travel

Most likely to:Become a pastry chef

JEROME GAGNER

Achievements unlocked:Taught

himself to read and speak Arabic;

deployed numerous large-scale

applications in languages such as Java,

PHP, .Net, Ruby, etc.

Geeks out on:Camping, flight

simulators, SCALA, server-side and

front-end technologies

LILIA GUTNIK

Achievements unlocked:Talk

her prank adventures on The M

developed GovStat from the gr

Geeks out on:Data visualizati

Keanu Reeves, stand-up comed

Most likely to:Sleep with a K

Reeves body pillow

GIACOMO FERRARI

Achievements unlocked:3D printed

Iron Mans helmet and arc reactor; co-

created Socratas charting library

Geeks out on:3D printing, electronics,

web development

Most likely to:Invent a working flux

capacitor, blow himself up in a lab

STEVE GREENBERG

Achievements unlocked:Recently had

his first child, helped design Microsofts

error reporting system

Geeks out on:Cycling, family,

remodeling his house

KARIN HELLMAN

Achievements unlocked:Iv

it through a year in the U.S. wit

getting fat.

Geeks out on:Climbing, color

painting, Photoshop

MeetSocratasEngineers

When we say geek out, we mean

it. Socratas engineering team hosts

a bright, quirky cast of characters

who are known for much more than

creating great software.


20/21


OHN KEW

chievements unlocked:Worked on

number of evolutionary computation

ojects; developed code used on

llions of servers around the world

eeks out on:Family, illustratingcroorganisms, microbiology

AYN LESLIE-COOK

Achievements unlocked:My kids

dont seem like theyll turn out to be

criminals.

Geeks out on:Family, statistical

modelling, swimming, theatre

ANTHONY NOWELL

Achievements unlocked:Recently had

his first child, built a small application to

assist his mother-in-law with her daycare

business

Geeks out on:Coding, family, webframeworks

PAUL PARADISE


shape Socrata from the very beginning

Geeks out on:Computer hardware,

computer networking, triathlon

JEFF SCHERPELZ

Achievements unlocked:Led Socrata

through several versions of its front

end (including conversion from FLEX to

Javascript); designed and built his own

house

Geeks out on:Cars, food, knitting

ASON KROLL

chievements unlocked:Built a

stem to identify the best type of

ucation for over 15,000 careers and

d schools offering such education

eeks out on:Computer science,

onomics, music, running, statistics,

nnis

CHRIS METCALF


Socrata grow from a startup into a

company

Geeks out on:APIs, developer tools,

home improvement, micro-brewing,

photography

BRIAN OLDFIELD

Achievements unlocked:Accepted to

grad school

Geeks out on:Bitcoin mining, websitedesign

DAN RATHBONE

Achievements unlocked:Travelled

and worked all over the world

Geeks out on:Database technologies,hiking, running, triathlon

Most likely to:Wander off to explore

and then be found living in a hut in the

middle of nowhere

CLINT TSENG

Achievements unlocked:Taught

math and English in India; worked on

a project deployed to 80+ countries

and the international space station

Geeks out on:Design, guitar, music,

movies, sailing

Most likely to:Freak that this was

written in Comic Sans

eet Socratas Engineers


21/21

[email protected]

(206) 340-8008

83 S. King St., Ste. 107

Seattle, WA 98104

www.socrata.com

twitter.com/socrata

facebook.com/socrata
https://twitter.com/socratahttps://www.facebook.com/socratahttps://www.facebook.com/socratahttps://twitter.com/socrata

Documents

[Socrata] Open Innovation - Volume 2