Northwestern data visualization - why why why

Northwestern Data Visualization| @gabegaster | 2015 may

what is data science?

who is a data scientist?

review of literature

who is a data scientist?“a scientist who can code”

• lower barrier to attack new problems

• lower barrier to attack new problems • repeatable analysis

• lower barrier to attack new problems • repeatable analysis • freedom to think about problems new ways

using emerging technologies to approach problems scientifically

which were difficult to answer before

computing has progressed

1950computing has progressed

cost of new analysis

cost of new analysis years

todaycomputing has progressed

hoursv

same person thinking about the problem can conduct experiments to answer it

hoursv

open-source code

standing on shoulders of giants

open-source code

reinventing the wheel

open-source code

reinventing the wheel

knowing what is possible

doing something useful

HOW WHY

usingnew good

the righttools

usingnew good

the rightasking whytools

usingnew good

the rightasking whytools WHY

usingnew good

the rightasking whytools WHY

why why whywhat is data science?

science is about asking why

science is about asking whystart there

an anecdote

an examplefrom the real world

an example

• what

what data do you have? how can we make a network?

• what

what data do you have? how can we make a network?

goal: save money

goal: save moneytask: find needle in the haystack (without poking yourself)

turn over to plaintiffdon’t

turn over to plaintiff

adverse inference

give away trade secrets

adverse inference

give away trade secrets

goal: save moneyprototype

goal: save moneyprototype — design for lawyers

Sexier. Less nerdy. Tailored. design for transparency

http://www.daegis.com/judicial-acceptance-of-tar/

another examplecontests

another example

classify schizophrenia w MRItask:

why?classify schizophrenia w MRItask:

improve understanding of disease

improve understanding of diseasehow?

improve understanding of diseasehow? … outside contest purview

why? outside contest purview

kaggle

kagglegetting data

& making usable

kagglegetting data

& making usable

timeline of contest

Accuracy of Classification

timeline of contest

what is AUC?AU

AUCwhat is AUC? Area Under Curve

what curve?

what curve? Receiver Operating Characteristic

balances:

balances: True Positive RateFalse Positive Rate

AUCwhat is AUC?

Area Under Curve

why?AUCwhat is AUC?

Area Under Curve

why?…

AUCwhat is AUC?

Area Under Curve

why?…

upshot:

AUCwhat is AUC?

Area Under Curve

why?…

choice of metric matters a LOT

upshot:

in practice

AUCwhat is AUC?

Area Under Curve

timeline of contest

random guess

timeline of contest

random guess

basic SVM

timeline of contest

random guess

basic SVM

timeline of contest

goal: depends on why

random guess

basic SVM

random guess

basic SVM

timeline of contest

AUC turned out to place 9th — because overfitting

timeline of contest

AUC turned out to place 9th — because overfitting

very common problem

timeline of contest

worth it?

We need to reduce the costs of Service Requests. They are too expensive.

Thousands of engineers around the world, 24-7 read through emails and hardware log files to determine

the cause of failure of a server. This is an expensive process. We've tried to automate it. We can now

automatically resolve 7% of new Service Requests. But we want more. That's why we bought a few

super computers with TBs of memory.

client

an example !

from the industrial internet

Why? Why do you need to set up a hadoop architecture to do clustering? What will this help you achieve? !!!!!How do you handle Service Requests? !

client

Why? Why do you need to set up a hadoop architecture to do clustering? What will this help you achieve? !!!!!!!

client

clienttools are not everything

but it is important to know the right tool for the job

don’t start w hadoop unless you have to. !

probably you don’t have to.

client

How did you automate resolving Service Requests? !!!!!!!!!!!

client

A group of senior engineers thought about different use cases and came up with a list of conditions that, if any

are met, lead to predetermined solutions.

client

are met, lead to predetermined solutions. !

Took a year to create. !

client

We’ve been keeping track of every solved request for several years now.

client

We’ve been keeping track of every solved request for several years now.

from sklearn import naive_bayes as nb!nb.GaussianNB().fit(historical_requests,!! ! ! ! ! ! historical_decisions)

client

This works really well! But we can’t use it. !!!!!!

client

Oh. Why is that?

client

Engineers don’t trust the predictions.

Oh. Why is that?

client

Engineers don’t trust the predictions.

Oh. Why is that?

an example

just for fun

an example

just for fun

(a bit more depth this time)

a typical trip for me

Bus transit times = a LIE

Chicago is a grid city

Difficult

Public Transit on the grid

=+ Diagonals

Difficult

Public Transit on the grid

=+ Diagonals

2+ buses = FAIL

Adding bikes to public transit

show how has divvy changed where people

can go

viz Goal:

show how has divvy changed where people

can goshow where people

actually go

viz Goal:

How to show where people go from here?

one way is with an arrow

How to show where people go from here?

good for abstract relationships

one way is with an arrow

hard to decipher

lines between pts?

@flowingdata

emphasizes traffic

@flowingdata

lines between pts?(the lines superimpose)

emphasizes traffic

@flowingdata

lines between pts?

beautiful map(the lines superimpose)

emphasizes traffic

@flowingdata

lines between pts?

beautiful map(the lines superimpose)

— but not suited for this goal

lines between pts?

how to represent stations?

lines between pts?

on a maphow to represent stations?

but how?

lines between pts?

on a maphow to represent stations?

can use gradient — to show gradual differences

between stationsLondon transit map

@mySociety

@mbostock

or use natural borders?

London transit map

@mySociety

what regions?

each point is related to the closest station

what regions?

—> Voronoi

what regions?

—> Voronoihuh?

what regions?

—> Voronoihuh? http://alexbeutel.com/webgl/voronoi.html

what regions?

—> Voronoihuh? http://alexbeutel.com/webgl/voronoi.html

Find the closest station — that’s my region!

Czech beer

starbucks

@gabegaster | http://bit.ly/1pdP2Tb

recap:

recap: stations are voronoi tiles

@gabegaster | http://bitly.com/bundles/gabegaster/1

bubbles

too many edges to show…

too many edges to show… !

what about using color?

how touse color?

colors v colors

colors !

colors

@mySociety

@mbostock

how touse color?

• two color scale

colors v colors

how touse color?

binned v gradient

colors v colors

• two color scale

• hard to read • differences subtle

gradient

how touse color?

• two colors not many • binned not gradient

binned v gradient

colors v colors

how touse color?

• two colors not many • binned not gradient

binned v gradient

colors v colors

binned

how touse color?

• two colors not many • binned not gradient • transparent empty bin

binned v gradient

colors v colors

binned

how touse color?

• two colors not many • binned not gradient • transparent empty bin • iterate

binned v gradient

colors v colors

binned

• dispersion (where people can / do go)

with these choices, the viz emphasizes:

• exploration

what can we learn from this?

urban vs lake frontcommute vs party

vs train

there’s me, too.hipster vs yuppie

Great! This looks great! !

But it’s too much at once.

make it interactive

divvy.datasco.pe

new question

How are stations different?

who uses it

when is the station usedwho uses it

when is the station usedhow it used

who uses it

when is the station usedhow it used

who uses it

use the time signature of a station

Time Signature of a station

http://divvy.datasco.pe/multiline/

questions?

in conclusion

thanks!@gabegaster

Northwestern data visualization - why why why

Data & Analytics

Introduction to Data Visualization - ProPublica · Why visualize data How our brains work The origins of data visualization Some basic forms of data visualization How things can go

Why Are Indian Children So Short? - Harvard University · Why Are Indian Children So Short? Seema Jayachandran Northwestern University Rohini Pande Harvard University November 3,

Tableau Presantation.ppt [Kompatibilitetstilstand] · Why Tableau Tableau a great Tool for visualization, BI and data analytics. • Ideal for complex visualization • Interactive

Geographic Data Science - Lecture IIIdarribas.org/gds15/slides/lecture_03.pdf · 2016. 2. 19. · Lecture III (Geo-)Visualization Dani Arribas-Bel. Today Visualization What and why

Why Data Visualization is Only the Tip of the Iceberg

Why Data Visualization is Important in Delivering Actionable Insight

Visualization Analysis Why analyze? & Designtmm/talks/minicourse14/vad20alum-4x4.pdf · UBC Alumni/Industry Lecture Feb 27 2020, Vancouver BC Visualization: deÞnition & motivation

Data Visualization. Outline What? Why? How? Outline What? Why? How?

Why We Need Tax Reform – and Why We Probably Won’t Get It Robert Coen Professor Emeritus of Economics Northwestern University Woman’s Athletic Club Chicago,

Mihnea Popa - Northwestern Universitympopa/talks/AMS.pdfMihnea Popa (Northwestern)One-forms AMS Meeting, Lansing March 14, 2015 8. Interest in studying one-forms Why currently interesting?

Remote and in situ visualization why is relevant - HPC … · Remote and in situ visualization: why is relevant? • researchers need to visualize the data produced by numerical simulations

Scientiﬁc visualization with ParaView - Compute Canada · Rasmol molecular visualization bugaboo ... Why ParaView for this workshop? When I was developing the ﬁrst version of

Time-variable Deformation in the New Madrid Seismic Zone (why there, why now?) Eric Calais, Andy Freed, Purdue University Seth Stein, Northwestern University

Web 2.0 Data Visualization with JSF.ppt - SYS-CON Mediadw.connect.sys-con.com/session/23/Web_2 0_Data_Visualization_with... · 6 Why Data Visualization with JSF • JSF Simplifies

Data Visualization - GitHub Pages · Why Data Visualization? ... If there are longitudes and latitudes in your data, try out geographical visualization Ways of obtaining maps qmap(),

Why Peace? • Initial Connection • NC Data Visualization

RESULTS PRODUCT OVERVIEW VISUALIZATION & ANALYSIS Product Sheets/13-R… · VISUALIZATION & ANALYSIS RESULTS PRODUCT OVERVIEW WHY CMG? CMG is devoted to providing the ultimate customer

The Heatmap - Why is Security Visualization so Hard?

Why do people not learn from ﬂood disasters? Evidence from Vietnam’s northwestern ... · 2017-08-23 · from Vietnam’s northwestern mountains ... water management failures being

Why data visualization is nice... · Why data visualisation is nice By Dave and Rebecca, UTS Library david.litting@uts.edu.au rebecca.dale@uts.edu.au