View
221
Download
1
Category
Preview:
Citation preview
From data to business advantage
Rafal LukawieckiStrategic Consultant
Project Botticelli Ltd
rafal@projectbotticelli.com
@rafaldotnet
Objectives
The information herein is for informational purposes only and represents the opinions and views of Project
Botticelli and/or Rafal Lukawiecki. The material presented is not certain and may vary based on several factors.
Microsoft makes no warranties, express, implied or statutory, as to the information in this presentation.
Portions © 2014 Project Botticelli Ltd & entire material © 2014 Microsoft Corp unless noted otherwise. Some
slides contain quotations from copyrighted materials by other authors, as individually attributed or as already
covered by Microsoft Copyright ownerships. All rights reserved. Microsoft, Windows, Windows Vista and other
product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The
information herein is for informational purposes only and represents the current view of Project Botticelli Ltd as of
the date of this presentation. Because Project Botticelli & Microsoft must respond to changing market conditions,
it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft and Project Botticelli
cannot guarantee the accuracy of any information provided after the date of this presentation. Project Botticelli
makes no warranties, express, implied or statutory, as to the information in this presentation. E&OE.
Introduction to BI & Big Data
DAX
MDX
Data Mining
Excel BI
projectbotticelli.com/ppt
Register on
projectbotticelli.com
[data + analytics + people] @ speedMicrosoft data dividend formula
Microsoft transformation
Mobile-first Cloud-first
Data-driven
Microsoft transformation
Cloud-firstMobile-first
Water pumps
Outbreaks
Public domain picture. Wikimedia. http://commons.wikimedia.org/wiki/File:Snow-cholera-map.jpg
Transformative opportunity
Business analytics
Good BI is key to business analytics
Integrate
+ Cleanse
Model +
Enrich
Visualise
Query
Share
Power
Query
Power
PivotPower View
& Power
Map
Power BI
Q&A
Power BI
Sites +
SharePoint
Power BI for cloud collaboration and new experiences
Excel as the BI tool for everyone
Self-service, cloud
IT
SharePoint + SQL/APSIT scalability & control
Excel user-driven
Corporate self-service, on-prem
Power BI for cloud collaboration and new experiences
Hybrid
IT
SharePoint + SQL/APSIT scalability & control
Big data, or just complex data?
velocity
variety complexity
volume
Data
interpretingpreparing
Today’s big data, tomorrow’s little dataComplexity vs. current capabilities
FAA International Flight Service Station, Honolulu, Hawaii, 1964 (Public Domain Image)
So… what is big data?
Machine learning
Machine learning = data mining?
Best together
Data wrangling (munging), retrieval
+ storage
Data mining & machine learning
Statistics
Big data
Domain Common big data scenarios
Financial services Modeling true risk
Threat analysis and fraud detection
Trade surveillance
Credit scoring and analysis
Media & Entertainment Recommendation engines
Ad targeting
Search quality
Abuse and click fraud detection
Retail Point of sales transaction analysis
Customer churn analysis
Sentiment analysis
Telecommunications Customer churn prevention
Network performance optimization
Call Detail Record (CDR) analysis
Network failure prediction
Government Cyber security (botnets, fraud)
Traffic congestion and re-routing
Environmental monitoring
Antisocial monitoring via social media
Healthcare Genomics research
Cancer research
Health pandemics early detection
Air quality monitoring
Do you need it?
Process
Understand & change
data
Discover patterns, build & validate models
Change business
People
Data expert
Data scientist
Domain expert
Start of an engagement
Data, sucks
slightly
Are there any useful patterns?
Unclear business
goals
Example: fraud
Does enough data show
examples of fraud?
Are the predictable patterns of fraud?
Can we reduce
fraud? What is fraud?
In-house intelligence
Understand & change
data
Discover patterns, build & validate models
Change business
What tools do data scientists use?Purple=data analyst role
SQL 42%!
#1 data science tool
Chart from "2013 Data Science Salary Survey" (ISBN 978-1-491-94914-6)
© 2014 O'Reilly Media, used with permission.
For more info, and great titles on data science, visit oreilly.com
My analytical toolkit at Project Botticelli
main tools
secondary
only if I can’t avoid it
Chart from "2013 Data Science Salary Survey" (ISBN 978-1-491-94914-6)
© 2014 O'Reilly Media, used with permission.
For more info, and great titles on data science, visit oreilly.com
might try
My toolkit (chronologically)
○ SQL Server
○ DB engine
○ SSAS for data mining
○ Excel
○ now + Power Query
○ R and RStudio
○ Stats
○ Great charts
○ Curve fitting
○ Rattle for data mining
○ Mahout in Hadoop
○ HDP, HDInsight, or just *nix
Hadoop
○ Evaluating H2O + Spark now
○ Python 3
○ PyCharm IDE on OS X
○ Visual Studio with Microsoft
Python tools on Windows
○ Azure ML
○ Data mining
Algorithm Description
Decision Trees Finds the odds of an outcome based on values in a training set, presents visually
Association Rules Identifies relationships between cases
Clustering Classifies cases into distinctive groups based on any attribute sets
Naïve Bayes Clearly shows the differences in a particular variable for various data elements
Sequence
Clustering
Groups or clusters data based on a sequence of previous events
Time Series Analyzes and forecasts time-based data combining the power of ARTXP (developed
by Microsoft Research) for short-term predictions with ARIMA for long-term accuracy.
Neural Nets Seeks to uncover non-intuitive relationships in data
Linear Regression Determines the relationship between columns in order to predict an outcome
Logistic
Regression
Determines the relationship between columns in order to evaluate the probability that
a column will contain a specific state
Algorithm Description
Random
Forests
Like decision tree, but can be more accurate, and
difficult to understand
Boosting Like random forest, but using any other algorithm (not
just DT), “boosts” model accuracy for less frequent
items
Survival
Analysis
Finds the risk of an outcome given periods of time
Ensemble Combination of multiple models (in SQL or R)
Microsoft tech for big data
Prebuilt & performance-tuned appliance
Linear scale-out to petabytes of data
MPP design & in-memory columnstorefor 100x speed improvement
Dedicated region for Hadoop
Joining relational & non-relational datawith Polybase
Analytics Platform System (APS)
MPP SQL Server
Hadoop
PolyBase
Apache Hadoop distribution
Developed by Hortonworks & Microsoft
Integrated with Microsoft BI
Microsoft HDInsight
Part 1: the job
Big, fast, or
complex
data
HDInsight
Tabular
OLAP
SQL
010101010101010101
1010101010101010
01010101010101
101010101010
Interaction,
exploration,
reporting,
visualisationAPS +
Polybase
Hadoop cluster
Yahoo! Hadoop cluster, about 2007.
Source: http://developer.yahoo.com. Picture used with permission.
Hadoop cluster
Buster Cluster, an early research project
by Miles Osborne, University of
Edinburgh, School of Informatics.
Picture used with permission.
http://homepages.inf.ed.ac.uk/miles/
Cloudrent-a-Hadoop-cluster, or:
“Supercomputer for cents”
Windows Azure HD Insight
Processing logic in HDInsight 3.0 & 3.1Hadoop 2.2/2.4: Interactive, online, stream, or batch(Hadoop 1.x was batch process only)
Hadoop data science
Collaborative filtering,
recommenders, clustering,
singular value decomposition,
parallel frequent pattern mining,
naive Bayes, decision tree
Part 2: the results
Azure ML = data science in
Turning data into advantage
Summary
projectbotticelli.com
BI video tutorials, PPTs, and articles
15% Off: 15PRAGUE2014
Valid until end of November 2014
Follow: @rafaldotnet
Email: rafal@projectbotticelli.com
Discover: rafal.net
The information herein is for informational purposes only and represents the opinions and views of Project Botticelli and/or Rafal Lukawiecki. The material presented is not certain and may vary based on several factors. Microsoft makes no warranties,
express, implied or statutory, as to the information in this presentation.
Portions © 2014 Project Botticelli Ltd & entire material © 2014 Microsoft Corp unless noted otherwise. Some slides contain quotations from copyrighted materials by other authors, as individually attributed or as already covered by Microsoft Copyright
ownerships. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and
represents the current view of Project Botticelli Ltd as of the date of this presentation. Because Project Botticelli & Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and
Microsoft and Project Botticelli cannot guarantee the accuracy of any information provided after the date of this presentation. Project Botticelli makes no warranties, express, implied or statutory, as to the information in this presentation. E&OE.
Recommended