Embed Size (px)
Citation preview
Crowdsourced non-probability sampling – A telecom industry showcase
Confidential
P3 insight GmbH @ IPSDS
Mannheim, 20.02.2016
20.02.2016
Confidential
P3 is a global provider of engineering and technical consulting services
P3 group provides technical consulting, management support, engineering and testing services.
Main markets are telecommunications, automotive and aviation.
P3 is currently entering new markets like energy and storage
MARKETS
P3 group Facts and Figures
GLOBAL PRESENCE
In [Mio €]
P3 GROUP REVENUE
P3 group consists of 25 subsidiaries
P3 group is internationally well positioned to ensure close contact to our customers
Local presence in 12 countries at 37 locations in Europe, North America, Australia, Asia and Africa
2.500 employees from 45 nations
¾ of our people are engineers (of different disciplines)
P3 GROUP FACTS AND FIGURES
BarcelonaESDenver
US
ToulouseFR
Wolfsburg DE
UlmDE
AachenDE
IngolstadtDE
Detroit US
VancouverCA
TokioJP
MontréalCA
BedminsterUS
BerlinDE
SolmsDE
MumbaiIN
PekingCN
KapstadtZA
BangaloreIN
ManchesterGB
StuttgartDE
HamburgDE
BelgradSER
20.02.2016
Confidential
P3 provides consulting, engineering and testing services for the ICT market
P3 communications is an independent and credible provider of
• technical consulting
• ICT network engineering services
• device, network and service testing
P3 communications is well known for excellence, swiftness, quality and innovative nature of its services and solutions
MARKETS
P3 communications Facts and Figures
Main markets are Europe, North America, India and Oceania
Business in more than 50 countries
GLOBAL BUSINESS
In [Mio €]
P3 COMMUNICATIONS REVENUE
P3 communications has subsidiaries in the USA, Australia, India and Serbia
Local presence at 15 own office locations worldwide
More than 600 employees
Founded P3 insight GmbH in 2013
P3 COMMUNICATIONS FACTS AND FIGURES
20.02.2016
2016 / 02 / 09 Confidential 4
P3 performs its yearly public benchmark of mobile networks with well-established model in European and Australian market
Since 2001 P3 with its media partners publishes yearly mobile network service quality benchmarks with strong focus on customer experience
Since 2002 P3, the international leader in mobile benchmarking with a solid and worldwide accepted testing methodology, carries out the tests
Focus markets: DE, AT, CH, UK, ES, NL, SE & AU
Example report: http://www.connect-testmagazine.com/network-tests/
WHAT IS IT?
Public Benchmark
P3 is cooperating with top telecommunication media partners in respective countries like Connect magazine
Connect is Europe's largest magazine for testing of telecommunications products and publishes the yearly Public Benchmark results
MEDIA COOPERATION
VISIBLE OUTCOME
Drive Testing Crowdsourcing
2016 / 02 / 09 Confidential 5
P3 provides leading-edge network testing to operators on a worldwide scale
Drive Testing – Impressions
Drive Testing Crowdsourcing
20.02.2016 Confidential
P3 provides a leading-edge passive and distributed crowdsourced network on a worldwide scale
Crowdsourcing – Impressions
Drive Testing Crowdsourcing
Confidential
P3 solved the challenge of crowdsourced measurements to get accurate location information without harming end users’ battery consumption
While walking around people generate data points for us which we aggregate and enhance to relevant mobile network performance data
PASSIVE DRIVE/WALK TEST
Crowdsourcing – Collected Data
EXAMPLE NEW YORK
A user going for a run, or navigating with TomTom App/Google Maps creates a passive walk/drive test for you
„
20.02.2016
Drive Testing Crowdsourcing
Confidential
Next to technical information we collect usage as well
Each app start creates a data point
Various app-related observers collected (network information is also part of the records)
BACKGROUND
Crowdsourcing – Collected Data
APP USAGE
20.02.2016
We look over the “virtual shoulder” of our users and see what apps are popular, are using heavy data, …
„
Drive Testing Crowdsourcing
20.02.2016 Confidential
In benchmarking business we have concepts comparable to classical sampling techniques. What can we learn/transfer from classical survey techniques?
DRIVE TESTING
Test Methods and established Sampling Strategies compared
CROWDSOURCING
A route is selected carefully Participants roam around “randomly”
A defined – more lab-like – environment is used Wide variety of devices and tariffs
Expensive and time consuming Delta cost for one sample is virtually zero
You have a complete sampling frame
You can select a random sample from your population
You can generalize your results from a random sample
Can be more expensive and time-consuming than convenience or purposive sampling
Used when there isn’t an exhaustive population list available
Can be effective when trying to generate ideas and getting feedback, but you cannot generalize your results to an entire population with a high level of confidence
More convenient and less costly, but doesn’t hold up to expectations of probability theory
PROBABILITY SAMPLING IN CLASSICAL SURVEYS NON-PROBABILITY SAMPLING
Adaption of representative polling kick started with pivotal polling mishap in 1936 by Literary Digest
Despite polling 10m people (2.4m answered) the sampling error in the Literary Digest poll was 19%, the largest ever in a major public opinion poll
A badly chosen big sample is much worse than a well-chosen small sample
Watch out for selection bias and non-response bias
20.02.2016 Confidential
Enhancing convenience sampling polls with statistical techniques to adjust estimates and result in accurate predictions
THE RAISE OF REPRESENTATIVE SAMPLING
The Drivers for Representative / Non Representative Sampling
Standard methods in modern representative polling (e.g. Random Digit Dialling) are suffering from increasingly high non-response rates
Increasing need for post-sampling adjustments
Adjustment methods to post-stratify non-representative/convenience samples become of utterly importance
One extreme case of a convenience sample that predicted the outcome of the 2012 U.S. presidential election was of Xbox gaming console users*) using an extremely elaborate estimator
DRIVER FOR NON-REPRESENTATIVE SAMPLING
Literary Digest Poll
Franklin D. Roosevelt - 43% Alf Landon – 57%
Election
62% 38%
19% Delta
*) Wang, W., et al. Forecasting elections with non-representative polls. International Journal of Forecasting (2014)
… with proper statistical adjustments, non-representative polls are able to yield accurate presidential election forecasts, on par with those based on traditional representative polls
Once again, [we are] asking .. one out of four [voters], representing every county in the United States -- to settle November's election in October. When the last figure has been totted and checked, if past experience is a criterion, the country will know to within a fraction of 1 percent the actual … vote …
20.02.2016 Confidential
Develop appropriate post-stratification methods and leverage from the statistical properties of our convenience sampling method
Network quality samples collected on end user smartphones are more correlated
Many influencing factors for network quality can be measured on the smartphone (time of day, location, smartphone type, tariff limitations) and used for re- weighting
Non-response bias non existent. We have 100% response rate once the app is installed
Selection bias on influential dimension (principle components on service quality) is low
We have hard technical data, no emotions, feelings etc.
No “undecided yet”
No issues because of “socially desirable response patterns“
No “bad question bias” conclusions
CROWDSOURCED SAMPLING FOR BENCHMARKING
Challenges and Chances
Identify principle components on service quality KPIs
Develop weighting model (space, time, devices, …)
Develop metric to judge how samples are distributed in space
Develop post stratification strategy
TASKS
Confidential
List of links to publicly available app with embedded P3 data collection framework
CURRENT APP PORTFOLIO
Our growing app portfolio
20.02.2016
DB Netzradar
https://itunes.apple.com/de/app/db-netzradar/id945408814?l=de&ls=1&mt=8
https://play.google.com/store/apps/details?id=de.db.fv2020
Sunrise Mobilnetz / Mobile Network
https://itunes.apple.com/de/app/sunrise-mobilnetz/id1001112739?mt=8
https://play.google.com/store/apps/details?id=ch.sunrise.cfa
Connect
https://itunes.apple.com/de/app/connect/id910960897?mt=8
https://play.google.com/store/apps/details?id=com.p3group.insight.connect
PC Magazin Speed Test
https://itunes.apple.com/de/app/pc-magazin-speedtest/id1003706363?l=en&mt=8
https://play.google.com/store/apps/details?id=com.p3group.insight.pcmagazin
Who‘s Calling?
https://play.google.com/store/apps/details?id=de.phl.whoscalling
P3 Feedback Demo
https://play.google.com/store/apps/details?id=com.p3group.feedback.demo
P3 Feedback
https://play.google.com/store/apps/details?id=com.p3group.feedback
?
Your Contact
e-Mail: [email protected] Internet: www.p3-group.com/p3-insight/insight
WEB
DISCLAIMER This document contains P3 insight GmbH proprietary information and shall at all times remain the property of P3 insight GmbH. It shall be distributed and used only by the subcontractor staff registered on the distribution list of this document. The subcontractor is not allowed to copy this document without P3 insight GmbH prior written consent. It shall not be used, distributed or copied by any other third party without P3 insight GmbH prior written consent.
Peter Seidenberg E-Mail: [email protected] Phone: +49 151 571 33 444 Ralf Klüber E-Mail: [email protected] Phone: +49 151 571 33 293 Michael Wennesheimer E-Mail: [email protected] Phone: +49 151 571 33 027
YOUR CONTACT
GERMANY - AACHEN P3 insight GmbH Am Kraftversorgungsturm 3 D-52070 Aachen Germany
P3 communications GmbH Am Kraftversorgungsturm 3 D-52070 Aachen Germany
USA – NEW JERSEY P3 Communications Inc. 412 Mt. Kemble Ave., Suite G20s Morristown, NJ 07960 USA
AUSTRALIA – SYDNEY P3 communications Pty Ltd 123 Epping Road, Suite 401 North Ryde NSW 2113 Australia
20.02.2016 Confidential
