Understanding mobility: Consent and capture of geoloca1on data in web surveys

Understanding mobility: Consent and capture of geoloca1on

data in web surveys

Sco4 D. Crawford [1] Colleen McClain [2] Robert H. Young [1] Toben F. Nelson [3]

[1] Survey Sciences Group, LLC; [2] Michigan Program in Survey Methodology; [3] University of Minnesota, Twin CiOes

Mobile Devices: A Disruptor Technology in Survey Research

•  We must adjust – but will be rewarded with new opportuniOes! – “In the moment” surveys – Self-‐administered biomarker collecOon – Sound and image captures – “Internet of Things” integraOon (i.e. thermostat readings, electricity use, pedometers, etc.)

– Geoloca'on capture

2

What is GeolocaOon? Is it the same as GPS?

3

1

2 3

4

Easing in to GeolocaOon Capture Our Domains of Interest

A. Is it possible? • What would respondents think about it?

B. Does it work? • Can we actually capture the data? • Do respondents allow it?

C. What are the best pracOces? • How do we handle device prompts? • Is standard consent form appropriate and adequate? • Should we ask permission explicitly? • Do we capture more than once?

Quality: How well did it work? • What does the data look like? Is it complete?

• Can it cause any error? Can it help idenOfy error?

4

Study: Baseline QuesOonnaire (April 2013)

•  Random sample of 8,000 U of Minnesota students •  Baseline quesOonnaire (T12) –  Alcohol, Drugs, Mental Health and related behaviors & experiences

– Web Survey Length •  Mean=24.3 minutes; Median=22 minutes

–  AAPOR RR#2: 28% –  Other

•  Prenote email requested users complete the survey on a desktop or laptop computer

•  All Email (Invite, 3 reminders to NRs) •  Sweepstakes incenOve for $500 cash

5

Study: Rapid Response QuesOonnaires •  Follow-‐up Surveys of Responders to Baseline –  Short version of key measures from baseline –  No incenOve used, all email communicaOons, references made to survey being designed for a mobile device

6

Time Month in 2013 Length AAPOR RR#2

T14 June Mean=3.9 min; Median=3 67%

T16 August Mean=3.63 min; Median=3 66%

T18 October Mean=3.83 min; Median=3 61%

Study: Sample CharacterisOcs

48.6

72.9

8.6 15.6 17.9 25.5 32.4 38.3

77.6

7.5 14.2 15.4 23.4

39.4

Male White Class: 1st Year

Class: 2nd Year

Class: 3rd Year

Class: 4th Year

Class: Grad Student

Baseline Sample vs. Baseline Responders

Sample Baseline

7

All differences were significant in a Χ2 test p<0.01

Quality: How well did it work?

In Rapid Response surveys – no further differences. They were the same as Baseline responders.

More White

Less Male Older

Hypothesis 1a: Will respondents cooperate (hypotheOcally)

at T14 Rapid Response? •  Hypothesis 1a: When asked, a majority of those responding will cooperate with a hypotheOcal request to collect geolocaOon data.

8 A. Is it possible?

Yes! 58% accepted when asked the hypotheOcal quesOon.

SUPPORTED …but, 42% did say NO…

The “HypotheOcals”… Were there any data quality concerns?

•  No differences demographically •  No differences on any substanOve measures captured, including: – Past 2 weeks binge drinking – NegaOve consequences from alcohol use – Past 4 hours drinking or drug use

9 Quality: How well did it work?

Will respondents cooperate with an ACTUAL geolocaOon request?

•  Hypothesis 1b: When asked, a majority of those responding will cooperate with a request to collect actual geolocaOon data.

•  So how do we get permission? – GeolocaOon data is above and beyond what a typical survey respondent would expect or even understand is being collected when they agree to complete a Web survey, thus, it should be described in the consen1ng process for the survey…

10 B. Does it work?

RequesOng GeolocaOon Data: Devices Already Do This – But Beware

•  W3C GeolocaOon API SpecificaOons require permission1

•  …While the technology currently requests permission prior to capturing this type of data, we do not have control over that and we cannot guarantee that it will be maintained. We believe that the automated request provided by the technology is not sufficient and it should be supplemented.

11

1) h4p://www.w3.org/TR/geolocaOon-‐API/#security

C. What are the best pracOces?

The SoluOons We Considered… Add Language to the Survey Consent Form

Add a Separate Geoloca'on Specific Consent Ques'on

12 C. What are the best pracOces?

In addi1on to the ques1ons in this brief survey we would like to collect data on the loca1on where you are comple1ng this survey using features available in desktop computers and mobile devices. You will be asked whether you will allow loca1on data to be collected and you may choose not to allow collec1on of loca1on data.

We would like to understand more about where respondents are when they par1cipate in surveys. We would like to collect informa1on made available by your computer/mobile device on your geographic loca1on. Do you accept or decline our request to collect your loca1on? o  Yes, you may collect geographic

data o  No you may not collect

geographic data

Two Hypotheses

•  H2a: Respondents who are asked in a separate quesOon for permission to allow geolocaOon capture will be less likely to consent to capture than those who agree as part of the main consent form. –  The “If you ask it, they will say no!” hypothesis

•  H2b: Respondents who consent to capture will be more likely to actually provide geolocaOon data if they are consented with a separate quesOon. –  The “Surprise, we are tracking you!” backfire hypothesis.


The Consent Experiment Treatments

14

Consent to Survey with GeolocaOon Text

Start Survey & GeolocaOon Capture

Consent to Survey with GeolocaOon Text Consent to GeolocaOon


(if Consented)

Treatment A: Consent Form Only

Treatment B: Geoloca'on Consent Ques'on

Consent to Survey Start Survey

Treatment C: Control (T16 only)


B. Does it work?

Did respondents consent to the survey? (T16)

15

Consent to Survey 96% (n=136)


Consent to Survey 96% (n=151) Consent to GeolocaOon


(if Consented)

Treatment A: Consent Form Only (n=142)

Treatment B: GeolocaOon Consent QuesOon (n=157)

Consent to Survey 93% (n=135) Start Survey

Treatment C: Control

B. Does it work?

But what about the extra step with the Consent QuesOon? (T16)

16

Consent to Survey 95.8% (n=136)



Consent to Survey 96% (n=151)

Consent to GeolocaOon

60% Agreed (n=90) 40% said NO or blank


(if Consented)

Consent to Survey 93.1% (n=135) Start Survey

Treatment C: Control


This replicated in T18 data collec'on with 67% agreeing to par'cipate.

Treatment B: Geoloca'on Consent Ques'on (n=157)

The “If you ask it, they will say no!” Result

•  H2a: Respondents who are asked in a separate quesOon for permission to allow geolocaOon capture will be less likely to consent to capture than those who agree as part of the main consent form. – The “If you ask it, they will say no!” hypothesis


SUPPORTED

Did we capture geolocaOon data? Possible outcomes

18

•  LaOtude / Longitude data and related data is received

1. Success: We Capture Data

•  Permission Denied or Permission Unknown

2. Permission Error: Error Code Received

•  And no informaOon as to why

3. No Data: We Get Nothing


Consent to Survey •  96.9% (n=189)

Consent to GeolocaOon • 66.7% Agreed (n=130) • 33.3% said NO or leu blank

GeolocaOon Captured? • Success: 49.2% (n=64) • Permiss Denied: 14.6% (n=19) • No Data: 36.2% (n=47)

Did we capture data auer consent? (T18)

Consent to Survey (with GeolocaOon text) •  97% (n=212)

GeolocaOon Captured? • Success: 20% (n=43) • Permiss Denied: 29% (n=62) • No Data: 51% (n=107)

19





Final Usable GeolocaOon Data •  n=64 cases

•  32.8% of sample

Net: More Data With GeolocaOon Consent QuesOon

20

Final Usable GeolocaOon Data •  n=43 cases

•  19.6% of sample





Hypothesis 2b Result

•  H2b: Respondents who consent to capture will be more likely to actually provide geolocaOon data if they are consented with a separate quesOon.



SUPPORTED

Hypothesis 2c: Where geolocaOon providers demographically the same?

•  H2c: Sample characterisOcs and substanOve measures collected from those who successfully provide geolocaOon data will be similar to those who did not.

Ø No differences found in gender, race/ethnicity and year in school.


Hypothesis 2d: What did this do to survey break-‐off?

•  H2d: Survey break-‐off rates among those who successfully provide geolocaOon data will be similar to the breakoff rates of those who did not.


•  Compared to the control, no significant impact (very small sample sizes) –  T16: 2.3% Treat A / 1.8% Treat B / 0% Treat C (control) –  T18: 5.5% Treat A (n=12) / 3.3% Treat B (n=6)

SUPPORTED

Hypothesis 2e: What did this do substanOve responses?

•  H2e: SubstanOve responses provided by those who successfully provide geolocaOon data will be similar to responses provided by those who did not.


H2e Results – So Far…

•  Ongoing analysis here… some mixed early news: – No differences on most measures – Respondents more than twice as likely to have drank alcohol within the past four hours (10.5% vs. 4.2%) if they successfully provided geolocaOon data (p<0.01) •  But not other drugs…


Hypothesis 3: Mobile vs. Non-‐Mobile

•  H3a: Mobile users (phones and tablets) will consent (when asked explicitly) to geolocaOon capture at a higher rate than non-‐mobile users.

•  No difference found: – 66% of non-‐mobile users consented – 69% of mobile users consented



NOT SUPPORTED

Hypothesis 3: Mobile vs. Non-‐Mobile

•  H3b: Mobile users will end up providing a higher rate of successful geolocaOon captures than non-‐mobile users.



H3b – Does mobile lead to a higher capture success rate?

28

27% 46%

17%

46% 56%

8%

0%

20%

40%

60%

80%

100%

Non-‐Mobile Respondents

Mobile Respondents

No Data

Error Capture

Successful GeolocaOon

NOTE: Difference was significant in a Χ2 test at p<0.01


Yes SUPPORTED

And what did we get? LaOtude/Longitude

•  Lat/Long Accuracy Data –  100% complete – Accuracy to within 5 to 95000 meters (16.4 feet to 59 miles)

•  InteresOng arOfact –  5 cases with 22,000 meters and 1 case with 95,000 meters all with the SAME Lat/Long point

–  Lat/Long point corresponds to a large 512 unit apartment building

– All Respondents with this point were using Windows


And what did we get? AlOtude Data

•  AlOtude – 25.6% with a non-‐NULL value; Min: 21 -‐ Max: 302 – Mean: 254 meters (833 feet) •  Per Wikipedia: Eleva'on of Minneapolis is 830 feet


So what does the data look like?


Study LimitaOons

•  Student populaOon at one University •  Low response rate with some nonresponse bias •  Short quesOonnaire •  Small sample sizes

32

Next Steps

•  Evaluate differences using baseline data •  Build and test some models to be4er evaluate substanOve difference(s)

•  How can we improve consent rate further? •  How can we reduce geolocaOon failure to provide data?

•  Explore what geolocaOon data itself can do to tell us about data quality (i.e. do those who stay sOll provide be4er data?)

•  What else can we collect?

33

Thank You!

•  QuesOons?

•  Contact: Sco4 D. Crawford [email protected] 734-‐527-‐2150 (office) 734-‐395-‐8790 (cell)

34

Documents

Understanding mobility: Consent and capture of geoloca1on data in web surveys