Advancing Paradata

Embed Size (px)


Advancing Paradata. How to i mprove your household surveys : Lessons from analysis of 2012 UK Poverty and Social Exclusion Survey paradata Karen Bell School for Policy Studies University of Bristol. What this session might help you with (a bit):. Understanding the usefulness of paradata - PowerPoint PPT Presentation

Text of Advancing Paradata

Paradata and administering the UK Poverty and Social Exclusion survey

Advancing ParadataHow to improve your household surveys: Lessons from analysis of 2012 UK Poverty and Social Exclusion Survey paradata Karen BellSchool for Policy StudiesUniversity of Bristol

What this session might help you with (a bit):Understanding the usefulness of paradataConsidering how to capture paradataImproving survey designImproving survey administration techniquesImproving training of survey administratorsCommissioning survey administration agenciesBreaking down the barriers between qualitative and quantitative methodologyResisting our replacement by virtual human researchers

OverviewStandardised interviewing generally utilised in preference to conversational techniques in household surveys

Analysis of survey paradata from the 2012 Poverty and Social Exclusion Survey shows, in practice, even standardised survey interviewing often involves extensive unscripted conversation between the interviewer and the respondent

The unscripted conversation was useful and necessary for enhancing accuracy, cooperation and ethicality BUT when not carried out sensitively and flexibly unscripted conversation appeared to be unhelpful for survey reliability and the achievement of ethical standards

Better training in conversational techniques is necessary, even when applying standardised interviewing methodologies

Standardised V Flexible Interviewing

Standardised interviewing reduces interviewer related error, cost, interview length, interviewer training time, and coding time (Fowler and Mangione 1990). Flexible interviewing standardises the meaning of questions so increasing accuracy (Conrad and Shrober 1997).RapportWhether rapport enhances data quality or not also contested. Weiss (1969) rapport causes respondents to ingratiate themselves to interviewers. Holbrook et al. (2003) rapport motivates respondents to give thoughtful, honest responses.

Appropriate sensitivity and adherence to everyday conversational norms, is necessary for any successful interview. MethodThe research drew on interview transcript paradata - the additional data that can be captured during the process of producing a survey statistic (Kreuter 2013: 3).

Data from the 2012 UK Poverty and Social Exclusion Survey (PSE-UK) - a household survey conducted in Britain, carried out by the University of Bristol and administered by NatCen. Survey interviewers instructed to take a rigorously standardised approach during the interview. A sample of 23 survey interviews were selected by the administering organisation for audio-recording. All the interviewers that agreed to participate were experienced interviewers. Examined the extent and content of deviations from the scripted interview though (quantitative) behaviour coding of interviewer behaviour and (qualitative) framework analysis.

Behaviour coding frame - InterviewerCodeDescriptionEWExact Wording Interviewer read question exactly as wordedSCSlight Change Interviewer read question with slight change that did not affect question meaningMCMajor ChangeInterviewer made changes to the question that either altered, or could have altered, the meaning of the questionSQSkipped Question Interviewer entirely omitted an applicable questionNDPNon-Directive Probe Follow-up question/information was not leadingDPDirective ProbeFollow-up question/information was leadingINIInaudible InterviewerInterviewer was not audible on the recordingACAdequate Clarification Explanation was in line with accurate interpretationICInadequate ClarificationExplanation was not in line with accurate interpretationIRIInterprets Response InadequatelyInterviewer mishears, misunderstands etcResultsWhen a question was asked:55% of questions were read with the exact wording (EW)37% were read with a slight change (SC)7% were read with a major change (MC)1.6% were skipped (SQ). When a probe was made:52% of the time it was a non-directive probe (NDP), as considered to be ideal, 48% of the time a directive probe was used (DP). When a clarification was made in response to a request: 79% of the time it was adequate 21% of the time it was inadequate (IC). How standardised are interviews?The threshold occurrence of non-ideal interviewer behaviour (i.e. major change to question wording, or skipping a question) is suggested to be 15% (Oksenberg et al. 1991; Fowler, 1992). In this analysis, the percentage of non-ideal interviewer behaviour was 9.9%. What is the content of these deviations? (interviewer behaviour only)

Giving emotional supportGiving advice or informationComplimenting the participantMaking jokesSharing the research processRevealing personal information

1. Giving emotional support

F: my sister got diagnosed with lung cancer on Monday evening and shes having a Petra scan later todayI: is that, is it serious? F: Yeah its lung cancer, soWere just trying to find out where its spread to, so were at that stage. I: Oh, Im so sorry. F: Yeah I: Okay. Thats quite a worrying time isnt it, til you find out a proper diagnosis really. F: Yeah and we can start

2. Giving advice or information

I: Oh, theyre nice clocks as well, arent they? Nice ones. Have you had them valued?M: No. No I havent had them valued. I: It might be an idea. M: Yeah. I: Itll invalidate your contents insurance. M: Oh right, okay. I: I was told that the other day, if youve got something thats not, that you havent mentioned, not only do they not pay out but they dont pay out at all.M: Oh right. Oh okay...M: Theres a loophole for everything isnt there?I: Yeah, soM: Ill look in to that.

3. Complimenting the participant

I: Did you make the papier-mch pieces?F: I did [chuckles].I: They're beautiful.F: Do, do you like them?I: Yes [laughs]...I think they're lovely.F: They started life as little pop, little pop bottles.I: Sweet.F: You know the littleI: Yes.F: Wee pop bottles you, children have? I: It's absolutely lovely.F: Thank you [chuckles]. They were a bit fun.

4. Making Jokes

I: Okay, Id like to ask some questions about your use of time. F: Has my husband sent you [Laughing]. I: This is what the recordings for! [Laughing]F: Hes been dying to find this information out for years [Both laughing]

I: For the next question I would like you to imagine that youve come across an item in a shop, or on the internet, that youd really like to have Its not an essential item its an extra. If this happened in the next month how restricted would you feel about buying it? F: I wouldnt, one. Hence my Tefal ActiFry in the corner. [Laughing]I: Have you used it? I: Once or twice. I: Once or twice, what the day you bought it and the day after that? F: Hmm, probably. [Laughs] I: Its always the same isnt it. Like a bread maker, oh yes. F: Under the stairs. [Both laughing]

5. Sharing the research process

F: How many people will you be seeing tonight?I: Oh, well if that computer's giving me a bit of jip, I'll I'll call it a draw. I still say there's something wrong with it and they keep saying, oh it's just the security. Every computer Ive, the one before that was brilliant. It used to start up in no time andF: Yeah.I: Never had any trouble with it. This one keeps freezing.F: Not fun is it?

6. Revealing personal informationI: I drive; I do a lot of mileage. So I drive old cars now because Ive wrecked two or three good cars [male respondent laughs]. Ive got the estate car which I normally use when Ive got my dogs. And I use the little car when Im doing longer distances 'cause it's cheaper to run.M: Yeah.I: But they're old cars and Ill drive them 'til they're finishedM: Yeah.I: and get the scrap value, get another oneM: Yeah.I: and then when Im not doing as much mileage, Ill probably get a better car unless I win the lottery.M: Yeah.

Not enough standardisation?I: I interviewed a family of four yesterday and they werent on, in a great area, what do they call, like a council house. Two children living at home but they were all working and no benefits at all. Which is really unusual, not even child benefit because the children were too old. And, ermF: Its not that unusual, because when we were growing up thats what happened to us. I: Its very unusual now. F: [laughs] Is it? I: I interview 40 or 50 households a month and its very unusual, very, very unusual. And, erm, but I mean one was, the daughter was a team leader of 30 odd in this pick and pack place for [NAME OF WORKPLACE], and had to work bank holidays as unpaid, one shift every bank holiday is unpaid, and take home 1400/1500 a month. Its a lot of work for

Too much standardisation?F: I think I told you last time that my niece was murdered...I: Oh right.F: She was murdered three years ago.I: Right.F: DATE her killer was sentenced to 50 years and his sisterI: Yes.F: SAME DATE gave birth to a little girl.I: Okay.M: Ive got to the next question about household problems [unclear phrase] enter one to continue, so...I: Yes, oh sorry, just press one.

All of the interviewers sampled, at some point, took a conversational approach and this was often carried out in a skilled manner, though not always. Survey organisations training appeared to focus primarily on developing interviewers standardised interviewing skills. Perhaps, left interviewers insufficiently prepared with regard to those moments where more conversational interviewing skills were needed. Illustrates the wide variety of interviewing approaches adopted, even when interviewers are instructed to use standardised interviewing methods in training. Whether we use conversational or standardised approaches, the level of skill of the interviewer is extremely important, both in terms of increasing accuracy and facilitating cooperation,