Critical Lessons for Training Bilingual

Interviewers on a Longitudinal Study

May 20, 2012

Presentation to the 67th Annual Conference of the American Association for

Public Opinion Research

Prepared by

Rebecca Weiner ● Joe Baker ● Cleo Jacobs Johnson ● Shawn Marsh

2

Introduction

Five trainings on Building Strong Families

(BSF) in-home assessment held between 2009

and 2011

Adjustments were made with each successive

training to improve certification rates

Challenges certifying and retaining bilingual

staff

A bilingual-only training was held to address

these challenges

Multiyear, multisite project sponsored by the

Administration for Children & Families

Goal: To learn whether well-designed voluntary

interventions can help unwed parents build

stronger relationships with each other and

their children

Data collection: 2005 to 2011

3

Study Overview: Building Strong Families

4

Building Strong Families: Sites

Seven sites offered marriage and relationship

counseling:

– Georgia

– Maryland

– Louisiana

– Florida

– Indiana

– Oklahoma

– Texas

Hispanic sample concentrated in Georgia,

Florida, Texas, and Oklahoma

5

Study Participants

Couples were eligible to participate if they were

– Expecting a child OR had a child younger than three

months old

– Not married, or married after conception of the focal child

Couples were randomly assigned to a control or

program group

– Control group: 5,098 participants

– Program group: 5,108 participants

– Total: 10,206 participants

1,260 participants indicated that their primary

language was Spanish

6

Couples were followed for three years after entering

the study:

Data Collection Mode Time Frame

Enrollment and Baseline Data

Collection

Paper and

pencil

2005–2008

15-Month Follow-Up CATI 2006–2009

36-Month follow-Up CATI 2008–2011

In-Home Assessments In person 2009–2011

Data Collection: Overview

7

In-Home Assessment: Overview

Purpose: To measure the quality of the parent-

child relationship

Trained field interviewers conduct

assessments in respondents’ homes with focal

child and mother and/or father

Interviewers follow a written protocol and

conduct multiple assessments

Administration time: approximately 1 hour

8

In-Home Assessment: Protocol

Assessment Dimension Measured

Peabody Picture Vocabulary Test 4

(PPVT-4)

Receptive Language

Development (English)

Test de Vocabulario en Imagenes

Peabody (TVIP)

Receptive Language

Development (Spanish)

Walk a Line Slowly Self-Regulation

Two Bags Task Quality of Parent-Child

Interaction

Toddler Attachment Sort-45

(TAS-45)

Parent-Child Attachment

9

Initial Training Model: All Staff

Three days of training

Direct instruction, demonstration, paired

group practice

Live and video certification

– Certification conducted with volunteer families

on second day of training

– Interviewers submitted a video of an assessment

with a volunteer family after training

– Mathematica staff scored administration

10

Initial Training Model: Bilingual Staff

English monolinguals and bilingual observers

attended same training

– Entire training conducted in English

– Majority of trainees were English monolinguals

– Bilingual trainees received one extra hour of training

on bilingual protocol

– Bilingual trainees required to submit a post-training

certification video in English AND Spanish

Assumption: bilingual interviewers can

translate what they learned when administering

Spanish protocol

11

Initial Training Results: Bilingual Staff

Two bilingual staff attended training

One trainee failed certification

One trainee passed certification, but later left the

project

– Felt that assessment was more complicated than she

expected

– Did not produce enough work

12

Adjustments to Training Model

Project staff assisted field manager with

interview process and hiring decisions

– Hired staff with skill set best suited to the task

Trainees were sent a video of Mathematica staff

administering assessment with a volunteer

family before training

– Trainees had a better understanding of what they would

be doing and the flow of the protocol

13

Adjustments to Training Model

Lengthened training to four days

– More time devoted to challenging assessments

Increased paired practice time

– More time to address areas of concern before

certification

Moved certification to last day of training

– More time to become familiar with protocol

Asked trainees to schedule a family for video

certification before training

– Fewer delays with at-home certification

14

Less than half of the bilingual staff were

certified during the first four trainings

Training Date # Bilingual Trainees

# Passed

Certification

February 2009 2 1

June 2009 4 2

December 2009 2 1

July 2010 1 0

Total 9 4

Challenges Certifying and Retaining

Bilingual Staff

15

Challenges Certifying and Retaining

Bilingual Staff

Bilingual staff did not have time to practice TVIP

or bilingual protocol during training

– Difficulty with protocol evident in post-training

certification video

– TVIP was often scored incorrectly

– Difficulty with protocol resulted in low certification and

retention rates

We decided to hold a bilingual staff-only training

to address these challenges

16

Adjustments to Training Model: Bilingual

Staff

Hired staff that could travel to all sites

– Few qualified applicants lived local to BSF sites

– Hired staff with best skills, regardless of location

– Traveled with them to sites with most need

Pre-training video demonstration in Spanish

Adjustments to Training Model: Bilingual

Staff

17

Recruited bilingual staff to assist with practice

and certification

All practice sessions conducted in Spanish

Recruited bilingual volunteer families for

certification

Devoted more time to TVIP practice and created

a “TVIP Window” to assist with scoring

TVIP Window: Example

18

19

Bilingual Training: Results

All staff were certified!

20

Bilingual Training: Results

Staff from the bilingual-only training

– Had never attended a bilingual-only training previously

and found it beneficial

– Completed 66% of bilingual in-home assessments

– Worked more efficiently than previously trained

bilingual staff

21

Lessons Learned

Bilingual staff benefit from a training geared

specifically to them

Language of administration has an effect on

performance

Interviewers hired to travel to sites worked more

efficiently than those hired local to sites

22

Recommendations for Training Bilingual

Field Interviewers

Training should allow for bilingual trainees to

practice in Spanish

Spanish-speaking project staff should assist

with training

Spanish-speaking families should be used for

certification

23

Thank You

Thank you to the Administration for Children &

Families of the U.S. Department of Health and

Human Services

Thank you to the BSF field interviewers and the

families participating in the BSF study

For additional information, email Rebecca Weiner

at rweiner@mathematica-mpr.com