Comparing interactive online and

face-to-face repertory grid interviews

in terms of construct quality and

practicability

Matthias Kaulartz & Mark Heckmann University of Bremen, Germany

EPCA, Brno, June 21, 2014

ca.$45%$of$*me$spent&on&grid&administra.on&

movie&pilot&&300&par.cipants&

h"p://www.intanges.com/relaunch/index.php?id=8&L=1=

Tourism&in&EU&&1,000&par.cipants&

(& )&

...&

So>ware&for&automated&administra.on&

...&large&scale&via&the&web&

Research&on&automated&grid&administra.on&is&sparse&

•  Research&by&Gaines&&&Shaw&in&the&80s&(1980;&e.g.&PEGASUS),&but&no&systema.c&research&on&outcome&quality&

•  Van&de&Kerkhof&et&al.&(2009,&p.&430):&– “Quite&some&constructs&men.oned&were&unclear&to&the&analysts&of&the&research&team.&Many&were&nonVbipolar&....&Moreover,&in&a&faceVtoVface&interview&the&interviewer&can&check&if&two&constructs&are&similar&to&the&respondent.”&

Current&situa.on&

Automated&grid&administra.on&is$used$by&commercial&companies&and&par.ally&in&research,&

but&there&is&virtually&no$research$on&outcome&quality&of&online&grid&administra.on&

Research&ques.ons&• How&can&we&do&“good”&online&repertory&grid&administra.on?&•  Is&it&possible&to&get&results&comparable&to&faceVtoVface&interviews?&

&

Online&grid&situa.on&

No&chance&to&ask&for&clarifica.on&

Quality&of&verbal&labels&crucial&to&infer&meaning&

Here:&Quality&of&constructs&refers&to&quality&of&verbaliza.on&

Content&analysis&of&constructs&

nonV&bipolarity& repeated&

constructs&

poor&verbaliza.ons&

mortality&

Virtual&interviewer&as&

harness&

unintended&ra.ng&

behaviour&

Pretest&&

Iden.fy&main&usability&issues&to&consider&in&so>ware&design&

The&gridhub&so>ware&

Study&Design&•  n&=&80&•  Two&groups&(faceVtoVface&&&online)&•  31&+&49&subjects&&

•  Average&age:&24,1&years&•  Average&number&of&shoes:&21,6&• !&516&constructs&in&total&•  Three&expert&raters&rate&construct&on&four&dimensions&of&construct&“quality”&

1.&Understandability&(of&verbaliza.on)&

Is&the&seman.c&verbaliza.on&of&the&constructs&clear&to&understand&for&a&third&person?&

&Example:&Good:&“This&shoe&makes&me&feel&comfortable&–&makes&me&feel&uncomfortable”&Bad:&“This&shoe&is&so&‘shoeVish’&–&this&one&isn’t”&

2.&OneVdimensionality&

Has&the&construct&just&one&dimension&or&does&one&of&the&poles&include&more&than&one&aspect?&

Example:&Good:&& &“This&shoe&is&for&daily&use&–&only&for&& &&&special&occasions”&

Bad:&& &“Feels&comfortable,&looks&good&and&is&& &&&cheap&–&…”&

3.&Appropriate&level&of&abstrac.on&(with&respect&to&aim&of&study)=

&&Highly&abstract&

Very&concrete&

$

„I&like&it&–&I&don‘t&like&it“&

$

„Shoe&–&Gumboot“&&

“Makes&me&friendly&and&selfVconfident&V&…”&

4.&Overall&ra.ng&of&construct&quality&

“Intui.ve”&score&by&expert&raters&with&experience&in&grid&studies&using&content&analysis&

Examples:&Good:&&“Makes&me&feel&happyVgoVlucky&–&&& &Makes&me&feel&weighted”&

Bad:&&“High&or&low&heels&–&Medium&height&heels”&

Dimensionality:&Is&the&construct&oneVdimensional?&

***&***&Δ=33,5%&

***&Δ=29,3%&Δ=27,0%&Δ=31,8%&

***&

Understandability:&How&understandable&is&the&construct&for&the&rater&on&a&scale&from&1&to&5?&

*&*&d=.28&

*&d=.26&

d=.27&

Reverse&effect&(!)&

Level&of&Abstrac.on:&Is&the&level&of&abstrac.on&in&the&medium&range?&

How&is&the&overall&quality&of&construct&verbaliza.on?&

*&*&

*&*&

d=.46&

d=.33&

d=.42&d=.28&

Results&in&a&nutshell&•  Online&grid&performs&significantly&worse&on&oneVdimensionality&and&overall&quality&•  No&significant&differences&for&abstrac.on&•  Reverse&effect&for&understandability&(?)&

!&While&faceVtoVface&interviews&achieve&bever&results,&online&administra.on&is&able&to&generate&quality&verbaliza.on&of&constructs&

What’s&next?&• Which&design&elements&help&to&…&

&…&generate&oneVdimensionality?&&…&systema.cally&steer&the&abstrac.on&level?&

•  Differences&with&regard&to&other&grid&measures&of&interest&(e.g.&complexity)&•  How&can&we&make&online&grid&administra.on&(more)&fun?&

Gaines,&B.&R.,&&&Shaw,&M.&L.&G.&(1980).&New&direc.ons&in&the&analysis&and&interac.ve&elicita.on&of&personal&construct&systems.&Interna?onal=Journal=of=ManCMachine=Studies,&13(1),&81–116.&

Honey,&P.&(1979).&The&repertory&grid&in&ac.on:&How&to&use&it&to&conduct&an&awtude&survey.&Industrial=and=Commercial=Training,&11(11),&452–459.&doi:10.1108/eb003756.&

Russell,&C.&G.,&&&Cox,&D.&N.&(2003).&A&computerised&adapta.on&of&the&repertory&grid&methodology&as&a&useful&tool&to&elicit&older&consumers’&percep.ons&of&foods.&Food=Quality=and=Preference,&14(8),&681–691.&

Van&de&Kerkhof,&M.,&Cuppen,&E.,&&&Hisschemoller,&M.&(2009).&The&repertory&grid&to&unfold&conflic.ng&posi.ons:&The&case&of&a&stakeholder&dialogue&on&prospects&for&hydrogen.&Technological=Forecas?ng=and=Social=Change,&76(3),&422–432.&

&&&

References&

Thanks!