8/11/2019 principles of survey - part 1.pdf

1/3

S o f t w a r e E n g i n e e r i n g N o t e s v o l 2 6 n o 6 N o v e m b e r 2 0 0 1 P a g e 1 6

A C M S I G S O F T

P r in c i p le s o f S u r v e y R e s e a r c h

P a r t 1 T u r n in g L e m o n s in to L e m o n a d e

Sh ad L aw ren ce Pf leeg er Barb ara A . K i tch en h am

Systems/So f tware , I n c . Dep ar tmen t o f Co m p u ter Sc ien ce

Wa shington, DC, US A Keele Univers i ty , Staf fordshire , UK

s. p f leeger@ieee.org barbara@c s, keele .ac , uk

S e t t in g t h e s c e n e

S urveys a re p robab ly the m o s t com m o nly-used re sea rch m e thod

wor ld-wide . S urvey work i s v is ib le no t on ly because we see m any

exam p les o f i t in sof tware eng inee r ing re sea rch , bu t a l so because

we a re o f ten a sked to pa r t i c ipa te in surveys in our p r iva te

capaci ty, as e lectors , consum ers , or service users . Thi s wide-

spread use o f su rveys m ay g ive us the im pres s ion tha t su rvey-

based re sea rch i s s t ra igh t fo rward , an easy op t ion fo r re sea rche rs

to ga the r im por tan t in form a t ion about p roduc t s , con tex t ,

p roces ses, worke rs and m ore . In our pe rsona l expe r ience wi th

apply ing and eva lua t ing re sea rch m e thods and the i r re su l t s , we

ce r ta in ly d id no t expec t to encounte r m a jor p rob lem s wi th a

survey tha t we p lanned , to inves t iga te i s sues a s soc ia ted wi th

technolo gy adopt ion . Th i s a r t i c le and subsequent ones in th i s

se r ie s desc r ibe how w rong we were . We do no t wan t to g ive the

im pres s ion tha t the re i s any way of tu rn ing a bad survey in to a

good one ; i f a su rvey i s a l em on, i t s t ays a l em on. However , we

be l ieve tha t l ea rn ing f rom our m is takes i s the way to m ake

lem on ade f rom lem ons . S o th i s s e r i e s o f a r t i c le s sha res wi th you

our l e s sons l ea rned , in th e hope of im prov ing survey re sea rch in

sof tware eng inee r ing .

W e b e g a n o u r i n v e s t i g a t i o n b y r e v i e w i n g w h a t i s k n o w n a b o u t

technolo gy t rans fe r. We fou nd tha t , a few yea rs ago , Ze lkowi tz ,

Wal lace and Bink ley (1998) surveyed prac t i t ione rs to de te rm ine

the i r co nf idence in d i f fe ren t types o f em pi r ica l eva lua t ions a s the

bas i s fo r t echnolo gy adopt ion dec i s ions . The i r f ind ings conf i rm ed

wh a t we have long suspec ted : the ev idence p roduced by the

resea rch com m u ni ty to suppor t t echnology adopt ion is no t the

k ind o f ev idence be ing sought by p rac t it ione rs. To bu i ld on

Ze lkow i tz e t a l . ' s work , we con tac ted the pub l i she r o f Appl ied

S of tware Deve lopm ent , a news le t t e r whose reade rsh ip inc luded

pr im ar i ly sof tware p ro jec t m anagers . We wanted to do a fo l low-

u p s u r v e y o f m a n a g e r s , t o f i n d o u t w h a t k i n d s o f e v al u a ti o n s t h e y

m a k e o f p r o p o se d t e c h n o l o g ie s , a n d w h a t k i n d s o f e v i d e n c e th e y

re ly on fo r the i r t echnology dec i sions . However , wh en we

a d m i n i s t e r e d o u r s u r ve y

and ana lyzed the re su l t s , we rea l i zed tha t we had m ade e r ro rs in

survey des ign , cons t ruc t ion , adm in i s t ra t ion and ana lys i s tha t

rende red our re su l t s inconc lus ive a t bes t.

Never the le s s . our l em on o f a su rvey gave us som e use fu l ins igh t s

in to the r igh t and wrong ways to do survey re sea rch . Th i s a r t i c le

i s the f i r s t o f s ix pa r t s about how to do use fu l su rvey re sea rch .

We use our ow n work , p lus the work o f seve ral o the r re sea rche rs ,

to i l lus t ra te the a t t rac t ions an d p i t fa l ls o f the survey t echn ique .

T o u n d e r s t a n d h o w s u r v ey s f it i n t h e l a r g e r s c h e m e o f e m p i r i c

inves t iga tion , we inv i t e you to read the s e r i e s we wro te , s t a r t ing

Decem ber 1995 , in S of tware Eng inee r in g Notes . The re ,

desc r ibed s eve ra l em pi r i ca l m e thods , inc lud ing case s tud i

fac to r ana lyses , expe r im ents and surveys , to he lp you dec i

wh ich t echn ique is appropr ia te in wh ich s i tua t ion . Here ,

a s s u m e y o u h a v e d e c i d e d to d o a s u r v e y , a n d w e f o c u s o n h o w

organ ize , adm in i s te r and ana lyze one so tha t you ge t use f

meaningful resul ts .

W h at our se r ies w i ll d iscuss

This f i r s t ins ta l lm ent o f our s e r i e s desc r ibes th ree surveys : t

repor ted in the l i t e ra tu re and the one we a t t em pted recen t ly . W

provide you wi th a gene ra l ove rv iew of each survey ; we wi l l u

pa r t i cu la rs o f each survey in subsequent ins ta l lm e nts a s exam pl

o f th e s t at e o f t h e p r a c ti c e a n d h o w i t m i g h t b e i m p r o v e d . W

also sugges t gene ra l re fe rence m a te r ia l fo r s tudy ing surv

techniques .

S ubsequent ins ta l lm ents wi l l focus on these i s sues :

S urvey des ign , inc lud in g d i s cus s ions o f s am ple s

and re sponse ra te

Ques t ionna i re des ign and cons t ruc t ion

I s sues in survey adm in i s t ra t ion , inc lud i

m ot iva t ing re spondents , su rveyo r b ia s , p re - and p i

t e s t s , su rvey docum enta t ion , and survey re l i ab i l

a n d v a l i d it y

Da ta analys is .

P o p u l a t io n a n d s a m p l i n g

W h a t i s a s u r v e y ?

To beg in , l e t us rev iew exac t ly wh a t a su rvey i s. A survey is n

jus t the ins t rum ent ( the ques t ionna i re o r check l i s t ) fo r ga the r i

in form a t ion . I t i s a com p rehens ive sys tem for co l l ec t i

in form a t ion to desc r ibe , com pare o r exp la in knowledge , a t t i tud

and behav ior . Thus , the survey ins t rum ent i s pa r t o f a l a rg

survey proces s w i th c lea r ly -de f ined ac t iv i t i e s:

1. Set t ing specif ic , mea surab le object ives

2 . P lann in g and s chedul ing the survey

3 . Ensu r ing tha t appropr ia te re sources a re ava i l ab le

4 . Des ign ing the survey

5 . P repa r ing the da ta co l l ec t ion ins t rum ent

8/11/2019 principles of survey - part 1.pdf

2/3

S o f t w a r e E n g i n e e r i n g N o t e s v o l 2 6 n o 6 N o v e m b e r 2 0 0 1 P a g e 1 7

CMSIGSOFT

6 . V a l i d a t i n g t h e i n s t r u m e n t

7 . S e l ec t i ng pa r t i c i pam s

8 . A d m i n i s t e ri n g a n d sc o r i n g t h e i n s t r u m e n t

9 . A n a l y z i n g t h e d a t a

10 . Rep or t i ng t he r e su l t s .

T he r e a r e s eve r a l d i f f e r en t t ypes o f su r veys . S ur veys can be e i t he r

supe r v i sed o r no t , depend i ng on t he ob j ec t i ves and t he r e sour ces

ava i l ab l e . I f supe r v i sed , we can a s s i gn one su r vey r e sea r che r t o

e a c h r e s p o n d e n t , t o e n s u r e t h a t t h e r e s p o n d e n t u n d e r s t a n d s e a c h

q u e s t i o n a n d p r o v i d e s a n a n s w e r . T e l e p h o n e i m e r v i e w s a re o f te n

o f t h is t y p e , w h e r e a q u e s t i o n e r w o r k s o n e - o n - o n e w i t h a

r e s p o n d e m t o e l i c it a n s w e rs . O r a s u r v e y c a n b e a d m i n i s te r e d t o a

gr oup , w i t h a su r vey r e sea r che r ava i l ab l e t o c l a r i f y and e l abor a t e

o n t h e i n s t r u c ti o n s i n t h e s u r v e y i n s t ru m e m . S o m e s u r v e y s a r e

semi - supe r v i sed , wher e a r e se a r che r exp l a i ns t he ob j ec t i ves and

f o r m a t , p e r h a p s w o r k i n g t h r o u g h s o m e s a m p l e q u e s t io n s , b u t t h e n

l e a v e s th e r e s p o n d e m s t o p r o v i d e i n f o r m a t i o n o n t h e i r o w n .

W e a r e a l l f a m i l i a r w i t h u n s u p e r v i s e d s u r ve y s : t h e a u t o m a t e d -

v o i c e t e l e p h o n e c a l l s i n t h e m i d d l e o f d in n e r , o r t h e m a i l e d

q u e s t i o n n a i r e ( s o m e t i m e s w i t h a p r i z e o r a p o u n d c o i n ) r e q u e st i n g

r esponses abou t l i f e s t y l e , t r ave l hab i t s , o r f ood p r e f e r ences , f o r

e x a m p l e .

A s n o t e d a b o v e , t h e f i r s t s te p i n b e g i n n i n g a n y s u r v e y r e s e a rc h ( o r

any r e sea r ch , f o r t ha t ma t t ed) i s s e t t i ng ob j ec t ives . E ac h

ob j ec t i ve is s i mpl y a s t a t em ent o f t he su r vey ' s exp ec t ed ou t comes .

F or i ns t ance , a su r vey may hope t o i den t i f y t he mos t use f u l

f e a t u re s o f a f r o m - e n d d e v e l o p m e n t t oo l , o r th e m o s t c o m m o n

t r a i n i n g n e e d s f o r n e w h i r e s . I t i s v e r y i m p o r t a n t t h a t t h e

s t a t em em of ob j ec t i ves i nc l ude de f i n i t i ons o f a ll po t en t i a l l y

a m b i g u o u s t e rm s . T h e s e d e f in i t io n s d r i v e n o t o n l y t h e

i n s t r u m e n t ' s d e s i g n a n d d e v e l o p m e n t b u t a l so t h e r e s p o n d e n t s '

u n d e r s t a n d i n g o f th e t e r m i n o l o g y .

W h e r e d o t h e se o b j e ct iv e s c o m e f r o m ? S o m e t i m e s t h e y a r e

d e r i v e d f r o m a p e r c e i v e d n ee d . S o m e o n e i n y o u r o r g a n i z a t i o n

w a n t s t o k n o w s o m e t h i n g a b o u t t h e o r g a n i z a t io n , t h e s t a t e o f t h e

pr ac t i ce , t he e f f ec t i venes s o f a new t echno l ogy , o r the e xpe r i ence

base o f i t s mem ber s , f o r examp l e . A t o t he r t i mes , an ob j ec ti ve

m a y b e d e t e r m i n e d b y a l i t e r a t u r e s e a r c h t o f i n d o u t w h o i s

w r i t i n g a b o u t w h i c h n e w t e c h n o l o g i e s a n d w h e r e t h e r e a r e g a p s i n

cur r en t know l edge . E xp er t s can a l so p r ov i de ob j ec t ives . F or

e x a m p l e , r e s e a r c h e r s p l a n n i n g a s u r v e y a b o u t s o f tw a r e t e st i n g

t echno l og i es may i n i t i a l l y canvass t e s t i ng expe r t s t o de t e r mi ne

w h a t i n f o r m a t i o n is n e e d e d a b o u t w h i c h te c h n o l o g ie s . T h i s k i n d

of so l i c i ta t i on i s somet i mes don e i n a f ocus g r oup , wher e

pa r t i c i pan t s b r a i ns t o r m abou t f u t u r e poss i b i l i t i e s o r d i f f e r en t

o u t c o m e s . W h e n t h e r e i s d i s a g r e e m e n t a b o u t w h a t t h e o b j e ct iv e s

s h o u l d b e , s o m e t i m e s w e f o r m a c o n s e n s u s p a n e l t o e n c o u r a g e

p e o p l e t o c o n v e r g e o n o n e o r t w o i d ea s .

I t i s e s sen t ia l t o devo t e enoug h t i m e t o t he ob j ec t i ves so t ha t t hey

a r e c l ea r an d m easur ab l e , becau se t he ob j ec t ives a r e e s sen t i a l f o r

a l l subsequen t su r vey p r oces s ac t i v i ti e s . F i r s t , t he ob j ec ti ves

d e t e r m i n e w h a t t h e s u r v e y w i ll a s k , o f w h a t p o p u l a t io n , a n d w h a t

i n f o r m a t i on w i l l be co l lec t ed . S econd , t he ob j ec t i ves a r e usua l

r e p h r a s e d a s r e s e a r c h q u e s t i o n s o r h y p o t h e s e s , a n d t h e y h e l p

s u g g e s t w h i c h a r e t h e d e p e n d e n t a n d i n d e p e n d e n t v a r i a b le s o f t

i nves t i ga ti on . T h i r d , t he ob j ec t ives con f i r m whe t h e r a su r vey

t he appr opr i a t e t ype o f emp i r i ca l i nves t i ga t ion . T h a t i s , i f t

ob j ec t i ves a r e f uzzy , t hen we may r equ i r e a d i f f e r en t r e sea r

me t hod .

Descr iption o f our su rvey

T o see how ea sy i t can be t o f a l l in t o s eve r a l su r ve y tr aps , i t

u s e f u l t o u n d e r s t a n d t h e e v o l u ti o n o f o u r o w n s u r v e y . W e h

n o t i ce d t h a t m a n y n e w s l e tt e rs o f t e n i n c l u d e r e a d e r s u r v e y f o r m

s o m e o f w h o s e q u e s t i o n s a n d a n s w e r s c o u l d p r o v i d e u s e f u l i n s i g

i n t o m a n a g e r s ' d e c i s i o n - m a k i n g p r o c e ss e s . W e a p p r o a c h e d t

p u b l i s h e r o f

Applied Software Development;

h e w a s e a g e r

c o o p e r a t e w i t h t h e r e s e a r c h c o m m u n i t y , a n d h e a g r e e d t o i n s e r t

o n e - p a g e s u r v e y i n t h e n e w s l e t t e r a n d g a t h e r t h e r e s p o n s e s f o r u

As a r e su l t , we t ook t he f o l l owi ng s t eps :

1 . W e d e s i g n e d a s u r v e y f o r m a n d a s k e d s e v e r a l o f o

co l l eagues t o c r i t i que i t. T h e su r vey a sk

r e s p o n d e n t s t o e x a m i n e a l is t o f t e c h n o l o g i e s a n d t

u s i f t h e t e c h n o l o g y h a d b e e n e v a l u a t e d a n d i f i t h

b e e n u s e d. I f i t h a d b e e n e v a l u a t e d , w e a s k e d t

r e s p o n d e n t s t o d i s t in g u i s h b e t w e e n a s o

eva l ua t i on , such a s a su r vey o r f ea t u r e ana l ys i s , a

a h a r d e v a l u a ti o n , s u c h a s f o r m a l e x p e r i m e n t

case s t udy .

2 . W e t e s t e d t h e r e s u l ti n g s u r v e y f o r m o n a c o l l e a g

a t L u c e n t T e c h n o l o g i e s . W e a s k e d h i m t o f i ll o

t h e s u r v e y fo r m a n d g i v e u s f e e d b a c k o n t h e c l a r

o f th e q u e s t i o n s a n d r e s p o n se s , a n d o n t h e t i m e

t o o k h i m t o c o m p l e t e th e f o r m . B a s e d o n h i s v e

pos i t i ve r eac t i on t o our ques t i onna i r e , we submi t t

a s l i gh t l y r ev i sed su r vey to t he n ews l e t t e r pub l i she r

3 . T h e pub l i she r t hen r ev i sed our su r vey , sub j ec t t o o

appr ova l , so t ha t i t woul d f i t on on e page o f h

newslet ter .

4 . T h e s u r v e y f o r m w a s i n c l u d e d i n a l l c o p ie s o f

s u m m e r 1 9 9 9 i s s u e o f Applied Softwa

Development.

O f t h e s e v e r al t h o u s a n d p o s s i b le r e c i p i e n t s o f

App lied Softwa

Development

o n l y 1 71 r e s p o n d e d b y s e n d i n g t h e i r s u r v e y f o

back ; t hus , our re sponse r a t e was l ow , whi ch i s typ i ca l i n t h

t y p e o f s u r v ey . T h e s t a f f a t

App lied Software Developm

t r ans f e r r ed t he da t a f r om t he su r vey shee t s t o a sp r eadshee

w h i c h b e c a m e t h e b a s is f o r o u r a n a l y s i s.

Descr ipt ion of the Lethbridge surv ey

I n 1 9 9 8 , L e t h b r i d g e ( 1 9 9 8 a n d 2 0 0 0 ) c o n d u c t e d s u r v e y s t o h e

h i m u n d e r s t a n d t h o s e a r e a s w h e r e p r a c t i t i o n e r s f e e l t h e y n e

m o r e o r b e t t e r e d u c a t io n . T h e g o a l o f t h e s u r v e y s w a s t o p r o v

i n f o r m a t i o n t o e d u c a t i o n a l i n s t i t u t i o n s a n d c o m p a n i e s a s t h

8/11/2019 principles of survey - part 1.pdf

3/3

A C M S I G S O F T S o f t w a r e E n g i n e e ri n g N o t e s v o l 2 6 n o 6 N o v e m b e r 2 0 0 1 P a g e 1 8

plan curr icula and t rain ing programs. A secondary goal involved

providing da ta that w i l l assi s t educators and pract it ioners in

evaluat ing e xis t ing and propose d curricula.

Lethbr idge and his team recrui ted par t icipants for the surveys in

two ways: by approaching com panies d i rectly and asking them to

par t icipate, and by adver t i s ing for par t icipants on the Web. To

determ ine the e f fects of formal educat ion, Lethbr idge presented

the responden ts wi th a l i s t of topics related to comp uter science,

mathe mat ics and business . For each topic, the respondent was

asked How m uch d i d you learn abou t t h is i n your fo rmal

educat ion? The choices for answe rs ranged on a s ix-point Liker t

scale f rom learned nothing to learne d in depth . Other

quest ions included

W hat i s your current know ledge about th i s

cons i der ing w hat you have l earned on t he j ob as w el l

as forgot ten?

Ho w useful has th is speci f ic mater ial been to you in

your career?

Ho w useful wou ld i t be (or have been) to learn mo re

about th i s (e.g . addi t ional courses)?

Ho w mu ch inf luence has learning the mater ial had

on yo ur th inking ( i .e . your approach to problem s and

your general maturi ty) , whethe r or not you have

di rect ly used the detai l s of the m ater ial? Please

consider inf luence on both your career and other

aspects of you l ife.

Descript ion o f the Finnish study

Recent ly , Ropponen and Lyyt inen (2000) descr ibed thei r

exanf inat ion of r i sk ma nagem ent practices. The y adm inistered a

survey addressing two ove ral l questions:

* W hat a re t he componen t s o f so f tware deve l opmen t

r i sk?

Wha t r i sk managem en t p rac t ices and env i ronmen t a l

cont ingencies help to address these components?

To f ind out the answe rs , the researchers m ai led a quest ionnai re to

each o f a p re- se l ec t ed sampl e o f members o f t he F i nn i sh

Informat ion Processing Associat ion whose job t i tl e was

'manager or equivalent. How ever , they sent the quest ionnai re to

at most two mana gers in the sam e company.

Ropponen and Lyyt inen asked twenty quest ions about r i sk by

present ing scenar ios and asking the respondents to rate thei r

occurrence wi th a f ive-point Liker t scale, ranging f rom hardly

ever to almost always. For examp le, the scenar ios included

Yo ur project i s canceled before complet ing i t .

and

Subco nt racted tasks in the project are performed as expected.

The researchers posed addi t ional quest ions relat ing to

organizat ional character i s t ics , such as the organizat ion ' s s ize,

industry , type of systems developed, and

contr ctu l

ar r angemen

They also sought technology character i s t ics , such as the newne

of the technology, the com plexi ty and n ovel ty of technologic

solut ions, and the process technologies used. Final ly , they ask

quest ions about the responden ts themselves: thei r exper ien

wi th d i f ferent s izes of projects , thei r educ at ion, the so f tware use

and exper ience wi th project management .

References for survey techniqu es

Subsequent instal lments of th i s ser ies wi l l focus on surv

research in sof tware engineer ing. To f ind mo re gener

informat ion about surveys, we recommend the fo l lowi

publications.

Arlene Fink,

The Survey Handbook

Sage Publ icat ions, 1995.

Th i s comprehens i ve book exp l a i ns how t o ask su rv

questions, how to design survey s, how to sample

surveys, how to analyze survey data, and how to repo

on surveys.

Li nda Bourque and Eve F i e l der ,

How to Conduct Se

administered and Mail Surveys

Sage Publ icat ions, 1995.

This book looks in detai l at sel f -administered and ma

surveys.

Mark Li twin,

How to Measure Survey Reliabil i ty and Validit

Sage Publications, 1995.

This book def ines survey rel iabi l i ty and val id i ty a

exp l a ins how t o m easu re t hem.

Bibliography

Let hb ri dge , T i mot hy , A su rvey o f t he r e l evance o f com pu t

science and sof tware engineer ing educat ion,

Proceedings o f t

llth International Conference on Software Engineering

I E E

Com puter Society Press , 1998.

Lethbr idge, T imothy, W hat knowledg e i s impo rtant to a sof twa

professional ,

IEE E Computer

May 2000.

Ropponen , J . and K. Lyy t inen , Com ponen t s o f so f t wa

devel opmen t r i sk : How t o add ress t hem. A p ro j ec t manag

survey,

IEEE Transactions on Software Engineering

26(2

February 2000.

Zelkowi tz, Marvin V. , Dolores R. Wal lace and David Binkle

'Understanding the cul ture clash in sof twa re engineer in

technology t ransfer , Universi ty of M aryland tec hnical repor t ,

June 1998.