Writing a ‘Behavioral’CSCW Paper

Dave RandallUniversity of Siegen, Germany(with thanks to John Thomas)

Introduction

This talk covers: • What might a ‘behavioural’ study be? • Important things to do for all behavioral studies• Important things to do for each kind of behavioral

study• Some example problems in papers and how you

might avoid them or solve them

CSCW, CHI and related conferences

• Increasingly eclectic view of our intellectual problems

• More and more ‘acceptable’ domains of study

• E.g. mobilities; internet applications, social media

CSCW and related conferencesReviewers are tough and thorough

– Papers are generally (but not always) reviewed by the leading expert(s) on what you are writing about

– They know the ‘house style’– They know the literature– But there is an element of chance as well- there are intellectual disputes

in CSCW. – If you are going to enter into these disputes, get the argument right.

Best advice: “do good work”– Nothing will save you if you don’t, so the time to think about these

issues is while deciding what to work on, designing studies, etc. – Write it up in a way the community will appreciate– Don’t be stupid or lazy

An (invented) example“In this paper, we follow Daniel Dennett’s ‘intentional’ stance. Dennett argues

that peoples’ attitudes, beliefs and desires can also be found in other, non- human phenomena such as animals and machines. Hence, human beings are like machines because all have these attitudes, beliefs and desires”

• Being an expert in an interdisciplinary field is not easy, but don’t claim to know about things you don’t know about !!!

The Happy Facts…

the number of people who will see your paper at all is probably depressingly small (average no. of citations in CSCW/CHI type conferences 12-15)

BUT you can increase the chances that it will be noticed

– Workshops, panels, demonstrations, SIGs, notes, courses….We are focusing here on papers, but keep other options

in mind

Regardless of TYPE of Behavioral Paper

Things to do: 1. POSITION your paper with respect to other literature.2. Can you relate your domain to a wider area of concern?3. NEVER be too critical of other work- why be a hostage to fortune?4. BUILD ON other work4. Make the ‘gain’ clear. Why is your work important?5. Is the gain empirical/ conceptual/ both?6. Spell out your methodology8. Evidence your findings.9. Tell a good story: An important goal. 10.Make sure your version is as near to finished as possible (although CSCW

allows more generosity)

Regardless of TYPE of Behavioral Paper

Things to do:

11.Treat reviews as useful contributions

12.In the CSCW conference context, this is vital

13.In related conferences, it is still important.

14.If reviewers don’t understand what you are trying to say, this could be your fault!

15.Of course, it may not be …. *sigh*

Regardless of TYPE of Behavioral Paper

Some possible aims:1. Comparison- are you confirming; disconfirming, or discovering something

new?

2. Conceptual development- do you have an interesting and useful way of thinking about the issues?

3. Methodological innovation

4. New domain of study

5. New kind of problem- e.g. the virtual world; social media

6. Discussion paper (only well-known people dare this one- feelings can get very strong)

Looking at Behaviour

Basically, three ways:

• Measuring it

• Observing it

• Talking to people about it

Some Types of Behavioral Studies

SurveyInterviewsEthnography/Field StudyLab StudyParticipatory DesignModeling

Surveys, questionnaires and lab studies

• By definition, surveys involve aggregation, and thus sufficiently large populations (sampling/ confidence limits/ correlation)

• They are more often ‘attitudinal’ rather than ‘behavioural’ but can in principle be either, or mixed.

• Do not do this work unless you are statistically competent!!!!

• Can be used to ‘triangulate’ with other methods

– Previous literature in this area– Real world tasks that are important– Design considerations for technology

Surveys etc“The case against Lucia de Berk was built on a suspicious pattern: there were 9

incidents on a ward where she worked, and Lucia was present for all of them. This could be suspicious, but it could be a random cluster, best illustrated by the “Texas Sharp Shooter” phenomenon ….

This is plainly foolish. All across the world, nurses are working on wards, where patients die, and it is inevitable that on one ward, in one hospital, in one town, in one country, somewhere in the world, you will find one nurse who seems to be on a lot when patients die. And did the idea that there was a killer on the loose make any sense, statistically, for the hospital as a whole? There were 6 deaths over 3 years on one key ward where Lucia supposedly did her murdering. In the 3 preceeding years, before Lucia arrived, there were 7 deaths. So the death rate on this ward went down at the precise moment that a serial killer – on a killing spree – moved in.”

Surveys etc“Even more bizarre was the staggering foolishness by some of the statistical

experts used in the court. One, Henk Elffers, a professor of law, combined individual statistical tests by taking p-values – a mathematical expression of statistical significance – and multiplying them together. This bit is for the nerds: you do not just multiply p-values together, you weave them with a clever tool, like maybe ‘Fisher’s method for combination of independent p-values’. If you multiply p-values together, then chance incidents will rapidly appear to be vanishingly unlikely. Let’s say you worked in twenty hospitals, each with a pattern of incidents that is purely random noise: let’s say p=0.5. If you multiply those harmless p-values, of entirely chance findings, you end up with a final p-value of p < 0.000001, falsely implying that the outcome is extremely highly statistically significant. With this mathematical error, by this reasoning, if you change hospitals a lot, you automatically become a suspect.”

(Ben Goldacre: ‘Bad Science’)

Surveys and questionnaires

• The Bible Code

“The Bible code report above lists the terms found … and then gives each term’s r-value in all of scripture, and it’s r-value in the matrix. The r-value is a statistical term for the odds at which that term appeared … for the matrix above; the combined r-value is 12.240, which yields odds of about 1.7 trillion to one.”

“A paper of Witztum, Rips and Rosenberg in this journal in 1994 made the

extraordinary claim that the Hebrew text of the Book of Genesis encodes events which did not occur until millennia after the text was written. In reply, we argue that Witztum, Rips and Rosenberg's case is fatally defective, indeed that their result merely reflects on the choices made in designing their experiment and collecting the data for it. We present extensive evidence in support of that conclusion. We also report on many new experiments of our own, all of which failed to detect the alleged phenomenon.”

Interviews- talking to people• Transcription is important (it’s your evidence!)

• Reviewers will want to know about: Piloting; coding; objectivity and bias sampling. YOUR ANSWER DEPENDS ON YOUR ANALYTIC POSITION

• How many participants is enough? Think about homogeneity and heterogeneity.

• Representativeness tends to be less of an issue

• Think about concept formation, or coding (data needs to be organized)

• Rhetorically, intersperse transcript data with argumentation

Interviews- talking to people• As one said, ‘we’re not that interested. If we didn’t want you doing this work we

wouldn’t have anything to do with you. We get something from it and, as long as we do we’re happy with you.’ As another said, ‘my relationship with the university is brilliant, excellent. I don’t really care what you’re up to as long as I can see the benefit’. None of this is to reject the importance of formal ethical structures. It does, however, point to maintenance issues that exist over and above formal statements of responsibility. These, it has to be said, have to do with personal contact between researchers and participants and on the nature of relationships in the community. Indeed, one of the ways in which participants assess university work is on the basis of judgements they make about individuals involved in the research process. They make frequent reference in our data to the difference between rather anonymous figures who ‘flit in and out, repair stuff and then disappear’ in contrast to, ‘X .. who is such a nice guy .. He’s always very helpful … and he can be pressured [said with a smile]’. We might also point to the fact that our own investigations of these issues prompt, to a degree, positive feedback on relations with the university, as in, ‘it’s nice when you guys are here .. sitting here with coffee and cake … and you always make certain to fit in with all the other things we have to do … that’s very good.’

Ethnography• There are different schools of thought about how ethnography or

fieldwork should be done. It is normal to provide a ‘theoretical’ backdrop (ethnomethodology; grounded theory; DCog; activity theory; ANT)

• CSCW tends (but not exclusively) towards a ‘conversational’ or ‘documentary’ version of evidence

– i.e. that the evidence provided is from recordings of talk (Because it’s difficult to provide a protocol for fieldnotes)

• Increasingly, photographic, screendump and video evidence is used(Note also, digital/ virtual/ trace ethnography)

Ethnography

• Detail, detail, detail !!!!

• Ethnography has no purpose if it only provides generalities

– “We found that people did not always follow the written procedures.” Describe *when* and *how* people avoided those procedures.

Ethnography

• It is the relationship between the particular and the general that is important- your study must fit a ‘CSCW relevant’ issue e.g. a technology

• You can compare data from other cases

• You can compare concepts

• You can provide ‘implications’; evaluate, or critique.

The 3-second ethnography1. ‘Dave’ gets on bus, which is half empty. Finds a place halfway back on the

left, immediately behind a girl who is aged about 12-13 years.2. Notices that a 40 year old man sits next to the girl (1 sec). Dave leans

forward, but says and does nothing.3. The man says to the girl, ‘Hiya, how are you?’ and the girl looks away, out of

the window4. The girl stands up and pushes past the man. He says (loudly), ‘I’m sorry, I

thought I knew you ...’ 5 She says nothing and ignores him, walks towards the back of the bus where

she finds an empty seat where she sits down. (3 secs)(Randall et al, Fieldwork for Design)

6. 5 minutes later, the man stands up to get off the bus. The girl follows him, and gets off at the same bus stop.

Participative Design

• Rather than “studying” people, you partner with potential users and together you design a system

• The ‘behaviour’ here lies in the design process• • Important to report on: – What you learned– How that impacted the design– How your findings may also have implications for other

systems

• You should describe the resulting system

Participative Design

• It is good if you can combine/ triangulate this with additional studies of people using the system

• You should present evidence that there is a real problem to solve– “We wanted to use participative design to incorporate

mobile technology into tea ceremonies.” But why?? What evidence is there that tea ceremonies *need* mobile technology in the first place?

Innovative methods

• Some papers are based on ‘doing something different’– e.g. cultural probes; ‘living labs’

• Assert a problem-– Ethnographic studies are time-consuming and do not meet real-world

needs of rapid innovation

– Users sometimes lack knowledge and imagination

• Assert a solution- new methods, justified through a case study

Rhetorics• Your reviewers need you to be clear about what you want to say and

why it matters

• State your aims clearly in your introduction

• Thread your argument through the paper- there is nothing wrong with careful (but limited) repetition. E.g. ‘as already stated’.

• If not a native speaker, find someone who is

• If writing with others, ensure to integrate ‘style’

Rhetorics• Use references consistently

• Check the literature FROM THE CONFERENCE YOU ARE AIMING FOR and cite it

• Sum up the ‘gain’ in your discussion and conclusion and, if you are generalising, do it here

• Avoid being controversial (big names sometimes get away with this)

Rhetorics- Discussion and Future Work

• Together with the introduction, this is important “framing” information for your paper

• One common pitfall is “underselling” --- that is, not making it clear what part your paper plays in a larger story– Implications for designing CSCW systems– Replicating or generalising – Showing that there are important exceptions to what has been found

before

• Another common pitfall is “overselling” --- claiming importance that is not justified

Questions+ useful links

libgen.info

http://en.bookfi.org