Teaching advanced research and scholarship skills to

postgraduate social science students

Paper presented at the 6th Leeds Learning and Teaching Conference,

January 8th 2009

Linda Evans,School of Education

What do we mean by ‘advanced’ research and scholarship skills? ‘advanced’ is a relative term ‘skills that research students and, indeed,

some academics often find difficult and challenging’ (Evans, 2002)

The basis of my evaluation: my teaching/supervision experience my experience of reviewing submitted bids and

journal articles - and reading published ones! Evans, L. (2002) Reflective Practice in Educational Research:

developing advanced skills, London, Continuum

What specific advanced skills?

Categorisation/classification Conceptual clarity and definitional

precision Formulating theory/theoretical

perspectives and propositions theoretical models

Critical analysis We ask for it, but seldom teach it.

A MA student’s attempt at critical analysis:

an uncorrected extract from a draft essay Evan’s comment that ‘amongst teachers who respected it and

responded well to it, ‘extended’ professionality in headteachers was an effective motivator.’ (Evans, 1998, p130) struck a particular chord with me because I am experiencing a greater sense of motivation since changing schools in September. My new headteacher constantly encourages ongoing reflection on individual practice with a view to ensuring that each child reaches their potential. I am experiencing a much greater sense of motivation as her professionality is very much in tune with my own. Although my last headteacher at my previous school was more of an extended professional than his predecessor, he was unable to filter out his extended professionality to the rest of the school due to his poor delegation skills: he wasted valuable time dealing with things others could easily have managed. At my new school, from the leadership team to the heads of department, the headteacher’s professionality is clear. As a consequence the extended professionals at my new school are more motivated than those with a similar degree of professionality at my previous school.

‘Teaching’ critical analysis:the rationale

Students generally have little or no conception of what is meant by ‘critical analysis’.

They often underestimate the depth and degree of originality that we are, ideally, hoping to see in their work.

They consider themselves to be incorporating critical analysis when, in fact, they are merely offering superficial comments or observations.

They usually are able to identify critical analysis when shown work that includes examples of it.

They naturally find it extremely challenging and difficult to do.

They derive a tremendous sense of achievement from having managed to do it well.

What do we mean by ‘critical analysis’?

questioning what you read in books and articles finding flaws in what you read in books and articles identifying contradictions in arguments and/or

theoretical perspectives thinking of ways in which arguments and/or theoretical

perspectives may be enhanced or developed further explaining intelligently why you approve of, or support,

a particular argument or theoretical perspective explaining intelligently why you disapprove of, or have

reservations about, a particular argument or theoretical perspective

originality of thought – your own ideas developing sound, rational argument and/or discussion

A teaching tool: identifying examples of critical

analysis On a one-to-one basis, using the student’s

own draft essays as a resource.In group teaching: Using extracts from past students’ essays Using extracts from journal articles Using extracts from my own work, to

allow me to explain my thinking and thought processes.

The Literature ReviewPurpose of the literature review: to provide background and

contextual information against which to set your argument

to fuel your argument The purpose is not to show the

tutor how much you’ve read!

Reviewing the Material The literature is a tool. Use it to support your argument. Use it to provide evidence of perspectives

that you can argue against. Use conflicting views and contradictions to

develop discussion. Draw on a reasonably wide range of

authors and texts. Paraphrased references are as good as –

and often better than – quotes.

Using the Literature as a Tool

Do not simply present information found in the literature.

Comment on it thoughtfully and intelligently. Indicate whether you agree with it:

to what extent; why; with what reservations or qualifications.

Argue against it. Support it – but give your reasons for doing so. Suggest modifications to it - but give your reasons

for doing so. Pick holes in it. Compare it with your personal experience. Compare it with conflicting

views/evidence/information. Analyse it critically.

Riddell, S. (1989) Exploiting the Exploited? The Ethics of Feminist Educational Research, in R.G Burgess (ed.) The Ethics of Educational Research, Lewes, Falmer, pp. 77-99

After talking to one mother and father for nearly an hour and a half, I was trying to bring the interview to a close with what I thought was a fairly innocuous question. I asked:‘Are you generally happy with the education Stephen’s getting?’

Mr Gammage, a policeman, replied:‘Well apart from this bloody irresponsible strike that’s been going on…’

Instead of bringing the interview to a calm conclusion, this resulted in a half hour’s denunciation of the entire teaching profession. Analyzing how I responded to male attempts to establish power in the interview, I found that I generally backed off from conflict, and often found it difficult to probe for more detailed answers when what they said was very brief. Although I did not probe the women’s responses either, they certainly offered me a far more intimate view of themselves. Of the three fathers whom I interviewed by themselves, one blocked my questions completely and gave yes/no responses to everything, and another answered very briefly. The headmasters whom I interviewed also provided proficient examples of school politics in operation, spending a very long time avoiding my questions.

From: Evans, L. (2002) Reflective Practice in Educational Research: developing advanced skills, London, Continuum: (Chapter 7: Telling it as it Is)

As a woman – and a feminist – I feel able to question with impunity that may well be denied a man the researcher’s conclusion that these reported incidents demonstrate ‘male attempts to establish power in the interview’. My immediate response to reading this and similar passages in Riddell’s paper was to categorise them as evidence of partisanship and of the author’s having a feminist chip on her shoulder. Certainly, that the researcher’s experiences reflect male dominance – or attempts to dominate - is one possible interpretation, but it is by no means the only interpretation. Ascribing motives to others - particularly in the absence of their own accounts and explanations for them – is an extremely inexact science that, at best, may be described as conjecture and, at worst, as arrogant assumption. It is highly susceptible to misinterpretation that leads to the generation of spurious conclusions. We are treading on thin ice if we subsume our unsubstantiated interpretations of the motives behind the nature of their responses within our data. This involves stepping over the line - albeit a thin one - that separates the manner in which data are collected and provided and the data themselves; it blurs the distinction between process and product. Based on the evidence available in her paper, and bearing in mind that, like any paper, this is unable to provide us with the full picture, I formulated the impression that the author is too ready to ascribe sexist or stereotypical gender-related motives to her research subjects. Her male interviewees seem unable to put a foot right: if they are loquacious and responsive she evidently interprets their responses as domineering, and their reticence is evidently interpreted as attempts to sabotage the interview through non-cooperation. This reflects what I interpret as her blinkered viewpoint, and in a later section I examine ways in which she might have analysed and presented her data in a more balanced way.

From: Evans, L. (2002) Reflective Practice in Educational Research: developing advanced skills, London, Continuum: (Chapter 7: Telling it as it Is)

Riddell’s analysis contrasts sharply with Luttrell’s (2000, pp.513-514) description of how what was effectively the removal of her blinkers allowed her to see and re-analyse her research data in a new light:

Identifying my reluctance to deal with strong emotions and mixed feelings about mothers – what could be called counter-transference in my fieldwork relationships – marked a major breakthrough in my research process. I again returned to all the interview material, and through this recursive process I discovered a range of maternal images and mixed feelings that the women had expressed, but that I had minimized in my analysis of the links between structure, culture and agency.

From: Evans, L. (2002) Reflective Practice in Educational Research: developing advanced skills, London, Continuum: (Chapter 7: Telling it as it Is)

Riddell’s interpretation of her male interviewees’ behaviour would be justifiable if she had applied appropriate controls to, or at least consideration of, other variables together with incorporation of consideration of evidence that supported or conflicted with her interpretation before systematically assessing rival explanations. She presents no evidence of having adopted such a rigorous approach to her analysis.

From: Evans, L. (2002) Reflective Practice in Educational Research: developing advanced skills, London, Continuum: (Chapter 7: Telling it as it Is)

Yet I have to acknowledge another side to the arguments I have presented within my criticism of Riddell’s analysis.

In his outline explanation of consensus theory Bridges (1999, p. 606) warns: Philosophers have tended to be sceptical of consensus theory as an

account of what it means for a belief to be true… Researchers ought perhaps to share especially in this scepticism of

consensus, since it is, arguably, their particular function as intellectual citizens to challenge the easy and self-perpetuating consensus that society creates for itself. At the very least they have to observe that what any group of people believe or agree to be true may rest on, for example, unexamined tradition, the hegemony of a dominant class, the suppression or self-censorship of dissenting opinion or collective hysteria – all the things that intellectuals and researchers have traditionally been expected to subvert.

Applying this consideration to my criticism of Riddell’s work, I am forced to accept the possibility that, in presenting my interpretation of her analysis as the reflection of a blinkered viewpoint, and my own as reflecting the more consensual form of truth, I may have fallen into the trap, to which Bridges refers, of perpetuating a consensus that fails to question - because it fails to see – the hegemony of one or more specific societal groups. In a sense, moreover, I seem to be trapped in a ‘Catch 22’-type situation; by contending that Riddell’s viewpoint is blinkered does not my own viewpoint then risk becoming so?

Morale, Job Satisfaction and Motivation:

Research Questions What factors influence morale?

What factors influence job satisfaction?

What factors influence motivation?

Activity: examine the raw research data presented as example quotes, in order to find answers to the above research questions.

Conceptual clarity and definitional precision

I do not tell students in advance that this is my teaching focus.

Students better appreciate the need for it and the potential difficulties created by its absence if they discover these for themselves.

The substantive focus of the teaching activity is, where possible, appropriate to the content of the module they are following.

Conceptual clarity

adds precision and rigour

facilitates construct validity

is a tool for achieving critical analysis

Construct validity ‘There’s glory for you’, said Humpty Dumpty. ‘I don’t know what you mean by “glory”,’ Alice

said. Humpty Dumpty smiled contemptuously. ‘Of

course you don’t – till I tell you. I meant “there’s a nice knock-down argument for you”.’

‘But “glory” doesn’t mean “a nice knock-down argument”,’ Alice objected.

‘When I use a word’, Humpty Dumpty said in a rather scornful tone, ‘it means just what I choose it to mean – neither more or less.’

(From: Through the looking Glass, by Lewis Carroll)

Construct validity involves consensual acceptance and

understanding of specific terms is threatened when researchers and

research subjects do not share the same interpretation and understanding of key constructs

is threatened when two parties are communicating at cross purposes

can impact upon the whole process of data collection, data analysis, and dissemination

Developing your capacity for pursuing construct

validityThe process has four elements:1. identification of key concepts for

clarification2. conceptualisation and construct

clarification 3. communicating and matching

constructs 4. conceptual clarity and dissemination

Identification of key concepts for clarification

Conceptualisation is the process whereby we develop concepts.

It is the process of attaching meaning to words.

It is also the product of having attached meaning to words: your conceptualisation of ‘milk’, for

example, is the meaning of milk that you have formulated

Identification of key concepts for clarification

Identify what you consider to be the key concepts within your research.

Decide which of these need clarifying: Which concepts are potentially ambiguous? Which concepts are susceptible to being

interpreted differently by different people? The more nebulous the concept, the

more likely it is to need clarifying. Concepts of tangible objects are less

susceptible to multiple interpretation than are intangible concepts.

a study of the causes of student disaffection

Conceptualisation & construct clarification

Examine your selected key concepts: Decide how you interpret or define them. What precisely do I mean by …? It is not enough simply to ‘know’ what you

mean by something – you must be able to communicate that meaning.

Formulate conceptual interpretations or definitions of your key concepts.

Aim to achieve definitional precision.

Achieving definitional precision

What is a definition? stipulative definitions

stipulate, in precise and unambiguous words, what you mean when you use a particular word

ostensive definitions where a word is defined by pointing to the

objects to which it exclusively refers e.g. in defining “education” one might point to

particular activities, or to what goes on in particular schools

Recognising stipulative definitions

Which of these is a stipulative definition of a chair? A chair is a piece of furniture intended for sitting on. A chair is a seat. A chair has four legs, a seat and a back and is

sometimes padded. It is usually made of wood or plastic but other materials can be used.

A chair is intended to seat people. A chair is a piece of furniture that people may sit

on.

Recognising stipulative definitions

Stipulative definitions should explain precisely and unambiguously what something is.

They should not simply describe something. A stipulative definition should be exclusive –

applicable only to what it is that is being defined.

A stipulative definition should encompass explanation of all of the features that are integral to what is being defined.

It should incorporate consideration of the necessary conditions for whatever is being defined.

Stipulative definition of a chair

A chair is a piece of furniture intended for sitting on. A chair is a seat. A chair has four legs, a seat and a back and is

sometimes padded. It is usually made of wood or plastic but other materials can be used.

A chair is intended to seat people. A chair is a piece of furniture that people may sit

on. A chair is a piece of furniture with a back and is

intended to seat one person at a time.

Formulating Stipulative Definitions

You do not necessarily need to formulate your own definition of something.

There’s no point in re-inventing the wheel. Examine the range of definitions available in the

literature. Adopting an existing definition is perfectly

acceptable. But you should be able to explain why you choose to

adopt it – what you like about it Adapting an existing definition is perfectly

acceptable. But you should be able to explain why you choose to

adapt it – what you like and don’t like about it; why it doesn’t quite meet your needs without being altered.

Knowing how to formulate a definition from scratch will help you to adapt an existing one.

Formulating Stipulative Definitions

A stipulative definition should encompass explanation of all of the features that are integral to what is being defined.

Start by identifying the necessary conditions for or properties of whatever it is that you wish to define.

First list as many conditions or properties as you consider relevant.

Then narrow down the list to necessary conditions or properties.

What do we mean by ‘necessary conditions’?

Necessary conditions are all those components that constitute what the ‘thing’ is.

They are components that, if missing, would change the nature of the ‘thing’.

Listing necessary conditions or properties:

the example of a chair Ask yourself, for something to be called a

chair, what does it need to have? Initially list as many things as spring to mind.

In the case of every thing on your list ask yourself, do all chairs have this?

Cross off anything that prompts you to answer, ‘no’. Try to think of an example of a chair that has

something that you haven’t listed. Ask yourself, does it need to have this in order

to be called a chair? If the answer is ‘yes’ add that thing to your list.

Properties of a chair/necessary conditions for something to be a

chair Initial list might include:

has a seat is a piece of furniture has a padded seat has four legs is meant for sitting on

Refining the list In the case of everything on the list

ask yourself, do all chairs have this? has a seat is a piece of furniture has a padded seat has four legs is meant for sitting on

Refining the list Reduce the list to include only the properties

that apply to all chairs: has a seat is a piece of furniture is meant for sitting on

Try to think of an example of a chair that has something that you haven’t listed.

Ensure that the property that you think of is common to all chairs.

Add it to your list.

Necessary conditions or properties:

some examples Necessary conditions for something to

be a chair categorised as furniture designed as a seat intended to seat one person at a time a back a seat part legs or a pedestal

Incorporate everything in the list into a stipulative definition.

Punishment - necessary conditions

Flew (philosopher):Punishment must be: unpleasant for an offence or a supposed offence of an offender or of a supposed offender carried out intentionally carried out by those with authority to do so.

Formulate a stipulative definition of punishment.

Stipulative definition of punishment

Punishment is an action directed at an offender or supposed offender by one or more persons with the authority to carry out the action, with the deliberate intention of causing unpleasantness to the offender or supposed offender, in response to an offence or supposed offence committed by the targeted offender or supposed offender.

Punishment is a deliberate action carried out by someone in authority to carry it out on, and causing unpleasantness to, an offender or a supposed offender for an offence or a supposed offence.

Formulating stipulative definitions

List the necessary conditions or properties. Decide what category of phenomena is the concept that you

are defining It is a/the …

Arrange all of the necessary conditions or properties into a statement that defines the concept, beginning with reference to the category of phenomena.

Pay particular attention to the meaning of every single word. Ensure that the phrases within the definition convey the

precise sense that you intend them to convey. Test your definition by actively seeking examples of the

concept being described that are not covered by the definition.

Modify your definition until you can find no more examples of the concept being described that are not covered by the definition.

Achieving conceptual clarity

The key point is that you must communicate to others what you mean when you refer to a certain term that is susceptible to misinterpretation.

You may do this be presenting a definition or simply by presenting your interpretation of a term.

Conceptual interpretations are as acceptable as stipulative definitions.

Stipulative definitions are intellectually challenging to formulate

rewarding sense of achievement

Stipulative definitions may constitute a form of critical analysis and may earn you extra marks …

… but only if you formulate (or adapt) them effectively.

Definitions and Conceptual Interpretations

What’s the difference between the two? Stipulative definitions:

stipulate precisely what something is; are clear and unambiguous; relate exclusively to what is being defined.

Conceptual interpretations: typically describe, rather than define; may include information that supplements what

would be included in a stipulative definition; tell us what something looks like, rather than

what it is; are what many people would categorise as

ostensive definitions.

The difference between stipulative definitions and conceptual

interpretationsMorale morale is: a state of mind encompassing all of the

feelings determined by the individual’s anticipation of the extent of satisfaction of those needs which s/he perceives as significantly affecting his/her total work situation.

morale has levels; it ranges from high to low and these levels may fluctuate as circumstances and situations affect them. Morale in a work context is about how people feel about their work. Morale is future-oriented, so it relates to people’s feeling about their work in the future – whether they anticipate being happy or unhappy at work…

(from: Evans, L. (2002) Reflective Practice in Educational Research, London, Continum, p. 66)

The difference between stipulative definitions and conceptual

interpretationsDefinition or interpretation? Professional development consists of all natural

learning experiences and those conscious and planned activities which are intended to be of direct or indirect benefit to the individual, group or school and which contribute, through these, to the quality of education in the classroom. It is the process by which, alone and with others, teachers review, renew and extend their commitment as change agents to the moral purposes of teaching; and by which they acquire and develop critically the knowledge, skills, planning and practice with children, young people and colleagues through each phase of their teaching lives.

From: Day, C. (1999) Developing Teachers: the challenges of lifelong learning, London, Falmer, p. 4)

Definition or Interpretation?

Conceptual interpretation: Professional development consists of all natural

learning experiences and those conscious and planned activities which are intended to be of direct or indirect benefit to the individual, group or school and which contribute, through these, to the quality of education in the classroom.

Stipulative definition: It is the process by which, alone and with others,

teachers review, renew and extend their commitment as change agents to the moral purposes of teaching; and by which they acquire and develop critically the knowledge, skills, planning and practice with children, young people and colleagues through each phase of their teaching lives.

How to criticise a definition formulated by

someone else List what you consider to be the

necessary conditions for, or properties of, the thing being defined.

Examine the definition against this list. Does the definition cover everything?

if not, is it the definition or your list that’s deficient ? Does the definition incorporate reference to

conditions or properties that are not on your list?

if so, is it the definition or your list that’s deficient ? Formulate a critique.

Explain why you consider the definition to be deficient.

Activity Formulate a stipulative definition of a

concept that is key to your work. Don’t tell anyone what concept you are

defining. In small groups, each person is to read

out her/his definition, but without the name of the concept being defined.

The rest of the group must work out what concept is being defined.

An alternative activity Prepare two sets of cards:

1. with the name of a concept on the card2. with the stipulative definition of the

concept on the card – but not the name of the concept

Students to work in groups, sorting the cards into pairs, matching the concepts with their definitions.

Categorisation and coding The process of grouping, sorting and classifying: of

formulating and identifying categories and placing phenomena into these categories.

Codes categorise information at different levels. Categorisation is probably the most difficult

research/scholarship skill An invaluable analytical tool

allows you to analyse your own findings allows you to criticise other people’s analyses and

categorisations Useful for essay/dissertation writing:

subheadings The difficulty lies in formulating categories that:

don’t overlap are on the same coding ‘level’ (as each other).

From: Evans, L. (2002) Reflective Practice in Educational Research: developing advanced skills, London, Continuum: (Chapter 8: Tools of the Trade: developing coding and categorisation skills)

What makes effective categorisation so difficult are the two requirements that categories be both exhaustive and exclusive. Gillham (2000a, p. 60) explains: ‘(A) requirement for the derivation of categories is that they should be exhaustive. But another requirement is that they should be exclusive, i.e. that the kind of statements that go into one category clearly belong there and couldn’t really go anywhere else’.

An example from a student’s work

well-organised trustworthiness and

honesty approachable positive attitudes interpersonal skills strong leadership qualities abilities to make decisions innovative knowledge of the staff and

children

knowledge of teaching and learning

knowledge of the school vision supportive high expectations encouragement of team

work a leading professional lead by setting an

example development of the staff

characteristics of effective school leadership:

From: Evans, L. (2002) Reflective Practice in Educational Research: developing advanced skills, London, Continuum: (Chapter 8: Tools of the Trade: developing coding and categorisation skills)

This list is flawed in several ways that I identify below. As it stands it will not contribute towards the development of theory without further treatment. To move on to a deeper level of analysis the student who compiled it will need to refine and reduce it by sorting into categories. To develop into an analytical researcher – an ‘extended’ professional – you need to move beyond the level of simply presenting research findings as lists compiled from raw data. You need to delve deeper.

Categorisation skills recognition and elimination of

overlap; recognition and incorporation of

outlier and atypical cases; application of appropriate levels of

the basis of categorisation, presentation of categories.

Recognition and elimination of overlap

The number of categories you have is an important clue: too many categories implies a flawed

categorisation; categories are likely to overlap.

Use a systematic approach to identifying and eliminating overlap.

I suggest two complementary approaches: systematic comparative pairing putting things into drawers

Systematic comparative pairing

Involves systematically comparing each of your categories with each other, in turn, to examine whether or not they overlap

In the case of each pairing the comparison is a two-way process:

category A is compared with B category B is compared with A

ask the same question twice in relation to each pairing: is this category a sub-category of the other? (A of B, then B of

A) or is this category subsumed within the other? (A of B, then B of

A) If the answer to both is, ‘No’, then ask:

could this category be considered to be essentially the same thing as the other?

Putting things into drawers Involves the pictorial representation of your

categorisation as a chest of drawers one drawer for each category generated – labelled

appropriately Examine your units of data. ‘Put away’ the data into whichever drawers they

belong in The purpose is to ascertain if specific units of data

may feasibly be placed into more than one drawer. If so, the categorisation is flawed.

It involves overlap. Effective categorisation will be represented by drawers

in only one of which any specific portion or unit of data – at whatever level of analysis you are working – belongs.

Putting things into drawers

spoons forks

teaspoons knives

cutlery cutting implements

Issues that matter to education professionals, in their work

contextsFigure 2: Sorting data into categories of issues that matter to people: an example of pictorial representation of categorisation as labelled drawers.

1

6

25

3

4

7

8