Question setting

CIEA Insight is a key support for CIEA members.

Through Insight you can tap into a range of materials in one place that can:

• help you develop your own knowledge and skills in specific areas of assessment• help you develop your team’s knowledge and skills • give you access to information that we hope you will find stimulating and informative.

Question setting

This short CIEA Insight will give you a glimpse of some key aspects of question setting

More detailed information on the subject is available to CIEA members in the members area of www.ciea.org.uk

This Insight will cover

Question typesSome general points on good practiceThings to avoid

Question setting

Whether you are setting questions for assessment in your own centre, class or tutor group or involved in setting questions for public examinations, there are still good and bad approaches to the process.

Question setting

Compare two main characters from different works of literature from different centuries.

Good question?

Question setting

Compare two main characters from different works of literature from different centuries.

“There is no evidence that Lady Macbeth liked fish; Tarka the Otter loved it.”

“Whereas Tarka was despatched with a spade,it was a balcony wot done for Lady Macbeth.”

Question setting

Poorly written questions have a negative impact on the learners who are faced with them

Poorly written questions also create frustration amongst assessors

Types of Question

Summative AssessmentMultiple choiceShort answerExtended answer / essay

Continuous AssessmentTasksEnd of unit testsExtended essays

Types of Question

Knowing which type of question to use, when, is a skill in itself

Multiple Choice Questions

Benefits•Objective marking – requires no interpretation from marker•Quick to answer•Test a breadth of knowledge quickly•Marking is fast and consistent

Disadvantages•Limited testing of skills•Need very tight focus•Longer to set than other question types•Setting is a specialist skill

Multiple Choice Questions

Terminology

Question = stem

Correct answer = key

Choices = options

Incorrect answers = distractors

Multiple Choice Questions - Example

The capital city of Spain is (STEM)

A Barcelona (DISTRACTOR)

B Madrid (KEY)

C Valencia (DISTRACTOR)

D Aragon (DISTRACTOR)

Multiple Choice Question Stems

Stems can be written in various ways

Among these are:

Completing a statementThe capital city of Spain is ...

Logic and reasoningWhich of the following statements can be justified using the information in the passage alone?

Odd one outEach of the following is a cause of drought except ...

Negative Stems

Using the negative in a Stem can be confusing

There are three things wrong with this question

Which of the items in the following list is not one of the major organs of the human body?

A SkinB HeartC LegD Pancreas

Vague and Confusing Stems

Which of the items in the following list is not one of the major organs of the human body?

A SkinB HeartC LegD Pancreas

Some learners will focus on the word “organ” and simply pick one out

The word “major” has not been defined

The answer “Leg” is too obvious

Vague and Confusing Stems

This question stem is confusing

Which of these is not a good reason for the world economy to slow down in the event of a major natural disaster?

Vague and Confusing Stems

Which of these is not a good reason for the world economy to slow down in the event of a major natural disaster?

It uses the negative

The main thrust of the question is separated from the opening

It is difficult to word a correct answer and even harder to write distractors

Carefully Worded Odd One Out Questions

It is valid to test the ability to recognise which items in a list stand out in some way

Using the format:

Each of the following is ... except

can be a good way around the confusing negative stem problem

True / False Questions

True / False questions are not true multiple choice questions

e.g. A basic principle of physics is that energy can neither be created nor be destroyed

True or False?

True / False Questions

A better approach would be to ask:

A basic principle is that energy

A can be created but not destroyedB can be destroyed but not createdC can be destroyed and createdD cannot be destroyed or created

True / False Questions

A basic principle is that energy

A can be created but not destroyedB can be destroyed but not createdC can be destroyed and createdD cannot be destroyed or created

Even here the correct answer is the only one to contain the word “cannot” so it might need further revision

Multiple Choice Questions Summary

You have now seen some of the benefits of multiple choice questions.

They can:

•be objective •be answered quickly•test a breadth of knowledge•be marked quickly and consistently

Short Answer Questions

SAQs can be marked

By human markers - possibly subject specialistsBy human markers who are not specialists

On screenOn line

Automatically - if the list of possible outcomes is finite and easily referenced

Short Answer Questions

Be clear in your demands

Avoid ambiguitye.g. Say how glaciers move

SLOWLY

And the old maths classic:

Short Answer Questions

are

useful for developing points slightly further than multiple questions would allow

a good way to explore a learner’s knowledge quickly

easily administered and marked - but not always automatically

are not

Multiple Choice QuestionsEssay titles

If the point you wish to test is ideally suited to another method then use the other method

Extended Answer Questions

Traditional essay style questions

Used to test knowledge in depth

Need to be carefully worded to test what is actually required

Writing Extended Answer Questions

e.g. Give an account of “Macbeth”

This pretends to be an essay but is nothing more than a request for a candidate to write out the plot.

A low order skill would actually be tested

Writing Extended Answer Questions

e.g. Macbeth is a good man who allows himself to be tricked by the forces of evil.

Discuss the above statement with reference to at least three scenes from the play.

This is clearer in its demands and gives an element of structure to the task which was not present in the previous question.

Structuring Extended Answer Questions

Having said that ...

Don’t turn an Extended Answer Question into nothing more than a series of SAQs (Short Answer Questions)

This can be the case if the question has

• too many parts

e.g. Give two reasons why Britain sent troops to France in 1914 and three reasons why the USA entered the war and two outcomes of the treaty of Versailles.

Deriving Extended Answer Questions

The question is being asked in order for a candidate to demonstrate knowledge and/or skills

The question should come naturally from a programme of study and not be tacked on artificially

Extended Answer Questions

can

assess knowledge in depthprobe into skills

should not

lead to repetitive low order tasksbe unconnected to a programme of study

Showing Off

It might be interesting to set questions for the local pub quiz which test esoteric knowledge

BUT

Question setters should resist the temptation to show the world how clever they are by setting candidates ridiculously difficult tasks

If there is a clear form of words then use it

Remember what you are testing

The question is being asked for a clear reason

It is in clear language

It asks what it seems to ask

Deciphering the question should not be the test

Teaching to the tests or testing what is taught?

Whilst the role of assessment remains very important in school, college and workplace education and training, we should all remember that assessment is not an end in itself.

Keep a sensible balance between teaching / training and the various forms of assessment.

The CIEA hopes to engage members in wide ranging discussion about the nature and the future of assessment.

You can find out more about question setting in the members’ area of the CIEA website - www.ciea.org.uk