The Fire ProtectionThe Fire Protection

Professional Engineer Professional Engineer

Exam andExam and

YOUYOU

Purpose of the ExamPurpose of the Exam

•• Prove the engineer is competent Prove the engineer is competent

enough to practice independently enough to practice independently

without adversely affecting the health, without adversely affecting the health,

safety, or welfare of the publicsafety, or welfare of the public

•• ASSESS MINIMUM COMPETENCYASSESS MINIMUM COMPETENCY

•• Identify “adequate” engineers not the Identify “adequate” engineers not the

“best” engineer“best” engineer

Historical Exam StructureHistorical Exam Structure

•• Part machine scored (Part machine scored (scantronscantron))

•• Part hand scored (essay)Part hand scored (essay)

•• 8 problems with 10 parts each8 problems with 10 parts each

•• Limited choice of which problems to Limited choice of which problems to

workwork

Current Exam StructureCurrent Exam Structure

•• All machine scoredAll machine scored

•• 80 problems 80 problems –– Each independent of Each independent of

othersothers

•• Must attempt all problems, no choicesMust attempt all problems, no choices

Goals for Future ExamsGoals for Future Exams

•• Reduce need for reference materials to Reduce need for reference materials to the exam site to the SFPE Handbook of the exam site to the SFPE Handbook of Fire Protection EngineeringFire Protection Engineering

•• Test candidate’s knowledge of fire Test candidate’s knowledge of fire protection engineering principles, not protection engineering principles, not the ability to look up code requirementsthe ability to look up code requirements

•• Eliminate need to revise exam items Eliminate need to revise exam items when codes are revisedwhen codes are revised

•• Facilitate onFacilitate on--demand computerdemand computer--based based testingtesting

Examination Development

Process Overview

Equating of

Examination

Question

Writing &

Review

Examination

Assembly &

Review

Examination

Scoring

Standard

Setting

Study

Need for

Examination

Identified

Task Analysis

Performed(PAKS)

Examination

Specification

Developed

Examination

AdministrationAfter Cut Score Established

Exam Specification

•• Types of fire protection analysis Types of fire protection analysis –– 10 q10 q

•• Info sources for FP analysis Info sources for FP analysis –– 6 q6 q

•• Fire protection management Fire protection management –– 8 q8 q

•• Fire dynamics Fire dynamics –– 8 q8 q

•• Human behavior Human behavior –– 4 q4 q

•• Water based fire suppression Water based fire suppression –– 10 q10 q

•• Special hazard systems Special hazard systems –– 4 q4 q

•• Detection and alarm Detection and alarm –– 7 q7 q

•• Smoke management Smoke management –– 4 q4 q

•• Explosion protection Explosion protection –– 3 q3 q

•• Passive building construction Passive building construction –– 10 q10 q

•• Means of egress Means of egress –– 6 q6 q

Ranked by EmphasisRanked by Emphasis•• Types of fire protection analysis Types of fire protection analysis –– 10 q10 q

•• Water based fire suppression Water based fire suppression –– 10 q10 q

•• Passive building construction Passive building construction –– 10 q10 q

•• Fire protection management Fire protection management –– 8 q8 q

•• Fire dynamics Fire dynamics –– 8 q 8 q

•• Detection and alarm Detection and alarm –– 7 q 7 q

•• Info sources for FP analysis Info sources for FP analysis –– 6 q6 q

•• Means of egress Means of egress –– 6 q6 q

•• Human behavior Human behavior –– 4 q4 q

•• Special hazard systems Special hazard systems –– 4 q4 q

•• Smoke management Smoke management –– 4 q4 q

•• Explosion protection Explosion protection –– 3 q3 q

The Life Cycle of an Exam The Life Cycle of an Exam

QuestionQuestion

•• Drafted by rank & file SPFE member (YOU)Drafted by rank & file SPFE member (YOU)

•• Reviewed/edited by “subject matter experts” on Reviewed/edited by “subject matter experts” on

SFPE Licensing (Exam) CommitteeSFPE Licensing (Exam) Committee

•• Beta tested at SFPE Annual Meeting (YOU)Beta tested at SFPE Annual Meeting (YOU)

•• Put into the Problem Bank at NCEESPut into the Problem Bank at NCEES

•• Pulled from Bank to assemble examinationPulled from Bank to assemble examination

•• Given on examinationGiven on examination

•• ReRe--evaluated based on exam performanceevaluated based on exam performance

Item WritingItem Writing

Item Writing NomenclatureItem Writing Nomenclature

Item = entire questionItem = entire question

Stem = problem statementStem = problem statement

Options = multiple choicesOptions = multiple choices

Key = right answerKey = right answer

Distracters = wrong answersDistracters = wrong answers

Example ItemExample Item

Stem Stem -- Which of the following temperatures Which of the following temperatures in in °°°°°°°°C is most nearly equivalent to a C is most nearly equivalent to a temperature of 60 temperature of 60 °°°°°°°°F?F?

Options:Options:

Key Key (A)(A) 1515

Distracter Distracter (B)(B) 3333

Distracter Distracter (C)(C) 5050

Distracter Distracter (D)(D) 5151

Solution to ExampleSolution to Example

(60 (60 -- 32)×5/9 = 15.532)×5/9 = 15.5 Option (a)Option (a)

Distracters:Distracters:

(b) did not subtract 32 from 60(b) did not subtract 32 from 60 ( = 33)( = 33)

(c) multiplied by 9/5 (c) multiplied by 9/5 ( = 50)( = 50)

(d) added 32 to 60 (d) added 32 to 60 ( = 51)( = 51)

Writing Good Items Writing Good Items -- DOsDOs

•• Work in about 6 minutesWork in about 6 minutes

•• One correct answerOne correct answer

•• Simple and direct languageSimple and direct language

•• Code information given in stemCode information given in stem

•• Plausible optionsPlausible options

•• Mutually exclusive optionsMutually exclusive options

•• Common errors make good distractersCommon errors make good distracters

•• All options the same lengthAll options the same length

•• Most items include the term “is most Most items include the term “is most nearly”nearly”

Writing Good Items Writing Good Items –– DON’TsDON’Ts

•• Code LookCode Look--up up

•• Negative wordingNegative wording

•• Extremes (always/never/only/every)Extremes (always/never/only/every)

•• Soft wording (much more/somewhat Soft wording (much more/somewhat more/much less)more/much less)

•• Avoid grouping options too closelyAvoid grouping options too closely

•• Use “None of the above” sparinglyUse “None of the above” sparingly

•• Extraneous clues to the correct answerExtraneous clues to the correct answer

•• Sets of questionsSets of questions

ExamplesExamples

Based on the description of the excavated channel, Based on the description of the excavated channel,

Manning’s roughness coefficient should not exceedManning’s roughness coefficient should not exceed

A. 0.016A. 0.016

B. 0.020B. 0.020

C. 0.042C. 0.042

D. 0.055D. 0.055

ExceedExceed

Flaw: Say answer is (B), 0.020, then (C) and Flaw: Say answer is (B), 0.020, then (C) and

(D) are also true(D) are also true

Which of the following factors could enter into a Which of the following factors could enter into a

decision about the type of circuit to be used?decision about the type of circuit to be used?

A. Weather at the plant siteA. Weather at the plant site

B. Terrain at the plant siteB. Terrain at the plant site

C. None of the aboveC. None of the above

D. All of the aboveD. All of the above

All or NoneAll or None

Flaw: “All of the above” cannot be true since it Flaw: “All of the above” cannot be true since it

includes “None of the above”includes “None of the above”

Redundant control data highways for a DCS are Redundant control data highways for a DCS are

to be extended with fiber optic cable. Which of the to be extended with fiber optic cable. Which of the

following installation practices is the most following installation practices is the most

important?important?

BalancedBalanced

A. Redundant fiber optic cables should be the same A. Redundant fiber optic cables should be the same

length to minimize time delay errors.length to minimize time delay errors.

B.B.Fiber optic cables should not be run near and parallel Fiber optic cables should not be run near and parallel

to alternating current cables.to alternating current cables.

C.C.Redundant fiber optic cables should take separate Redundant fiber optic cables should take separate

paths to reduce common mode interferencepaths to reduce common mode interference

D.D.Redundant fiber optic cables should take separate Redundant fiber optic cables should take separate

paths to reduce risk of mechanical damage.paths to reduce risk of mechanical damage.

Tests practice; Qualitative;

Balanced length of options

In extrapolating data to determine the 50In extrapolating data to determine the 50-- or 100or 100--

year flood from a series of much shorter duration, year flood from a series of much shorter duration,

the probable error or standard deviation of the the probable error or standard deviation of the

estimated flood canestimated flood can

Short distractersShort distracters

A.A. be determined from a linear regression of Q versus Tbe determined from a linear regression of Q versus Trr..

B.B. be determined from a linear regression of Q versus log Tbe determined from a linear regression of Q versus log Trr..

C.C. be determined from a quadratic regression of Q versus log be determined from a quadratic regression of Q versus log

TTrr..

D. NOT be determined because the method of extrapolating D. NOT be determined because the method of extrapolating

the curve is arbitrary and has no basis in the physics of the the curve is arbitrary and has no basis in the physics of the

hydrologic system, even though such extrapolation is hydrologic system, even though such extrapolation is

routinely performed by engineering practitioners and routinely performed by engineering practitioners and

regulatory bureaucrats.regulatory bureaucrats.

Flaw: Key is longer than distracters -

more qualified and specific

Flaw: This item

is too long.

Code LookupCode Lookup

Per NFPA 101, Per NFPA 101, Life Safety CodeLife Safety Code, the , the maximum permissible travel distance maximum permissible travel distance (ft) in a new, sprinkler protected, (ft) in a new, sprinkler protected, business occupancy is:business occupancy is:

(a) 75(a) 75

(b) 100(b) 100

(c) 200(c) 200

(d) 300(d) 300

Why don’t we want it?Why don’t we want it?

•• It only tests whether a candidate can It only tests whether a candidate can

find the applicable requirement in NFPA find the applicable requirement in NFPA

101101

•• It doesn’t assess whether the candidate It doesn’t assess whether the candidate

knows what travel distance is or how to knows what travel distance is or how to

measure itmeasure it

Better VersionBetter Version

In the figure below, the travel distance In the figure below, the travel distance

from Room 1 is equal to:from Room 1 is equal to:

(a)(a) AA

(b)(b) BB

(c)(c) A + BA + B

(d)(d) A + CA + C

Why we want it?Why we want it?•• Assesses the Candidate’s understanding of Assesses the Candidate’s understanding of the concept of travel distancethe concept of travel distance

•• Assesses the Candidate’s ability to measure Assesses the Candidate’s ability to measure travel distancetravel distance

•• Does not require a “compliance Does not require a “compliance determination”determination”

•• Does not require the candidate to have a Does not require the candidate to have a code book to look up an allowable travel code book to look up an allowable travel distancedistance

You are testYou are test--wise if you can wise if you can

select the key of the following select the key of the following

items based on clues given in items based on clues given in

the item without knowing the the item without knowing the

content.content.

ARE YOUARE YOU

TEST WISE?TEST WISE?

What is the primary purpose of the cluss in What is the primary purpose of the cluss in frumpaling?frumpaling?

A. remove clussA. remove cluss--prangsprangsB. patch tremailsB. patch tremailsC. loosen cloughsC. loosen cloughsD. repair plumotsD. repair plumots

ClussCluss

Someone who knows nothing about the content Someone who knows nothing about the content

can still pick A as the key because it’s the only can still pick A as the key because it’s the only

option to include a word (cluss) found in the option to include a word (cluss) found in the

stem.stem.

The fribbled breg will ninter best with anThe fribbled breg will ninter best with an

A. mors.A. mors.

B. ignu.B. ignu.

C. derst.C. derst.

D. sortar.D. sortar.

Fribbled bregFribbled breg

TestTest--wise candidates will select B because it is wise candidates will select B because it is

the only option that grammatically flows from the only option that grammatically flows from

the stem.the stem.

Why does the sigla frequently overfesk the Why does the sigla frequently overfesk the

trelsum?trelsum?

A. All siglas are mellious.A. All siglas are mellious.

B. Siglas are always votial.B. Siglas are always votial.

C. The trelsum is usually tarious.C. The trelsum is usually tarious.

D. No trelsa are directly feskable.D. No trelsa are directly feskable.

SiglaSigla

TestTest--wise candidates can identify C as the key wise candidates can identify C as the key because all the other options contain words (i.e. because all the other options contain words (i.e. all, always, no) associated with false all, always, no) associated with false statements.statements.

Trassign normally occurs under which of the Trassign normally occurs under which of the following conditions?following conditions?

A. when dissles frullA. when dissles frullB. when lusp trasses the vomB. when lusp trasses the vomC. when the belgo lisks tariousC. when the belgo lisks tariousD. when the viskal flans, if the viskal is zortilD. when the viskal flans, if the viskal is zortil

and the hackshe is plaffedand the hackshe is plaffed

TrassignTrassign

TestTest--wise candidates will pick D as the key because it wise candidates will pick D as the key because it

is the only option to specify a condition and the stem is the only option to specify a condition and the stem

directs candidates to look for a condition. It is also the directs candidates to look for a condition. It is also the

longest.longest.

What probable causes are indicated when doss What probable causes are indicated when doss

occurs in a compots?occurs in a compots?

A. The sabs foped and the doths tinzed.A. The sabs foped and the doths tinzed.

B. The kredgs roted with the rots.B. The kredgs roted with the rots.

C. The rakogs were not accepted in the sluth.C. The rakogs were not accepted in the sluth.

D. The polats were thenced in the sluth.D. The polats were thenced in the sluth.

DossDoss

This one is relatively tricky, yet some testThis one is relatively tricky, yet some test--wise wise

candidates may be able to deduce that A is the candidates may be able to deduce that A is the

only option with multiple only option with multiple causescauses (sabs (sabs andand

doths).doths).

The nintering function of the ignu is most effectively The nintering function of the ignu is most effectively

performed in connection with theperformed in connection with the

A. arazma tol.A. arazma tol.

B. fribbled breg.B. fribbled breg.

C. groshing statol.C. groshing statol.

D. frallied stantels.D. frallied stantels.

IgnuIgnu

This one is also tricky, but some testThis one is also tricky, but some test--wise candidates wise candidates

will remember that will remember that ignusignus and and fribbled bregsfribbled bregs were linked were linked

in the second item and thus select B here. This is in the second item and thus select B here. This is

called cueing. It can be difficult to detect, but you called cueing. It can be difficult to detect, but you

should always check for it.should always check for it.

Testing the TestTesting the Test

Fairness GoalsFairness Goals

•• Scoring not subjective Scoring not subjective

•• Different exam administrations yield Different exam administrations yield

similar gradessimilar grades

•• Exam results on individual questions Exam results on individual questions

show questions are able to distinguish show questions are able to distinguish

adequate and inadequate performanceadequate and inadequate performance

Exam to ExamExam to Exam

•• Repeat some questions (about ¼)Repeat some questions (about ¼)

•• Compare two groups on the questions Compare two groups on the questions

in commonin common

•• Norm the two groupsNorm the two groups

•• Serious Implications for Exam SecuritySerious Implications for Exam Security

•• FPE Exam more difficult to norm than FPE Exam more difficult to norm than

other because of small numbersother because of small numbers

Fair QuestionsFair Questions

•• 40 to 80% of test takers get question right40 to 80% of test takers get question right

�� Less indicates problem too hardLess indicates problem too hard

�� More indicates problem too easyMore indicates problem too easy

•• Wrong answers spread across Wrong answers spread across distractorsdistractors

•• Positive relationship between performance Positive relationship between performance

on question and performance on entire on question and performance on entire

examexam

What Happens if a What Happens if a

Question is Bad?Question is Bad?•• If many test takers get the same wrong If many test takers get the same wrong answer, both the correct and the answer, both the correct and the “popular” wrong answer are accepted “popular” wrong answer are accepted and exam is and exam is rescoredrescored..

•• If problem is too hard or perverse then If problem is too hard or perverse then everyone is given credit for the problem everyone is given credit for the problem and the exam is and the exam is rescoredrescored..

•• The question leaves the bank to be The question leaves the bank to be reworked or retiredreworked or retired

Philosophical IssuesPhilosophical Issues

for Exam Writersfor Exam Writers

•• Forcing all problems to be independent Forcing all problems to be independent limits the complexity of problems that limits the complexity of problems that can be on examcan be on exam

•• Time limits favor qualitative questions Time limits favor qualitative questions over calculation questionsover calculation questions

•• Qualitative questions tend to become Qualitative questions tend to become trivial if there is an single unarguable trivial if there is an single unarguable correct answercorrect answer

Questions?Questions?