Assessing Measurement quality in Assessing Measurement quality in quantitative studies.quantitative studies.

Presented By:Presented By:

Mrs. Christy SimpsonMrs. Christy Simpson

Professor, Maternity nursingProfessor, Maternity nursing

Definition:Definition:Quantitative Data:Quantitative Data:

Information collected in a quantified (numeric ) Information collected in a quantified (numeric ) form.form.Quantitative ResearchQuantitative Research::

The investigation of phenomena that lend The investigation of phenomena that lend themselves to precise measurement and themselves to precise measurement and quantification, often involving a rigorous and quantification, often involving a rigorous and controlled design.controlled design.Quantitative Analysis:Quantitative Analysis:

Manipulation of numeric data through statistical Manipulation of numeric data through statistical procedures for the purpose of describing procedures for the purpose of describing phenomena or assessing the magnitude and phenomena or assessing the magnitude and reliability of relationships among them.reliability of relationships among them.

Measurement:Measurement:

Quantitative studies derive data through the Quantitative studies derive data through the measurement of variables.measurement of variables.

Measurement involves the assignment of Measurement involves the assignment of numbers to represent the amount of an attribute numbers to represent the amount of an attribute present in an object or person using a specified set present in an object or person using a specified set of rules.of rules.

Principles of Measurement:Principles of Measurement:

Classical measurement theory e.g Psychosocial Classical measurement theory e.g Psychosocial constructs such as depression or social support.constructs such as depression or social support.

Alternative measurement theory or Item Response Alternative measurement theory or Item Response theory e.g Cognitive constructs, Achievement or theory e.g Cognitive constructs, Achievement or ability.ability.

Advantages of measurement:Advantages of measurement:

- Measurement is a language of - Measurement is a language of communication.communication.

- Numbers are less vague than words and - Numbers are less vague than words and therefore can communicate information more therefore can communicate information more correctly. e.g Obese than 80Kgcorrectly. e.g Obese than 80Kg

Errors of Measurement:Errors of Measurement: Instruments that are not perfectly Instruments that are not perfectly

accurate yield measurements containing accurate yield measurements containing some error.some error.

With in classical measurement theory, any With in classical measurement theory, any observed (Obtained) score can be observed (Obtained) score can be decomposed conceptually in to two parts :decomposed conceptually in to two parts :

a) An error componenta) An error component b) A true componentb) A true component Obtained score = true score Obtained score = true score ± error± error

Many factors contribute to errors of Many factors contribute to errors of measurement:measurement:Some are random or variable, others are Some are random or variable, others are systematic, which represent bias.systematic, which represent bias.

The most common influences on The most common influences on measurement error are:measurement error are:

1.1. Situational contaminants:Situational contaminants: Scores can be affected by the conditions Scores can be affected by the conditions

under which they are produced.e.g. A under which they are produced.e.g. A participant’s awareness of an observer’s participant’s awareness of an observer’s presence (reactivity). presence (reactivity).

Other environmental factors Other environmental factors are:Temperature , lighting etc.are:Temperature , lighting etc.

2.Transitory personal factors:2.Transitory personal factors:

A person’s score can be influenced by A person’s score can be influenced by such temporary personal states as fatique, such temporary personal states as fatique, hunger, anxiety or mood.hunger, anxiety or mood.

3. Response set biases:3. Response set biases:

Relatively enduring characteristics of Relatively enduring characteristics of respondents can interfere with accurate respondents can interfere with accurate measures. E.g social desirability, measures. E.g social desirability, acquiescenceacquiescence

4. Administration Variations:4. Administration Variations:

Alterations in the method of collecting Alterations in the method of collecting data from one person to the next.data from one person to the next.

Errors cont’dErrors cont’d5. Instrument Clarity:5. Instrument Clarity: If the directions for obtaining measures If the directions for obtaining measures

are poorly understood, then scores may be are poorly understood, then scores may be affected by misunderstanding. E.g. Self - affected by misunderstanding. E.g. Self - report instrument may be interpreted report instrument may be interpreted differently by different respondents.differently by different respondents.

6. 6. Item Sampling:Item Sampling: Errors can be introduced as a result of the Errors can be introduced as a result of the

sampling of items used in the measure.sampling of items used in the measure.7. 7. Instrument format:Instrument format: Technical characteristics Technical characteristics

of an instrument. E.g open ended questions of an instrument. E.g open ended questions yield different information than closed ones.yield different information than closed ones.

Criterion to assess the quality of Criterion to assess the quality of quantitative instrument:quantitative instrument:

1.1. Reliability:Reliability:

An instrument’s reliability is the An instrument’s reliability is the consistency with which it measures the consistency with which it measures the target attribute. target attribute.

The less variation an instrument The less variation an instrument produces in repeated measurements, the produces in repeated measurements, the higher its reliability.higher its reliability.

The three key aspects of reliability:The three key aspects of reliability:

Stability , Internal consistency and equivalenceStability , Internal consistency and equivalence

Stability:Stability: The stability of an instrument is the extent The stability of an instrument is the extent

to which similar results are obtained on two to which similar results are obtained on two separate occasions.separate occasions.

Assessments of an instrument’s stability Assessments of an instrument’s stability involve procedures that evaluate test – retest involve procedures that evaluate test – retest reliability.reliability.

e.g. Administer the same measure to a e.g. Administer the same measure to a sample twice and then compare the scores sample twice and then compare the scores by computing a reliability coefficient, which is by computing a reliability coefficient, which is an index of the magnitude of the test’s an index of the magnitude of the test’s reliability. Statistical analysis is correlation –reliability. Statistical analysis is correlation –coefficient. coefficient.

How to read a correlation coefficient:How to read a correlation coefficient:

Two relationships:Two relationships:

1.1. Positive relationship:Positive relationship:

The possible values for a correlation The possible values for a correlation coefficient ranges from – 1.00 through .00 coefficient ranges from – 1.00 through .00 to + 1.00.to + 1.00.

Positive relationship value should be 1Positive relationship value should be 1

e.g:e.g:Anxiety scale - Administer the scale twice Anxiety scale - Administer the scale twice with 2 weeks durationwith 2 weeks duration

Negative Relationship:Negative Relationship:

When two variables are inversely related, When two variables are inversely related, increases in one variable are associated with increases in one variable are associated with decreases in the second variable.decreases in the second variable.

The value of negative relationship is -1.The value of negative relationship is -1.

e.g:e.g: IQ is more in tall person. IQ is more in tall person.

The higher the coefficient, the more stable The higher the coefficient, the more stable the measure.the measure.

The reliability coefficient is higher for short The reliability coefficient is higher for short term retests than longterm reteststerm retests than longterm retests

Internal consistency:Internal consistency:

Scales designed to measure an attribute Scales designed to measure an attribute ideally are composed of items that measure that ideally are composed of items that measure that attribute and nothing else.attribute and nothing else.

An instrument may be said to be internally An instrument may be said to be internally consistent or homogeneous to the extent that its consistent or homogeneous to the extent that its measure the same trait. measure the same trait.

e.g Depression scalee.g Depression scale

The most widely used method for evaluating The most widely used method for evaluating internal consistency is coefficient alpha or internal consistency is coefficient alpha or Cronbach’s alpha.Normal range of value is .00 Cronbach’s alpha.Normal range of value is .00 and +1.00and +1.00

Equivalence:Equivalence: The degree to which two or more The degree to which two or more

independent observers or coders agree about independent observers or coders agree about the scoring on an instrument.the scoring on an instrument.

Inter rater reliability can be assessed. Inter rater reliability can be assessed. When ratings are dichotomus, Following When ratings are dichotomus, Following equation is used to calculate the proportion of equation is used to calculate the proportion of agreements.agreements.

Number of agreementNumber of agreementNumber of agreement + disagreementsNumber of agreement + disagreementsThe statistics used is The statistics used is Cohen’s KappaCohen’s Kappa which which

adjust for chance agreements. adjust for chance agreements. Multi rater Multi rater KappaKappa when more than two raters. when more than two raters.

Factors affecting reliability:Factors affecting reliability:

More items tapping the same concept should More items tapping the same concept should be added.be added.

Items that have no discriminating power Items that have no discriminating power should be removedshould be removed

Validity:Validity:

It is the degree to which an instrument It is the degree to which an instrument measures what it is supposed to measure.measures what it is supposed to measure.

A measuring device that is unreliable cannot A measuring device that is unreliable cannot possibly be valid.possibly be valid.

Validation efforts should be viewed as Validation efforts should be viewed as evidence gathering enterprises.evidence gathering enterprises.

The more evidence gathered, using various The more evidence gathered, using various methods to assess validity, the stronger the methods to assess validity, the stronger the inference. inference.

Types of validity:Types of validity:

1.1. Face validity:Face validity:

Refers to whether the instrument looks as though it Refers to whether the instrument looks as though it is measuring the appropriate.is measuring the appropriate.

Scale is established by consulting the experts and Scale is established by consulting the experts and person with a same diseaseperson with a same disease

2. 2. Content Validity:Content Validity:

Concerns the degree to which an instrument has Concerns the degree to which an instrument has an appropriate sample of items for the construct an appropriate sample of items for the construct being measured and adequately covers the being measured and adequately covers the construct domain.construct domain.

Content validity is relevant for both affective Content validity is relevant for both affective and cognitive measuresand cognitive measures

Content Valid cont’dContent Valid cont’d

An content validity is necessarily based on An content validity is necessarily based on judgement.No objective methods to ensure judgement.No objective methods to ensure content validity.content validity.

Use a panel of substantive experts to Use a panel of substantive experts to evaluate and document the content validity of evaluate and document the content validity of new instruments.Validation by minimum of new instruments.Validation by minimum of three.three.

Calculate the Content Validity index,(CVI) Calculate the Content Validity index,(CVI) Experts rate items on a 4 – point scale of Experts rate items on a 4 – point scale of relevance, the item(I) CVI is computed as relevance, the item(I) CVI is computed as the number of raters giving a rating of the number of raters giving a rating of either 3 or 4 , divided by the number of either 3 or 4 , divided by the number of experts.I-CVI of .80 is considered an experts.I-CVI of .80 is considered an acceptable value.acceptable value.

Scale CVI (S) CVI can be also done.Scale CVI (S) CVI can be also done.

3. Concurrent Validity:3. Concurrent Validity:

Concurrent Validity refers to a Concurrent Validity refers to a measurement device’s ability to vary directly measurement device’s ability to vary directly with a measure of the same construct or with a measure of the same construct or indirectly with a measure of an opposite indirectly with a measure of an opposite construct. It allows you to show that your construct. It allows you to show that your test is valid by comparing it with an already test is valid by comparing it with an already valid test. valid test.

4. Criterion – Related validity:4. Criterion – Related validity:

Determines the relationship between an Determines the relationship between an instrument and an external criterion.instrument and an external criterion.

The instrument is said to be valid if its The instrument is said to be valid if its scores correlate highly with scores on the scores correlate highly with scores on the criterion.criterion.

Two types of criterion related validity:Two types of criterion related validity:

a) Predictive validity : Refers to the adequacy a) Predictive validity : Refers to the adequacy of an instrument in differentiating between of an instrument in differentiating between people’s performance on some future people’s performance on some future criterion. e.g , High school grades for nursing criterion. e.g , High school grades for nursing school performanceschool performance

b)b) Construct validity:Construct validity:

It is a key criterion for assessing the quality of It is a key criterion for assessing the quality of a study. a study.

sometimes also called factorial validity, has to sometimes also called factorial validity, has to do with the logic of items which comprise do with the logic of items which comprise measures of social concepts.measures of social concepts.

The key construct validity questions:The key construct validity questions:

- What is this instrument really measuring?- What is this instrument really measuring?

- Does it adequately measure the abstract - Does it adequately measure the abstract concept of interest concept of interest

Construct cont’dConstruct cont’d

A good construct has a theoretical basis A good construct has a theoretical basis which is translated through clear operational which is translated through clear operational definitions involving measurable indicators. definitions involving measurable indicators.

It involves logical analysis and hypothesis It involves logical analysis and hypothesis test.test.

Methods of construct validity:Methods of construct validity:

1.Known groups Technique:1.Known groups Technique:

The instrument is administered to groups The instrument is administered to groups hypothesized to differ on the critical hypothesized to differ on the critical attribute because of some known attribute because of some known characteristics. characteristics.

E.g Anxiety among primi & Multi in labour.E.g Anxiety among primi & Multi in labour.

2. Hypothesized Relationship:2. Hypothesized Relationship:

Testing hypothesized relationships, often on Testing hypothesized relationships, often on the basis of theory. E.g Smoking ---Cancerthe basis of theory. E.g Smoking ---Cancer

3. Convergent and Discriminant Validity:3. Convergent and Discriminant Validity:

An important construct validation tool is a An important construct validation tool is a procedure known as the Multitrait – procedure known as the Multitrait – multimethod matrix method which involves multimethod matrix method which involves convergence and Discriminiability.convergence and Discriminiability.

Convergence is evidence that different Convergence is evidence that different methods of measuring a construct yield methods of measuring a construct yield similar results.e.g Self report,Observation similar results.e.g Self report,Observation etc.etc.

Discriminiability is the ability to differentiate Discriminiability is the ability to differentiate the construct from other similar constructs.the construct from other similar constructs.

e.g. Psychological & Physical problems (HIV)e.g. Psychological & Physical problems (HIV)

4. 4. Factor Analysis:Factor Analysis:

It is a method for identifying clusters of related variables – It is a method for identifying clusters of related variables – that is ,dimensions underlying a central construct.that is ,dimensions underlying a central construct.

It is a statistical procedure for identifying unitary clusters of It is a statistical procedure for identifying unitary clusters of items.items.

e,g Assess nursing students confidence in caring mentally ill e,g Assess nursing students confidence in caring mentally ill patients. patients.

Criteria for screening and diagnostic Criteria for screening and diagnostic instruments:instruments:

Sensitivity and SpecificitySensitivity and SpecificitySensitivity Sensitivity is the instrument’s ability to is the instrument’s ability to

identify a case correctly.(Its rate of yielding identify a case correctly.(Its rate of yielding true positives)true positives)

True positives divided by positives, (Smokers True positives divided by positives, (Smokers who had high cotinine / all real smokers)who had high cotinine / all real smokers)

Specificity Specificity is the instrument’s ability to is the instrument’s ability to identify non cases correctly.(Its rate of identify non cases correctly.(Its rate of yielding true negatives)Teenagers reported yielding true negatives)Teenagers reported that they did not smoke,True negatives / all that they did not smoke,True negatives / all negatives.negatives.

Self Reported Self Reported smokingsmoking

Positive Positive

CotinineCotinine

NegativeNegative

CotinineCotinine

TotalTotal

Yes , SmokedYes , Smoked A (True A (True positive) 20positive) 20

B (False B (False positive)10positive)10

A+B =30A+B =30

No,Did not No,Did not smokesmoke

C (False C (False negative)20negative)20

D(True D(True negative)50negative)50

C+D = 70C+D = 70

A+C=40A+C=40 B+D=60 B+D=60 A+B+C+DA+B+C+D

100100

Urinary cotininie level

Sensitivity = A/(A+C) = .50, Specificity = D/ (B+D) = .83 ( Percentage)

Positive predictive value = A/(A+B) =.67 Negative predictive value =D/(C=D)=.71

Likelihood ratio –Positive (LR+) = Sensitivity/(1- Specificity) = 2.99

Likelihood ratio – Negative(LR_) = (1- sensitivity) / specificity =.60

LR Summarizes the relationship specificity and sensitivity in a single number.

Other criteria to assess quantitative Other criteria to assess quantitative measures:measures:

1.1. EfficiencyEfficiency

One aspect of efficiency is the number of One aspect of efficiency is the number of items on the instrument. Long instruments items on the instrument. Long instruments tend to be more reliable than shorter ones.tend to be more reliable than shorter ones.

Spearman – Brown formula , to Spearman – Brown formula , to estimate how reliable the scale would be estimate how reliable the scale would be with fewer itemswith fewer items

There are other 6 criteria to check the There are other 6 criteria to check the quality and it is related to reliability and quality and it is related to reliability and validity.validity.

Six CriteriaSix Criteria1.1. Comprehensibility:Comprehensibility: Subjects and researchers should be Subjects and researchers should be

able to comprehend the behaviors required able to comprehend the behaviors required to secure accurate and valid measures.to secure accurate and valid measures.

2. Precision:2. Precision: An instrument should discriminate An instrument should discriminate

between people with different amounts of between people with different amounts of an attribute as precisely as possible.an attribute as precisely as possible.

3. Speededness:3. Speededness: Researchers should allow adequate Researchers should allow adequate

time to obtain complete measurements time to obtain complete measurements without rushing the measuring process.without rushing the measuring process.

Criteria cont’dCriteria cont’d4. Range:4. Range: The instrument should be capable of

achieving a meaningful measure from the smallest expected value of the variable to the largest.

5. Linearity: A researcher normally strives to construct

measures that are equally accurate and sensitive over the entire range of values.

6. Reactivity: Instrument should avoid affecting the

attribute being measured.

Conclusion:Conclusion:• Quantitative Research studies are more Quantitative Research studies are more

common common • Easy to do and analyze Easy to do and analyze • Quality of the instrument must be assessed.Quality of the instrument must be assessed.• Reliability and validity are the main qualities.Reliability and validity are the main qualities.• Measure carefully to make the study findings Measure carefully to make the study findings

more relevant to use it for nursing or more relevant to use it for nursing or midwifery practice.midwifery practice.