Chapter ThirteenChapter Thirteen

Data Collection and Measurement

Measurement• The process by which categories or

numbers are used to reflect or indicate concepts and constructs

• A concept is a general idea not directly observable in the real world

• A construct is a concept specified in such a way that it is observable in the real world

Levels of a Research Study• Theoretical - interconnected propositions or

statements of relationship between concepts

• Conceptual - statements of relationships between two or more constructs

• Operational - indicates how each of the constructs will be measured or operationalized. It refers to the indicators used to reflect the constructs as well as to the procedures used to collect & analyze data

Theoretical Substruction

• The dynamic thinking process used to move from the theoretical level to the operational or measurement level of a study

• It illustrates the hierarchical order among the major constituents of a study

• It identifies the foundational elements of a study, determines the relationships among the elements, & presents this in a diagram

Measurement• Measurement is the linkage between the

conceptual and the operational levels of a research project

• Two key issues in this linkage: validity or the the congruence between a concept and the indicators of the concept, and reliability or the extent to which an instrument yields similar results on repeated measures

ValidityValidity

• Face validity..on the face of it...• Content validity…reflects the dimension

implied by the concept• Concurrent validity…correlation of one

measure with another• Predictive validity...predict accurately• Construct validity…distinguishes participants

who differ on the construct• Internal validity…treatment produces changes

in dependent variable

Validity Cont.

• Internal validity…treatment produces changes in dependent variable

• External validity…extrapolation from study to the other groups in general

• In qualitative research…”credibility” is the issue

Validity in Qualitative Research

• A qualitative study is credible when it presents descriptions of experiences that the people having had that experience immediately recognize as their own

• … the best test of rigor in qualitative work is when the researcher creates “true-to-life, and meaningful portraits, stories, & landscapes of human experiences…” (Sandelowski, 1993)

Rigor in Qualitative Research

• Keep careful records• Avoid the holistic fallacy• Guard against elite bias• Don’t be taken over by respondent

Reliability

• Instruments ability to produce the same results on repeated measures

• Terms such as dependability, consistency, stability & accuracy are often used interchangeably

• accuracy reflects the instrument’s ability to measure the true value (free from random measurement error) being measured

Reliability in Quantitative Reliability in Quantitative ResearchResearch

• Reliability is a relative term, expressed as a correlation …1.00 (perfect reliability) to 0.00 (absence of reliability)

• Reliability coefficients of .70 are acceptable (Nunnally, 1978)

• Estimates of reliability need to be determined each time the instrument is used

Three Attributes of Reliability

• Stability

• Internal Consistency

• Equivalence

Stability

• Concerned with consistency of results with repeated measures

• Test-retest procedures - response should be identical on both occasions assuming the variables measured remain the same at the two testing times

• Gillis (1997) tested the reliability of the ALQ using the test-retest procedure

Internal Consistency

• Refers to the homogeneity of the instrument or the ability of the items in the instrument to measure the same variable

• Items are strongly correlated to each other• The > intercorrelations, the > internal consistency• Measures to test internal consistency: KR-20,

item-total correlations, split-half method, cronbach’s alpha

Equivalence

• Degree of agreement among 2 or more different observers using the same measurement tool, or

• Degree of agreement among 2 or more alternate forms of an instrument or tool

• Determined by correlating the 2 scores with each other

• Interrater reliability may be determined several times in a study

Reliability in Qualitative Research

• In qualitative research replication is not possible because the circumstances & individuals can never be the same at some later time

Measurement Error

• Any deviation from the true value• True value is the underlying exact quantity

of a variable at any given time• Variables change over time & any measure

will vary slightly from 1 day to the next• Measure are made up of the

following:Measure=TV+ (SE+RE)

Measurement Error Cont.

• Systematic error…non-random error that systematically over- or under-estimates a value (eg., persons not answering a question are given the lowest value

• Random error…random fluctuations around the true value. Not a problematic…should average out.

Tips for Reducing Tips for Reducing Measurement ErrorMeasurement Error

• Take average of several measures• Use different indicators• Use random sampling procedures• Use sensitive measures• Avoid confusion in wordings• Error check data carefully• Reduce subject/experimenter expectations

Levels of MeasurementLevels of Measurement

The level of measurement achieved is important because it constrains the type of statistical analysis that can be performed on your data.

• Nominal• Ordinal• Ratio

The Effects of Reduced Levels The Effects of Reduced Levels of Measurementof Measurement

• Underestimating the relative importance of a variable if it is poorly measured

• The greater the reduction in measurement precision, the greater the drop in correlations between variables

• Precisely measured variables will appear to be more important than poorly measured ones

Data Collection• Process of gathering data from identified

participants to answer a research question• A variety of quantitative & qualitative methods

are available depending upon research question• indexes or scales, biochemical & physiological

measures, projective techniques, delphi techniques, unstructured interviews, focus groups, observation sessions, historical documents

Item AnalysisItem Analysis

• Good indexes “discriminate well” • Example of test item development

– test graded, students divided into upper and lower quartile

– examine performance on each question– select those questions that discriminate best

Discrimination of ItemsDiscrimination of ItemsPercent Correct Each Item

Bottom Top

Question # 25% 25% 1 40.0 80.0

2 5.0 95.0 3 60.0 55.0 4 80.0 80.0 5 10.0 40.0 6 20.0 60.0

Selecting Index ItemsSelecting Index Items

• Review conceptual definition• Develop measures for each dimension• Pre-test index• Pilot test index

Tips for Wording Likert Items

• The “and” alert: avoid multiple dimensions• Strongly Agree on right hand side 9-points

– response set issue• Avoid negatives like “not” simply use negative

wording. • Vary strength of wording to produce variation

in response• Exercise….items for a euthanasia index

Other ScalesOther Scales

• Semantic Differential: Here a variety of anchors are used and people place themselves or others on a continuum: shy/outgoing; bookworm/social butterfly

Other Scales Cont.

• Magnitude Estimations: subjects use numbers or line lengths to indicate perceptions. Very good for comparisons: yields ratio level measures. Comparing liking of teachers; seriousness of crimes; liking of one community compared to another one, etc.

Other Scales (cont’d)

• Visual Analogue Scales (VAS) measure the intensity of participant’s sensations & feelings about the strength of their attitudes, beliefs, & opinions about specific stimuli such as fatigue, pain, health, etc.

• Usually a 100 mm line is used with anchor words or phrases at each end

Delphi technique

• A panel of experts used for multiple data collection, analysis and processing

• Obtains the opinions of experts without the financial cost or inconvenience of bringing expert

• opinions of a variety of experts are condensed into precise statements people together

Physiological Measures

• Particularly appropriate in studies designed to assess the impact of nursing interventions on bodily functions

• Provide objective & sensitive measurements that are difficult for the participant to distort

• e.g. vital signs, % body fat, muscle strength, salivary enzyme levels, serum glucose, etc.

Observational Measurement• Well suited to phenomena that are best viewed from

a holistic rather than a reductionistic perspective

• Observations maybe structured, unstructured, or semi-structured; occur in natural or controlled settings

• To be scientific they must meet four critieria: consistent with study objectives; systematic & standard plan for recording; checked & controlled; related to scientific concepts or theories

Interviews

• A face-to-face verbal interaction to illicit information from the respondent usually through direct questioning

• structured, semi-structured, nonstructured• Advantage of probing, in-depth data, used

with participants who are not literate• Limited by time, cost , sample size