Sampling and Unit of Sampling and Unit of AnalysisAnalysis

EDL 714 Fall 2010EDL 714 Fall 2010

Related but Related but distinct…distinct…

Sample: the people, places, things, Sample: the people, places, things, phenomena you will be collecting data phenomena you will be collecting data about or from. about or from.

Unit of analysis: that which you seek to Unit of analysis: that which you seek to make claims about. make claims about.

They are NOT automatically the same thing. They are NOT automatically the same thing. Sorting out this distinction is absolutely Sorting out this distinction is absolutely fundamental. Quite frequently your sample fundamental. Quite frequently your sample will be strategically used to collect data will be strategically used to collect data regarding your unit of analysis, but will not regarding your unit of analysis, but will not be the same thing. be the same thing.

Let’s talk out some examples…Let’s talk out some examples…

3 research 3 research potholes potholes to avoid…to avoid…StructuringStructuring a research question in such a way a research question in such a way

that you can neither appropriately study it, or that you can neither appropriately study it, or make useful claims about.make useful claims about.

SelectionSelection and use of measurements, and use of measurements, instruments, or methods that are neither instruments, or methods that are neither credible nor rigorous.credible nor rigorous.

ChoosingChoosing a sample that does not allow clear a sample that does not allow clear and valid links to your research questions, and valid links to your research questions, nor generalization or transfer. nor generalization or transfer.

In research, sampling is destinyIn research, sampling is destiny (Kemper, et al., (Kemper, et al., 2003)2003)

Guidelines for Guidelines for samplingsampling

Apply to qualitative, quantitative, and Apply to qualitative, quantitative, and mixed method designs. mixed method designs.

Curtis, Gesler, Smith, & Washburn Curtis, Gesler, Smith, & Washburn (2000)(2000)

Kemper, Stringfield, & Teddlie (2003)Kemper, Stringfield, & Teddlie (2003)

#1… logic chain#1… logic chainThe sampling plan should be clearly linked The sampling plan should be clearly linked to the logical argument forged by the to the logical argument forged by the relationship of the research questions to relationship of the research questions to the conceptual framework and literature the conceptual framework and literature review. review.

Essentially… does the sampling plan Essentially… does the sampling plan logically support answering the research logically support answering the research question(s) in a credible and valid question(s) in a credible and valid manner? Can you actually address your manner? Can you actually address your question(s) about your unit(s) of analysis?question(s) about your unit(s) of analysis?

#2… sufficient #2… sufficient datadata

The sampling plan will generate the The sampling plan will generate the necessary and sufficient data, or evidence, necessary and sufficient data, or evidence, to understand and make claims about the to understand and make claims about the phenomenon under study (the latter being phenomenon under study (the latter being your unit of analysis). your unit of analysis).

All claims/conclusions must be evidence-All claims/conclusions must be evidence-based. based.

Is your study descriptive, interpretive, Is your study descriptive, interpretive, comparative, etc.? Your research question comparative, etc.? Your research question determines this for the most part. determines this for the most part.

#3... Credibility #3... Credibility (validity) (validity)

The sample allows the possibility that credible The sample allows the possibility that credible inferences can be drawn from the data. So not inferences can be drawn from the data. So not only is there a logical basis for making claims, only is there a logical basis for making claims, but there is also a chance your claims are but there is also a chance your claims are credible. (Logic and credibility are related but credible. (Logic and credibility are related but not the same.)not the same.)

Sample allows for the likelihood that other Sample allows for the likelihood that other potential causal factors or variables are potential causal factors or variables are accounted for. accounted for.

Conclusions are likely representative of the Conclusions are likely representative of the reality of the participants. reality of the participants.

#4… ethics#4… ethicsYour sampling strategy does not Your sampling strategy does not compromise the rules, regulation, and compromise the rules, regulation, and ethics regarding research with human ethics regarding research with human subjects (participants). subjects (participants).

Participation in the study is worth the Participation in the study is worth the time and involvement of participants. time and involvement of participants.

#5… you can do it#5… you can do itThe sampling plan is feasible and “do-The sampling plan is feasible and “do-able”able”

You can actually access the sample/data You can actually access the sample/data you plan to access.you plan to access.

It is physically, logistically, temporally, It is physically, logistically, temporally, spatially, metaphysically possible for spatially, metaphysically possible for the researcher to do what they plan. the researcher to do what they plan.

#6… extension#6… extensionConclusions of the study can transfer or Conclusions of the study can transfer or generalize to other contexts, circumstances, generalize to other contexts, circumstances, or participantsor participants

Analytical generalization is possible Analytical generalization is possible (transferability)(transferability)

Statistical generalization is possible Statistical generalization is possible (generalization)(generalization)

Practical relevance or applicability is possiblePractical relevance or applicability is possibleRequires a strong connection to practice and/or Requires a strong connection to practice and/or policypolicy

#7… balance#7… balanceThe sampling plan is practical.The sampling plan is practical.

The sampling plan is efficient.The sampling plan is efficient.

The sampling plan is necessary and The sampling plan is necessary and sufficient.sufficient.

The sampling plan is logical. The sampling plan is logical.

Sampling Sampling techniquestechniques

(use this language)(use this language)Probability (random) and purposive Probability (random) and purposive (non-random) strategies (non-random) strategies

Large-scale or experimental Large-scale or experimental quantitative studies will typically use quantitative studies will typically use probability sampling methodsprobability sampling methods

Case studies, mixed methods, or Case studies, mixed methods, or qualitative studies will typically use qualitative studies will typically use purposive sampling strategiespurposive sampling strategies

Probability Probability sampling sampling (random)(random)Sample: the smaller group, from a Sample: the smaller group, from a

larger population, from which data is larger population, from which data is obtained.obtained.

Population: the entire group from which Population: the entire group from which data data could becould be obtained. obtained.

Defining the population is an essential Defining the population is an essential task in selecting an appropriate sample. task in selecting an appropriate sample.

Random samplingRandom sampling

Population: all members of a group of Population: all members of a group of interest.interest.

Sample: a segment of that population.Sample: a segment of that population.

Inferential statistics seeks to Inferential statistics seeks to generalizegeneralize from samples to populations. from samples to populations.

Random sampling is the means by Random sampling is the means by which we try to achieve an unbiased which we try to achieve an unbiased sample.sample.

Variations on random Variations on random samplingsampling

Simple random sampleSimple random sample

Stratified random sampleStratified random samplePopulation divided by stratification variablesPopulation divided by stratification variablesEqual percentage then randomly drawn from Equal percentage then randomly drawn from each stratumeach stratum

Multistage random sampleMultistage random sampleTypically used in larger scale studiesTypically used in larger scale studiesStrata may be introduced by researcherStrata may be introduced by researcher

Random cluster sampleRandom cluster sampleSampling of pre-existing clusters of a Sampling of pre-existing clusters of a population already belonging to a group (i.e. population already belonging to a group (i.e. subgroup, grade level, single-parent families)subgroup, grade level, single-parent families)

Random sampling Random sampling and research in and research in

schoolsschoolsAssumptions of “random” assignment Assumptions of “random” assignment in quasi-experimental designsin quasi-experimental designs

Best approximations of equivalent Best approximations of equivalent groupsgroups

Matched-pair designMatched-pair design

Sample sizeSample size

Random samples may still include Random samples may still include sampling errors (in fact they likely do).sampling errors (in fact they likely do).

In general, the larger the sample the In general, the larger the sample the smaller the sampling errors are.smaller the sampling errors are.

In general, the larger the sample the In general, the larger the sample the greater the precision (think consistent greater the precision (think consistent replicability) of the results. replicability) of the results.

In general, think of a sample size of 25-In general, think of a sample size of 25-30 as our minimum for inferential stats. 30 as our minimum for inferential stats.

Sample sizeSample size

Increasing sample size produces Increasing sample size produces diminishing returns. diminishing returns.

The smaller the anticipated difference The smaller the anticipated difference in a population, then the larger the in a population, then the larger the sample size should be.sample size should be.

Even small samples can identify Even small samples can identify significant differences. significant differences.

For populations with very limited For populations with very limited variability, small samples can be very variability, small samples can be very precise. precise.

Sample sizeSample size

The more variable the population, the The more variable the population, the larger the sample size should be. larger the sample size should be.

When studying something rare (or of low When studying something rare (or of low occurrence), then larger samples are occurrence), then larger samples are usually required.usually required.

Large sample size cannot control for a Large sample size cannot control for a biased sample. It doesn’t change. biased sample. It doesn’t change.

Results from large samples can be Results from large samples can be potentially misleading, even if potentially misleading, even if statistically significant. statistically significant.

Purposive Purposive sampling (non-sampling (non-

random)random)Convenience samplingConvenience sampling

Extreme/deviant case and/or typical case Extreme/deviant case and/or typical case samplingsampling

Confirming/disconfirming casesConfirming/disconfirming cases

Homogenous casesHomogenous cases

Stratified purpose (quota)/random Stratified purpose (quota)/random purposive purposive

Optimistic and snowball samplingOptimistic and snowball sampling

Convenience Convenience samplingsampling

Most common (i.e. the teachers in my Most common (i.e. the teachers in my own building).own building).

Sample drawn from group easily Sample drawn from group easily accessible to researcher.accessible to researcher.

May not provide the best sample for May not provide the best sample for answering the research question(s).answering the research question(s).

Often result in spurious conclusions. Often result in spurious conclusions.

Extreme/deviant Extreme/deviant case,case,

Typical caseTypical caseSampling plans specifically designed to Sampling plans specifically designed to answer the research question at hand answer the research question at hand through highly strategic sampling through highly strategic sampling decisions. decisions.

Focus on Focus on outliersoutliers with extreme cases (i.e. with extreme cases (i.e. non-Spanish speaking adolescent ELLs with non-Spanish speaking adolescent ELLs with no formal educational experience).no formal educational experience).

Focus on Focus on archetypesarchetypes with typical cases (i.e. with typical cases (i.e. Spanish speaking ELLs with prior formal Spanish speaking ELLs with prior formal educational experience). educational experience).

Confirming/Confirming/disconfirming disconfirming

casescasesConfirming case designs seek specific Confirming case designs seek specific samples or cases that already fit a samples or cases that already fit a known pattern (i.e. a 55 year old known pattern (i.e. a 55 year old divorced female superintendent with divorced female superintendent with grown children).grown children).

Disconfirming case designs seek the Disconfirming case designs seek the opposite; cases that are clear opposite; cases that are clear exceptions to a pattern (i.e. a 45 year exceptions to a pattern (i.e. a 45 year old married female superintendent with old married female superintendent with school-age children).school-age children).

Homogenous Homogenous casescases

Identification of cases from a specific Identification of cases from a specific subgroup with common characteristics subgroup with common characteristics (i.e. female African-American high (i.e. female African-American high school students who did well in math). school students who did well in math).

Stratified purposive Stratified purposive (quota),(quota),

Random purposiveRandom purposiveStratified purposive: splitting a non-random Stratified purposive: splitting a non-random sample into smaller units based upon sample into smaller units based upon specified criteria (i.e. a non-random specified criteria (i.e. a non-random matched-pair design). The goal is to compare matched-pair design). The goal is to compare across groups within the larger group. across groups within the larger group.

Random purposive: randomizing within a Random purposive: randomizing within a larger purposive sample. Can be used in larger purposive sample. Can be used in quasi-experimental designs. Can add some quasi-experimental designs. Can add some credibility, but does not contribute to credibility, but does not contribute to statistical generalizability. Does allow for statistical generalizability. Does allow for certain pre/post designs to be used within certain pre/post designs to be used within the context of overall purposive sampling. the context of overall purposive sampling.

Both are useful for mixed-methods designs. Both are useful for mixed-methods designs.

ActivityActivityRefer to your prospectus self-design as needed.Refer to your prospectus self-design as needed.

Identify, assess and evaluate your sampling Identify, assess and evaluate your sampling plan based upon the guidelines for sampling we plan based upon the guidelines for sampling we have just discussed (use the worksheet have just discussed (use the worksheet provided).provided).

Define your sampling plan using the language Define your sampling plan using the language we have just covered. we have just covered.

Describe how your sampling plan supports Describe how your sampling plan supports investigating your unit of analysis. investigating your unit of analysis.

Share with a colleague. Praise and push. Share with a colleague. Praise and push.