SelectQuestion

ReviewResearch

DevelopHypothesis Research

Design

Measurement

Data Analyses CommunicateResults

Ethics

Overview of Research Process

Research Designs

Measurement

Data Analyses

Max Precision

Max Context

Scaling

Reliability & Validity

Max Generality

Current Focus

Qualitative

Runkey & McGrath typology

Particular Behavior SystemsUniversal Behavior Systems

ObtrusiveOperations

UnobtrusiveOperations

NaturalSettings

ContrivedSettings

FieldStudies

FieldExperiments

LabExperiments

MaximumContext

MaximumPrecision

MaximumGenerality

Formal Theory

Sample Surveys

SettingIndependent

Behavior notmeasured

ComputerSimulations

Runkel& McGrath, 1972

ExperimentalSimulations

How to do field research?

What is field research?• Examples

– Field Studies• Cross sectional

– Field Experiments• E.g., Longitudinal, prog evaluation

• Similarity and differences from– Other methods of data collection

• Large-scale (Sample) Surveys

– Methods statistical analyses• Correlational

Particular Behavior SystemsUniversal Behavior Systems

ObtrusiveOperations

UnobtrusiveOperations

NaturalSettings

ContrivedSettings

FieldStudies

FieldExperiments

LabExperiments

MaximumContext

MaximumPrecision

MaximumGenerality

Formal Theory

Sample Surveys

SettingIndependent

Behavior notmeasured

ComputerSimulations

Runkel& McGrath, 1972

ExperimentalSimulations

Why do field research?: General reasons

Describe Phenomena How satisfied are the employees

Establish standards How satisfied are our employees compared to another organization

Establish value added by a program

The effect of the new benefits program on employee satisfaction

Make decisions Should we continue with the new benefits program

Validate/test intuitions

Everyone else is using the new benefits program, is it any good?

Identify source of problem & potential solutions

Why are our employees dissatisfied? How to increase their satisfaction?

Describe

Predict

Explain

• Type of organizational change & development• E.g., self & peer evaluation of oral presentation

(Radhakrishnan & Yang, 2006)

• Two-way (symbolic) communication channel between employees & organization via content and conduct – e.g., UT Employee Survey

• Cox, T. Jr (2001). Creating the Multicultural Organization: A Strategy for Capturing the Power of Diversity San Francisco, CA: Jossey Bass

Why do field research?Organization-specific reasons

After deciding why you are doing field research, decide

how you will collect data

• Types of Data Collection Methods – Numerical vs. Non-numerical– Oral/Written vs. Observational– Behavioral vs. non-behavioral

• Each of the above types of data can be collected via all or some of the following– Questionnaires/Surveys– Observation (Archival)– Interviews

Methods of data collection• Bias in any one method is overcome if you

use multiple methods– Cf choosing research designs

• Some methods are better suited for measuring certain kinds of concepts – E.g., willingness & ability should determine

use of self report• Stereotype research

• Amount of resources used by method– Researchers resources– Participants’ resources

Time & resources restrict you to certain methods of data

collection

• Questionnaires • E.g., Field study, cross-sectional data

• Archival data• E.g., Field studies, Sample (large scale) Surveys

– If using, justify measures w/logic & research• e.g., ESL indicators

• Qualitative (non-numerical) data will take too long for collection & analyses

Instructor-Generated Exampleof a Questionnaire

• Hypothesis based on Rode et al., 2005, AOMLE – Additional control variable

• Renner, M. & Mackin R. (2000). A life stress instrument for classroom use in M. Ware & D. Johnson (Eds.) M. Handbook of demosntrations and activities in the teaching of psychology: Vol 1 Lawrence Erlbaum: Marwah, NJ.

Before designing your questionnaire identify

• Research hypothesis

• Predictor, criterion & explanatory variables

• Pre-existing measures of predictor & criterion variable– Bonus if you have measure of explanatory

variable

Why identify pre-existing measures for your questionnaire?• Examples of pre-existing measures

– Found in books on Reserve at CIRHR library– Psycinfo database:

• Search: Measures OR Questionnaires AND your topic keyword

• Why use pre-existing measures– Improves statistical reliability of your study– Improves validity of your study

• Disadvantages of pre-existing measures– E.g., UT study

How pre-existing measures improve validity

• Validity– Content based on definition of concept– Content can be based on qualitative data

generated by potential participants • E.g., critical incidents for ethnic harassment (EH)

measure (Schneider, Hitlan, & Radhakrishnan 2000) but see Swim et al EH measure

– Not all constructs need participant-generated data

• e.g., answers to an exam

How pre-existing measures improve reliability

• Reliability– If measure is tested on samples similar to your

sample, then you can be confident in the measure

• Schneider et al., 2000

– Can reasonably expect hypothesis to be supported if concepts are reliably measured

Pre-existing measures used in the instructor’s example

• Satisfaction measures – cited in Rode et al., 2005

• Performance Measure– Cited in Rode et al, 2005

• Control Variables– Citizenship replaced by primary language

question which is more appropriate– Not feasible to collect IQ measure in context– Stress measure

• Described in Renner & Mackin, 2000 Instructor slightly modified stem based on previous research (Schneider et al., 2000)

After deciding on measures, structure questionnaire

1. Content of Study information Sheet & Consent form

– See methodology assignment guidelines

2. Logic of ordering

3. Assess criterion variable first in cross-sectional study

4. Attractiveness via Visual Layout– Headings, Font size, White Space

More issues to consider when structuring questionnaire

5. Number of control variables & length of survey

– Shortening pre-existing measures is tempting but might damage reliability and validity.

6. Assessing sensitive variables– E.g., Class demonstration survey; UT survey

7. Ease of data analyses– Numbering sections & items– Number of Open-ended questions

Issues the Instructor faced when designing the examplar

questionnaire• Sensitive Variables

– Dropping additional demographic variable due to sample size

• What if the hypothesis is not supported– Restricted range on the GPA variable– Arguments to use stress as a control

variable vs. an antecedent

While or After designing questionnaire develop

sampling plan• Sampling plan depends whether you

want maximum precision, maximum context or maximum generality– E.g., maximum generality then need

random, large, representative sample

Particular Behavior SystemsUniversal Behavior Systems

ObtrusiveOperations

UnobtrusiveOperations

NaturalSettings

ContrivedSettings

FieldStudies

FieldExperiments

LabExperiments

MaximumContext

MaximumPrecision

MaximumGenerality

Formal Theory

Sample Surveys

SettingIndependent

Behavior notmeasured

ComputerSimulations

Runkel& McGrath, 1972

ExperimentalSimulations

Some terms in the area of sampling

• Population:– Group you are interested in obtaining data from

and studying.

• Sample:– Representative number of respondents from the

population that you sample.

• Actual sample:– The actual number of participants from your

sample that complete and return your survey

Types of Sampling You Can Hope vs. Actually do

• Every person in the population has exactly the same probability of being included in the sample to avoid bias.

• Sample is representative of the larger population.

• Representativeness can be checked by comparing the characteristics of a sample to those of the population – e.g., gender, age, tenure

Random Sampling

One Possible Modification of Random Sampling

• Stratification sampling:– Population divided into groups called strata.– Random selection from within groups.– Ensures representation on some critical factor

in the sample (e.g., gender, job category).

A Second Possible Modificationof Random Sampling

• Cluster sampling:– Participants chosen as members of a group

rather than as individuals.– Randomly select work teams, organizations,

factories, plans, facilities, etc.

Convenience Sampling(AKA what you will end up

doing for this course)

• Selection of participants based on easy availability or accessibility.

• Snowball or chain sampling – people who know people.

How to get a good sample size

• Provide incentives before or after.

• Indicate support from stakeholders.

• Convincing reason to complete it.

• Promise of feedback.

• Reminders.

• Personalize correspondence.

• Return envelope with postage / web-survey

What you learned today• Is your study a field study (or field expt)

or a sample survey?• Will you administer the questionnaire

yourself or collect archival data?• For both data collection methods you

need to use data collected with, or collect data with pre-existing valid & reliable measures– How to find reliable & valid measures– Why use them

• How to design a good questionnaire

• What sampling plan you can hope to use– How to get a large enough sample with the

sampling plan you will use

What you learned today