Upload
fatima-cale
View
212
Download
0
Tags:
Embed Size (px)
Citation preview
ARIZONA IN THE GREEN ECONOMY
ARIZONA IN THE GREEN ECONOMY
Lessons Learned in Designing an Effective Business
SurveySession 4
April 26, 2011
CREDITS AND DISCLAIMER
2
This report was funded by a grant awarded by the U.S. Department of Labor’s Employment and Training Administration. The report does not necessarily reflect the official position of the U.S. Department of Labor. The Department of Labor makes no guarantees, warranties, or assurances of any kind, express or implied, with respect to such information, including any information on linked sites and including, but not limited to, accuracy of the information or its completeness, timeliness, usefulness, adequacy, continued availability, or ownership. State of Arizona has the copyright to this report. Internal use by an organization and/or personal use by an individual for non-commercial purposes is permissible. All other uses require the prior authorization of the copyright owner.
SESSION OVERVIEW:PREPARING FOR A SURVEY Defining the survey purpose
Identifying survey recipients
Defining key concepts
Designing the survey instrument
Preparing the survey cover letter
DEFINING THE SURVEY PURPOSE
4
4 COMMON ANALYTIC RESEARCH METHODS
1. Archival Use of available
administrative records to analyze the past or forecast the future
Observations used from a time period before a research question is posed
2. Experiment/Quasi-experiment
“Pilot project” compared against the “real world”
Controls for a single variable Helps to factor out other
explanations for the occurrence of a few observations
3. Case study In-depth examination of a
particular story Review multiple influences Develop theories about possible
causal explanations
4. Survey Primary data collection Surveys typically focus
gathering a larger number of responses that can be analyzed using statistical methods
Interviews and focus groups that can be “de-constructed” and easily compared (often provide in-depth companion info for surveys)
5
DEFINING THE SURVEY PURPOSE:ANSWERING EXPLORATORY QUESTIONS
Gathering and reporting facts or opinions
Examples from the Arizona Green Survey: How many Arizona companies have “green”
economic activity? What proportion of Arizona’s employment is
dedicated to “green” economic activity? How rapidly are employers planning to add
“green” jobs? What are the wages provided to workers in these
“green” jobs?
DEFINING THE SURVEY PURPOSE:TESTING SPECIFIC HYPOTHESES
Assessing whether patterns emerge as expected
Examples from the Arizona Green Survey: In which industries does Arizona have a high
concentration of green activities? How much faster do companies with green jobs grow
relative to all other companies? How much more do jobs requiring green skills pay
than other jobs in the same occupation?
BIASES THAT MUST BE UNDERSTOOD
The selection of questions reflect the biases and perspectives of the researchers The questions being posed often include inherent
biases
Answers to survey questions reflect the perspective of respondent
In business surveys, multiple respondents may answer differently
The selected sample may not necessarily reflect the general population
Difficulty in getting “accurate” responses from larger and smaller businesses
IDENTIFYING SURVEY RECIPIENTSWho are the businesses that you are surveying?
9
DATABASES FOR BUSINESS LISTS Quarterly Census of
Employment and Wages Availability Suitability of contacts
Partners and stakeholders Availability Reliability
Proprietary vendors (e.g., InfoGroup; D&B) Cost Accuracy and
completeness
10
SELECTING A VALID SURVEY FRAME
Determining how to narrow the list Geography, industry, size, business age,
available contact info, etc.
Cleaning up the list Time, access to information, accuracy of
alternative data sources
Determining appropriate contacts within the company Related to questions being asked
11
GETTING THE RIGHT SAMPLE SIZE Identify sub-groups of survey recipients (“stratification”)
Identify potential biases within sub-groups and the need for “oversampling” some groups
Estimate expected response rates and overall sample size required
Establish quotas for sub-groups in the sample (balancing resources with desire for of statistical validity)
Adjust the sample size to reflect sub-group quotas as well as expected overall response rate
12
DEFINING KEY CONCEPTS13
WHAT ARE YOU TRYING TO MEASURE? Operationalizing key concepts
“Firm” vs. “establishment” “Jobs” “Employment” “Wages”
Identifying terms with “shared definitions”
Understanding the limits What can you expect respondent to ‘know’?
Selecting appropriate time periods The power of memory The limits of projecting the future (OR the problem with
“hypothetical questions”)14
EXAMPLE: DEFINING “GREEN” JOBS
Examples Of Green Activities
Making Goods or Providing Services
Improving Processes in Existing Goods or Services
Energy generation Producing renewable energy
Improving fuel efficiency
Energy efficiency in buildings, equipment, appliances, or vehicles
Building long-lasting batteries
Implementing smart grid technologies
Pollution reduction/removal, green-house gas reduction, recycling and reuse
Producing industrial scrubbers or generating nuclear energy
Managing energy proactively
Natural resources conservation
Organic farming Conserving soil, water, or wildlife
Environmental compliance, education and training, and public awareness
Training in regulatory compliance
15
AN EXAMPLE: “RENEWABLE ENERGY” What are key elements of the “renewable energy”
industry? Which segments do we focus on? How do we handle nuclear energy or other segements?
What are related industries? …activities? …occupations?
How do you handle suppliers that might not be unique to solar such as… Maintenance and repair? Accountants and attorneys? Educational institutions Utilities? Investors?
What occupations would want to include? 16
ANOTHER EXAMPLE:OPERATIONALIZING “INNOVATION” IN RENEWABLE ENERGY
Defining innovation Innovation—introducing something new—mostly
derives from technological advances that ultimately lead to productivity and prosperity gains.
Communities must provide an appropriate foundation to realize those gains, including: A strong educational system A solid physical infrastructure, A base of research and development activities, Access to financial resources, and other factors.
Science and engineering—in the educational system and in the workforce—are of particular importance.
17
Source: Adapted from Arizona Indicators, Morrison Institute for Public Policy
http://arizonaindicators.org/innovation
SELECTED WAYS TO MEASURE INNOVATION
Human capital University entrance
exam scores Graduate students
and postdoctorates Employed science,
engineering & health doctorate holders
Patents granted
Financial capital Research and
development Innovation grants Venture capital
High tech employment High tech industries High tech
occupations
18Source: Arizona Indicators, Morrison Institute for Public Policy
HOW COULD FURTHER RESEARCH HELP US BETTER UNDERSTAND “RENEWABLE ENERGY” RELATED TO…?
…Arizona’s educational system?
…the state’s physical infrastructure?
…research and development activities?
…access to financial resources?
…the workforce’s preparation for emerging jobs in this area?
19
DESIGNING A SURVEY INSTRUMENTCreating and aggregating information about a large number of respondents
20
5 ALTERNATIVE INTERVIEW METHODS
Interview type Advantages Disadvantages
Personal interview
•Stronger commitment to completing the interview.•Greater confidence in finding the right recipient•Longer interviews tolerated
•More costly•Greater potential for creating a non-representative sample
Telephone surveys
•Quicker turn-around •Wide availability of telephone numbers•Availability of CATI to address complex questions•Role skilled interviewers to elicit more complete answers
• Negative perception of surveyors as “telemarketers”•Challenges of getting past “gatekeepers” (e.g., administrative assistants, caller ID)
Mail surveys •Relative expense per response•Ability to use only names and address to reach respondent•Can use photos•Answer at respondent’s leisure•Viewed as less intrusive
•Survey takes longer•Response rates lower•Respondents are more likely to have “something to say” •Less ability to correct “misunderstanding” of questions
21
5 ALTERNATIVE INTERVIEW METHODS(CONTINUED)
Interview type Advantages Disadvantages
Email surveys •Inexpensive with quick turnaround•Simple•Allows attachments
•Cannot handle complex logic•Requires having email addresses•Perceptions about viruses and spam•Potential missed messages due to “junk” mail filters•Difficult to manage sample due to ease of forwarding•Loss of anonymity•Response bias/inability to generalize
Web surveys •High speed in getting responses•Low cost•Use of email to generate responses•Flexibility in creating complex logic•More responsive to ‘sensitive’ questions•Greater ability to use open-ended questions•Respondent inputs data
•Increase survey “quitting”•Restricting response to the “sample” •Even greater difficulty in generalizing results
22
TIPS IN SURVEY DESIGN
Keep it Short and Simple (KISS)
Keep questions simple and specific
Start with an interesting question What makes a question
interesting?
Ask “answerable” questions
Taking care with question skipping
Recognize that surveys “reveal a story” about the research Encourage a “logic flow”
for questions Understand how
question order might affect response 23
TIPS IN WRITING QUESTIONS
Allow don’t know, not applicable, or OTHER options Avoids “skipping” and early
termination
Use forced-choice questions whenever possible
Create appropriate answer categories How many categories?
Ensure that answer options are complete AND mutually exclusive
When to use open-ended questions Provide context for specific questions or
survey You get 1 or 2 shots at most
Recognize respondents’ desire to please Will affect responses Omit the middle alternative and
focus on “intensity” Provide option for “no opinion” or
“not applicable”
Watch for “double-barreled” or “negative” questions? About people or policies that might be
controversial Using words that unintentionally evoke
emotion24
THE “STORY” BEHIND THE SURVEYWHY DID WE CHOOSE THIS STRUCTURE?
25
KEY GOALS IN PRE-TESTING THE SURVEY INSTRUMENT
Variation
Meaning
Task difficulty
Respondent interest and attention
Flow and naturalness of the sections
The order of questions
Impact of skip patterns
Timing
Overall respondent interest and attention
Respondent reaction
Specific Questions Instrument as a Whole
26
PREPARING THE SURVEY COVER LETTER
27
TIPS FOR A GREAT COVER LETTER
Keep the letter short Personalize it to an
individual whom you want to respond
Tell the recipient how they could benefit directly (or pay them for their time)
Tell them how long it will take and BE HONEST
Give them options for responding
Provide a deadline within a week of receipt
Say something memorable (funny is good, but not offensive)
Use the “right” logo Careful about who signs
the letter
28
DEVELOPING YOUR OWN SURVEYAn Exercise29
RESOURCE IN DESIGNING AN INNOVATION SURVEY
30
WORKING GROUP EXERCISE: DESIGNING A BUSINESS SURVEY ON INNOVATION IN RENEWABLE ENERGY
1. Issues to explore: How many Arizona renewable
energy companies are
“innovative”?
What are the key characteristics of
“innovative” renewable energy
companies?
2. Hypotheses to test: “Innovative” companies were
much more likely to add jobs likely
to add jobs during the recession.
“Innovative” companies need
different kinds of skills and training
than do other renewable energy
companies.
What are the key talking
points in the letter?
Who should sign it and
why?
Design an Instrument– 2 survey design groups
Prepare a Cover Letter -- for each group
31
Who should get the survey?
What subgroups are there?
How should the survey be
adapted to the subgroups?
Determine Sample Frame-- for each group
THANK YOU