Innovation Surveys: Advice from the Oslo Manual

www.uis.unesco.org

Innovation Surveys:Advice from the Oslo Manual

South Asian Regional Workshop on Science, Technology and Innovation StatisticsKathmandu, Nepal

6-9 December 2010

www.uis.unesco.org

Ch 8 OM - Innovation Survey Procedures

Guidelines - collection and analysis of innovation data;

Comparable results;

Particular circumstances may require other methodology comparability!

www.uis.unesco.org

Populations

The target population:The target population:

• Business enterprise sector (goods-producing and services industries);

• At a minimum, all statistical units with at least ten employees.

www.uis.unesco.org

Populations

The frame population:The frame population:

• Units from which a survey sample or census is drawn = frame population;

• Basis: last year of the observation period;

• Ideal frame = up-to-date official business register established for statistical purposes NSOs;

• If the register forms the basis for several surveys (innovation, R&D, business), the information collected can be restricted to issues specific to innovation.

www.uis.unesco.org

Survey methods

Mandatory Mandatory surveys increase response rates;

Census or sample survey?Census or sample survey?

• Sample surveys should be representative of the basic characteristics of the target population (industry, size, region) a stratified sample is necessary;

• Census, though costly, might be unavoidable in some cases (legal requirement, small frame population, inclusion of all units in the frame with a certain number of employees).

www.uis.unesco.org

Survey methods

Domains Domains (sub-populations) are subsets of the sampling strata;

• Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active;

• Guidelines:» Same statistical units and classifications;

» Consistent methods;

» Documentation of deviations in data treatment or differences in the quality of the results (from the domains).

www.uis.unesco.org

Survey methods

Sampling techniques:Sampling techniques:

• Stratified sample surveys (reliable results): based on the size and principal activity of the units;

• Sampling fractions should not be the same for all strata: the sampling fraction of a stratum should be higher for more heterogeneous strata and for smaller strata.

Cross-sections: Cross-sections: standard approach - new random sample drawn from a given population;

Alternative/supplementary approach: panel datapanel data.

www.uis.unesco.org

Survey methods

Suitable respondents:Suitable respondents:

• Various methods: postal surveys, electronic surveys, personal interviews;

• Questions are very specialised and can be answered by only a few people in the unit;

• It is highly recommended to make a special effort to It is highly recommended to make a special effort to identify respondents by identify respondents by namename before data collection before data collection starts.starts.

www.uis.unesco.org

Survey methods

The questionnaire:The questionnaire:

• Pre-test before fieldwork;

• Simple and short;

• Order of the questions;

• Questions on qualitative indicators - binary or ordinal scale;

• International innovation surveys: translation and design of the questionnaire;

• Short-form questionnaires for units with little/no innovation activity previously reported.

www.uis.unesco.org

Survey methods

Combination of Innovation and R&D surveys:Combination of Innovation and R&D surveys: Reduction in the overall response burden; Scope for analysing the relations between R&D and

innovation activities at the unit level; Efficient method of increasing the frequency of innovation

surveys; It is possible to obtain reliable results for R&D expenditures;

Longer questionnaire; Units not familiar with the concepts of R&D and innovation

may confuse them; The frames for the two surveys are generally different.

www.uis.unesco.org

Survey methods

Guidelines for conducting combined surveys:Guidelines for conducting combined surveys:

• The questionnaire should have two distinct sections;

• Individual sections for R&D and innovation should be smaller than in separate surveys;

• Comparisons of results from combined surveys with those from separate innovation surveys should be done with care;

• Surveying methods should be reported;

• Samples to carry out such surveys should be extracted from a common business register.

www.uis.unesco.org

Estimation of results

Weighting methods:Weighting methods:

• The simplest one is weighting by the inverse of the sampling fractions of the sampling units, corrected by the unit non-response;

• Stratified sampling technique with different sampling fractions weights should be calculated individually;

• Commonly based on the number of enterprises in a stratum;

• In international and other comparisons, be sure that In international and other comparisons, be sure that the the samesame weighting method is used. weighting method is used.

www.uis.unesco.org

Estimation of results

Non-response:Non-response:

• Unit non-response: a reporting unit does not reply at all;

• Item non-response: response rate to a specific question / % of blank or missing answers among the reporting units;

» Disregarding missing values and applying simple weighting procedures based only on the responses received implicitly assumes that non-respondents are distributed in the same way as respondents biased resultsbiased results;

» Possibility: imputation methodsimputation methods to estimate missing values on the basis of additional information.

www.uis.unesco.org

Presentation of results

Descriptive analysis:Descriptive analysis:• Description of the

statistical units in terms of their innovative or non-innovative activities without drawing any conclusions about the underlying survey or target population;

• No generalisation of the results;

• Unit non-response rate is of minor importance.

Inferential analysis:Inferential analysis:

• Drawing of conclusions about the target population;

• The results should give a representative estimation of the situation;

• Weighted results;

• Unit non-response rate is very important.

www.uis.unesco.org

Presentation of results

Errors:Errors:

• Random errors due to the random process used to select the units;

• Systematic errors containing all non-random errors (bias);

» Results’ variance: it is recommended to calculate both (average) values for innovation indicators and also their coefficients of variation and/or confidence intervals;

» Results presentation: metadata (including information on data collection procedure), sampling methods, procedures for dealing with non-response and quality indicators.

www.uis.unesco.org

Frequency of data collection

Innovation surveys: every two years;

When not economically feasible three or four years;

Surveys must always specify an observation period for questions on innovation;

• The length of the observation period for innovation surveys should not exceed three years nor be less than one year.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts

Information system specificities:

• Relative weakness of statistical systems

• Absence of linkages between surveys and data sets; lack of official business registers information from other surveys cannot be used;

• Involvement of NSOs;

• When lacking, basic variables about firms’ performance can be included in the innovation survey - to enable further analysis.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts

General methodological considerations:

• Questionnaire design: » Separated sections - different respondents;

» Guidance / definitions;

» Language and the translation of technical terms;

• Survey application:» In-person;

» Trained personnel.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts

General methodological considerations:

• Frequency:» Every three to four years (e.g., timed to CIS rounds);

» Try to update a minimum set of variables every year;

• The purpose of surveys needs to be clearly stated and the questions clearly formulated;

• An adequate legislative base for the collection of innovation statistics can help ensure the success of such an exercise;

• The results should be published and distributed The results should be published and distributed widely.widely.

www.uis.unesco.org

Thank you!

http://www.uis.unesco.org

l.marins@uis.unesco.org

(CIS: http://www.oecd.org/dataoecd/37/39/37489901.pdf)