Recommended Practices for the

design of business surveys

questionnaires Stefania Macchia

macchia@istat.it

(Istat, National Accounts and Economic Statistics Department)

NTTS 2015 Conference

Bruxelles, 10-12 March

2

ISTAT Business Statistical Portal implements a new approach for the organization and

management of data collection processes. It constitutes a single entry point for web-

based data collection from enterprises according to a “business-centric” perspective.

Process 1

Businesses register

Process n

Businesses

Statistical Portal

Data capturing systems

Communication

The context

High integration among all the phases of the productive processes: survey design, data

collection, frame, design of electronic questionnaires, elaboration, estimations, informative

«return»

Rationalization and intensification of the informative flows among «stakeholders» involved in

statistical processes (production lines, statistical registers, data capturing systems, dissemination and

communication, statistical units)

Introduction of new management procedures characterised by high interrelation degree among Portal,

production lines, production support facilities

High potential in terms of systemic efficiency

Main

Fea

ture

s:

Harmonisation of business surveys questionnaires

3

The new Business Portal architecture implies the acceleration of the harmonization

of survey questionnaires, both in terms of concepts treated and features of

electronic questionnaires, with the perspective of:

reducing respondent burden in terms of duplication of information requested

adopting a common language of communication with enterprises through a

standard approach for questionnaires design, which takes into consideration the

peculiarities of the different business surveys questionnaires (SBS and STS),

with attention to the variables definitions management and to the structure of

questions to collect information

enhancing the level of quality (in terms of response rates, but also

homogeneity of collected data).

Manual of Recommended Practices

for the design of business surveys

questionnaires

Manual of Recommended Practices for the design of

business surveys questionnaires

Not all the aspects of the structured process to design questionnaires have been treated

because, in this context, the aim was not to plan new surveys ex novo, but to restructure

and harmonize surveys to be included in Business Portal.

The manual is concentrated on:

The definition of the questionnaire schemes

All the aspects of the questionnaire design related to the data capturing technique

4

GSBPM v.5:

2.2 Design variable descriptions

2.3 Design collection

Questionnaire Electronic Questionnaire

design design

‘The sub-process 2.2 may need to run in parallel with sub-

process 2.3, as the definition of the variables to be collected, and

the choice of collection instrument may be inter-dependent’

Recommended Practices for the design of business

surveys questionnaires

In defining Recommended Practices, it has been considered how some aspects of the

design of questionnaires and of the features to be implemented in the electronic

questionnaire have an impact on the steps of the response process, represented in a

‘hybrid response model’ based on a cognitive approach.

5

Encoding in Memory / Record Formation

Selection / Identification of Respondent(s)

Assessment of Priorities (Motivation)

Release of the Data

Comprehension

Retrieval

Judgement

Communication.

The questions wording, their structuring, the layout

and all the instructions and information at disposal

of respondents related to each question impact on

the comprehension and retrieval processes.

The skipping and consistency rules implemented in

electronic questionnaire support for the judgement

step

the electronic questionnaire design impact on the

communication process, as it predefines the

structures according to which responses have to be

given.

the questionnaire design impact on the Encoding

in memory/Record formation step because as more

the way data are stored in the respondent data base is

considered in structuring questions, as more the effort

needed to provide data is contained.

Ibrid response model (2000 Sudman, 2001 Wilimack,

Nichols)

Manual of Recommended Practices for the design of

business surveys questionnaires

6

First section of the Manual:

General principles and particular ones related to the CAWI technique (shortly described)

Principles defined according to a strategy which should:

Encourage responses

Contain responses errors

Reduce the effort necessary to provide answers

Give respondents a feedback

Very important general principle, consistent with an integrated approach:

contain information to be collected, using sources already available like:

Administrative/fiscal sources

Data collected in other surveys

Official registers.

Manual of Recommended Practices

7

The set of recommendations provided are organised in paragraphs according to specific

homogeneous themes:

1) Design of questionnaire sections/Web pages

Suggestions are given on how designing the sections, taking into considerations the Web

pages, which in turn, can be composed by more than one computer screen-shots.

2) Questions design

Identification of the structure to be used to provide the answer: suggestions are provided on

quantitative questions (how to manage decimal values, scale factors, measure units, etc.),

closed questions (when using radio-buttons, drop-down-boxes, check-boxes), free texts

questions to be coded (when it is preferred to use assisted coding functions) and, finally,

questions organised in tables/arrays.

3) Use of classifications

When classifications are used to classify textual responses, suggestions are given on

different algorithms for textual matching which can be used.

Manual of Recommended Practices

8

4) The structure for variables definitions

Harmonising the definitions in terms of their content is surely the first problem to be faced

but how this content is expressed and structured has in impact on the comprehension

process as well. For this reason a common modular structure for definitions of variables

has been defined

5) The management of skipping rules

The automatic management of skipping rules is a strength of data collection computer

assisted techniques. Concerning business surveys questionnaires, different suggestions

are given for skipping rules related to questions belonging to the same section/web page or

to different section/web pages.

6) The checking plan

Suggestions are given on the management of the consistency rules, taking into

consideration aspects, like: rules regarding variables of the same web-page or different

ones, number of checking rules on a single web-page, relation between skipping and

checking rules, etc.

7) The screen design

Standards already defined for software package used in the Business Portal architecture

have been adopted. Some details have been provided on graphic symbols to be used to link

definitions, instructions, helps, etc.

Manual of Recommended Practices

9

8) The guide and instructions for respondents

A set of documents is always provided to respondents (e.g. brief description of the survey,

normative aspects concerning confidentiality, compilation guide, etc.). Common structures,

templates and access modes have been designed for these documents, because they can

have an impact on the comprehension process as well.

9) Questions to estimate the perceived respondent burden

The migration of surveys questionnaires in the new generalised sw system for electronic

questionnaires implementation is a good occasion to collect information on respondent

burden directly from respondents. For this purpose a set of questions has been proposed,

which distinguish possible burden due to difficulty in retrieving the requested information

and in filling in the questionnaire. The responses to these questions could be analysed

together with other indications derivable from paradata.

10) The use of paradata for data collection monitoring and optimising questionnaire

design

The Business Portal already produces some indicators for the monitoring purpose. A set of

other indicators has been identified which can be used by the researcher to optimise the

questionnaire design, specifically when some ‘alerts’ are highlighted by the answers given to

questions on the respondent burden (e.g. number of entries for web-pages, connection time

for the entire questionnaire or for each web-page, length of navigation routings, etc.).

10 10

The structure for variables definitions The harmonisation of the

contents to be observed

In business statistics some variables can be collected in different surveys

questionnaires. This can depend on different informative needs (structural and short-

term surveys), while in other situations it could sometime result to be redundant.

On the other hand, big business agencies are often involved in more surveys.

Definitions of the same variables could be different:

in terms of their content because EU regulations contain different specifications for structural

and short-term surveys, or

in terms of wording, because expressed differently by each survey responsible.

Contents harmonisation in terms of variables definitions and questions wording is

the pre-requirement for a two steps activity:

Improvement of the harmonisation and comparability of collected data

Reduction of redundancies and respondent burden

11 11

A common modular structure for definitions of variables has been defined,

which will be a standard for all variables definitions.

The same informative content can be presented to respondents through different wordings, different

structures and different layouts.

All these aspects have an impact on the comprehension process.

The structure for variables definitions The harmonisation of the

contents to be observed

A Definition (core of the definition = real meaning, expressed in a

simple, unambiguous and synthetic way)

B Inclusions/Exclusions clauses

C National peculiarities

D Peculiarities of sectors/surveys

This structure is a useful tool for the improvement of harmonisation and comparability of collected

data. It makes it easier:

converging on the core of the definition

identifying differences of informative needs and, consequently, structuring questions so as to

reduce redundancies in data collection

12 12

Two variables, used by different survey, have been considered with the

purpose of harmonising definitions and testing the standard structure:

Number of employees

Turnover

Sources used to produce harmonised definitions

Eurostat Regulations on SBS and STS surveys

Definitions already used in Istat surveys (SBS and STS)

Indications produced by ESSNet project on consistency (WP3)

Definitions of Istat Business register

Definitions of Business census.

The structure for variables definitions The harmonisation of the

contents to be observed

REFERENCES

14

• J.Bethlehem, S.Biffignandi (2011) Handbook of Web Surveys, eds Wiley

• Eurostat (2006) Handbook of Recommended Practices for Questionnaire Development and

Testing in the European Statistical System

http://ec.europa.eu/eurostat/ramon/statmanuals/files/Handbook_of_Practices_for_Quest.pdf

• Eurostat (2014) Memobust handbook - Handbook on Methodology of Modern Business

Statistics http://www.cros-portal.eu/content/memobust

• N.Fazio, M.Murgia, A.Nunnari (2013) The Business Statistical Portal: a new way of

organising and managing data collection process for business surveys in ISTAT Proceedings

of the UNECE Seminar on Data Collection, Ginevra

• Sudman, S., D. K. Willimack, E. Nichols and T.L.Mesenbourg (2000) “Exploratory

Research at the U.S. Census Bureau on the Survey Response Process in Large Companies,” Proceedings of the Second International Conference on Establishment Surveys, American

Statistical Association,