What is actually measured in process evaluations for ... 3.pdf · (3) explore the relationship between effectiveness and the implementation process. Methods: Pubmed, EMBASE, PsycINFO,

What is actually measured in process evaluations for

worksite health promotion programs: a systematic review

Wierenga D, Engbers LH, van Empelen P, Duijts S, Hildebrandt VH, van Mechelen W.

BMC Public Health 2013, 13(1):1190

3

R1R2R3R4R5R6R7R8R9

R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39

42

Chapter 3

Abstract

Background: Numerous worksite health promotion program (WHPPs) have been

implemented the past years to improve employees’ health and lifestyle (i.e., physical

activity, nutrition, smoking, alcohol use and relaxation). Research primarily focused on the

effectiveness of these WHPPs. Whereas process evaluations provide essential information

necessary to improve large scale implementation across other settings. Therefore, this

review aims to: (1) further our understanding of the quality of process evaluations alongside

effect evaluations for WHPPs, (2) identify barriers/facilitators affecting implementation, and

(3) explore the relationship between effectiveness and the implementation process.

Methods: Pubmed, EMBASE, PsycINFO, and Cochrane (controlled trials) were searched

from 2000 to July 2012 for peer-reviewed (randomized) controlled trials published in English

reporting on both the effectiveness and the implementation process of a WHPP focusing

on physical activity, smoking cessation, alcohol use, healthy diet and/or relaxation at work,

targeting employees aged 18-65 years.

Results: Of the 307 effect evaluations identified, twenty-two (7.2%) published an additional

process evaluation and were included in this review. The results showed that eight of those

studies based their process evaluation on a theoretical framework. The methodological

quality of nine process evaluations was good. The most frequently reported process

components were dose delivered and dose received. Over 50 different implementation

barriers/facilitators were identified. The most frequently reported facilitator was strong

management support. Lack of resources was the most frequently reported barrier. Seven

studies examined the link between implementation and effectiveness. In general a positive

association was found between fidelity, dose and the primary outcome of the program.

Conclusions: Process evaluations are not systematically performed alongside effectiveness

studies for WHPPs. The quality of the process evaluations is mostly poor to average,

resulting in a lack of systematically measured barriers/facilitators. The narrow focus on

implementation makes it difficult to explore the relationship between effectiveness and

implementation. Furthermore, the operationalization of process components varied

between studies, indicating a need for consensus about defining and operationalizing

process components.

R1R2R3R4R5R6R7R8R9R10R11R12R13R14R15R16R17R18R19R20R21R22R23R24R25R26R27R28R29R30R31R32R33R34R35R36R37R38R39

43

Systematic review of implementation studies

3

Introduction

Employees with unhealthy lifestyle behaviors and overweight or obese employees are less

productive at work, show a decreased work ability and take more sick days compared to

employees with a healthy lifestyle [1-4]. An unhealthy lifestyle can be characterized by one

or more of the following behaviors; low physical activity levels, an unhealthy diet, smoking,

frequent alcohol use and poor levels of relaxation (i.e. mental health and vitality). Because

employed adults spend about half of their workday waking hours at their workplace, the

worksite may be an effective setting to increase employees health and productivity [5-8].

Therefore, in the past decade, numerous worksite health promotion programs (WHPPs)

have been conducted to improve employees’ health and lifestyle. However, not all of these

programs were successful [5-7].

In order to change employees’ health and lifestyle, programs need to work as intended

(and therefore avoid theory failure). Additionally, they must be effectively transferred

from research to practice and be maintained over time [9]. Programs often fail to reach

potential participants adequately due to a lack of adoption, communication, or program

sustainability which could lead to low participation levels (in other words, program failure)

[10]. Furthermore, research starts to recognize the importance of evaluating different

implementation outcomes (such as recruitment, dose delivered, dose received, fidelity,

satisfaction, and maintenance) and the contextual factors that hinder or facilitate the

implementation of a WHPP [11-14]. These contextual factors are related to characteristics

of the context, organization, implementer, program and participants [11, 12, 15]. Since

WHPPs are often comprehensive interventions with multiple components, it is difficult to

determine the overall level of implementation and investigate which specific intervention

components have been successful [14, 16].

A review by Durlak and Dupre (2008) showed that higher levels of implementation

improve program outcomes, suggesting that there should be an adequate focus on the

implementation process [13]. However, most research focuses primarily on measuring the

effects of a WHPP, with an occasional process evaluation performed after implementation.

However, systematic process evaluations can produce valuable insights into the interpretation

of the (lack of) effects of an intervention by identifying successful and unsuccessful

program components, thereby allowing researchers to optimize their program [16-19].

Furthermore, process evaluations can help to identify barriers and/or facilitators influencing

the implementation process, while taking into account the different actor levels at which

R1R2R3R4R5R6R7R8R9


44

Chapter 3

these factors play a role [16, 18]. These are valuable outcomes that can be used to improve

program implementation in the future and across other settings. Hence, effect evaluations

should be accompanied by systematic and real-time process evaluations [20, 21].

Murta et al. (2007) looked at the quality of process evaluations accompanying controlled

trials studying individual based stress management interventions. They showed that process

evaluations are often incomplete and not systematically conducted, meaning that they lacked

a theoretical framework and were not planned prior to implementation [22]. However, the

quality of the process evaluations that were included in this review was insufficient to make

sound conclusions in terms of best predictors for successful interventions. Additionally,

Murta’s review only focused on evaluations of occupational stress management

interventions published between 1977 to 2003. So their conclusion cannot directly be

transferred to recent WHPP focusing on other lifestyle behaviors. In their review, Murta also

concluded that the framework of Steckler and Linnan proved to be a useful tool to conduct

process evaluations. But in this framework little attention has been given to the great variety

of contextual factors that can either hinder or facilitate the implementation process. We

therefore recently proposed a framework for a systematic and comprehensive process

evaluation based on several theoretical frameworks to gain insight into the implementation

process (figure 1) [11, 12, 14, 15, 23]. The four main aspects of this framework relate to

determinants of implementation that may influence the implementation process, and the

implementation process itself (i.e. adoption, implementation and continuation). These four

main aspects are operationalized using a combination of the framework of Steckler and

Linnan for process evaluations and the RE-AIM framework [14, 23]. So, in order to gain

insight in the implementation process, eight different process components at three different

actor levels (macro-level: organization and management; meso-level: implementer; micro-

level: participant) need to be evaluated using a mixed methods approach [11]. Of these eight

components, six components focus on implementation (reach, recruitment, dose delivered,

dose received, fidelity, and satisfaction), one component (maintenance) on continuation and

the eighth component (context) refers to the determinants of implementation. Since many

implementation determinants can be identified, context is further defined by categorizing

the barriers and/or facilitators into five main categories: 1) characteristics of the socio-

political context, 2) characteristics of the organization, 3) characteristics of the implementer,

4) characteristics of the intervention program, 5) characteristics of the participant [11, 12,

15].


45


3

Some reviews have focused on process evaluations and relevant implementation

barriers and/or facilitators [10, 12, 13, 22, 24, 25]. However, these reviews focused on

domains other than WHPPs [12, 13, 24, 25], limited the identification of barriers and/or

facilitators to those affecting WHPP participation levels [10], or focused only on process

evaluations accompanying stress management interventions [22]. To our knowledge, there

has been no systematic review that has examined process evaluations for worksite health

promotion programs targeting lifestyle change among employees. The aim of this review was

therefore to: (1) further our understanding of the quality of process evaluations alongside

effect evaluations for worksite health promotion programs (WHPPs), (2) identify barriers/

facilitators affecting implementation, and (3) explore the relationship between effectiveness

and the implementation process.

Characteristics of the socio-political context

Characteristics of the organization

Characteristics of the innovation

Characteristics of the adopting person/user

Adoption

Implementation

Continuation

Implementation strategy

Innovation process Innovation determinants

Figure 1. Theoretical framework

R1R2R3R4R5R6R7R8R9


46

Chapter 3

Methods

Literature search and study selection

For this systematic review, peer-reviewed studies were eligible for inclusion when they

reported an effect evaluation as well as a process evaluation for a worksite health promotion

intervention focusing on stimulating a healthy lifestyle (physical activity, nutrition, smoking,

alcohol use and relaxation) published in English between 2000 and July 2012. The literature

search was conducted in two steps.

Step 1: a literature search was in the online databases Pubmed, EMBASE, PsycINFO, and

the Cochrane Central Register of Controlled trials for peer-reviewed WHPP effect evaluations

published in English from 2000 to July 2012. Searches included the following combination of

keywords: (Randomized controlled trial OR controlled trial) AND (worker* OR employee* OR

worksite OR work environment) AND (worksite health promotion OR lifestyle intervention)

AND (absenteeism OR sickness absence OR body mass index OR lifestyle OR health behavior

OR cholesterol level). To ensure no studies were overlooked this search string was repeated

in each database with the addition of the following keywords for each lifestyle behavior

separately: ‘physical activity’, ‘smoking’, ‘alcohol use’, ‘healthy diet’ and ‘relaxation’. Since

we were only interested in studies focusing on change in actual behavior rather than change

in attitude, social norm or self-efficacy keywords were added on outcome measures related

to actual behavior change. All the keywords were specified for each database using Mesh

or Thesaurus terms. The complete search strategy for each database and lifestyle behavior

are presented in appendices 1. The inclusion criteria for the first stage of the selection

process were: (1) a randomized controlled (RCT) or controlled trial (CT) published in English

between 2000 and July 2012, (2) an evaluation of the effects of worksite health promotion

interventions focusing on physical activity, smoking cessation, alcohol use, healthy diet and/

or relaxation at work, and (3) targeting employees aged 18-65 years. With respect to the

second inclusion criteria it should be noted that studies investigating interventions that

primarily aim to promote a healthy lifestyle as well as interventions that primarily aim to

prevent musculoskeletal disorders of which the intervention contains one of the lifestyle

components mentioned above were included. The first author (DW) performed an initial

selection based on the titles and abstracts of all papers reporting on effect evaluation.

The abstracts of the effect evaluations were presented to the fourth author (SD), who was

blinded for authors, affiliations, journal and year of publication. In a consensus meeting

between both authors (DW and SD), a final selection of effects evaluations was made. When


47


3

an abstract contained insufficient information or when the authors disagreed, the full paper

was retrieved and read. When disagreement persisted, the second author (LE) was asked to

decide about eligibility.

Step 2: DW checked whether the selected effect evaluations were accompanied by

a peer-reviewed process evaluation. The inclusion criteria for the second stage of the

selection procedure were: (1) published in English between 2000 and July 2012, and (2)

reporting on implementation/process outcomes. DW first contacted the corresponding

authors by e-mail to ask whether the effect evaluation had been accompanied by a paper

on implementation/process outcomes. If the corresponding author did not respond or

could not provide the requested information, DW first checked the full text paper of the

effect evaluation for references to a related process evaluation (in the reference list and/or

text). If nothing was found, DW searched all four databases using the title of the study (if

known), trial registration number and the names of all authors for an additional formative,

implementation or process evaluation. If based on title and abstract it was unclear whether

the paper included information on the implementation process of the intervention, the full

text paper was retrieved and read.

The full papers were retrieved for all eligible effect evaluation papers paired with a

published process evaluation paper. These complete studies were independently assessed

for eligibility by the first three authors (DW, LE and PE) before being included in the review.

During this final round, LE and PE were blinded for authors, affiliations, journal and year of

publication.

All references used in eligible effect evaluations and process evaluations were checked

by the first author (DW) for other relevant publications that might have been missed in the

electronic search (‘snow ball’ procedure).

Methodological quality assessment

In order to answer the first research question, a methodological quality assessment

was performed. The methodological quality of all papers included in the review was

independently assessed by the first three authors (DW, LE and PE) using a checklist (Table 1).

Disagreements between reviewers were discussed and resolved during consensus meetings.

The criteria for the assessment of the included effect evaluations were based on the

methodological guidelines for systematic reviews developed by the Cochrane Back Review

Group [26]. These guidelines were developed for RCT’s studying low back pain. Some

Cochrane criteria were therefore omitted or adapted to fit the studies included in this

R1R2R3R4R5R6R7R8R9


48

Chapter 3

review. Other reviews focusing on worksite interventions have used an adapted version

of this guideline [5, 27]. The final criteria list consisted of three main categories: internal

validity (n=8), descriptive criteria (n=5) and analysis (n=3). Items were scored as positive (+),

unsatisfactory (+/-) or negative (-). If an item was not applicable, this was stated. The quality

of studies was considered to be ‘above average’ if the overall validity score (V) was above

50%, and a score above 75% was considered to represent relatively good quality [26, 28].

The overall validity score (V) was based on the eight internal validity criteria (V1-V8). When

items were not applicable (N/A), they were not included in the percentages.

Given the absence of a standardized assessment form, we defined our own criteria for

the methodological quality assessment of process evaluations on the basis of our proposed

conceptual framework, shortly explained in the introduction and published elsewhere [11].

These criteria are described in detail in table 1. The final criteria list included nine items.

Items were scored as positive (+), unsatisfactory (+/-) or negative (-). If an item was not

applicable, this was stated. The quality of studies was considered to be ‘above average’ if the

overall validity score (T) was above 50%, and a score above 75% was considered to represent

relatively good quality. This validity score (T) was based on the nine internal validity criteria

(T1-T9). Non-applicable items (N/A) were not included in the percentages.

Table 1. Criteria list for the methodological quality assessment of the studies and definitions of the criteria

Effect evaluationsInternal validity / study designV1 Randomization procedure Positive if a random (unpredictable) assignment procedure

sequence of subjects to the study groups was used and if there was a clear description of the procedure and adequate performance of the randomization

V2 Similarity of companies Positive if they controlled for variability in included companiesV3 Similarity of study groups Positive if the study groups were similar at the beginning of the

studyV4 Dropout Positive if the percentage of dropouts during the study period did

not exceed 20% for short-term follow-up (≤ 3 months) or 30% for long-term follow-up (> 3 months) and adequately described

V5 Timing of outcome measurement

Positive if timing of outcome assessment was identical for intervention and control groups and for all important outcomes assessments.

V6 Blinding Positive if the person performing the assessments was blinded to the group assignment

V7 Co-interventions Positive if co-interventions were avoided or comparable.V8 Outcome Positive when data on outcome was selected with standardized

methods of acceptable quality


49


3

Descriptive criteriaD1 Eligibility criteria (in- and

exclusion criteria)Positive if in- and exclusion criteria of participants were specified

D2 Baseline characteristics Positive if an adequate description of the study groups was given for demographic variables: gender, age, type of work, hours a week working, education level, baseline main outcome measures

D3 Company characteristics Positive if an adequate description of the included companies was given (type of industry, organizational characteristics)

D4 Intervention Positive if an adequate description was given of the interventions(s): number of intervention aspects, type of interventions, frequency of sessions, intensity of intervention(s)

D5 Follow-up Positive if a follow-up of 6 months or longer was described.AnalysisA1 Sample size Positive if an adequate sample size calculation was describedA2 Confounders Positive if the analysis controlled for potential confoundersA3 Intention to treat Positive if the intervention and control subjects were analysed

according to the group belonging to their initial assignment, irrespective of non-compliance and co-interventions.

Process evaluationsT1 Model used for evaluation Positive if a theoretical framework for the evaluation was used and

adequately described.T2 Level of evaluation Positive if implementation was evaluated on 2 or more levels (i.e.

macro, meso, micro)T3 Definition of outcome

measure (process components)

Positive if the definition of the outcome measures (process variables and barriers and/or facilitators) were accurately described

T4 Reported process variables a. Positive if four or more process evaluation variables are evaluated (in process evaluation)

b. Positive if barriers or facilitators on 1 or more levels are presented

T5 Data collection Positive if 2 or more techniques for data collection were used (triangulation).

T6 Timing of data collection Positive if measurements of barriers and/or facilitators were performed pre-, during and after implementation.

T7 Quantitative outcome measures

Positive if data on quantitative outcome was selected with methods of acceptable quality and data on multiple process components was measured.

T8 Qualitative data a. Positive if study design for qualitative data (theoretical framework, participant selection, setting, data collection) were adequately described

b. Positive if qualitative data was analyzed by two researchers.

T9 Outcome related to implementation of intervention

Positive if outcomes (barriers and/or facilitators) are related to the quality of implementation

R1R2R3R4R5R6R7R8R9


50

Chapter 3

Data extraction

The first author (DW) extracted the data using a predefined format. For the effect

evaluations, information was extracted about study design, company type, study

population, intervention content, and intervention goal. In addition, although not the main

aim of this review, the proportion of affected primary outcomes was extracted to provide

some information about the effectiveness of the program. For the process evaluations

information was extracted about data collection, methods, timing of measurements, level

of evaluation, type of evaluation, linking effect to implementation outcomes, model for

process evaluation, reported process components and reported barriers and/or facilitators.

During data extraction, the process components reported in the studies were classified

under the eight process components (recruitment, reach, dose delivered, dose received,

fidelity, satisfaction and maintenance, and context) on the basis of the definitions proposed

by Wierenga et al. [11, 14]. If a study used another framework/model with additional

components, these were classified separately. The barriers and/or facilitators found in

the studies were assigned to five categories based on the review by Fleuren et al. (2004):

socio-political context, organization, implementer, program and participants [12, 15]. When

something was unclear, advice was asked from the second (LE) and third author (PE). The

complete data extraction form can be obtained from the corresponding author.

Results

Study selection

The initial computerized search identified 9112 articles (see flow chart in figure 2). Initial

screening of the titles and abstracts produced 704 potentially relevant articles looking at

all lifestyle behaviors. 8408 articles were excluded because they were not (R)CT related to a

WHPP with the aim of changing employee lifestyles. We excluded 397 of the 704 potentially

relevant articles on the grounds that articles were found in multiple databases (Pubmed,

EMBASE, PsycINFO, Cochrane Central Register of Controlled Trials), or in the different

searches based on lifestyle behavior. Interventions not related to any of the five health

behaviors or not targeting employees were also excluded.

A total of 307 full-text articles were retrieved and checked for related process evaluation

articles, resulting in the exclusion of another 277 articles. The main reasons for exclusion

were: no additional published process evaluation for the same study, not targeting

employees, or no lifestyle component. Thirty studies reporting on both the effects and the


51


3

implementation process of the same WHPP were checked for eligibility. After reading the

papers, eight studies were excluded because, contrary to expectations, they did not perform

a process evaluation. No articles were added after snowballing the references in the

selected studies. Finally, twenty-two studies met the selection criteria and were included in

this review (table 2; Appendices 2).

49

because, contrary to expectations, they did not perform a process evaluation. No articles were

added after snowballing the references in the selected studies. Finally, twenty-two studies met the

selection criteria and were included in this review (table 2; Appendices 2).

Figure 2. Flowchart of study selection process

Intervention and study characteristics

The second column in table 2 presents the general study characteristics in order to give an overview

of the type of studies included. Half of the studies were conducted in the USA [29-39], and the other

half in Europe [40-50], mostly in the Netherlands [40, 41, 45, 47-49]. Nineteen (86%) studies used a

randomized controlled trial design [29, 30, 33-36, 38-50] and three (14%) a controlled trial design

[31, 32, 37]. Fourteen (63%) studies reported on interventions targeting physical activity (PA) [29-34,

40-44, 48-50]. Two of these studies focused mainly on preventing musculoskeletal disorders [40, 44],

but the intervention included a PA component and so these studies were included in the review.

Twelve studies (55%) reported on the effect and process evaluation of interventions targeting

healthy nutrition [29, 32-37, 41, 45, 47-49]. Furthermore, five (23%) studies reported interventions

for smoking and tobacco use [36-39, 41], and two (9%) studies focused on relaxation [46, 48]. None

of the twenty-two studies targeted alcohol use.


9112 of records identified through database searching

0 of additional records identified through other sources

9112 records screened

307 of records after duplicates removed and full-text articles

assessed for eligibility

8549 records excluded

22 of studies included in the qualitative synthesis

285 of full-text articles excluded:

- no additional process evaluation - intervention not related to any of the lifestyle behaviours - target population not employees

Figure 2. Flowchart of study selection process

Intervention and study characteristics

The second column in table 2 presents the general study characteristics in order to give an

overview of the type of studies included. Half of the studies were conducted in the USA

[29-39], and the other half in Europe [40-50], mostly in the Netherlands [40, 41, 45, 47-

49]. Nineteen (86%) studies used a randomized controlled trial design [29, 30, 33-36, 38-

50] and three (14%) a controlled trial design [31, 32, 37]. Fourteen (63%) studies reported

on interventions targeting physical activity (PA) [29-34, 40-44, 48-50]. Two of these studies

focused mainly on preventing musculoskeletal disorders [40, 44], but the intervention

included a PA component and so these studies were included in the review. Twelve studies

(55%) reported on the effect and process evaluation of interventions targeting healthy

nutrition [29, 32-37, 41, 45, 47-49]. Furthermore, five (23%) studies reported interventions

for smoking and tobacco use [36-39, 41], and two (9%) studies focused on relaxation [46,

48]. None of the twenty-two studies targeted alcohol use.

R1R2R3R4R5R6R7R8R9


52

Chapter 3

Tabl

e 2.

Cha

ract

eris

tics

of th

e st

udie

s in

clud

ed in

this

revi

ewG

ener

al s

tudy

info

rmati

onPr

oces

s ev

alua

tion

info

rmati

on

Refe

renc

eD

esig

n eff

ect e

valu

ation

: Ran

dom

ized

co

ntro

lled

tria

l (RC

T) o

r Co

ntro

lled

tria

l (CT

)Ty

pe o

f com

pany

St

udy

popu

lati

onIn

terv

enti

on c

onte

ntIn

terv

enti

on g

oal

Prop

orti

on o

f aff

ecte

d pr

imar

y ou

tcom

es

Dat

a co

llecti

on m

etho

d of

pro

cess

ev

alua

tion

(PE)

Tim

ing

of p

roce

ss e

valu

ation

(PE)

Proc

ess

eval

uati

on (P

E) e

valu

ation

leve

ls:

Mac

ro, m

eso,

mic

roTy

pe o

f pro

cess

eva

luati

on (P

E)Eff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Des

crib

ed o

r N

ot d

escr

ibed

Mod

el u

sed

for

eval

uati

onM

easu

red

proc

ess

com

pone

nts:

Defi

nitio

n us

ed in

the

stud

y, -

=not

des

crib

ed

Dri

esse

n et

al.

(201

0,20

11)

[29-

31]

Net

herl

ands

Des

ign

effec

t ev

alua

tion

: RCT

Type

of c

ompa

ny: U

nive

rsity

, rai

lway

tr

ansp

orta

tion,

air

line,

ste

elSt

udy

popu

lati

on: B

lue

and

whi

te c

olla

r In

terv

enti

on c

onte

nt:

Wor

ksite

par

ticip

ator

y er

gono

mic

s pr

ogra

mIn

terv

enti

on g

oal:

Prev

ent l

ow b

ack

pain

and

ne

ck p

ain

Prop

orti

on o

f aff

ecte

d pr

imar

y ou

tcom

es:

2 of

14

Dat

a co

llecti

on m

etho

d PE

:M

ixed

met

hods

Tim

ing

PE: D

urin

g an

d po

st-in

terv

entio

nPE

eva

luati

on le

vels

: Mes

o, M

icro

Type

of P

E: E

valu

ation

of i

nter

venti

ons

and

impl

emen

tatio

n pr

oces

sEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Not

des

crib

ed

Mod

el u

sed

for

eval

uati

on:

Adj

uste

d ve

rsio

n of

Lin

nan

and

Stec

kler

Mea

sure

d pr

oces

s co

mpo

nent

s:-

Recr

uitm

ent:

inte

rven

tion

depa

rtm

ents

, wor

king

gro

ups

form

ed, w

orki

ng

grou

p m

embe

rs fo

r tr

aini

ng, r

espo

nse

rate

bas

elin

e qu

estio

nnai

re-

Reac

h: w

orks

ite v

isits

by

trai

ner,

atten

danc

e ra

tes

impl

emen

ters

, att

enda

nce

rate

s of

impl

emen

ters

in tr

aini

ng-

Dos

e de

liver

ed: p

erce

ived

impl

emen

tatio

n of

the

inte

rven

tion

acco

rdin

g to

im

plem

ente

rs-

Dos

e re

ceiv

ed: p

erce

ived

impl

emen

tatio

n of

the

inte

rven

tion

of th

e em

ploy

ees

- Fi

delit

y: E

xten

t to

whi

ch th

e in

terv

entio

n w

as d

eliv

ered

as

inte

nded

- Sa

tisfa

ction

: of i

mpl

emen

ters

abo

ut im

plem

enta

tion

proc

ess

and

inte

rven

tion,

and

of e

mpl

oyee

s ab

out t

he in

terv

entio

n-

Cont

ext:

Per

ceiv

ed b

arri

ers

and/

or fa

cilit

ator

s to

impl

emen

tatio

n of

in

terv

entio

n-

Mai

nten

ance

: -

Gro

enev

eld

et

al. (

2010

, 201

1)[3

2-34

]

Net

herl

ands

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Con

stru

ction

St

udy

popu

lati

on: B

lue

colla

r at

ris

k fo

r CV

DIn

terv

enti

on c

onte

nt:

Indi

vidu

ally

bas

ed li

fest

yle

inte

rven

tion

by

mea

ns o

f moti

vatio

nal i

nter

view

ing

Inte

rven

tion

goa

l: Ch

ange

phys

ical

acti

vity

, die

t and

sm

okin

g be

havi

orPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 8o

f 15a

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mes

o, M

icro

Type

of P

E: E

valu

ation

of i

nter

venti

ons

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: D

escr

ibed

for

body

wei

ght

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- D

ose

deliv

ered

: Num

ber

of s

essi

ons

perf

orm

ed b

y im

plem

ente

r an

d nu

mbe

r of

item

s di

scus

sed

duri

ng s

essi

on-

Dos

e re

ceiv

ed: N

umbe

r of

ses

sion

s att

ende

d by

par

ticip

ants

- Fi

delit

y: A

dher

ence

to in

terv

entio

n pr

otoc

ol b

y im

plem

ente

rs a

nd th

e qu

ality

of t

he in

terv

entio

n de

liver

y-

Satis

facti

on: P

artic

ipan

ts o

pini

on o

n th

e co

mpe

tenc

e an

d sk

ills

of th

e im

plem

ente

r. Im

plem

ente

rs o

pini

on o

n th

eir

own

com

pete

nce

and

skill

s on

de

liver

y of

inte

rven

tion

and

over

all d

ifficu

lty o

f del

iver

ing

inte

rven

tion

- Re

crui

tmen

t, R

each

, Mai

nten

ance

, Con

text

: -


53


3

Fren

ch e

t al.

(201

0) [3

5, 3

6]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Bus

gar

ages

Stud

y po

pula

tion

: Blu

e co

llar

Inte

rven

tion

con

tent

: Env

ironm

enta

l wor

ksite

ob

esity

pre

venti

on in

terv

entio

nIn

terv

enti

on g

oal:

Impr

ove

heal

thfu

l foo

d ch

oice

s an

phy

sica

l acti

vity

leve

lsPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 4

of 2

0

Dat

a co

llecti

on m

etho

d PE

: Qua

ntita

tive

Tim

ing

PE: P

ost-

inte

rven

tion

PE e

valu

ation

leve

ls: M

icro

Ty

pe o

f PE:

Eva

luati

on o

f int

erve

ntion

sEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Not

des

crib

ed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- D

ose

deliv

ered

: Am

ount

of i

nter

venti

ons

mad

e av

aila

ble

duri

ng th

e in

terv

entio

n pe

riod

- D

ose

rece

ived

: Par

ticip

ation

rate

s of

em

ploy

ees

in in

terv

entio

n-

Recr

uitm

ent,

Rea

ch, F

idel

ity, S

atisf

actio

n, M

aint

enan

ce, C

onte

xt: -

Dis

hman

& W

ilson

et

al.

(200

9, 2

010)

[3

7, 3

8]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Ret

ail

Stud

y po

pula

tion

: Whi

te c

olla

rIn

terv

enti

on c

onte

nt:

Soci

al-e

colo

gic

inte

rven

tion

by p

erso

nal a

nd

team

goa

l-setti

ngIn

terv

enti

on g

oal:

Incr

ease

leis

ure-

time

phys

ical

acti

vity

Pr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 3

of 3

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Dur

ing

and

post

-inte

rven

tion

PE e

valu

ation

leve

ls: M

eso,

Mic

roTy

pe o

f PE:

Eva

luati

on o

f int

erve

ntion

s an

d im

plem

enta

tion

proc

ess

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: D

escr

ibed

for p

hysi

cal a

ctivi

ty le

vels

Mod

el u

sed

for

eval

uati

on:

Dur

lak

& D

upre

Mea

sure

d pr

oces

s co

mpo

nent

s:-

Dos

e de

liver

ed: A

mou

nt o

f int

erve

ntion

del

iver

ed-

Dos

e re

ceiv

ed: A

mou

nt o

f int

erve

ntion

rece

ived

by

parti

cipa

nts

- Fi

delit

y: D

egre

e to

whi

ch th

e in

terv

entio

n w

as im

plem

ente

d as

pla

nned

- Co

ntex

t: Im

plem

enta

tion

barr

iers

- Re

crui

tmen

t, R

each

, Sati

sfac

tion,

Mai

nten

ance

: -

Yap

et a

l. (2

009,

20

10) [

39-4

1]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: C

TTy

pe o

f com

pany

: Man

ufac

turi

ng p

lant

Stud

y po

pula

tion

: Blu

e co

llar

Inte

rven

tion

con

tent

:Ta

ilore

d e-

mai

l int

erve

ntion

Inte

rven

tion

goa

l: In

crea

se in

tenti

onal

ph

ysic

al a

ctivi

tyPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 2

of 2

Dat

a co

llecti

on m

etho

d PE

: Qua

litati

ve

Tim

ing

PE: P

ost-

inte

rven

tion

PE e

valu

ation

leve

ls:

Mic

ro

Type

of P

E: E

valu

ation

of i

nter

venti

ons

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: N

ot d

escr

ibed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- D

ose

rece

ived

: Use

of i

nter

venti

on-

Satis

facti

on: S

atisf

actio

n w

ith th

e in

terv

entio

n co

mpo

nent

s-

Recr

uitm

ent,

Rea

ch, D

ose

deliv

ered

, Fid

elity

, Mai

nten

ance

, Con

text

: -

Gils

on e

t al.

(200

7,

2008

) [42

, 43]

Uni

tes

King

dom

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Uni

vers

itySt

udy

popu

lati

on: W

hite

col

lar

Inte

rven

tion

con

tent

:Ro

ute

and

task

-bas

ed w

alki

ng In

terv

enti

on

goal

: Im

prov

e w

ork

day

step

cou

nts

and

heal

th

stat

usPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 1o

f 4

Dat

a co

llecti

on m

etho

d PE

:Q

ualit

ative

Ti

min

g PE

: Pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mic

roTy

pe o

f PE:

Eva

luati

on o

f int

erve

ntion

s an

d im

plem

enta

tion

proc

ess

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: N

ot d

escr

ibed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- Re

crui

tmen

t: S

trat

egie

s to

app

roac

h an

d att

ract

em

ploy

ees

- Sa

tisfa

ction

: Exp

erie

nces

dur

ing

inte

rven

tion

- Co

ntex

t: B

enefi

ts/p

ositi

ves

and

prob

lem

s/ba

rrie

rs a

ssoc

iate

d w

ith

parti

cipa

tion

in in

terv

entio

n-

Reac

h, D

ose

deliv

ered

, Dos

e re

ceiv

ed, F

idel

ity, M

aint

enan

ce: -

R1R2R3R4R5R6R7R8R9


54

Chapter 3

Goe

tzel

, DeJ

oy,

Wils

on e

t al.

(200

7, 2

009,

201

0,

2011

) [44

-48]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: C

TTy

pe o

f com

pany

: Man

ufac

turi

ng

Stud

y po

pula

tion

: Blu

e an

d w

hite

col

lar

Inte

rven

tion

con

tent

: Env

ironm

enta

l wei

ght

man

agem

ent i

nter

venti

on

Inte

rven

tion

goa

l: Im

prov

e ph

ysic

al a

ctivi

ty

and

heal

thy

eatin

gPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 6

of 8

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Dur

ing

and

post

-inte

rven

tion

PE e

valu

ation

leve

ls: M

acro

, Mes

o, M

icro

Type

of P

E: E

valu

ation

of i

nter

venti

ons

and

impl

emen

tatio

n pr

oces

sEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Not

des

crib

ed

Mod

el u

sed

for

eval

uati

on:

Inte

grati

ve m

odel

Mea

sure

d pr

oces

s co

mpo

nent

s:-

Cont

ext:

Man

agem

ent s

uppo

rt, j

ob/t

ask

fact

ors

(phy

sica

l and

psy

chol

ogic

al

dem

ands

of s

peci

fic jo

bs),

envi

ronm

enta

l fac

tors

(phy

sica

l wor

k en

viro

nmen

t and

the

soci

al-o

rgan

izati

onal

env

ironm

ent

- Re

crui

tmen

t, R

each

, Dos

e de

liver

ed, D

ose

rece

ived

, Fid

elity

, Sati

sfac

tion,

M

aint

enan

ce: -

Lem

on, E

stab

rook

et

al.

(201

0, 2

011)

[4

9, 5

0]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Hos

pita

lSt

udy

popu

lati

on: W

hite

col

lar

Inte

rven

tion

:Ec

olog

ical

and

env

ironm

enta

l mul

tilev

el

inte

rven

tion

Inte

rven

tion

goa

l: Pr

even

t wei

ght g

ain

by

targ

eting

hea

lthy

eatin

g an

d ph

ysic

al a

ctivi

tyPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 0

of 1

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Pre

-, du

ring

, and

pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mac

ro, M

eso,

Mic

roTy

pe o

f PE:

Eva

luati

on o

f int

erve

ntion

s an

d im

plem

enta

tion

proc

ess

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: D

escr

ibed

for

Body

Mas

s In

dex

Mod

el u

sed

for

eval

uati

on:

RE-A

IMM

easu

red

proc

ess

com

pone

nts:

- Re

crui

tmen

t: A

mou

nt o

f rec

ruite

d se

tting

s fo

r pa

rtici

patio

n an

d nu

mbe

r th

at a

gree

d to

take

par

t, re

pres

enta

tives

of p

artic

ipati

ng s

etting

s (A

dopti

on)

- Re

ach:

Par

ticip

ation

rate

s, re

pres

enta

tiven

ess

of p

artic

ipan

ts,

char

acte

risti

cs o

f par

ticip

ants

and

non

-par

ticip

ants

- D

ose

deliv

ered

: Im

plem

ente

d in

terv

entio

n co

mpo

nent

s-

Dos

e re

ceiv

ed: U

se o

f int

erve

ntion

com

pone

nts

- Fi

delit

y: E

xten

t to

whi

ch th

e pr

ogra

m w

as d

eliv

ered

as

inte

nded

at p

rogr

am

and

indi

vidu

al le

vel (

Impl

emen

tatio

n)-

Mai

nten

ance

: Ext

ent t

o w

hich

the

beha

vior

or

polic

y of

inte

rest

mai

ntai

ned

over

the

long

term

at i

ndiv

idua

l and

insti

tutio

nal l

evel

- Co

ntex

t: M

anag

emen

t sup

port

, job

/tas

k fa

ctor

s (p

hysi

cal a

nd p

sych

olog

ical

de

man

ds o

f spe

cific

jobs

), en

viro

nmen

tal f

acto

rs (p

hysi

cal w

ork

envi

ronm

ent a

nd th

e so

cial

-org

aniz

ation

al e

nviro

nmen

t-

Effec

tiven

ess:

Beh

avio

ral o

utco

mes

and

oth

er o

utco

mes

incl

udin

g im

pact

on

qua

lity

of li

fe-

Satis

facti

on: -

And

erse

n et

al.

(201

1) [5

1-53

]

Den

mar

k

Des

ign

effec

t ev

alua

tion

: RCT

Type

of c

ompa

ny: O

ffice

Stud

y po

pula

tion

: Whi

te c

olla

rIn

terv

enti

on c

onte

nt:

Prog

ress

ive

resi

stan

ce tr

aini

ng (e

xerc

ise

prog

ram

)In

terv

enti

on g

oal:

Relie

ve n

eck/

shou

lder

pai

nPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 4

of 6

Dat

a co

llecti

on m

etho

d PE

:Q

uanti

tativ

eTi

min

g PE

: Pre

-inte

rven

tion

PE e

valu

ation

leve

ls: M

icro

Type

of P

E: E

valu

ation

of i

nter

venti

ons

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: N

ot d

escr

ibed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- Co

ntex

t: P

rogn

ostic

fact

ors

for

adhe

renc

e to

inte

rven

tion

- Re

crui

tmen

t, R

each

, Dos

e de

liver

ed, D

ose

rece

ived

, Fid

elity

, Sati

sfac

tion,

M

aint

enan

ce: -


55


3

Hau

kka,

Peh

kone

n et

al.

(200

9, 2

010)

[5

4, 5

5]

Finl

and

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Mun

icip

al k

itche

nsSt

udy

popu

lati

on: B

lue

colla

r In

terv

enti

on c

onte

nt:

Parti

cipa

tory

erg

onom

ics

inte

rven

tion

Inte

rven

tion

goa

l: D

ecre

ase

phys

ical

and

m

enta

l wor

kloa

dPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 0

of 8

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Pre

-, du

ring

, and

pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mes

o, M

icro

Type

of P

E: E

valu

ation

of i

nter

venti

ons

and

impl

emen

tatio

n pr

oces

sEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Not

des

crib

ed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- D

ose

deliv

ered

: Tim

e an

d vi

sits

mad

e to

wor

ksite

- D

ose

rece

ived

: Par

ticip

ation

rate

s -

Satis

facti

on: G

ener

al o

pini

ons

rega

rdin

g pr

oces

s, e

xper

ienc

es o

f the

in

terv

entio

n,

- Co

ntex

t: B

enefi

ts o

r di

fficu

lties

exp

erie

nces

of t

he p

roje

ct, f

acili

tatin

g fa

ctor

s fo

r im

plem

enta

tion,

sup

port

and

tim

e ne

eded

for

deve

lopm

ent,

or

gani

zatio

n of

wor

k ta

sks

duri

ng in

terv

entio

n-

Recr

uitm

ent,

Rea

ch, F

idel

ity, M

aint

enan

ce:-

Sore

nsen

, Hun

t (2

005,

200

7) [5

6,

57]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Man

ufac

turi

ngSt

udy

popu

lati

on: B

lue

colla

rIn

terv

enti

on c

onte

nt: W

orks

ite in

terv

entio

n ta

rgeti

ng fr

uit a

nd v

eget

able

con

sum

ption

, re

d m

eat c

onsu

mpti

on, m

ultiv

itam

in u

se a

nd

phys

ical

acti

vity

Inte

rven

tion

goa

l: Ca

ncer

pre

venti

onPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 2

of 4

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mes

o, M

icro

Type

of P

E:Ev

alua

tion

of in

terv

entio

ns a

nd

impl

emen

tatio

n pr

oces

sEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Not

des

crib

ed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- Re

ach:

Pro

gram

aw

aren

ess

- D

ose

deliv

ered

: Del

iver

ed in

terv

entio

ns-

Dos

e re

ceiv

ed: P

rogr

am p

artic

ipati

on

- Co

ntex

t: M

anag

emen

t con

tact

s re

gard

ing

orga

niza

tiona

l cha

nges

, fac

tors

th

at in

fluen

ces

impl

emen

tatio

n-

Recr

uitm

ent,

Fid

elity

, Sati

sfac

tion,

Mai

nten

ance

: -

Bere

sfor

d et

al.

(200

0, 2

001,

201

0)

[58-

60]

Uni

ted

Stat

es

Des

ign

effec

t ev

alua

tion

: RCT

Type

of c

ompa

ny: H

ospi

tals

, edu

catio

nal,

gove

rnm

enta

l, pr

ofes

sion

al a

genc

ies,

co

nstr

uctio

n, m

anuf

actu

ring

, fina

ncia

l, re

tail,

w

hole

sale

, ser

vice

Stud

y po

pula

tion

: Blu

e an

d w

hite

col

lar

Inte

rven

tion

con

tent

: Env

ironm

enta

l and

in

divi

dual

str

ateg

ies

Inte

rven

tion

goa

l: In

crea

se fr

uit a

nd v

eget

able

in

take

Pr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 2

of 2

b

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mes

o, M

icro

Type

of P

E: E

valu

ation

of i

nter

venti

ons

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: D

escr

ibed

for

frui

t and

veg

etab

le

inta

ke

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- Re

ach:

See

ing

or re

adin

g th

e in

terv

entio

n m

ater

ials

(use

of i

nter

venti

on)

- D

ose

deliv

ered

: Doc

umen

tatio

n of

inte

rven

tion

activ

ities

by

impl

emen

ters

- D

ose

rece

ived

: Ave

rage

inte

rven

tion

expo

sure

per

em

ploy

ee-

Cont

ext:

Wor

ksite

cha

ract

eris

tics

- Re

crui

tmen

t, F

idel

ity, S

atisf

actio

n, M

aint

enan

ce: -

Sore

nsen

, Hun

t et

al. (

2007

, 201

0)

[61,

62]

Uni

ted

Stat

es

Des

ign

effec

t ev

alua

tion

: RCT

Type

of c

ompa

ny: C

onst

ructi

on

Stud

y po

pula

tion

: Blu

e co

llar

Inte

rven

tion

con

tent

: Tai

lore

d te

leph

one-

deliv

ered

and

mai

led

inte

rven

tion

Inte

rven

tion

goa

l: Pr

omot

e sm

okin

g ce

ssati

on

and

incr

ease

frui

t and

veg

etab

le c

onsu

mpti

onPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 3

of 3

Dat

a co

llecti

on m

etho

d PE

:Q

ualit

ative

Ti

min

g PE

: Pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mic

roTy

pe o

f PE:

Eval

uatio

n of

inte

rven

tions

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: N

ot d

escr

ibed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- D

ose

deliv

ered

: Num

ber

of in

terv

entio

ns d

eliv

ered

- D

ose

rece

ived

: Rec

eivi

ng in

terv

entio

n an

d m

ater

ials

- Fi

delit

y: E

xten

t to

whi

ch th

e in

terv

entio

n w

as im

plem

ente

d-

Satis

facti

on: P

artic

ipan

t sati

sfac

tion

with

inte

rven

tion

- Re

crui

tmen

t, R

each

, Mai

nten

ance

, Con

text

: -

R1R2R3R4R5R6R7R8R9


56

Chapter 3

Stee

nhui

s et

al.

(200

4) [6

3, 6

4]

Net

herl

ands

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Gov

ernm

enta

l and

larg

e co

mpa

nies

Stud

y po

pula

tion

: Whi

te c

olla

r In

terv

enti

on:

Envi

ronm

enta

l int

erve

ntion

s in

clud

ing

labe

ling

of h

ealth

y fo

ods,

food

sup

ply

prog

ram

and

ed

ucati

onal

pro

gram

abo

ut h

ealth

y fo

od

Inte

rven

tion

goa

l: In

crea

se a

vaila

bilit

y an

d kn

owle

dge

of h

ealth

y fo

ods

Prop

orti

on o

f aff

ecte

d pr

imar

y ou

tcom

es:

1 of

3

Dat

a co

llecti

on m

etho

d PE

:Q

ualit

ative

Tim

ing

PE: P

ost-

inte

rven

tion

PE e

valu

ation

leve

ls: M

eso

Type

of P

E:Ev

alua

tion

of in

terv

entio

ns a

nd

impl

emen

tatio

n pr

oces

sEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Not

des

crib

ed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- Sa

tisfa

ction

: man

ager

s op

inio

ns o

n in

terv

entio

n -

Cont

ext:

diffi

culti

es w

ith im

plem

enta

tion,

per

ceiv

ed b

enefi

ts o

f int

erve

ntion

- Re

crui

tmen

t, R

each

, Dos

e de

liver

ed, D

ose

rece

ived

, Fid

elity

, Mai

nten

ance

: -

Sore

nsen

, Q

uinti

liani

et a

l. (2

010)

[65,

66]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: C

TTy

pe o

f com

pany

: Tru

ckin

g te

rmin

als

Stud

y po

pula

tion

: Blu

e co

llar

Inte

rven

tion

: Tel

epho

ne a

nd p

rint

-del

iver

ed

inte

rven

tion

Inte

rven

tion

goa

l: Pr

omot

e to

bacc

o us

e ce

ssati

on a

nd im

prov

e w

eigh

t man

agem

ent b

y he

alth

y nu

triti

onPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 1

of 2

Dat

a co

llecti

on m

etho

d PE

:Q

uanti

tativ

eTi

min

g PE

: Pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mes

o, M

icro

Type

of P

E: E

valu

ation

of i

nter

venti

ons

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: D

escr

ibed

for

smok

ing

cess

ation

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- D

ose

deliv

ered

: Num

ber

of d

eliv

ered

inte

rven

tions

- D

ose

rece

ived

: Per

cepti

on o

f rec

eive

d in

terv

entio

n co

mpo

nent

s an

d m

ater

ials

(Eng

agem

ent i

n in

terv

entio

n co

mpo

nent

s)-

Satis

facti

on: H

elpf

ulne

ss o

f int

erve

ntion

- Re

crui

tmen

t, R

each

, Fid

elity

, Mai

nten

ance

, Con

text

: -

Stod

dard

, Hun

t et

al. (

2003

, 200

5)

[67,

68]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Ret

ail

Stud

y po

pula

tion

: Whi

te c

olla

r (t

eens

)In

terv

enti

on c

onte

nt: B

ehav

iora

l tob

acco

co

ntro

l int

erve

ntion

Inte

rven

tion

goa

l: D

ecre

ase

smok

ing

prev

alen

cePr

opor

tion

of a

ffec

ted

mai

n ou

tcom

es: 0

of 2

Dat

a co

llecti

on m

etho

d PE

:Q

uanti

tativ

eTi

min

g PE

: Dur

ing

and

post

-inte

rven

tion

PE e

valu

ation

leve

ls: M

eso,

Mic

roTy

pe o

f PE:

Eval

uatio

n of

inte

rven

tions

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: N

ot d

escr

ibed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- D

ose

deliv

ered

: Am

ount

of i

nter

venti

on d

eliv

ered

- D

ose

rece

ived

: am

ount

of i

nter

venti

on re

ceiv

ed, p

artic

ipati

on ra

tes,

m

otiva

tion

for

parti

cipa

tion

in in

terv

entio

n, p

rogr

am a

war

enes

s-

Recr

uitm

ent,

Rea

ch, F

idel

ity, S

atisf

actio

n, M

aint

enan

ce, C

onte

xt: -

Volp

p, K

im e

t al

. (20

09, 2

011)

[6

9, 7

0]

Uni

ted

Stat

es

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Mul

tinati

onal

St

udy

popu

lati

on: N

ot d

escr

ibed

Inte

rven

tion

con

tent

: Offe

ring

fina

ncia

l in

centi

ves

for

smok

ing

cess

ation

Inte

rven

tion

goa

l: Im

prov

e sm

okin

g ce

ssati

on

rate

sPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 1

of 1

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Pos

t-in

terv

entio

n PE

eva

luati

on le

vels

: Mic

roTy

pe o

f PE:

Eva

luati

on o

f int

erve

ntion

sEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Des

crib

ed fo

r sm

okin

g ce

ssati

on

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- Re

ach:

Pro

gram

aw

aren

ess

- D

ose

deliv

ered

: Am

ount

of i

nter

venti

on d

eliv

ered

- D

ose

rece

ived

: am

ount

of i

nter

venti

on re

ceiv

ed, p

artic

ipati

on ra

tes,

m

otiva

tion

for

parti

cipa

tion

in in

terv

entio

n-

Recr

uitm

ent,

Fid

elity

, Sati

sfac

tion,

Mai

nten

ance

, Con

text

: -


57


3

Has

son

et a

l. (2

005,

201

0) [7

1,

72]

Swed

en

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Inf

orm

ation

tech

nolo

gy,

med

iaSt

udy

popu

lati

on: W

hite

col

lar

Inte

rven

tion

con

tent

: Web

-bas

ed h

ealth

pr

omoti

on a

nd s

tres

s m

anag

emen

t tra

inin

gIn

terv

enti

on g

oal:

Dec

reas

e un

wan

ted

stre

ss

and

prom

ote

heal

th a

nd re

cove

ryPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 5

of 5

Dat

a co

llecti

on m

etho

d PE

:Q

ualit

ative

Tim

ing

PE: D

urin

gPE

eva

luati

on le

vels

: Mic

roTy

pe o

f PE:

Eva

luati

on o

f int

erve

ntion

sEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Not

des

crib

ed

Mod

el u

sed

for

eval

uati

on:

Not

des

crib

edM

easu

red

proc

ess

com

pone

nts:

- D

ose

rece

ived

: Fre

quen

cy o

f rep

lyin

g to

the

scre

enin

g to

ol-

Cont

ext:

Fac

tors

that

det

erm

ine

use

of p

rogr

am-

Recr

uitm

ent,

Rea

ch, D

ose

deliv

ered

, Fid

elity

, Sati

sfac

tion,

Mai

nten

ance

: -

Verm

eer

et a

l. (2

011)

[73,

74]

Net

herl

ands

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Wor

ksite

caf

eter

ias

of

hosp

itals

, uni

vers

ities

, pol

ice

depa

rtm

ents

, co

mpa

nies

Stud

y po

pula

tion

: Whi

te c

olla

rIn

terv

enti

on c

onte

nt:

Envi

ronm

enta

l int

erve

ntion

intr

oduc

ing

a sm

all

hot m

eal i

n ad

ditio

n to

the

exis

ting

size

and

a

prop

ortio

nal p

rici

ng s

trat

egy

in c

afet

eria

sIn

terv

enti

on g

oal:

Stim

ulat

e w

orke

rs to

re

plac

e th

eir

larg

er m

eal w

ith a

sm

alle

r m

eal

Prop

orti

on o

f aff

ecte

d pr

imar

y ou

tcom

es:

0 of

2

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Pre

-, du

ring

and

pos

t-in

terv

entio

nPE

eva

luati

on le

vels

: Mes

o, M

icro

Type

of P

E:Ev

alua

tion

of in

terv

entio

nsEff

ect o

f im

plem

enta

tion

on

outc

ome

mea

sure

: Not

des

crib

ed

Mod

el u

sed

for

eval

uati

on:

Bara

now

ski &

Sta

bles

and

Rog

ers

Mea

sure

d pr

oces

s co

mpo

nent

s:-

Recr

uitm

ent:

Att

racti

ng a

genc

ies,

impl

emen

ters

, or

pote

ntial

par

ticip

ants

fo

r co

rres

pond

ing

part

s of

the

prog

ram

- D

ose

rece

ived

: Ext

ent t

o w

hich

par

ticip

ants

vie

w o

r re

ad th

e m

ater

ials

that

re

ach

them

- Fi

delit

y: E

xten

t to

whi

ch th

e pr

ogra

m is

impl

emen

ted

as d

esig

ned

- M

aint

enan

ce: K

eepi

ng p

artic

ipan

ts in

volv

ed in

the

prog

ram

mati

c an

d da

ta c

olle

ction

, and

ext

ent t

o w

hich

par

ticip

ants

con

tinue

to d

o an

y of

the

activ

ities

- Co

ntex

t: A

spec

ts o

f the

env

ironm

ent o

f an

inte

rven

tion

- Co

ntam

inati

on: e

xten

t to

whi

ch p

artic

ipan

ts re

ceiv

ed in

terv

entio

ns fr

om

outs

ide

the

prog

ram

and

the

exte

nt to

whi

ch th

e co

ntro

l gro

up re

ceiv

es th

e tr

eatm

ent

- Re

sour

ces:

Mat

eria

ls o

r ch

arac

teri

stics

of a

genc

ies,

impl

emen

ters

, or

parti

cipa

nts

nece

ssar

y to

att

ain

proj

ect g

oals

- Re

ach,

Dos

e de

liver

ed, S

atisf

actio

n:-

Stri

jk e

t al.

(201

1,

2012

) [75

-77]

Net

herl

ands

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Aca

dem

ic h

ospi

tal

Stud

y po

pula

tion

: Whi

te c

olla

rIn

terv

enti

on c

onte

nt: V

italit

y in

terv

entio

n co

nsis

ting

of w

eekl

y yo

ga a

nd w

orko

ut

sess

ion,

wee

kly

unsu

perv

ised

aer

obic

exe

rcis

e,

free

frui

t dur

ing

sess

ions

and

thre

e vi

sits

of

pers

onal

vita

lity

coac

hIn

terv

enti

on g

oal:

Impr

ove

lifes

tyle

beh

avio

rs

Prop

orti

on o

f aff

ecte

d pr

imar

y ou

tcom

es:

3 of

10

Dat

a co

llecti

on m

etho

d PE

:Q

uanti

tativ

eTi

min

g PE

: Dur

ing

and

post

-inte

rven

tion

PE e

valu

ation

leve

ls: M

eso,

Mic

roTy

pe o

f PE:

Eval

uatio

n of

inte

rven

tions

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: N

ot d

escr

ibed

Mod

el u

sed

for

eval

uati

on:

Stec

kler

and

Lin

nan

Mea

sure

d pr

oces

s co

mpo

nent

s:-

Reac

h: p

ropo

rtion

of w

orke

rs p

artic

ipati

ng in

inte

rven

tion

- D

ose

deliv

ered

: num

ber

of in

terv

entio

n co

mpo

nent

s de

liver

ed b

y im

plem

ente

rs-

Dos

e re

ceiv

ed: E

xten

t to

whi

ch th

e w

orke

rs w

ere

enga

ged

in th

e in

terv

entio

n (a

tten

danc

e ra

tes)

- Fi

delit

y: E

xten

t to

whi

ch th

e in

terv

entio

n w

as im

plem

ente

d as

pla

nned

- Sa

tisfa

ction

: Wor

kers

atti

tude

tow

ards

the

inte

rven

tion

- Co

ntex

t: O

rgan

izati

onal

and

env

ironm

enta

l fac

tors

con

cern

ing

the

inte

rven

tion

- Re

crui

tmen

t, M

aint

enan

ce:-

R1R2R3R4R5R6R7R8R9


58

Chapter 3

Verw

eij e

t al.

(201

1, 2

012)

[7

8-80

]

Net

herl

ands

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Uni

vers

ity, b

ank,

nur

sing

ho

me,

spi

ce fa

ctor

y, p

acka

ging

com

pany

, m

unic

ipal

ity, c

onsu

mer

goo

ds c

ompa

nySt

udy

popu

lati

on: B

lue

and

whi

te c

olla

r In

terv

enti

on c

onte

nt: O

ccup

ation

al h

ealth

gu

idel

ine

Inte

rven

tion

goa

l: pr

even

ting

wei

ght g

ain

by

incr

easi

ng P

A, d

ecre

asin

g se

dent

ary

beha

vior

, in

crea

sing

frui

t con

sum

ption

or

redu

cing

en

ergy

inta

ke d

eriv

ed fr

om s

nack

sPr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 2

of 6

Dat

a co

llecti

on m

etho

d PE

: Mix

ed m

etho

dsTi

min

g PE

: Dur

ing

and

post

-inte

rven

tion

PE e

valu

ation

leve

ls: M

eso,

Mic

roTy

pe o

f PE:

Eval

uatio

n of

inte

rven

tions

and

im

plem

enta

tion

proc

ess

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: D

escr

ibed

for

body

wei

ght a

nd

wai

st c

ircum

fere

nce

Mod

el u

sed

for

eval

uati

on:

Stec

kler

and

Lin

nan

Mea

sure

d pr

oces

s co

mpo

nent

s:-

Recr

uitm

ent:

Sou

rces

and

pro

cedu

res

used

to a

ppro

ach

and

attra

ct

pote

ntial

par

ticip

ants

, the

num

ber

of ra

ndom

ized

OPs

, and

the

num

ber

of

empl

oyee

s th

at fi

lled

out t

he b

asel

ine

ques

tionn

aire

- Re

ach:

Num

ber

of e

mpl

oyee

s w

ho a

tten

ded

the

coun

selin

g se

ssio

ns,

reas

on fo

r m

isse

d co

unse

ling

sess

ions

and

the

perc

enta

ge o

f dro

p-ou

ts

incl

udin

g re

ason

- D

ose

deliv

ered

: The

num

ber

of in

terv

entio

n m

ater

ials

or

com

pone

nts

actu

ally

del

iver

ed b

y O

Ps, a

nd th

e du

ratio

n an

d fo

rm o

f the

cou

nsel

ing

sess

ions

- D

ose

rece

ived

: The

ext

ent t

o w

hich

par

ticip

ants

use

mat

eria

ls, r

esou

rces

, or

tech

niqu

es re

com

men

ded

by p

rogr

am-

Fide

lity:

The

ext

ent t

o w

hich

the

inte

rven

tion

was

del

iver

ed a

s pl

anne

d: if

O

Ps a

dher

ed to

the

guid

elin

e an

d ad

equa

tely

per

form

ed b

ehav

ior

chan

ge

coun

selin

g-

Satis

facti

on: P

artic

ipan

ts a

ttitu

des

tow

ard

the

cont

ent,

use

and

lim

itatio

ns

of th

e gu

idel

ine

(OP)

, or

tow

ard

the

inte

rven

tion,

mat

eria

ls a

nd O

P (e

mpl

oyee

)-

Cont

ext:

Org

aniz

ation

al c

hara

cter

istic

s th

at a

ffect

inte

rven

tion

impl

emen

tatio

n, in

clud

ing

phys

ical

, soc

ial,

politi

cal,

and

econ

omic

feat

ures

- M

aint

enan

ce: -

Jorg

ense

n et

al

.(20

11, 2

012)

[8

1-83

]

Den

mar

k

Des

ign

effec

t eva

luati

on: R

CTTy

pe o

f com

pany

: Hos

pita

ls, c

lean

ing

com

pani

es, l

arge

bus

ines

s w

ith in

-hou

se

clea

ning

ser

vice

sSt

udy

popu

lati

on: B

lue

colla

r In

terv

enti

on c

onte

nt: P

hysi

cal c

oord

inati

on

trai

ning

Inte

rven

tion

goa

l: Im

prov

e ph

ysic

al

coor

dina

tion

trai

ning

or

cogn

itive

beh

avio

ral

reso

urce

s Pr

opor

tion

of a

ffec

ted

prim

ary

outc

omes

: 3

of 6

Dat

a co

llecti

on m

etho

d PE

:Q

uanti

tativ

eTi

min

g PE

: Dur

ing

inte

rven

tion

PE e

valu

ation

leve

ls: M

eso

Type

of P

E: E

valu

ation

of i

nter

venti

ons

Effec

t of i

mpl

emen

tati

on o

n ou

tcom

e m

easu

re: N

ot d

escr

ibed

Mod

el u

sed

for

eval

uati

on:

Ada

pted

ver

sion

Ste

ckle

r &

Lin

nan

Mea

sure

d pr

oces

s co

mpo

nent

s:-

Recr

uitm

ent:

Pro

cedu

res

used

to a

ppro

ach

and

attra

ct p

artic

ipan

ts-

Dos

e de

liver

ed: i

nter

venti

on d

eliv

ered

- D

ose

rece

ived

: adh

eren

ce to

inte

rven

tion

(att

enda

nce

rate

s)-

Fide

lity:

Qua

lity

of in

terv

entio

n de

liver

y-

Cont

ext:

Una

ntici

pate

d ev

ents

at t

he w

ork

plac

e-

Reac

h, S

atisf

actio

n, M

aint

enan

ce: -

RCT:

Ran

dom

ized

con

trol

led

tria

l; CT

: Con

trol

led

tria

l; PE

: pro

cess

eva

luati

on; M

acro

leve

l: co

mpa

ny/m

anag

emen

t; M

eso

leve

l: Im

plem

ente

r; M

icro

: em

ploy

eea 3

out

com

es w

ere

sign

ifica

nt a

t sho

rt a

nd lo

ng te

rm, 3

out

com

es o

nly

at s

hort

term

, 2 o

utco

mes

onl

y at

long

term

b bo

th o

utco

mes

wer

e si

gnifi

cant

ly im

prov

ed a

t sho

rt te

rm, h

owev

er th

ey w

ere

not s

tatis

tical

ly s

igni

fican

t at l

ong

term

but

stil

l im

prov

ed in

favo

r of i

nter

venti

on

grou

p


59


3


Table 3 shows the methodological quality scores of the included studies. Initial disagreement

between the reviewers about 50 of the 448 effect evaluation criteria (11%) and 46 of the

207 process evaluation criteria (22%) were mainly attributable to the interpretation of the

items. The differences concentrated primarily on drop-out (V4), company characteristics

(D3), intention to treat (A3), data collection for process evaluation (T5), and the timing of

data collection for process evaluation (T6). Full agreement was reached after two discussion

sessions, thereby completing the scoring process.

The methodological quality scores for the effect evaluations of the studies ranged from

37.5% to 100%. Fifteen (68%) effect evaluations were considered to be ‘above average’

(>50%) [29-33, 37, 39-41, 43, 44, 46, 48, 48-50]. Only six of these effect evaluations (27%)

were relatively good (quality score >75%) [31, 40, 41, 43, 48, 49]. The quality scores for

the process evaluations ranged from 12.5% to 100%. Eight process evaluations (36%) were

relatively good (quality score >75%) [51-58].

Only three (14%) studies scored relatively good (quality score >75%) on both the process

as well as the effect evaluation with respect to their methodological quality [51, 57, 58]. In

four (18.2%) studies the methodological quality of both the effect as well as the process

evaluation was poor (quality score 50% or less) [36, 38, 42, 45]. No relation was observed

between the quality of effect evaluations and process evaluations.

R1R2R3R4R5R6R7R8R9


60

Chapter 3Ta

ble

3. O

vera

ll sc

ores

of t

he m

etho

dolo

gica

l qua

lity

of th

e in

clud

ed s

tudi

es

Firs

t aut

hor

(Yea

r)M

etho

dolo

gica

l qua

lity

asse

ssm

ent c

rite

rion

– e

ffec

t ev

alua

tion

sM

etho

dolo

gica

l qua

lity

asse

ssm

ent c

rite

rion

– p

roce

ss

eval

uati

onV

1V

2V

3V

4V

5V

6V

7V

8Va

lidit

y sc

ore

(V) i

n %

T1T2

T3T4

T5T6

T7T8

T9Va

lidit

y sc

ore

(T) i

n %

Dri

esse

n et

al.

(201

0, 2

011)

**

++

+/-

+/-

++

++

87,5

++

++

+/-

++

++

94,4

Gro

enev

eld

et a

l. (2

010,

201

1) *

+N

/A+

+/-

++/

-N

/A+

83,3

-+

++

-+

+N

/A+

75Fr

ench

et a

l. (2

010)

*+/

-+

+/-

N/A

+N

/A+/

-+

75-

N/A

--

-+/

-+/

-N

/A-

14,3

Dis

hman

& W

ilson

et a

l. (2

009,

201

0)+

+/-

+/-

-+

-+/

-+

56,2

5+

++

++

++

++

100

Yap

et a

l. (2

009,

201

0) *

N/A

N/A

++

+N

/AN

/A+

100

--

--

--

N/A

+-

12,5

Gils

on e

t al.

(200

7, 2

008)

+N

/A+

-+/

--

-+

50-

-+

--

+/-

N/A

+/-

-25

Goe

tzel

, DeJ

oy, W

ilson

et a

l. (2

007,

200

9-20

11) a

***

N/A

+/-

+/-

-+

N/A

+/-

+75

++

++/

-+

++

+-

83,3

Lem

on, E

stab

rook

et a

l. (2

010-

2011

)+/

-+

+/-

++

--

+62

,5+

++

++

++

++

100

And

erse

n et

al.

(201

1)+

N/A

++

++

++

100

--

+/-

--

-+

N/A

+31

,25

Hau

kka,

Peh

kone

n et

al.

(200

9,20

10)

+-

++

+N

/A-

+71

,4-

++

++/

-+

++

+83

,3So

rens

en, H

unt e

t al.

(200

5, 2

007)

-+

++/

-+/

--

-+

50-

++

+-

++

+-

66,6

7Be

resf

ord

et a

l. (2

000,

200

1, 2

010)

*-

++

-+

--

+50

-+

++/

-+/

-+

+N

/A+

75So

rens

en, H

unt e

t al.

(200

7, 2

010)

-N

/A+

++

--

+/-

50-

-+/

--

-+/

-+/

-N

/A+/

-25

Stee

nhui

s et

al.

(200

4)-

+/-

+/-

-+

--

+37

,5-

--

+/-

--

N/A

-+/

-12

,5So

rens

on, Q

uinti

liani

et a

l. (2

010)

_+

++

+/-

--

+56

,25

-+

+-

-+

+N

/A+

62,5

Stod

dard

, Hun

t et a

l. (2

003,

200

5)-

++

N/A

+-

-+/

-50

-+

+/-

+/-

-+

+N

/A-

50Vo

lpp,

Kim

et a

l. (2

009,

201

1)

+-

++

+-

-+

62,5

+/-

-+/

--

+/-

++

++

61,1

Has

son

et a

l. (2

005,

201

0)+

-+

++

--

+62

,5+

-+

-+

++

N/A

+/-

68,7

5Ve

rmee

r et

al.

(201

1)+/

--

-+/

-+/

--

-+

31,2

5+

++

++

++

+-

88,8

9St

rijk

et a

l. (2

011,

201

2)+

++

+/-

++

++

93,7

5+

++

++/

-+

+N

/A-

81,2

5Ve

rwei

j et a

l. (2

011,

201

2)+

-+

++

++

+87

,5+

++

++

++

++

100

Jorg

ense

n et

al.

(201

1, 2

012)

+-

+-

++/

-+

+68

,75

+-

++

+/-

-+

N/A

-56

,25

a Con

trol

led

tria

l*

two

effec

t arti

cles

sco

red

toge

ther

** tw

o pr

oces

s ev

alua

tions

sco

red

toge

ther

***

thre

e eff

ect a

rticl

es s

core

d to

geth

er a

nd 2

impl

emen

tatio

n ar

ticle

s sc

ored

toge

ther

N/A

, not

app

licab

le. +

, pos

itive

. +/-

, not

suffi

cien

t; -,

neg

ative

. All

tria

ls a

re r

ando

miz

ed t

rial

s ex

cept

for

the

tria

ls in

dica

ted

with

a s

uper

scri

pt ‘a

’. Th

e m

axim

um s

core

for

met

hodo

logi

cal q

ualit

y of

effe

ct e

valu

ation

s is

8 (b

ased

on

valid

ity s

ectio

n V1

-V8)

. The

max

imum

sco

re fo

r m

etho

dolo

gica

l qua

lity

of p

roce

ss e

valu

ation

is 9

(bas

ed o

n se

ction

T1-

T9).


61


3

Tabl

e 3.

Ove

rall

scor

es o

f the

met

hodo

logi

cal q

ualit

y of

the

incl

uded

stu

dies

Firs

t au

thor

(Yea

r)M

etho

dolo

gica

l qua

lity

asse

ssm

ent c

rite

rion

– e

ffec

t ev

alua

tion

sM

etho

dolo

gica

l qua

lity

asse

ssm

ent c

rite

rion

– p

roce

ss

eval

uati

onV

1V

2V

3V

4V

5V

6V

7V

8Va

lidit

y sc

ore

(V) i

n %

T1T2

T3T4

T5T6

T7T8

T9Va

lidit

y sc

ore

(T) i

n %

Dri

esse

n et

al.

(201

0, 2

011)

**

++

+/-

+/-

++

++

87,5

++

++

+/-

++

++

94,4

Gro

enev

eld

et a

l. (2

010,

201

1) *

+N

/A+

+/-

++/

-N

/A+

83,3

-+

++

-+

+N

/A+

75Fr

ench

et a

l. (2

010)

*+/

-+

+/-

N/A

+N

/A+/

-+

75-

N/A

--

-+/

-+/

-N

/A-

14,3

Dis

hman

& W

ilson

et a

l. (2

009,

201

0)+

+/-

+/-

-+

-+/

-+

56,2

5+

++

++

++

++

100

Yap

et a

l. (2

009,

201

0) *

N/A

N/A

++

+N

/AN

/A+

100

--

--

--

N/A

+-

12,5

Gils

on e

t al.

(200

7, 2

008)

+N

/A+

-+/

--

-+

50-

-+

--

+/-

N/A

+/-

-25

Goe

tzel

, DeJ

oy, W

ilson

et a

l. (2

007,

200

9-20

11) a

***

N/A

+/-

+/-

-+

N/A

+/-

+75

++

++/

-+

++

+-

83,3

Lem

on, E

stab

rook

et a

l. (2

010-

2011

)+/

-+

+/-

++

--

+62

,5+

++

++

++

++

100

And

erse

n et

al.

(201

1)+

N/A

++

++

++

100

--

+/-

--

-+

N/A

+31

,25

Hau

kka,

Peh

kone

n et

al.

(200

9,20

10)

+-

++

+N

/A-

+71

,4-

++

++/

-+

++

+83

,3So

rens

en, H

unt e

t al.

(200

5, 2

007)

-+

++/

-+/

--

-+

50-

++

+-

++

+-

66,6

7Be

resf

ord

et a

l. (2

000,

200

1, 2

010)

*-

++

-+

--

+50

-+

++/

-+/

-+

+N

/A+

75So

rens

en, H

unt e

t al.

(200

7, 2

010)

-N

/A+

++

--

+/-

50-

-+/

--

-+/

-+/

-N

/A+/

-25

Stee

nhui

s et

al.

(200

4)-

+/-

+/-

-+

--

+37

,5-

--

+/-

--

N/A

-+/

-12

,5So

rens

on, Q

uinti

liani

et a

l. (2

010)

_+

++

+/-

--

+56

,25

-+

+-

-+

+N

/A+

62,5

Stod

dard

, Hun

t et a

l. (2

003,

200

5)-

++

N/A

+-

-+/

-50

-+

+/-

+/-

-+

+N

/A-

50Vo

lpp,

Kim

et a

l. (2

009,

201

1)

+-

++

+-

-+

62,5

+/-

-+/

--

+/-

++

++

61,1

Has

son

et a

l. (2

005,

201

0)+

-+

++

--

+62

,5+

-+

-+

++

N/A

+/-

68,7

5Ve

rmee

r et

al.

(201

1)+/

--

-+/

-+/

--

-+

31,2

5+

++

++

++

+-

88,8

9St

rijk

et a

l. (2

011,

201

2)+

++

+/-

++

++

93,7

5+

++

++/

-+

+N

/A-

81,2

5Ve

rwei

j et a

l. (2

011,

201

2)+

-+

++

++

+87

,5+

++

++

++

++

100

Jorg

ense

n et

al.

(201

1, 2

012)

+-

+-

++/

-+

+68

,75

+-

++

+/-

-+

N/A

-56

,25

a Con

trol

led

tria

l*

two

effec

t arti

cles

sco

red

toge

ther

** tw

o pr

oces

s ev

alua

tions

sco

red

toge

ther

***

thre

e eff

ect a

rticl

es s

core

d to

geth

er a

nd 2

impl

emen

tatio

n ar

ticle

s sc

ored

toge

ther

N/A

, not

app

licab

le. +

, pos

itive

. +/-

, not

suffi

cien

t; -,

neg

ative

. All

tria

ls a

re r

ando

miz

ed t

rial

s ex

cept

for

the

tria

ls in

dica

ted

with

a s

uper

scri

pt ‘a

’. Th

e m

axim

um s

core

for

met

hodo

logi

cal q

ualit

y of

effe

ct e

valu

ation

s is

8 (b

ased

on

valid

ity s

ectio

n V1

-V8)

. The

max

imum

sco

re fo

r m

etho

dolo

gica

l qua

lity

of p

roce

ss e

valu

ation

is 9

(bas

ed o

n se

ction

T1-

T9).

Process evaluation design

The third and fourth columns in table 2 list the characteristics of the included process

evaluations. Most studies (50%) used a mixed methods approach (qualitative and

quantitative) to look at the implementation process [51-56, 58-62]. With regard to the

number of measurements, ten (45%) studies conducted a post-process evaluation only [29,

55, 59, 61-67]. Three (14%) studies collected process evaluation information at three points

in time: before, during and after the intervention [54, 56, 60].

Only two (9%) studies evaluated the implementation process at all three actor levels

(the macro-, meso- and micro-levels, see above) [53, 60]. The majority (50%) of the studies

evaluated the implementation process at both the meso- and micro-levels [51, 52, 54-59,

61, 67, 68]. A minority of the studies (n=8; 36%) reported using a theoretical framework to

guide their process evaluation [47, 51-53, 57, 58, 60, 69]. Of these eight studies, four used

the framework of Steckler and Linnan (or an adapted version of that framework) [51, 57,

58, 69], and the remaining four studies applied either the integrative model [53], the RE-

AIM model [60], a model based on Durlak and Dupre[52], or a model based on Baranowski

and Stables, in combination with Rogers’ framework for the diffusion of innovations [56].

It should be noted that the integrative model is originally a model for the development of

workplace environmental interventions and not specifically for process evaluations [53].

Reporting of process evaluation components

The fourth column of table 2 lists the predetermined process components that were

measured. Seven of these components (reach, recruitment, dose delivered, dose received,

fidelity, satisfaction and maintenance) measure the degree of implementation. The eighth

component, context, maps the barriers and/or facilitators that affect implementation. The

average number of process evaluation components found in the studies was 3.9, ranging

from 1 to 8. Ten (45%) studies evaluated fewer than four process components [29, 53, 62-

64, 66-68, 70, 71]. Six (27.5%) studies reported five or more process components [51, 56-58,

60, 69]. The studies focused mainly on dose received (82%) [29, 51, 52, 54-63, 65, 67-69,

71], dose delivered (68%) [29, 51, 52, 54, 55, 57-62, 65, 67, 68], and context (68%) [52, 54-

57, 60, 61, 64, 66, 69-74]. Fewer than half of the studies (41%) looked at fidelity [51, 52,

56-60, 65]. Nine (41%) studies measured satisfaction [51, 54, 57-59, 63-66]. Relatively few

studies reported on reach (n=7; 32%) [51, 55, 57, 58, 60, 61], or recruitment (n=5; 23%) [56,

58, 60, 64, 69]. In addition, two (9%) studies looked at maintenance [56, 60].

R1R2R3R4R5R6R7R8R9


62

Chapter 3

Implementation barriers and/or facilitators

Next, we identified barriers and/or facilitators which can affect implementation. However,

the barriers and/or facilitators were often not systematically measured by means of

questionnaires or interviews and often only observed and documented on the basis of the

researchers’ experience. Nevertheless, fifty-four different barriers and/or facilitators were

obtained from nineteen (86%) studies (table 4) [52, 54-57, 59-64, 66, 68-74]. On average, the

studies described 6.5 [range 1-22] barriers and/or facilitators. None of the studies reported

a context analysis prior to implementation. The majority of barriers and facilitators in the

five categories as a whole were reported only in a single study.

Fewest barriers and/or facilitators were reported in the category ‘characteristics of the

socio-political context’. Only two (9%) studies reported a barrier or facilitator in this category

(these studies are listed in table 4). The largest number of barriers and/or facilitators (n=17)

was found in the category ‘characteristics of the organization’. Management support was

found most frequently, having been mentioned in eight studies (36%). Strong management

support was described as a facilitator, whereas unbalanced or lack of management support

was found to be a barrier. The definition of management support varied widely. Another

frequently reported barrier was the lack of financial, staffing or material resources (n=5

studies; 26%). In the category ‘characteristics of the intervention’ fifteen barriers and/

or facilitators were identified, with ‘compatibility of intervention with the organization’

being the most commonly reported facilitator in eight studies (42%). The facilitator

‘relative advantage of the intervention’ was reported in five (26%) studies. In the category

‘characteristics of the implementer’ twelve barriers and/or facilitators were identified; the

factor ‘available time of the implementer’ was most commonly reported (n=5; 26%). In the

category ‘characteristics of the user’, eight barriers and/or facilitators were identified with

‘time constraints’ was frequently reported as a barrier to participation. Furthermore, we

found that high work demands and a high workload were also a barrier to participation.


63


3

Table 4. Reported barriers (B) and/or facilitators (F) in the studies included in this review

Main categoriesDescription of the determinants for implementation B/F

Characteristics of the socio-political context

1. Compatibility of program with societal developments (attention for health in society) [74]

F

2. Competitive business environment [53] BCharacteristics of the organization

3. Organizational reorganization: reorganization due to take over by another company [68]

B

4. Lack of resources: financial, personnel, material (e.g., equipment, facilities) resources or lack of space or facilities [54, 57, 66, 72, 74]

B

5. Organizational culture:(a) Senior leaders emphasized the need to implement the intervention keeping the organizational culture in mind [53] (b) Intervention did not fit the organizational culture [55, 72] (c) The organizational culture emphasized goal setting and tracks progress towards achieving goals [53] (d) Worksite culture supported social interaction among workers and between workers and managers [55]

F B F F

6. Organizational size:(a) In a large organization (1000+ employees) there were numerous competing priorities and it was challenging to maintain visibility [60] (b) In a small organization (<500 employees) it is challenging to assemble a critical mass of potential participants for participation in the intervention [60] (c) Small organizations tend to receive more intervention components per employee than larger organizations [61]

B

B

F

7. Amount of company locations: Different company locations at which the intervention needs to be delivered [74]

B

8. Organization’s awareness of perceived benefits of investment [74], and awareness of relevance and economics of health and employee wellness [53, 60]

F

9. Company image: the program gives the organization a positive image since it shows that the organization cares about their employees [66]

F

10. Perceived responsibility of employer towards workers health and wellbeing [74] F11. High staff turnover rate among employees made it difficult to provide adequate

exposure to the intervention [68, 69]B

12. Good collaboration between persons/ structures/ services/ collaborative partners within or outside departments and organizations [54, 66, 72]

F

13. Conflicting relationship between management and researchers[68] B14. General good organizational support for health promotion [53] F15. Poor psychosocial work environment consisting of the following the

subcomponents: influence at work, work pace quantitative work demands, interpersonal relations [70]

B

16. History of social interaction: Worksite has a history of bringing employees together for social activities and a history of positive social interaction between worker and management [55]

F

17. Management support: (a) Strong (upper) management support for intervention and general health promotion efforts at the organization [55, 60, 68, 72, 73] (b) Unbalanced management support for intervention [55, 68] (c) Managers encouraging workers to attend intervention [55] (d) Experienced management support are different for junior employees and senior employees [64] (e) formal approval of upper management before start of intervention [57] (f) Lack of perceived management support by implementers on site [74] (h) Management commitment and willingness to provide employees with release time from their usual duties to attend intervention [55]

F B F B F BF

R1R2R3R4R5R6R7R8R9


64

Chapter 3

Characteristics of the organization

18. Management participation and engagement: (a) Active management participation and involvement alongside and with workers [55, 73] (b) Active management engagement in planning [55]

F F

19. Relationship between management and employees: Respectful relationship between management and worker [55]

F

Characteristics of the implementer

20. Job position of implementer:[74] (a) Self-employed (advantage of managing his or her own time) (b) Internal position (facilitating in scheduling appointments) (c) external position

F F B

21. (Perceived) Support for implementers:[74](a)Poor support from co- implementers (b) Support for implementers to change their routines (applicable when implementer is an occupational physician) [74]

B F

22. Collaboration between implementers: lack of possibility to exchange experiences between implementers [74]

B

23. Available time of implementer: (a) Sufficient time available to implement intervention [56, 66, 72, 74] (b) The intervention involved extra work on top of the heavy workload of the regular duties of the implementer [66] (c) planning difficulties of implementers with planning al contacts in the intervention period [59]

F B B

24. Expectations of implementer: implementers expectations were met [74] F25. Absence of a project leader/ leading person/ ambassador[72] B26. Implementers’ compliance with intervention protocol [52] F27. Staff turnover among implementers: drop out of implementers (without replacing

them) [69, 72]B

28. Absence of decision maker among implementers: among the implementers there lacked a person who was entitled to make decision at department level [72]

B

29. High perceived Level of control for intervention delivery by provider/implementer [60]

F

30. Low level of engagement of implementers in planning, promoting and providing feedback on intervention activities [55]

B

31. Personnel characteristics of implementer: sufficient skills, knowledge and competence to implement guideline or intervention correctly [55, 59, 74]

F

Characteristics of the intervention program

32. Degree of rewards: either financial reimbursement or other incentives [53, 68] F33. Compatibility and alignment of intervention with:

(a) organizations mission statement/business goals/ institutional policy change [53, 60, 68, 74](b) policy, culture, norms and current practices of organization [56, 58, 66, 72] (c) Ease of integration of intervention in working live [64]

F F F

34. The intervention fit implementers current work [74] F35. Intervention is part of the worksites integral health policy and seen as a pilot for

future health promotion policy instead of independent project [57]F

36. Relative advantage: intervention is advantageous compared to the current situation and no negative consequences were observed and the company, managers, implementers and participants benefit from participation [54, 56, 66, 72, 74]

F

37. Time: Project took more time than expected due to high workload of administration and planning[74]

B

38. Complexity: Intervention was not too difficult or complex to implement and execute [56, 59, 72, 74]

F

39. Observability of positive results of the intervention [74] F

Table 4. Continued


65


3

Characteristics of the intervention program

40. Risk and uncertainty level/Triability: the degree to which an innovation can be adopted/implemented with minimal risk [56]

F

41. Conflicting interest between worksite and intervention [66] B42. Timing of intervention activities: intervention activities coincide with scheduled

breaks [68]F

43. Technical problems (e.g., equipment breaks down) [54, 69] B44. Degree of incorporation of program communication and interventions into already

established communication channels or existing worksite events/meetings [53, 55]F

45. Presence of advisory board: well-functioning advisory board [55] F46. Ease of access to the program by bringing the program to participants and making

participation free or inexpensive [53]F

Characteristics of the participant

47. Needs of participants: [a] Positive personal preferences for program [63][b] No need for intervention (e.g., already being healthy) [74] [c] Positive program expectation [71], [d] Prior failed attempts to maintain a healthy lifestyle [62]

F B F B

48. Current workload and work structure/schedules: volume of daily tasks, overtime work, shift work, part-time work, irregular work schedules, shifts of different lengths, time-pressures [53, 60, 64, 68]

B

49. Work demands: Workers were unable to participate since they could not leave their work due to work demands, obligations and limited free time and flexibility to leave immediate work area [55, 57, 60, 64]

B

50. Time constraints of participants: lack of time, time constraints and willingness to make time to participate at work [53, 54, 57, 62, 63, 74]

B

51. Amount of peer leaders: Few peer leaders due to geographically separated worksites made it difficult to establish group cohesion[68]

B

52. Lack of social support: [a] No interaction with the entire workforce to build worksite-wide social norms and social support) [68] [b] Peer support: difficult to engage in behavior not considered normal by peers [64]

B B

53. Lack of motivation of workers to participate in intervention[54] B54. Participants self-efficacy: Low to medium self-efficacy is a barrier for participation

[70]B

Degree of implementation and program effectiveness

Only seven (31.8%) studies evaluated the association between implementation and program

outcomes [52, 58-62, 67]. These studies generally found that the level of implementation

in terms of high fidelity and dose was positively associated with a positive change in their

primary outcome measures (body weight, waist circumference, Body Mass Index, fruit and

vegetable intake, physical activity levels, smoking cessation). This was analyzed by means

of linear or logistic regression analysis [52, 58-62, 67], latent growth modeling [52], dose or

as-treated analysis [52, 58-62, 67], analysis of variance [52, 58-62, 67], mixed model logistic

regression analysis or linear mixed model regression analysis [52, 58-62, 67], chi square

tests [52, 58-62, 67], and multilevel linear regression analysis [52, 58-62, 67]. Of the seven

studies, three studies found that higher participation levels (dose received) significantly and

R1R2R3R4R5R6R7R8R9


66

Chapter 3

positively impacted their primary outcome measures (body weight, waist circumference,

physical activity levels and smoking cessation) [52, 58, 62].

In the study on personal and team goal-setting with the aim of increasing leisure-

time physical activity, Dishman and Wilson found that high implementation groups had a

greater increase in vigorous physical activity over the three time points than did the low

implementation sites. For dose received no significant change over time was observed for

any of the outcome measures (walking, moderate physical activity, vigorous physical activity)

[52]. The results of the study of Volpp and Kim on improving smoking cessation rates by

offering financial incentives for smoking cessation, the results showed that the attendance

levels to the cessation programs were higher among quitters than among non-quitters

in the intervention group. Additionally, quitters in the intervention group attended more

than two times as many sessions as non-quitters in the intervention group [52, 58-62, 67].

Finally, in the study of Verweij on preventing weight gain by implementing an occupational

health guideline, it was found that employees with higher attendance and satisfaction levels

significantly reduced their waist circumference and body weight compared to employees

with lower attendance and satisfaction levels[52, 58-62, 67].

Discussion

The aims of this review were to (1) further our understanding of the quality of process

evaluations alongside effect evaluations for WHPPs, (2) identify barriers/facilitators

affecting implementation, and (3) explore the relationship between effectiveness and the

implementation process.

Prior to discussing the main findings, it should be noted that only a small number of

studies that evaluated the effectiveness of a WHPP included a process evaluation relating

to the implementation of that WHPP. Of the 307 effect evaluations identified in this review,

only twenty-two (7.2%) published an additional process evaluation. With respect to the first

aim we can conclude that the quality of process evaluations alongside effect evaluations of

WHPPs was generally poor to average and a systematic approach was lacking. This makes it

difficult to draw firm conclusion about reliably identifying implementation barriers and/or

facilitators (second aim) or about the relation between effectiveness and implementation

(third aim). Murta et al. (2007) found the same difficulties in a systematic review. They

concluded that process evaluations alongside workplace stress-management interventions


67


3

were mainly poor to average, also making it difficult to identify reliable determinants of

effective intervention implementation [22].

The process evaluations covered by our review lacked a theoretical basis, and the

most frequently measured process components were dose delivered and dose received.

This reflects the primary interest of researchers in actual intervention delivery and in

participation levels (in other words, quantitative outcomes), rather than in how an

intervention is delivered, the quality of delivery (fidelity), maintenance, or the reasons

respondents participated or not. Possible explanations may be that researchers are more

trained in quantitative methods of research and less in qualitative methods and analysis.

Furthermore, good qualitative research takes a lot of time and energy and is therefore more

expensive than quantitative research [75].

The included process evaluations tended to operationalize the measured process

components in different ways, even when using the same framework for evaluation. For

instance, four studies used Steckler and Linnan’s framework as a guideline for their process

evaluation [51, 57, 58, 69]. However, in some of these studies, actual attendance levels were

placed under reach [51, 57, 58], whereas another study included attendance levels as part of

the dose received [69]. The definitions for ‘reach’ and ‘dose received’ as defined by Steckler

and Linnan are: ‘proportion of intended target audience that participates in an intervention’

and ‘the extent to which participants actively engage with the intervention’ respectively [14].

The different approaches to operationalization in studies of these components could be

explained by these somewhat ambiguous descriptions in Steckler and Linnan’s framework.

What is the difference between ‘participating’ and ‘actively engaging with’ an intervention?

We can assume that ‘participating in an intervention’ involves a focus on participation and

non-participation regardless of the frequency, duration and intensity of participation and that

these aspects may actually be taken into account when measuring active engagement with

the intervention. Since several studies had different approaches to the operationalization

of process components, it was a challenge to translate the definitions used into general

terms. We therefore adopted a working definition for each process component based on

a previously published framework in order to interpret our findings [11]. For example, we

defined reach as ‘the proportion of the target audience that is aware of the intervention’ and

dose received as ‘the proportion of participants that actually participates in the intervention’,

including frequency, duration and intensity [11]. We therefore distinguished between

awareness and actual participation, the latter including level of participation. However, due

to the operationalization ambiguity in Steckler and Linnan’s framework, others might define

R1R2R3R4R5R6R7R8R9


68

Chapter 3

these components, and therefore interpret the findings, differently. This limitation was also

mentioned in a comparable review by Durlak and Dupre (2008), revealing that researchers

still do not take the time and effort necessary to develop consensus about the best way to

perform process evaluations in different settings [13].

Our review of the studies provided evidence of various barriers and/or facilitators that

could influence the implementation process. However, partly as a result of the relatively

low quality of the process evaluations, no systematic examination of barriers and/

or facilitators affecting implementation was possible. Barriers and/or facilitators were

often not systemically measured and only observed and documented on the basis of the

researchers’ experience. This raises the question of how reliable the results of this review

are with respect to the identified barriers and/or facilitators (table 4). Since the barriers

and/or facilitators identified in this review clearly overlap with other reviews focusing on

these factors, we can conclude that our results should be reliable [10, 12, 13, 22, 24, 25]. For

example, Sangster-Gormley also identified ‘active management participation’ as a facilitator

for implementation [25].

Another notable finding was that researchers evaluated barriers and/or facilitators

mostly after implementation and only a few studied them during implementation. However,

interventions are more likely to be successful if potential barriers and/or facilitators are

assessed beforehand so they can be anticipated, facilitating implementation. Although

most studies failed to observe barriers and/or facilitators systematically, some barriers and/

or facilitators were more frequently reported in several studies. This suggests that these

are important factors which researchers, practitioners and implementers need to take

into account during a WHPP. In the category ‘characteristics of the socio-political context’,

only one barrier (‘competitive business environment’) and one facilitator (‘compatibility

of program with societal developments’) were reported in all twenty-two studies. This

could suggest that this category is virtually disregarded, despite its importance for a

thorough understanding of these socio-political factors prior to implementation, since the

intended user of the intervention is part of an organization, which in turn is part of a wider

environment [12]. However, it is also possible that intervention developers and researchers

already anticipate on socio-political issues and try to take these into account throughout the

development and implementation process.

Most barriers and/or facilitators were reported within the category ‘characteristics of

the organization’. This is not surprising since most implementation research focuses at this

level as this is essential for continuation. Moreover, the studies in this review are selected


69


3

because they focus on organizations. Frequently mentioned implementation barriers were:

‘lack of resources’, ‘no fit of the intervention with organizational culture’ and ‘unbalanced

or lack of manager support’. For example, when the intervention program requires a fitness

center but there is no center nearby, this intervention is not the best fit with the company

and other options, such as sports activities (possibly outdoors) in the immediate vicinity of

the company, should be explored. Frequently mentioned organizational facilitators for WHPP

implementation were: ‘organizations awareness of perceived benefits and relevance’; ‘good

collaboration with all persons involved’; ‘strong and formal management support’; and

‘active management participation and engagement’. The researcher or implementer could

work on these factors with the aim of optimizing implementation, for example by organizing

manager meetings before the start of the intervention so that everyone is informed. However,

another frequently mentioned facilitator for implementation was: ‘an organizational culture

that emphasizes goal setting and supports social interaction’. Nevertheless, it is unclear what

researchers and implementers can do when the organization culture does not fulfill this

criterion before the start of the study, especially since organizational culture is a complex

phenomenon, which is not easily changed [76]. We suggest that the best way to deal with

this ‘problem’ is to adapt the intervention where possible to ensure an optimal fit with the

current organizational culture. In addition we would advise researchers and implementers

to take the time to assess the organizational culture and include organizational determinants

in their process evaluation [77].

Turning to the category ‘characteristics of the implementer’, the results of this review

indicate that the most important facilitators for implementers are: ‘sufficient time’, ‘skills’,

‘knowledge’ and ‘competence’. When implementers experienced ‘planning difficulties’ or

a ‘heavy workload’, this represented a barrier to implementation. However, these barriers

can be overcome by making sure that the implementer receives enough support from

stakeholders within the organization and enough administrative support. In the category

‘characteristics of the intervention program’, fifteen barriers and/or facilitators were

reported. The three most frequently reported facilitators in this category were: ‘relative

advantage’, ‘compatibility’ and ‘complexity’. These three factors are part of Rogers’

Diffusion of Innovations theory, and they are therefore known facilitating factors which

could explain why researchers focused on these factors [78]. However, we also identified

barriers relating to the intervention characteristics, including: ‘conflicting interests between

worksite and interventions’ and ‘technical problems’ which are not part of a known theory.

Fewest barriers and/or facilitators were reported in the category ‘characteristics of the

R1R2R3R4R5R6R7R8R9


70

Chapter 3

participant’. The most commonly reported barrier to participation was ‘time constraints of

participants’, suggesting that employers should give employees time off to participate in

the interventions. This facilitator is also a reflection of the facilitator ‘management support

and engagement’ which was mentioned earlier. Two frequently reported perceived barriers

for participants were ‘work demands’ and ‘current workload’. However, as for many of the

other found barriers, it is difficult to address these two factors. A possible suggestion to

overcome these barriers could be that at onset of the study, employers should stress out

that employees are allowed to participate under working hours and that if they experience

difficulties in participating due to work demands and workload they should discuss this with

their manager.

The reported barriers and/or facilitators in this review are comparable with the results of

other reviews mentioned in this paper [10, 12, 24, 25]. This suggests that the findings of this

review are generalizable to other settings. It would be beneficial to explore all the barriers

and/or facilitators listed here before implementation so that researchers and implementers

can anticipate possible barriers, incorporate possible facilitators and perhaps adjust their

program accordingly. A new Dutch instrument was proposed recently that makes it possible

to map barriers and/or facilitators beforehand, and therefore possibly help to facilitate

the actual implementation of WHPPs. It also serves as a monitoring instrument enabling

program adjustments before and during implementation (TNO R10625; Fleuren et al. 2012).

Unfortunately, it was not possible to rank the 54 reported barriers and/or facilitators

in order of importance because some determinants were only identified once. This could

suggest that the factor was specific to the type of intervention. In addition, the relation

between the 54 reported barriers and/or facilitators is unclear. This limitation was also

experienced by Fleuren et al. (2004), showing that more research is needed into barriers

and/or facilitators affecting the implementation process [12].

Despite the low quality of most included process evaluations, the findings of this

review do suggest that higher levels of implementation are associated with better program

outcomes. However, the few studies that investigated this association showed a narrow

focus on implementation, mainly addressing the link between dose received and/or fidelity

in relation to program outcomes. Whereas it is so important to use all aspects of process

evaluations for the interpretation of the (lack of) effects of a program [18, 19].

A limitation, which is inherent to writing a systematic review, is publication bias or

studies being overlooked. In other words, our overview of the literature may not be

complete. We tried to avoid this pitfall by selecting four different databases, both medical


71


3

and psychological, using broad search terms and checking each lifestyle (i.e. physical activity,

nutrition, smoking, alcohol use and relaxation) separately and by checking the references in

the studies we included. Clearly, our decision to restrict our search to process evaluations in

combination with effect evaluations in a (randomized) controlled trial design for WHPP has

caused a number of process evaluations to fall out of scope for this review. Although we do

acknowledge this, our aim was to gain insight into the relationship between implementation

quality and effectiveness. Importantly, this is also a tendency that has been suggested by

implementation journals, such as Implementation Science.

A best-evidence synthesis falls outside the scope of this review. Moreover, it could not be

performed since we included only the studies that could be paired with a process evaluation.

This means that not all known and relevant controlled trials in the field of WHPPs focusing

on healthy lifestyle are included in this review.

Conclusion and implications for future research

Given the fact that relatively few process evaluations were found by comparison with the

high number of RCT’s, and taking into account that not all process evaluations are published,

it is safe to conclude that process evaluations are still not as high on the research agenda as

effect evaluations. The importance of conducting an effect evaluation as well as a process

evaluation is increasingly being advocated [22, 79]. It does appear that lately more process

evaluations are being conducted, but this review shows that they are in general of low to

average quality due to the lack of a systematic approach.

Durlak and Dupre (2008) noted that “implementation matters” and “science cannot

study what it cannot measure accurately and cannot measure what is does not define” [13].

Our findings support this observation, and moreover, suggest that we should be asking:

what are we measuring, when we measure implementation at all? Without a standardized

approach to the operationalization of process evaluation components, it is difficult to

faithfully replicate studies, identify implementation barriers and/or facilitators, or assess the

methodological quality of process evaluations. This indicates that a general framework for

process evaluations is required that researchers, implementers, reviewers and practitioners

can use to evaluate and assess the quality of the implementation of a WHPP [80, 81]. In

order to create a general framework, it is essential that all relevant stakeholders subscribe to

a consensus about the terminology and operationalization of relevant process components

by developing a taxonomy for process evaluations. This taxonomy could constitute a

fundamental first step towards the standardization of process evaluations in multiple fields,

R1R2R3R4R5R6R7R8R9


72

Chapter 3

and lead to program reproducibility [80]. Furthermore, it will allow researchers to compare

and assess the quality of process evaluations, systematically identify implementation

barriers and/or facilitators and possibly link implementation to program outcomes.

In short, this review demonstrates the need for a systematic approach to process

evaluation as a way of improving WHPP implementation. In order to set the first steps into

this direction, we suggest that future process evaluations need to apply the framework

Wierenga et al. 2012 we proposed in the introduction of this review and should also use

this framework to assess the methodological quality of the studies. This framework allows

mapping the complete implementation process and takes into account determinants of

implementation and provides the necessary explicit operationalization of each component

[11].


73


3

Appendices 1

Search strategies used for PubMed

Lifestyle behavior Strategy

Physical activity ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR physical activity OR exercise OR physical fitness) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))

Nutrition ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR Nutrition OR healthy eating OR food OR diet OR food services OR nutrition policy) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))

Smoking ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR smoking OR Tobacco OR smoking cessation OR tobacco use) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))

Alcohol use ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR Alcohol drinking OR Alcohol intake OR alcohol consumption) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))

Relaxation ((Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR “stress management” OR “job stress” OR “work stress” OR “psychological stress” OR “life stress” OR “emotional stress” OR distress OR “mental suffering” OR “relaxation therapy” OR relaxation OR meditation OR mind-body OR “mind-body therapies” OR “leisure activities” OR mindfulness OR “mental healing” OR “exercise movement techniques” OR “breathing exercises” OR yoga OR “tai ji” OR pilates OR engagement OR vitality OR detachment) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress OR relaxation)) AND ((English OR Dutch[Language])) AND (“2000/01/01”[Date - Publication]: “2012/08/01”[Date - Publication]))

R1R2R3R4R5R6R7R8R9


74

Chapter 3

Search strategies used for EMBASE


Physical activity ‘controlled study’/exp OR ‘randomized controlled trial’/exp OR ‘intervention study’/exp AND (‘worker’/exp OR ‘employee’/exp OR ‘personnel’/exp OR ‘workplace’/exp OR ‘work environment’/exp) AND (‘health promotion’/exp OR ‘lifestyle modification’/exp OR ‘physical activity’/exp OR ‘exercise’/exp OR ‘fitness’/exp) AND (‘absenteeism’/exp OR ‘sickness absence’/exp OR ‘medical leave’/exp OR ‘body mass’/exp OR ‘body weight’/exp OR ‘lifestyle’/exp OR ‘life style’/exp OR ‘quality of life’/exp OR ‘health’/exp OR ‘health behavior’/exp OR ‘blood pressure’/exp OR ‘cholesterol blood level’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py

Nutrition ‘controlled study’/exp OR ‘randomized controlled trial’/exp OR ‘intervention study’/exp AND (‘worker’/exp OR ‘employee’/exp OR ‘personnel’/exp OR ‘workplace’/exp OR ‘work environment’/exp) AND (‘health promotion’/exp OR ‘lifestyle modification’/exp OR ‘nutrition’/exp OR ‘food’/exp OR ‘diet’/exp OR ‘catering service’/exp OR ‘nutrition policy’/exp) AND (‘absenteeism’/exp OR ‘sickness absence’/exp OR ‘medical leave’/exp OR ‘body mass’/exp OR ‘body weight’/exp OR ‘lifestyle’/exp OR ‘life style’/exp OR ‘quality of life’/exp OR ‘health’/exp OR ‘health behavior’/exp OR ‘blood pressure’/exp OR ‘cholesterol blood level’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py

Smoking ‘controlled study’/exp OR ‘randomized controlled trial’/exp OR ‘intervention study’/exp AND (‘worker’/exp OR ‘employee’/exp OR ‘personnel’/exp OR ‘workplace’/exp OR ‘work environment’/exp) AND (‘health promotion’/exp OR ‘lifestyle modification’/exp OR ‘smoking’/exp OR ‘tobacco’/exp OR ‘smoking cessation’/exp OR ‘tobacco dependence’/exp) AND (‘absenteeism’/exp OR ‘sickness absence’/exp OR ‘medical leave’/exp OR ‘body mass’/exp OR ‘body weight’/exp OR ‘lifestyle’/exp OR ‘life style’/exp OR ‘quality of life’/exp OR ‘health’/exp OR ‘health behavior’/exp OR ‘blood pressure’/exp OR ‘cholesterol blood level’/exp OR ‘stress’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py

Alcohol use ‘controlled study’/exp OR ‘randomized controlled trial’/exp OR ‘intervention study’/exp AND (‘worker’/exp OR ‘employee’/exp OR ‘personnel’/exp OR ‘workplace’/exp OR ‘work environment’/exp) AND (‘health promotion’/exp OR ‘lifestyle modification’/exp OR ‘drinking behavior’/exp OR ‘alcohol’/exp OR ‘alcohol consumption’/exp) AND (‘absenteeism’/exp OR ‘sickness absence’/exp OR ‘medical leave’/exp OR ‘body mass’/exp OR ‘body weight’/exp OR ‘lifestyle’/exp OR ‘life style’/exp OR ‘quality of life’/exp OR ‘health’/exp OR ‘health behavior’/exp OR ‘blood pressure’/exp OR ‘cholesterol blood level’/exp OR ‘stress’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py

Relaxation trial OR randomized AND controlled AND trial OR controlled AND trial AND (‘worker’/exp OR workers OR ‘employee’/exp OR employees OR workforce OR worksite OR ‘work site’ OR ‘work environment’/exp) AND (‘health’/exp AND promotion OR worksite AND ‘health’/exp AND promotion OR ‘lifestyle’/exp AND intervention OR ‘stress management’/exp OR ‘job stress’/exp OR ‘work stress’/exp OR ‘psychological stress’/exp OR ‘life stress’/exp OR ‘emotional stress’/exp OR ‘distress’/exp OR ‘mental suffering’ OR ‘relaxation therapy’/exp OR ‘relaxation’/exp OR ‘meditation’/exp OR ‘mind body’ OR ‘mind-body therapies’/exp OR ‘leisure activities’/exp OR mindfulness OR ‘mental healing’/exp OR ‘exercise movement techniques’/exp OR ‘breathing exercises’/exp OR ‘yoga’/exp OR ‘tai ji’/exp OR ‘pilates’/exp OR engagement OR vitality OR detachment) AND (‘absenteeism’/exp OR ‘sickness’/exp AND ‘absence’/exp OR sick AND leave OR body AND ‘mass’/exp AND index OR body AND ‘weight’/exp OR ‘lifestyle’/exp OR ‘life’/exp AND style OR quality AND of AND ‘life’/exp OR ‘health’/exp AND ‘behavior’/exp OR ‘blood’/exp AND ‘pressure’/exp OR ‘cholesterol’/exp AND level OR ‘stress’/exp OR ‘relaxation’/exp) AND ([Dutch]/lim OR [English]/lim) AND [2000-2012]/py


75


3

Search strategies used for PsycINFO


Physical activity (KW=(trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or physical activity or exercise or physical fitness or ‘active living’) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level)) AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)

Nutrition KW=(trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or nutrition or food or diets or ‘healthy eating’ or ‘food services’ or ‘nutrition policy’) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level) AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)

Smoking KW = (trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or smoking OR Tobacco or Tobacco smoking or smoking cessation or tobacco use) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress) AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)

Alcohol use KW = (trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or ‘drinking behavior’ OR alcohol OR ‘alcohol consumption’ OR ‘alcohol drinking’) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress) AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)

Relaxation KW = (trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or ‘work site’ or ‘work environment’ or workplace or ‘work place’ or personnel or working conditions or business organizations) and (health promotion or worksite health promotion or lifestyle intervention or “stress management” OR “job stress” OR “work stress” OR “psychological stress” OR “life stress” OR “emotional stress” OR distress OR “mental suffering” OR “relaxation therapy” OR relaxation OR meditation OR mind-body OR “mind-body therapies” OR “leisure activities” OR mindfulness OR “mental healing” OR “exercise movement techniques” OR “breathing exercises” OR yoga OR “tai ji” OR pilates OR engagement OR vitality OR detachment) and (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress OR relaxation)AND (Limiters - Published Date: 20000101-20120731; Language: Dutch, English; Document Type: Journal Article)

R1R2R3R4R5R6R7R8R9


76

Chapter 3

Search strategies used for Cochrane Central Register of Controlled trials


Physical activity (Trial or randomized controlled trial or controlled trial) and (Worker or workers or employee or employees or workforce or worksite or “work site” or “work environment”) and (health promotion or worksite health promotion or lifestyle intervention or physical activity or exercise or physical fitness) and (absenteeism or sickness absence or sick leave or body mass index or body weight or lifestyle or life style or quality of life or health or health behavior or blood pressure or cholesterol level):ti,ab,kw from 2000 to 2012 (Word variations have been searched)

Nutrition (Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR Nutrition OR healthy eating OR food OR diet OR food services OR nutrition policy) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level):ti,ab,kw from 2000 to 2012 (Word variations have been searched)

Smoking (Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR smoking OR Tobacco OR smoking cessation OR tobacco use) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress):ti,ab,kw from 2000 to 2012 (Word variations have been searched)

Alcohol use (Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR Alcohol drinking OR Alcohol intake OR alcohol consumption) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress):ti,ab,kw from 2000 to 2012 (Word variations have been searched)

Relaxation (Trial OR randomized controlled trial OR controlled trial) AND (Worker OR workers OR employee OR employees OR workforce OR worksite OR “work site” OR “work environment”) AND (health promotion OR worksite health promotion OR lifestyle intervention OR “stress management” OR “job stress” OR “work stress” OR “psychological stress” OR “life stress” OR “emotional stress” OR distress OR “mental suffering” OR “relaxation therapy” OR relaxation OR meditation OR mind-body OR “mind-body therapies” OR “leisure activities” OR mindfulness OR “mental healing” OR “exercise movement techniques” OR “breathing exercises” OR yoga OR “tai ji” OR pilates OR engagement OR vitality OR detachment) AND (absenteeism OR sickness absence OR sick leave OR body mass index OR body weight OR lifestyle OR life style OR quality of life OR health OR health behavior OR blood pressure OR cholesterol level OR stress OR relaxation):ti,ab,kw from 2000 to 2012 (Word variations have been searched)


77


3

References 1. Kirsten W: Making the link between health and productivity at the workplace--a global perspective.

Ind Health 2010, 48(3):251-255.

2. Jans MP, van den Heuvel SG, Hildebrandt VH, Bongers PM: Overweight and Obesity as Predictors of Absenteeism in the Working Population of the Netherlands. J Occup Environ Med 2007, 49(9):975-980.

3. Robroek SJ, van den Berg TI, Plat JF, Burdorf A: The role of obesity and lifestyle behaviours in a productive workforce. Occup Environ Med 2011, 68(2):134-139.

4. Alavinia SM, Molenaar D, Burdorf A: Productivity loss in the workforce: Associations with health, work demands, and individual characteristics. Am J Ind Med 2009, 52(1):49-56.

5. Engbers LH, van Poppel MN, ChinA.Paw MJ, van Mechelen W: Worksite health promotion programs with environmental changes: a systematic review. Am J Prev Med 2005, 29(1):61-70.

6. Ni Mhurchu C, Aston LM, Jebb SA: Effects of worksite health promotion interventions on employee diets: a systematic review. BMC Public Health 2010, 10(10):62.

7. Jepson RG, Harris FM, Platt S, Tannahill C: The effectiveness of interventions to change six health behaviors: a review of reviews. BMC Public Health 2010, 8(10):538.

8. Proper KI, Koning M, van der Beek AJ, Hildebrandt VH, Bosscher RJ, van Mechelen W: The effectiveness of worksite physical activity programs on physical activity, physical fitness, and health. Clin J Sport Med 2003, 13(2):106-117.

9. Kristensen TS: Intervention studies in occupational epidemiology. Occup Environ Med 2005, 62:205-210.

10. Robroek SJ, van Lenthe FJ, van Empelen P, Burdorf A: Determinants of participation in worksite health promotion programmes: a systematic review. Int J Behav Nutr Phys Act 2009, 6:26.

11. Wierenga D, Engbers LH, van Empelen P, Hildebrandt VH, van Mechelen W: The design of a real-time formative evaluation of the implementation process of lifestyle interventions at two worksites using a 7-step strategy (BRAVO@Work). BMC Public Health 2012, 12(1):619.

12. Fleuren M, Wiefferink K, Paulussen T: Determinants of innovation within health care organizations: literature review and Delphi study. Int J Qual Health Care 2004, 16(2):107-123.

13. Durlak JA, DuPre EP: Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 2008, 41:327-350.

14. Steckler A, Linnan L: Proces Evaluation for Public Health Interventions and Research: San Francisco: Jossey-Bass; 2002.

15. Paulussen TH, Wiefferink K, Mesters I: Invoering van effectief gebleken interventies. In Gezondheidsvoorlichting en gedragsverandering. Edited by Brug,J.,Van Asseman,P.& Lechner,L. Assen: van Gorcum: 2007:.

16. Craig P, Dieppe P, Macintyre S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008, 337(1655):979-983.

17. Saunders RP, Evans MH, Joshi P: Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health promotion practice 2005, 6(2):134-147.

18. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, RIPPLE Study Team: Process evaluation in randomised controlled trials of complex interventions. BMJ 2006, 332(7538):413-416.

19. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks PA: The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med 2004, 27(1):3-12.

R1R2R3R4R5R6R7R8R9


78

Chapter 3

20. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp N, Smith JL: The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med 2006, Feb(21 Suppl 2):S1-8.

21. Mittman BS: Creating the evidence base for quality improvement collaboratives. Ann Int Med 2004, 140(11):897-901.

22. Murta SG, Sanderson K, Oldenburg B: Process evaluation in occupational stress management programs: a systematic review. Am J Health Promot 2007, 21(0890-1171; 0890-1171; 4):248-254.

23. Glasgow RE, Vogt TM, Boles SM: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999, (89):9-1322.

24. Koppelaar E, Knibbe JJ, Miedema HS, Burdorf A: Determinants of implementation of primary preventive interventions on patient handling in healthcare: a systematic review. Occup Environ Med 2009, 66(6):353-360.

25. Sangster-Gormley E, Martin-Misener R, Downe-Wamboldt B, Dicenso A: Factors affecting nurse practitioner role implementation in Canadian practice settings: an integrative review. J Adv Nurs 2011, 67(6):1178-1190.

26. Furlan AD, Pennick V, Bombardier C, van Tulder M, Editorial Board, Cochrane Back Review Group: 2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group. Spine (Phila Pa 1976) 2009, 34(18):1929-1941.

27. Proper KI, Staal BJ, Hildebrandt VH, van der Beek AJ, van Mechelen W: Effectiveness of physical activity programs at worksites with respect to work-related outcomes. Scand J Work Environ Health 2002, 28(2):75-84.

28. van Tulder M, Furlan A, Bombardier C, Bouter L, Editorial Board of the Cochrane Collaboration Back Review Group: Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine (Phila Pa 1976) 2003, 28(12):1290-1299.

29. French SA, Harnack LJ, Hannan PJ, Mitchell NR, Gerlach AF, Toomey TL: Worksite environment intervention to prevent obesity among metropolitan transit workers. Prev Med 2010, 50(4):180-185.

30. Dishman RK, DeJoy DM, Wilson MG, Vandenberg RJ: Move to Improve: a randomized workplace trial to increase physical activity. Am J Prev Med 2009, 36(2):133-141.

31. Yap TL, Davis LS, Gates DM, Hemmings AB, Pan W: The effect of tailored E-mails in the workplace. Part I. Stage movement toward increased physical activity levels. AAOHN J 2009, 57(7):267-273.

32. Goetzel RZ, Baker KM, Short ME, Pei X, Ozminkowski RJ, Wang S, Bowen JD, Roemer EC, Craun BA, Tully KJ, Baase CM, DeJoy DM, Wilson MG: First-year results of an obesity prevention program at The Dow Chemical Company. J Occup Environ Med 2009, 51(2):125-138.

33. Lemon SC, Zapka J, Li W, Estabrook B, Rosal M, Magner R, Andersen V, Borg A, Hale J: Step ahead a worksite obesity prevention trial among hospital employees. Am J Prev Med 2010, 38(1):27-38.

34. Sorensen G, Barbeau E, Stoddard AM, Hunt MK, Kaphingst K, Wallace L: Promoting behavior change among working-class, multiethnic workers: results of the healthy directions--small business study. Am J Public Health 2005, 95(8):1389-1395.

35. Beresford SA, Thompson B, Feng Z, Christianson A, McLerran D, Patrick DL: Seattle 5 a Day worksite program to increase fruit and vegetable consumption. Prev Med 2001, 32(3):230-238.

36. Sorensen G, Barbeau EM, Stoddard AM, Hunt MK, Goldman R, Smith A, Brennan AA, Wallace L: Tools for health: the efficacy of a tailored intervention targeted for construction laborers. Cancer Causes Control 2007, 18(1):51-59.

37. Sorensen G, Stoddard A, Quintiliani L, Ebbeling C, Nagler E, Yang M, Pereira L, Wallace L: Tobacco use cessation and weight management among motor freight workers: results of the gear up for health study. Cancer Causes Control 2010, 21(12):2113-2122.

38. Stoddard AM, Fagan P, Sorensen G, Hunt MK, Frazier L, Girod K: Reducing cigarette smoking among working adolescents: results from the SMART study. Cancer Causes Control 2005, 16(10):1159-1164.


79


3

39. Volpp KG, Troxel AB, Pauly MV, Glick HA, Puig A, Asch DA, Galvin R, Zhu J, Wan F, DeGuzman J, Corbett E, Weiner J, Audrain-McGovern J: A randomized, controlled trial of financial incentives for smoking cessation. N Engl J Med 2009, 360(7):699-709.

40. Driessen MT, Proper KI, Anema JR, Knol DL, Bongers PM, van der Beek AJ: Participatory ergonomics to reduce exposure to psychosocial and physical risk factors for low back pain and neck pain: results of a cluster randomised controlled trial. Occup Environ Med 2011, 68(9):674-681.

41. Groeneveld IF, Proper KI, van der Beek AJ, van Mechelen W: Sustained body weight reduction by an individual-based lifestyle intervention for workers in the construction industry at risk for cardiovascular disease: results of a randomized controlled trial. Prev Med 2010, 51(3-4):240-246.

42. Gilson N, McKenna J, Cooke C, Brown W: Walking towards health in a university community: a feasibility study. Prev Med 2007, 44(2):167-169.

43. Andersen LL, Saervoll CA, Mortensen OS, Poulsen OM, Hannerz H, Zebis MK: Effectiveness of small daily amounts of progressive resistance training for frequent neck/shoulder pain: randomised controlled trial. Pain 2011, 152(2):440-446.

44. Haukka E, Pehkonen I, Leino-Arjas P, Viikari-Juntura E, Takala EP, Malmivaara A, Hopsu L, Mutanen P, Ketola R, Virtanen T, Holtari-Leino M, Nykanen J, Stenholm S, Ojajarvi A, Riihimaki H: Effect of a participatory ergonomics intervention on psychosocial factors at work in a randomised controlled trial. Occup Environ Med 2010, 67(3):170-177.

45. Steenhuis I, Van Assema P, Van Breukelen G, Glanz K, Kok G, De Vries H: The impact of educational and environmental interventions in Dutch worksite cafeterias. Health Promot Int 2004, 19(3):335-343.

46. Hasson D, Anderberg UM, Theorell T, Arnetz BB: Psychophysiological effects of a web-based stress management system: a prospective, randomized controlled intervention study of IT and media workers [ISRCTN54254861. BMC Public Health 2005, 5:78.

47. Vermeer WM, Steenhuis IH, Leeuwis FH, Heymans MW, Seidell JC: Small portion sizes in worksite cafeterias: do they help consumers to reduce their food intake? Int J Obes (Lond) 2011, 35(9):1200-1207.

48. Strijk JE, Proper KI, van der Beek AJ, van Mechelen W: A worksite vitality intervention to improve older workers’ lifestyle and vitality-related outcomes: results of a randomised controlled trial. J Epidemiol Community Health 2012, 66(11):1071-1078.

49. Verweij LM, Proper KI, Weel AN, Hulshof CT, van Mechelen W: The application of an occupational health guideline reduces sedentary behaviour and increases fruit intake at work: results from an RCT. Occup Environ Med 2012, 69(7):500-507.

50. Jorgensen MB, Faber A, Hansen JV, Holtermann A, Sogaard K: Effects on musculoskeletal pain, work ability and sickness absence in a 1-year randomised controlled trial among cleaners. BMC Public Health 2011, 11:840.

51. Driessen MT, Proper KI, Anema JR, Bongers PM, van der Beek AJ: Process evaluation of a participatory ergonomics programme to prevent low back pain and neck pain among workers. Implement Sci 2010, 5:65.

52. Wilson MG, Basta TB, Bynum BH, DeJoy DM, Vandenberg RJ, Dishman RK: Do intervention fidelity and dose influence outcomes? Results from the move to improve worksite physical activity program. Health Educ Res 2010, 25(2):294-305.

53. Wilson MG, Goetzel RZ, Ozminkowski RJ, DeJoy DM, Della L, Roemer EC, Schneider J, Tully KJ, White JM, Baase CM: Using formative research to develop environmental and ecological interventions to address overweight and obesity. Obesity (Silver Spring) 2007, 15 Suppl 1:37S-47S.

54. Pehkonen I, Takala EP, Ketola R, Viikari-Juntura E, Leino-Arjas P, Hopsu L, Virtanen T, Haukka E, Holtari-Leino M, Nykyri E, Riihimaki H: Evaluation of a participatory ergonomic intervention process in kitchen work. Appl Ergon 2009, 40(1):115-123.

55. Hunt MK, Barbeau EM, Lederman R, Stoddard AM, Chetkovich C, Goldman R, Wallace L, Sorensen G: Process evaluation results from the Healthy Directions-Small Business study. Health Educ Behav 2007, 34(1):90-107.

R1R2R3R4R5R6R7R8R9


80

Chapter 3

56. Vermeer WM, Leeuwis FH, Koprulu S, Zouitni O, Seidell JC, Steenhuis IH: The process evaluation of two interventions aimed at portion size in worksite cafeterias. J Hum Nutr Diet 2012, 25(2):180-188.

57. Strijk JE, Proper KI, van der Beek AJ, van Mechelen W: A process evaluation of a worksite vitality intervention among ageing hospital workers. Int J Behav Nutr Phys Act 2011, 8:58.

58. Verweij LM, Proper KI, Hulshof CT, van Mechelen W: Process evaluation of an occupational health guideline aimed at preventing weight gain among employees. J Occup Environ Med 2011, 53(7):722-729.

59. Groeneveld IF, Proper KI, Absalah S, van der Beek AJ, van Mechelen W: An individually based lifestyle intervention for workers at risk for cardiovascular disease: a process evaluation. Am J Health Promot 2011, 25(6):396-401.

60. Estabrook B, Zapka J, Lemon SC: Evaluating the implementation of a hospital work-site obesity prevention intervention: applying the RE-AIM framework. Health Promot Pract 2012, 13(2):190-197.

61. Beresford SA, Shannon J, McLerran D, Thompson B: Seattle 5-a-Day Work-Site Project: process evaluation. Health Educ Behav 2000, 27(2):213-222.

62. Kim A, Kamyab K, Zhu J, Volpp K: Why are financial incentives not effective at influencing some smokers to quit? Results of a process evaluation of a worksite trial assessing the efficacy of financial incentives for smoking cessation. J Occup Environ Med 2011, 53(1):62-67.

63. Yap TL, Busch James DM: Tailored e-mails in the workplace. AAOHN J 2010, 58(10):425-432.

64. Gilson N, McKenna J, Cooke C: Experiences of route and task-based walking in a university community: qualitative perspectives in a randomized control trial. J Phys Act Health 2008, 5 Suppl 1:S176-82.

65. Hunt MK, Harley AE, Stoddard AM, Lederman RI, MacArthur MJ, Sorensen G: Elements of external validity of tools for health: an intervention for construction laborers. Am J Health Promot 2010, 24(5):e11-20.

66. Steenhuis I, van Assema P, Reubsaet A, Kok G: Process evaluation of two environmental nutrition programmes and an educational nutrition programme conducted at supermarkets and worksite cafeterias in the Netherlands. J Hum Nutr Diet 2004, 17(2):107-115.

67. Quintiliani L, Yang M, Sorensen G: A process evaluation of tobacco-related outcomes from a telephone and print-delivered intervention for motor freight workers. Addict Behav 2010, 35(11):1036-1039.

68. Hunt MK, Fagan P, Lederman R, Stoddard A, Frazier L, Girod K, Sorensen G: Feasibility of implementing intervention methods in an adolescent worksite tobacco control study. Tob Control 2003, 12 Suppl 4:IV40-5.

69. Jorgensen MB, Faber A, Jespersen T, Hansen K, Ektor-Andersen J, Hansen JV, Holtermann A, Sogaard K: Implementation of physical coordination training and cognitive behavioural training interventions at cleaning workplaces--secondary analyses of a randomised controlled trial. Ergonomics 2012, 55(7):762-772.

70. Andersen LL: Influence of psychosocial work environment on adherence to workplace exercise. J Occup Environ Med 2011, 53(2):182-184.

71. Hasson H, Brown C, Hasson D: Factors associated with high use of a workplace web-based stress management program in a randomized controlled intervention study. Health Educ Res 2010, 25(4):596-607.

72. Driessen MT, Groenewoud K, Proper KI, Anema JR, Bongers PM, van der Beek AJ: What are possible barriers and facilitators to implementation of a Participatory Ergonomics programme? Implement Sci 2010, 5:64.

73. DeJoy DM, Bowen HM, Baker KM, Bynum BH, Wilson MG, Goetzel RZ, Dishman RK: Management support and worksite health promotion program effectiveness. Ergonomics and Health Aspects 2009, (5624):13-22.

74. Verweij LM, Proper KI, Leffelaar ER, Weel AN, Nauta AP, Hulshof CT, van Mechelen W: Barriers and Facilitators to Implementation of an Occupational Health Guideline Aimed at Preventing Weight Gain Among Employees in the Netherlands. J Occup Environ Med 2012, 54(8):954-960.


81


3

75. Pope C, Ziebland S, Mays N: Qualitative research in health care. Analysing qualitative data. BMJ 2000, 320:114-116.

76. Cameron KS, Quinn RE, Robert E: Diagnosing and changing organizational culture: Based on the Competing Values Framework: San Fransico: Jossey-Bass; 2006.

77. Weiner BJ, Lewis MA, Linnan LA: Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. Health Educ Res 2009, 24(2):292-305.

78. Rogers E.M.: Diffusion of innovations: New York: Free Press; 2003.

79. Nielsen K, Randall R, Holten A, Gonzálezc ER: Conducting organizational-level occupation health interventions: What works? Work & Stress 2010, 24(3):234-259.

80. Abraham C, Michie S: A taxonomy of behavior change techniques used in interventions. Health Psychol 2008, 27(3):379-387.

81. Pedersen LM, Nielsen KJ, Kines P: Realistic evaluation as a new way to design and evaluate occupational safety interventions. Safety Science 2012, 50(1):48-54.

Documents

What is actually measured in process evaluations for ... 3.pdf · (3) explore the relationship between effectiveness and the implementation process. Methods: Pubmed, EMBASE, PsycINFO,