1

The effect of synchronous tele-mental health on depression: a meta-analysis

Student Name: Luqi Kong

Major: Applied Statistics and Decision Making

2016.03.10

2

Contents Introduction .................................................................................................................................... 3

Method ............................................................................................................................................ 3

Effect Size Computation for individual studies .............................................................................. 3

Summary Effect Size Computation ................................................................................................. 4

Fixed effects Model ...................................................................................................................... 4

Heterogeneity .............................................................................................................................. 5

Random effects Model ................................................................................................................ 5

Conclusions and implications ......................................................................................................... 7

References ....................................................................................................................................... 7

Appendix ......................................................................................................................................... 9

3

Introduction Depression is one of the most widespread mental problems, cited as the leading cause of

disability (Lopez et al. 2001). However, only a small percentage of patients with depression

receive mental treatment. Major barriers that prevent patients from mental therapy include

distance, costs, travel time and emotional concerns such as stigma (Osenbach et al. 2013).

Telemental health, the provision of psychological services using telecommunication

technologies (APA 2013), has been introduced as a new solution to overcome the barriers of

traditional mental therapy. In past case studies, program evaluations and random control trials,

TMH has demonstrated positive outcomes: reduced costs, increased satisfaction and improved

clinical outcome (Richardson et al. 2010).

This paper of interest is to examine whether telemental health via synchronous technologies

(video conferencing or telephone conferencing) has a positive impact on reducing depression

and to what extent the symptoms are alleviated.

Method Meta-analysis technique is used to statistically synthesize results from 11 studies extracted from

PubMed, Google Scholar and PsycINFO databases. The software used is R. The 3 main screening

criterion for including articles are: the mental problem is depression; the intervention modality

is either video conferencing or telephone conferencing; the experiment is a random control trial.

The detailed publication description is shown in Appendix 1.

The data in each study includes the following variables: sample size in the treatment group 1n ,

sample size in the control group 2n , mean of depression level (depression level is quantified by

certain measures) 1X in the treatment group, mean of depression level 2X in the control

group, standard error of depression level 1S in the treatment group and standard error of

depression level 2S in the control group. For each study, standard mean difference between two

groups is calculated as the effect size (ES). Further details of this computation will be discussed

later in the Effect size computation for individual studies part. Data and result is shown in

Appendix 2.

Using effect size and standard error computed in each study, summary effect size across all 11

studies can be calculated. Two models are used in this paper (discussed in Summary effect size c

omputation part). One is fixed effects model, which assumes that there is no true effect size

differences across studies. Another is random effects model, which assumes that different study

has different true effect size.

Effect size computation for individual studies Effect size is introduced to quantify the difference between two groups. It can refer to mean

difference, risk ratios, odd ratios or risk difference. In this paper, standardized mean difference

(difference of means between two groups divided by the within-study standard deviation) is

4

used to measure different levels of depression between patients who undergo telemental

therapy (treatment group) and those who use traditional ways (control group).

All the studies included in this paper use two independent groups. Let µ1 and σ1 be the

population mean and standard deviation of the treatment group, µ2 and σ2 be the population

mean and standard deviation of the control group. Assuming the two groups have the same

variance, then σ1 = σ2 = σ. The true (population) standard mean difference is defined as

1 2SMD

. There are mainly 2 sample estimates of the true standard mean difference.

One is Cohen’s d, another is Hedge’s g. Cohen’s d is calculated as

1 2

within

X Xd

S

, where

2 2

1 1 2 2

1 2

( 1) ( 1)

2within

n S n SS

n n

, assuming σ1 = σ2 = σ.

The variance of is d computed as2

1 2

1 2 1 22( )d

n n dV

n n n n

.

The standard error of d is d dSE V d dSE V .

However, if the sample is small, Cohen’s d has a tendency to overestimate the true population

parameter. Hedge’s g improves Cohen’s d by introducing a correction factor3

=1-4 1

Jdf

,

where 1 2 2df n n . Hence, g J d , 2

g dV g V , gSE Vg . 95% confidence

interval is 1.96* , 1.96* ]g gg SE g SE .

The calculated result is shown in detail in Appendix 2.

Summary Effect Size Computation

Fixed effects Model Fixed-effect model assumes that all studies have a common effect size. Therefore, the observed

difference can all be explained by the inherent sampling error in each study. To get the most

precise estimate (minimize variance) of the population effect, a weight iW is assigned to each

study, which is calculated as the inverse of the within-group variance. The summary standard

mean difference M is the weighted average of Hedge’s g in each study. The variance of the

summary effect size is computed as the reciprocal of the sum of the absolute weights. The

computing steps are as follows:

5

1i

g

WV

, i=1

1

k

i

k

i

i

W g

M

W

,

1

1M k

i

i

V

W

,

M MSE V , M

MZ

SE , The 95% confidence interval for

the summary effect is 1.96* , 1.96*M MM SE M SE

The result computed is shown in Appendix 3. Forest plot (Figure 3.1), Funnel Plot (Figure 3.2)

and Galbraith plot (Figure 3.3) are drawn to show the results visually.

To sum up, telemental therapy has an impact on reducing depression with a small p value

0.0001. The summary effect size is -0.1904, indicating that telemental therapy reduces

depression score by 0.1904 on average. Under 95% confidence, the true effect size will lie

between -0.2877 and -0.0930.

Heterogeneity A critical point of the fixed effect model is that we assume that the true effect sizes across

studies are the same. However, the assumption is too strong. Mostly, there are variations in the

true effect sizes, defined as heterogeneity. The observed variation often incorporates both

heterogeneity (between-study difference) and random error (within-study difference).

The method to estimate the extent of heterogeneity is as follows:

a) Calculate the total amount of variation across studies: 2

12 2

1 1

1

= ( )

k

ik ki

i i ki i

i

i

W g

Q W g M W g

W

b) Estimate the expected value of Q , which indicates how much the observed effects

would be expected to vary from each other if the true effect was actually the same

across all studies. df (degrees of freedom) is used to estimate this parameter.

c) Compute the ratio ( 2I ) of the excess variation Q - df and total variationQ . Then we

could know what proportion of the observed variation reflects true differences in effect

size.

The computed Q is 15.67. 2I for the 11 included studies is 39.43%. According to the tentative

benchmark provided by Higgins et al (2003), values on the order of 25%, 50% and 75% each

suggests low, moderate and high heterogeneity. 39.43% indicates that there might be

moderate differences of effect sizes across studies. In the next part, random effects model will

be introduced to address this issue.

Random effects Model

While fixed-effect model assumes a common effect size, random-effect model allows for

differences of true effect sizes across studies. Under random-effect model, the observed

6

difference in effect sizes can be explained by both the inherent sampling error ( i ) in each study

and the true differences between studies ( i ).

The 11 studies included were operated by different researchers independently. The samples

have different demographic characteristics, including age, occupation, medical record, etc. The

intervention and comparison modalities are also not the same, which has a potential to

influence the outcome. Due to these differences, it’s unlikely that the 11 studies would be

equivalent.

To calculate the variance under random-effect model, we need to know both the between-study

variance ( 2 ) and within-study variance ( 2 ). The method for estimating the within-study

variance (Using 2

withinS as a sampling estimate for the population parameter 2 ) is illustrated

above. This part will focus on computing the between-study variance.

The between-study variance 2 is defined as the variance of true effect sizes across an infinite

number of studies, assuming we know the true effect size of each study. However, we cannot

have infinite number of studies and even know the true effect size of each study. In order to

estimate 2 , 2T is introduced calculated as follows:

2 Q dfT

C

, where

2

12

1

1

k

iki

i ki

i

i

W g

Q W g

W

, 1df k ,

2

1

1

1

k

iki

i ki

i

i

W

C W

W

Under the random effects model, the weight iW assigned to each study is the inverse of its

variance, including within-study variance Vg and between study variance 2T . The summary

effect size is then calculated as 1

1

=

k

i i

i

k

i

i

W Y

M

W

, where

2

1iW

Vg T

. The variance of the

summary effect size is estimated as

1

1M k

i

i

V

W

.

The result is shown in Appendix 4.

To sum up, random effects model also suggests that telemental therapy has an impact on

reducing depression. The summary effect size is -0.1570, indicating that telemental therapy

reduces depression score by 0, 1570 on average, a bit lower than that computed under fixed

effects model. Under 95% confidence, the true effect size will lie between -0.2976 and -0.0165.

7

Conclusions and implications In conclusion, telemental health has a positive influence in reducing depression levels. Under

fixed effects model, the depression score is alleviated by 0.1904 on average, with effect size 95%

CI [-0.2877, -0.0930]. Under random effects model, the reduction is 0.1570, with 95% CI [-

0.2976, -0.0165].

Further implications include:

a) Subgroup analysis: do meta-analysis on different intervention modalities (VC or TC) and

different comparison groups (CAU vs FTF)

b) Meta-regression: examine the impact of moderator variables (Specific characteristics in

each study) on study effect size using regression-based techniques.

c) Bayesian network meta-analysis: compare multiple treatments based on estimates from

different studies.

References 1. American Psychological Association. (2013). GUIDELINES FOR THE PRACTICE OF

TELEPSYCHOLOGY. Retrieved from

http://www.apa.org/practice/guidelines/telepsychology.aspx

2. Borenstein, M., Hedges, L. V., & Higgins, J. T. (2009). Introduction ot Meta-Analysis. A

John WIley and Sons.

3. Dorstyn, D., Mathias, J., Denson, L., & Robertson, M. (2012, November). Effectiveness of

Telephone Counseling in Managing Psychological Outcomes After Spinal Cord Injury: A

Preliminary Study. Archives of Physical Medicine and Rehabilitation, 93(11), 2100-2108.

doi:http://dx.doi.org/10.1016/j.apmr.2012.06.002

4. Frueh, B. C., Monnier, J., & Yim, E. (n.d.). A randomized trail of telepsychiatry for post-

traumatic stress disorder. J Telemed Telecare, 13(3), 142-147.

5. Heckman, T. G., & Carlson, B. (2007). A randomized clinical trial of two telephone-

delivered, mental health interventions for HIV-infected persons in rural areas of the

United States. AIDS Behav, 11(1), 5-14.

6. Higgins, J. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003, Sep 6). Measuring

inconsistency in meta-analyses. BMJ, 327(7414), 557-560.

doi:10.1136/bmj.327.7414.557

7. Lopez, A. D., Mathers, C. D., & Ezzati, M. (2006, May 27). Global and regional burden of

disease and risk factors, 2001: systematic analysis of population health data. Lancet,

367(9524), 1747-57.

8. Lovell, K., Cox, D., & Haddock, G. (2006). Telephone administered cognitive behaviour

therapy for treatment of obsessive compulsive disorder:randomized controlled non-

inferiority trial. BMJ, 333(7574), 883.

8

9. Ludman, E. J., Simon, G. E., Tutty, S., & Von, K. M. (2007). A randomized trial of

telephone psychotherapy and pharmacotherapy for depression: continuation and

durability of effects. J Consult Clin Psychol, 75(2), 257-266.

10. Mohr, D. C., Ho, J., & Duffecy, J. (2012). Effect of telephone-administered vs face-to-face

cognitive behavioral therapy on adherence to therapy and depression outcomes among

primary care patients: a randomized trail. JAMA, 307(21), 356-361.

11. Moreno, F., & Chong, J. (2012). Feasiblility and Acceptability of clinic-based

telepsychiatry for low-income hispanic primary care patients. Telemedicine and e-

Health, 18(4), 297-304.

12. Napolitano , M. A., Babyak, M. A., Palmer, S., Tapson, V., Davis, R. D., & Blumenthal, J. A.

(2002, Oct). Effects of a telephone-based psychosocial intervention for patients awaiting

lung transplantation. Investigational Study of Psychological Intervention in Recipients of

Lung Transplant(INSPIRE) Investigator, 122(4), 1176-84.

13. Nelson, E. L., Barnard, M., & Cain, S. (2003). Treating childhood depression over

videoconferencing. Telemed J E Health, 9(1), 49-55.

14. Osenbach, J. E., O'Brien, K. M., Matthew, M., & Smolenski, D. J. (2013, July 13).

Synchronous telehealth technologies in psychotherapy for depression:a meta analysis.

Depression and Anxiety, 1053-1067. doi:10.1002/da.22165

15. Richardson, L. K., Frueh, B. C., Grubaugh, A. L., & Egede, L. (2009, Sep 1). Current

Directions in Videoconferencing TeleMental. Clin Psychol, 16(3), 323-338.

doi:10.1111/j.14682850.2009.01170x

16. Simon, G. E., Ludman, E. J., & Tutty, S. (2004). Telephone psychotherapy and telephone

care management for primary care patients starting antidepressant treatment:a

randomized controlled trail. JAMA, 292(8), 935-942.

17. Strachan, M., Gros , D. F., & Ruggiero, K. J. (2012). An integrated approach to delivering

exposure-based treatment for symptoms of PTSD and depression in OIF/OEF

veterans:preliminary findings. BehavTher, 43(3), 560-569.

9

Appendix

1

Appendix 1: Publication Description

Study Intervention Intervention

Modality Control

Modality

Depression Outcome Measure

Sample

Dorstyn et al.(2012) 15 weekly, 90 minutes treatment sessions of CBTT

TC CAU DASS-21 39 outpatients from a spinal injuries rehabilitation center

Ludman et al. (2007) 16 weekly, 90 minutes treatment sessions of CBTT

TC CAU HSCL 393 outpatients from a psychology clinic

Strachan et al.(2011) 8 weekly sessions of CBTT VC FTF BDI 31 combat veterans

Simon et al.(2004) 17 weekly, 90 minutes treatment sessions of CBTT

TC CAU HSCL 348 outpatients from a primary care clinic

Moreno et al(2012) 6 monthly Webcam sessions with the psychiatrist

VC CAU MADR 132 adults with a diagnosis of depression from a community clinic

Heckman and Carlson (2007)

19 weekly, 90 minutes treatment sessions of CBTT

TC CAU BDI 215 HIV-positive individuals

Frunch et al(2007) 14 weekly, 90 minutes treatment sessions of CBTT

VC FTF BDI 21 combat veterans

Lovell et al(2006) 10 weekly sessions of CBTT TC FTF BDI 68 outpatients from a psychogical clinic

Mohr et (2012) 18 weekly sessions of CBTT TC FTF HAMD 293 outpatients from a primary care clinic

Napolitano et al. (2002) 8 weekly sessions of CBTT TC CAU GHQ(depression)

71 patients awaiting lung transplant

Nelson et al. (2003) 8 weekly sessions of CBTT VC FTF CDI 28 children age 8 to 14

Note: CBTT, cognitive-behavioral telehealth therapy; TC, teleconferencing; VC, videoconferencing; CAU, care as usual; FTF, face to face; DASS-21,

depression, anxiety and stress scale; HSCL, Hopkins symptoms checklist; BDI, beck depression inventory; MADR, Montgomery-Asberg Depression

Rating Scale; HAMD, Hamilton depression rating scale; GHQ, general health questionnaire; CDI, children’s depression inventory.

2

Appendix 2: Compute Effect Size and Standard Error for individual studies

Study Name 1n 1X 1S 2n

2X 2S g gSE 95% CI

Dorstyn et al.(2012)

20 3.44 8.23 19 4.48 5 -0.15 0.32 [-0.78,0.48]

Ludman et al. (2007)

198 0.68 0.55 195 0.85 0.65 -0.28 0.10 [-0.48,-0.08]

Strachan et al.(2011)

18 16.8 13.3 13 17.5 12.6 -0.05 0.38 [-0.77,0.66]

Simon et al.(2004)

172 0.69 0.5 176 0.93 0.68 -0.40 0.11 [-0.61,-0.19]

Moreno et al(2012)

64 9.52 10.73 68 13.46 12.57 -0.33 0.18 [-0.68,0.01]

Heckman and Carlson (2007)

108 20.23 6.96 107 20.73 6.83 -0.07 0.14 [-0.34,0.20]

Frueh et al (2007)

9 2.89 8 12 1.42 6.1 0.20 0.44 [-0.66,1.07]

Lovell et al(2006)

35 11.2 8 33 9.3 8.5 0.23 0.24 [-0.25,0.70]

Mohr et (2012)

141 12.51 15.69 152 13.58 14.59 -0.07 0.12 [-0.30,0.16]

Napolitano et al. (2002)

36 14.09 3.07 35 13.33 2.22 0.28 0.24 [-0.19,0.75]

Nelson et al. (2003)

14 6.71

4.78

14 11.65

11.63

-0.54 0.38 [-1.29,0.21]

3

Appendix 3: Summary Effect Size under Fixed effects model

Table 3.1: Summary Result

Effect Size (Standard Mean Difference) -0.1904

Standard Error 0.0497

Z value -3.8331

P value 0.0001

95% Confidence Interval [-0.2877,-0.0930]

Figure 3.1: Forest Plot under Fixed effects model

Study Weight ES 95%CI

4

Appendix 3: Summary Effect Size under Fixed effects model

Figure 3.2: Funnel Plot under Fixed effects model Figure 3.3: Galbraith Plot under Fixed effects model

Standard

Error

Z Statistic

1/SE

A funnel plot is a scatterplot of treatment effect against

a measure of study size. It is used primarily as a visual

aid for detecting bias or systematic heterogeneity

(Wikipedia).

A Galbraith plot is produced by first calculating the

standardized estimates or z-statistics by dividing

each estimate by its standard error (SE). The

Galbraith plot is then a scatter plot of each z-statistic

(vertical axis) against 1/SE (horizontal axis). It is

used to examine heterogeneity (Wikipedia).

Effect Size

5

Appendix 4: Summary Effect Size under Fixed effects model

Table 4.1: Summary Result

Effect Size (Standard Mean Difference) -0.1570

Standard Error 0.0717

Z value -2.1901

P value 0.0285

95% Confidence Interval [-0.2976,-0.0165] 2T ( estimated amount of total heterogeneity ) 0.0192

Degree of freedom 10

Q 15.6701 2I ( total heterogeneity / total variability ) 39.43%

Figure 4.1: Forest Plot under Random effects model

Study Weight ES 95%CI

6

Appendix 4: Summary Effect Size under Fixed effects model

Figure 4.2: Funnel Plot under random effects model Figure 4.3: Galbraith Plot under random effects model

Standard

Error

1/SE Effect Size

Z Statistic