lable at ScienceDirect

Chinese Nursing Research 4 (2017) 192e200

Contents lists avai

Chinese Nursing Research

journal submission page: http://nr.suo1.cn/site/index.php

Original article

Effectiveness of flipped classrooms in nursing education: Systematicreview and meta-analysis

Cui Tan a, *, Wei-Gang Yue b, Yu Fu a

a Sichuan Nursing Vocational College, Chengdu, Sichuan 610100, Chinab The First Affiliated Hospital of Lanzhou University, Lanzhou, Gansu 730000, China

a r t i c l e i n f o

Article history:Received 17 April 2017Received in revised form28 June 2017Accepted 5 August 2017Available online 14 October 2017

Keywords:Flipped classroomInverted classroomReversed classroomNursing educationSystematic reviewMeta-analysis

* Corresponding author.E-mail address: 340031132@qq.com (C. Tan).Peer review under responsibility of Shanxi Medic

https://doi.org/10.1016/j.cnre.2017.10.0062095-7718/© 2017 Shanxi Medical Periodical Press. Puorg/licenses/by-nc-nd/4.0/).

a b s t r a c t

Background: The flipped classroom has generated considerable interest in nursing education in the lastfew years, especially in higher education in China. However, research to date has been insufficient toconfirm the effectiveness of the flipped classroom approach. Since most of the Chinese studies arepublished in Chinese journals, they may not be easily accessible to international researchers. Therefore,this meta-analysis aimed to review the effectiveness of a flipped classroom in nursing education both inEnglish and Chinese databases.Objective: The aim of the present study was to identify the robust available evidence about the effec-tiveness of flipped classrooms in nursing education through a systematic review and meta-analysis.Methods: A systematic search of English databases, including PubMed, EMBASE, Science Director,CINAHL and Google Scholar, and Chinese databases, including Chinese National Knowledge Infrastructure(CNKI), WanFang Data, VIP Information and Chinese Biomedical Literature (CMB), were conducted toidentify peer-reviewed studies that met the inclusion criteria. Two reviewers independently performedstudy identification with Endnote X7 software and used Excel to extract data. The risk of bias in theincluded studies was assessed with Cochrane Collaboration's risk of bias tool, and a meta-analysis wasconducted using RevMan 5.3 software following the guidelines proposed by PRISMA reporting standardswith the protocol register No. CRD42016041826.Results: A total of 29 studies were included in systematic review. There was a significant post-intervention improvement in academic performance both in knowledge with a pooled random-effectsstandardized mean difference of 1.13 and skills with a pooled random-effects standardized mean dif-ference (SMD) of 1.68. Students' self-learning abilities were also improved with a pooled random-effectsSMD of 1.51 compared with traditional lecture. In the subgroup analysis, we found that the effect sizeshad high fidelity in terms of nursing degrees and research settings. There was a high rating of flippedclassroom pedagogy from teaching evaluations, study satisfaction, study attitude and improvement incritical thinking and problem-solving skills.Conclusions: These results indicated that a flipped classroom might help nursing students improve inknowledge, skills, attitudes, self-learning, study satisfaction, critical thinking and problem-solving skills.We recommend adopting this approach for nursing education reform. There is a call for robust empiricalresearch and unification of appraisal standards for further support of the effectiveness of the flippedclassroom. Furthermore, a meta-regression analysis is also recommended to explore the sources ofheterogeneity in included studies.© 2017 Shanxi Medical Periodical Press. Publishing services by Elsevier B.V. This is an open access article

under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

al Periodical Press.

blishing services by Elsevier B.V. Th

1. Introduction

There is a gap between nursing education and actual practicesuch that newly graduated nursing students struggle to meetworkplace requirements.1 To address this gap, there has been acall for a radical transformation in nursing education involvingeducators designing learning experiences that enhance education

is is an open access article under the CC BY-NC-ND license (http://creativecommons.

C. Tan et al. / Chinese Nursing Research 4 (2017) 192e200 193

quality and ensure that graduates are prepared for today's com-plex nursing practice at different health care facilities.1e3 Newinnovative teaching strategies are critical to integrate into nursingeducation. The literature tells us that one of the primary compo-nents of effective teaching is student engagement and thatengagement is critical for learning.4,5 Reflecting this change is therise in the flipped classroom that is an alternative pedagogy thatshifts from traditional lectures to active learning6 and would offeropportunities to integrate clinical experiences within theclassroom.

The flipped classroom is an instructional strategy that provides anew methodology and modality for teaching and learning. It in-volves a role change for instructors as a way of minimizing theamount of direct instruction in their teaching practice whilemaximizing one-to-one interaction and more cooperative andcollaborative contribution to the teaching process, which canimprove and encourage social interaction, teamwork and culturaldiversity among students. The roles of students have a corre-sponding change from passive participants to active participants.1

The modern flipped classroom began in 2007 in a high schoolchemistry course in Colorado.7 It could be argued that the flippedclassroom has been in existence within the broader educationalsphere for a number of years. Flipped classrooms take what waspreviously class content and replace it with what was previouslyhomework, which now takes place within the class.8 In addition,this approach fosters student ownership of learning through thecompletion of preparatory work and being more interactive duringactual class time. Proponents of the flipped class suggest that thispedagogy allows students to learn at their own pace, and they mayhave flexibility as to when they engage with electronic resources. Itfrees up actual class time for robust discussion and associatedproblem-solving activities, which could be initiated by the stu-dents, not the staff member. This pedagogy may help students toimprove self-efficacy through self studying, problem-solving andcommunication skills, which are skills that are recommended bythe Institute of Medicine report from 2010.2

Many studies have already reported that the flipped classroomhad a positive effect on education outcomes, such as acceleratingself-learning, improving academic performance,9,10 developingcritical thinking skills,11 and increasing positive feedback.9,12

However, some of the studies also showed no significant differ-ences in final exam scores between students undertaking tradi-tional lectures and those in the flipped classroom13,14 and even hadless satisfaction with the flipped class approach due to increasingthe amount of out-of-class preparation time.6,15 Meanwhile, stu-dents and staff are reluctant to abandon the lecture-basedapproach because it requires little active student participation.4

Despite acknowledgement of the positive attributes of the flip-ped classroom, few articles used a robust scientific method to verifyeducational outcomes as they relate to improved student learningand attitude.16,17 Rigorous experimental research on the flippedclassroom in higher education is very limited.14,18 Other nursescholars have conducted systematic reviews to address flippedclassroom models in nursing education and found that a meta-analysis was impossible due to the limited number and heteroge-neity of studies.19,20 Therefore, research to date is insufficient toconfirm the effectiveness of the flipped classroom approach.

The flipped classroom has generated considerable interest innursing education in recent years. Although experimental researchon the flipped classroom published in English is limited,14,18 therehas been a recent influx of empirical evidence on the flippedclassroom in Chinese nursing education.21 Since Chinese articlesmay not be easily accessible to international researchers due tolanguage barriers, this review aims to examine whether the flippedclassroom is more effective than the traditional lecture through

synthesizing both English and Chinese studies. The followingresearch questions are addressed in this paper:

What effects of the flipped classroom compared with thetraditional lecture have been reported in nursing education?

To what extent do the effects of flipped classrooms relate toknowledge, skills and attitudes compared with traditional lecturesin nursing education?

What is the rating of self-report questionnaires as to the effec-tiveness of the flipped classroom?

2. Methods

We conducted this meta-analysis according to the guidelinesproposed by the Preferred Reporting Items for Systematic Reviewsand Meta-Analyses (PRISMA) standards of quality22 for reportingthis review.

2.1. Protocol and registration

This protocol was registered in PRISMA with register No.CRD42016041826 and is available from http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID¼CRD42016041826.

2.2. Inclusion and exclusion criteria

Studies were selected for inclusion if the study had (a) partici-pants were in a higher education program including associate de-gree students, undergraduates, and postgraduates, (b) arandomized controlled trial design, (c) a flipped classroomapproach in the intervention group, (d) traditional lecturesapproach in the control group, (e) measurable outcome indexes,and (f) publication in Chinese and English. Exclusion criteriaincluded that the study (a) was conducted with students in disci-plines other than nursing or register nurses, (b) was a non-randomized controlled trial, (c) had an intervention that utilizedflipped classroom and at the same time accepted other teachingmethods, or (d) had non-extractable outcome data.

2.3. Data sources and searches

The English databases searched included PubMed, EMBASE,Science Director, CINAHL, Google scholar, and the Chinese data-bases searched included CNKI, WanFang Data, VIP and CMB. Weidentified peer-reviewed studies in accordance with the inclusioncriteria from inception to November 2016. To identify as manyeligible studies as possible, we broadened search terms andstrategies. Search terms were modified together with informaticsand combined with Boolean operators as follows: ‘nurs*’ AND(‘flipp* classroom’ OR ‘invert* classroom’ OR ‘revers* classroom’).The reference lists of the eligible articles included after the elec-tronic search were also manually searched. Two researcherssearched the databases together and independently selected theeligible articles. Articles with the same authors were included aslong as the research participants were from different areas. Anydiscrepancies were resolved through discussion with the thirdresearcher.

2.4. Data extraction process and quality assessment

Data extraction and article quality assessment were performedby two researchers independently. Disagreements were resolved byconsensus. The following information from the studies wasextracted based on a predesigned coding manual form: authors andyears of publication, sample size, educational degrees, subjects,withdrawal rate, and intervention and outcome measures.

C. Tan et al. / Chinese Nursing Research 4 (2017) 192e200194

The quality of the included studies was based on the six generalsources of bias described in the Cochrane Handbook for SystematicReviews of Interventions. The following quality items werechecked: adequacy of the generation of the allocation sequence,concealment of allocation, blinding procedures, incompleteoutcome data, selective outcome reporting, and other sources ofbias. These items were obtained from the published reports, andthe authors were contacted if additional information was required.Themethodological quality of these domains was assessed by usingthe following scoring: (1) ‘low risk of bias’, when plausible biasunlikely alters the results, (2) ‘unclear risk of bias’ when plausiblebias raises some doubt about the results and (3) ‘high risk of bias’when plausible bias seriously weakens confidence in the results.23

2.5. Statistical analyses

Review Manager 5.3 software24 was used to test the data forheterogeneity and to carry out the meta-analysis. As continuousdata from different scales were extracted, the standardized meandifference (SMD) was calculated for effect size based on samplesize25 and 95% confidence intervals for each study, and for thepooled studies using variance analysis. Weighted mean difference(WMD) and 95% confidence intervals (CI) were calculated forcontinuous data from the same scale. A two-sided P-value of lessthan 0.05 was regarded as significant for all analyses. There weretwo meta-analysis models. A fixed-effects model was used to pooldata if there was no heterogeneity, and otherwise we used arandom-effects model. Heterogeneity was considered significantfor a P-value of Cochran's Q statistic <0.1 or an I2 > 50%.26 I2 is thepercentage of variation attributed to heterogeneity and is easilyinterpreted. An I2 statistic of 25%e50% was considered low, 50%e75% was considered moderate, and �75% was considered high. Ifheterogeneity was found, we conducted a sensitivity analysis toassess if this heterogeneity significantly altered the results of the

Fig. 1. Flow diagram o

meta-analysis. We performed the sensitivity analysis by excludingstudies one by one, and then recalculating the pooled estimates forthe remaining studies. However, this process did not significantlyalter the results. Subgroup analyses were planned according to pre-set variables, including educational degree and research setting.Publication bias was assessed through funnel plots, and funnel plotshapes did not reveal obvious evidence of asymmetry.

3. Results

3.1. Search results

The search yielded 497 relevant studies that were downloadedinto EndNote X7. After the removal of duplicates, the 244 residualreferences were reviewed based on the title and abstract. Then, 73full text studies were assessed for eligibility. Subsequently, a total of29 eligible studies were included in the meta-analysis. Fig. 1 pro-vided a flow chart illustrating our search strategy and output.

3.2. Characteristics of included studies

Twenty-nine articles9,10,12,27e52 were included in thisreview. They were all Chinese articles published from 2014 to 2016.Sixteen studies measured theoretical examinationscores,10,30,31,36,38e40,43e50,52 and sixteen studies measured skillexamination scores.9,10,12,31,32,34e37,39,41,43,46,48,51,52 Fifteen studiesinvestigated the students' self-learning through question-naires.27e29,31,33e36,38e41,43,45,47 One study42 was not included in themeta-analysis because its outcome indexes were investigated onlyby questionnaires. The sample sizes ranged from 36 to 320 par-ticipants, and the pooled sample size was 3694 (experimentalgroup ¼ 1896, control group ¼ 1798). Fifteen studies were con-ducted in undergraduates, and 14 studies were conducted inassociated degree students. Research settings included 25 schools

f included studies.

C. Tan et al. / Chinese Nursing Research 4 (2017) 192e200 195

(86.2%) and 4 clinical areas (13.8%). Table 1 shows the character-istics of the 29 included studies.

3.3. Study quality

Fig. 2 shows the risk of bias assessment of the 29 includedstudies. The majority of the studies had a low risk of bias across thesix domains. Most of the studies reported cluster randomizedsampling or just mentioned it used “randomized” for allocationconcealment without details. One study33 adequately generatedrandom numbers. Another two studies were separately generatedby odd and even numbers based on the college entrance exami-nation scores37 and student ID numbers.49 The allocation sequenceof the two studies30,34 was based on the preferences of theresearchers.

Two studies10,33 reported using blind implementation, butbecause of the nature of the given intervention, it was not feasibleto blind the students and tutors. Therefore, performance bias wasthe only bias category causing high risk in nearly all studiesincluded. One study reported missing data with reason,33 theothers reported complete outcome data.We assessedwhether eachstudy was free from selective outcome reporting by checkingwhether all outcomes mentioned were adequately reported in theresults section. All studies adequately reported the results. Allstudies seemed free from “other sources of bias” as defined in theCochrane Collaboration's domain-based evaluation.

3.4. The effectiveness of the flipped classroom in nursing education

The effects of the flipped classroom were measured by scoresaccording to the teacher-constructed examination or self-report

Table 1Characteristics of included studies.

Study (year) Sample size (T/C) Education degrees Subject

Chen et al 201652 51/51 Associated degree students Medical nursChen 201650 64/51 Associated degree students PhysiologyDeng 201647 34/32 Undergraduate Nursing instrFeng et al 201528 50/49 Associated degree students Obstetric nurGao 201610 72/68 Associated degree students Nursing techGuo et al 201648 40/40 Associated degree internship Clinical areaHu 201543 55/55 Associated degree students Obstetric nurJi et al 201641 120/122 Undergraduate Medical nursKou et al 201636 30/30 Undergraduate internship Clinical areaLiu 201630 53/51 Undergraduate Nursing theoLiu et al 201629 70/70 Undergraduate InterventionaLiu et al 201631 153/146 Higher vocational internship Nursing techLu 201651 98/96 Associated degree students Nursing founLuo 201642 64/61 Associated degree students Medical nursSong et al 210634 30/30 Undergraduate Nursing techTao et al 201612 32/30 Undergraduate Medical nursTian 201639 193/129 Undergraduate Nursing techTian et al 201540 32/32 Undergraduate GerontologyXie 201544 50/50 Associated degree students Pediatric nurYan et al 20159 55/54 Associated degree students Nursing techYang 201638 64/64 Undergraduate Preventive mYe et al 201633 43/42 Undergraduate internship Clinical areaYe et al 201632 53/52 Undergraduate Nursing techYin 201627 67/64 Associated degree students N/AZhang 201637 18/18 Undergraduate internship Clinical areaZhao et al 201646 115/114 Undergraduate Nursing founZhao 201645 57/57 Associated degree students Health assessZhong et al 201649 36/36 Undergraduate Medical nursZong et al 201635 101/100 Associated degree students Nursing foun

Note: T: flipped classroom pedagogy; C: traditional lecture pedagogy; N/A: Non withdrawscores; c: self-leaning questionnaire; d: teaching evaluation questionnaire; e: satisfactionproblem solving questionnaire.

questionnaires. Major outcome indexes were (a) theoretical ex-amination scores, (b) skills examination scores, (c) self-learningquestionnaires, and (d) questionnaires used for teaching evalua-tion, student satisfaction, student attitudes, critical thinking andproblem-solving skills.

3.4.1. The effectiveness of interventions on knowledgeWe used the theoretical examination scores to check the stu-

dents' academic performance in knowledge. Sixteen studies testedthe theoretical examination scores with pooled 2103 participants(experimental group ¼ 1099, control group ¼ 1004). Thirteenstudies showed statistically significant differences between theflipped classroom and traditional lecture. There was high hetero-geneity (I2 ¼ 93%, P < 0.00001). Thus, a random-effects model wasused. The pooled effect size showed a significant difference intheoretical examination scores [SMD ¼ 1.13, 95% CI (0.76, 1.49),P < 0.00001] in favor of flipped classrooms compared with tradi-tional lectures (Fig. 3). The results indicated that the flippedclassroom improved students' academic performance in knowl-edge. Due to the high heterogeneity among studies, we used sub-group analysis by education level. There was no significantdifference in the subgroups (P ¼ 0.85, I2 ¼ 0%), which means thatthe heterogeneity was from intra-group differences rather thaninter-group differences. A sensitivity analysis was carried out toverify the reliability of the results. This analysis was performedusing sequential omission of individual studies one by one, and thepooled effect size that favored the flipped classroom group did notchange the effects observed in the initial analysis. The funnel plot ofthe 16 studies with theoretical examination scores is shown inFig. 4. The shape of the funnel plot is symmetrical indicating nosignificant publication bias.

Withdraw rate Intervention Outcome measures

T C

ing N/A FC LBC a, bN/A FC LBC a, d

uction N/A FC LBC a, c, f, gsing N/A FC LBC c, enology N/A FC LBC a, b, c, f, h

N/A FC LBC a, b, esing N/A FC LBC a, b, cing technology N/A FC LBC b, d

N/A FC LBC a, b, cry N/A FC LBC a, dl therapy N/A FC LBC a, c, enology N/A FC LBC a, b, cdation N/A FC LBC bing N/A FC LBC e, gnology N/A FC LBC b, c, eing & nursing foundation N/A FC LBC b, fnology N/A FC LBC a, b, c, dnursing N/A FC LBC a, csing N/A FC LBC a, gnology N/A FC LBC b, eedicine N/A FC LBC a, c, d

1 FC LBC cnology N/A FC LBC b, d

N/A FC LBC a, c, e, f, hN/A FC LBC b

dation N/A FC LBC a, b, dment N/A FC LBC a, c, eing technology N/A FC LBC a, ddation N/A FC LBC b, c

al; FC: flipped classroom; LBC: lecture-based classroom; a: theoretical scores; b: skillquestionnaire; f: study attitude questionnaire; g: critical thinking questionnaires; h:

Fig. 2. Risk of bias.

Fig. 3. Forest plot in theoretical examination scores.

Fig. 4. Funnel plot in theoretical examination scores.

C. Tan et al. / Chinese Nursing Research 4 (2017) 192e200196

C. Tan et al. / Chinese Nursing Research 4 (2017) 192e200 197

3.4.2. The effectiveness of interventions on skillsWe used the skills scores to check the students' academic per-

formance in skills. Sixteen studies tested the skill examinationscores with a pooled 2349 participants (experimentalgroup ¼ 1216, control group ¼ 1133). Thirteen studies showedstatistically significant differences between the flipped classroomand the control groups in skills scores. There was high heteroge-neity (I2 ¼ 95%, P < 0.00001). Thus, a random-effects model wasused. The pooled effect size showed a significant difference in skillsscores [SMD ¼ 1.68, 95% CI (1.24, 2.12), P < 0.00001] in favor of aflipped classroom compared with traditional lectures. The resultsindicated that the flipped classroom improves students' academicperformance in skills with an effect size of 1.68. Subgroups by ed-ucation level and research setting were analyzed (Fig. 5), and theresults showed that there was small inter-group heterogeneity(I2 ¼ 11.9%). Thus, the majority of heterogeneity were still fromintra-group variability. A sensitivity analysis was performed usingsequential omission of individual studies. After excluding onestudy,21 the heterogeneity was obviously diminished (I2¼66%) inundergraduates, but the pooled effect size favored the flippedclassroom [SMD¼ 1.65, 95% CI (1.41, 1.89), P < 0.00001] and did notchange the effects observed in the initial analysis. The funnel plotfor the 16 studies in the skill examination score analysis is shown inFig. 6. This shape was mostly symmetrical indicating no significantpublication bias.

3.4.3. The effectiveness of interventions on self-learning skillFifteen of the articles tested the self-learning skill of students

with pooled 2119 participants (experimental group ¼ 1099, con-trol group ¼ 1020). The effect size of 1.51 [95% CI (1.02, 2.01),P < 0.00001] means that students agree that flipped classroomcan improve their self-learning skill. There is a relatively smallheterogeneity existing according to the evaluation instrumentsthrough subgroup analysis, however, it is not the main heteroge-neity sources (Fig. 7). The funnel plot for the 15 studies on theself-learning questionnaires analysis is shown in Fig. 8. The shapeof the funnel plot is symmetrical indicating no significant publi-cation bias.

Fig. 5. Forest plot in skill

3.4.4. The effectiveness of interventions assessed by questionnairesOverall, nineteen articles investigated the students' evaluations

of the flipped classroom pedagogy through self-reported ques-tionnaires in terms of teaching effectiveness,30,32,38,39,41,46,49,50

student satisfaction,9,27e29,34,42,45,48 student study atti-tudes,10,12,27,47 critical thinking skills,42,44,47 and problem-solvingskills.10,27 According to the questionnaire results, most students atthe flipped classroom had positive comments about the flippedclassroom. At least 90.4% of reported studies indicated an aboveaverage rating for students' study enthusiasm in the flipped class-room approach, and 93.8% of students selected above average thatflipped classrooms accelerate the communication between facultyand students. In all, 87% of the students reported that theypreferred this approach and wanted to continue this approach inother subjects. There was also a statistically significant improve-ment in student satisfaction,29,34,48 student interest,27,47 and clin-ical thinking38,42 in the flipped classroom compared with thetraditional lecture.

4. Discussion

At first glance, evidence in favor of the flipped classroom seemsstrong because all studies indicated that flipped classroom studentsoutperformed those in traditional lecture classrooms, especially inimproving students' academic performance both in knowledge andskills and self-learning ability. These results are in accordance withthe findings of the previous narrative review related to foreignnursing education.6,16,20 In the evaluation of teaching satisfactionand study attitude, these results also showed a relatively higherrating for the flipped classroom compared with the traditionallecture, which was contrary to someWestern studies6,53 that foundimproved examination scores and diminished student satisfaction.Several possible reasons may explain why Chinese students preferflipped classrooms. First, usually, in traditional Chinese nursingeducation, students are not allowed to use cellphones and com-puters during the lecture. Second, the traditional lecture is quiterepressed since most students are not allowed to discuss theiropinions and always remain quiet. Third, students have few op-portunities to communicate with the faculty after class, and the

examination scores.

Fig. 6. Funnel plot in skill examination scores.

Fig. 7. Forest plot in self-learning questionnaire.

C. Tan et al. / Chinese Nursing Research 4 (2017) 192e200198

flipped classroom may let students feel the faculty are moreapproachable. However, compared with the different culture inWestern higher education, their traditional education has alreadyadopted electronic equipment and encouraged them to share theiropinions freely in the class, so they may not be curious about thispedagogy style, which may be why it received a low satisfactionrating due to the workload for class preparation.

Overall, in the pool of included studies, there was obvious het-erogeneity, which may have resulted from the following factors.First, the different educational levels of the students resulted in thecriteria for admission of nursing students being different. Second,the pooled studies focused on different nursing subjects. The thirdreason concerns the examination methods. Twenty-nine studiesemployed different appraisal tools. There were no unified criteriafor the evaluation of the effectiveness of the flipped classroom.Fourth, each study employed a diverse range of preparatory and in-class active learning strategies as part of the flipped classroom

intervention. These different teaching methods and objectives mayhave resulted in different effects, and this diversitymay be themaincause of the heterogeneity among those studies.

Initially, the researcher was planning to synthesize the out-comes of questionnaires quantitatively. However, most of thestudies used self-constructed questionnaires with a post-test andlacked comparison with traditional lectures. Meanwhile, the typesof measurable data are different. Thus, the synthesis was notfeasible. The researcher just described the aforementioned aspectsrelated questionnaire surveys. However, most studies showedpositive results for the flipped classroom.

5. Strengthens and limitations

In this review, we used a statistical method to validate theeffectiveness of the flipped classroom in nursing education since allof the review papers before were narratives. It provided relatively

Fig. 8. Funnel plot on self-learning questionnaires.

C. Tan et al. / Chinese Nursing Research 4 (2017) 192e200 199

robust evidence to address the concerns of prior researchers.Meanwhile, this review was reproducible in terms of data extrac-tion and synthesis. In addition, we registered in PRISMA with aprotocol beforewe conducted this review, which decreased the biasin this review. In terms of limitations, no studies published in En-glish met the inclusion criteria, and some of the Chinese studieswere of poor quality. Fortunately, the sensitivity analyses showedthat the outcomes of the study were reasonably stable.

As a result, the effectiveness of the flipped classroom was pos-itive. However, two studies in the final pool that found no signifi-cant differences were excluded due to insufficient data. Therefore,we suppose that the extent of effectiveness may have diminished ifwe included them. On the other hand, in the measurable outcomeindexes, knowledge can be objectively evaluated by theoreticalexamination, but the appraisal of skills and self-reported ques-tionnaires are somehow subjective. Most studies did not reportwhether the implementation was blind or the surveys wereanonymous, which is an important caveat when addressing surveyresults.

For future researchers, we recommend strengthening theinstructional framework of flipped classroom pedagogy andformulating unified evaluation criteria. Additionally, there is a needfor rigorous randomized controlled trial (RCT) designs for futurestudies to reduce the heterogeneity of the included articles.We alsosuggest that future reviewers perform meta-analyses to find sour-ces of heterogeneity in this area.

6. Conclusions

This review indicated that the flipped classroom might helpnursing students improve in knowledge, skills, self-learning, studysatisfaction and enthusiasm. However, students' critical thinking,problem-solving, and decision-making skills should be furthervalidated by robust studies. Flipped classroom pedagogy is rec-ommended, especially in Chinese nursing education reform. Sinceall of the included articles were in Chinese, the review highlightsthe lack of evidence about flipped classrooms in Western nursingeducation. Compared with the outcomes of Western studies of the

effectiveness of the flipped classroom, Chinese studies showedmore positive feedback.

Most importantly, to explore the rationale behind flippedclassroom models, factors that will influence student performancealso need to be included. Nursing educators could then translatethese models in more strategic ways to transform the teachingelearning process.

Conflicts of interest

The authors declare that there are no conflicts of interest withregard to the present work. This research received no specific grantfrom any funding agency in the public, commercial or not-for-profitsectors.

Acknowledgements

The authors would like to thank Dr. Brian S. Bantugan, who gaveguidance and always opened the door whenever I had questionsabout this study. We would also like to thank Ma'am Xiao Li, whoprovided the electronic libraries resources for the literature searchand gave her suggestions about search strategies.

The authors would like to express gratitude to all who havegiven generously of their time to edit this paper.

References

1. Benner P, Sutphen M, Leonard V, Day L. Educating Nurses: A Call for RadicalTransformation. San Francisco, CA: Jossey-Bass; 2009.

2. IOM (Institute of Medicine). The Future of Nursing: Focus on Nursing Education.Washing ton, DC: The National Academic Press; 2010.

3. Yacout DAESA, Shosha AAEFA. Nursing students' perceptions towards flippedclassroom educational strategy. J Am Sci. 2016;12:62e75.

4. Barkley EF. Student Engagement Techniques: A Handbook for College Faculty. SanFrancisco, CA: Jossey-Bass; 2009.

5. Coates H. Student Engagement in Campus-based and Online Education: UniversityConnections. New York: Routledge; 2006.

6. Missildine K, Fountain R, Summers L, Gosselin K. Flipping the classroom toimprove student performance and satisfaction. J Nurs Educ. 2013;52:597e599.

C. Tan et al. / Chinese Nursing Research 4 (2017) 192e200200

7. Bergmann J, Aaron S. Remixing chemistry class: two Colorado teachers makepodcasts of their lectures to free up class time for hands-on activities. LearnLeading Technol. 2009;36:22e27.

8. Pierce R, Fox J. Vodcasts and active-learning exercises in a “flipped classroom”

model of a renal pharmacotherapy module. Am J Pharm Educ. 2012;76:196.9. Yan LX, Xie HY. Application of flipped classroom mode in comprehensive

nursing technology. J Nurs Train. 2015;7:626e628 (in Chinese).10. Gao Y. Flipped teaching with mobile teaching platform for fundamental

nursing skill. J Nurs. 2016;8:13e15 (in Chinese).11. Horn MB. The transformational potential of flipped classrooms. Educ Next.

2013;13:78e79.12. Tao WW, Liu YN, Ding SZ, Sha LY, Liu Y. Effect of flip classroom and micro-

lectures on educational environment and academic emotions of nursing stu-dents. Chin Nurs Manage. 2016;16:1221e1225 (in Chinese).

13. Geist MJ, Larimore D, Rawiszer H, Al S, Abdellatif W. Flipped versus traditionalinstruction and achievement in a baccalaureate nursing pharmacology course.Nurs Educ Perspect. 2015;36:114e115.

14. Harrington SA, Bosch MV, Schoofs N, Beel-Bates C, Anderson K. Quantitativeoutcomes for nursing students in a flipped classroom. Nurs Educ Perspect.2015;36:179e181.

15. Simpson V, Richards E. Flipping the classroom to teach population health:increasing the relevance. Nurse Educ Pract. 2015;15:162e167.

16. O'Flaherty J, Phillips C. The use of flipped classrooms in higher education: ascoping review. Int Higher Educ. 2015;25:85e95.

17. Bernard JS. Nurse Educators' Transition to Flipped Classroom: An InterpretiveDescription Study. East Tennessee State University; 2015. Electronic Theses andDissertations. Paper 2603 http://dc.etsu.edu/etd/2603.

18. Heyborne W. To flip or not to flip? Analysis of a flipped classroom pedagogy ina general biology course. J Coll Sci Teach. 2016;45:31.

19. Betihavas V, Bridgman H, Kornhaber R, Cross M. The evidence for ‘flipping out’:a systematic review of the flipped classroom in nursing education. Nurse EducToday. 2016;38:15e21.

20. Presti CR. The flipped learning approach in nursing education: a literaturereview. J Nurs Educ. 2016;55:252e257.

21. Kang DQ, Yue SJ. Bibliometric analysis on application of flipped classroom innursing teaching journal of nursing. J Nurs. 2016;23:22e25 (in Chinese).

22. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reportingitems for systematic reviews and meta-analyses: the PRISMA statement. PLoSMed. 2009;6, e1000097.

23. Higgins JPT, Green S, eds. Cochrane handbook for Systematic Reviews of In-terventions. vol. 4. John Wiley & Sons; 2011.

24. Cochrane Collaboration. Review Manager (RevMan). Copenhagen: The NordicCochrane Centre; 2014. Version 5.2.3.

25. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale,NJ: Lawrence Erlbaum Associates; 1988:20e26.

26. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency inmeta-analyses. BMJ. 2003;327:557e560.

27. Yin WQ. The application value of flipped classroom in nursing education. NewCurriculum Res. 2016;3:83e85 (in Chinese).

28. Feng R, Xu H, He SZ, Zhang YH. The implementation and evaluation of flippedclassroom in gynecologic and obstetric nursing. J Nurs. 2015;22:15e17 (inChinese).

29. Liu MX, Peng GH, Kong Y. Practice of flipped classroom model in teaching ofnursing for interventional therapy. Hainan Med. 2016;27:1013e1016 (inChinese).

30. Liu F. Application of flipped classroom teaching mode based on micro lecture inteaching of “introduction of nursing science”. Chin Nurs Res. 2016;30:3406e3408 (in Chinese).

31. Liu P, Xue M, Ma LD, LI J. The effect of flipping classroom teaching in improvingthe self-directed learning ability of higher vocational nursing students. Chin JNurs Educ. 2016;13:109e112 (in Chinese).

32. Ye YS, Zhao J, Zhang ZX, Ye L, Wang WJ, Kong LL. The application of students-based flipping teaching in Fundamental Nursing technology. J Clin Nurs.2016;1:69e71 (in Chinese).

33. Ye YS, Zhao J, Bao CY, Ye L, WangWJ, Xie YQ. Influence of student-based flippedteaching on clinical practical performance of nursing undergraduates. J Nurs.2016;1:16e19 (in Chinese).

34. Song YL, Liu XT, Liu W, et al. Application of case method-based flipped class-room in practice teaching of fundamental of nursing and its evaluation. J Nurs.2016;9:8e11 (in Chinese).

35. Zong MZ, Cong RZ, Li FJ, Liu YH. The practice of micro lecture and flippedclassroom in fundamental nursing. World Latest Med Inf (Electro Version).2016;3:198 (in Chinese).

36. Kou D, Gan H, Chen QH, He R, Li Y. The application of flipped classroom inintern nursing students. Tianjin J Nurs. 2016;4:338e339 (in Chinese).

37. Zhang XH. The evaluation of effectiveness of flipped teaching mode in clinicalteaching of undergraduates nursing students. Curr Nurs. 2016;6:131e133 (inChinese).

38. Yang SG. Evaluation of the flipped classroom model applied in preventivemedicine teaching. China Med Educ Tech. 2016;5:577e580 (in Chinese).

39. Tian ZJ. Application of Flipped Instruction in the Course of Fundamental Nursing.Taiyuan: Shanxi Medical University; 2016.

40. Tian ZJ, Jin RH, Liu CF. Application of flipped instruction in the course ofgeriatric nursing. J Chin Nurs Educ. 2015;12:333e335 (in Chinese).

41. Ji HX, Liang A, Zhang HY. Application of flipped classroom mode in surgicalnursing technology. Mod Vocational Educ. 2016;14, 152e152 (in Chinese).

42. Luo LR. A study of flipped classroom in critical thing and study satisfaction ofassociated degree nursing students. Health Vocational Educ. 2016;34:106e108(in Chinese).

43. Hu HF. Flipped classroom in gynecologic and obstetric nursing. Time Educ.2015;20:86 (in Chinese).

44. Xie ZM, Li YJ. The effective analysis of flipped classroom combine with microvideo mode in strengthening critical thinking of nursing students. J Exam.2015;47:171e174 (in Chinese).

45. Zhao MY. Application of micro lecture and flipped classroom in healthassessment of nursing students. Health Vocational Educ. 2016;34:83e85 (inChinese).

46. Zhao H, Yao PB, Liu YH, Wu MC. Application of micro lecture-based and flippedclassroom mode in fundamental nursing. J Snake. 2016;28:108e109 (inChinese).

47. Deng WF. Application of flipped classroom teaching mode based onconstructive theory in the course of introduction of nursing. Chin J Prac Nurs.2016;32:1429e1433 (in Chinese).

48. Guo W, Yu Y, Liu JL, Gai QM, Sang Y. Application of flipped classroom mode invascular surgical teaching of nursing students. Today Nurse. 2016;11:162e163(in Chinese).

49. Zhong ML, Jiang DY, Wei QM, Jiang ZX, Chen SZ, Hu RJ. The effects of flippedclassroom in primary life support education. Chin J Nurs Educ. 2016;13:273e276 (in Chinese).

50. Chen W. Application of micro lecture based flipped classroom in physiologylaboratory class of associated degree nursing students. J Med Theor Pract.2016;29:1255e1256 (in Chinese).

51. Lu MY. The usefulness of flipped classroom in fundamental nursing of highervocational education. Nurs Pract Res. 2016;20:129e130 (in Chinese).

52. Chen L, Cao XP, Yuan AD, Luo Y, Liang SY. A study of flipped classroom modelon medical nursing of higher vocational education. Chin Gen Pract Nurs.2016;12:1277e1278 (in Chinese).

53. Della-Ratta CB. Flipping the classroom with team-based learning in under-graduate nursing education. Nurse Educ. 2015;40:71e74.

How to cite this article: Tan C, Yue W-G, Fu Y. Effectiveness offlipped classrooms in nursing education: Systematic review andmeta-analysis. Chin Nurs Res. 2017;4:192e200. https://doi.org/10.1016/j.cnre.2017.10.006