Embed Size (px)
Citation preview
Efficacy of Online Laboratory Science Courses
Rebecca J. Rowe1 & Lori Koban2& Amy J. Davidoff1 & Kathryn H. Thompson1
# Association for Educational Communications & Technology 2017
AbstractThe overall goal of this study was to evaluate student experiences in online laboratory courses in order to inform the design andimprovement of lab activities in a distance education program. Students were surveyed about their satisfaction and perceptions ofusability and learning in both hands-on (at home) and computer-based simulation (virtual) labs in a variety of natural sciencecourses. We also attempted to evaluate the effectiveness of several online chemistry courses taught with either hands-on kits orvirtual laboratory activities, and examined the performance of students concurrently enrolled in lecture and laboratory chemistrycourses versus those enrolled in a lecture only courses. The majority of survey respondents felt their online laboratory experiencewas the same as or better than their prior experiences in the traditional setting. Survey data also show that students believe theirlaboratory experiences reinforced and improved their understanding of concepts presented in lectures and the textbook, and thusmay have helped them perform better on course assessments. Our data on performance suggest that students enrolled in onlinescience courses do as well or better than their peers enrolled in traditional courses. The data also suggest that students who takelecture and laboratory concurrently outperform their lecture-only peers, independent of course (i.e., general or organic chemistry)or delivery method (i.e., online or traditional).
Keywords Online learning . Hands-on laboratory . Virtual laboratory . Inquiry-based . Student performance . Student perception
Introduction
Decades of research and practice on teaching and learning sci-ence have provided educators with a framework based on inqui-ry—learning science by engaging in the scientific process
(Bransford et al. 2000; National Research Council 1996, 2000,2005). This includes laboratory experiences that provide stu-dents with the opportunity for hands-on manipulation of mate-rials and equipment to learn the techniques (practical skills) ofacquiring data, as well as the methods of experimental design,analysis/evaluation and problem solving, and the broader out-comes of reinforcing conceptual knowledge and improving sci-entific communication skills (National Research Council 2000;NSTA 2007). Goals for laboratory experiences have been pub-lished for the disciplines of biology, chemistry, and physics(American Association of Physics Teachers 1998; AmericanChemical Society 2011; Dikmenli 2007; Woodfield et al.2005), and there is agreement among these groups that face-to-face laboratory experiences reinforce the topics and conceptsintroduced in the lecture and foster critical thinking skills. Bruckand Towns (2013), in their national survey of undergraduatechemistry faculty, reported that both general and organic chem-istry teachers believe that laboratory activities support the lecturecontent. Students concurrently enrolled in lecture and laboratorysections tend to have better learning outcomes and retentionwhen compared to their lecture-only peers (Matz et al. 2012).
With the rapid and extensive adoption of online education,research has begun to explore the nature and effectiveness of
Electronic supplementary material The online version of this article(https://doi.org/10.1007/s41686-017-0014-0) contains supplementarymaterial, which is available to authorized users.
* Rebecca J. [email protected]
Lori [email protected]
Amy J. [email protected]
Kathryn H. [email protected]
1 University of New England, 11 Hills Beach Road,Biddeford, ME 04005, USA
2 University of Maine Farmington, 111 South Street,Farmington, ME 04938, USA
Journal of Formative Design in Learninghttps://doi.org/10.1007/s41686-017-0014-0
teaching and learning in the digital world. Several reports haveappeared in the literature providing evidence that online edu-cation engages students (Chen et al. 2010) and provides anenvironment that promotes student learning (Allen andSeaman 2010; Means et al. 2010). Adoption of the onlineformat to teach laboratory science courses has been morechallenging because of the perceived need to provide anequivalent laboratory experience. However, the potentialcost-savings of online learning is affecting nearly every scien-tific field, despite the view that traditional lab experiences areneeded to provide the practical skills to conduct advanced(i.e., graduate level and professional) research (Rivera 2014;Waldrop 2013). And others argue that many students in thetraditional laboratory setting come to lab unprepared, and thencomplete the exercise/experiment without ever really thinkingabout what they are learning (Josephsen and Kristensen 2006;Woodfield and Catlin 2004).
Alternative laboratory experiences have been used by someinstructors as a supplement in conjunction with face-to-facelaboratories (e.g., Makransky et al. 2016; Woodfield et al.2005). The laboratory component can be either a Bvirtuallaboratory^ using computer-based simulations or a Bhands-onlaboratory^ where the students perform actual experiments out-side the classroom setting (at home). For example, online sim-ulations have been used to familiarize students with their chem-istry labs before they come to the physical laboratory to performexperiments (Dalgarno et al. 2009). Delivery of a distance ed-ucation laboratory component through these modalities (virtualor at-home) has several advantages, which include essentiallyunlimited access and the ability to repeat the experiments(Conway-Klaassen et al. 2012). Studies of online labs usedmainly as supplemental instruction have provided evidence thatthese may be helpful for increasing lecture test scores, improv-ing students’ attitudes and preparedness for the hands-on lab,and strengthening conceptual knowledge (Dalgarno et al. 2009;Liu 2006; Woodfield and Catlin 2004, 2005).
Virtual laboratory experiences can range from simple videosand games to graphing and 2D simulations to interactive 3Dvirtual reality experiences. Simulations, as mathematicalmodels of processes in the physical world, allow users to ma-nipulate parameters and can be used by faculty to customizelaboratories in various disciplines—e.g., virtual dissections,chemical reactions, laws of motion experiments (Welsch2015). In addition to the advantages of cost savings, improvedsafety, and lower environmental impact (i.e., no hazardouswaste disposal), virtual chemistry labs can help students visu-alize structures and processes at the molecular level (Johnstone1982) and allow types of experiments not possible in a standardundergraduate laboratory—e.g., quantum chemistry(Woodfield and Catlin 2004, 2005). Most studies evaluatingsimulations have focused on conceptual understanding, andthe evidence is promising for advancing this goal (Honey andHilton 2011). Students using SimuLab in an inorganic
chemistry course found it to be a motivating tool that enhancedtheir skills and helped them see the practical application of theirknowledge (Josephsen and Kristensen 2006). A recent studymodeled the impact of 3D virtual reality-based instruction in anintroductory chemistry course and found this to be effective inenhancing spatial ability and self-efficacy, with positive im-pacts on achievement (Merchant et al. 2012).
Distance education has employed hands-on labs that can beexperienced at home. These labs are favored for teaching manip-ulative, practical skills and exposing students to open-ended sit-uations that foster inquiry and design skills (Ma and Nickerson2006). Hands-on kits available from multiple vendors can pro-vide students with standards-based experiments while providinguniversities advantages in terms of cost and safety, with manu-facturers usually assuming liability (Welsch 2015).
Attempts to compare the effectiveness of these alternatives,especially with to face-to-face laboratory experiences, have beencomplicated, in part because studies have not used standardizededucational objectives as the criteria for measuring success, andin part because of small sample sizes (Ma and Nickerson 2006).Nevertheless, a number of studies in various disciplines havereported results suggesting that a virtual or at-home laboratorycan be comparable to the traditional laboratory in terms of learn-ing outcomes (Casanova et al. 2006; Dobson 2009; Feig 2010).Brinson (2015) completed a comprehensive review of empiricalstudies comparing the effectiveness of traditional and non-traditional (online) labs, using a model he developed for catego-rizing learning outcomes. While learning outcomes and evalua-tion tools varied considerably among studies, a majority of thesesuggested that non-traditional labs are equally or more effectiverelative to traditional labs (in all outcome categories, indepen-dent of evaluation instrument), with conceptual knowledge be-ing the most frequently used measure of effectiveness.
In the present study, wewere interested in evaluating whichfeatures of our virtual and hands-on laboratory experiences areimportant in adding pedagogical value to online sciencecourses. We used a formative evaluation to survey studentsatisfaction and perceptions regarding usability, content deliv-ery, and learning in a university program offering a variety ofonline laboratory science courses (i.e., biology, general chem-istry, organic chemistry, microbiology, anatomy, physiology,and physics). We also attempted to evaluate the effectivenessof online chemistry courses taught with either remote hands-on or virtual laboratory activities and examined the perfor-mance of students concurrently enrolled in a lecture and lab-oratory course versus those enrolled in a lecture-only course.
Methods
The overall goal of this study was to evaluate student experi-ences in online laboratory courses in order to inform the de-sign and improvement of online/distance education laboratory
J Form Des Learn
experiences. These courses are part of a distance educationprogram of science prerequisites for the health professionsand were developed to incorporate the same science standardsand best practices used in face-to-face laboratory sciencecourses—i.e., with experiences that are engaging and ground-ed in scientific thinking and practice. In order to evaluate thesenew course designs, we surveyed students’ perceptions of theeffectiveness, convenience, quality, and usability of the labo-ratory experiences.
Evaluating Student Perceptions
Survey Development and Structure
We developed a survey to evaluate student satisfaction andperceptions regarding content delivery and learning in onlinelaboratory courses. The survey is presented in Appendix A.The study design and survey were approved by theInstitutional Review Board. Survey questions were developedby the authors and reviewed by university curriculum person-nel in order to verify the clarity of the questions and answers.The survey questions emerged from those aspects of a sciencelaboratory experience that have been found to be important(American Association of Physics Teachers 1998; Bruck andTowns 2013; Dikmenli 2007; Woodfield et al. 2005).Generally, the questions explored the usability of the onlinelaboratory and the extent of the connection between the laband the lecture. The survey consisted of 20 items, including amixture of Likert scale responses, Yes/No, and fill-in ques-tions. Background information was also collected includingsome demographic factors and items exploring computer use.
Pre-survey reviews used to validate our instrument includ-ed individuals with experience in curriculum/course designand faculty with experience in laboratory sciences. For exam-ple, survey questions were reviewed by members of the uni-versity curriculum staff to verify the clarity of question andanswer options and the face validity of the items. Followingthe initial review, faculty with experience in laboratorycourses was asked to complete the survey as if they werestudents completing the survey. This simulation occurredwithout any questions or concerns arising. General descriptivestatistics for the survey were generated using Survey MonkeyGold. The last survey question was an open response questionasking students to Bplease provide additional comments orfeedback about your experience in the laboratory.^ The openresponses were manually coded and categorized into themesby the corresponding author. Reliability of the survey re-sponses was addressed by asking similar but unique questionsand evaluating consistency among those questions. This isdescribed in more detail in the BResults^ sections pertainingto individual questions.
Survey Enrollment
In order to ensure a significant number of respondents andgather reliable data, we sent out the survey to all studentswho successfully completed one of the online science labora-tory courses offered through the distance education programduring the period of June 1, 2010, to May 31, 2013. These areundergraduate courses in the following disciplines: biology,general chemistry, organic chemistry, microbiology, anatomy,physiology, and physics. The students were taking these on-line courses in order to fulfill major requirements for theirhome institutions or as prerequisites for their desired profes-sional programs. There were 2972 students in this cohort.Students were contacted through e-mail with an explanationof the purpose and process of the survey which included thefollowing information: Principal Investigator’s contact infor-mation, short introduction as to the purpose of the study, enddate for data collection, and the link to the survey. Studentswere informed that the survey responses would be anony-mous, and only aggregate data would be presented. All par-ticipation was voluntary. The survey took approximately20 min to complete. The survey remained open for 2 weeksfor each student.
Survey Analyses
General descriptive statistics for the survey were generatedusing Survey Monkey Gold. The last survey question wasan open response question asking students to Bplease provideadditional comments or feedback about your experience in thelaboratory.^ The open responses were manually coded andcategorized into themes by the corresponding author. We ex-tracted a subset of the survey results from the chemistrycourses (General Chemistry I and II and Organic ChemistryII), in order to control for inter-instructor and inter-disciplinevariability (Tables 4 and 5 in Appendix E). These data weretabulated separately for comparison with both the full aggre-gated data set, as well as the performance data (see BResults^section).
Evaluating Student Performance
In order to assess the relationships among student perceptionsand student performance, further analyses were carried outwith students enrolled in general chemistry and organic chem-istry courses. This enabled us to enhance the reliability of ourdata by controlling for inter-discipline and inter-instructorcourse variability, since these courses were all designed anddelivered by the same faculty member (corresponding author).This restricted sample included 160 students who successfullycompleted an online course, either General Chemistry I (70students) or Organic Chemistry II (90 students), during thepast 3 years, and students who completed Organic
J Form Des Learn
Chemistry II taught in a traditional manner at another univer-sity (107 students). From this convenience sample, we obtain-ed data for the following variables: Final Laboratory Grade,Final Lecture Grade, Laboratory Type, and Course Format(i.e., online or traditional).
Hands-On and Virtual Laboratories
The study was designed, in part, to explore the efficacy oflaboratory experiences in online chemistry courses using ahands-on (at-home) laboratory format or a virtual laboratoryformat. Hands-on laboratory activities were part of the onlinegeneral chemistry course, in which students conducted theexperiments, unsupervised, using the chemicals and equip-ment in custom-built at-home laboratory kits. The studentsused their observations to write laboratory reports and answerassessment questions, similar to those activities associatedwith traditional laboratory experiments, but unsupervised di-rectly by a content expert. Virtual laboratory activities werepart of the online organic chemistry course in which studentscompleted the experiments on their computers (again with nodirect expert supervision). The students built reaction appara-tus, ran chemical reactions, and analyzed reaction productsusing spectroscopy, all through an interactive computer-based simulation program (Woodfield and Catlin 2004) whichwe refer to as Bvirtual^ laboratory experiments. The studentscollected and analyzed data to write laboratory reports andanswer assessment questions.
Performance Analyses
Student performance in lecture and laboratory was analyzedusing linear regression analysis and non-parametric testing. Inaddition, student performance in lecture was compared acrossall three chemistry course formats, with or without laboratory,using a two-way ANOVAwith Tukey follow-up tests for mul-tiple comparisons.
Results
Demographics of Survey Respondents
Surveys were sent to students who completed the online un-dergraduate science laboratory courses between the years2010 and 2013. Of the total 2972 surveys emailed to the stu-dents, 90 student emails were undeliverable. Of the remaining2882 students presumed to have received the survey, 386responded, yielding a 13.4% response rate (see Appendix Bfor full results of demographic and computer use questions).Only 22.2% of respondents were taking these courses to com-plete an undergraduate or post-baccalaureate degree, whereasover half (54.5%) were in a graduate or professional program.
The remaining 23.3% were taking the courses to satisfy pre-requisite requirements for graduate or professional programsor for self-enrichment. Over 70% were non-traditional stu-dents (defined by the National Center for EducationStatistics (NCES; https://nces.ed.gov/pubs/web/97578e.asp)as those aged 25–39) and females outnumbered males twoto one.
The first two survey questions asked students to select thelast online course they completed through their distance edu-cation program and to select the corresponding laboratorycomponent (i.e., hands-on or virtual laboratory activities).Students were instructed to answer the remaining survey ques-tions based on the course they selected. In the last question ofthe survey, which was an open response question, studentswere given the opportunity to provide additional commentsor feedback about their laboratory experience. Of the 386students who responded to this survey, 189 (49%) providedfeedback or comments. These open responses were manuallycoded and categorized into themes by the corresponding au-thor. The themes that emerged were effectiveness, quality, andconvenience.
Of the courses involved in the survey, five courses utilizedhands-on lab kits (at-home labs) and five used computer-based virtual labs. For the subset of data extracted and usedto control for inter-discipline and inter-instructor variability,students were taking either online general chemistry, in whichthey engaged in hands-on, unsupervised, at-home laboratoryactivities, or online organic chemistry, in which they conduct-ed virtual experiments on their computers, again with no directexpert supervision.
Description of Laboratory Experiencesfor Online Courses
Hands-on General Chemistry Laboratory
The students in online general chemistry used custom-builtBLabPaqs,^ purchased from Hands on Labs (HOL; http://holscience.com/). The experiments selected for the LabPaqsprovide the student with a Bcomprehensive hands-on laboratoryexperience^ that is academically aligned with the laboratory ex-perience obtained by students at traditional colleges and univer-sities according to theHOLwebsite. These LabPaqs are designedto reinforce the concepts and topics described in the lecture com-ponent that are usually difficult for the students to grasp. Forexample, students are able to determine the stoichiometry of aprecipitation reaction using techniques normally used in the tra-ditional lab setting, which include the scientific method (includ-ing recording observations), correct use of laboratory glasswareand equipment including digital balance, gravity filtration, labo-ratory calculations, and waste disposal. This one experiment al-lows the student to demonstrate competency in several key
J Form Des Learn
concepts, such as stoichiometry, limiting reactant, theoretical,actual, and percent yields that are used throughout the course.
Owing to the nature of these at-home labs, the students areengaged in the process and can reflect on what they are doingand why. There is a greater risk with the hands-on labs that theexperiment will not go as planned owing to experimenter er-ror. The LabPaqs are designed to contain enough chemicalsand supplies to allow the student to repeat the experiment.When a student’s experiment does not go as planned, thiscan be turned into a positive learning experience by whichthe student can gain valuable experience and confidencewhich will be helpful down the road.
Virtual Organic Chemistry Laboratory
The students in Organic Chemistry II used simulated/virtuallab software that is installed on their computers. The experi-ments are Brealistic and sophisticated^ (Woodfield et al. 2005)and place the students in a 2D virtual laboratory where theyare able to make decisions that will determine the outcome ofthe experiments (Woodfield and Catlin 2004). This importantattribute of the virtual laboratory software places the respon-sibility of the decision-making process on the student, whichis comparable to the decision-making process of students inthe traditional laboratory setting.
One important aspect of the simulated organic chemistryexperiments is that they are designed to work as long as thestudent follows the experimental procedure. Therefore, it isimportant to keep the student engaged by giving them theopportunity to reflect on why they are doing the experiment,what they should expect the outcome to be, and how they canensure the expected outcome is achieved. Keeping the stu-dents engaged in the process and allowing them time to reflectare instrumental in the development of their independent crit-ical thinking skills. To keep the students engaged, the proce-dure is written in such a way to allow students to record theirobservations as they are happening. This is accomplished byasking the student to run thin-layer chromatography (TLC)and describe how the reaction is progressing from reactant toproduct. Another aspect of student engagement occurs duringthe work-up of the experiment. During the liquid-liquid ex-traction, the student needs to record which layer contains thedesired product. This is important to know since saving thewrong layer would result in isolating a product that is not thedesired product. Knowing where the desired product is at alltimes allows the student to show they understand how thestructure of the product directly affects its polarity and thusincreases the solubility in one layer (solvent) over the other.
What happens when an experiment does not go as plannedin either the simulated organic chemistry laboratory or thetraditional organic chemistry laboratory? This is where thereflection comes in. In the simulated laboratory, students havethe luxury of reflecting onwhat went right and what did not go
as planned. They are able to go over their experiment anddetermine the step where the reaction went off the projectedpath. They are then able to re-do the experiment and correcttheir mistakes on the spot. This reduces the student’s frustra-tions and creates a positive learning experience. In the tradi-tional laboratory, students have very few options when theirexperiments do not go as planned. Traditional organic labora-tories have significant time constraints. Students have a setamount of time to start the experiment, work-up and analyzethe data, and clean up. They do not have the luxury of startingover if their experiment did not go as planned. This can lead tofrustration and a poor lab grade, especially since the studentdoes not have time to figure out what went wrong.
Student Perceptions
The following results for student perceptions of their onlinecourse experiences are based on the survey data from all of ourprogram’s science courses with laboratory that includesGeneral Biology I and II, General and Organic Chemistry Iand II, Physics I, Microbiology, Anatomy, and Physiology.Table 1 summarizes responses collated from Likert-formatted questions, and Table 2 illustrates responses from afill-in question format.
Student Perceptions of the Usability of OnlineLaboratories
We constructed several survey questions (see Appendix A forentire survey questionnaire) to assess student perceptions ofthe ease of use (i.e., usability) of the online course laboratoryactivities. Based on our survey results (Table 1), more than88% of the respondents who conducted course experimentsusing either a virtual laboratory or a hands-on lab kit felt thatthe virtual labs were easy to install and use, and the hands-onlab instructions were easy to understand.
One survey question provided students the opportunity toreport any challenges they experienced accessing the virtual lab-oratory experiments or online laboratorymanual for the hands-onexperiments. Of the 379 responses to this question, 40 students(10.6%) reported they experienced one of the following chal-lenges: unable to install the virtual laboratory software or todownload the hands-on lab manual, unable to open the laborato-ry software or laboratory manual once it was installed, or com-puter froze or crashed when opening the software or laboratorymanual. Seventy-six percent of the students reported that they didnot experience any technical difficulties. The remaining 13.4%reported experiencing different technical challenges than thosestated above. These responses are summarized in Appendix C.Since these challenges varied with no appreciable trend for anyone challenge, they were not explored further.
Of the 189 responses to the open response question, themajority of these students were enrolled in a laboratory course
J Form Des Learn
that utilized online (virtual) experiments, rather than at home,hands-on experiments (see Appendix D for results of openresponse question). Nineteen responses were about the usabil-ity of the virtual laboratory software: nine students mentionedthat it was easy to use, two mentioned that it was not easy touse, three mentioned that there was a small learning curve, andfive mentioned that the software worked well in general, butsuffered a bit from poor image quality or mouse sensitivity.Regarding whether or not the hands-on laboratory instructionswere easy to understand, ten of the students’ responses to theadditional-comment question were relevant: four said theywere easy to understand, four said they were easy to under-stand but would likemore instructions, and two said they werenot easy to understand.
Overall, students found the laboratory instructions readilyaccessible and experienced few technical problems, indepen-dent of whether their course utilized hands-on or virtual labo-ratory activities.
Perceptions of Online vs Traditional LaboratoryScience Courses
Two survey questions were designed to assess the students’experiences with the laboratory associated with their online
course as compared to their experiences in traditional labora-tory courses.Most of the students (97.4%) responded that theyhad taken a college or university science course with a labo-ratory in a traditional classroom setting so they could makethis comparison.
Perceptions about the time required to complete an exper-iment varied widely. Approximately half of the students feltthat the time required to complete a laboratory experiment fortheir online course was comparable to the time required tocomplete an experiment in a traditional laboratory setting(Table 1).While nearly 40% did not agree with this, the reasonfor disagreement was not surveyed. Nineteen studentscommented that the online (virtual) laboratories took less timethan traditional laboratories. Reasons provided by the studentsfor the time-savings were twofold: first, timers could be accel-erated so that reactions could be sped up; second, time re-quired for setting up and cleaning up experiments wasreduced.
When students compared their laboratory experiences intheir online courses to their traditional laboratory courses,two thirds reported that the online laboratory experience wasas good or better than their traditional laboratory experience(Table 2). Eighty-three students who responded to the openresponse question specifically commented on their virtual and
Table 1 Student perceptions of usability and learning
Survey questions Numbera Strongly agreeor agree
Neither agreenor disagree
Strongly disagreeor disagree
Usability
• The laboratory software was easy to install/download. 368 88.0% 4.1% 7.8%
• The laboratory software was easy to use. 365 88.5% 4.4% 7.1%
• The laboratory instructions were easy to understand. 317 88.3% 5.4% 6.3%
• Time required to complete online lab assignments comparableto time spent in traditional laboratory.
381 48.3% 12.1% 39.6%
Learning enhancement
• Lab experiments reinforced topics from lecture and text. 385 83.4% 9.4% 7.3%
• Laboratory experiments helped me with the understanding oflecture concepts.
385 72.7% 15.6% 11.7%
• Lecture material helped me understand the purpose of the lab experiment
383 81.7% 9.9% 8.4%
• Completing the lab assignments helped me to do better on lecturequizzes and exams.
385 61.3% 21.0% 17.7%
a The sample size for each question is 386 (the total number of students who took the survey) minus the number of students who either did not answer thequestion or who said the question did not apply
Table 2 Perceptions of students with both online and traditional lab experiences
Survey question Numbera BMuch better than^or Bbetter than^
BAbout thesame as^
BMuch worse than^or Bnot as good as^
My online lab experience was ___________my experience in atraditional classroom lab
384 29.7% 36.7% 33.6%
a The sample size for each question is 386 (the total number of students who took the survey) minus the number of students who either did not answer thequestion or who said the question did not apply
J Form Des Learn
traditional laboratory experiences. Of these, the majority(73%) said that their virtual laboratory experience was as goodor better than their traditional laboratory experience.Recurring reasons for this included the ability to repeat exper-iments, the ease of changing parameters, the stress-free envi-ronment, the ability to perform the experiment withoutwasting materials, the ability to focus exclusively on contentinstead of logistical details, and generally feeling the labora-tory was more interactive. Several comments in favor of tra-ditional laboratories cited the advantage of having peers pres-ent and a laboratory assistant available for interactions, and apreference for hands-on learning.
Perceptions of Synergism Between Lectureand Laboratory Concepts
As shown in Table 1, the majority of students (83.4%) indi-cated that the online laboratory experiments reinforced thetopics discussed during the online lecture or in the textbook.These survey questions were reinforced by other answers tothe BLearning enhancement^ questions in Table 1, where amajority (72.7%) also felt that completing the laboratory ex-periments helped them better understand the lecture material.Furthermore, most students (81.7%) believed that the lecturematerial helped them understand the purpose of the laboratoryexperiments, and many (61.3%) perceived that completing thelaboratory assignments helped them to do better on lecturequizzes and exams. These findings are further supported byresponses to the open response question, where 12 studentsspecifically commented on how well the online laboratoriesreinforced the lecture concepts, and 3 students mentioned thatthe lecture reinforced the concepts of and/or prepared them forthe online laboratory.
In order to control for inter-instructor variability anddiscipline-based variability, we extracted the survey resultsfrom students who were enrolled in chemistry courses.These courses were all developed by and taught by the leadauthor of this paper, and performance data are also presentedin Figs. 1, 2, and 3. Analyzing the subset of results from thesurvey data of the chemistry courses provided some interest-ing outcomes. The data querying students about whether theyfelt labs enhanced their learning of the lecture and textbookmaterials by reinforcing concepts and making connectionsvaried between general chemistry courses (with hands-on lab-oratory exercises) and the organic chemistry course (with thevirtual laboratory component). A large majority of students inthe virtual organic chemistry laboratory (76–88%) reportedthat the labs improved their understanding of the lecture ma-terial and vice versa, including that their performance on quiz-zes and exam were enhanced (Table 5 in Appendix E). Thesefindings are similar in students who completed either generalchemistry course (both of which utilized hands-on laboratoryexperiments), with the notable exception that only 52% of the
general chemistry students felt that the labs improved theirassessment scores (Table 4 in Appendix E). When perfor-mance data was analyzed between students who took thesecourses either with or without a laboratory component, wefound that the final course grade for organic chemistry stu-dents was similar in both groups, whereas having a lab im-proved student performance in general chemistry course(whether they completed the lab at home or in a traditionalmanner in a classroom with other students and the professor(Fig. 3).
Survey Validation and Reliability
As stated in the methods, prior to administering our survey, wevetted it through professionals with knowledge of online cur-riculum and laboratory sciences, as a means of validating oursurvey with some fidelity. It is recognized that there are morerigorous methods to validate surveys, but since our questionswere straightforward and we feel confident that they are rea-sonable for an initial screening of student perceptions regard-ing their experiences with online courses. The reliability ofour survey results was assessed by the consistency amongthe responses to questions in the BUsability^ and BLearningenhancement^ questions (Table 1) and BOpen response^ ques-tion (Appendix D). By assessing the consistency among threeLikert scale questions (each indicating that 88% of the stu-dents report ease of software installation/use and instruction)and while the majority responses in the Open response ques-tion indicated that they did not experience any technical diffi-culties (76%). Furthermore, there is good corroborationamong the three Likert scale questions in the Learning en-hancement category that interrogate whether the lab and lec-ture components reinforce one another (73–83% agreedstrongly or agreed), and these were confirmed in the openresponses. Collectively, we have confidence that our surveycaptured the fundamentals of student perception of usability ofsoftware and learning enhancement of laboratory activities.
Student Performance
Results describing student performance are based on the con-venience sample data. This sample reflects a subset of studentswho completed a chemistry course, either online generalchemistry I, online organic chemistry II, or organic chemistryII taught in a traditional classroom setting. This subset waschosen so that we could evaluate performances while reducingvariability due to differences among science disciplines andinstructors. This subset of students was further subdivided interms of the format of the laboratory experiments, such thatstudent taking the online organic chemistry course used avirtual laboratory program while those taking the online gen-eral chemistry course had hands-on (at-home) laboratory ex-periments. The organic chemistry course taught in a face-to-
J Form Des Learn
face campus setting utilized traditional hands-on laboratoryexperiments. Another advantage for using this particular sub-set of students is that all of the courses were developed andtaught by the same instructor, so that instructor variabilitycould be minimized.
Student Performance in Online Chemistry Courses;Lecture and Laboratory Components
Data from 70 students in an online general chemistry course,who used a hands-on laboratory at home, were analyzed toexamine the relationship between Final Lecture Grade andFinal Laboratory Grade (Fig. 1a). A similar analysis was car-ried out with data from 90 students in an online organic chem-istry course, in which students used software for a virtuallaboratory (Fig. 1b). These analyses were then compared toregression results based on data from 107 students in a tradi-tional (face-to-face) organic chemistry course (Fig. 1c).
Regression analyses of these data indicate a strong positivecorrelation between student performance in lecture and labo-ratory in the traditional organic chemistry course and a modestpositive correlation between these variables in the online
organic chemistry course (see R2 values in Fig. 1c, b,respectively). On the other hand, lecture grades were not wellcorrelated with laboratory grades in the online general chem-istry course (see R2 value in Fig. 1a). Therefore, we furtheranalyzed the data from the online courses by separating thestudent performances by quadrants. In Fig. 2, each variablewas broken into an upper and lower half. In both scatterplots,the vertical line was placed at the median Final LaboratoryGrade, and the horizontal line was placed at the median FinalLecture Grade. Figure 2a illustrates data from the studentswho conducted hands-on laboratory experiments, and Fig.2b scatterplot illustrates data from a course with virtual labo-ratory experiments. Both scatterplots contain representation inall four quadrants. The four quadrants are marked LL, LU,UL, and UU. The first letter in each pair indicates whetherthe data points are in the lower or upper half of the FinalLaboratory Grade, and the second letter pertains to the FinalLecture Grade. For example, a data point in the UL quadrant isabove the median Final Laboratory Grade and below the me-dian Final Lecture Grade.
For each laboratory type, the quadrant distribution is shownin Table 3. Nonparametric testing indicates that the percent
50 60 70 80 90 10040
60
80
100
Final Lab Grade
Fina
l Lec
ture
Gra
de
Hands-On Laboratory
LU
LL UL
UL
50 60 70 80 90 10040
60
80
100
Final Lab Grade
Fina
l Lec
ture
Gra
de
Virtual Laboratory
LU
LL UL
UL
a b
Fig. 2 Quadrant analyses. a Comparing laboratory grades and lecturegrades based on upper and lower limits using the medians to draw thedemarcation lines for students in an online general chemistry course withhands-on laboratory experiments (n = 70 students). b Comparing
laboratory grades and lecture grades based on upper and lower limitsusing the medians to draw the demarcation lines for students in anorganic chemistry course with virtual laboratory experiments (n = 90students). See text for quadrant definition details
50 60 70 80 90 100
40
60
80
100
Final Lab Grade
Fina
l Lec
ture
Gra
deHands-On Laboratory
y = 0.56x + 29.24R2 = 0.106
50 60 70 80 90 100
40
60
80
100
Final Lab Grade
Fina
l Lec
ture
Gra
de
Virtual Laboratory
y = 0.07x + 26.33R2 = 0.508
50 60 70 80 90 100
40
60
80
100
Final Lab Grade
Fina
l Lec
ture
Gra
de
Traditional Laboratory
y = 1.27x - 25.39R2 = 0.729
a b c
Fig. 1 Regression analyses of lecture and laboratory performance in theonline and traditionally taught chemistry courses. a Online generalchemistry with Bhands-on^ laboratory experiments (n = 70 students). bOnline organic chemistry with Bvirtual^ laboratory experiments (n = 90students). c Traditionally taught organic chemistry with Btraditional^
laboratory experiments (n = 107 students). Data represent individualstudent performances in lab (final lab grade) and final lecture exam.Inserts depict slope and y-intercept of the regression line and R2 valuesfor each data set
J Form Des Learn
distribution for the quadrants for hands-on laboratories is notstatistically different than the percent distribution for the quad-rants describing virtual laboratories, χ2
3 ¼ 2:937; p ¼ 0:4 Forexample, there was no statistical difference between highestperforming students in both the lecture and lab (UU) whencomparing students using a hands-on lab (30%) to those usinga virtual lab (40%).
Student Performances in Online and TraditionalEnvironments: Impact of Laboratory
To assess the influence of laboratory activities on final lecturegrades, and compare those results for both online and tradi-tional courses, data from three subsets of student outcomeswere analyzed (online general chemistry, online organicchemistry, and traditional organic chemistry).
Our data indicate that students performed better in courseswith hands-on or traditional laboratory experiments comparedto lecture only (Fig. 3). A similar trend was observed with thevirtual laboratory, but the data were not significant (p = 0.09).It should be noted that the overall grades for the online chem-istry course with a virtual laboratory were higher than any ofthe other courses. In addition, even in a lecture-only course,
students enrolled in an online organic chemistry courseoutperformed students in the other courses (gray bars).
Discussion
Based on a survey of student perceptions, this study providessupport for the notion that natural science courses can be ef-fectively taught using an online/distance education platform.Laboratory experiences provide additional value to these on-line courses and can be delivered digitally (virtual labs) orusing hands-on lab kits for Bat-home^ convenience. Our sum-mative student data on performance suggest that students en-rolled in online science courses do as well or better than theirpeers enrolled in traditional courses.
Student Perceptions of Online Laboratory Experiences
The findings presented here provide useful information onhow to design and deliver a positive online laboratory ex-perience. Laboratory software must be easy to install, user-friendly, and intuitive, yet challenging and able to provideexperiences similar to the traditional laboratory. Likewise,digital laboratory manuals that accompany hands-on labkits must also be user-friendly and intuitive. It is vital tobear in mind that online students are completing their lab-oratory assignments (either virtual or hands-on) withoutdirect supervision from content experts.
The majority of survey respondents felt their online labo-ratory experience was the same as or better than their priorexperiences in the traditional setting. One reason that emergedfrom students’ comments was the convenience of accessingthe laboratories whenever and wherever they were needed. Inmany cases, this is what made it possible for students to com-plete their required or prerequisite laboratory science courses.Student comments were particularly positive about the virtuallaboratory for several reasons, all associated with the ease ofsetup and use. These virtual labs are time saving compared totraditional labs that require physical setup and cleanup, andcan be conducted multiple times while changing the parame-ters in an efficient manner. This allows students to spend moretime understanding the objectives of the laboratory experi-ences. Laboratory experiments that involve timed processes,such as reactions, can be accelerated in the virtual format.Recurring comments also indicate that students find virtuallaboratory experiments to be more interactive and engaging.
Overall, the students’ responses regarding the quality of theonline laboratory experience were positive. These includedhaving a stress-free environment, the ability to perform theexperiments without wasting materials, the ability to focusexclusively on content instead of logistical details, and gener-ally feeling the laboratory was more aligned with the lecturematerial. Some students mentioned having a preference for
01050
60
70
80
90
100Fi
nal L
ectu
re G
rade Lecture & Lab
Lecture Only*
Gen Chem O-Chem O-Chem
Online TraditionalCourse format
Fig. 3 Final lecture grades were compared between two online courses(general chemistry and organic chemistry) and a course taught in atraditional format (organic chemistry). Student grades were also dividedbased on concurrent enrollment in lecture and laboratory versus lectureonly. Data represent mean ± SE, n = 70–107 students/course. Asteriskindicates significant difference among final lecture grades of studentstaking the lecture only. Bars indicate significant differences betweengroups. Data were analyzed with a two-way ANOVA with Tukeymultiple comparisons follow-up tests (significance was set at α = 0.05)
Table 3 Quadrant distribution (see text for definitions of quadrant labels)
LL LU UL UU
Hands-on 44% 11% 14% 30%
Virtual 43% 9% 8% 40%
J Form Des Learn
hands-on learning, and this can be accomplished in an onlinecourse by using at home lab kits. Other comments, however,did cite the advantage of having direct interactions with bothpeers and the professor/lab instructor in the traditional sciencelaboratory setting.
In terms of effectiveness, our data show that students be-lieve that their laboratory experiences helped them understandthe concepts presented in lectures and that the lecture materialhelped them grasp the purpose of the laboratory experiments.Furthermore, students stated that completing the laboratoryexperiments reinforced specific topics covered in the lecturesand textbook readings, and thus may have helped them per-form better on course assessments (quizzes and exams).Instructors in online science courses, like their counterpartsusing face-to-face instruction, use the laboratory experienceboth to illustrate particular techniques and methods, and toprovide students with an experiential way to learn specificcourse concepts. This reinforcement provides an opportunityfor students to develop a deeper understanding and better re-tention of course material.
The open-ended response question also gave the studentsthe opportunity to inform us of several areas where we couldimprove our online courses in order to enhance the student’sonline experience. These include real-time technical supportand the lack of more tutorials that help to explain difficulttopics. Real-time technical support is a hard issue to address.This is owing to the fact that our students are able to takeonline courses from anywhere in the world. With all of thedifferent time zones, there is a possibility that all technicalsupport offices are closed and not accessible to all of ourstudents when needed. Solutions to the most common prob-lems, such as forgotten passwords, or other common issuescan be found in the Frequency Asked Questions (FAQ) on theprogram’s home page. To address student needs for more tu-torials on difficult subject matter, we can turn to the textbookpublishers. With the advances of electronic textbooks (e-books), these tutorial or videos are embedded directly in thee-book. In addition to these, the publishers often provide otheronline resources that can be included into the learning man-agement system (LMS). Adoption of books that are contentrich and have adaptive learningwould address these needs andenhance the online experience. The biggest concern expressedby online students is the lack of peer-to-peer and student-instructor interactions. Ways that we are planning to addressthis concern include the use of chat rooms or study lounges,virtual office hours, and discussion boards.
Overall Student Performance in Online vs TraditionalLaboratory Courses
The data presented in Fig. 1 indicate a positive correlationbetween lecture and laboratory grades for students enrolledin either an online or traditional organic chemistry course,
compared to students enrolled in an online general chemistrycourse (R2 values = 0.508, 0.729, and 0.106, respectively).Wespeculate this finding most likely reflects a process of studentlearning and maturation (as opposed to being subject matter orformat specific). Students usually take a year of general chem-istry with corresponding laboratory before they take organicchemistry. This allows students in general chemistry an op-portunity to figure out how to make use of the laboratoryexperience to help them understand the topics discussed inthe lecture, and vice versa. Therefore, students taking organicchemistry have already developed the skills needed to appre-ciate how laboratory experiments are designed to reinforcecourse concepts discussed in lecture, whereas students in gen-eral chemistry are just learning how to make this connection.
A closer inspection of the R2 values for Fig. 1b, c shows thecorrelation between lecture and laboratory performance isstronger for students in the traditional course than those takingorganic chemistry online, with a virtual laboratory (same in-structor for both courses). This is perhaps explained by thefact that the instructor in the traditional organic chemistrycourse has the ability to directly supervise students in thelaboratory. This allows for immediate feedback on the prog-ress of the experiment and an opportunity to address concernsor questions that may arise during the lab period. Studentstaking the laboratory online do not have the ability to obtainimmediate feedback from their instructor or their peers.Instructors for online laboratories must rely on their studentsto concisely explain (after the fact) any issues they had duringthe laboratory. Several students’ comments in the open re-sponse survey question stated they preferred the traditionallaboratory over the virtual laboratory simply because of theavailability of the laboratory instructor and peers.
Student Performance: Concurrent Enrollmentin Lecture and Laboratory vs Lecture Only
The data presented in Fig. 3 show that students who takelecture and laboratory concurrently outperform their lecture-only peers, independent of course (i.e., general or organicchemistry) or delivery method (i.e., online or traditional).These data are in agreement with the observations of Matzet al. (2012) who compared students taking general chemistrywith or without a concurrent laboratory. When the material inthe lecture and laboratory reinforce each other, students gain adeeper understanding, retain the material longer, and performbetter on the assessments.
Interestingly, we found that the students enrolled in theonline organic chemistry course performed significantly betterthan those in either the traditional organic chemistry course orthe online general chemistry course (Fig. 3). Most of the stu-dents taking the online science courses are nontraditional stu-dents who have earned a BA/BS degree, and those takingorganic chemistry all must have already taken at least one
J Form Des Learn
prior science course (i.e., general chemistry). Therefore, theseresults are perhaps not surprising. It is likely that these stu-dents are more mature, more motivated, and have alreadydemonstrated success in science. The traditional organicchemistry students have also had general chemistry (and may-be other science courses), but they are typically younger andhave not yet earned a degree. The online general chemistrystudents may or may not have had any science courses beforetaking that course (even though they are likely to have earneda bachelor’s degree).
Study Limitations
The authors did not have access to the grades for all the surveyparticipants; therefore, we restricted the analyses of the onlinenatural science courses to student perceptions of usability andeffectiveness of their laboratory experiences. The survey ques-tion asking students to compare their online experience rela-tive to their traditional lab experience was too general—i.e., itdid not include specific aspects of the lab (e.g., quality, effec-tiveness, ease of use, etc.). Assessment of student performancewas limited to chemistry courses in which lecture and labora-tory grades were available (taught by the corresponding au-thor). Additionally, it would be an improvement in the studydesign if a comparison between virtual, hands-on and tradi-tional laboratory experiments could be conducted for the sameonline lecture course.
Conclusions
This study supports the notion that effective, efficient teachingand learning instruction is possible outside the traditional lab-oratory setting. For students taking online science courseswith laboratory, it was important to determine whether stu-dents’ experiences and perceptions of the laboratory compo-nent of the course were similar to those in the traditionalclassroom/laboratory setting. Most of the students surveyedare nontraditional students who have limited educational op-tions because of geography, employment, and/or family obli-gations. Online science courses with laboratory provide a flex-ible option for completing science prerequisite courses forprograms in the health professions.
This study also provides evidence that online laboratoryscience courses can meet the educational needs of these stu-dents. Our data show that the majority of students conductinglaboratory experiments through online science courses feltthat the online laboratories were easy to use and comparableto the experiments in a traditional laboratory setting, but also,they saved valuable time. Students perceived that the labora-tories helped them to do well in the lecture portion of thecourse, and that the lecture, in turn, helped them understandthe purpose of the laboratories. These perceptions are
supported by summative laboratory and lecture assessmentscores. Finally, content analysis of the students’ open-endedcomments revealed an overall satisfaction with the effective-ness, quality, and convenience of the laboratory experience.
Acknowledgements The authors would like to thankMary L. Schwanke,Ph.D. for her expert editing and advice during the writing of thismanuscript.
Compliance with Ethical Standards
Conflict of Interest The authors declare that they have no conflict ofinterest.
References
Allen, I. E., Seaman, J. (2010). Class difference$: online education in theUnited States. Sloan Consortium. https://www.onlinelearningsurvey.com/reports/class-differences.pdf. Accessed June 2006.
American Association of Physics Teachers. (1998). The goal of introduc-tory laboratories. College Park. Retrieved from http://www.aapt.org/Resources/policy/goaloflabs.cfm. Accessed April 2016.
American Chemical Society. (2011). Importance of hands-on laboratoryactivities. Retrieved from https://www.acs.org/content/acs/en/policy/publicpolicies/education/computersimulations.html.Accessed September 2017.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How peoplelearn: Brain, mind, experience, and school (expanded ed).Washington, D.C.: National Academy Press.
Brinson, J. R. (2015). Learning outcome achievement in non-traditional(virtual and remote) versus traditional (hands-on) laboratories: Areview of the empirical research. Computers and Education, 87,218–237. https://doi.org/10.1016/j.compedu.2015.07.003.
Bruck, A. D., & Towns, M. (2013). Development, implementation, andanalysis of a national survey of faculty goals for undergraduatechemistry laboratory. Journal of Chemical Education, 90(6), 685–693. https://doi.org/10.1021/ed300371n.
Casanova, R. S., Civelli, J. L., Kimbrough, D. R., Heath, B. P., & Reeves,J. H. (2006). Distance learning: A viable alternative to the conven-tional lecture-lab format in general chemistry. Journal of ChemicalEducation, 83(3), 501.
Chen, P.-S. D., Lambert, A. D., & Gudiry, K. R. (2010). Engaging onlinelearners: The impact of web-based learning technology on collegestudent engagement. Computers & Education, 54, 1222–1232.
Conway-Klaassen, J. M., Wiesner, S. M., Desens, C., Trcka, P., &Swinehart, C. (2012). Using online instruction and virtual laborato-ries to teach hemostasis in a medical laboratory science program.Clinical Laboratory Science, 25(4), 224–229.
Dalgarno, B., Bishop, A. G., Adlong, W., & Bedgood Jr., D. R. (2009).Effectiveness of a virtual laboratory as a preparatory resource fordistance education chemistry students. Computers & Education,53(3), 853–865.
Dikmenli, M. (2007). Biology student teachers’ ideas about purpose oflaboratory work. Asia-Pacific Forumon Science Learning andTeaching, 10(2), 1–14 Article 9. https://www.ied.edu.hk/apfslt/v10_issue2/dikmenli/dikmenli7.htm.
Dobson, J. (2009). Evaluation of the virtual physiology of exercise labo-ratory program. Advances in Physiology Education, 33, 335–342.
Feig, A. (2010). An online introductory physical geology laboratory:From concept to outcome. Geosphere, 6(6), 1–10.
Hands-On Lab Website. (n.d.). http://holscience.com/.
J Form Des Learn
Honey, M. A., & Hilton, M. L. (2011). Learning science through com-puter games and simulations. Washington, D.C.: NationalAcademic Press.
Johnstone, A. H. (1982). Macro-and micro-chemistry. School ScienceReview, 64(227), 377–379.
Josephsen, J., & Kristensen, A. K. (2006). Simulation of laboratory as-signments to support students’ learning of introductory inorganicchemistry. Chemical Education Research and Practice, 7(4), 266–279.
Liu, X. (2006). Effects of combined hands-on laboratory and computermodeling on student learning of gas laws: A quasi-experimentalstudy. Journal of Science Education and Technology, 15(1), 89–100.
Ma, J., & Nickerson, J. V. (2006). Hands-on, simulated, and remotelaboratories: A comparative literature review. ACM ComputingSurveys (CSUR), 38(3), 7.
Makransky, G., Thisgaard, M. W., & Gadegaard, H. (2016). Virtual sim-ulations as preparation for lab exercises: Assessing learning of keylaboratory skills in microbiology and improvement of essential non-cognitive skills. PLoS One, 1-11, e0155895. https://doi.org/10.1371/journal.prone.0155895.
Matz, R. L., Rothman, E. D., Krajcik, J. S., & Banaszak, M. M. (2012).Concurrent enrollment in lecture and laboratory enhances studentperformance and retention. Journal of Research in ScienceTeaching, 49(5), 659–682.
Means, B., Toyama, Y., Murphy, R., Bakia, M., and Jones, K. (2010).Evaluation of evidence-based practices in online learning: a meta-analysis and review of online learning studies. Washington DC: USDepartment of Education, Office of Planning, Evaluation, andPolicy Development. https://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf. Accessed September2017.
Merchant, Z., Goetz, E. T., Keeney-Kennicutt, W., Kwok, O., Cifuentes,L., & Davis, T. J. (2012). The learner characteristics, features ofdesktop 3D virtual reality environments, and college chemistry in-struction: A structural equation modeling analysis. Computers &Education, 59, 551–568.
National Center for Education Statistics (NCES). (n.d.). https://nces.ed.gov/.
National Research Council. (1996). National Science EducationStandards. Washington, D.C.: The National Academies Press.
National Research Council. (2000). Inquiry and the National ScienceEducation Standards: A guide for teaching and learning.Washington, D.C.: The National Academies Press.
National Research Council. (2005). How students learn: History, mathe-matics, and science in the classroom. Washington, D.C.: TheNational Academies Press.
National Science Teachers Association. (2007). NSTA position state-ment: the integral role of laboratory investigations in science instruc-tion. http://www.nsta.org/about/positions/laboratory.aspx. AccessedSeptember 2017.
Rivera, C. (2014). For some students, virtual labs replace hands-on sci-ence experiments. Los Angeles Times. Retrieved from http://www.latimes.com/local/education/la-me-college-labs-20141115-story.html. Accessed September 2017.
The National Center for Education Statistics (NCES). (n.d.). Retrievedfrom https://nces.ed.gov/pubs/web/97578e.asp.
Waldrop, M. M. (2013). Education online: The virtual lab. Nature, 499,268–270 http://www.nature.com/news/education-online-the-virtual-lab-1.13383.
Welsch, D. (2015). The effectiveness of online labs. Online LearningTips , Amer i can Pub l i c Unive r s i ty Sys tem. h t tps : / /onlinelearningtips.com/2015/06/the-effectiveness-of-online-science-labs/. Accessed September 2017.
Woodfield, B. F., & Catlin, H. R. (2004). The virtual ChemLab project: Arealistic and sophisticated simulation of inorganic qualitative analy-sis. Journal of Chemical Education, 81(11), 1672–1678.
Woodfield, B. F., Andrus, M. B., Anderson, T.,Miller, J., Simmons, B., &Stanger, R. (2005). The virtual ChemLab project: A realistic andsophisticated simulation of organic synthesis and organic qualitativeanalysis. Journal of Chemical Education, 82(11), 1728–1735.
J Form Des Learn
