stoeckel evaluation report - Boise State Universityedtech2.boisestate.edu/martastoeckelrogers/docs/evaluation_report.pdf · Engineering & Physical Science Responses ... over linear

An Evaluation of Desmos for Use in Physical Science Courses Submitted to John Thompson, PhD Boise State University

ABSTRACT

Prepared by Marta R. Stoeckel-‐Rogers EdTech 505: Evaluation for Educational Technologists

2 Desmos in Physical Science

Table of Contents Summary .............................................................................................................. 3 Program Description ........................................................................................... 3 Program Objectives .......................................................................................................... 4 Program Components ....................................................................................................... 4 Evaluation Method ............................................................................................... 4 Participants ....................................................................................................................... 4 Procedures ........................................................................................................................ 4 Data Sources .................................................................................................................... 5 Results .................................................................................................................. 5 Linear Motion Assessment ................................................................................................ 5 Classroom Observations ................................................................................................... 6 Student Survey ................................................................................................................. 6 Discussion ............................................................................................................ 9 Linear Motion Assessment ................................................................................................ 9 Classroom Observations ................................................................................................... 9 Student Survey ................................................................................................................. 9 Overall Conclusions ........................................................................................................ 10 Project Cost ........................................................................................................ 10 Appendix A: Linear Motion Assessment ......................................................... 11 Appendix B: Student Survey ............................................................................ 14 Appendix C: Open-Ended Survey Item Responses ........................................ 16 Notes on Interpretation and Analysis of Open-Ended Survey Items .............................. 16 Engineering & Physical Science Responses .................................................................. 17 Physics Responses ......................................................................................................... 18

Desmos in Physical Science 3 Summary In preparation for a bring your own device (BYOD) program scheduled to begin in the spring of 2015, the use of Desmos (desmos.com), a web-based tool for creating and analyzing graphs, was piloted in physical science courses. The courses that utilized Desmos are largely lab-based, meaning that students regularly complete basic data analysis focused on the creation and analysis of graphs, in order to learn new concepts in science. The goals of the program were to use Desmos to help students to generate science content knowledge, to teach students to identify and describe trends in data, to assist students in finding the equation for a line of best fit on a graph. The program was also attempting to identify a tool for creating and analyzing graphs that students would find easy to use. The program evaluation included several tools. First, students took both a pre-test and a post-test over content taught using data analysis activities built around the use of Desmos. Second, classroom observations were conducted to determine whether students were able to successfully complete tasks using Desmos, identify particular challenges students encounter when using Desmos, and collect information on student attitudes towards Desmos. Finally, students completed a survey regarding their perceptions of Desmos. Each of these data sources were used in two physical science courses. Engineering & Physical Science is a one-trimester course required of all 9th grade students. Physics is a year-long elective targeted at 12th grade students interested in pursuing college to study a STEM field. The evaluation reached favorable conclusions regarding the use of Desmos in physical science courses. Students’ acceptable growth on the test of science content, indicating that they were able to generate science content knowledge based on data analysis completed using Desmos. Classroom observations indicate that students were able to successfully create graphs, identify and describe trends, and determine equations for a line of best fit from within Desmos. A combination of classroom observations and the results of the student survey suggest that students find Desmos to be accessible and easy to use. Based on this data, Desmos should be used in the physical science curricula in the future with some modifications to the program to address issues identified in the evaluation. Program Description During the fall of 2014, Desmos, an online graphing calculator intended for educational uses, was piloted for use in physical science courses at Tartan High School. Tartan is located in a suburb of St. Paul, Minnesota and serves a diverse population of approximately 1200 students. During the pilot, Desmos was used by students in Engineering & Physical Science, a one-trimester course required of all 9th grade students, and Physics, a year-long elective targeted at 12th grade students interested in future study in STEM fields. Both courses are taught using lab-based methods, meaning students must frequently engage in data analysis, including graphing and graph interpretation. Students in Engineering & Physical Science were required to use Desmos for all graphs normally a part of the course while students in Physics were required to use Desmos for the first three weeks of the course, then given their choice of different graphing tools.

In the spring of 2015, Tartan will begin a formal BYOD program, leading many of the school’s science teachers to consider options for producing graphs digitally. Desmos was selected for a pilot and evaluation because it works on a wide variety of


devices. During the pilot, students used a mix of school-owned and student-owned devices, ranging from smartphones to desktop computers, to create and analyze graphs in Desmos. This evaluation covers the use of Desmos in a single, 12-week long trimester. If the program is deemed promising, the program will continue with any necessary modifications as a part of Tartan’s BYOD program. If the evaluation finds significant issues with the use of Desmos, it will be abandoned after the fall trimester and other tools will be examined. Program Objectives This program is intended to support students in developing graph interpretation skills, a key component of both scientific literacy and the scientific process. This program is also intended to support the development of science content knowledge through hands-on, student-centered strategies.

1. Students will be able to generate science content knowledge through data analysis, including graph interpretation.

2. Students will be able to recognize and describe trends in data. 3. Students will be able to determine the equation for a line of best fit. 4. Students will consider the graphing tool easy to use.

Program Components The use of Desmos is intended to supplement the larger program of instructional practices used in physical science courses at Tartan. These courses are heavily lab-based, meaning that students must frequently interpret data and produce graphs. This need led to numerous opportunities scattered throughout the trimester for students to use Desmos to create and analyze graphs. Students were provided with instruction over the basic functions, including entering a data table to produce a scatterplot and producing a line of best fit, as part of the first lessons where Desmos was used. Students used a variety of tools to access Desmos. On several occasions, a cart of Windows netbooks was brought into the classroom for students to run Desmos from. Students were also given the option of connecting their own devices to the classroom wi-fi to access Desmos. While a few students had their own laptops or tablets, most used smartphones to access Desmos. Evaluation Method Participants

All participants in this evaluation came from two groups. 30 students were enrolled in Engineering & Physical Science, a required one-trimester course for all 9th grade students. 115 students were enrolled in Physics, a year-long elective targeted at 12th grade students intending to pursue college education in a STEM field. These students were selected by virtue of having been randomly assigned to an instructor interested in using Desmos during the fall term. Procedures

The evaluation was conducted during the first term in which Desmos was utilized in Physics and Engineering & Physical Science, meaning both the instructor and the students were learning to use the tool. Students in both courses were provided with instruction on using Desmos to create a graph from a series of data points and finding the

Desmos in Physical Science 5 equation for a line of best fit for the data. Students in Engineering & Physical Science were required to use Desmos for all scatter plots produced during the course. Students in Physics were required to use Desmos for graphs of lab data associated with instruction over linear motion, then allowed to use the graphing tool of their choice for the remainder of the course. Data Sources

The first data source is the linear motion assessment, included in appendix A, given to all students enrolled in Physics. The assessment was given as a pre-test during the first week of the program, then repeated as part of a post-test during week four of the program. This assessment includes several questions which explicitly require students to use graph interpretation skills. The remaining questions address science content knowledge developed using lab exercises emphasizing graph analysis. During these lab exercises, students were required to use Desmos to produce graphs which were then used to develop a student understanding of key science concepts. This tool primarily assessed the first two objectives, namely students’ ability to generate science content knowledge via data analysis and students’ ability to recognize and describe trends in data. A score of 60%, or 8/13, on this assessment is considered mastery by the teachers who developed the assessment.

Next, classroom observations were conducted while students used Desmos through out the program. The evaluator first tracked whether students were able to successfully describe a trend in experimental data and determine the equation for a line of best fit when using Desmos, as described in two of the program objectives. The observations also attempted to assess whether students found Desmos to be easy to use by watching how much support students required beyond the initial instruction and how many students selected to use Desmos when given a choice.

The final data source was the student survey included in appendix B. This survey was delivered to all students in Physics and Engineering & Physical Science to record their attitudes regarding how well the program met its objectives. Students completed the survey during the school day to encourage a high response rate. A random selection of just over half of the students in these courses completed the survey. The survey was given during week 12 of the program. Results Linear Motion Assessment

Table 1 summarizes the results of the linear motion assessment used in Physics. Results are given for both the pre-test and the post-test as percentages of a total score. The gain represents the increase individual students achieved from the pre-test to the post-test. Note that all statistics increased from the pre-test to the post-test. One of the most important pieces of data from this table is the median gain of 30.8%, indicating acceptable growth in student understanding of the content over the course of instruction. Table 1: Linear Motion Assessment Results MEDIAN MAXIMUM MINIMUM RANGE PRE-TEST 46.2% 69.2% 0.0% 69.2% POST-TEST 76.9% 100% 61.5% 38.5% GAIN 30.8% 84.6% 15.4% 69.2%


Histograms of student scores for the pre-test and the post-test are shown in figure

1. Scores to the right of the orange bar achieved the benchmark for mastery while scores to the left fell short. Note that on the pre-test, only 19 out of 113 students scored at or above mastery. On the post-test, however, all students scored at or above mastery. Figure 1: Linear Motion Assessment Scores

Classroom Observations When using Desmos, students were able to consistently create a graph based on experimental data and describe any trends in the data in accordance with lesson objectives. In addition, students were able to determine the equation for a line of best fit.

During most lessons, students required minimal support beyond the initial instruction over the use of Desmos. In an experiment involving particularly small values, students did need assistance in setting an appropriate range to display. Most of the technical support provided by the instructor was related to Internet connectivity and other issues not directly related to Desmos.

When 12th grade students were provided the opportunity to select a tool for creating graphs, 72% of lab groups opted to use Desmos. Each group typically had several reasons for selecting Desmos. Nearly all groups cited the ease of use of Desmos, the ability to save and share graphs, and the ability to use Desmos on any internet-enabled device, including a smartphone. Student Survey

The student survey contained six Likert scale items where a 1 indicated strongly disagree and a 5 indicated strongly agree. Items were worded such that agreement indicated a positive attitude towards the use of Desmos. Table 2 contains the text of these items along with the identifiers used in the graphs of student responses.

0

5

10

15

20

25

30

35

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Num

ber o

f Stu

dent

s

Score (out of 13)

Linear Motion Assessment Results

Pre-Test

Post-Test

Scores at or Above Mastery Scores Below Mastery

Desmos in Physical Science 7 Table 2: Student Survey Question Summary Q1 It is easy to plot data in Desmos. Q2 Creating a graph in Desmos helps me to understand what my data means. Q3 It is easy to find the equation for a line using Desmos. Q4 When I use Desmos, I can focus more on what the graph means than on

making the graph. Q5 Desmos is a helpful tool for my science class. Q6 Desmos should continue to be used in science.

Figure 2 contains a histogram the total number of students who selected each

response for the Likert scale items. Most students selected either neutral or agree for all of these items, suggesting an overall neutral to positive attitude towards Desmos. Figure 2: Number of Responses by Question

Figure 3 provides the mean response for each Likert scale item. Mean responses are provided for students in each course, as well as for the overall population included in the evaluation. The mean responses range from 3.27 to 4, reinforcing the overall weakly positive attitude towards Desmos seen in figure 2. It is worth noting that students enrolled in Engineering & Physical Science tended to display a slightly more positive attitude towards Desmos than students enrolled in Physics.

0

5

10

15

20

25

30

Q1 Q2 Q3 Q4 Q5 Q6

Num

ber o

f Res

pons

es

Number of Responses by Question

Strongly Disagree (1)

Disagree (2)

Neutral (3)

Agree (4)

Strongly Agree (5)


Figure 3: Mean Responses by Question and Course

Figure 4 summarizes responses to the open-ended survey items by categorizing

each response as primarily positive or primarily negative towards the use of Desmos. In this section of the survey, students demonstrated a much more strongly positive attitude towards Desmos than in the Likert scale items with 82% of all respondents demonstrating an overall positive attitude in the open-ended questions. As seen in table 3 and figure 3, students enrolled in Engineering & Physical Science again showed a slightly more positive attitude towards Desmos than students enrolled in Physics. Figure 4: Open-Ended Response Summary

0.00 1.00 2.00 3.00 4.00 5.00

Q1

Q2

Q3

Q4

Q5

Q6

Mean Response

Mean Responses by Question and Course

Engineering & Physical Science Physics

All Students

0.00

20.00

40.00

60.00

80.00

100.00

Engineering & Physical Science

Physics Overall

Perc

ent o

f Res

pons

es

Open-Ended Response Summary

Primarily Positive

Primarily Negative

Desmos in Physical Science 9 Discussion This evaluation was intended to determine the effectiveness of Desmos in physical science courses at Tartan High School. The goal is to determine whether Desmos will be an appropriate complement to the BYOD program Tartan is scheduled to implement in spring 2015. The program has four identified objectives:

1. Students will be able to generate science content knowledge through data analysis, including graph interpretation.

2. Students will be able to recognize and describe trends in data. 3. Students will be able to determine the equation for a line of best fit. 4. Students will consider the graphing tool easy to use.

Based on the pre-test and post-test administered to students enrolled in Physics, classroom observations of students using Desmos, and the results of a survey completed by students, the program is meeting its intended objectives and should be continued with some modifications. Detailed discussion based on each data source follows. Linear Motion Assessment The scores on the linear motion assessment suggest that program successfully met objective 1 in that students were able to demonstrate a mastery of science content knowledge taught through lab exercises utilizing Desmos. The program also met objective 2 since on the assessment students were able to describe trends in the graphs included in the assessment. As indicated by figure 1 in the results section, only 19 out of 113 students achieved a score recognized as mastery on the pre-test, while all students scored at or above the cutoff for mastery on the post-test. In addition, the median score increased from 46.2% on the pre-test to 76.9% on the post-test with the median student’s score increasing 30.8% during the unit. These increases fall within the acceptable range for the Physics course. Classroom Observations The classroom observations were intended to assess objectives 2, 3, and 4. During these observations, all students were able to identify and articulate key patterns in experimental data after producing a graph using Desmos, providing additional data that the program has met objective 2. These observations also provide evidence that the program has met objective 3 in that all students were able to successfully produce a line of best fit and find the equation for that line when using Desmos. The observations also provide evidence that the program has met objective 4 in that students needed minimal support using Desmos beyond the initial instruction provided. Student Survey While the student survey includes elements of all program objectives, it most directly addresses student perceptions on objective 4, providing data on whether students perceive Desmos as easy to use. The survey suggests that students are overall positive, though not enthusiastic, toward the use of Desmos in their science classes. The response count in figure 2 shows that for all Likert scale items, most students selected either neutral or agree where agreement indicates a positive attitude towards Desmos. The mean responses in figure 3 are consistently above neutral. There is room for students to develop stronger agreement with the statements, but the program is on the right track with regard to objective 4.

10

Desmos in Physical Science

The open-ended questions included in the survey provide stronger evidence that the program has met objective 4. Figure 4 includes what fraction of the open-ended responses demonstrate an overall positive or negative attitude toward Desmos with 82% of the responses falling into the positive category. When asked what students like about using Desmos, many students described the tool as “easy” or “quick and easy”. A few students described Desmos as “confusing” when asked what they do not like about using it, but the responses also suggest that additional instruction may remedy the confusion. It is worth note that students enrolled in the 9th grade course, Engineering & Physical Science, tended to demonstrate more positive attitudes towards Desmos on both the Likert scale and open-ended items than the students enrolled in Physics, a 12th grade course; these differences can be seen in figures 3 and 4. Based on the open-ended responses, this may be in part due to what students consider an alternative for creating and analyzing graphs. The students in Engineering & Physical Science frequently compared Desmos to creating graphs by hand while the students in Physics frequently compared Desmos to using a TI-Nspire graphing calculator, a tool that few 9th grade students have. Overall Conclusions Given the positive results described above, the use of Desmos in Tartan’s physical science courses should continue. There is sufficient evidence that all four objectives are currently being met by the program. There are, however, opportunities for improvements. On the open-ended survey items, several students commented that they found Desmos confusing. Given the number of students who described Desmos as easy to use, additional instruction on the use of Desmos may not be an appropriate solution. However, students could easily be provided with resources including brief screencasts and illustrated how-to guides describing how to perform common tasks within Desmos. Another crucial theme from the open-ended portion of the student survey is the importance of the device used to run Desmos. During this program, most students used either personal smartphones or school-owned Windows netbooks. Many students commented that it was difficult to type or use many of the advanced features from a smartphone. A number of students also found that the small screen size of a smartphone made it difficult to get a clear view of the graph. When using netbooks, many students were frustrated by the long start-up time and the Internet connectivity issues frequently encountered with these particular devices. When Tartan’s BYOD program is formally implemented, students should have access to personal Chromebooks, tablets, or laptops which can be expected to significantly improve the student experience using Desmos. Project Cost

The primary cost in this evaluation was the daily rate for Marta Stoeckel, the principal evaluator. This includes five days of classroom observations in addition to the development and analysis of other data sources. The primary operational cost was photocopies of the student survey and content knowledge assessments. ITEM COST Marta Stoeckel: 10 days at $500/day $5,000 Operational Costs (photocopies, office supplies, etc) $100 Total Costs $5,100

Desmos in Physical Science 11

Appendix A: Linear Motion Assessment

12



14


Appendix B: Student Survey

Student Survey: Desmos in Physics & PhysicalScience

1. It is easy to plot data in Desmos.Mark only one oval.

1 2 3 4 5

Strongly Disagree Strongly Agree

2. It is easy to find the equation for a line using Desmos.Mark only one oval.

1 2 3 4 5


3. Creating a graph in Desmos helps me to understand what my data means.Mark only one oval.

1 2 3 4 5


4. When I use Desmos, I can focus more on what the graph means than on making thegraph.Mark only one oval.

1 2 3 4 5


5. Desmos is a helpful tool for my science class.Mark only one oval.

1 2 3 4 5



Powered by

6. Desmos should continue to be used in science.Mark only one oval.

1 2 3 4 5


7. What do you like about using Desmos to create graphs?

8. What do you dislike about using Desmos to create graphs?

9. Do you have any other thoughts in the use of Desmos in physics and physicalscience?

16


Appendix C: Open-Ended Survey Item Responses Notes on Interpretation and Analysis of Open-Ended Survey Items

The following pages contain student responses to the open-ended survey items in a table format. Each row represents the responses provided by a single student and responses have been separated by the course students are enrolled in. Blank cells indicate that the student did not respond to the item.

Each response was rated by the evaluator as primarily positive or primarily negative towards Desmos itself. Since this evaluation is concerned with Desmos itself, not the technology used to access it, this rating was based only on comments about Desmos as much as possible. In ambiguous cases, responses to the Likert scale items were considered to identify the student’s general attitude as positive or negative.


Engineering & Physical Science Responses Four surveys did not contain responses to these items. Those surveys were not included in the analysis. Q7: What do you like about using Desmos to create graphs?

Q8: What do you dislike about using Desmos to create graphs?

Q9: Do you have any other thoughts on the use of Desmos in physics and physical science?

Evaluator’s Rating

Its easy Nothing I like it Positive Its a easy way to put things into a clean graph

Its difficult at first

No Positive

I like messing around with it

Nothing No Positive

I don't really use it I prefer to make graphs by hand

Nothing Nope Negative

It's quicker than making one yourself

Some people don't know how to do it

No Positive

It was easier to find the slopes and graphing

At first, it was kinda confusing to use

No Positive

Easy Hard to find some things

No Positive

It makes the graph quick

Nothing Nope Positive

It takes a lot less time using Desmos than making your own

It gets kind of confusing sometimes

Positive

It's quick and easy Positive It makes the graph for me, so I don't have to take a while figuring out how to graph the lines

The netbooks are really slow

It should be continued being used.

Positive

18


Physics Responses Twelve surveys did not contain responses to these items. Those surveys were not included in the analysis. Q7: What do you like about using Desmos to create graphs?




Its fun to learn what different graphs mean

It's time consuming Nope Positive

It's different, useful when you don't have a graphing calculator

It's weird to use at first

nope Negative

When it actually works, I can zoom in on data

I never understood how to use it and I find it really confusing to graph things

I actually prefer using Google graphs, I think it's way easier, but I don't think you should eliminate Desmos because everyone prefers different things

Negative

It is helpful when a calculator is not available

It is sometimes hard to access

No

Negative

It is similar to Excel; easy to use

Certain things they don't have compared to Excel or maybe I just don't know

N/A Positive

Its simple to read and create them

Sometimes its difficult with extra settings

It could only be beneficial, not counter beneficial

Positive

Its relatively easy once its used a couple times and its easier to highlight and move data compared to Microsoft Excel

Its confusing at first to get the equation out and typing in data was somewhat tricky too

None really Positive


Q7: What do you like about using Desmos to create graphs?




It really helps me to visualize things

Nothing much. Maybe associating the picture with velocity/acceleration graphs is a little difficult.

I think it would be a great tool for the future.

Positive

I like the visual of it because I am a visual learner

Its kind of difficult to know how to do everything on it

Nope Positive

You just input numbers and a graph is made

On a mobile device it is very confusing

Much easier on a computer than on a mobile device

Positive

You can use it anywhere, you don't need the unreliable net books

Its a bit confusing Positive

The graph is very detailed

Its complicated to make

N/A Negative

More accurate than other programs

Hard to find what you're looking for

N/A Negative

I like that Desmos allows us to see the graphs and equations for real-time data

I have not found much dislike other than a few complications w/ the interface

Positive

Its simple and I can do it on my phone

Its small print so a little challenging to type and navigate

Nope Positive

Fast, easy, accurate

Hard to get perfect line of best fit

Nope Positive

20






I like that I'm able to find about everything I need when I do it correctly

Making sure that I do it correctly because if its wrong all the work I did along with it was wrong

No Positive

Lots of features, has everything you need mathematically

Positive

Allows me to interpret graphs in different ways

It can be confusing at times

No Positive

It is an easy tool to use

Great tool to introduce students to

Positive

It makes a graph I can do it just as easy on my calculator

No Negative

It shows you a picture of your data

Its on the computer, the name is weird, and its overly complicated

A TI-Nspire CAS is a far superior item for this level of physics than Desmos

Negative

It is extremely easy to make the graph and it is easy to understand

N/A It goes faster than on Google Docs, so I like it better, it should be used more often

Positive

It is a way that is simple if you know how to use it, but if you don't have instruction on what to do it isn't very self-explanatory

Its a pain to use it when I have a graphing calculator already

Negative






I like that you are able to create multiple lines on the same plane

Using the iPhone app things are hidden in menus

Positive

Its confusing to use sometimes

We should spend more time on how to use Desmos exactly

Negative

It helps to understand what is happening with the data

It is sometimes hard for me to understand

Positive

It gives us more time to work on other parts of what we are doing

Sometimes it would not let me enter data if I used it on my phone

Positive

It is a fairly easy tool to make accurate graphs quickly

Can be difficult to find the equation the first time or 2 you use it

Desmos should be used but it should have walk throughs on how to use it

Positive

Lots of functions to use

The app No Positive

Not really because the graph has to be extremely precise

That it always ends up in a decimal

Nope Positive

I like the convenience and accuracy as opposed to hand drawing them

The fact that I have trouble creating graphs in the first place

I have no other thoughts

Positive

The graphs are accurate and easy to interpret

The numbers get mixed up and can be hard to put in equations

Positive

22






Makes it easy to interpret the data and understand what it means

Can be hard to make depending on the circumstances

It should continue to be used!

Positive

It is generally user friendly

I was unfamiliar with the program at first, so it took a little while to get the hang of it

It should continue to be used

Positive

I don't find it helpful

It's too much work, I rather use formulas

No Negative

Its handy and better than other things

At first is hard to figure out how to type the stuff in

No Positive

Its a good visual aid

Have to use internet, easier to use a graphing calculator

N/A Negative

It is faster and more accurate

Positive

It makes everything easy

Positive

Documents

stoeckel evaluation report - Boise State Universityedtech2.boisestate.edu/martastoeckelrogers/docs/evaluation_report.pdf · Engineering & Physical Science Responses ... over linear