Using Log File Analysis to Evaluate Instructional Design

Using Log File Analysis toEvaluate Instructional Design

Kenneth W. Fansler

Rodney P. Riegle

Illinois State University04/10/23


What is a Web server log file? Records activity on a Web server “Who” visited What files were requested What time/date site was visited What page was visited first, last Not “blogs” (Web logs)


Elements of a log file How to access log files How to generate log file reports Log file analysis software prices and features The limits of log file analysis The ethics of log file analysis How to avoid unsound inferences from log files Toward a Model of Online Instructional Design

Analytics


Elements of a log file Simple text file Generated by

Web serversoftware

Tracks visitors’activity on aWeb site

Example Difficult to read

in raw format Controlled by

server admin.


How to access log files

Contact Web server administrator Files are VERY LARGE

Burn to CD or DVD network download

Ask for range of files by date (semester, year, etc.)


How to generate log file reports Use commercial

reporting softwarelike WebTrends®

Configure thesoftware to reportonly what you wantto see

Ease of use andexpense varies

Example


Log file analysis software prices and features Variety of software packages All perform same relative

basic function Some provide advanced

reporting functionality All rely on log files Price ranges from

free to >$500 Product list in paper


The limits of log file analysis

Difficult if not impossible to track usage by “real” person

IP address recorded Limited data recorded on authenticated visitors

Log files record only “request” transactions (client requests page from server)

No Flash No Java apps No cgi applications


The limits of log file analysis

Log file analysis software usually designed with business, not education, in mind

Results shown in static images (no motion) Easy to misinterpret the results Not useful to students’ self-evaluation


The ethics of log file analysis

Privacy Any identifiers must be kept confidential

Unsound Inferences Reporting information without revealing limits Tempting to report data as shown in analysis software


How to avoid unsound inferences from log files

Log file analysis can be misleading

Examples of possible misconceptions based on data analysis

1. Site and page hits are an indicator of the number of visits Different from “page hits” Includes multimedia files related to that html page



Examples of possible misconceptions based on data

2. All page or site hits are generated by students Search engine “spiders” or “robots” scour web Course authors’ visits are recorded Outside guests who may simply be browsing

3. Hits from a single Internet Protocol (IP) address reflect visits from a single user

Multiple users sharing a single computer One user moving from machine to machine Re-routing of IP addresses through ISPs Firewalls



Examples of possible misconceptions based on data

4. The “average time viewed” statistic accurately measures the amount of time someone spent on the Web page

“Walk-aways” Multiple browser windows Printers Timeouts


Toward a Model of Online Instructional Design Analytics Log file analysis currently driven by commercial

interests Customer behavior Raw traffic Purchasing statistics

Online student behavior more difficult to analyze Did learning occur? Engaged interactivity between student and instructor (email,

phone, etc.) Engaged interactivity among students (chats, discussions) Is “Student X” actually “Student X”?


Toward a Model of Online Instructional Design Analytics

Results Skepticism towards use of log files to analyze online

instruction Continued debate over general effectiveness of online

instruction Problems appear insurmountable to non-experts Inhibited research and theory in regard to using log file

analysis to evaluate online courses If problems can be solved, it will have a huge impact on

instructional evaluation and design



What’s needed? Identify what is currently possible Identify missing characteristics that could improve data

collection Create theoretical model of new tool(s) Work with commercial software designers to integrate

needed characteristics into new software



Characteristics necessary to improve data collection:

1. Log file data must be displayed in 4 dimensions 3 spatial dimensions plus time Replace static charts with motion charts (movies)

2. File data must be tied to individual students More accurate tracking of student activity Navigation tied to final grades, evaluations



Characteristics necessary to improve data collection:

3. Log file data must be integrated with more file types Flash, java applets, cgi programs, email Student navigation can be altered by a single email or discussion

4. Information generated must be accessible in real time by students

Useful to students who want to improve their performance Compare to other students’ behavior


Toward a Model of Online Instructional Design Analytics Questions still left to ask to support investigation

of new characteristics

1. Do different Web server software applications (IIS, Apache, etc.) log activity differently?

2. Do courseware packages (WebCT, eCollege, Blackboard, etc.) produce a secondary log that more closely tracks individual users’ activity?

3. Will newer versions of Web server software and Web client software work more closely together to provide more detailed logs (such as single-user activity)?


Questions, Suggestions

Ken Fansler ([email protected]) Rod Riegle ([email protected])

Documents

Using Log File Analysis to Evaluate Instructional Design