Learning system design considerations in creating an online learning environment

38 Performance Improvement • OCTOBER 2001

Leadership facilitators in a U.S. military organization needed a Web-based

learning environment. The overall aim of the project was to design a proto-

type that supported leadership facilitators’ knowledge development in the

content area of motivation. This learning environment was designed to pro-

vide new learning opportunities for facilitators to build content knowledge through

interactive learning activities and experiences with other members within the facilita-

tors’ community. For this learning environment to continually improve and be effective

in the long run, issues concerning diffusion of the technology and ongoing learning

environment development needed to be addressed.

The iterative design process used to develop this Web-based application is illustrated

in Figure 1. The human performance technology (HPT) phases of analysis, design,

development, implementation, and evaluation are often thought of as unique, stand-

alone events. This project was managed as a concurrent solution design process, incor-

porating elements of strategic thinking, change creation, work environment design, and

rapid prototyping.

HPT consultants analyze and evaluate performance to effectively design and develop

human performance improvement solutions that reduce the gaps between current and

desired performance. With today’s technologies and the speed of business, designers

have no time to figure out what works and what doesn’t, or to understand why some-

thing works when it does.

The components of the iterative design process address a wide array of performance

analysis, solution design, development, implementation, and evaluation issues. The

use of this compressed design process allows for increased efficiency in making ongo-

ing refinements and increased effectiveness in making improvements to solution pro-

totypes. The designers of this project used the learning environment design principles

identified in the next section to drive the major tasks and to ensure project quality.

Learning System Design ConsiderationsIn Creating an OnlineLearning Environmentby Scott P. Schaffer and Shawn M. Overcast

Additionally, actions considered necessary for ongoing sup-port of the web-based environment were summarized in twomajor areas: diffusion of the technology into the facilitatorcommunity, and continuous improvement and adaptationof the site by the community.

Learning Environment Design Principles

The project team agreed on three broad areas of learningenvironment design. These components address solutionanalysis, design, development, implementation, evaluation,and diffusion issues. They are useful as guides to match pro-ject needs with learning environment features. For the sakeof brevity, these principles have been listed, but not fullyelaborated below.

Analyzing

• Identify people who are accountable for learners’ resultsand engage them in the design goal-setting process.

• Identify components and levels of the organizationalsystem where performance-based results depend onlearning achievement.

• Determine measures of learning success and perfor-mance on an individual and organizational basis.

• Plan how you will evaluate this success (pre- and post-learning).

• Identify nontraining, work environment factors thatcould contribute to the success or failure of the Web-based learning environment.

• Create an awareness and knowledge of the analysisprocess.

• Plan for diffusion and adaptation of the learning envi-ronment within the organization.

Designing

• Create learning objectives from performance objectives,based on task, job, and performance analyses.

• Create activities that gain learners’ attention by engagingthem in the learning process.

• Create activities that show learners how the instructionwill benefit them.

• Provide ways for learners to practice learning activitiesand get feedback on their performance.

• Create ways for learners to assess their own performancein a way that simulates the actual performance, as muchas possible.

• Create ways for learners to access a common knowledgebase of resources on a given topic.

• Provide ways for learners to interact with instructors andwith other learners.

• Provide ways for learners to gain multiple perspectiveson a particular problemsolving approach.

• Provide a means for learners to access knowledge basedon previous learners’ experiences.

• Engage members of the learning community or organiza-tion in adapting the learning environment for a better fitwithin the current culture.

Continuously Improving

• Collect data to measure the learners’ success and specificlearning environment design processes.

• Identify nontraining, work environment factors thatcould contribute to the success or failure of the Web-based learning environment.

• Adapt learning design processes and adapt environmentto optimize learners’ performance.

• Collect data to measure performers’ and organizations’success in meeting goals as related to learning activitiesand objectives.

These key principles were used as the foundation for apreprototype evaluation plan. The plan included ques-tions derived from a variety of performance improvementtheories and models. A solution design decision aidassisted in evaluating the effectiveness of the learning andperformance environment. Figure 2 is an excerpt from thisdecision aid.

To accomplish the objectives of this particular project, theiterative design phases (performance analysis, prototypedesign, and prototype redesign) were integrated with thedesign principles to produce a functional prototype.

Performance Analysis

Performance analysis is a process of defining the results theleadership community expects and facilitator practices thatcould lead to achievement of those results. Current anddesired facilitator results, practices, and attitudes were ana-lyzed during this phase. Measurable results, critical perfor-mance and learning measures, achievable facilitatorpractices, and work environment factors and conditions thatsupport or hinder performance were profiled. Prototypedesign began after performance objectives and the flow ofinformation in the leadership facilitator community wereidentified. The results of analysis guided the planning forthe evaluation, design, and implementation of the learningand performance environment solution.

Performance Improvement • Volume 40 • Number 9 39

Figure 1. Iterative Design Process.

PrototypeDesign

PerformanceAnalysis

Re-Design


Figure 2. Solution Design Decision Aid.

Prototype Design and Redesign (Continuous Improvement)

Participatory design processes are effective in buildingstakeholder and user commitment to the success of applica-tion systems. For this project, each design iteration andimprovement was based on discussions with key sponsors,stakeholders, and potential users. Participants and design-ers collaborated to develop the product, identify possibleproduct improvements, and generally provide guidance inthe ongoing design of the product.

During this concurrent design and evaluation process, inter-views and focus groups were conducted, during whichvisual prototypes clarified concepts. Initial prototypes werePowerPoint™ slides, which were transformed into a proto-type Website using simple Web-design tools. To assist inthis transformation, members of the target audience pro-vided initial feedback regarding the site look and feel andoverall concept. Following initial feedback, sponsors held a“joint application design” (JAD) meeting, during which thefirst relatively primitive Web-based site was reviewed live.They reviewed the site screen by screen, each accompaniedby a set of structured interview questions. The purpose ofthis meeting was to collect data to assist in design decision-making. A major goal of this design iteration was to ensurethat the organizational learning and performance objectivesof the site were addressed.

Following this meeting, revisions were made accordingly toprepare the site for usability testing and evaluation. Theevaluation process included, but was not limited to, assess-ment of the aesthetic quality of the interface, and—follow-ing redesign—assessment of the content, learning activities,and achievement of objectives.

The first level of evaluation was with facilitators at a largetraining unit with more than 100 facilitators. Once facilita-tors learned the basic goals of the site, they went to the siteand provided feedback. Following this process, these evalu-ation processes were also completed:• Learner/Performer Evaluation—A survey was placed on

the Website to gather input from facilitators regardingtheir reactions to site content and learning activities, aswell as to glean suggestions for improvement. Users alsohad the opportunity to provide feedback on the sitediscussion board.

• Individual interviews with novice users were conductedto observe a typical user interaction and resolve any bar-riers to navigation and learning. The user went througheach activity and made suggestions for the content andlayout of the site.

Specific strategies and recommendations related to utiliza-tion and ongoing support of the prototype were proposed.These strategies and recommendations were focused in two

key areas: adopting and diffusing the product within the tar-get organization, and continuously improving the designand usefulness of the current prototype once adopted by thetarget organization.

Adopting and Diffusing the Product: ImplementationStrategies. The most well-designed products often gounused or unappreciated within organizations because theylack sponsorship and adequate attention to the diffusion ofthe new technology. During the performance analysis phaseof this project, many nontraining, work environment factorsthat could contribute to the success or failure of the Web-based learning environment were identified. Critical diffu-sion and work environment factors are listed below tohighlight some of the biggest challenges to project successwith this particular target audience. Recommendations rel-ative to these factors have also been included.

Redefining/Restructuring and Clarifying Relationships/Roles. These two critical stages in the diffusion of a tech-nology within an organization have been documented byresearchers in many different organizational settings(Rogers, 1995). Following initial design, development, andevaluation of the Web-based learning environment, it wasexpected that members of the facilitator community wouldbegin the process of redefining/restructuring and clarifyingroles related to this technology (see also Wenger, 1998).

The following recommendations were put forth to assist inthe initial adoption and redefining/clarifying of the proto-type. Long-term evaluation of the site is possible only afteradoption and utilization of the prototype by members of thefacilitator community:• Identify decisionmakers who will advocate the site—key

stakeholders within the facilitator community who takeresponsibility for the product. The project sponsor and


Continuous improvement andinstitutionalization of thelearning environment withinthe leadership facilitatorcommunity depended on thequality of the design process,participation of the stake-holder groups in develop-ment, and the degree towhich it helped them achievedesired performance results.

individual training units will be responsible for identi-fying key adopters, change agents, and problemsolverswho will champion the use of this innovation by mem-bers of the facilitator community. These role players willbe active participants in the design, redesign, and evalu-ation of the application.

• Identify implementation managers who will diffuse thecritical aspect of the early design and development ofany new product. These are early adopters that mustassume the burden of “selling” the product to others, aswell as taking on much of the responsibility for its suc-cess. One strategy that stakeholders may use is to inte-grate this tool with other performance support toolscurrently in use.

• Introduce the site during ongoing activities already exist-ing in the environment.

• Introduce the site during initial facilitator mentoring.• Have trainers and mentors introduce the site as a tool for

individual and job development. The site could helpexperienced facilitators mentor and coach less-experi-enced facilitators. Facilitators may be positively influ-enced to use the site if their mentors encourage it as auseful way to develop skills and build new knowledge.

• Introduce the site as part of facilitator training.Introducing the site as a tool that can be used by facilita-tors to build knowledge and skills related to job dutiesand responsibilities will increase its potential impactand overall effectiveness.

Recommendations

Resources for Ongoing Design and Development

Identification of adequate resources—human and financial—to support the maintenance and continued growth of theapplication is crucial to success. This is an often-neglectedaspect of computer-based application development, espe-cially if the technology requires frequent updating. Web-basedtechnologies are easier to update than other technologies andthus represent a potential long-term cost savings.

Recommendations: • Form a team responsible for learning and redefining the

Website. Although time and monetary requirements are min-imal, it will be essential to identify key personnel who willbe responsible for supporting ongoing maintenance of thesite. Internal facilitators who are familiar with the facilitatorcommunity and current practices can be trained in changeand redesign processes. A technical support resourceresponsible for updating and refining software and hardwarewill also be required for continual site improvement.

• Conduct prototype review groups with potential usersand opinion leaders. Continual feedback from the learn-ing community is helpful in redefining and redesigningthe site to support and build on current course content.

Incentives

Incentives and reward systems are one of the least under-stood yet most effective methods of improving performancein the workplace. The benefits of participating in the learn-ing environment include improved performance, self-development, and professional growth.

Recommendations:• Official “release” time for participation in learning

activities, and recognition for participation are highlyrecommended. The costs of release time can be consid-erable but the potential benefits of participation, interms of improved performance, are well worth the cost.

• Facilitators should receive positive consequences andencouragement from their command to use the site aspart of their ongoing skill development.

Process Management

Technical and human resource management will berequired to ensure consistent, supportive, and performance-oriented system implementation. The inter-relationship ofthis new process with other facilitator processes is part ofthe role of project leaders.

Recommendations:• Leaders of this project within the facilitator community

must be fully engaged in the process of creating, refining,communicating, and using knowledge that is supportedby technology.

• Periodic feedback should be elicited from facilitators tocontinually improve and add areas of interest to facilita-tors who are actually using the site.

Equipment

Hardware and software, physical information infrastruc-ture, and documentation will be required to support initialand ongoing system development.

Recommendation: • Current computer hardware and software must meet

minimum specifications. This is probably the mostexpensive (monetarily) of all implementation factors butcan pay off in learning and performance effectiveness inthe long term.

Continuous Improvement of the Learning andPerformance Environment

The major components of the Web-mediated learning andperformance environment address the learning objectivesrequired to achieve selected facilitator performance require-ments, or best practices. By addressing the best practices


that were identified through the performance analysis, thisenvironment has a greater likelihood of having a positiveimpact on facilitator, performer, and organization readiness.

There are several components in this environment.Descriptions of these learning environment componentsand recommendations for continuous improvement andmaintenance follow.

Practice

The purpose of this element is to enhance content knowl-edge and problemsolving ability in the area of motivationthrough a variety of self-assessment/content mastery activi-ties. These activities include multiple choice, matching and“essay”/case study response formats. They should enhanceauthentic learning and critical thinking skills.

For ongoing development, the following steps are recom-mended:• Monitor and update self-assessment items to reflect cur-

rent course content.• Continually develop new items that challenge users and

reflect current research and practice in the area of moti-vation.

• Ensure relevant, corrective feedback for each item.• Add tutorials that provide a review of classroom mater-

ial for each level of the course.• Identify current problems to which facilitators can apply

basic problemsolving skills, thus providing the tools tosolve larger issues/concerns.

• Monitor case study submissions to ensure that learningopportunities go beyond those presented in the classroom.

• Continually add new case studies. Expert facilitators canupdate and maintain relevant case studies.

• Review submissions by peers and revise as necessary.• Revise case studies as field experiences change or as cur-

rent case studies become outdated.• Hold expert forums in the chat room and archive

interview transcripts.

Discussion Board

The discussion board will provide current learners with aplatform to build usable knowledge for current and futurefacilitators and leaders. Features of such an interactiveboard would include a message board with a chat feature toshare ideas, experiences, and discussions, with synchro-nous and asynchronous communication options.

Ongoing development would need to cover two areas. 1. Cleaning up/Monitoring

• Continually monitor information for accuracy.• Delete outdated or incorrect information.• Review other discussion boards that may better suit

user requirements.

2. Maintaining Interactions• Periodically post new discussion topics.• Encourage participation by “graduates” who are now

on the job.• Provide positive consequences for facilitators to

interact with others on the discussion board.• Reward expert facilitators or subject matter experts

who participate in scheduled discussions.Help

The help component provides assistance and reduces frus-tration in navigating through the site. Features include a sitemap, technical help with frequently asked questions (FAQ),and site tips.

This function needs ongoing updating and revision: • Monitor and delete/update FAQs. Facilitators should be

encouraged to ask questions of their peers and torespond to questions appropriately.

• Revise site map/navigational tips based on revisions andadvancements to the site.

Knowledge Base

The knowledge base provides learners with a repository ofresources for known facts, theories, and procedures. It willalso promote construction of new knowledge of current andfuture facilitators through critical reviews of the resources.

Features of a knowledge base include a library with motiva-tion-related readings, instructional resources, and links toother organization Websites.

For ongoing development, the following points are important:• Monitor and update/delete “dead links.”• Links should reflect current course content and should

not present views conflicting with current curriculum.• Expert contributions should be archived and continu-

ously updated.• Archived models/theories should be created and easily

accessed.• A drop-down menu can be created so that reactions to

information found in Web links can be accessed undereach link in the library.

Evaluation

Evaluation provides an opportunity for the user to continuallydevelop the site. Users complete a profile that identifies themby location and level of experience, among other factors, andsolicits their feedback on the site. This profile is also suitableas a Level 1 (Kirkpatrick, 1994) reaction evaluation tool.

For ongoing development, these steps are required:• Log and continuously monitor evaluation responses to

assist in usefulness of data in continuous site improvement.


• Survey questions (as well as the profile) should bereviewed based on changes to the site and evaluationdata expected.

• As updates are made to the site, face-to-face evaluationsshould be conducted to observe interactions and toreceive feedback from facilitators regarding ease of navi-gation and usefulness of activities.

Site Technical Maintenance

The designers have provided sufficient information in a sitemaintenance guide for an information technologist tounderstand the basic architecture and technical features ofthis product for the purpose of continual technical develop-ment. Where applicable, possible alternative applications tothe ones used in the prototype design have been suggested.

There are various popular online course development soft-ware applications. WebCT, Blackboard CourseInfo, PhoenixPathlore, and Construe are just a few examples.Alternatively, this site has been created with a variety ofCOTS (commercial off-the-shelf) products that alloweddesign flexibility for ultimate user customization.

Reflections

The term “blended solutions” has recently become popularamong HPT practitioners. This project models a form ofblended design in which multidimensional perspectives con-verged during completion of a design project. These perspec-tives have roots in the disciplines of communications andpsychology and spring forth from the applied fields of man-agement science, learning and performance systems design,and information systems design. Project team memberswhose skills and experiences were shaped by these appliedfields worked collaboratively to gain client engagement andcontinued participation in the design and redesign processes.Design principles served as a way to plan for the concurrentevaluation of the design process. Continuous improvementand institutionalization of the learning environment withinthe leadership facilitator community depended on the qual-ity of the design process, participation of the stakeholdergroups in development, and the degree to which it helpedthem achieve desired performance results.

References

Dick, W., & Carey, L. (1996). The systematic design of instruc-tion (4th ed.). Reading, Massachusetts: Addison-Wesley.

Ely, D.P. (1999). Conditions that facilitate the implementa-tion of educational technology innovations. EducationalTechnology, pp. 23–27.

Gilbert, T. (1978). Human competence: Engineering worthyperformance. Amherst, Massachusetts: HRD Press, Inc.

Kaufman, R. (1998). Strategic planning plus: An organiza-tional guide (revised ed.). Newbury Park, California: SagePublications.

Keller, J. (1987). The systematic process of motivationaldesign. Performance & Instruction, 26 (9), 1–8.

Kirkpatrick, D.L. (1994). Evaluating training programs: The4 levels. San Francisco: Berrett-Kohler.

Nielsen, J. (2000). Designing web usability: The practice ofsimplicity. Indianapolis: New Riders Publishing.

Rogers, E.M. (1995). Diffusion of innovations (4th ed.). NewYork: Free Press.

Wedman, J., & Graham, S. (1998). The performance pyra-mid. The Journal of Continuing Higher Education, 46 (3),8–20.

Wenger, E. (1998). Communities of practice: Learning as asocial system [Online]. Soquel, California: CommunityIntelligence Labs. Available: http://www.co-i-l.com/coil/knowledge-garden/cop/lss.shtml.

Scott P. Schaffer has led many performance improvement projects in theeducation, financial services, and high-technology sectors as a performanceconsultant. Areas of expertise include: employee selection, competencymodeling, leadership and career development, and performance systemdesign. Currently a doctoral student in the Instructional Systems program atFlorida State University, he is also a research associate for the LearningSystems Institute. Scott has published professionally and presented at sev-eral professional conferences including ISPI and ASTD. Scott may bereached at [email protected].

Shawn M. Overcast has a keen interest in helping organizations developtheir performance capacity. Research and professional interests include thedevelopment of learning and performance support tools. Shawn has engagedclients in a variety of performance improvement arenas, including online learn-ing and performance community, electronic performance support systems,and leadership evaluation. She is a graduate of the Instructional Systems pro-gram at Florida State University, and is a research associate for the LearningSystems Institute. Shawn may be reached at [email protected].


Individual CD: $69

❏ CD 1 ❏ CD 2 ❏ CD 3

❏ CD 4 ❏ CD 5

❏ Complete Set of 5: $159

Special Applications of

HPT CD: $89

❏ SCD 1

Listen to the presentations and review the speaker’s handouts simultaneously for the performance improvementsubjects below.

CD1Evaluation and Continuous Improvement Focus: Evaluating performance and performance improvement interventions. This CD features 20 presentationsand handouts related to this topic.

Process or Tool InterventionsFocus: Selecting, designing, developing and delivering work processes and performance support tools. Includesresults achieved by the intervention. This CD features 33 pre-sentations and handouts related to this topic.

CD2Analysis of Performance Improvement Problems Focus: Identifying performance expectations, gaps, requirements and needs. This CD features 19 presentationsand handouts related to this topic.

CD3

The Performance Improvement PractitionerFocus: Developing professional and personal skillsets for internal and external practitioners. This CD features25 presentations and handouts related to this topic.

Name:

Organization:

Address:

City, State/Province:

Post Code: Country:

Tel: E-mail:

ORDER YOUR CD-ROMS NOW!

Submit form with payment to: ISPI, 1400 Spring Street, Suite 260

Silver Spring, MD 20910 USA

Tel: 301-587-8570 / Fax: 301-587-8573

www.ispi.org

Performance Improvement Presentationsand Handouts on CD-ROM

CD4Instructional InterventionsFocus: Selecting, designing, developing and delivering instructional interventions. This CD features 20 presenta-tions and handouts related to this topic.

CD5Organization and Strategic InterventionsFocus: Integrating performance improvement interventionsfrom multiple classifications or categories to have animpact at the organization level. This CD features 30 pre-sentations and handouts related to this topic.

SCD1Culture and Change ManagementFocused on harnessing the power of culture and change to improvean organization’s bottom-line results. Learn to integrate culture-dri-ven “best practices” into your organization—this CD features 34 pre-sentations and handouts related to this topic.

SPECIAL APPLICATIONS OF HUMAN PERFORMANCE TECHNOLOGY (HPT)

The Business of HPT Focus: Getting HPT into organizations and managingthe HPT business as either an internal or externalsenior professional. This CD features 27 presentationsand handouts related to this topic.

The above CD-ROMs are $69 each or$159 for the complete set of five.

The Special Applications of HPT CD-ROM is available for $89 each.*All CD-ROM presentations were recorded at various ISPI Conferences.

Total Amount: US$____________

❏ Check Enclosed

❏ AMEX ❏ VISA ❏ MC Exp. Date ____________

Card # _____________________________________

Name on Card _______________________________

Signature ___________________________________

Total Amount:

Documents

Learning system design considerations in creating an online learning environment