Embed Size (px)
Citation preview
DOCUMENT RESUME
ED 474 555 IR 021 910
AUTHOR Cap, Ihor
TITLE Motivational Design Quality, Internal Benchmarking andStatistical Analysis of Corporate Information Materials: APilot Study.
PUB DATE 2002-10-00
NOTE 21p.; Paper presented at the Annual General Meeting of theAmerican Society for Quality Manitoba Section (Winnipeg,Manitoba, Canada, October 30, 2002).
PUB TYPE Reports - Research (143) Speeches/Meeting Papers (150)EDRS PRICE EDRS Price MF01/PC01 Plus Postage.
DESCRIPTORS Employers; Employment Services; Evaluation Methods; ForeignCountries; *Material Development; Mental Retardation;Nonprofit Organizations; *Printed Materials
IDENTIFIERS Manitoba; Motivational Design
ABSTRACT
This paper reports on an exploratory pilot study at NetworkSouth Enterprises (NSE) (Manitoba), a non-profit, community-basedorganization with the purpose of providing employment services for adultswith mental disabilities. The study used a focus group, an open-endedquestionnaire, and a 36-item InfoMMS scale to determine the motivationalappeal of NSE print materials and contents. Seven employers took part in thepre-benchmarking phase of the study to determine any difficulties with thecurrent performance levels of NSE corporate information materials. Anotherseven employers were later used to determine the performance levels and theeffects of newly developed treatment materials and contents. The studyprovided vital internal pre-benchmarking stage pretest data for: (1)
estimating the reliability of the InfoMMS administration; (2) estimating theperformance levels or usefulness of existing corporate materials; (3)
clarifying design, development, and use of future corporate informationmaterials and preferences; and (4) providing a ready bank of baseline controldata against which future materials and associated benchmarking partner datacan be compared. (MES)
Reproductions supplied by EDRS are the best that can be madefrom the original document.
MOTIVATIONAL DESIGN QUALITY, INTERNALBENCHMARKING AND STATISTICAL ANALYSISOF CORPORATE INFORMATION MATERIALS: A
PILOT STUDY
By
Ihor Cap, Ph.D.
A Paper Presented at the Annual General Meeting of The AmericanSociety for Quality Manitoba Section October 30, 2002, held at
The Viscount Gort Hotel,Winnipeg, Manitoba, Canada
Thor Cap is Program Development Coordinator with Manitoba Education, Training and Youth,Apprenticeship Branch. He holds an Education Specialist degree and Ph.D. in ComprehensiveVocational Education from the Florida State University and a Masters of Education in InstructionalTechnology from the University of Manitoba. He is a Member of the Ukrainian Academy of Artsand Sciences in Canada, and Chair of the American Society for Quality - Manitoba Section.
PERMISSION TO REPRODUCE ANDDISSEMINATE THIS MATERIAL HAS
BEEN GRANTED BY
I. Cap
TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)
12 Wisconsin St.Winnipeg, Manitoba
Canada R2M [email protected]
2
U.S. DEPARTMENT OF EDUCATIONOffice of Educational Research and Improvement
EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)
el This document has been reproduced asreceived from the person or organizationoriginating it.
Minor changes have been made toimprove reproduction quality.
Points of view or opinions stated in thisdocument do not necessarily representofficial OERI position Cr policy.
BEST COPY AVALABLE
2
INTRODUCTION
When you choose t o read a printed p age, you think there is some intrinsic qualityabout it, and this conception of the "quality" is part of a motivational structure whichestablishes priorities about your reading experience. To ask what an individual wants to readis to a large extent to ask what this person wants to experience. When this journey must beplanned, the effective designer asks what catches a reader's eye or what does a reader finduseful, memorable and gratifying, for these are the essential features which nourish qualityand give rise to a climate for reading.
Many businesses, corporate conglomerates, profit and non-profit organizations arejust beginning to reflect a renewal of interest to examine or incorporate these motivationalaspects in the process of designing their promotional material, and or parallel practice. Theyhave a huge role t o p lay i n m arketing themselves and their services to their stakeholders.They can be quite boisterous and quick about letting us know about what marketing mediathey use to dispense such information. Brochures, employer information packages, posters,newspaper ads, video commercials are not unusual, and more recently web sites among themare frequently used. They are a little quieter, however, when it comes t o telling u s a boutwhether the contents of their messages actually have any appeal to their stakeholders. NotNetwork South Enterprises, Inc.(NSE). That's because NSE has been busily engaged inupgrading their marketing initiatives in the hopes of improving their services. They not onlywant employers to receive their messages, but also expect employers to perceive thesemessages as motivationally appealing. As such, this NSE Business Advisory Councilvolunteer research consultant was specifically charged to focus on improving what worked,and provide recommendations to rectify what didn't while ensuring that the needs of NSE arebest being served.
NSE is a non-profit, community based organization begun in 1991. NSE is fundedprimarily through the generous support of Human Resources Development Canada andManitoba Department of Family Services. It was founded with the principal purpose ofproviding employment services for adults with mental disabilities. Their mission, then, is tosupport adults with mental disabilities to work and participate in the community, where theyare respected and rewarded for their e fforts, s kills and a ccomplishments. T hey d o t his b yworking cooperatively with employers throughout the city of Winnipeg who share the samevision. Together they are making significant inroads in achieving the mission. Over 600disabled workers were placed to date. That's a tremendous achievement and commitment toaction.
This exploratory pilot study was limited to using a focus group, an open-endedquestionnaire and a 36-item InfoMMS scale. The latter scale attempts to document themotivational appeal of print reading materials and contents in an objective format so that theresearcher is able to understand and interpret the employer's experience as accurately aspossible. As such, seven employers took part in the prebenchmarking phase of this study todetermine any difficulties with the current performance levels of NSE corporate informationmaterials. Another seven employers were later used to determine the performance levels andthe effects o f n ewly d eveloped treatment m aterials and c ontents. P urposive o r j udgmentalsampling was employed because NSE Business Advisory Council members appealed to theirown judgement and experience to select people for the sample who would best attend to theresearch questions of the study. Owing to these limitations, the results of this inquiry should
3
3
be used primarily to inform and stimulate future research. As indicated above, the study useda two foci approach to internal benchmarking and product improvement. Quantitative andqualitative. The former using objective instruments, and the latter employing an open-endedquestionnaire, a focus (reference) group of employers and a Business Advisory Council fromNSE to guide the design, development and evaluation decision-making process. The report issubdivided into the pre- benchmarking stage and benchmarking stage. An analysis anddiscussion of results, observations and concluding comments are provided.
PRE-BENCHMARKING STAGE
The Research Design
The prebenchmarking stage o f research w as c onducted i n M arch 2 3, 1 999 u sing asmall focus group of employers and a posttest only group design. This was required to set thestage for development. A Focus Group session was held to establish baseline measurementdata of existing NSE corporate information materials and contents from an employer'sperspective. The measurement data in this stage is called baseline because they s how theemployers' natural responses to existing NSE materials and contents without experimentalmanipulation. The Focus Group also responded to three open ended survey questions aboutthe existing materials. What did they like best about the existing NSE Employer InformationPackage? What did they like least about it? What recommendations did they have forimprovement? A face to face discussion followed immediately afterward, the results ofwhich were tabulated in Table 4 of the Appendix. This researcher examined over 30Corporate Information Brochures in the interim.
The benchmarking stage of research was conducted in November 29, 2000 using asmall alternate group, a small experimental group and a posttest two-group crossover orcounterbalanced design. In the counterbalanced design, two treatments (i.e., 2 newlydesigned brochures) were administered in November 29, 2000 to each employer. The orderof treatments was varied across subjects to control possible confounding of treatment effectswith order effects. That is, employers received treatments in either of the sequences X1, X2 orX2, X1 depending on what side of the table they sat on when they walked in to theboardroom. Each treatment was followed up with a means measurement of the study'sdependent variable, the InfoMMS scale. Comparisons were then made on the effects of thesetwo treatments using the March 23, 1999 baseline group data as the "control." The basicpattern of the counterbalanced design can be diagrammed as follows:
Period 1 XI Alternate Brochure --0' 0 X2 Experimental BrochureIP' 0
Period 2 X2 Experimental Brochure .°. 0 X1 Alternate Brochure °. 0
Where: X1= Alternate treatment brochureX2 = Experimental treatment brochure0 = means measurement or observation (posttest)
The two periods in this pilot test were the same lengths (of time). Thus, both treatmentswere administered in each time period rendering the comparisons between treatments free of
4
4
period effects. In other words, rotating the sequence in which different treatments werepresented t o subjects was done in an effort to control for extraneous carry-over variables,such as practice and fatigue effects. Relevant to the choice of design under discussion, Borgand Gall (1989) indicate the following advantages of employing counterbalancedexperiments.
The advantage of assigning subjects to several treatments is that the experimentcan be done with fewer subjects. Thus, subject recruitment is easier, andfinancial expenses of conducting the experiment may be reduced. Anotheradvantage is that the statistical analysis of the data is more sensitive because eachsubject is "matched" with himself across treatments. (p.709)
The Model
The study used the ARCS research-supported motivation model which considers fouressential human characteristics and the motivational strategies associated with them (Keller,1987a, 1987b). They include: (1) strategies that would mobilize and sustain theATTENTION o f y our audience, ( 2) strategies t hat w ould facilitate RELEVANCE t o t heirpersonal needs or goals, (3) strategies to assist CONFIDENCE, or the audience's expectancyfor successful understanding of contents, and (4) SATISFACTION strategies linked to theprocess or outcomes of the experience.
Instrumentation
A posttest analysis was used to ascertain the perceptions of employers as to whetheror not the contents presented in NSE's existing Employer's Information Package weremotivationally appealing or useful. Evidence of usefulness was based upon participants'responses to the modified Instructional Materials Motivation Survey (Keller, 1990). Thevalidity of this scale's four motivational components has been supported by concurrentvalidity and field testing of the ARCS model which defines each of these components(Keller, 1987a, p.2). Minor wording changes to this scale were completed by this researcherto facilitate interpretation of information contents rather than instructional contents. As such,the instrument, a 36-item questionnaire, surveyed four variable employer motivations (i.e.,attention, relevance, confidence, and satisfaction) in relation to the information contentsgiven. 0 verall reactions t o the 3 6 statements on this questionnaire were rated on a Likertscale (1 or A=not true, 5 or E=very true). The scoring procedure was reversed for statementskeyed in negatively. The reliabilities of the modified scale, hereafter referred to as theInformation Materials Motivation Survey (InfoMMS), within the March 23, 1999 referencestudy sample showed good internal consistency. The revised InfoMMS Scale proved to bevery sensitive. While reliability of one subclass scale (Confidence) was only moderatelyhigh, the remaining subclass scale r eliabilities w ere h igh for the Attention and R elevancecategories, and very high for the Satisfaction category. Evidently, the InfoMMS Scale gainsin effectiveness by the inclusion of all its content areas (subscales). Reliabilities as measuredby Cronbach's alpha coefficient are given in Table 1.
Pearson's Skewness (nonsymmetry) coefficient, given by SK=3(mean -
median)/standard deviation, for the distribution of the InfoMMS scores was found to be .41.
5
5
This suggests that the attribute termed motivational appeal or usefulness of contentspresented was symmetrically distributed and approaching normality within the March 23,1999 baseline study sample. The fact that coefficient values must fall between -3 and 3, witha value of 0 yielding a perfectly symmetrical distribution undoubtedly encourages thisfinding (Freund & Williams, 1982, p.59). Accordingly, the tendency for InfoMMS rawscores to show a normal distribution adds considerably to any description or interpretation ofstudy data. The attribute was considered a comprehensive or representative set of interrelatedideas pertaining to the four variable motivational categories within the InfoMMS instrument:Attention, Relevance, Confidence and Satisfaction.
Statistical Analysis
Descriptive statistics were used to determine central tendency. Missing data within agiven dimension of the 36-item InfoMMS scale were replaced with the overall mean - i.e.,the sample mean across all cases for a given scale item making up the scale. This wasperformed for 2 Attention subscale items (#17 and #22) in the Alternate Group for Caseparticipant number 9 in the November 29, 2000 study results. Analysis of variance(ANOVA) was used to investigate the significance of motivational differences betweentreatment conditions that were being compared. The ANOVA replaced the t-test because "Fand t are directly related and the analysis of variance is an extension of the test" (McCall,1975, p.263) to accommodate two or more groups. The index Effect Size (ES), defined as"the mean difference between the treated and control subjects divided by the standarddeviation of the control group," (Smith, Glass & Miller, 1980, p.68) was used to evaluate themagnitude of experimental and alternate group brochure effects against the reference (focus)study group (used here as the "control") in standard deviation units. The reference studygroup was convened at the pre-benchmarking stage session as part of the Focus Group inMarch 23, 1999 to establish measurement baseline data of existing NSE corporateinformation materials and contents in a format (i.e., Information Materials MotivationSurvey and Open Ended Questionnaires) for ease of subsequent comparison at thebenchmarking stage (i.e., November 29, 2000 treatment group effects determination andadvanced analyses). ES outcomes and standards were already reviewed by Schermer (1988)and adopted by this researcher to facilitate consistency in the interpretation of outcomes. Inquantitative terms, point size estimates of less than .2 were considered "small" effects, .5 as"medium" in size and higher than .5 as "large." With these benchmarks in mind, an estimateof the effect magnitude was calculated over the posttest InfoMMS measure. In general, theresearcher set a hypothesis presented in null form b ased o n standard e ducational researchconventions. Namely, that there would be no motivational brochure appeal differencebetween the alternate brochure and experimental brochure employers. The reader shouldknow that the researcher's preferred outcome was to show no motivational brochure appealdifference between alternate and experimental group employers since these were newlydesigned brochures. The intent was to obtain equal appeal between both groups of readers.The hypothesis would be rejected or not rejected according to a standard decision rule, asdescribed by Brewer (1986, p.5-23).
If p is less than or equal to a, the Ho is rejected
If p is greater than a, the Ho is not rejected.
6
In this context, the probability value was increased to the .10 level of significance in order toreduce the chances of making a Type II error. Along with manual calculations, all data wereanalyzed using the Statistical Package for the Social Sciences version 8.
March 23,1999 Small Focus (Reference) Group Study Results
The InfoMMS was used to establish reference or baseline group data and to ascertainthe overall "usefulness" of contents presented or motivational appeal engendered by theexisting NSE Employer Information Package as perceived by the employers. Descriptivestatistics and overall reactions to the InfoMMS questionnaire are shown in Table 1 below.
Table 1 - Descriptive Baseline Statistics Relevant to March 23, 1999 Small Focus(Reference) Group Employer Session (N=7)
Variable PossibleScore
MeanScore/(%)
SD Median Mode MeanLikertRating
Cronbach' sAlpha
InfoMMS 180 126 (70%) 22.16 123 120 3.5 0.9171Attention 60 38.71 (65%) 7.70 38 28 3.2 0.7673Relevance 45 29 (64%) 7.72 29 25 3.2 0.8368Confidence 45 37.57 (84%) 4.47 40 41 4.2 0.6417Satisfaction 30 20.86 (70%) 5.55 20 20 3.5 0.9199
Table 1 reveals that all of the components were at the moderate to high level in eachmotivational category. Together, the four motivational categories averaged 126 points(standard deviation = 22.16) for a moderate percentage performance of 70%. From the"sample percentage ± 2 Standard Errors " (Freedman, Pisani & Purves, 1978), we learn thatthe 95% confidence interval for this percentage ran from 66% to 74%. On this basis, therewas no convincing evidence that the population percentage or percentage performance inother samples would exceed a high degree of acceptance or fall below a moderate level ofacceptance should the same or similar NSE Employer Information materials continue to beused. Here, the "population percentage" means the percentage of all female or maleemployers between 20 to 59 years of age who prefer to read in English, have no disabilitythemselves, may or may not have a family member with a disability and know someone witha disability.
The author labeled percentages between 0 and 50 as very low, 51 to 70 as low, 71 to 86 ashigh and above 86 as very high for the purposes of educational research. These shortclassifying phrases facilitated consistency in the interpretation of acceptance of brochurecontents (i.e., direction and intensity of appeal) or percentage performances for theinstrument used in this study.
7
7
BENCHMARKING STAGE
November 29, 2000 Small Alternate Group and Experimental Group Study Results
The InfoMMS was used to ascertain the overall "usefulness" of contents presented ormotivational appeal engendered by the Alternate and Experimental Group Employerbrochures. The InfoMMS scale surveyed four variable employer motivations (i.e., attention,relevance, confidence, and satisfaction) in relation to the contents given. Overall reactions tothe four motivational categories contained within the InfoMMS questionnaire were rated ona Likert scale (1 or A=not true; 5 or E=very true). The resulting performance levels (i.e.,uppercase or capitalized for Alternate Group and lowercase for Experimental Group) werecompared in terms of the scale's four motivational components and portrayed on an inverted-U curve as Figure 1 below.
COMBINED AUDIENCE CHARACTERISTICS
PercentagePerformance
Very Hi 1
Hi 801-
Moderate (:1
/ 6I
40I
1
Lo 201
I
011
Very Lo2 3
Lo4
Hi very Hi
Motivational Appeal (Usefulness) Level(Mean ratings on 5-point Likert scale)
Figure 1. November 29, 2000 InfoMMS POSTTEST GRAPH SHOWING MOTIVATIONAL APPEALOF ALTERNATE GROUP (N=7) AND EXPERIMENTAL GROUP (N=7) BROCHURE ACCEPTANCE LEVELS ASPERCEIVED BY EMPLOYERS
It reveals that the Alternate Group Employer brochure mobilised and sustained theAttention of employers to a high degree (of acceptance). The 49.9 average (or 4.2 Likertrating) along this dimension translated into a high percentage performance of 83%. Thecontents were Relevant to their personal needs or goals to a high degree to an establishedaverage of 34.1 points (3.8 Likert rating) or a percentage performance of 76%. Confidence,or the employers' expectancy for successful recall or understanding of contents averaged avery high 40.6 (4.5 Likert rating or 90%), and Satisfaction with the process or outcomes ofthe experience averaged a high 22 (3.7 Likert rating or 73%). It is of interest that all of thecomponents were at high to very high levels in each motivational category. None of theemployers were undermotivated in the process or as a result. Together, the four motivationalcategories averaged 146.57 points (standard deviation = 13.58) for a high percentageperformance of 81%.
8 BIEST COPY AVARLA 112
8
The Experimental Group Employer brochure mobilised and sustained the Attentionof employers to a high degree (of acceptance). The 43.6 average (or 3.6 Likert rating) alongthis dimension translated into a percentage performance of 73%. The contents were Relevantto their personal needs or goals to a high degree to an established average of 33 points (3.7Likert rating) or a percentage performance of 73% as well. Confidence, or the employers'expectancy for successful recall or understanding of contents averaged a high 36.6 (4.1Likert rating or 81%), and Satisfaction with the process or outcomes of the experienceresulted in a low performance averaging 18.4 points (3.1 Likert rating or 61%). Motivationalperformance levels ranged from low to high. Together, the four motivational categoriesaveraged 131.57 points (standard deviation = 16.98) for a high percentage performance of73%.
The overall Alternate Group InfoMMS sample mean 146.57 or overall samplepercentage 81% estimates the average of a population, but to indicate the precision of thisestimate this researcher gave a confidence interval. From Freedman, Pisani, & Purves (1978,p.345) we learn that the interval "sample percentage ± 2 Standard Errors" has a confidencelevel of about 95%. The 95% confidence interval for this percentage ran from 75% to 88%,as illustrated in Figure 2 below. Thus, the researcher can be reasonably confident ofdetecting a high to very high overall percentage performance between 75% and 88%.
2SE 2SE4 <
75% 81% 88%Samplepercent
Figure 2. 95% Confidence interval on overall Alternate Group InfoMMSSample Percentage.
The range of this estimate is exact given a normal population distribution and "... isapproximately correct for large n in other cases" (Moore & McCabe, 1989, p.512). On thisbasis, there was no convincing evidence that the population percentage or percentageperformance in other samples would exceed a very high degree of acceptance or fall below ahigh level of acceptance should the same or similar Alternate Group Employer brochurematerials continue to be used. Here, the "population percentage" means the percentage of allfemale or male employers between 20 to 59 years of age who prefer to read in English, haveno disability themselves, may or may not have a family member with a disability and knowsomeone with a disability.
The overall Experimental Group I nfoMMS s ample mean 131.57 o r overall samplepercentage 73% estimates the average of a population, but to indicate the precision of thisestimate this researcher gave a confidence interval here as well. The 95% confidence intervalfor this percentage ran from 64% to 82%, as illustrated in Figure 3 below. Thus, theresearcher can be reasonably confident of detecting a low to high overall percentageperformance ranging between 64% and 82%.
9
9
2SE 2SE4 k-
64% 73% 82%Samplepercent
Figure 3. 95% Confidence interval on overall Experimental Group InfoMMSSample Percentage.
Accordingly, there was no convincing evidence that the population percentage or percentageperformance in other samples would exceed a high degree of acceptance or fall below a lowlevel of acceptance should the same or similar Experimental Group Employer brochurematerials continue to be used. The low performance in Satisfaction appears to undermine theoverall high percentage performance of the Experimental Group Employer brochure. Here,the "population percentage" means the percentage of all female or male employers between20 to 59 years of age who prefer to read in English, have no disability themselves, may ormay not have a family member with a disability and know someone with a disability.
SUPPLEMENTARY RESULTS
Observed Significance LevelsTable 2
ANOVAs for the Posttest on Information Materials Motivation Survey (InfoMMS) as aDependent Variable
Sum ofSquares
MeanSquare
df FValue
p
InfoMMS Between Groups 787.500 787.500 1 3.333 0.093 *Within Groups 2835.429 236.286 12
3622.929 13
A Between Groups 138.286 138.286 1 4.478 0.056 *Within Groups 370.571 30.881 12
13
R Between Groups 4.571 4.571 1 .282 0.605Within Groups 194.857 16.238 12
199.429 13
C Between Groups 56.000 56.000 1 6.497 0.026 *Within Groups 103.429 8.619 12
159.429 13
S Between Groups 44.643 44.643 1 1.957 0.187Within Groups 273.714 22.810 12
318.357 13
* Significant at the .10 alpha level
10
10
Only 14 observations were used in this analysis. The omnibus analysis of variancerevealed a significant difference by group, df=1, F=3.333, p=0.093. It can be seen in Table 2on the analysis of variance that the group variable was significant at the .10 alpha level.Thus, the null hypothesis of no motivational brochure appeal difference between thealternate and experimental conditions was rejected according to the decision rule.Supplementary ANOVA results also showed significant differences between the groups onthe Attention (p=0.056) and Confidence (p=0.026) sub-component categories. The analysisdid not reveal any significant differences for the Relevance (p=0.605) and Satisfaction(p=0.187) sub-component categories.
Effect Sizes
The initial observations show that the two treatment brochures differ significantly inusefulness but tell us little about their effectiveness. On this basis, it is premature to concludethe treatments (that is, the newly designed experimental and alternate brochures andcontents) had any effect. So, to estimate how large the treatment effects could really be, thisresearcher compared the average effect sizes for the Alternate and Experimental Groupsagainst the reference data from the Focus Group session which was used as the baseline"control." These are outlined in Table 3 below.
Table 3
SPSS Summary Statistics for Post Study Groups on Information Materials Motivation Survey(InfoMMS) as a Dependent Variable versus Reference (Focus) Study "Control" Group
Reference"Control" Group
(N=7)March 23 1999 Study
Alternate Group
(N=7)Nov 29, 2000 Study
Experimental Group
(N=7)Nov 29, 2000 Study
Attention Factor X=38.7 X=49.86 X=43.57SD=7.70 SD=4.74 SD=6.27
ES=1.5 ES=.63Relevance Factor X=28.8 X=34.14 X=33
SD=7.72 SD=3.58 SD=4.44ES=.69 ES=.54
Confidence Factor X=37.6 X=40.57 X=36.57SD=4.47 SD=3.51 SD=2.23
ES=.66 ES=.23Satisfaction Factor X=20.9 X=22 X=18.43
SD=5.55 SD=4.44 SD=5.09ES=.2 ES=-.45
IMMS Totals X=126 X=146.57 X=131.57SD=22.16 SD=13.58 SD=16.98
ES=.93 ES=.25
II
11
Effects of Alternate Group against Reference "Control" Group
There was an overall effect size gain of .93 standard deviation units - a large effect.Thus, the ES of .93 also demonstrates that Alternate Group InfoMMS change has beenincreased to a large degree. An estimate of the effect magnitude was calculated over theposttest InfoMMS measure, per motivational usefulness factor as well. Effect size gains weredistributed as follows: 1.5 standard deviations (SDs) for Attention - a large effect, .69 forRelevance a large effect, .66 for Confidence - a large effect, and .2 SDs for Satisfaction -approaching a small effect. More importantly, these measures consistently reflect some gainsin each of the brochure content areas.
The immediate overall effect size gain of .93 on the InfoMMS scale also means thatthe "average" employers receiving the untreated baseline brochure who scored at the 50thpercentile level would see their usefulness (or motivational appeal) scores rise to the 82ndpercentile level if provided the alternate treatment brochure. This represents a gain of 32percentile points over the untreated "control" condition. It also suggests that the overalleffect of the alternate treatment brochure is very engaging. As for interpreting effect sizes byway of percent of nonoverlap (See Cohen, 1988, pp.21-23) of the alternate treated group'sscores with those of the untreated baseline "control" group, the ES of .93 indicated anonoverlap of about 52% in the two distributions.
Effects of Experimental Group against Reference "Control" Group
There was an overall effect size gain of . 25 s tandard d eviation units a m oderateeffect. Thus, the ES of .25 also demonstrates that Experimental Group InfoMMS change hasbeen increased to a moderate degree. As well, an estimate of the effect magnitude wascalculated over the posttest InfoMMS measure, per motivational usefulness factor. Effectsize gains were distributed as follows: .63 standard deviations (SDs) for Attention - a largeeffect, .54 for Relevance - a large effect, .23 for Confidence - a moderate effect, and -.45SDs for Satisfaction - a negative effect. Except for the moderately negative Satisfactionmeasure, these data consistently reflect some gains in each of the brochure content areas.
The immediate overall effect size gain of .25 on the InfoMMS scale implies a 50th tothe 60th percentile gain in the direction and intensity of usefulness (or motivational appeal)scores by the average experimental treatment brochure employers. This represents a gain ofonly 10 percentile points over the untreated "control" condition. It also suggests that theoverall effect of the experimental treatment brochure is not that motivating. Interpreting theeffect size by way of percent of nonoverlap of the experimental treated group's scores withthose of the untreated baseline "control" group, the ES of .25 indicated a nonoverlap of about18% in the two distributions.
DISCUSSION OF RESULTS AND FINDINGS
Summary of Overall Study Findings
There appear to be four major findings in this exploratory pilot study. The overallfindings of this study were that
12
12
1. In general, both newly designed treatment brochures experienced positive effectsize gains when compared to the untreated baseline "control" condition contents.
2. The immediate overall effect size gain of .25 on the experimental treatmentbrochure InfoMMS scale implies a 50th to the 60th percentile gain in the directionand intensity of usefulness (or motivational appeal) scores by the averageexperimental treatment brochure employers. The gain of only 10 percentile pointsover the untreated "control" condition suggests that the overall effect of theexperimental treatment brochure was not that motivating. The ES of .25 alsoindicated a nonoverlap of about 18% of the experimental treated group's scoreswith those of the untreated baseline "control" group.
3. There was an immediate overall effect size gain of .93 on the alternate treatmentbrochure InfoMMS scale. This means that the "average" employer receiving theuntreated baseline brochure who scored at the 50th percentile level would see theirusefulness (or motivational appeal) scores rise to the 82nd percentile level ifprovided the alternate treatment brochure. The gain of 32 percentile points overthe untreated "control" condition suggests that the overall effect of the alternatetreatment brochure was very engaging. The ES of .93 also indicated a nonoverlapof about 52% of the alternate treated group's scores with those of the untreatedbaseline "control" group.
4. Supplementary ANOVA results s howed a significant difference b y group. T hisconfirms that employers who received the alternate treatment brochure perceivedit as significantly more useful (or motivationally appealing) than the experimentaltreatment brochure.
Observations
The overall F-test result showed significant differences between the Alternate Groupemployers' and Experimental Group employers'. From the point of view of the ARCS model,employers perceived Alternate Group materials and contents much more positively becausethey significantly mobilized and sustained their attention by a much higher degree, thecontents were perceived as more relevant to their personal needs or goals, their confidencelevels for successful understanding of contents were significantly higher and as a result, theywere m ore s atisfied w ith the p rocess o r o utcomes o f t his experience. Thus, there is somesupport for Arredondo's (1991, p.22) truism that "if it doesn't get and keep their attention, itcan't be meaningful, memorable, or activating." The proportionally greater effect size gainsin all ARCS model subcomponent motive factors favouring the Alternate group support thisview. Effect size gains for the Alternate Group brochure and its contents as a whole was .93standard deviation units (- a large positive impact with about 52% nonoverlap) compared to.25 standard deviation units (-a slightly moderate positive effect with about 18% nonoverlap)for the Experimental Group brochure. In retrospect, while both brochures engendered greater"usefulness" or motivational appeal among employers than the initial Baseline "Control"
13
13
Group employer information materials, it seems clear that the results favoured the alternatebrochure group of employers.
Conclusions and Recommendations
The results of this exploratory study may have practical implications. It has beenfound that corporate produced information materials can be more readily accepted into aclient's motivational schema if the target audience itself has input into the change (orlearning) process. As such, it is not inconceivable that a wider degree of generality mayeventually b e found i n s imilar research s ituations and s amples. Given the complexities ofcorporate, employer and researcher values and the prohibitive costs of undertakingeducational research, it is recommended that a focus group be used as a qualitativesupplement to any corporate decision-making change effort dealing with formativeevaluation. The focus group gave useful feedback and suggestions for revision of existingmaterials and contents. This phase of the study provided vital internal pre-benchmarkingstage "pretest" data for (1) estimating the reliability of the InfoMMS questionnaire, (2)estimating the performance levels or usefulness of existing corporate materials, (3) clarifyingdesign, development and use of future corporate information materials and preferences, and(4) providing a ready bank of baseline "control" data from which future materials andassociated benchmarking partner data can be compared against. Researchers, practitionersand program planners are encouraged to use these findings in conjunction with a focus groupin the design of future activities related to corporate information materials and contentsdevelopment. For all practical intends, formative production and evaluation efforts shouldseriously consider using multiple samples and repeated testing procedures to ensurematerials are interpreted in the way intended. Central to this post-benchmarking process isthe calculation and reporting of effect sizes which provide objective and standardizedmeasures of mean differences attributed to the exposure of groups treated to a modificationin comparison to a reference (baseline) or "control" group t hat does not receive a qualitydesign change opportunity. Newly established effect sizes should be discussed foreffectiveness and further usefulness. The results of these analyses showed that theexperimental brochure and the alternate brochure in particular were more useful than thebaseline "control" brochure. The degree of usefulness was represented by an effect size of.95 for the Alternate Group and .29 for the Experimental Group, both of which were positive.For the larger effect size, the difference in the distributions was obvious and there was verylittle overlap. For the smaller effect size, the two distributions overlap a great deal and theeffect was difficult to see. Accordingly, the experimental brochure had a weaker effect and issusceptible to further instability. Thus and so, the alternate group effect sizes arerecommended as the re-calibrated standard (i.e., updated future reference benchmark).Alternate group data s howed that the employer was more likely to react positively on theposttest InfoMMS measure if the employer performs in the high range on eachsubcomponent ARCS factor. Additional supporting evidence showed that the 95%population percentage for the Alternate Group was 81%, and the likely range was 75% to88%. In other words, the p ercentage p erformance i n other s amples would not tend to fallbelow a high level of acceptance should the same or similar Alternate Group employerbrochure materials continue to be used. This tendency can be verified by a larger sample ofsubjects to provide a narrower precision of the likely range. However, unlike significance
14
14
tests, effect-size statistics are not dependent on sample size which makes their reportingessential to any research. It has been shown that to remain successful, promotional materialsand contents need to achieve and maintain motivational acceptance standards in the highrange, but only continued review processes will ensure continued effectiveness andusefulness.
In the final analysis, the presence of a negative aftereffect on audience responsemight be real. Such effects may or often go unnoticed on the radar screen because they tendto occur at the subcomponent level as opposed to a global one. So, it is incumbent upondesigners and researchers to measure for these aftereffects or to measure the phenomenonsusceptible of curtailing ill effects in favour of improving product and materials effectivenessor usefulness. Those who are dazzled by "attractive" or "smart" looking materials andcontents may unintentionally markdown the "tame" looking ones when the latter mayproduce the most desirable effect/s. More research is needed to explain the sometimesuneasy choices that an audience makes. This study speaks to that void. The use ofdescriptive and inferential statistics, corporate advisory committees and focus groups eachhave something equally valuable to contribute to product development, and therefore shouldbe used in conjunction with each other. The general conclusion to be drawn from thisexploratory formative evaluation study is that the same or similar uses of internalbenchmarking and statistical analysis provide one avenue to fostering quality in themotivational design of corporate information materials.
15
15
REFERENCES
Arredondo, L. (1991). How to present like a pro:Getting people to see things your way.New York: McGraw-Hill, Inc.
Borg, R.W., & Gall, D.M. (1989). Educational research: An introduction (5th ed.). WhitePlains, NY: Longman.
Brewer, K. J. (1986). Introductory statistics for researchers. Tallahassee, Florida: BellwetherPress, a Division of Burgess International Group, Inc.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ:Laurence Erlbaum.
Freedman, D., Pisani, R., and Purves, R. (1978). Statistics. New York, N.Y.: W. W.Norton & Company.
Freund, E.J., & Williams,J.F. (1982). Elementary business statistics: The modernapproach (4th ed.). Englewood Cliffs, New Jersey: Prentice-Hall.
Keller, M.J. (1987a). Strategies for stimulating the motivation to learn.Performance & Instruction, 26(8), 1-7.
Keller, M.J. (1987b). The systematic process of motivational design.Performance & Instruction, 26(9), 1-8.
Keller, M.J. (1990). Manual for the Instructional Materials Motivation Survey.Unpublished manuscript, Occassional Paper (draft), Florida State University,Department of Educational Research, Tallahassee.
McCall, B.R. (1975). Fundamental statistics for psychology. New York: Harcourt BraceJovanovich.
Moore, S.D., & McCabe, P.G. (1989). Introduction to the practice of statistics.New York: W.H. Freeman and Co.
Schermer, J. (1988). Visual media, attitude formation, and attitude change in nursingeducation. ECTJ, 36(4), 197-210.
Smith, L.M., Glass, V.G., & Miller, I.T. (1980). The benefits of psychotherapy.Baltimore, Maryland: The John Hopkins University Press.
1.6
16
APPENDIXA
Small Focus (Reference) Group Baseline Session DataAnd Brochure Descriptions
17
Table 4.Descriptions of Open Ended Survey and Qualitative Portion of March 23, 1999 SmallFocus (Reference) Group Employer Session (N=7) Data by Frequency of Occurrence
What did you like best about the existing NSE Employer InformationPackage? (8 1/2" x 11" format)
Accomplishments/Achievements (N=2)Comprehensive Information (N=2)Honesty/Real Life Examples (N=2)History of Org. (N=2)
Mission Statement (provides objectives to work with) (N=2)
Employer's responsibilities/Needs of Participants (N=2)Systematic/Logical Sequence/Methodology (N=2)Easy reading (N=2)Newsletter (N=1)Frequently Asked Questions by Employers (N=1)Liked it all (N=1)What did you like least about it?
Repetition (N=2)Literary usage lacks passionate appeal (N=2)About NSE info, introduced to late (N=1)First two pages somewhat daunting, tempted to skip them (N=1)The organisation chart, role, reporting, etc., unclear (N=1)Few graphics (N=1)Few examples (N=1)Design (N=1)Better understanding of message/information required (N=1)No uniqueness in service (N=1)Did not like much at all about it. (N=1)
Employer Recommendations for Improvement
Include Pictures/Graphs/Illustrations and/or the like (N=3)List/Describe Successes/Testimonials (N=2)List Current Employers/Companies using programs (provides prospective
employers greater feeling of confidence) (N=2)
About NSE info up front (N=1)Impact of NSE (N=1)Use colour (N=1)Back to back use of Information on Sheets (N=1)Supply Employer references at time of application (N=1)Use greater appeal in contents (N=1)Uniqueness/differences in services (N=1)Format Preference
Pamphlet/Brochure = (N=3)
Existing 8 1/2 x 11" = (N=3)
No Response (N=1)
.18BEN COPY AVELABLIE
18
Facilitator Face to Face Portion of Focus Group Session Results
Package is easy to readPackage is well organised, it evolves or follows a logical orderContent should be condensedMore examples of clients we work with, less info on each company, but
more company examplesGet endorsements from clientsCompanies interested in knowing companies have used our servicesMention how long we have worked with a company, how long consumer
employed by companyExamples/testimonials spread throughout documentUse company logosRemove repetition from literatureUse more visualsInfo package was clear, it tells you what you need to knowPurpose of the org has to 1st thing you seeUse referencesNetwork South Logo/mission statement should be on front of the folderBox important informationInclude info on screening applicants
Do they fill out application forms (give example)Do we have evaluations (give example)
:BEST COPY /WAXLIKE
19
19
Some Elements Incorporated Into The Experimental Brochure
Panel 1 Outside Front Cover (Far Right)3/4 " corporate logo (left justified), name of corporation (rightjustified) alongside corporate logocolours and hues of corporation adopted throughoutsingle message tag line (right justified 3 inches below logo. Itreads "Connecting your business to supported employment services"Also 3 black and white 1" x 1 1/4" pictures positioned below tagline vertically aligned and separated by 1/8" space eachBottom of brochure caption reads "An Information Guide ForEmployers"
Panel 2 Left Inside CoverWho Are We? 1/4" Header followed by informationPhilosophy 1/8" Sub- header and information
Mission 1/8" Sub-header and informationHistory 1/8" Sub-header and information
Panel 3 Middle of Inside CoverWhy Hire From Us? 1/4" HeaderBenefits 1/8" Sub-header and informationEmployer Services 1/8" Sub-header and bulleted informationStaff 1/8" and information
Panel 4 Far Right Inside Cover
Two employer endorsements with statements enclosed in shaded boxPrograms 1/4" Header with bulleted information
Panel 5 Outside Cover (Far Left)
Our Employer Partners Have Included... (Employers listed)
Panel 6 Middle Outside CoverDesign and Layout donated by acknowledgements (top of panel)
Printing Courtesy of acknowledgements (top of panel)1/2" corporate logo bottom of panel with corporate name and contactaddress points etc...
Note: Inside Cover contents positioned NSE Inc. as experts in thefield.
BEST COPY AVAREABIS
20
Some Elements Incorporated Into The Alternate Brochure
Panel 1 Outside Front Cover (Far Right)line bar on top on top of panel
2" x 2" corporate logo (top center location)
colours and hues of corporation adopted throughoutsingle message tag line (5 1/2" below top of panel). It reads"Building Positive Relationships With Local Business"
Bottom of panel caption spells out corporation's name
Line bar on bottom of panel
Panel 2 Left Inside CoverMission Statement 1/4" Header onlyPhilosophy 1/8" Sub-header and informationRole and Mission 1/8" Sub-header and informationHere Are The Facts: 1/8" uppercase Sub-header followed by 3 bulletsof information
Panel 3 Middle of Inside CoverNetwork South Enterprises 1/2" Header across top of panel
Here's How We Work uppercase 1/8" Sub-header followed by 4 bulletsof informationSoft Watermark of corporate logo spanning panels 2 through 4Employer Services 1/8" uppercase Sub-header followed by 2 bullets ofinformation
Panel 4 Far Right Inside CoverEmployer Benefits 1/8" uppercase sub-header and small descriptionfollowed by 4 bullets of informationContact information bottom of panel
Panel 5 Outside Cover (Far Left)
Here is what people have to say about Network South Enterprises Inc.Statement on top of panel followed by 1 Employer endorsement, 1Parent endorsement, 1 NSE Graduate endorsement and 1 NSE EmploymentConsultant endorsement
Panel 6 Middle Outside CoverBrochure Set-Up and Design Courtesy of Acknowledgements (top centerof panel)
Brochure Printing Courtesy of acknowledgements (center of panel)
NSE corporate name spelled out in addition to corporate support andsponsorship acknowledgement statements (bottom of panel)
Note: Inside Cover contents used less descriptive informationstatements and relied more on bulleted information points. No picturesor visuals outside the watermark supported the contents.
21 131ES7 COPY AVAHILAIBILIE
RE
PRO
DU
CT
ION
RE
LE
ASE
- S
peci
fic
Doc
umen
t
1. D
OC
UM
EN
T I
DE
NT
IFIC
AT
ION
:
4/0
Ma
Aul
C4
ro.d
ecal
m; vq
j-Ss
fYi
U.S
. Dep
arim
ento
fEdu
catio
nO
ffic
e of
Edu
cato
nal R
esea
rch
and
Impr
ovem
ent (
OE
RI)
Nat
iona
l Lib
rary
of
Edu
catio
n (N
LE
)E
duca
tiona
l Res
ourc
es I
nfor
mat
ion
Cen
ter
(ER
IC)
RE
PRO
DU
CT
ION
RE
LE
ASE
Page
1 o
f 2
SE
RIC
(Spe
cifi
c D
ocum
ent)
Erl
adoi
tlfitt
atts
trib
ritd
iti ta
rs
Titl
e:
Aut
hors
:
Cor
pora
te S
ourc
e:[MOTIVATIONAL DESIGN QUALITY, INTERNAL BENCHMARKINC AND
STATISTICAL
ANALYSIS OF CORPORATE INFORMATION MATERIALS: A PILOT STUDY
ihor
Cap
Not
App
licab
le'P
ublic
atio
n D
ate:
II. R
EPR
OD
UC
TIO
N R
EL
EA
SE:
In o
rder
to d
isse
min
ate
as w
idel
y as
pos
sibl
e tim
ely
and
sign
ific
ant m
ater
ials
of
inte
rest
to th
e ed
ucat
iona
l com
mun
ity, d
ocum
ents
ann
ounc
ed in
the
mon
thly
abst
ract
jour
nal o
f th
e E
RIC
sys
tem
, Res
ourc
es in
Edu
catio
n (R
IB),
are
usua
lly m
ade
avai
labl
e to
use
rs in
mic
roR
che,
rep
rodu
ced p
aper
cop
y, a
nd e
lect
roni
c m
edia
, and
sold
thro
ugh
the
ER
IC D
ocum
ent R
epro
duct
ion
Serv
ice
(ED
RS)
. Cre
dit i
s gi
ven
to th
e so
urce
of
each
doc
umen
t, an
d, if
rep
rodu
ctio
n re
leas
e is
gran
ted,
one
of
the
follo
win
g no
tices
is a
ffix
ed to
the
docu
men
t.
If p
erm
issi
on is
gra
nted
to r
epro
duce
and
dis
sem
inat
e th
e id
entif
ied
docu
men
t, pl
ease
CH
EC
K O
NE
of
the
follo
win
g th
ree
optio
ns b
elow
and
sig
n at
the
botto
m o
fth
e pa
ge.
The
sam
ple
stic
ker
show
n be
low
will
be
affi
xed
to a
ll L
evel
I d
ocum
ents
The
saw
pk s
ticke
r sh
own
belo
w w
ill b
e af
fixe
d to
all
Lev
el 2
A d
octu
neat
sT
he s
ampl
e tic
ker
show
n be
low
will
be
affi
xed
to a
ll L
evel
2B
s
docu
men
tsPE
RM
ISSI
ON
TO
RE
PRO
DU
CE
AN
D D
ISSE
MIN
AT
E T
HIS
MA
TE
RIA
L H
AS
BE
EN
GR
AN
TE
D B
YPE
RM
ISSI
ON
TO
RE
PRO
DU
CE
AN
D D
ISSE
MIN
AT
E T
HIS
ME
RIC
CO
LL
EC
TIO
N S
UB
SCR
IBE
RS
ON
LY
HA
S B
EE
NA
TE
RIA
L I
N M
ICR
OFI
CH
E, A
ND
IN
EL
EC
TR
ON
IC M
ED
IA F
OR
GR
AN
TE
D B
Y
PER
MIS
SIO
N T
O R
EPR
OD
UC
E A
ND
DIS
SEM
INA
TE
TH
ISM
AT
ER
IAL
IN
MIC
RO
FIC
HE
ON
LY
HA
S B
EE
N G
RA
NT
ED
BY
SAM
PLE
SAM
PLE
SAM
PLE
TO
TH
E E
DU
CA
TIO
NA
L R
ESO
UR
CE
S IN
FOR
MA
TIO
N C
EN
TE
R(E
RIC
)F
TO
TH
E E
DU
CA
TIO
NA
L R
ESO
UR
CE
S IN
OR
MA
TIO
N C
EN
TE
R(E
RIC
)
TO
TH
E E
DU
CA
TIO
NA
L R
ESO
UR
CE
S IN
FOR
MA
TIO
N C
EN
TE
R(E
RIC
)
file
://U
:\ica
pkho
med
tive\
WPD
AT
AV
ASQ
%20
mis
c\lh
or's
%20
Mot
ivat
iona
l%20
Des
ign%
20Pr
esen
tatio
n\R
EPR
OD
UC
TIO
N%
2...
2002
-12-
31
RE
PRO
DU
CT
ION
RE
LE
ASE
- S
peci
fic
Doc
umen
t
Leve
l 1Le
vel 2
A
Che
ck h
ere
for
Leve
l I r
elea
se, p
erm
ittin
g re
prod
uctio
n an
ddi
ssem
inat
ion
in m
icro
fiche
or
othe
r E
RIC
arc
hiva
l med
ia (
e.g.
,el
ectr
onic
) an
d pa
per
copy
.
Che
ck h
ere
for
Leve
l 2A
rel
ease
, per
mitt
ing
repr
oduc
tion
and
diss
emin
atio
in m
icro
fiche
and
in e
lect
roni
c m
edia
for
ER
IC a
rchi
val c
olle
ctio
nsu
bscr
iber
s on
ly..
Page
2 o
f 2
Leve
l 2B
C.
Che
ck h
ere
for
Leve
l 213
rel
ease
, per
mitt
ing
repr
oduc
tion
and
diss
emin
atio
n in
mic
rofic
he o
nly.
Doc
umen
ts w
ill b
e pr
oces
sed
as in
dica
ted
prov
ided
rep
rodu
ctio
n qu
ality
per
mits
.If
perm
issi
on to
rep
rodu
ce is
gra
nted
, but
no
bos
is c
heck
ed, d
ocum
ents
will
be
proc
esse
d at
Lev
el I.
I he
reby
gra
nt to
the
Edu
catio
nal R
esou
rces
Inf
orm
atio
n C
ente
r (E
RIC
) no
nexc
lusi
ve p
erm
issi
on to
rep
rodu
ce a
nd d
isse
min
ate
this
doc
umen
t as
indi
cate
d ab
ove.
Rep
rodu
catio
n fr
om th
e E
RIC
mic
rofi
che
or e
lect
roni
c m
edia
by
pers
ons
othe
r th
an E
RIC
em
ploy
ees
and
its s
yste
m c
ontr
acto
rs r
equi
res
perm
issi
on f
rom
the
copy
righ
tho
lder
. Exc
eptio
n is
mad
e fo
r no
npro
fit r
epro
duct
ion
by li
brar
ies
and
othe
r se
rvic
e ag
enci
es to
sat
isfy
info
rmat
ion
need
s of
edu
cato
rs in
resp
onse
to d
iscr
ete
inqu
irie
s.S
igna
ture
:az
Prin
ted
Nam
e/P
ositi
onfr
itle:
lhor
Cap
/Pro
gram
Dev
. Coo
rd./D
r.
Org
aniz
atio
n/A
ddre
ss..
24T
elep
hone
:
1(20
4) 9
45-3
346
E-m
ail A
ddre
ss:
lihor
ca p
g ho
t mai
l.cor
h
FA
X:
Dat
e:
EF
F-0
1311
(R
ev. 2
/200
1)
tias.
FQrr
n
file
://M
icap
\hom
cdri
vc\W
PDA
TA
\ZA
SQ%
20m
isc1
lhoe
s%20
Mot
ivat
iona
l%20
Des
ign%
20Pr
esen
tatio
n\R
EPR
OD
UC
TIO
N%
2...
2002
-12-
31
HI.
DO
CU
ME
NT
AV
AIL
AB
ILIT
Y I
NFO
RM
AT
ION
(FR
OM
NO
N-E
RIC
SO
UR
CE
):
If p
erm
issi
on to
rep
rodu
ce is
not
gra
nted
to E
RIC
, or,
if y
ou w
ish
ER
IC to
cite
the
avai
labi
lity
of th
e do
cum
ent f
rom
ano
ther
sou
rce,
ple
asep
rovi
de th
e fo
llow
ing
info
rmat
ion
rega
rdin
g th
eav
aila
bilit
y of
the
docu
men
t. (E
RIC
will
not
ann
ounc
e a
docu
men
t unl
ess
it is
pub
licly
ava
ilabl
e, a
nd a
dep
enda
ble
sour
ce c
an b
e sp
ecif
ied.
Con
trib
utor
s sh
ould
als
obe
awar
e th
at E
RIC
sele
ctio
n cr
iteri
a ar
e si
gnif
ican
tly m
ore
stri
ngen
t for
doc
umen
ts th
at c
anno
t be
mad
e av
aila
ble
thro
ughb
ED
RS.
IV. R
EFE
RR
AL
OF
ER
IC T
O C
OPY
RIG
HT
/RE
PRO
DU
CT
ION
RIG
HT
S H
OL
DE
R:
If th
e ri
ght t
o gr
ant t
his
repr
oduc
tion
rele
ase
is h
eld
by s
omeo
ne o
ther
than
the
addr
esse
e, p
leas
e pr
ovid
e th
e ap
prop
riat
e na
me
and
addr
ess:
V. W
HE
RE
TO
SE
ND
TH
IS F
OR
M:
Send
this
for
m to
the
follo
win
g E
RIC
Cle
arin
ghou
se:
ER
IC C
lear
ingh
ouse
on
Info
rmat
ion
& T
echn
olog
ySy
racu
se U
nive
rsity
621
Skyt
op R
oad,
Sui
te 1
60Sy
racu
se, N
Y 1
3244
-529
0E
-Mai
l eri
cft@
eric
ir.s
yr.e
du31
5-44
3-36
40 1
-800
-464
-910
7Fa
x: 3
15-4
43-5
448
Sen
d F
orm
Cle
ar F
orm
file
://U
Nca
p\ho
med
rive
\WPD
AT
AV
ASQ
%20
mis
e\Ih
or10
/020
Mot
ivat
iona
l%20
Des
ign%
20Pr
esen
tatio
n\R
EPR
OD
UC
TIO
N%
2...
2002
-12-
31
