Quality Assessment of Multimedia Materials B. S. BHATIA Former Dir. DECU-ISRO [email protected]

Quality Assessment Quality Assessment of of

Multimedia MaterialsMultimedia Materials

B. S. BHATIAFormer Dir. [email protected]

Quality Assurance of Multimedia MaterialsQuality Assurance of Multimedia MaterialsInitiated By CEMCAInitiated By CEMCA

Quality Assessment - The Quality Assessment - The NeedNeed

Availability of Multimedia Material is Availability of Multimedia Material is increasing at a very rapid rateincreasing at a very rapid rate

Access has increased – due to computer and Access has increased – due to computer and internet spreadinternet spread

The number of content generators both The number of content generators both private and government increasing rapidlyprivate and government increasing rapidly

Consumer is paying for lack of qualityConsumer is paying for lack of quality Therefore Quality Assessment Standards and Therefore Quality Assessment Standards and

Certification is essentialCertification is essential CEMCA initiated the Quality Assurance for CEMCA initiated the Quality Assurance for

Multimedia Materials projectMultimedia Materials project

Project Evolution StagesProject Evolution StagesFirst Round Table– Bangalore 7First Round Table– Bangalore 7thth August August

20072007

GUIDELINES :GUIDELINES : The process should be inclusive.The process should be inclusive. Engage stakeholders from different Engage stakeholders from different

countries.countries. Provide guidelines for self assessment Provide guidelines for self assessment

by developers and users.by developers and users. certification and standards covered certification and standards covered

follow. follow.

Project Evolution StagesProject Evolution Stages

First Round Table– Bangalore 7First Round Table– Bangalore 7thth August 2007 August 2007 Second Round Table – Delhi 11Second Round Table – Delhi 11thth October 2007 October 2007 Core group meetingsCore group meetings

2828thth December – Delhi Meeting December – Delhi Meeting 2828thth January Tel con call with Malaysia January Tel con call with Malaysia 2020thth- 21- 21stst March- Ahmedabad Meeting. March- Ahmedabad Meeting. 2525thth March KL Presentation March KL Presentation



Project Evolution StagesProject Evolution Stages

Third Round Table– Delhi 9Third Round Table– Delhi 9thth June 2008 June 2008 Presentation of the version zero of the Presentation of the version zero of the

framework and approachframework and approach Modifications suggested to include quality Modifications suggested to include quality

indicatorsindicators


Multimedia Learning Materials Multimedia Learning Materials

Multimedia material refers to computer Multimedia material refers to computer

based e-learning material which may be based e-learning material which may be

made available on-line or off-line. It may be made available on-line or off-line. It may be

used either by an individual or group for used either by an individual or group for

learning with or without the intervention of a learning with or without the intervention of a

facilitator/mentor.facilitator/mentor.

Quality is Fitness of purpose at Quality is Fitness of purpose at minimum cost to Societyminimum cost to Society

“


Assessment ApproachAssessment Approach

Product AssessmentProduct Assessment Process AssessmentProcess Assessment People AssessmentPeople Assessment

To ensure consistent and continuous good To ensure consistent and continuous good quality output, Process Assessment quality output, Process Assessment supported by Product and People supported by Product and People assessment considered to be most assessment considered to be most appropriate.appropriate.

PROCESS PROCESS ASSESMENTASSESMENT



Proposed QAMLM Process diagramProposed QAMLM Process diagram

Analysis

Need for MLM

LearnerLearner

TaskTask

ContentContent

ContextContext

Design

Instructional Strategy

Visualal

TechnicalTechnical

Prototype testingPrototype testing

Development

Story boardStory board

Multimedia Multimedia elementselements

ProgrammingProgramming

Process Process documentationdocumentation

TestingTesting

IPR issuesIPR issues

Implementation StrategyStrategy

Structure & Structure & MechanismsMechanisms

Evaluation

StrategyStrategy

FrameworkFramework

MechanismsMechanisms

ImprovementsImprovements


Analysis phaseAnalysis phase

Captures requirements, sets expectationsCaptures requirements, sets expectations

– Needs AnalysisNeeds Analysis – – Analysis of ‘the gap between what is Analysis of ‘the gap between what is

and what is expected/desired’and what is expected/desired’ – Context Analysis – Context Analysis – Refers to the analysis of the learning Refers to the analysis of the learning

environment.environment.– Learner Analysis – Learner Analysis – Refers to the analysis which brings Refers to the analysis which brings

out a clear understanding on “who” the learners or target out a clear understanding on “who” the learners or target audience are.audience are.

– Task Analysis – Task Analysis – Refers toRefers to defining the goals and defining the goals and objectives for the multi-media materials that are to be objectives for the multi-media materials that are to be prepared.prepared.


Design phase…1Design phase…1

Sets the blueprint, defines the frameworkSets the blueprint, defines the framework

– Instructional DesignInstructional Design is the systematic process of is the systematic process of translating general principles of learning and instruction translating general principles of learning and instruction

into plans for instructional materials and learning.into plans for instructional materials and learning.

– Visual DesignVisual Design - - refers to the application of proper refers to the application of proper methods to produce a product that is both artistic and methods to produce a product that is both artistic and functional. This technique is very important in order to functional. This technique is very important in order to communicate the message. communicate the message.


Design phase…2Design phase…2

Sets the blueprint, defines the frameworkSets the blueprint, defines the framework

– Technical Design Technical Design considersconsiders Product TypeProduct Type Product StandardsProduct Standards Medium of deliveryMedium of delivery End user scenarioEnd user scenario NavigationNavigation Technical aspects of Production of media elementsTechnical aspects of Production of media elements

– Prototyping – Prototyping – Developing aDeveloping a representative sample representative sample of all the key features that will be used in the of all the key features that will be used in the entire material.entire material.


Development phaseDevelopment phase

Creation, assembly and integration of Creation, assembly and integration of media elementsmedia elements – Storyboarding / ScriptingStoryboarding / Scripting– ProductionProduction of media elements – Text, video, audio, of media elements – Text, video, audio,

illustrations, graphics, animations – 2D / 3Dillustrations, graphics, animations – 2D / 3D– Integration or assembly of all media elementsIntegration or assembly of all media elements– ProgrammingProgramming where applicable where applicable– SCORM or AICC complianceSCORM or AICC compliance – Testing or Quality AssuranceTesting or Quality Assurance for Content and overall for Content and overall

productproduct


Implementation phaseImplementation phase

Putting the plan into actionPutting the plan into action

Implementation StrategyImplementation Strategy

Implementation Structure and Implementation Structure and MechanismsMechanisms


Evaluation phaseEvaluation phase Measure effectiveness, Recommendations Measure effectiveness, Recommendations

for product improvementfor product improvement

Evaluation frameworkEvaluation framework

Evaluation mechanismsEvaluation mechanisms

Improvement mechanismsImprovement mechanisms

Quality Assurance Framework based Quality Assurance Framework based on ADDIE : Analysis-Design-on ADDIE : Analysis-Design-

Development- Implementation-Development- Implementation-Evaluation (Version 1.0)Evaluation (Version 1.0)

ACTIVITY INPUTAND/ OR PROCESSES

OUTCOMES QUALITY INDICATORS

I ANALYSIS 1.1 Need Defining the learning needs

identifying the needs from the perspective of different stakeholders (learners, teachers, subject experts industries/practitioners, policy makers)

Need assessment statement/document

1.1 Needs are clearly stated and comprehensive. 1.2 Needs are appropriately prioritised.

1.2 Context Collecting data on contextual variables Learning Environment • Individual /Group• Formal / Informal• Facilitated / self learn• Individual / Group• Technical facilities • Access to internet • Software / Hardware specifications socio cultural aspects

Contextual profile 1.3 Context is clearly and fully mapped.






1.3 Learner Collecting Data with respect to learners vis-a- vis their academic levels and attributes like stills, motivation, visual literacy, language competency, learning styles, special needs (differently-abled) etc.

Learner profile 1.4 Learner profiles are adequately captured.

1.4 Task Stating purpose(s) of the MLM• Education• Training • Enrichment • Awareness• Skill development• Any other • Identifying usage aspects of MLM as standalone/ series, supplementary, integrated and / or any other

Task definition documents

1.5 Primary purpose of MLM is clearly stated.






1.5 Content Creating content – outline

Content – outline 1.7 Content outline is indicative of the scope of the MLM.

Generating content for design (raw content) Verifying that content is cognitively appropriate, factually accurate, complete, sensitive and inclusive (gender, class, caste, religion, ethnic groups, environmental factors, etc.)

Appropriately validated raw content

1.8 Raw content has been validated for appropriateness and accuracy.

Classifying content into facts, concepts, principles, processes, procedures, etc. identifying Learning Domains as cognitive and /or psychomotor

Identifying Learning Domain

Classified content

1.9 content is accurately classified for design treatment as per learning domain.


I. ANALYSIS I. ANALYSIS

S.NO.

INDICATORS DESCRIPTORS SCORE

1.1 Needs are clearly stated and comprehensive. (*)

1. Needs are not stated and stakeholders are not defined.

2. Needs are somewhat clearly stated, but stakeholders are not defined.

3. Needs are clearly stated and few stakeholders are defined.

4. Needs are very clearly stated and most stakeholders are well defined.

5. Needs are very clearly stated and all stakeholders are very well defined.

1. Poor

2. Average

3. Good

4. Very good

5. Excellent

1.2 Needs are appropriately prioritized.

1. Needs are not prioritized / wrongly prioritized.

2. Needs are prioritized to some extent, but inputs used are not clear.

3. Needs are prioritized to a large extent and indicate usage of some inputs.

4. All needs are appropriately prioritized and indicate usage of most inputs.

5. All needs are appropriately prioritized and indicate usage of all inputs (primary and secondary)

1. Poor 2. Average

3. Good

4. Very good

5. Excellent


I. ANALYSIS I. ANALYSIS

1.3 Needs are appropriately prioritized.

1. Context is not mapped.2. Context is somewhat clear, but only partially

mapped. (e.g. learning environment mapped but socio-cultural aspects and / or technical facilities not considered. )

3. Context is clear, though not fully mapped. ( e.g. Learning environment mapped and technical facilities determined, but socio-cultural aspects not considered.)

4. Context is very clear and well mapped. (e.g. socio- cultural aspects well mapped along with the learning environment and technical aspects..)

5. Context is very clear, well mapped and is clearly reflected in the profile document so as to guide the Design and Development stages.

1. Poor 2. Average

3. Good

4. Very good

5. Excellent


SUMMARY OF SUMMARY OF INDICATORS (Version INDICATORS (Version

1.0)1.0)PHASE NO. OF

INDICATORSCRITICAL

Analysis 9 5

Design 11 9

Development 6 4

Implementation 3 1

Evaluation 4 2

Total 33 21


PRODUCT PRODUCT ASSESMENTASSESMENT


Part A : Pre-requisite Part A : Pre-requisite Data for Assessment of Data for Assessment of

MLMMLMS.NO

.QUALITY

INDICATORSDESCRIPTIONS SCORE

Part A : Information on MLM Comments

A1 Reference Code

A2 Module Title

A3 Subject Area

A4 Coverage

A5 Keywords

A6 Language

A7 Target audience • Pre school• Primary school• Lower Secondary school • Upper Secondary School• Undergraduate • Post Graduate • Special needs (specify :_________)•Others (specify : e.g. Farmers)• Not Stated



MLMMLMS.NO

.QUALITY

INDICATORSDESCRIPTIONS SCOR

EA8 Minimum hardware

Requirements stated :• RAM Stated• Processor stated • HDD Stated • Multimedia requirements stated (e.g. CD/DVD, speakers, mike etc.)• Internet speed stated • Other requirements stated (specify : ______)• Not stated

A9 Minimum software Requirements stated :

• OS Stated (e.g. window version)• Browsers version / Type• Plug in software stated • Other software stated (specify : _________)• Not stated

A10 MLM validated by SME • Yes ● No

A11 Warranty • Warranty provided • No warranty provided

A12 MLM conformant with prevalent and applicable standards

• SCORM• Web 2.0• Open source • Other (not stated)



MLMMLMS.NO

.QUALITY

INDICATORSDESCRIPTIONS SCORE

A 13 Provides support for special needs ( i.e. physically challenged)

• Yes ● No

A 14 License conditions of the MLM stated

• Copyright protected • Creative commons with attribution • Creative commons non commercial • Any other • Not stated


Part B : Assessment Guide Part B : Assessment Guide for Multimedia Learning for Multimedia Learning

Materials Materials S.

NO.QUALITY INDICATORS 5 POINT

SCALE B 1 Learning Objectives (LO) are clearly stated (*)

B 2 Language is appropriate to target audience (*)

B 3 Pre – Requisites stated

B 4 Content is accurate and factual (*)

B 5 Content Meets objectives (*)

B 6 Content is structured

B 7 Scope of the content is sufficient

B 8 Clear instruction are available on how to use the content (*)

B 9 Content is easy to understand (*)

B10 MLM is interactive (*)


Part B : Assessment Guide Part B : Assessment Guide for Multimedia Learning for Multimedia Learning

Materials Materials S.

NO.QUALITY INDICATORS 5 POINT

SCALE B 11 Different learning styles are addressed

B 12 The use of media is appropriate (*)

B 13 The interface is user friendly (*)

B 14 The MLM is sensitive to gender and socio-cultural factors. (*)

B 15 Use of fonts and colour are appropriate.

B 16 Learner assessment is included in the MLM

B 17 Promotes collaborative learning

B 18 Learning support is available

B 19 Overall how do you rate the MLM?


TESTING TESTING

The scales were tested by a The scales were tested by a number of agencies in India and number of agencies in India and Malaysia. Malaysia.

All users benefited.All users benefited. Developers particularly Developers particularly

appreciated. appreciated. Some terminology found difficult.Some terminology found difficult. Work on version 1.5 started. Work on version 1.5 started.


VERSION 1.5VERSION 1.5

Explanations of indicators added. Explanations of indicators added. Some indicators modified added Some indicators modified added

or dropped. or dropped. A learner scoring sheet added. A learner scoring sheet added.


Quality Assurance Framework based on Quality Assurance Framework based on ADDIE : Analysis-Design-Development- ADDIE : Analysis-Design-Development-

Implementation-Evaluation (Version 1.5)Implementation-Evaluation (Version 1.5)


OUTCOMES

QUALITY INDICATOR

S

CHECKPOINTS

I ANALYSIS 1.1 Need Defining the learning

needs identifying the needs from the perspective of different stakeholders (learners, teachers, subject experts industries/practitioners, policy makers)



• Needs are defined from every stake holder’s point of view• Inputs from studies are considered while defining the need.• Needs are clearly stated for the reference of the development team


Contextual profile

1.3 Context is clearly and fully mapped.

• Context is very clear, well mapped with respect to the learning environment, technical facilities and socio- cultural aspects. • Context defined is clearly reflected and documented so as to guide the Design and Development stages.




ACTIVITY

INPUTAND/ OR PROCESSES

OUTCOMES

QUALITY INDICATOR

S

CHECKPOINTS

1.3 Learner

Collecting Data with respect to learners vis-a-vis their academic levels and attributes like stills, motivation, visual literacy, language competency, learning styles, special needs (differently-able) etc.

Learner profile

1.4 Learner profiles are adequately captured.

1.4 Task Stating purpose(s) of the MLM• Education• Training • Enrichment • Awareness• Skill development• Any other • Identifying usage aspects of MLM as standalone / series, supplementary, integrated and / or any other

Task definition documents

1.5 Primary purpose of MLM is clearly stated.

• Purpose of MLM is clearly stated with task well defined and includes suggestions for treatment at Design and Development stage. • Usage aspects of MLM are clearly stated, include suggestions for treatment and reflect how they will impact the design and development stages.



Implementation-Evaluation (Version 1.5)Implementation-Evaluation (Version 1.5)ACTIVIT

YINPUTAND/ OR

PROCESSESOUTCOME

SQUALITY

INDICATORSCHECKPOINTS

1.5 Content

Creating content – outline

Content – outline

1.7 Content outline is indicative of the scope of the MLM.

• Content outline is given and clearly indicates the scope of the MLM and incorporates suggestions that impact the design and development stages.

Generating content for design (raw content) Verifying that content is cognitively appropriate, factually accurate, complete, sensitive and inclusive (gender, class, caste, religion, ethnic groups, environmental factors, etc.)

Appropriately validated raw content

1.8 Raw content has been validated for appropriateness and accuracy.

• Raw content has been fully and thoroughly validated fro appropriateness and accuracy. (all features like content outline, logical sequencing, completeness factual correctness, sensitivity and inclusiveness are considered and validated.)• Check that all references are cited the source of the content is clearly captured while collating raw content.

Classifying content into facts, concepts, principles, processes, procedures, etc. identifying Learning Domains as cognitive and /or psychomotor

Identifying Learning Domain

Classified content

1.9 content is accurately classified for design treatment as per learning domain.

• Content is accurately classified, learning domain is appropriately identified and suggestions for design and development treatment are given.





OUTCOMES

QUALITY INDICATOR

S

CHECKPOINTS

I ANALYSIS 1.1 Need Defining the learning

needs identifying the needs from the perspective of different stakeholders (learners, teachers, subject experts industries/practitioners, policy makers)



• Needs are defined from every stake holder’s point of view• Inputs from studies are considered while defining the need.• Needs are clearly stated for the reference of the development team


Contextual profile

1.3 Context is clearly and fully mapped.

• Context is very clear, well mapped with respect to the learning environment, technical facilities and socio- cultural aspects. • Context defined is clearly reflected and documented so as to guide the Design and Development stages.


SUMMARY OF SUMMARY OF INDICATORSINDICATORSPROCESS PROCESS

PHASE NO. OF INDICATORS

(Version 1.0)

NO. OF INDICATORS

(Version 1.5)

Analysis 9 9

Design 11 12

Development 6 8

Implementation (Delivery consideration)

3 3

Evaluation 4 3

Total 33 35


SUMMARY OF SUMMARY OF INDICATORSINDICATORSPRODUCT PRODUCT

PHASE NO. OF INDICATORS

(Version 1.0)

NO. OF INDICATORS

(Version 1.5)

Prerequisite 14.0 -

Indicators 19 16

Binary - 4



THANK YOUTHANK YOU

Documents

Quality Assessment of Multimedia Materials B. S. BHATIA Former Dir. DECU-ISRO [email protected]