Embed Size (px)
Citation preview
www.ikmnet.com
Developing Testing
Rajiv Unnikrishnan - Business Head - SkillProfiler
www.skillprofiler.com [email protected]
www.ikmnet.com
Developing a test
• Content preparation
• Choice of authoring tool
• Scoring logic and methodology
• Report generation
• Effective usage of test results
www.ikmnet.com
CONTENT
PREPARATION
www.ikmnet.com
Contents identification
• Identify the most critical skills
• Break them into sub skills
• Create questions for each sub skill
• Ascertain the assessment method…
Adaptive
Oracle 9i
www.ikmnet.com
Adaptive Methodology
It is an optimal method for soliciting and analyzing feedback and for adjusting our actions based upon the feedback.
www.ikmnet.com
• Can be compared to a High Jump event• Bar height (question difficulty) is continuously
adjusted based upon prior jumps (answers) until the maximum cleared height (knowledge) is identified.
Adaptive Model
www.ikmnet.com
Guess My Number GameObjective: To guess the secret number in as few guesses as possible
• selects a secret number from 1 to 100• Player 2 makes a guess at the secret number• Player 1 provides FEEDBACK response of ‘higher’ or ‘lower’• Play continues until Player 2 guesses the secret number.
You have used Adaptive Model even as a kid
www.ikmnet.com
Non-Adaptive Strategy
Is the secret number 25?-higherIs the secret number 80?-lowerIs the secret number 26?-higherIs the secret number 30?-higherIs the secret number 70?-lowerIs the secret number 35?-higherIs the secret number 68?-lowerIs the secret number 67?-lowerIs the secret number 39?-higherIs the secret number 61?-lowerIs the secret number 60?-lower
Is the secret number 1?-higherIs the secret number 2?-higherIs the secret number 3?-higherIs the secret number 4?-higherIs the secret number 5?-higherIs the secret number 6?-higherIs the secret number 7?-higherIs the secret number 8?-higher -Is the secret number 9?-higherIs the secret number 10?-higherIs the secret number 11?-higher
Is the secret number 60?-higherIs the secret number 40?-I said it was higher than 60!Is the secret number 30?-What are you, an idiot?Is the secret number 20?-Let’s play Monopoly
Hit or Miss Method Slow but Steady Method I Have No Clue Method
www.ikmnet.com
The Adaptive Model
Is the secret number 50
higher
Is the secret number 75
lower
Is the secret number 62
lower
Is the secret number 56higher
Is the secret number 59
lower
Is the secret number 58 YES
Model: Each guess adapts to the feedback of prior guessesStrategy: Maximize value of feedback
www.ikmnet.com
E = MC2Hey Albert,
How Much is 2 + 2 ?
• Secure– Random selection of questions from the
appropriate difficulty pool. – Knowledge of answers results in more
difficult follow-up questions
• Efficient– No time is wasted on questions that are
too hard or too easy.
IT T
esti
ng
- U
SP
Benefits of Adaptive…
www.ikmnet.com
Choice
0f
Authoring Tool
www.ikmnet.com
Authoring Tool
• A software environment used for developing, implementing and deploying assessment contents.
• At this stage the need is for a tool to translate our design and prepared content into the requirement we envisioned
www.ikmnet.com
Easy You’ve got to be
kidding!
MediumVeryEasy
Hard
Degree of Difficulty
6.7
IT T
esti
ng
- U
SP
Adaptive System Driver…
www.ikmnet.com
• Online Web Based
• Online LAN Based
• Online PC Based
• Offline Based
Seamless Technology…
www.ikmnet.com
SCORING LOGIC
&
METHODOLOGY
www.ikmnet.com
PARADIGM SHIFT IN SCORING
Replace Binary Feedback (Correct / Wrong)
with
Multi-Dimensional feedback
(Absolutely Correct, Somewhat Correct, partially correct, reasonably wrong, wrong, not even close)
www.ikmnet.com
BRINGING ACCURACY TO SCORING
So, how do we get multi-dimensional feedback?
Multiple Correct Responses
Multiple choice questions with 1-3 correct answers out of 5 possible answers Each answered weighted for degree of accuracy or inaccuracy (perfectly correct, partially correct, wrong, not even close)
A ABD CEB ABE DEC ACE ADED ACD BDEE BC CDEAB BDAC BEAD BCDAE BCEABC CD
25 Answer Combinations
identifies a different level of knowledgehas its score based upon contained answersleads to a different level of follow-up question
Each answer combination
www.ikmnet.com
Sample Adaptive Model Question
Which of the following are states in the United States of America?
a. Texas
d. Washington, D.C.
b. Parisc. California
e. Jupiter
Subject: U.S. Geography
www.ikmnet.com
Sample Adaptive Model Question
Answer Combinations Reflect Partial Understanding
California and Texas
Not Even Close answer; loses most credit and leads to easiest follow-up question
Texas
Demonstrates no understanding of US geography; the test taker is most likely guessing
Jupiter
Full credit answer Leads to most difficult follow-up question
Paris, Jupiter and Washington D.C.
Credit for selecting Texas, loss of credit for not selecting California
Less wrong than Texas and Paris; reflects partial understanding of US geography
Texas and Washington D.C.
www.ikmnet.com
We assign the ‘secret number’ in each sub-skill a knowledge level designation
Detailed Feedback based Reporting
www.ikmnet.com
REPORT
GENERATION…
www.ikmnet.com
ORACLE9i DATABASE ADMINISTRATION - CompleteORACLE9i DATABASE ADMINISTRATION - Complete
Assessment Results
www.ikmnet.com
Applying The Test…
www.ikmnet.com
Recruit
Training &
Development
Skill Inventory
Internal Certification
Skill Assessment Model
Hire the best talent
Assess & train on skill gap areas
Skill-wise Segmentation
Skill-wise Competency Mapping
Beneficiaries are Business vertical
Beneficiaries are HR
www.ikmnet.com
Custom Learning Perspective
www.ikmnet.com
ORACLE9i PL/SQL PROGRAMMING
GREEN BAR – Portion of group showing strength in the sub-topic
YELLOW BAR – Portion of group showing proficiency in the sub-topic
RED BAR – Portion of group showing weakness in the sub-topic
Group Knowledge Audits
Applying The Adaptive Model
www.ikmnet.com
ORACLE9i PL/SQL PROGRAMMING
Customize Classroom Training
Applying The Adaptive Model
Remove these sub-topics
Emphasize these sub-topics
www.ikmnet.com
Build Homogeneous Classes
Class C only requires training in 5 of the 12 defined skills for this curriculum.
Applying The Adaptive Model
www.ikmnet.com
Sub-topics listed in order of weakest to strongest
Proposed project team is less than ideal. 90% of team is weak in this critical sub-topic and the team lacks an expert to mentor them (no green bar). Restructuring of team is indicated.
Skills ManagementAfter training, build optimum project teams
Applying The Adaptive Model
www.ikmnet.com
20%
30%
45%
5%
Excellent
Good
Fair
Poor
Skill Inventory… a click away
Java Talent pool
www.ikmnet.com
After Training
Certification Incentive for training
Validate training
Career advancement
Establish criteria for project participation
Identify mentors
Applying The Adaptive Model
www.ikmnet.com
Certification
External Certification Provided by outside 3rd party does not reflect client specific knowledge sets Clients concerned External Certification makes employees more marketable
Percentiles Relative measure that changes with the knowledge of other individuals rather than with the knowledge of the test taker.
Raw Scores Requires long calibration study to fixed a knowledge level (how much knowledge does a score of X represent) and must be normalized across multiple test subjects (does a score of X indicate the same level of knowledge in all test subjects?) Reflects an aggregate across all sub-skills. (Do two individuals with the same raw score have the same set of skills?)
Applying The Adaptive Model
www.ikmnet.com
Capitalize on the Adaptive Method’s unique ability to qualify sub-skills
1
2
3
4
5
6
7
89
10
11
12
13
14
15
16 1
2
3
4
5
6
7
89
10
11
12
13
14
15
16
Assessment is broken down into sub-skills that collectively cover the entire scope of the subject
1
2
3
4
5
6
7
89
10
11
12
13
14
15
16
Sub-Skills are group by knowledge level
Each sub-skill is classified as a weakness, proficiency or strength
Establish certification criteria based upon the extent of subject coverage
Percent of sub-skills that are not weakPercent of sub-skills that are strong
Applying The Adaptive Model
www.ikmnet.com
Assign weights to each sub-skill reflecting the specifics of your environment.
Assign weights to each knowledge level
Arrive at level wise and measure each one against the specified skill Certify based on knowledge & skill levels
Certification Process
www.ikmnet.com
Summary
• Determine the most critical skill
• Break them into sub skills• Create questions based on adaptive (difficulty based)
• Choose the right authoring tool
• Tests should be scalable technologically
• Report generated should be elaborate
• Focus to be on Skill Assessment Model (SAM)
