Perfecting test with Questionmark

2010 Users Conference Miami

PERFECTING THE TEST THROUGH QUESTION ANALYSIS

DELL CONFIDENTIAL

ABOUT THE PRESENTER

NEELOV KAR

Program Owner for Project Management courses Led the Evaluation Team and introduced QM in Perot Systems Account Manager in Perot Systems since 1998 Worked in other organizations in senior positions and has a total of

22 years of IT experience PMP, ISO 9000 Lead Assessor, Six Sigma and Certified Quality

Analyst Certified Instructor for Orientation, Leadership Workshop and

Project Management courses. Graduation from Friendship University, Moscow and MBA from BIT

Mesra

DELL CONFIDENTIAL


WELCOME TO DELL SERVICES

3

“Dell Services is an one stop shop for all IT Services delivering its services worldwide in 183 countries with an employee strength of 41000”.

DELL CONFIDENTIAL

4

Building on years of successful collaboration, together we create Dell Services

Serving the public sector, large enterprises and small- and medium-size businesses all over the world

More than 41,000 experienced and skilled technology and business-services professionals

Approximately $8B revenues from enhanced services and support Best-in-class suite of intelligent, end-to-end services and business

solutions to reduce IT complexity and lower costs

Global Presence

DELL CONFIDENTIAL

5

The Integrated Services PortfolioFrom product support to business solutions, we can help you achieve your business outcomes

SupportServices

Managed Services

Cloud /as-a-Service

ApplicationsServices

Business ProcessServices

ITConsulting

Business Consulting

Extended Warranty End User Client

SoftwareCustom

DevelopmentRevenue Cycle

OutsourcingEnterprise

ArchitectureStrategy

Consulting

Enhanced Support Data Center

Enterprise Management

SoftwareModernization Policy

AdministrationService

Management

Organizational Change

Management

NetworkBusiness Continuity Software

Business Intelligence

Physician Services

Data Center Infrastructure

Process Re-Engineering

Information Assurance Desktop

Enterprise and Industry

Applications

Engineering Services

Outsourcing

Enterprise Applications

Rationalization

Supply Chain Re-Engineering

Hosting Servers and Storage Testing Customer

ManagementEnd User

ComputingClinical

Transformation

Management Government Office Virtualization/ Cloud Integration

Data Management

Business Continuity /

Disaster RecoveryDELL CONFIDENTIAL

Dell Services: Foundation of Technology With Industry Capabilities

HealthcareGovernmentCommercial

Hospitals

Physicians

Health Insurance

Life Sciences

Pharmaceutical

Defense

Homeland Security

Intelligence

Civilian

State and Local

Education

Retail Banking

Payments and Clearing

Insurance

Capital Markets

Manufacturing

Freight and Logistics

Telecommunications

Travel and Transport

Energy

Retail

Technology

6DELL CONFIDENTIAL

7

Helping You SucceedSetting Dell apart through our breadth of portfolio, ecosystem of partners and systematic approach to innovation

Trusted by you:

Over 2 million desktops managed

12.8 million desktops and notebooks supported

1.4 million servers and storage systems supported

Over 5,000 SaaS customers

Ranked #1 for Professional Services in North America Healthcare*

Ranked #1 in Customer Service**

Using customizable technology to deliver business-focused solutions

Leveraging internet technology, we will be at the forefront of cloud and as-a-service delivery

Using standardization to drive down costs and drive up efficiency, from the desktop to the data center

Evolving with the industry to bring you the newest technology and more flexible consumption models

*By revenue from Gartner’s Industry IT Services Worldwide Market Share Database; August 14, 2009 **TBR's Q209 Service and Support Customer Satisfaction Study

DELL CONFIDENTIAL

DELL CONFIDENTIAL 8

Responsible for Design, Development & Delivery of Project Management and Leadership courses for Dell Services across the globe

ISO 9000 certified since 2007 Awarded “2009 Best Place to Learn” by ASTD Dallas

Chapter We use

– QM for Level 2 Assessment– SurveyMonkey for Level 1 feedback– Adobe Captivate for CBL development– PeopleSoft for Training data management– Saba as LMS

Learning & Development

ADDIE MODEL

DELL CONFIDENTIAL 9

DELL CONFIDENTIAL 10

Learn to use the different Perception Report to comprehensively analyze the assessment

Understand how item analysis Report helps in identifying the badly constructed questions

Learn how Dell Services have done the analysis to refine the test

Learning Objectives


P3MM Fundamentals Course 3 Series with 17 VILT Modules Exam after each series Launched in December 2008 Cohort 1, 2 & 3 as Pilot, participants were

experienced project managers Used Cohort 1, 2 & 3 Exam results and

participant feedback to analyze the test questions

The Case

Done through QM– Question Statistics Report– Item Analysis Report– Indices to look for:

– Discrimination Index

– Item Difficulty Level

Combination of SurveyMonkey and QM– Correlation between Test Score and Demography (Role,

Account, BU etc) QM and Course Delivery

– Correlation between Test Score and Course Content – Correlation between Test Score and Instructor

TEST ANALYSIS

DELL CONFIDENTIAL

Completed Schedule 2980% or over 12 41%Average time 00:20:05

Completed Cost 2180% or over 4 19%Average time 00:19:43

Completed Scope 2180% or over 17 80%Average time 00:09:54

Completed All 2080% or over 10 50%Average time 00:35:57

TEST RESULTS

Raised a red flag that there is some problem with Cost TestDELL CONFIDENTIAL

Course TotalScope Management 12 9 75% 3 25%Cost Management 16 8 50% 6 38% 2 13%Schedule Management 34 27 79% 1 3% 5 15% 1 3%

Majority Correct Polarized Scattered Majority Wrong

ANALYZED DISTRIBUTION OF ANSWERS

CategoriesMajority CorrectPolarizedScatteredMajority Wrong

DELL CONFIDENTIAL


Test Analysis Report


Question Statistics Report

Scattered


Question Statistics Report

PolarizedMicrosoft Office Excel Worksheet


Item Analysis Report







Analyzing the Question and Feedback

DELL CONFIDENTIAL


COACHING REPORT


Assessment Overview Report


P3MM FUNDAMENTALS COURSE HAS BEEN STABILIZED AFTER FEBRUARY 2009 AND RUNNING SMOOTHLY SINCE THEN………


THANK YOU

26