2016 Baldrige New Examiner Orientation

Participant Guide & Notebook

2016 Baldrige New Examiner Orientation

This 2016 Baldrige New Examiner Orientation Participant Guide was developed and produced by the Baldrige Performance Excellence Program at the National Institute of Standards and Technology in Gaithersburg, MD.

Participant Guide

3

Course Overview

Purpose

As part of the on-boarding experience for first-year examiners, this classroom event will enable first-year examiners to confidently and proficiently participate in the Examiner Preparation course.

.

Learning Objectives

After completing this course, participants will be able to do the following:

Demonstrate working knowledge of the program, the evaluation process, and their role in the process.

Develop an Independent Review (IR) Worksheet for an award application using the Six-Step Evaluation Process, which encompasses the following: – Identifying and using an applicant’s key factors (KFs)

– Analyzing and synthesizing an applicant’s response to the Criteria, using the Approach, Deployment, Learning, and Integration (ADLI) and Levels, Trends, Comparisons, and Integration (LeTCI) evaluation factors

– Providing relevant, feedback-ready comments

– Using the Scoring Guidelines to holistically score an item to align with analysis

Identify personal strengths and opportunities for development based on feedback received during the course from peers and experienced examiners.

Participant Guide

4

Participant Guide

5

Agenda

Morning

Welcome and Introductions

Course Overview

Process Overview and the Examiner’s Role

Begin with the End in Mind

Code of Ethical Conduct

Importance of Key Factors

Program and Pre Work Q&A

Afternoon

Reconfigure Class Groups

1-on-1 Coaching

Wrap-Up

Participant Guide

6

Welcome

Baldrige Program Director

Points to Remember

Participant Guide

7

Award Process Overview & the Examiner's Role

Slide 1

The Award Process

Participant Guide

8

Examiner Development Model: On-Boarding

0–1 month

1–2 months

2–7 months

5–7 months

Prepare for 2nd year

Coaching ProcessPreparation Application Evaluation

Selection Notification

Baldrige Program

Orientation

Evaluation Process

Orientation

Case Study

New Examiner Orientation

Examiner Preparation

Team Assignment Planning for Year 2

Receive welcome e-mail.Select training week & wave preference.Participate in LinkedIn discussions for new examiners.

Complete online modules.Review sample feedback report . Read about Baldrige Award recipients.Participate in LinkedIn discussions for new examiners

Complete online modules.Complete “Are You Ready?” Participate in LinkedIn discussions for new examiners.

Read the Criteria.Complete prework.Participate in LinkedIn discussions for new examiners.

Complete classroom training.Peer learning, reflection, & feedback

Complete classroom training.Prepare for the awardapplication evaluation.Peer learning, reflection, & feedback

Complete IR Scorebook for assigned applicant.Peer learning, reflection, & feedbackTeam leader coaching

Reflect on experience.Seek feedback from examiner peers & team leader.

What are you learning?What are your strengths and

opportunities?What are your next steps?

Points to Remember

Participant Guide

9

Code of Ethical Conduct

Principles 2011

Baldrige Performance Excellence Program | www.nist.gov/baldrige

Code of Ethical Conduct

• Protect the integrity of the award process

• Exhibit professional conduct at all times

• Protect the promise of confidentiality

• Protect the program’s intellectual property

Points to Remember

Participant Guide

10

Participant Guide

11

Code of Ethical Conduct Members of the Board of Examiners for the Malcolm Baldrige National Quality Award pledge to uphold their professional principles in the fulfillment of their responsibilities as defined in the administration of Public Law 100-107, the Malcolm Baldrige National Quality Improvement Act of 1987. The Board of Examiners Code of Ethical Conduct establishes accountability for these principles.

Four principles are the foundation of the Code of Ethical Conduct. These principles empower examiners to make effective decisions with great confidence.

Principle 1: Protect the Integrity of the Award Process

Examiners will make sound decisions related to conflicting or competing interests, as well as do their part to ensure that all applicants are evaluated consistently. Violations of this principle include, but are not limited to, the following examples:

Representing conflicting or competing interests or placing themselves in such a position where their interest may be in conflict–or appear to be in conflict–with the purposes and administration of the Baldrige Award. This includes being employed by, being a supplier or customer of, having a financial interest in, or having a consulting arrangement with a competitor or competing interest, present or future.

Approaching an organization they have evaluated for personal gain or accepting employment from an organization they have evaluated within five years of the evaluation

Using information gained from sources other than the award applicant, such as information gathered from the press, websites, social media, examiners, or any other outside sources

Intentionally communicating false or misleading information that may compromise the integrity of the award process or decisions therein

Principle 2: Exhibit Professional Conduct at All Times Examiners will conduct themselves professionally, guided by truth, accuracy, fairness, respect, and responsibility in all their interactions. Violations of this principle include, but are not limited to, the following examples:

Sharing their number of years of service as an examiner or promoting/advertising their services while performing as an examiner

Independently giving feedback to applicants regarding scoring or overall performance

Using Baldrige Program/Award logos in advertising or promotion. Note: Upon completion of the Examiner Preparation Course, examiners may use the following designation for one year, except on business cards: Examiner, Malcolm Baldrige National Quality Award (MBNQA).

Failing to respect the climate, culture, and values of organizations being evaluated when participating in site visits

Participant Guide

12

Principle 3: Protect the Promise of Confidentiality

Examiners will safeguard the confidences of all parties involved in the judging or examination of present or former applicants so that the integrity of the Baldrige Award or award process is not compromised. Violations of this principle include, but are not limited, to the following examples:

Disclosing the identity of or other information about the applicant to anyone other than the examiner’s team, the judge involved in the examination, or the Baldrige staff members involved in the examination during or at any time after the review process.

Copying applicant information of any kind. Note: Copying information from inside the Baldrige Program’s encrypted, secure database to other documents within the database is permitted as necessary.

Retaining applicant information beyond the Consensus Review (or Site Visit Review, if applicable)

Communicating applicant identity or other applicant information through e-mail, social media, or any other electronic or written means outside of the Baldrige Program’s secure database

Communicating applicant identity or other information via cell phone, VOIP, or wireless devices unless authorized by the applicant

Using or adapting applicant information subsequent to the review process, unless the information is publicly released by the applicant

Communicating with the applicant during Independent Review and/or Consensus Review

Communicating directly with the applicant during site visits about matters other than verifying and clarifying information in the consensus scorebook, unless the examiner is the team leader or backup team leader.

Requesting/reviewing individual customer, stakeholder, or workforce member data and information during the Site Visit Review process

Principle 4: Protect the Program’s Intellectual Property

Examiners will make sound decisions related to the use of Baldrige Program materials, trademarks, logos, and information contained within the program’s website. Violations of this principle include, but are not limited to, the following examples:

Establishing links from their own website to the National Institute of Standards and Technology (NIST) or Baldrige Performance Excellence Program (BPEP) website without making it clear that users will be taken to official NIST websites

Failing to acknowledge the use of trademarks owned by NIST, including those for NIST, the Quest for Excellence, and the Malcolm Baldrige National Quality Award, along with the statement indicating the trademark is registered by NIST

Making unauthorized copies of the Baldrige Criteria for Performance Excellence or any other Baldrige Program publications

Participant Guide

13

Furthermore, board members enhance and advance the Malcolm Baldrige National Quality Award as it serves to stimulate U.S. organizations to improve quality, productivity, and overall performance. All board members pledge to abide by this Code of Ethical Conduct.

Furthermore, board members enhance and advance the Malcolm Baldrige National Quality Award as it serves to stimulate U.S. organizations to improve quality, productivity, and

overall performance. All board members pledge to abide by this Code of Ethical Conduct.

Participant Guide

14

Participant Guide

15

What Would You Do?

Table activity.

Your Table:

Your Scenario(s):

At your tables, discuss the dilemma facing the examiner. Select a timekeeper and a reporter. Discuss the following questions:

What were the issues in this situation?

Which Code of Ethical Conduct principles apply to this situation?

What would you do?

You have 5 minutes to complete this activity.

Scenario #1:

I received a mailing from someone who claims to have been a Baldrige examiner for five years. For only $500, we can attend his one-day seminar on “How to Win the Baldrige Award.” His brochure promises ten secrets for winning. Because his approach to winning the Baldrige Award is so successful, he has been given permission to use the Award logo on his publications. He guarantees that our organization will receive at least a site visit, and the brochure implies that he has terrific connections. I definitely think we should sign up, don’t you?

Participant Guide

16

Scenario #2:

I received an award application from the Baldrige Program for an organization that I assisted in writing its Organizational Profile. After we finished writing the Organizational Profile, staff members felt they could finish writing the application without my further assistance. I don’t think I have a conflict with this applicant since I did not write the rest of the application. Since the information I learned about the organization is limited, I think that I will share it with other examiners in my class during training to help them better understand the process applicants use to write an award application. What do you think?

Scenario #3:

The Performance Management System of the application I am reviewing is extremely complex and difficult to understand. I think I should contact a friend of mine who is an expert in human resources. She has said that she has consulted with some of the best organizations, and perhaps she can help me understand this applicant’s approach. Or, if she’s not available, maybe I’ll just contact the applicant’s official contact point (OCP). He will definitely be able to explain it. What do you think? Is this a good idea?

Participant Guide

17

Scenario #4:

Part 1: While waiting for the rest of the team to join the teleconference for consensus, a team member shares information with you about the applicant your team is evaluating. She tells you she heard this information during a professional association luncheon. Apparently, the applicant has begun negotiations with a supplier to require adherence to the organization’s new “green” standards. The supplier does not agree but feels pressure to conform. Should the two examiners be discussing this information?

Part 2: When the consensus call begins, the team leader opens by informing the team that the applicant has just made a public announcement that it will be requiring new environmental standards for its suppliers. The process for deploying these standards can be found on the applicant’s Web site. Should this information be considered in the evaluation process?

Scenario #5:

Last year, I evaluated a large manufacturing organization that is outside my area of expertise. The application contained a number of terms unfamiliar to me, such as Kaizen, Pareto charts, PDCA, and Failure Modes and Effects Analysis (FMEA). How can I do a proper evaluation if I do not know what these terms mean? Can I do any kind of research to familiarize myself with these concepts?

Participant Guide

18

Participant Guide

19

Begin with the End in Mind:

The Feedback Report

Participant Guide

20

Participant Guide

21

Importance of Key Factors

Slide 2

Key Factors Definition

A key factor is an attribute of an organization or its environment that influences the way the organization operates and the key challenges it faces. It indicates areas of strategic importance. Examples may include:

The organization’s mission, vision and values

Strategic challenges

Workforce groups and segments

Key factors do not include descriptions of processes.

Slide 3

How To Use Key Factors

Participant Guide

22

Table activity: Directions.

1. Independently, read the Criteria requirements in P.1a – Organizational Environment (from the 2013-2014 Health Care Criteria for Performance Excellence)

2. Independently, read P.1a from your assigned organization’s Organizational Profile.

3. As a table, discuss the significance of the Key Factors identified for your organization on the matrix and compare them to the Key Factors for the other organization.

4. As a table, identify additional Key Factors that might facilitate better understanding of the organizations.

5. Appoint one person to report to the whole group, listing the key Factors identified by your table group.

How could such disparate Key Factors result in all three of these organizations receiving the Baldrige Award in one year?

Participant Guide

23

Program and Prework Q&A

Participant Guide

24

Participant Guide

25

1-on-1 Coaching

The goal of this module is for new examiners (and non-examiners) to explore the use of the Six-Step Evaluation Process with senior and alumni examiners. Discussions with experienced practitioners of the evaluation process will help new examiners better understand how to apply the process.

Process Steps

New Examiners and BETE participants are expected to share their pre work for item 5.2, describing their output and the rationale behind it. Senior and alumni examiners are expected to listen actively, ask questions about the rationale used, and provide guidance as needed, using their own examples if needed.

You have 3 hours and 15 minutes to complete this activity (includes a break).

Table introductions between new and senior/alumni examiners.

Six-Step Evaluation Process

– Step 1: Review the Criteria item. – Step 2: Select relevant key factors. – Step 3: Read the relevant section of the application. – Step 4: Identify around six strengths and OFIs. – Step 5: Draft feedback-ready comments.

Step 6: Determine the scoring range and score.

Materials Needed

Criteria booklet and case study Prework: Key Factors Worksheet and completed IR Worksheet for item 5.2 Reference tools: Step-by-Step for Independent Review and Comment Guidelines Easel and flipchart paper

Participant Guide

26

Participant Guide

27

Step 1: Review the Criteria Item

You have 20 minutes to complete this activity.

Learning Objectives:

Understand that “Ground yourself in the Criteria item” does not mean a rote reading of the requirements. It includes interpreting and studying the requirements in order to apply them appropriately.

Review your work with your coaching partner and receive feedback

Identify changes to this part of the evaluation practice that you will make, as appropriate.

Process Steps

Pair up with a coaching partner.

Explain what you think the main points in Criteria Item 5.2 are and why they are important, and then share your approach and rationale for accomplishing this step.

Ask for feedback.

Seniors will listen actively and ask questions to understand your thinking, and to encourage you to think about how your approach accomplishes the objective of the step and overall goal, pros and cons of different approaches, etc.

Take notes on your learnings and things you will do differently next time around.

Prepare to share your learnings with the large group

Participant Guide

28

Error! No text of specified style in document.(continued)

Points to Remember

Participant Guide

29

Step 2: Select the Relevant Key Factors

You have 20 minutes to complete this activity.

Learning Objectives:

Understand the importance of the thoughtful selection of KFs and their eventual use in constructing feedback.

Review your work with your coaching partner and receive feedback.

Identify changes to this part of the evaluation practice that you will make, as appropriate.

Process Steps

Pair up with a coaching partner.

Explain to your coaching partner the key factors you chose and why.

Ask for feedback

Seniors will listen actively and ask questions to understand your thinking, and to encourage you to think about how your approach accomplishes the objective of the step and overall goal, pros and cons of different approaches, etc.

1. Take notes on your learnings and things you will do differently next time around. 2. Prepare to share your learnings with the large group.

Participant Guide

30

Step 2: Select the Relevant Key Factors continued

Points to Remember

Participant Guide

31

Step 3: Read the Relevant Section of the Application

You have 20 minutes to complete this activity.

Learning Objectives:

Understand that “read the application” is not intended to be at the multiple requirement level, but rather is a holistic review.

Review your work with your coaching partner and receive feedback.

Identify changes to this part of the evaluation practice that you will make, as appropriate.

Process Steps

Provide a brief overview of your approach to reading and analyzing and rationale, along with the results of your analysis of the application, asking for feedback.

Ask for feedback

Seniors will listen actively and ask questions to understand your thinking, and to encourage you to think about how your approach accomplishes the objective of the step and overall goal, pros and cons of different approaches, etc.

1. Take notes on your learnings and things you will do differently next time around. 2. Prepare to share your learnings with the large group

Participant Guide

32

Step 3: Read the Relevant Section of the Application

(continued)

Points to Remember

Participant Guide

33

Step 4: Draft around Six Strengths and Opportunities for Improvement (OFIs)

You have 20 minutes to complete this activity.

Learning Objectives:

Understand that strengths and OFIs incorporate the Criteria and the applicant’s operating environment and approaches (or results).

Review your work with your coaching partner and receive feedback.

Identify changes to this part of the evaluation practice that you will make, as appropriate.

Process Steps

Refer to the sample to item 5.1. This may be a useful document to refer to in your coaching discussion.

Provide your coaching partner a brief overview of your approach to determining strengths and OFIs, and your rationale.

Ask for feedback.

Seniors will listen actively and ask questions to understand your thinking, and to encourage you to think about how your approach accomplishes the objective of the step and overall goal, pros and cons of different approaches, etc.

1. Take notes on your learnings and things you will do differently next time around. 2. Prepare to share your learnings with the large group.

Participant Guide

34

Step 4: Draft around Six Strengths and Opportunities for Improvement (OFIs) (continued)

Points to Remember

Participant Guide

35

Step 5: Draft Feedback-Ready Comments

You have 20 minutes to complete this activity.

Learning Objectives:

Understand the use of the Comment Guidelines.

Review your work with your coaching partner and receive feedback

Identify changes to this part of the evaluation practice that you will make, as appropriate.

Process Steps

Refer to the Comment Guidelines (see page 37). Your comments should be evaluated against this document.

Explain to your coaching partner your approach to this step and rationale.

Review your comments.

Ask for feedback.

Seniors will listen actively and ask questions to understand your thinking, and to encourage you to think about how your approach accomplishes the objective of the step and overall goal, pros and cons of different approaches, etc.

1. Take notes on your learnings and things you will do differently next time around. 2. Prepare to share your learnings with the large group.

Participant Guide

36

Step 5: Draft Feedback-Ready Comments (continued)

Points to Remember

Participant Guide

37

Step 6: Determine the Scoring Range and Score for the Item

You have 20 minutes to complete this activity.

Learning Objectives:

Understand that scoring is a holistic process of determining the range most descriptive of the applicant’s achievement level for the entire item, rather than for individual processes.

Review your work with your coaching partner and receive feedback

Identify changes to this part of the evaluation practice that you will make, as appropriate.

Process Steps

The process will be the same as in previous steps. In this step you will provide your scoring range and score, along with a brief overview of your approach to scoring and rationale, and ask for feedback.

Refer to the “Scoring Guidelines in the Criteria Booklet.

Ask for feedback.

Seniors will listen actively and ask questions to understand your thinking, and to encourage you to think about how your approach accomplishes the objective of the step and overall goal, pros and cons of different approaches, etc.

1. Take notes on your learnings and things you will do differently next time around. 2. Prepare to share your learnings with the large group. 3.

Step 6: Determine the Scoring Range and Score for the Item (continued)

Points to Remember

Participant Guide

38

Participant Guide

39

Wrap-Up

Reflection and Assessment

You have 10 minutes to complete this activity.

What did I learn? What did this day mean to me?

What are my strengths and opportunities?

What are my next steps? What do I need to change?

For Examiner Preparation: what questions still need to be answered?

Points to Remember

Participant Guide

40