COLORADO TRAINING STRATEGY …c.ymcdn.com/sites/ LEVELS OF LEARNING ... Bloom’s taxonomy action verbs are used in writing ... Asking students to raise their hands in response to

C O L O R A D O2 0 1 5 P U B L I C WA T E R S Y S T E M

T R A I N I N G S T R A T E G Y A D M I N I S T R A T I V EF R A M E W O R K P R O J E C T

----------

Scorecard for Evaluation of Training Under Colorado’s Drinking Water Operator

Training Framework

November 2013

ACKNOWLEDGMENTS

This project was managed by Elizabeth Carter from the Aurora Water for the Rocky Mountain Section of the American Water Works Association (RMSAWWA).

Thanks are extended to the Demonstration Project Review Team for their guidance, review, and insights on all project materials:

Joe Cowan

Mary Presecan

Melissa Elliott

Sarah J. Dominick

Lori Pivonka

Armando Herald

David Dani

Thanks also go to Joy Barrett and William Hogrewe for coordination with Boulder School instructors and review of project materials.

Appreciation is given to the following instructors for testing the scorecard and content mapping tools:

Joy Barrett

Bill Hogrewe

Boyd Hanzon

David Dani

Sidney Innerebner

Gayle A. Lammers

Kim Evezich

Laura Travis

Jay Mashburn

Gary Parham

Kirk Watson

THIS DOCUMENT PREPARED BY:

Jeff Oxenford, Oxenford Consulting, LLC, on behalf of RMSAWWA.

Contact Information: [email protected], (720) 353-4242

i

TABLE OF CONTENTS

APPROACH The Scorecard .........................................................................................................................1 Data Analysis ..........................................................................................................................2

RESULTS ....................................................................................................................................2 1. Training Design and Content ..............................................................................................3 2. Instructor Qualifications .....................................................................................................7 3. Evaluation Procedures ........................................................................................................9 4. Curriculum Pathways ........................................................................................................10

SUMMARY OF CONCLUSIONS AND RECOMMENDATIONS .....................................12 NEXT STEPS ............................................................................................................................12 APPENDIX A: COLORADO DRINKING WATER OPERATOR TRAINING

FRAMEWORK SCORECARD ..................................................................................15 APPENDIX B: WATER TREATMENT PATHWAY ..........................................................21 APPENDIX C: DISTRIBUTION PATHWAY ......................................................................28 APPENDIX D: UNDERSTANDING ABC CERTIFICATION EXAMS

AND LEVELS OF LEARNING ..................................................................................37 APPENDIX E: SCORECARD EVALUATION CRITERIA ...............................................41

ii

A scorecard was developed and tested for evaluation of training courses under Colorado’s Drinking Water Operator Training Framework. The project was sponsored by Rocky Mountain Section of the American Water Works Association (RMSAWWA) under contract to the Local Assistance Unit of the Colorado Department of Public Health and Environment (CDPHE). This report summarizes the piloting of the scorecard with the Rocky Mountain Water Plant Operators School (commonly referred to as the Boulder School).

APPROACH

The Scorecard

The scorecard was developed as a tool to assess compliance of trainers with Colorado’s Drinking Water Operator Training Framework (the Framework) and was designed to evaluate:

Training design and content,

Instructor qualifications, and

Evaluation procedures.

The scorecard was developed using the online SurveyMonkey tool. This tool allows trainers to use the scorecard online, obtain individual reports, and prepare summary reports for multiple trainers. The scorecard is provided in Appendix A.

In addition to the scorecard, a content mapping tool was developed (see appendices B and C) to identify how training content relates to job tasks identified in the Task 2 Report: Curriculum Pathway and Core Curriculum Program Plan. These job tasks correlate with Tier 9, Core Technical Content in the Framework’s water treatment and distribution curriculum pathways. For each job task that the training addressed, respondents were to identify at what level of learning the content was taught. To support using the content mapping tool the document, “Understanding ABC Certification Exams and Levels of Learning” was provided to trainers (see Appendix D).

The scorecard and content mapping tool were sent to the trainers at the Boulder School who taught the course in 2013. Trainers were requested to self-assess the course modules they taught for the school. Those teaching just one module were asked to consider only the module they taught. Trainers with multiple modules were asked to take all modules into consideration when filling out the scorecard. To expand the sample size and further test the instrument, trainers for other CDPHE classes (referred to as RFA [Request for Application] trainers) were asked to participate in the self-assessment.

1

Data Analysis

To evaluate scorecard results, draft criteria were developed based on Framework recommendations. These are provided in Appendix E. Responses from both the Boulder School and RFA trainers were analyzed against these criteria. When differences between these two groups of trainers were observed, the results for each group were considered separately.

Respondent information is summarized in Table 1.

Table 1: Response Rates

Number of responses 11 Number of responses from Boulder School trainers 8 Number of responses from RFA trainers 3 Percentage of responses from trainers with more than 10 years of trainer experience 64% Percentage of trainers with a BA degree or higher 82% Percentages of trainers certified as an operator 27% Course length of 4 hours or fewer 73% Number of responses to the Distribution System Pathway 5 Number of responses to the Water Treatment Pathway 10

RESULTS

The scorecard results were analyzed against the draft criteria. The analysis of the results was organized in the following sections:

1. Training Design and Content – with the following subsections a. Learning objectives b. Learning event planning c. Instructional methods

2. Instructor Qualifications 3. Evaluation Procedures 4. Curriculum Pathways

a. Water Treatment Pathway b. Distribution Pathway

The results are presented by

Criteria,

Results, and

Discussion of the results.

2

1. Training Design and Content

1a. Learning Objectives

Criteria

The Framework specifies the following criteria related to learning objectives:

The training provider has clear, specific, and measurable written statements of intended learning outcomes.

Learning outcomes are clear, specific, measurable, and reflect what learners will achieve for each learning event.

Bloom’s taxonomy action verbs are used in writing learning outcomes.

Results

Eight respondents agreed that they had clear, specific, and measurable learning objectives; 3 disagreed with this statement. All 3 who disagreed were Boulder School trainers (3 of 8, 38%).

Only one respondent (of 11) agreed to using Bloom’s taxonomy in developing learning objectives.

All respondents provided learning objectives for their training. However, when compared to the criteria of clear, specific, and measureable, as well as using Bloom’s action verbs, the following responses were documented.

Only 4 (36%) met the criteria of actionable and measurable objectives (2 Boulder School, 2 RFA). Examples included that participants will be able to

o Work simple math problems; o Understand how to access and use CoWarn resources; o Distinguish the advantages and disadvantages of four types of backflow

preventers. Be able to prepare a sample plan for cross connections; o Demonstrate the ability to safely change out a chlorine gas cylinder.

Seven examples indicated that the learning objectives do not meet the proposed criteria: o Three of the respondents frame the objectives as “will educate/ teach.” (This type

of objective describes the content to be covered, but not what the students can do with the content.)

o Four of the respondents stated learning objectives as: students will understand (identify, be familiar with). The term understand can mean “comprehend, apply, or analyze,” which makes it difficult to determine whether the objective has been met.

3

Discussion of the Results

Although the majority of trainers (73% all, 63% Boulder School) indicated that there are clear, specific, and measurable learning objectives, when the examples provided are compared to the criteria of being specific, measurable, and using Bloom’s action verbs, only 36% (25% for Boulder School) met this criteria.

1b. Learning Event Planning

Criteria

The Framework specifies the following criteria related to learning event planning:

Each learning event is planned in response to the identified needs of the intended learners.

o The training provider shall use a learning needs analysis to develop learning objectives and content for the training.

o The process shall include information about how the needs are periodically evaluated to support the currency and relevancy of the learning event content.

Identified needs form the basis for planned learning outcomes. o The training provider shall have a process to ensure that the selected content

logically supports the learning outcomes. o The training provider shall document the relationship between needs analysis and

planned learning outcomes.

The training provider presents complete, accurate, and timely information about the learning experience available to learners in advance of the learning event.

o The training provider shall make learners aware of the information about the learning experience in advance of the learning event.

o Information about the learning experience shall include event description, logistics, learning outcomes; any prerequisites, technology, and other material requirements; and how the learner will be assessed.

Results

Seven (5 from the Boulder School) of 11 respondents agreed with the statement that a learning needs analysis was used to develop the content, 3 disagreed, and 1 strongly disagreed.

All 11 agreed (7) or strongly (4) agreed with the statement that training materials are periodically reviewed.

Eight of 11 disagreed or strongly disagreed with the statement that information about the learning experience is available to the learners in advance of the training. The three RFA trainers and 5 of 8 Boulder school trainers disagreed or strongly disagreed with this statement.

4


The results from these questions indicate that the majority of trainers agree that a learning needs analysis was conducted and that the information is periodically reviewed. However, the extent of needs analysis or frequency of content updating is not clear.

The majority of trainers (73%) believe they would not meet the criteria of providing information about the learning experience in advance of the training. All RFA trainers believed they would not meet these criteria.

1c. Instructional Methods

Criteria

The Framework specifies the following criterion related to instructional methods:

Instructional methods accommodate various learning styles and are designed to promote interaction between and among learners, instructors, and learning resources to achieve the stated learning outcomes.

Results

Approximately 80% agreed or strongly agreed that course design accounts for different learning styles and engages participants. Seven of 8 Boulder School trainers agreed or strongly agreed with this statement.

As shown in Figure 1, 85% of the classroom structure is conducted with the full group. Six of 8 Boulder School instructors indicated that 100% of the time is spent in large groups.

.0010.0020.0030.0040.0050.0060.0070.0080.0090.00

Full group Small group (2-8) Individually

FIGURE 1: PARTICIPATION STRUCTURE: Percentage of time participants spend in the following instructional groupings (note: percentages must add up

to 100%)

5

As shown in Figure 2, content is delivered through presentations, on average, 80% of the time. For the Boulder School,

Two instructors used presentation 95–100% of the time;

Two instructors used 90–95% presentations, with up to 10% of the time for large group discussion and questions;

Two instructors used presentations 80% of the time, with time used for solving problems and answering questions; and

Two instructors spent significant amounts of time in small-group activities.

The following techniques were used to engage learners in the full group:

Instructor asks if there are any questions

Instructor asks the audience a question

Vocal quizzes are given

Asking students to raise their hands in response to instructor questions

Leaving a blank in a slide and having the audience “fill in” the blank

Asking attendees to share strategies they have used

Asking attendees about their system

Having a student solve a math problem in front of the class

.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

List

enin

g to

pr

esen

tatio

ns

Wat

chin

g de

mon

stra

tions

Par

ticip

atin

g in

a la

rge

grou

p (d

iscu

ssio

n or

as

king

…

Par

ticip

atin

g in

a s

mal

l gr

oup

(dis

cuss

ion,

ha

nds

on …

Con

duct

ing

indi

vidu

al

task

s

Oth

er

FIGURE 2: TYPES OF LEARNING ACTIVITIES: Percentage of time participants spend in different learning activities (note: percentages must add

up to 100%)

6

Small-group strategies include

Performing a calculation as a small group and then solving it in front of the class,

Touring a facility, and

Creating a presentation and/or action plan.


The data are mixed when evaluating the results against the criteria, Instructional methods accommodate various learning styles and are designed to promote interaction. Seven of 8 instructors believe that course design accounts for different learning styles and engages participants. However, 6 of 8 instructors spend 100% of their time in the large group, and only 4 (half) use strategies other than presentation and questions to the audience. Only two instructors use small-group activities. Accommodating different learning styles is difficult in the large-group setting as is promoting interaction during presentations.

2. Instructor Qualifications

Criteria

The Framework specifies the following criteria related to qualified instructors:

The training provider shall document that individuals involved in the design, development, delivery, and evaluation of learning events are qualified in their assigned roles.

o They must be competent in the learning event content, as measured by some combination of education, experience, and/or certification.

o They must be knowledgeable in instructional methods and learning processes, as measured by being credentialed and/or trained in planning and/or facilitating the learning event.

Individuals involved in the development and delivery of learning events shall remain current in subject matter material and learning methods.

o The training provider shall have a process to document professional development activities for ensuring that individuals involved in designing, developing, and delivering learning events remain current in subject matter material and learning methods.

Policies shall be in place for the development and delivery of learning events to demonstrate high standards of professional conduct.

o The training provider shall have a policy that does not discriminate or make explicit references of a discriminatory nature based on gender, ethnicity, religion, age, disability, socioeconomic status, and/or sexual orientation.

o The training provider shall have a policy that discloses to learners, prior to the start of the learning event, any instructor’s proprietary interest in any product,

7

instrument, device, service, or material discussed, as well as the source of any compensation related to the presentation.

o The training provider shall have a policy that addresses the ownership of intellectual property rights.

Results

More than 90% of respondents agreed or strongly agreed that they have practical, in-the-field experience with the content. Only 1 disagreed.

100% agreed or strongly agreed that they have the appropriate level of education and certification.

More than 90% agreed or strongly agreed that they are knowledgeable in instructional methods and the learning process. Only 1 disagreed.

100% agreed or strongly agreed that the trainers remain current in the subject matter.

100% agreed or strongly agreed that the instructors demonstrated a high standard of professional conduct.

55% agreed or strongly agreed that trainers obtain and document their professional development on developing and delivering learning events.

45% agreed or strongly agreed that trainers obtain and document their professional development on the subject matter material.

36% had a policy or procedure in place to ensure that the training does not discriminate; 1 stated that this does not apply.

55% disclosed the instructor’s proprietary interest; 3 (27%) identified that this does not apply.

36% agreed or strongly agreed with the statement that there is a policy or procedure to address intellectual property rights; 45% disagreed or strongly disagreed, and 2 state that this does not apply.

When asked to write comments on the necessary level of experience needed for a training program, comments included

Replaced non-operators with operators as instructors as long as they are good trainers,

There is no policy in place for level of experience needed,

Trainers are either a certified operator or PE [professional engineer], and

Familiarization with the concepts are needed to be a trainer.

8

Conclusions/Recommendations

The vast majority of trainers identified that they would meet criteria by having appropriate qualifications, practical experience, and knowledge in the subject matter and learning methods. When asked to provide comments on the necessary level of experience, all comments focused on technical/professional roles, not on development as a trainer. Trainers did not seem to acknowledge the skills/expertise required for instructional methods and learning processes.

All trainers agreed or strongly agreed that they remain current in the subject matter. However, only 50% agreed or strongly agreed with the statement that individuals should obtain and document professional development in the subject matter or learning methods. While not explicitly identified, it seems that there is a gap in providing documentation of professional development.

The responses indicate that policies and procedures to prevent discrimination, disclose proprietary information, and address intellectual property are not commonly in place. One comment stated that “as a single member firm, we don’t have policies other than good sense.” Given that the majority of trainers are typically individuals providing training rather than being under the auspices of a training company, one conclusion is that these policies may need to be put in place at the school level or from CDPHE.

3. Evaluation Procedures

Criteria

The Framework specifies the following criteria related to evaluation procedures:

The training provider measures what participants thought and believed about training programs.

The training provider evaluates student learning against the learning objectives.

The training provider evaluates the transfer of student learning to application on the job.

Results

100% agreed or strongly agreed that there was an evaluation of participation perceptions of the training program.

63% agreed or strongly agreed that there was an evaluation of student learning against the learning objectives.

Only 20% agreed or strongly agreed that there was a measure of post-course effectiveness.

When asked to describe evaluation procedures, the following techniques were identified:

Post training evaluation forms.

9

Ask for comments on what worked well and what could be improved in the training.

Provide pre- and post-tests.

Conclusions/Recommendations

The results indicate that all trainers achieved Criteria 1, an evaluation of what participants believed and thought about the training. Two-thirds identified using a pre- or post-test to evaluate learning against learning objectives. Only 20% attempted to measure post-course effectiveness, but no examples were provided of how this was done.

4. Curriculum Pathways

Two content mapping surveys were used to identify how training content relates to job tasks identified in Task 2 Report: Curriculum Pathway and Core Curriculum Program Plan. These job tasks correlated with Tier 9, Core Technical Content, in the Framework’s water treatment and distribution curriculum pathways.

4a. Water Treatment Pathway

Ten responses were provided for the water treatment pathway. Six Boulder School instructors described their specific modules. Two Boulder School instructors described greater effort, that is, Intermediate School and Industrial Wastewater Treatment. Two RFA instructors also provided comments.

Results

Individual results were analyzed for all respondents. Summary results were not used in the analysis because they were not generally informative.

Of the six Boulder School instructors who evaluated their modules, the following data were obtained:

An average of 11 job tasks identified per module.

The range was 5 to 26 topics.

Four respondents identified 8 or fewer topics.

Three respondents selected only one Bloom’s level (comprehension or application).

Two respondents identified 1–3 topics with more than one Bloom’s level.

One respondent identified 24 topics at all four levels.

For the two Boulder School instructors that evaluated larger courses, each identified more than 100 topics, one with 50% at all four levels, and one with only one level for all topics.

For the two RFA instructors, one identified 35 topics with 16 of them at four levels, and one identified 23 topics with 2 of them at four levels and 13 at two to three levels.

10


The results were quite variable with different instructors selecting a different number of topics and diverse levels of coverage. In general, trainers identified too many topics to cover in one class, or for students to retain that amount of knowledge. Also, some instructors selected many topics at multiple levels—again, too much material to be covered. It is clear from the results that there needs to be more information and training on these topics and levels of training. Training should include how the water treatment pathway can be tied to the learning objectives. Criteria need to be developed. Based on this observation, the criteria should include items such as how many topics and to what level should be covered in a certain course length.

4b. Distribution Pathway

Five responses were provided for the distribution pathway, 3 from Boulder School instructors and 2 from RFA instructors.

Results

Because of a glitch in the survey, instructors could only select one Bloom’s level for the distribution pathway. In almost all cases, comprehension and application levels were identified. Synthesis was identified for one topic by an RFA instructor.

Boulder School instructors identified addressing between 6 and 13 topics; the two RFA instructors identified 3 and 11 topics, respectively.

Conclusion

In general, respondents were more targeted to job tasks for the Distribution Pathway. Two of the three Boulder School instructors identified similar topics for sessions on instrumentation and control versus pump, motors, and valves. More data are needed on the Distribution Pathway.

11

SUMMARY OF CONCLUSIONS AND RECOMMENDATIONS

A general summary of the results is provided in Appendix E.

1. Lack of clarity on instruction methods. The results indicate that trainers believe they are using proper instructional methods. When asked, the majority stated that they include learning objectives, needs analysis, and have engaged students. This is not surprising since the model for training in the water industry has been the subject matter expert as the trainer, using PowerPoint presentations in a classroom setting. Asking questions at the end of the presentation is typically used to generate engagement. Using this model, the Boulder School has been operating well.

However, when applying the concepts of the Framework, training does not meet expectations. The Framework is offering a new paradigm for training. Terms such as learning needs analysis, learning objectives, engagement, and so on, are new to water industry training as well as the idea of being versed in technical content as well as instructional methods.

The gap exposed by applying the scorecard to existing training programs is the improvement in instructional design. There needs to be recognition of what constitutes best instructional practices. There also needs to be resources and training available to assist subject matter experts in adding instructional design to their skill sets.

2. Too much material being covered. Results using the content mapping tool suggest that training classes may be attempting to address too many topics at too many levels of Bloom’s taxonomy for the time allotted. This result is not unexpected. A common challenge in designing training has always been what to include and what to omit. In presentation/lecture-type course design, the tendency is to include as much content as possible. However, in applying the Framework and establishing clear learning objectives, the old adage of “less is more” may need to be applied to course design.

Results were also not clear on whether topics were covered in a cursory fashion or in a comprehensive way. This level of detail would be needed by operators in determining whether classes fully address the topics as identified in the curriculum pathways or if additional training sessions would be needed. This would also be needed to place courses in a training catalog or for use with search features on a training website.

3. Evaluation criteria need to be better defined. A key to achieving better results is to clearly define the criteria/expectations for training. Although the Framework provides some generalized concepts and goals, the criteria need to be more clearly defined. A collaborative process is needed to establish the criteria, similar to that used when developing the Framework.

4. The scorecard needs to be refined. Small changes can be made to the scorecard to better address issues identified in this report. Once the criteria are better defined, the scorecard should be fully aligned with the criteria.

12

NEXT STEPS

The Framework is providing new expectations for trainers in Colorado. In order to set the stage for these new expectations, trainers will need the following:

Clear statements of the expectations and criteria for determining whether the expectations have been met.

Policy statements such as intellectual property, non-discriminatory practices, and disclosure of financial interest.

Training on these expectations and policies, what they are, and how they will be measured.

Training on new concepts in the Framework, such as adult learning, instructional design, using Bloom’s taxonomy, and so forth.

1. Ideally the expectations, criteria, and policies would be developed before having trainers make changes to current training. However, since it will take some time to establish and finalize the criteria and policies, and because training most likely will need to occur before this is accomplished, the following approach is recommended:

2. Establish a stakeholder committee to clarify the criteria and expectations. This should be a joint effort between CDPHE and RMSAWWA and should use a stakeholder process such as that used in development of the Framework.

3. Develop policy statements for topics such as intellectual property, non-discriminatory practices, disclosure of financial interest, and so on. This could be done either by the Boulder School for their trainers or by CDPHE for all trainers.

4. Develop a train-the-trainer program. The program should include items such as

The new expectations of training as described in the Framework,

Areas were the Framework may be changing (i.e., better defined criteria) in the future and a process for informing instructors of future changes,

Concepts of adult learning,

Instructional design best practices,

Techniques to improve knowledge retention, and

Resources to improve instructional design. 5. Make changes to the current program. Since the Framework has significantly increased

the expectations related to training design and curriculum, resources (time and funding) will be needed to redesign the curriculum. Experience has shown that it takes significantly more time to develop a curriculum than building a PowerPoint presentation.

The curriculum pathways provide a new opportunity for operators to identify specific classes they may need to advance their knowledge. The pathways can also be used to categorize/classify courses on a training website and/or in a course catalog. The challenge will be in identifying how each course fulfills the knowledge expectations of a topic on the pathway. To accomplish this, the following approach is recommended:

13

1. Develop clear knowledge expectations for each topic in the curriculum pathways. 2. Establish criteria and a process for determining whether the courses meet knowledge

expectations. 3. Develop a training catalog and/or website that will allow operators to be able to search

for courses that address topics in the curriculum pathways.

14

APPENDIX A

COLORADO DRINKING WATER OPERATOR TRAINING FRAMEWORK SCORECARD

15

Colorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework Scorecard

This training scorecard was developed to support the implementation of the Colorado’s Drinking Water Operator Training Framework. The full framework can be downloaded at: http://www.rmsawwa.net/PDFs/A2C_Task_7_Final_070213.pdf The scorecard can be used by Colorado trainers to assess their compliance with the Framework and evaluates: 1. Training design and content 2. Instructor qualifications 3. Evaluation procedures The scorecard should also be used in combination with the appropriate curriculum pathways (water treatment or water distribution). This scorecard is being piloted with the Rocky Mountain Water Plant Operator School (commonly referred to as the Boulder School). Trainers are requested to conduct a selfassessment of their curriculum. For trainers teaching only one module, please consider only that module. For trainers that are involved with multiple modules, take all modules in consideration when filling out the scorecard. Results will be collected and analyzed and recommendations will be provided for improvement of the school. The effectiveness of the scorecard is also being evaluated. Feel free to provide any comments you may have to Jeff Oxenford at [email protected]. The training scorecard was developed for the Rocky Mountain Section of the AWWA under contract to the Local Assistance Unit of CDPHE.

GENERAL INFORMATION

1. Your Name:

2. Years you have provided training services (select one)

Training Scorecard

55

66

1 or less

nmlkj

15

nmlkj

510

nmlkj

1020

nmlkj

20 or more

nmlkj

16

Colorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework Scorecard3. Highest level of education (select one)

4. Are you a certified operator for (select all that apply)

5. Training course name (Can be a training program, specific course, or module within a course) – (write in response)

6. Course length (select one)

7. LEARNING OBJECTIVES: List the learning objectives for the training course (write in response)

8. PARTICIPATION STRUCTURE: Indicate the percentage of time participants will spend in the following instructional groupings (note: percentages must add up to 100%):

55

66

55

66

Full group

Small group (28)

Individually

High school degree

gfedc

Associates degree

gfedc

Bachelor degree

gfedc

Masters degree

gfedc

PhD.

gfedc

Not certified

gfedc

Water Treatment

gfedc

Water Distribution

gfedc

Small Water System

gfedc

1 hour or less

nmlkj

1 hour to 4 hours

nmlkj

4 hours to 8 hours

nmlkj

816 hours

nmlkj

More than 16 hours

nmlkj

17

Colorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework Scorecard9. TYPES OF LEARNING ACTIVITIES: Indicate the percentage of time participants will spend (note: percentages must add up to 100%)

10. Other learning activity (please specify)

11. MEASURES FOR OBJECTIVES: List methods and/or metrics used for determining if learning objectives are achieved (write in response)

12. AUDIENCE: Who is the intended audience for the training? Select your primary and secondary audiences for your training (select choices primary, secondary, not an audience)

Listening to presentations

Watching demonstrations

Participating in a large group (discussion or asking questions)

Participating in a small group (discussion, hands on activity, or small group tasks)

Conducting individual tasks

Other

55

66

55

66

Primary Secondary Not an audience

Water Treatment Level A nmlkj nmlkj nmlkj

Water Treatment Level B nmlkj nmlkj nmlkj

Water Treatment Level C nmlkj nmlkj nmlkj

Water Treatment Level D nmlkj nmlkj nmlkj

Water Distribution 4 nmlkj nmlkj nmlkj




Small Water System Operators nmlkj nmlkj nmlkj

Other (please specify)

18

Colorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework Scorecard13. CONTENT DESIGN

14. Provide examples of strategies used to engage participants (write a response)

15. INSTRUCTOR(S) QUALIFICATIONS

16. Provide an comments on the necessary level of experience needed for the training program (write a response)

Strongly agree Agree DisagreeStrongly disagree

A learning needs analysis was used to develop learning objectives and content for the training

nmlkj nmlkj nmlkj nmlkj

There are clear, specific, and measurable learning objectives nmlkj nmlkj nmlkj nmlkj

Bloom's taxonomy was used in the design of learning objectives nmlkj nmlkj nmlkj nmlkj

Course design accounts for different learning styles and engages participants nmlkj nmlkj nmlkj nmlkj

Information on the learning experience (learning outcomes, prerequisites, training environment…) is available to learners in advance of the learning event.


Training materials are periodically evaluated to ensure that the content remains current and relevant


55

66

Strongly agree

Agree DisagreeStrongly disagree

Does not apply

Trainer(s) have practical, in the field, experience with the content nmlkj nmlkj nmlkj nmlkj nmlkj

Trainer(s) have appropriate level of education and or certification nmlkj nmlkj nmlkj nmlkj nmlkj

Trainer(s) are knowledgeable in instructional methods and learning processes. nmlkj nmlkj nmlkj nmlkj nmlkj

Individual(s) involved in the development and delivery of learning events remain current in subject matter material and learning methods.

nmlkj nmlkj nmlkj nmlkj nmlkj

Trainer(s) obtain and document their professional development in designing, developing and delivering learning events


Trainer(s) obtain and document their professional development on the subject matter material


Individual(s) involved in the development and delivery of learning events demonstrate high standards of professional conduct.


A policy or procedure is in place to ensure that the training does not discriminate or make explicit references of a discriminatory nature based on gender, ethnicity, religion, age, disability, socioeconomic status, and/or sexual orientation.


Trainer(s) disclose to learners, prior to the start of the learning event, instructor’s proprietary interest in any product, instrument, device, service, or material discussed, as well as the source of any compensation related to the presentation.


There is a policy or procedure that addresses the ownership of intellectual property rights for content delivered in the training


55

66

19

Colorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework ScorecardColorado Drinking Water Operator Training Framework Scorecard17. EVALUATION PROCEDURES

18. Describe evaluation procedures use (write a response)

19. FEEDBACK: Please provide any comment or suggestions you have for improving this survey

Strongly agree

Agree DisagreeStrongly disagree

Not applicable

There is an evaluation form to evaluate participant's perception of the training experience (i.e. rate the training and trainer)


There are opportunities for participants to provide comments and feedback during the training (i.e. such that training can be modified midstream to better meet participants needs)


There is an evaluation of student learning against the learning objectives nmlkj nmlkj nmlkj nmlkj nmlkj

There is a postcourse evaluation of the effectiveness of student transfer and application of the learning from the classroom to the job


55

66

55

66

20

APPENDIX B

WATER TREATMENT PATHWAY

21

Water Treatment PathwayWater Treatment PathwayWater Treatment PathwayWater Treatment Pathway

1. Name

2. Name of training course

In this section you will be asked to identify any job task that your training class addresses. For any job task covered in your training, please identify the level to which you are teaching. The levels are briefly described below. For a more detailed discussion of the levels, please refer to the document on teaching levels attached. Comprehension comprehension includes understanding the meaning, translation, interpolation, and interpretation of instructions and problems as well as stating a problem in one’s own words. Application application is using a concept in a new situation or unprompted use of an abstraction. The operator applies what was learned in the classroom to new situations in the work place Analysis the operator separates material or concepts into component parts so that its organizational structure may be understood. The operator distinguishes between facts and inferences. Synthesis the operator is able to assimilate the information learned in the classroom into his/her work environment.

*55

66

*55

66

22

Water Treatment PathwayWater Treatment PathwayWater Treatment PathwayWater Treatment Pathway3. Monitor, Evaluate, and Adjust Treatment Processes For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).

Comprehension Application Analysis Synthesis

Chloramination gfedc gfedc gfedc gfedc

Chlorine disinfection with calcium hypochlorite gfedc gfedc gfedc gfedc

Chlorine disinfection with gaseous chlorine gfedc gfedc gfedc gfedc

Chlorine disinfection with sodium hypochlorite gfedc gfedc gfedc gfedc

Corrosion control gfedc gfedc gfedc gfedc

Disinfection with chlorine dioxide gfedc gfedc gfedc gfedc

Disinfection with ozone gfedc gfedc gfedc gfedc

Disinfection with ultraviolet light gfedc gfedc gfedc gfedc

Fluoridation gfedc gfedc gfedc gfedc

pH adjustment gfedc gfedc gfedc gfedc

Perform basic water mathematic calculations such as CT and dosage, and physical measurements

gfedc gfedc gfedc gfedc

Prepare and control chemical addition gfedc gfedc gfedc gfedc

Coagulation aid polymer and filter aid gfedc gfedc gfedc gfedc

Coagulation and enhanced coagulation gfedc gfedc gfedc gfedc

Flocculation tanks/basins gfedc gfedc gfedc gfedc

Clarification gfedc gfedc gfedc gfedc

Dissolved air flotation gfedc gfedc gfedc gfedc

Sedimentation basins gfedc gfedc gfedc gfedc

Bag filters gfedc gfedc gfedc gfedc

Cartridge filters gfedc gfedc gfedc gfedc

Diatomaceous earth filters gfedc gfedc gfedc gfedc

Granular media filtration and backwashing gfedc gfedc gfedc gfedc

Membrane filtration gfedc gfedc gfedc gfedc

Slow sand filters gfedc gfedc gfedc gfedc

Sludge conditioning gfedc gfedc gfedc gfedc

Sludge drying beds gfedc gfedc gfedc gfedc

Solids handling gfedc gfedc gfedc gfedc

Activated carbon gfedc gfedc gfedc gfedc

Aeration gfedc gfedc gfedc gfedc

Ion exchange gfedc gfedc gfedc gfedc

Iron & Manganese Removal gfedc gfedc gfedc gfedc

Reverse osmosis and air stripping gfedc gfedc gfedc gfedc

Softening gfedc gfedc gfedc gfedc

23


4. Laboratory Analyses For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).


Chemical feed rate adjustment gfedc gfedc gfedc gfedc

Instrumentation and control gfedc gfedc gfedc gfedc

Taste and odor control gfedc gfedc gfedc gfedc

Water treatment processes and design parameters gfedc gfedc gfedc gfedc

Analyze process control samples gfedc gfedc gfedc gfedc

Collect process control samples gfedc gfedc gfedc gfedc

Interpret lab results gfedc gfedc gfedc gfedc

Maintain logs and records gfedc gfedc gfedc gfedc

Chemical formulae and reactions gfedc gfedc gfedc gfedc

Chemical handling, storage and disposal gfedc gfedc gfedc gfedc

Concentrations (e.g. molarity and normality) gfedc gfedc gfedc gfedc

Instrument calibration and operation gfedc gfedc gfedc gfedc

Oxygen saturation gfedc gfedc gfedc gfedc

Quality control and assurance gfedc gfedc gfedc gfedc

Sampling procedures gfedc gfedc gfedc gfedc

Serial dilutions gfedc gfedc gfedc gfedc

Standard Methods for the Examination of water and wastewater gfedc gfedc gfedc gfedc

Streaming current testing gfedc gfedc gfedc gfedc

Alkalinity testing gfedc gfedc gfedc gfedc

Bacterial testing gfedc gfedc gfedc gfedc

Chlorine demand testing gfedc gfedc gfedc gfedc

Carbon dioxide residual testing gfedc gfedc gfedc gfedc

Chlorine residual testing gfedc gfedc gfedc gfedc

Fluoride testing gfedc gfedc gfedc gfedc

Hardness testing gfedc gfedc gfedc gfedc

Iron testing gfedc gfedc gfedc gfedc

Jar testing gfedc gfedc gfedc gfedc

Manganese testing gfedc gfedc gfedc gfedc

Ozone residual testing gfedc gfedc gfedc gfedc

pH testing gfedc gfedc gfedc gfedc

Temperature testing gfedc gfedc gfedc gfedc

Settleable solids testing gfedc gfedc gfedc gfedc

Turbidity testing gfedc gfedc gfedc gfedc

24


5. Comply with Drinking Water Regulations For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).

Zeta testing gfedc gfedc gfedc gfedc


Colorado Primary Drinking Water Regulations gfedc gfedc gfedc gfedc

Colorado Operators Certification Regulation No. 100 gfedc gfedc gfedc gfedc

CDPHE Laboratory Manual gfedc gfedc gfedc gfedc

CDPHE Guidance Manual gfedc gfedc gfedc gfedc

Safe Drinking Water Act gfedc gfedc gfedc gfedc

Regulatory monitoring gfedc gfedc gfedc gfedc

25

Water Treatment PathwayWater Treatment PathwayWater Treatment PathwayWater Treatment Pathway6. Operate and Maintain Equipment For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).


Diagnose, troubleshoot, and adjust facility processes gfedc gfedc gfedc gfedc

Interpret meter and/or gauge readings gfedc gfedc gfedc gfedc

Backflow prevention devices gfedc gfedc gfedc gfedc

Crossconnection control gfedc gfedc gfedc gfedc

Electrical controls gfedc gfedc gfedc gfedc

Flow measurement (e.g. weirs) gfedc gfedc gfedc gfedc

Hydraulic principles gfedc gfedc gfedc gfedc

Instrumentation and control gfedc gfedc gfedc gfedc

Mechanical principles gfedc gfedc gfedc gfedc

Pressure control gfedc gfedc gfedc gfedc

Respond to variations in operating conditions gfedc gfedc gfedc gfedc

Diagnose/troubleshoot equipment gfedc gfedc gfedc gfedc

Inspect equipment for abnormal conditions gfedc gfedc gfedc gfedc

Perform basic mathematical calculations, such as drawdown and pumping rate

gfedc gfedc gfedc gfedc

Blowers and compressors gfedc gfedc gfedc gfedc

Centrifugal pumps gfedc gfedc gfedc gfedc

Chemical feed systems gfedc gfedc gfedc gfedc

Chorinators and chlorine systems gfedc gfedc gfedc gfedc

Diffusers gfedc gfedc gfedc gfedc

Engines gfedc gfedc gfedc gfedc

Fittings gfedc gfedc gfedc gfedc

Hydraulic equipment gfedc gfedc gfedc gfedc

Intake structures gfedc gfedc gfedc gfedc

Mechanical drive systems gfedc gfedc gfedc gfedc

Mixers gfedc gfedc gfedc gfedc

Motors gfedc gfedc gfedc gfedc

Pipes gfedc gfedc gfedc gfedc

Positive displacement pumps gfedc gfedc gfedc gfedc

Valves gfedc gfedc gfedc gfedc

Perform routine maintenance functions gfedc gfedc gfedc gfedc

Read meters gfedc gfedc gfedc gfedc

Startup and shutdown gfedc gfedc gfedc gfedc

26

Water Treatment PathwayWater Treatment PathwayWater Treatment PathwayWater Treatment Pathway7. Perform Security, Safety, and Administrative Procedures For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).

8. Evaluate Characteristics of Source Water For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).


Administer compliance program gfedc gfedc gfedc gfedc

Administer emergency preparedness program gfedc gfedc gfedc gfedc

Administer maintenance program gfedc gfedc gfedc gfedc

Administer safety program gfedc gfedc gfedc gfedc

Develop and administer budget gfedc gfedc gfedc gfedc

Develop operation and maintenance plan gfedc gfedc gfedc gfedc

Identify potential safety hazards gfedc gfedc gfedc gfedc

Perform basic mathematical calculations, such as treatment costs gfedc gfedc gfedc gfedc

Plan work activities gfedc gfedc gfedc gfedc

Purchase supplies & equipment gfedc gfedc gfedc gfedc

Train staff gfedc gfedc gfedc gfedc

Maintain logs and/or records gfedc gfedc gfedc gfedc

Recordkeeping and reporting requirements gfedc gfedc gfedc gfedc

Operate safety equipment gfedc gfedc gfedc gfedc

Recognize unsafe work conditions gfedc gfedc gfedc gfedc

Chemical handling and storage gfedc gfedc gfedc gfedc

Confined space protocols gfedc gfedc gfedc gfedc

Electrical safety gfedc gfedc gfedc gfedc

Fire safety gfedc gfedc gfedc gfedc

Housekeeping gfedc gfedc gfedc gfedc

Lockout and tagout gfedc gfedc gfedc gfedc

Material Safety Data Sheets gfedc gfedc gfedc gfedc

Personal protective equipment gfedc gfedc gfedc gfedc


Algae control in reservoirs gfedc gfedc gfedc gfedc

Groundwater conditions gfedc gfedc gfedc gfedc

Hydrology gfedc gfedc gfedc gfedc

Normal characteristics of water gfedc gfedc gfedc gfedc

Sources and their characteristics gfedc gfedc gfedc gfedc

27

APPENDIX C

DISTRIBUTION PATHWAY

28

Distribution PathwayDistribution PathwayDistribution PathwayDistribution Pathway

1. Name

2. Name of training course

in this section you will be asked to identify any job task that your training class addresses. For any job task covered in your training, please identify the level to which you are teaching. The levels are briefly described below. For a more detailed discussion of the levels, please refer to the document on teaching levels attached. Comprehension comprehension includes understanding the meaning, translation, interpolation, and interpretation of instructions and problems as well as stating a problem in one’s own words. Application application is using a concept in a new situation or unprompted use of an abstraction. The operator applies what was learned in the classroom to new situations in the work place Analysis the operator separates material or concepts into component parts so that its organizational structure may be understood. The operator distinguishes between facts and inferences. Synthesis the operator is able to assimilate the information learned in the classroom into his/her work environment.

Distribution Pathway

*55

66

*55

66

29

Distribution PathwayDistribution PathwayDistribution PathwayDistribution Pathway3. System Information Components For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).


Assess system demand gfedc gfedc gfedc gfedc

Measure and evaluate head loss gfedc gfedc gfedc gfedc

Monitor alarms gfedc gfedc gfedc gfedc

Perform pressure testing gfedc gfedc gfedc gfedc

Read and evaluate chart results gfedc gfedc gfedc gfedc

Read and evaluate gauges gfedc gfedc gfedc gfedc

Read and evaluate meter results gfedc gfedc gfedc gfedc

Read blueprints, readings & maps gfedc gfedc gfedc gfedc

Record and evaluate data gfedc gfedc gfedc gfedc

Select materials gfedc gfedc gfedc gfedc

Select type of pipes gfedc gfedc gfedc gfedc

Size mains gfedc gfedc gfedc gfedc

Alarms gfedc gfedc gfedc gfedc

Fittings/piping gfedc gfedc gfedc gfedc

Hydrants gfedc gfedc gfedc gfedc

Joint restraint design gfedc gfedc gfedc gfedc

Pipeline assessment gfedc gfedc gfedc gfedc

Service connections gfedc gfedc gfedc gfedc

Shoring design gfedc gfedc gfedc gfedc

System layout gfedc gfedc gfedc gfedc

System map gfedc gfedc gfedc gfedc

Thrust block design gfedc gfedc gfedc gfedc

Treated water storage gfedc gfedc gfedc gfedc

Trenching and shoring gfedc gfedc gfedc gfedc

Valves gfedc gfedc gfedc gfedc

Water mains gfedc gfedc gfedc gfedc

Wells gfedc gfedc gfedc gfedc

Cross connection surveys/control gfedc gfedc gfedc gfedc

Crossconnections gfedc gfedc gfedc gfedc

Flow monitoring gfedc gfedc gfedc gfedc

Flow pattern / water age gfedc gfedc gfedc gfedc

Flushing program gfedc gfedc gfedc gfedc

Physical inspection gfedc gfedc gfedc gfedc

Volume gfedc gfedc gfedc gfedc30


4. Monitor Evaluate, and Adjust Disinfection For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).

5. Laboratory Analysis For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).

Sample site plan gfedc gfedc gfedc gfedc


pH nmlkj nmlkj nmlkj nmlkj

Temperature nmlkj nmlkj nmlkj nmlkj

Turbidity nmlkj nmlkj nmlkj nmlkj

Chlorine demand nmlkj nmlkj nmlkj nmlkj

Chlorine residual nmlkj nmlkj nmlkj nmlkj

Chlorination process nmlkj nmlkj nmlkj nmlkj

Dechlorination process nmlkj nmlkj nmlkj nmlkj

Disinfection byproducts nmlkj nmlkj nmlkj nmlkj

Chemical feeders nmlkj nmlkj nmlkj nmlkj

Chlorinators nmlkj nmlkj nmlkj nmlkj

Chemical handling nmlkj nmlkj nmlkj nmlkj


Bacteriological nmlkj nmlkj nmlkj nmlkj

Chain of custody nmlkj nmlkj nmlkj nmlkj

Chlorine demand nmlkj nmlkj nmlkj nmlkj

Chlorine residual nmlkj nmlkj nmlkj nmlkj

Hardness nmlkj nmlkj nmlkj nmlkj

pH nmlkj nmlkj nmlkj nmlkj

Temperature nmlkj nmlkj nmlkj nmlkj

Total Organic Carbon (TOC) nmlkj nmlkj nmlkj nmlkj

Turbidity nmlkj nmlkj nmlkj nmlkj

Microbiological/microscopic nmlkj nmlkj nmlkj nmlkj

Nitrification nmlkj nmlkj nmlkj nmlkj

Disinfection byproducts (DBP) nmlkj nmlkj nmlkj nmlkj

Bloodborne pathogens/infectious disease nmlkj nmlkj nmlkj nmlkj

31

Distribution PathwayDistribution PathwayDistribution PathwayDistribution Pathway6. Install equipment For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).


Hydrants nmlkj nmlkj nmlkj nmlkj

Meters nmlkj nmlkj nmlkj nmlkj

Piping nmlkj nmlkj nmlkj nmlkj

Service connections nmlkj nmlkj nmlkj nmlkj

Shoring nmlkj nmlkj nmlkj nmlkj

Taps nmlkj nmlkj nmlkj nmlkj

Valves nmlkj nmlkj nmlkj nmlkj

Water lines/mains nmlkj nmlkj nmlkj nmlkj

32

Distribution PathwayDistribution PathwayDistribution PathwayDistribution Pathway7. Operate Equipment For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).


Backflow prevention devices nmlkj nmlkj nmlkj nmlkj

Blowers and compressors nmlkj nmlkj nmlkj nmlkj

Boring equipment nmlkj nmlkj nmlkj nmlkj

Cathodic protection systems nmlkj nmlkj nmlkj nmlkj

Chlorinators nmlkj nmlkj nmlkj nmlkj

Controllers nmlkj nmlkj nmlkj nmlkj

Drives nmlkj nmlkj nmlkj nmlkj

Electrical controls nmlkj nmlkj nmlkj nmlkj

Electronic testing equipment nmlkj nmlkj nmlkj nmlkj

Engines nmlkj nmlkj nmlkj nmlkj

Fittings/Piping nmlkj nmlkj nmlkj nmlkj

Flow meters nmlkj nmlkj nmlkj nmlkj

Fuses nmlkj nmlkj nmlkj nmlkj

Generators nmlkj nmlkj nmlkj nmlkj

Hand and power tools nmlkj nmlkj nmlkj nmlkj

Heavy equipment nmlkj nmlkj nmlkj nmlkj

Hydrants nmlkj nmlkj nmlkj nmlkj

Hydraulic equipment nmlkj nmlkj nmlkj nmlkj

Inspection equipment nmlkj nmlkj nmlkj nmlkj

Meters nmlkj nmlkj nmlkj nmlkj

Motor controls nmlkj nmlkj nmlkj nmlkj

Motors nmlkj nmlkj nmlkj nmlkj

Pigging nmlkj nmlkj nmlkj nmlkj

Pneumatic equipment nmlkj nmlkj nmlkj nmlkj

Pressure relief valves nmlkj nmlkj nmlkj nmlkj

Pressure sensors nmlkj nmlkj nmlkj nmlkj

Pumps nmlkj nmlkj nmlkj nmlkj

SCADA/Process control computers nmlkj nmlkj nmlkj nmlkj

Service connections nmlkj nmlkj nmlkj nmlkj

Tapping equipment nmlkj nmlkj nmlkj nmlkj

Valves nmlkj nmlkj nmlkj nmlkj

Variable speed drives nmlkj nmlkj nmlkj nmlkj

Vaults/Manholes nmlkj nmlkj nmlkj nmlkj

Water mains nmlkj nmlkj nmlkj nmlkj33


8. Perform Maintenance For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).

Water storage nmlkj nmlkj nmlkj nmlkj


Check and evaluate capacity of equipment nmlkj nmlkj nmlkj nmlkj

Corrosion control nmlkj nmlkj nmlkj nmlkj

Inspect equipment for abnormal conditions nmlkj nmlkj nmlkj nmlkj

Leak detection nmlkj nmlkj nmlkj nmlkj

Perform pressure readings nmlkj nmlkj nmlkj nmlkj

Troubleshoot electrical equipment nmlkj nmlkj nmlkj nmlkj

Well inspection nmlkj nmlkj nmlkj nmlkj

34

Distribution PathwayDistribution PathwayDistribution PathwayDistribution Pathway9. Perform Security, Safety, and Administration Procedures For each job task that is being addressed in your training, select the level which you are teaching at (you can select multiple levels).


Clean Water Act nmlkj nmlkj nmlkj nmlkj

Colorado Drinking Water Regulations nmlkj nmlkj nmlkj nmlkj

Colorado Operators Certification Regulation nmlkj nmlkj nmlkj nmlkj

Compliance requirements nmlkj nmlkj nmlkj nmlkj

DMR (Discharge Monitoring Report) nmlkj nmlkj nmlkj nmlkj

Regulatory authority reports nmlkj nmlkj nmlkj nmlkj

Resource Conservation and Recovery Act nmlkj nmlkj nmlkj nmlkj

Safe Drinking Water Act nmlkj nmlkj nmlkj nmlkj

Sample site plan nmlkj nmlkj nmlkj nmlkj

Sanitary surveys nmlkj nmlkj nmlkj nmlkj

Spill response nmlkj nmlkj nmlkj nmlkj

Watershed regulations nmlkj nmlkj nmlkj nmlkj

Administer emergency preparedness program nmlkj nmlkj nmlkj nmlkj

Administer safety program nmlkj nmlkj nmlkj nmlkj

Develop and administer budget nmlkj nmlkj nmlkj nmlkj

Develop O&M plan nmlkj nmlkj nmlkj nmlkj

Operate atmosphere testing equipment nmlkj nmlkj nmlkj nmlkj

Operate safety equipment nmlkj nmlkj nmlkj nmlkj

Plan and organize work activities nmlkj nmlkj nmlkj nmlkj

Respond to complaints nmlkj nmlkj nmlkj nmlkj

Train staff nmlkj nmlkj nmlkj nmlkj

Write plans (O & M plans) nmlkj nmlkj nmlkj nmlkj

Chemical handling nmlkj nmlkj nmlkj nmlkj

Confined space entry nmlkj nmlkj nmlkj nmlkj

Customer relations nmlkj nmlkj nmlkj nmlkj

Electrical hazards nmlkj nmlkj nmlkj nmlkj

Excavating, trenching and shoring nmlkj nmlkj nmlkj nmlkj

Facility inspection nmlkj nmlkj nmlkj nmlkj

Facility security nmlkj nmlkj nmlkj nmlkj

Fire safety nmlkj nmlkj nmlkj nmlkj

Hazardous environment nmlkj nmlkj nmlkj nmlkj

Lockout/tagout nmlkj nmlkj nmlkj nmlkj

Natural and manmade disasters nmlkj nmlkj nmlkj nmlkj

OSHA compliance nmlkj nmlkj nmlkj nmlkj35


Personal protective equipment (PPE) nmlkj nmlkj nmlkj nmlkj

Process Safety Management nmlkj nmlkj nmlkj nmlkj

Quality Assurance and Quality Control nmlkj nmlkj nmlkj nmlkj

Risk management program nmlkj nmlkj nmlkj nmlkj

Spill response nmlkj nmlkj nmlkj nmlkj

Traffic/work zone nmlkj nmlkj nmlkj nmlkj

Vulnerability assessment nmlkj nmlkj nmlkj nmlkj

Records nmlkj nmlkj nmlkj nmlkj

36

APPENDIX D

UNDERSTANDING ABC CERTIFICATION EXAMS AND LEVELS OF LEARNING

It is important for trainers to understand both the nature of the ABC exams and the level of learning concept to provide operators with the most valuable and meaningful training offerings. An overview of certifications exams, identifying the level of learning required for job tasks, and how to use this information in training development is included in this appendix.

ABOUT CERTIFICATION EXAMS

The ABC water treatment certification exams evaluate an operator’s knowledge of tasks related to the operation of water treatment systems. The ABC Water Treatment Validation and Examination Committee determines the content of each exam based on the results of a national task analysis survey. To pass an ABC exam, an operator must demonstrate knowledge of the core competencies.

Four levels of certification exams are offered by ABC, with Class I or D being the lowest level and Class IV or A the highest level. The specifications for the exams are based on a weighting of the job analysis results to ensure they reflect the criticality of tasks performed on the job. The essential tasks and capabilities that were identified through this process are called the core competencies. Core competencies for the water treatment operator, water distribution operator, and small water system operator are shown in the chart to the right. The level of learning depends on the level of certification. For example, training for water treatment operators Level A should be taught at the Evaluation and Synthesis levels of learning.

LEVELS OF LEARNING

Levels of learning that are not addressed in the Framework are Teaching Knowledge and Teaching Evaluation.

Teaching Comprehension

Comprehension includes understanding the meaning, translation, interpolation, and interpretation of instructions and problems as well as stating a problem in one’s own words.

37

Key Words: comprehends, converts, defends, distinguishes, estimates, explains, extends, generalizes, gives examples, infers, interprets, paraphrases, predicts, rewrites, summarizes, and translates.

Example Activities to Demonstrate Comprehension:

In your own words, write down the steps for chlorine disinfection.

Report to the class what that paragraph about valve operation means.

Illustrate and label the components of a piston-type meter.

Write a brief outline to explain backflow to someone else.

Write a summary report of the importance of system surveillance.

Prepare a flow chart to illustrate the sequence of events in the water cycle.

Rewrite the principles of electrical thawing of frozen water services.

Key Questions to Demonstrate Comprehension:

Why should operators wear safety toe boots?

What is the purpose of the rate valve in a chlorination system?

How would you explain the purpose of thrust blocks?

What are the main points of managing of a water treatment facility?

What was the main idea of that section about fire hydrants?

What are the benefits of using steel pipe?

Illustrate a typical “tree” water distribution system.

Teaching Application

Application is using a concept in a new situation or unprompted use of an abstraction. The operator applies what was learned in the classroom to new situations in the work place.

Key Words: applies, changes, computes, constructs, demonstrates, discovers, manipulates, modifies, operates, predicts, prepares, produces, relates, shows, solves, uses.

Example Activities to Demonstrate Application:

Calculate the slope of the energy grade line.

Construct a model to demonstrate a typical hypo-chlorinator installation.

Make a diorama to illustrate the proper lane closure on a two-lane road using flaggers.

Create a handbook covering the essentials of operator safety.

Make up a game that rewards meter construction knowledge and skills.

Apply the formula for pump efficiency to calculate the efficiency of a given system.

Given the following components, construct a water meter.

Key Questions to Demonstrate Application:

38

What is the cross-sectional area of a pipe that is 10 in. in diameter?

Why is excessive water pressure to residential homes objectionable?

Calculate the pounds per square inch of pressure at the bottom of a tank if the water level is 33.11 feet deep.

What is the best location for a tap on a main?

How would you convert 80 degrees Fahrenheit to Celsius?

What type of personal protective equipment should an operator wear when handling hypochlorite?

Teaching Analysis

The operator separates material or concepts into component parts so that its organizational structure may be understood. The operator distinguishes between facts and inferences.

Key Words: analyzes, breaks down, compares, contrasts, diagrams, deconstructs, differentiates, discriminates, distinguishes, identifies, illustrates, infers, outlines, relates, selects, separates.

Example Activities to Demonstrate Analysis:

Troubleshoot a broken water meter by using logical deduction.

Recognize the mistakes in this diagram of dry tapping.

Design a questionnaire to gather information about the emergency preparedness of a population.

Classify the common contaminants of a water system and list the recommended treatments.

Construct a graph to illustrate the advantages and disadvantages of available disinfection methods.

Conduct an investigation to determine the best type of pipe for a given water system.

Use a Venn diagram to show how dry tapping and wet tapping are the same and different.

Read this emergency response plan and determine what is missing.

Key Questions to Demonstrate Analysis:

What should a supervisor do if an employee is performing work in an unsafe manner?

How would you take corrective action to fix a large water distribution system with inadequate pressure?

How are ball valves similar to diaphragm valves? How are they different?

What would the outcome be if a turbine water meter is assembled without a freeze plug? What would be the consequences of that outcome?

What are some of the problems with uncovered reservoirs?

How can you tell the difference between pipe cleaning swabs and pigs? List the strengths and weaknesses of each.

39

Teaching Synthesis

The operator is able to assimilate the information learned in the classroom into his/her work environment.

Key Words: propose, rearrange, reconstruct, reinforce, reorganize, revise, structure, substitute, validate.

Example Activities to Demonstrate Synthesis:

Recognize the mistakes in your utility’s diagram of dry tapping, and prepare a proposal to address the issues.

Redesign your utility’s emergency response plan.

Classify the common contaminants of your utility’s water system, and list the recommended treatments.

40

APPENDIX E

SCORECARD EVALUATION CRITERIA

Criteria Meets or Does Not Meet Recommendations Training Design and Content The training provider has clear, specific, and measurable written statements of intended learning outcomes for each learning event that are based on identified needs.

All trainers provided learning objectives; however, the majority focused on what is being taught, rather than what students would learn (i.e., Bloom’s action verbs, measurable outcomes).

More information/training should be provided to trainers about how to develop and utilize learning objectives in designing and evaluating training.

Each learning event is planned in response to the identified needs of the intended learners.

The majority of respondents stated that a learning needs analysis was conducted. All agreed that content is updated periodically.

More detail is needed on what constitutes a learning needs analysis, and how this is conducted and used. More detail is needed on the frequency and criteria for how content is updated.

Identified needs form the basis for planned learning outcomes.

This was not asked on the scorecard. A question should be added to the scorecard regarding the relationship of the needs analysis and learning outcomes.

The training provider presents complete, accurate, and timely information about the learning experience available to learners in advance of the learning event.

The majority of trainers would not meet these criteria.

Guidance is needed to determine what information about the learning experience should be provided in advance of a training program.

Instructional methods accommodate various learning styles and are designed to promote interaction between and among learners, instructors, and learning resources to achieve the stated learning outcomes.

Instructors believe they accommodate different learning styles. However, the majority of training is conducted in a large group, with limited engagement. Only two school instructors use small-group activities.

More information/training is needed on engagement strategies. Recommended percentages of large vs. small group and presentations vs. activities should be developed.

Instructor Qualifications The training provider shall document that individuals involved in the design,

The majority of trainers believe they meet these criteria, both on technical content

Specific detail is needed about qualifications. In particular,

41

development, delivery, and evaluation of learning events are qualified in their assigned roles.

and instructional design. When asked for what experience is necessary, technical content was identified but not instructional methods.

What technical experience is needed? Do you need to be an operator? What level of education/certification is

needed? What training/certification/experience

is needed for training design and delivery?

Individuals involved in the development and delivery of learning events remain current in subject matter material and learning methods.

All trainers believe they are current on the subject matter. Only half indicated that they obtain and document professional development.

Expectations need to be developed for what professional development is needed for trainers and if documentation is needed and how it is provided.

Policies are in place for the development and delivery of learning events to demonstrate high standards of professional conduct.

In general, policies and procedures seem to be lacking.

CDPHE and/or the Boulder School should develop these policies for the trainers to implement.

Evaluation Procedures The training provider measures what participants thought and believed about training programs.

All trainers met this requirement. No changes are needed.

The training provider evaluates student learning against the learning objectives.

Two-thirds state that this is done, with the most common method being pre- and post-testing. However, based on the comments earlier about learning objectives, there is room for improvement.

Training guidance on measurement against learning objectives is needed.

The training provider evaluates the transfer of student learning to application on the job.

This criterion was not met. One method that could be evaluated is the pass rate of school students.

42

Documents

COLORADO TRAINING STRATEGY …c.ymcdn.com/sites/ LEVELS OF LEARNING ... Bloom’s taxonomy action verbs are used in writing ... Asking students to raise their hands in response to