Course Evaluation Report - Eppley Instituteeppley.org/wp-content/uploads/uploads/file/62/Capstone Evaluation... · Capstone Course Evaluation Report July 2008 1 Introduction The Capstone

Course Evaluation Report Capstone Study in Facility Management

July 2008

Completed in accordance with Task Agreement J2420060049 of the National Park Service-Indiana University Cooperative Agreement CA: H2420060015

Stephen A. Wolter Executive Director

Christy McCormick Project Team

Eppley Institute for Parks & Public Lands Indiana University Research Park 500 N. Morton Street, Suite 101

Bloomington, IN 47404 812.855.4712

Timothy Harvey Asset Management Program Team Leader National Park Service

Elizabeth A. Dodson Training Manager WASO-PFMD National Park Service

http://eppic.eppley.org/team/NPS Identity/print/tif/white_bckgr/AH_sm_shaded_c.tif�

Acknowledgements Contributing Eppley Institute Staff Matthew Berry Zach Carnagey Amy Gregor Allene Lowery Christy McCormick Jeff Nix Stephen A. Wolter Contributing National Park Service Betsy Dodson Jeri Mihalic

This publication may not be duplicated without the permission of the Eppley Institute for Parks and Public Lands, acting on behalf of Indiana University. The National Park Service and federal agencies may duplicate it for training and administrative purposes, provided that appropriate written acknowledgement is given. No other state or local agency, university, contractor, or individual shall duplicate this publication without the permission of Indiana University.

Copyright 2008, the Trustees of Indiana University on behalf of the Eppley Institute for Parks and Public Lands

Table of Contents Introduction ..................................................................................................................... 1

The Facility Manager Leaders Program ...................................................................... 1

The Problem and Need ............................................................................................ 1

The Participants ....................................................................................................... 2

The Capstone Study in Facility Management .................................................................. 2

Course Specifics ......................................................................................................... 2

Process of Evaluation ..................................................................................................... 3

Methodology ................................................................................................................ 3

Evaluations of the Instructors ................................................................................... 3

Participant Course Evaluations ................................................................................ 3

Evaluations of the Instructors .......................................................................................... 4

The Audience .............................................................................................................. 4

The Course Coordinators ............................................................................................ 4

Guest Instructors ......................................................................................................... 4

Course Agenda and Corresponding Guest Speakers ................................................. 5

Quantitative Instructor Feedback ................................................................................. 7

Qualitative Instructor Feedback ................................................................................... 8

Participant Course Evaluations ....................................................................................... 9

Quantitative Course Feedback .................................................................................... 9

Qualitative Course Feedback .................................................................................... 10

Most Useful Activities ............................................................................................. 10

Least Useful Activities ............................................................................................ 10

Suggestions for Improvements .............................................................................. 10

Mentor Course Evaluations ........................................................................................... 11

Student/Mentor Communications .............................................................................. 11

Mentor Time and Interactions .................................................................................... 12

Benefits to the Mentor ............................................................................................... 15

Qualitative Mentor Feedback ..................................................................................... 16

Summary and Recommendations ................................................................................. 17

Course Content ......................................................................................................... 17

Recommendations .................................................................................................... 18

Capstone Course Evaluation Report

July 2008 1

Introduction

The Capstone Study in Facility Management presented April 14, 2008 to April 18, 2008 in Washington, D.C. was the final course in a series of a five-part, year-long course of study for the Facility Manager Leaders Program (FMLP). This four-day course focused on specific competencies that future leaders in Facility Management needed to develop. As the culminating course for the FMLP, students were asked to present their best work from the year-long course of study, experienced the legislative realities of asset management, and had the opportunity to meet with NPS chiefs and policy makers. The main focus was on the articulation and demonstration of new knowledge gained by successful completion of all prior FMLP coursework and a poster presentation with an accompanying portfolio. Additionally, students focused on life-long learning, leading change, and improving asset management in the NPS. This report will review the problem and need for the Facility Manager Leaders Program, a brief course description, the evaluation process, and the evaluation results for the Capstone Study in Facility Management

The Facility Manager Leaders Program

The Problem and Need Over the past thirty-five years, the National Park Service (NPS) facility management profession has evolved into a complex, mission-critical discipline. Being a facility manager in the 21st century requires insights, choices, and actions within a very competitive environment. Facility management in the Federal sector enjoys a much higher profile than ever before. In-depth studies conducted by the National Academy of Science’s Federal Facilities Council Report state:

“…facilities managers can no longer be regarded only as caretakers who bring unwelcome news about deteriorating facilities and the need for investments. As facilities management has evolved from tactical, building-oriented activities to a strategic, portfolio-based approach, the skills required by facilities management organizations have similarly evolved. … [This evolution] requires not only the technical skills (e.g., engineering, architecture, mechanical, electrical, contracting) found in traditional facilities engineering organizations but also business acumen and communication skills.”

Both the Department of the Interior (DOI) and the NPS have targeted the facility management profession as “mission-critical” because of the importance of maintaining the built environment and the high numbers of facility managers eligible to retire in the near future. In anticipation of the growing need for competent, able facility managers, the NPS has developed a comprehensive set of facility manager competencies to form a roadmap for development of our facility management workforce. Additionally, the Facility Manager Leaders Program (FMLP), was designed to build a core group of competent, up-and-coming facility managers to serve as leaders of facility management in the NPS in the years to come.

Capstone Study in Facility Management Evaluation Report

2 July 2008

The Participants The course audience included those charged with the facility function in the NPS. Students were geographically dispersed in parks across the country, representing various parks and experience levels. The course was designed for:

• New facility managers and facility managers new to the Service • Those in the Service who wanted to build experience in order to apply for a facility

management position and/or those seeking promotions to a facility management position

• Individuals who may be in other disciplines in the NPS but wish to pursue a career in facility management, long-term chiefs of maintenance, or facility management specialists who could benefit from facility management.

The Capstone Study in Facility Management

Course Specifics

The Capstone Study in Facility Management was a week-long course designed to be the culminating experience in the Facility Manager Leaders Program. It provided classroom and field experiences on leadership, political realities, and professionalism. Additionally, the course provided the students with the opportunity to demonstrate the learning they had amassed throughout the previous year through the Poster Presentations. The closing of the course was a goal-setting experience followed by a graduation ceremony. Specifically, the course provided the students an opportunity to learn and practice elements such as the following:

• Cultural and Natural Resource Issues as presented from the WASO perspective and their impact on the field of facility management

• Behavioral benchmarks for leading an organization or division • Implications of political realities and their impact on facility management and the NPS as

a whole • Identification of elements that make up a profession and the ability to apply them to

facility management • Creation of a leadership team toolkit • Demonstration of new knowledge gained as a result of the Facility Manager Leaders

Program


July 2008 3

Process of Evaluation

The information in this report comes directly from student feedback on the course and the instructor evaluations and team member observations of the Capstone Study in Facility Management course, held in Washington, D.C. from April 14, 2008 to April 18, 2008. A summary of the evaluations is provided in this report.

Methodology

The course was analyzed using feedback from the instructor and participant course evaluations. The details regarding how each of these components was implemented are included below. Evaluations of the Instructors The students were provided with a separate instructor evaluation form on which they could write their feedback and observations regarding each different instructor who participated throughout the course. The instructor evaluation tool allowed for a quantitative measurement of each instructor’s effectiveness, as well as space for qualitative comments about each instructor. These evaluations were collected on the final day of the course. The results from this evaluation tool are included in this report. Participant Course Evaluations At the conclusion of the course, the students were asked to complete a participant course evaluation. The evaluation tool, developed by the Eppley Institute, included both quantitative and qualitative questions regarding the course, the instructors, and improvements that could be made to the course.


4 July 2008

Evaluations of the Instructors

The Audience

This course was designed to meet the needs of those charged with execution of the facility function in the NPS. Course participants came from parks throughout the country and represented a variety of park types, sizes, and experience levels. The audience consisted of those accepted into the year-long Facility Manager Leaders Program (FMLP). The greatest number participating were those in the maintenance areas, followed by the facilities areas.

The Course Coordinators

The course coordinators provided the overall structure for the course and provided support and assistance for the guest instructors. The coordinators supplied the guest instructors with outlines and lesson plans, training aids, and instructional resources. They also provided the students with continuity and a common voice through which the key points for each session could be highlighted. A session on professionalism and lifelong learning was also presented by the course coordinators.

Guest Instructors

The guest instructors for Capstone were selected based on their position of leadership in the NPS. The servicewide chiefs for Natural and Cultural Resources spoke to the students about their particular divisions. Speakers from higher levels in the NPS were also asked to provide their perspectives on leadership and educated the students about the challenges they faced regarding political realities.


July 2008 5

Course Agenda and Corresponding Guest Speakers

The following provides the course agenda and the guest speakers for each session. Monday, April 14th, 2008 Time Session Guest Speaker(s) 8:30-9:30 am Course Introduction and Welcome Course Coordinators 10:00-11:30 Federal Highways and NPS Roads Tim Hudson 11:30-1:00 Lunch 1:00-2:30 pm Congressional and Legislative Affairs Don Hellmann 2:45-3:45 pm Concessions Management Debbie Harvey Ernest Jutte 3:45-4:15 pm One-Minute Paper and Conclusion of Day Christy McCormick Betsy Dodson Tuesday, April 15th, 2008 Time Session Guest Speaker(s) 8:30 – 9:00 am Start the Day Course

Coordinators 9:00 – 10:00 Leadership and Political Realities Sue Masica 10:00-10:15 Break 10:15 – 12:00 Leadership Case Study Costa Dillon 12:00 – 1:00 pm Lunch 1:00 – 2:00 Continue Leadership Study Costa Dillon 2:00 – 3:00 Character Discussion- Do I Really Care? Steve Wolter 3:00 – 3:15 Break 3:15 – 4:15 Leadership Development: Steve Wolter

What are my next steps? 4:15 – 4:30 One Minute Paper and Conclusion of the Day McCormick Jeri Mihalic


6 July 2008

Wednesday, April 16th, 2008

Time Session Guest Speaker(s) 8:00 – 8:15 am Start of the Day Coordinators 8:15 – 9:30 am Professionalism Workshop (Part One) Christy McCormick 9:30 – 9:45 am Break 9:45 – 10:45 am Rec Fees Program Carol Maass 10:45 – 12:00 Repair/Rehab Program Michele Proce 12:00 – 1:15 pm Lunch 1:15-2:45 pm Professionalism Workshop (Part Two) Betsy Dodson Steve Hastings Jackie Messer 3:00 – 4:30 pm Poster Presentation Practice and Critique Course Coordinators 4:30– 5:00 pm One-Minute Paper Christy McCormick Jeri Mihalic Thursday, April 17th, 2008 Time Session Guest Speaker(s) 8:00 – 8:30 am Start of the Day Coordinators 8:30-10:00 am Goal Setting and where am I going next? Students Mentors Course of Study Evaluation

10:00-11:30 am Poster Presentation - Preview & Evaluation FMLP students 11:30 – 12:30 Lunch 12:30-2:30 pm Poster Presentation - General Audience FMLP students 3:00-4:30 pm Graduation Ceremony 6:00-8:00 pm Celebration Dinner and Awards Dinner (Mr. Smith’s of Georgetown, 3104 M St, NW) Friday April 18th, 2008 Time Session Guest Speaker(s) 8:00 am Tour of Lincoln Memorial and WW II Memorial Sean Kennealy


July 2008 7

Quantitative Instructor Feedback

The course evaluation asked the participants to respond to three statements relating to the guest instructors: Statement 1: The instructor knew the subject matter well. Statement 2: The instructor made the subject matter interesting. Statement 3: The instructor encouraged student involvement. The evaluation rankings were scored on a Likert scale from 5 to 1, with 5 corresponding to Strongly Agree and 1 corresponding to Strongly Disagree. The following chart displays the average scores for each statement according to instructor. This chart reflects that the evaluations of most instructors were very high, with no response scoring below a 3. The majority of the scores were above 4, indicating the students were very satisfied with the experience level of the instructors, their ability to make the subject matter interesting, and their involvement of students in the course.

Figure 1: Instructor Quantitative Feedback

Responses to statements 1, 2, 3 reading left to right for each instructor

Overall Average

4.21


8 July 2008

Qualitative Instructor Feedback

The qualitative evaluation had only one question which asked for comments on the overall effectiveness of the instructor and suggestions for improvement. The responses were overwhelmingly positive, with a majority of the comments being notes of thanks or appreciation. There was very little constructive criticism or even suggestions for improvement. This could be because the students honestly felt there was little upon which to improve. It also could be that they read the question as just asking for overall effectiveness and skipped the part about requesting suggestions for improvement. Perhaps making these two separate questions, or adding another question, would be helpful in terms of receiving suggestions for improvement or feedback for the instructors. Almost every instructor received words of praise, thanks, and appreciation. Sample comments related to words of appreciation:

• “Very interesting and motivating.” • “Caring Instructor” • “Good information” • “Great Job” • “Thanks.”

There were also comments related to subject matter expertise, past experience, and suggestions for improvement. Sample comments related to subject matter expertise:

• “Great information, really valuable insight into congressional legislature and affairs.” • “Very valuable info, especially the WASO perspective views on RR.” • “Very Knowledgeable.” • “Great presentation, really valuable insight and info”

Sample comments related to management and park experiences:

• “Liked his personal experiences and ideas on how he empowered and involved employees.”

• “The subject and level of information was so relative to daily park issues it was really interesting.”

Comments related to suggestions for improvement:

• “A little disorganized” • “Information was not basic enough. Assumed we were all well versed in topic.” • “Level of information was above park level. Hard to relate to daily park activities.” • “I think there is a little disconnect between WASO program managers and the field…”


July 2008 9

Participant Course Evaluations

The evaluation instrument for the Capstone Study in Facility Management contained eight quantitative statements about the course. The statements included, among others, how clearly concepts were presented, the importance of the background of the speakers, and whether the facility met the needs of the participants. These statements were then followed by three qualitative questions which were designed to elicit comments from the participants about what was the most and least useful, and suggestions for improvements.

Quantitative Course Feedback

The quantitative section asked the students to rate the course according to eight statements about their learning, their leadership, the speakers, the political process of the NPS, the field experience, and the training facility. The statements were rated using a five-point Likert scale ranging from Strongly Disagree with a value of 1 to Strongly Agree with a value of 5. The mean score of 4.25 for the statements was very high. All eight statements received a score of at least 4.00, which shows that overall the students were very positive about the course. The highest rated statement showed that students were encouraged to participate and answer questions.

Figure 2: Mean Scores for the Quantitative Evaluation Statements

Overall Average

4.25


10 July 2008

Qualitative Course Feedback

The course evaluation also asked the students to respond to three open-ended questions about what was the most and least helpful aspect along with suggestions for improvements. These comments will be used to help improve future courses in the FMLP series of courses. Most Useful Activities The areas identified as most useful were leadership activities, speakers, and working with peers. What did you find most useful about the course?

• Political stuff and fund source info • WASO level people talking about their programs • The different speakers • Leadership sections, also the WASO speakers • DOI/WASO perspective speakers

Least Useful Activities The area which the students mentioned most frequently as being least useful was the trip to the Capitol. Other activities included the poster session, portfolio, and the existence of politics in the NPS. What did you think was the least useful in the course?

• Everything was helpful to some degree • The leadership stuff is a bit philosophical

Suggestions for Improvements The comments about improvements included easing up for this last week of classes and including more practical information. What would you improve about the Capstone Study as the fifth course in FMLP?

• I’d like fewer speakers and more in depth from each speaker, but I know that's more work for each speaker who already has a busy schedule.

• Let the students in the class suggest the topics to cover so anything missed during the year can be addressed.

• Outside DOI section maybe on new technologies etc. • Discussion on how we as a group of leaders need to get together to influence change.


July 2008 11

Mentor Course Evaluations

Mentor evaluations were collected for the first time for this course. Since student evaluations were collected it was deemed important to also look at the course from the mentor’s point of view to see if any additional recommendations could be found. Some questions will have 14 responses, while others only 13 due to the fact that some evaluations were only partially completed.

Student/Mentor Communications The first question asked mentors, on average, how often do you and your student interact? Thirteen of the 14 responses fell into the “once per week” or “once every few weeks” categories. The remaining response was “once every few months”. This shows that, while no one communicated with their student every day, the vast majority of mentors interacted with their students on a regular basis. The second question dealt with how mentors usually communicate with their students. As you can see in the figure below telephone and e-mail were by far the most used methods of communication, as one would expect. In person meetings and the e-portfolio site were the next most common methods of communication, but far behind telephone and e-mail.


12 July 2008

Figure 3: Method of Mentor and Student Communication The third question inquired: “Of the different communication methods used, which was used most frequently?” Only two choices were selected, with telephone being the overwhelming favorite with 11 responses and e-mail garnering 2 responses. Unsurprisingly, no other form of communication was chosen as the most frequently used.

Mentor Time and Interactions The next question was “how much time have you spent per month mentoring your student, on average?” 11 respondents answered that they spend 0-5 hours per week mentoring their student, while 2 responded that they spend 6-10 hours per week. There were no responses for 10-15 hours, or more than 15 hours per week. The figure below represents how mentors perceived the quality of their interaction with their students. Six mentors felt as though the interactions meet expectation both in terms of frequency and quality. Four responded that they would like more interaction, but the quality of the contact they did have was high. Finally, three mentors said that they would like more interaction, but felt as though the interaction that was there was not of a very high quality. There were no mentors that responded that they would like to have less interaction with their students, which is a very positive sign.

Figure 4: Quality of Student/Mentor Interaction


July 2008 13

The figure below illustrates how mentors and students worked to adjust goals and milestones when the students fell behind schedule. Half of the mentors responded by saying that this question was not applicable because their student did not fall behind schedule. The majority of the remaining mentors said that they worked to get the student back on track for the original schedule. One mentor actually altered their student’s goals to ensure success, and one simply fell behind and did not work on an alternative strategy with their student.

Figure 5: Student Schedule Adjustment and Goals The mentors were also asked “How they would describe the effort their student has put in thus far?” The results from this question were extremely positive. Seven of the mentors thought that their student’s effort “exceeded expectations”, and the remaining six believed that their student’s “meet expectations”. Not a single mentor thought that their student’s effort did not meet expectations.


14 July 2008

Another question that had extremely positive results was, “How responsive were the course instructors to any questions you had about the program?” Thirteen mentors stated that the instructors “responded quickly with the information I requested.” The remaining mentor said that they “responded slowly, but with the information I requested.” No one replied that the instructors did not get them the information requested. The graph below illustrates which areas were the most difficult for the mentors to deal with during the year. Fitting mentoring into their schedule was the biggest problem. Everyone is busy and has a lot they are responsible for, so finding time to mentor can be a challenge. Logging onto the FMLP e-portfolio site was second, followed by difficulties resulting from time zone differences. There were a variety of challenges the mentors faced this year, but none of them felt like there was too much paperwork involved.

Figure 6: Difficult Areas for Mentors


July 2008 15

Benefits to the Mentor Maybe the question with the most positive response was, “Do you feel being a mentor helped you grow, professionally or personally?” All thirteen mentors said “yes”, not a single one said “no” or “I’m not sure”. This shows that, at least from the mentor’s perspective, the program is worthwhile. The last question inquired whether or not the mentor would be interested in mentoring future FMLP students. As the figure below shows, a vast majority would be willing to do so, which says a lot about the success of the program.

Figure 7: Interest in Mentoring Again


16 July 2008

Qualitative Mentor Feedback

The majority of the questions, eight to be exact, had a comment area for qualitative feedback. The responses were overwhelmingly positive with a majority of the comments expressing what a great experience it was. There were very few suggestions for improvement. Below are some example comments, both positive and negative. Sample comments related to the instructors’ responsiveness:

• Every time that I called for assistance they were very helpful. • Very good in answering questions. • Great group of instructors with lots of passion for what they do.

Sample comments related to the students’ effort:

• With my student’s work load, he did an excellent job. • Very smart individual that worked real hard to balance all the things going on in her life. • My student remained very dedicated to timely execution of the assignments.

Sample comments related to the most difficult aspects of mentoring:

• E-portfolio site could be more user friendly. • Time zone differences were, in my opinion, the most difficult. • I would have preferred assignments by e-mailed specifically requested some. However I felt obligated to use the website.


July 2008 17

Summary and Recommendations

The evaluative feedback can be summarized by categorizing the information into the areas of: Course Content and Instructors. This summary appears below and forms the basis for the recommendations which follow.

Course Content

• The students appreciated, and rated very highly, hearing from speakers in high-level leadership positions. They enjoyed hearing and learning about the practical aspect of the work, the first-hand perspective and the opportunity to network with these high-ranking officials.

• Leadership activities were highly rated by most. The students believed that the exercises and information on leadership was valuable to them, and increased emphasis on leadership styles and case studies was listed as a suggestion for improvement.

Instructors

• All of the instructors in this course were rated quite high. The only time an instructor was not highly rated was because the student found the topic uninteresting or because it was perceived that it wasn’t tied directly to facility management. This illustrates again that students were looking for very practical information directly related to facility management.

• The instructors’ level of expertise was highly regarded by all of the students.

They appreciated having that type of resource available to them.

• The instructors and their styles, experience, and knowledge were the most frequently praised aspect of the course. Once again, the students in this course believed that the interaction with administrators who are in the field and on the job, who have years of experience working for the NPS, and can provide the practical perspective of facility management were the highlight of this course.


18 July 2008

Recommendations

The recommendations which follow were based on the above summaries and the conclusions drawn from the feedback provided by the students

• Because of the overwhelming support for the instructors, their background and their

positions within the NPS, it is recommended that these instructors be invited back to future courses or that in their place staff with similar qualifications, experience and positions be invited. It seems crucial that this level of quality instructor be utilized for this course. Furthermore, it seems evident that the quality of the instructors is the most important element of this course for the students.

• Because the quality of the instructors is so critical to the students in this course, use

additional instructors from upper level management positions in the NPS and the DOI.

• Continue to incorporate as much practical information from practitioners in the

field. Whether it’s leading a field trip, or a leadership discussion, or a “how-to” session, incorporate the practitioners whenever and wherever possible.

• Because the students seem to appreciate and value discussion and interaction

especially with the practitioners, and because they felt the week of class should be more relaxed, consider ways for more of the assignments to be completed beforehand so that the last week of class can focus more on discussion and interaction. It would also contribute to making the class less intense and/or more relaxed

• Because so many of the criticisms or concerns can be traced to a lack of

understanding of the intent or purpose of this week of class, every effort should be made to ensure that the course and class purposes and objectives are clearly understood by all students. At the outset of the course during the introduction, use an activity that involves and engages the students in the discussion of the objectives to allow them to absorb the meaning and importance of the objectives and of the course.

Documents

Course Evaluation Report - Eppley Instituteeppley.org/wp-content/uploads/uploads/file/62/Capstone Evaluation... · Capstone Course Evaluation Report July 2008 1 Introduction The Capstone