Audit of

District’s Information Technology Disaster Recovery Plan

April 11, 2014

Report #2014-03

E. Wayne Gent

Superintendent of Schools

School Board Members Audit Committee Members Chuck Shaw, Chair Noah Silver, CPA, Chair Frank A. Barbieri, Jr., Esq., Vice Chair David H. Talley, Vice Chair Marcia Andrews N. Ronald Bennett, CPA Karen M. Brill Michael Dixon, CPA Jennifer Prior Brown, Esq. Richard Roberts, CPA Michael Murgio Bill Thrasher, CGFO Debra L. Robinson, M.D. (Vacant) Representatives Frank A. Barbieri, Jr., Esq., School Board Member E. Wayne Gent, Superintendent of Schools JulieAnn Rico, Esq., General Counsel Stephanie Nance, Principal Representative Debra Wilhelm, CTA President

MISSION STATEMENT

The School Board of Palm Beach County is committed to providing a world class education with excellence and equity to empower each student to reach his

or her highest potential with the most effective staff to foster the knowledge, skills, and ethics required for responsible citizenship and productive careers.

Audit of

District’s Information Technology

Disaster Recovery Plan

Table of Contents

Page PURPOSE AND AUTHORITY 1 SCOPE AND METHODOLOGY 1 INFORMATION EXEMPT FROM PUBLIC DISCLOSURE 2 BACKGROUND 2 CONCLUSIONS 1. Business Impact Plan (BIA) Not Performed 4

2. Disaster Recovery Plans Not Fully Tested 5

3. Temperature and Humidity Requirements at Off-Site Tape Storage Facility 7

Did Not Meet Specifications

4. Back-up Tapes May Not Arrive at Designated Off-Site Location 8

5. Technology Disaster Recovery Plan Needs Improvement 9

6. Procedures for Off-Site Tapes to Designated Recovery Sites Needs Enhancement 10

7. Security Enhancements Needed for Off-Site Backup Tapes 11 APPENDIX Management’s Response 12

This page intentionally left blank.

M E M O R A N D U M TO: Honorable Chair and Members of the School Board E. Wayne Gent, Superintendent of Schools Chair and Members of Audit Committee FROM: Lung Chiu, CPA, Inspector General DATE: April 11, 2014 SUBJECT: Audit of District’s Information Technology Disaster Recovery Plan PURPOSE AND AUTHORITY Pursuant to the District’s Audit Plan of 2012-2013, we have audited the District’s Information Technology Disaster Recovery Plan. The primary objective of the audit was to assess the adequacy of the District’s Information Technology Disaster Recovery Plan for preserving the integrity of data backups and minimizing disruption to the District’s operations should disasters occur. SCOPE AND METHODOLOGY The audit was conducted in accordance with Generally Accepted Government Auditing Standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This audit was based on information and records obtained from four departments: (1) IT – Infrastructure, System Support & Security, (2) IT – Enterprise Applications, (3) IT – Technical Operations, and (4) Purchasing. Computer processed data was not used as part of this audit; therefore, we did not assess the reliability of this data. The scope and methodology of this audit included the following areas:

• Development of the Information Technology Disaster Recovery Plan • Plan testing and maintenance of both plans

THE SCHOOL DISTRICT OF LUNG CHIU, CIG, CPA SCHOOL BOARD PALM BEACH COUNTY, FLORIDA INSPECTOR GENERAL CHUCK SHAW, CHAIRMAN FRANK A. BARBIERI, JR, ESQ., VICE CHAIRMAN OFFICE OF INSPECTOR GENERAL MARCIA ANDREWS 3318 FOREST HILL BLVD., C-306 KAREN M. BRILL WEST PALM BEACH, FL 33406 JENNIFER PRIOR BROWN, ESQ. MICHAEL MURGIO (561) 434-7335 FAX: (561) 434-8652 DEBRA L. ROBINSON, M.D. www.palmbeachschools.org E. WAYNE GENT, SUPERINTENDENT

1

• Effectiveness of tape backup procedures, testing, and storage facility review • Contracts for hot site recovery and tape storage • Test results and action plans from hot sites

This review also included the review of the following:

• School Board Policies and District’s procedures • School Board IT User Standards and Guidelines Manual • Florida Statute 252.365 Section 1. Subsection (3b) and Florida Statute 119.021 • Chapter 1B-26.003, Florida Administrative Code Records Management – Standards

and Requirements Electronic Recordkeeping (11) (b) • Florida Agency for Enterprise Information Technology AEIT Rule 71A-1.012, Florida

Administrative Code

Audit conclusions were brought to the attention of staff during the audit so that necessary corrective actions could be implemented immediately. The draft report was sent to the departments for review and comments. The management response is included in the Appendix. We would like to thank staff for their cooperation and courtesy extended to us during the audit. The final draft report was presented to the Audit Committee at its April 11, 2014, meeting. INFORMATION EXEMPT FROM PUBLIC DISCLOSURE Pursuant to Florida Statute 281.301, certain security systems information is exempt from public access and disclosure. Moreover, in accordance with Government Auditing Standards, Section 7.41,

“…information related to computer security for a particular program may be excluded from publicly available reports because of the potential damage that could be caused by the misuse of this information…”

As such, this confidential and sensitive information has been excluded or redacted from this report. This information and the related audit findings have been provided to the Chief Operating Officer and Division of Information Technology for review and appropriate corrective actions. BACKGROUND District’s Continuity of Operations Plan (COOP). This is a District-wide planning process with the objective of maintaining continuity of the business processes across the District during/after a disaster. Damages to the District could range from data loss due to corrupted data to loss of computer operations and adverse impacts on other District operations from a hurricane, tornado, and fire, etc.

2

Business Impact Analysis. As part of the COOP, Service Level Agreements between the business processes and Information Technology are needed to determine which systems should be recovered first within specific timeframes. This requires the completion of a formal Business Impact Analysis which includes an inventory of all computer systems, a cost/benefit risk assessment to identify and include all the critical systems in the backup and disaster recovery arrangement. The Business Impact Analysis also assesses the risk for certain disasters to occur along with the cost to be incurred for the loss of each critical District process and computer system in the event of these disasters. The cost of the loss should always outweigh the cost of restoring the business process as the District does not want to spend more money on a disaster recovery solution than the financial loss or other consequences that would be suffered from a disaster. Critical applications to restore at a recovery facility should be identified in order to minimize the disruption of business operations should disaster occur. Technology Disaster Recovery Plan (DRP). DRP is a component of COOP and can only be successful with full engagement from the departments and schools for input and plan testing. DRP focuses on the continuity of the technology side of the District. Disaster recovery plans should be tested periodically and modifications be made to correct any problems. Overall, the District has established and implemented some parts of DRP. As of January 28, 2014, the District has the following agreements with three vendors for Disaster Recovery (DR) services:

• Vendor 1: This vendor (in an out-of-state location) provides DR facility for the District’s mainframe computer systems, such as the Student TERMS System. The annual cost of this contract during 2013 was about $43,708.

• Vendor 2: This vendor (in Florida) provides DR facility for the District’s enterprise systems such as PeopleSoft (Financial and HR/Payroll) and Educational Data Warehouse (EDW). The annual cost of this contract was approximately $64,306 during 2013.

• Vendor 3: This vendor (in Florida) provides off-site storage of backup tapes for the District’s computer systems. Total payment to this vendor during 2013 was approximately $57,044.

The District continuously replicates and transmits data for the enterprise systems, such as PeopleSoft, to Vendor 2. Backup tapes of the enterprise systems are also sent to the off-site storage facility with Vendor 3 in case the data transmitted to Vendor 2 is not available for any unforeseen reason.

Mainframe data for student information on the TERMS system is also backed up daily on tape and sent to the off-site storage facility managed by Vendor 3.

In case of a disaster, the off-site storage facility with Vendor 3 is responsible for sending the mainframe and enterprise systems backup tapes to the two DR sites with Vendor 1 and Vendor 2 respectively.

3

CONCLUSIONS The audit produced the following major conclusions. 1. Business Impact Analysis (BIA) Not Performed

Continuity of Operations Plan (COOP) is a District-wide planning process. The objective of the plan is to maintain continuity of the business processes in case of a disaster. The Technology Disaster Recovery Plan (DRP) is a subset of the COOP and focuses on the continuity of the technology side of the District in order to support the District’s schools and departments during a disaster. COOP is a sound business practice which should help to assure the District’s survival in the event of a disaster. A Business Continuity Plan (referred to as the Continuity of Operations Plan (COOP) at the School District) is required by Florida Statutes 252.365(3)(b) for Palm Beach County and all state agencies. Specifically, the statute states:

“The plan must include, at a minimum, the following elements: identification of essential functions, programs, and personnel; procedures to implement the plan and personnel notification and accountability; delegations of authority and lines of succession; identification of alternative facilities and related infrastructure, including those for communications; identification and protection of vital records and databases; and schedules and procedures for periodic tests, training, and exercises.”

The District is part of the local government and submitted a copy of its COOP version to Palm Beach County Office of Emergency Management in 2010. A second COOP draft version was started in October 2011 but had not been completed. Both the 2010 version and the 2011 draft require more involvement, feedback, testing and written approval from departments, schools, and senior management. Although a COOP requires the District to conduct a Business Impact Analysis (BIA), we found no evidence of such analysis by the District. The BIA should include an inventory of all computer systems in the School District and a cost/benefit risk assessment to identify and include all the critical systems in the backup and disaster recovery arrangement. The analysis should also include critical business processes, their associated downtime, and the risks to be incurred in the event of a disaster. These risks should justify the required availability of the processes and the related IT systems that are needed to support the processes. General services needed for schools/departments in an emergency were found in the Continuity of Operations Plan; however, the Business Impact Analysis was not performed. Also, the Internal Service Level Agreements with detailed requirements about system name, and system availability were not completed and implemented between the schools/ departments and Information Technology. It did appear that Information Technology attempted to obtain this information, but no input was received from the business side. Consequently, there is no assurance that the Information Technology is aware of the availability requirements of the business operations.

4

In the absence of a Business Impact Analysis, which should include an inventory of all computer systems and a risk assessment, there are increased risks that some critical applications will not be appropriately defined and included in the Disaster Recovery Plan. Consequently, the District may not be fully prepared for disasters because not all the applications are included in the Technology Disaster Recovery Plan. As indicated in the 2014-2016 District Technology Plan, work is still needed on further completion of the Continuity of Operations Plan. Recommendation A formal Business Impact Analysis should be conducted by the business/applications sides to inventory all computer systems, confirm the identification of critical processes and applications, and to further confirm that the identified Recovery Time Objectives (Tiers I, II, and III) remain appropriate and relevant as noted in the draft COOP. Management’s Response: The District's COOP plan was first approved and submitted to Palm Beach County Office of Emergency Management in 2010. The District initiated a review/revision of the COOP plan in 2013 with a final version being submitted to Palm Beach County office of Emergency Management in December 2013. The final COOP plan was approved by senior management as well as appropriate department leaders. The District developed an Essential Systems document that was reviewed and approved by management in 2011. IT will work with Operational and Academic divisions to update/revise the Essential Systems Documents and develop a full functioning Business Impact Analysis (BIA) which will include critical business processes, financial, operational and instructional risks to be incurred in the event of a disaster. (Please see page 13.)

2. Disaster Recovery Plans Not Fully Tested Disaster Recovery Plan testing allows users to test procedures and detect errors or gaps. Agency for Enterprise Information Technology Florida Administrative Code, AEIT Rule 71A-1.012(5) requires annual testing of the technology disaster recovery plans. Specifically,

“Information Technology Disaster Recovery Plans shall be tested at least annually; results of the annual exercise shall document those plan procedures that were successful and modifications required to correct the plan.”

Regular testing of the Technology Disaster Recovery Plan will ensure that:

1. The plans are updated. Both technical and functional tests need to be performed which requires resources for preparation time, reporting test results, and implementing an action plan.

2. Problems encountered are discussed.

5

3. Critical systems can be recovered and addressed. However, the District’s Disaster Recovery Plan does not require periodic testing. The District should ensure the system and data are restored within time frames the district has defined in the Business Impact Analysis. The functional team should ensure that the processes on the test plans for TERMS student information and PeopleSoft Financial and HR/Payroll, etc. are successful. We noticed that the functional team has not tested the PeopleSoft Financials and HR/Payroll business processes since July 2010, over three years ago. Also, functional testing for mainframe TERMS student information was last tested in July 2011 at the DR facility with Vendor 1, over two years ago. Moreover, lessons learned and action plans from testing sessions were not consistently documented and made available for management use. Consequently, potential problems detected during testing will not be addressed in the plan. When a disaster recovery plan is not tested regularly, there is an increased risk that restoration of technology operations could be delayed in the event of a disaster. Recommendation The District should:

• Incorporate an annual testing of the enterprise systems (PeopleSoft) and Student TERMS System at the disaster recovery facilities, as part of the District’s Disaster Recovery Plan (DRP) to identify and address any weaknesses. Restoration of backup tapes should also be part of this annual testing.

• Document and learn from DRP testing and address issues accordingly.

Management’s Response: A DR Functional and Technical test was performed at Vendor 1 site in December 2013 for the TERMS system. All associated documentation and restoration of backup tapes were completed. IT has scheduled the annual TERMS DR functional and technical test at the District's Disaster Recovery facility (Vendor 1) for July 2014. Restoration of backup tapes are included in the DR process. This process will continue annually. PeopleSoft functional DR testing: The PeopleSoft Team along with the business users will perform annual DR functional and technical tests through virtual connections to Vendor 2 on key business functions. (Please see page 13.)

6

3. Temperature and Humidity Requirements at Off-Site Tape Storage Facility Did Not Meet Specifications We visited the off-site storage facility with Vendor 3 on June 20, 2013. During the visit, we observed that the temperature of the media vault was 72.3 degrees Fahrenheit and the humidity was 42%. There was no automatic temperature monitoring or redundant power system to run the air conditioners for the vault. Consequently, there was no assurance that the air conditioning was functioning properly. The School District’s Request For Proposal (RFP) No. 09C-004L for the off-site tape storage facility states:

“In compliance with Florida Statute 119.021... these storage areas shall also be temperature and humidity controlled at all times and shall be physically separated from the paper records storage areas. As specified in Chapter 1B-26.003, Florida Administrative Code.., the temperature for such storage areas shall be maintained below 68 degrees Fahrenheit and the relative humidity controls shall remain between 20 and 30%. “

Also, page 82 Electronic Media and Archival Storage Environment of the RFP states,

“The storage area shall include storage racks, fire suppressant systems (non-liquid), and alarm systems. The successful proposer shall store media in a facility that meets the county commercial building codes and hurricane standards in which the facility is located.”

During our visit, we noted a log located outside the media vault which indicated that the fire suppression system was installed for the off-site tape media vault. However, the log indicated the system was not tested since January 19, 2009, which also appeared to be the installation date of the system. Inspections for this type of system should occur every six months, according to a manual for that same system. The environment and safety controls for the media room should be properly maintained to ensure the integrity of the data on the tapes, and the restoration of backup of critical District data will not be compromised. Recommendation We recommend the following issues be addressed at the current off-site tape media vault location:

• The media vault temperature and humidity be maintained properly. The vault should also be installed with the device for monitoring the temperature and humidity, and instant redundant power supply for air conditioning.

• There should be proper inspections of fire suppression systems.

7

The temperature and humidity monitoring system for the vault and an instant redundant power supply for air conditioning, fire, etc. should be a part of requirements for future Request for Proposal. Management’s Response: IT has been in communication with Vendor 3 including a surprise visit to see that related inadequacies were also corrected. Additionally, IT will work with experts in the field environmental control from District facilities management and establish a process for future inspections. Future competitive solicitation documents will include the requirements for remote monitoring of temperature and humidity and redundant power supply subject to cost considerations and fiscal responsibility. (Please see page 14.)

4. Back-up Tapes May Not Arrive at Designated Off-site Location The mainframe tapes containing TERMS student information and the District enterprise system tapes with PeopleSoft and EDW data, etc. are rotated daily to the designated off-site storage facility with Vendor 3. The business systems backup tape is an additional safeguard implemented by the Information Technology Department in case the data continuously transmitted to Vendor 2 is somehow corrupted in a disaster. We tested 13 backup tapes sent to the off-site media storage vault for a period of two days (June 17 and 18, 2013) and noted that two of the tapes (tape #102329 and tape #400475L4) for June 17, 2013, were not received by the off-site truck driver, and therefore were never sent to the off-site vault as presumed by District’s IT staff. The above enterprise system and mainframe tapes would not be available to restore critical District data if a disaster occurred at the District. There were no written procedures to ensure that all tapes scheduled to be shipped to the designated off-site storage facility with Vendor 3 arrived at the storage site. IT staff later stated that the tapes might not have been given to the driver and has since documented procedures to correct this issue. Recommendation Information Technology should develop procedures to ensure that all backup tapes scheduled for delivery arrive at the off-site storage facility for proper storage. Management’s Response: The tapes identified in the finding were not from the TERMS or the Mainframe systems. Therefore, the District was not at risk with the TERMS or the Mainframe systems as stated in the finding. The tapes identified in the audit are from the secondary (redundant) backup of the enterprise system pool of PeopleSoft and EDW database as per the data below from the tape backup

8

log. The primary backup of the enterprise system pool database is conducted by continuous electronic transmission to Vendor 2 The District IT staff monitors that transmission to ensure data integrity. However, we have taken steps with our internal and vendor processes to ensure that all tapes are accounted for and delivered in a timely fashion to Vendor 3. The data submitted earlier showed the tapes to be TSM server (non-mainframe) tapes. (Please see page 14.)

5. Technology Disaster Recovery Plan Needs Improvement The Disaster Recovery Plan (DRP) should support the business strategy outlined in the Continuity Operation Plan and contain a prioritized recovery strategy. While work has been performed on the Disaster Recovery Plan (DRP) and includes critical applications, the framework is not yet fully implemented. Specifically,

• The District has assigned only one employee the task of administering the DRP activities, among their numerous other duties.

• There is no evidence that the DRP was adopted by the School District, and therefore may not meet the needs.

• There is no DRP versioning process to ensure that the plan is kept up-to-date and indicate possible changes in procedures and responsibilities which should be communicated to all responsible parties. Without these procedures, there is no assurance all items in the plan are current. Moreover, there is no evidence of when the last review of the Disaster Recovery Plan occurred. The Technology Department should review the plan annually and make necessary adjustments.

• Technology employees utilize a SharePoint site to access the DRP, which increases the risk that the DRP will not be readily available should the SharePoint site not be accessible in a disaster.

• Internal Service Level Agreements with detailed requirements about system availability have not been completed and implemented between the Technology Department and the schools/departments. This increases the risk that the Technology Department may not be aware of the availability requirements of the business side.

Recommendation To ensure the DRP is readily available in a disaster and meets the business requirements for recovery times and priorities, the, District should:

• Know who are the employees currently assigned to the DRP program.

9

• Ensure that the DRP is reviewed and formally adopted. The same employees who sign the internal Service Levels Agreements detailing system availability requirements should sign the DRP.

• Implement procedures to certify all DRP documents are formally reviewed and approved by application users and updated at least annually.

• Ensure that the DRP is also in a format other than SharePoint and distributed to appropriate employees. Staff at the recovery facilities should have access to DRP documents.

• Ensure formal Service Level Agreements with detailed requirements about system availability be implemented between IT and schools and departments.

Management’s Response: We concur with the finding that IT has inadequate staffing levels to support the audit’s detailed recommendations. Flash Drives for the DR plan have been purchased and are being distributed to respective stakeholders. (Please see page 14.)

6. Procedures for Off-Site Tapes to Designated Recovery Sites Needs Enhancement The District has contracts for two disaster recovery facilities:

• Vendor 1: This vendor (in an out-of-state location) provides DR facility for the District’s mainframe computer systems, such as the Student TERMS System.

• Vendor 2: This vendor (in Florida) provides DR facility for the District enterprise systems such as PeopleSoft (Financial and HR/Payroll) and Educational Data Warehouse (EDW).

However, there are no written procedures from the District for the designated off-site storage facility with Vendor 3 to automatically ship the tapes to the disaster recovery facilities in the event of an emergency, without input from the District’s IT Division. For prior mainframe disaster recovery tests, District IT staff notified the off-site tape storage facility with Vendor 3 of the tape numbers and Vendor 1’s address to ship the tapes to. However, if IT staff is unable to provide the off-site tape facility with the exact tape numbers and address, no backup tapes would be shipped to the designated recovery facilities in the event of a disaster. Recommendation The District should develop and implement procedures to ensure that management at the off-site storage facility with Vendor 3 ships the appropriate tapes to the designated recovery addresses in the event of an emergency.

10

Appendix Management’s Response

12

Appendix Management’s Response

13

Appendix Management’s Response

14