NASA ASRS: Evolution Toward Full Electronic Processing

NASA Aviation Safety Reporting System (ASRS)and Patient Safety Reporting System (PSRS)

PM Challenge 2010

Lester GongIT Manager, Booz Allen Hamilton

Elisa TaubeResearch Manager, Booz Allen Hamilton

Linda J. ConnellDirector, NASA ASRS & PSRS

Jessica AriasPSRS Deputy PM, Booz Allen Hamilton

Used with Permission

Agenda

• Background: ASRS / PSRS & Report Processing• The Evolution Towards Full Electronic Processing• Solutions for the Evolution of Report Processing• Approach Methodology • Before & After: The Evolution of Report Processing• Current State of the Evolution• Next Steps / Future Enhancements

ASRS and PSRS

What are ASRS and PSRS?

• Two separate incident reporting systems: ASRS = Aviation Safety Reporting System PSRS = Patient Safety Reporting System

• Managed and operated at NASA Ames Research Center, Moffett Field, California

• Based on three guiding principles:VoluntaryConfidentialNon-punitive

The Aviation Safety Reporting System (ASRS)

• Established in 1976 through interagency collaboration between NASA and the Federal Aviation Administration (FAA): An independent system for aviation safety One of the first lines of defense in identifying

safety issues NASA chosen as “Honest Broker”

• ASRS database is a national asset of U.S. aviation safety data

The Aviation Safety Reporting System (ASRS)

• Receives, processes and analyzes voluntarily submitted incident reports from Pilots Air Traffic Controllers Flight Attendants Maintenance Technicians, and others

• Reports submitted to ASRS may describe both unsafe occurrences and hazardous situations

• ASRS's particular concern is the quality of human performance in the aviation system

The Patient Safety Reporting System (PSRS)

• Initiated in May 2000 via an Interagency Agreement with the Department of Veterans Affairs (VA)

• Modeled after the ASRS• A venue for frontline hospital staff to report patient

safety issues externally• Accepts reports from:

Physicians Nurses Ancillary staff (Pharmacy, Physical / Respiratory

Therapy) Other

• Collects and analyzes voluntarily submitted patient safety reports including: Close Calls Events Suggestions

• Strengthens the culture of safety• Supports the foundation of medical human

factors safety research

The Patient Safety Reporting System (PSRS)

Report Processing

What is Report Processing?

• Report Processing: the methodology for codification of incident reports to facilitate extraction from the database for learning and safety information

• Both programs follow a standard report processing model of rapid screening, multiple report matching, alert identification and codification

• Begins with the receipt of the report and ends with the final coded report entered in either the ASRS or PSRS Database

Report Processing Flow

Each report is read by two expert safety analysts within three working days

The Evolution Towards Full Electronic Processing

Why Evolve?…Paper Based for Decades

Report ProcessingUsers sent 100% of reports by U.S. Mail

All incoming reports were routed in hardcopy Analysts used a 12 page paper coding form to select relevant

information for database insertionCapitalize on data that is generally more available in

electronic form In 2002 ASRS began to receive airline ASAP reports sent

electronically that required in-house printing to enter the processing flow

Database “keyers” typed in all coded reports and narrative textPublic Accessibility to Data

Our public ASRS database was not searchable onlineAccess to data was by request only

ASRS Reporting Volume

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

5,000

'81 '82 '83 '84 '85 '86 '87 '88 '89 '90 '91 '92 '93 '94 '95 '96 '97 '98 '99 '00 '01 '02 '03 '04 '05 '06 '07

Smoothed / ForecastActual Intake

January 1981 – December 2008 ASRS receives an

average of 4,200 report/month - 193 per working day

Total Report intake for 2008 was 50,405

Why Evolve?…

• Increasing pressure for: Users to submit reports electronically Decreasing processing time / cost

Paper process has limited throughputReport Volume continues to increase

A Method that leverages electronically submitted reporting form data and reduces data input costs

Searchable databases to access relevant reports for research, rulemaking, and safety purposes

Why Evolve?…

• Identified Challenges: Need a well planned and smooth transition

Immediate cutover with no effect on high volume report production

No opportunity to cease or pause production for the evolution

Innovative solutions needed to be developedInternal staff must shift paradigm for report handling

including special cases such as alert level reports

Solutions for The Evolution of Report Processing

Solutions for Evolution

Launched development of 3 Essential Tools:

Electronic Report Submission (ERS) – A method for ASRS and PSRS Reporters to submit reports electronically

External, on website

Analyst Workbench – A browser based internal tool used by ASRS or PSRS Analysts to process reports through 25 stages

Internal for Processing

Database Online (DBOL) – Internet accessible, browser based self-serve application for users to search the databases

External, on website

Approach Methodology

Approach Methodology

• Used Standard Project Management Techniques1: Identified requirements Established clear and achievable objectives Balanced quality, scope, time, and cost concerns Accommodated concerns of key stakeholders

• In order to identify requirements and accommodate stakeholders, a Usage-Centered Approach2 was used

1 PMBOK Guide, p. 9, 102 Constantine, L., & Lockwood, L. (1999). Software for use.

Addison-Wesley-Longman, Inc., NY, NY., p. 23-25

Approach Methodology

• Usage-Centered Approach1 – focuses on the work that users are trying to accomplish and on what the software will need to supply via the user interface to help them accomplish it User centered design represents a shift of focus

from technology to people, from user interfaces to users. To design dramatically more usable tools, however, it is not just users who must be understood, but usage

1 Constantine, L., & Lockwood, L. (1999). Software for use. Addison-Wesley-Longman, Inc., NY, NY., p. 23

Approach Methodology

• Utilized Basic Elements of Usage-Centered Approach 1

Pragmatic Design Guidelines Model-driven design process (task models or use

cases, content models, role models) Organized development activities Iterative improvement Measures of quality

1 Constantine, L., & Lockwood, L. (1999). Software for use. Addison-Wesley-Longman, Inc., NY, NY., p. 24-68

Before & After: The Evolution of Report Processing

Solutions for Evolution

Launched development of 3 Essential Tools:Electronic Report Submission (ERS) – A method for ASRS and PSRS Reporters to submit reports electronicallyFrom Reporting Form to Electronic Reporting

External, on website

Analyst Workbench – A browser based internal tool used by ASRS or PSRS analysts to process reports through 25 stagesFrom Coding Form to Online Data Coding

Internal for Processing

Database Online (DBOL) – Internet accessible, browser based self-serve application for users to search the databasesFrom Formal Requests to Direct Access

External, on website

Electronic Report Submission (ERS)

• General Requirements Internet accessible, web browser based to accommodate

diverse geographic locations and computer workstations Mimics paper form; Intuitive and easy to use Ability to print form online for paper submission No caching or saving on user computer to protect

confidentiality Secure Transmission of data and “on the fly” encryption Autopopulates to new Workbench Tool Ability to reconstruct original report for Analysts in

electronic form for analysis

BEFORE -Paper Reporting Forms sent by U.S. Mail

AFTER -Reporting Forms Securely Submitted Online

http://asrs.arc.nasa.gov/

AFTER -Reporting Forms Securely Submitted Online

Analyst Workbench

• General Requirements Browser based & cross platform Customized interface based on role Single electronic application for completing

and tracking 25 steps in report processing Leverages staff “mental models” for analyzing

paper reports Reduce errors and keying labor hours by pre-

populating electronic data

AFTER

Before & After Report Processing

BEFORE

BEFORE -Coding Form – 12 Pages in Hardcopy

BEFORE -Javascript Data Entry GUI

AFTER -Dual Monitors, Synchronized Displays

AFTER -Analyst Workbench Main Menu

AFTER -Analyst Workbench Full Form Codification Screen

Database Online (DBOL)

• General Requirements: Internet accessible, web browser based self-serve

query application Enable stakeholder communities to construct complex

SQL queries utilizing a simple interface Export to variety of formats (Excel, CSV, Word) for

ease of data manipulation Rapid topical searches of over 150,000 records and

10 million rows of dataSimple approach (just text) or more advanced queries

are possible

BEFORE – Database Online (DBOL)

Internal tool - complex with a high requirement for data structure knowledge

AFTER – Database Online (DBOL)

Public tool – Intuitive, easy to use. Serves casual user and domain expert.

Current State of the Evolution

• On Day 1, October 16, 2006: First electronic report from ASRS website came within minutes

of launch

ASRS received 53 ERS reports in the first week

• ASRS received 50,405 total reports in CY 2008. Breakdown in table below*:

Totals Total %

Electronic 37,076 73.6%

Paper 13,329 26.4%

Totals 50,405 100.0%

Current state of Electronic Report Submission (ERS)

*Data thru October 2009

Current State of Database Online (DBOL)

• Database Online Metrics In 30 years, 7,100 Search Requests (SRs) have been directly

provided by ASRS Research Staff to various aviation organizations and agencies

Since July 2006, 58,857 data queries have been accomplished by external users since the launch of ASRS Database Online (DBOL)1

1Thru October 2009

May 5, 2009 – smooth transition from paper to electronic processing was accomplished

Report analysis staff have been active in quickly identifying issues and proposing innovative solutions

Productivity was improved. Average time to process a report initially has shown increased efficiency

Need for data keying and printing reports eliminated Report tracking requirements have dramatically

improvedEase of tracking location of each report in process

Current State of Analyst Workbench

Next Steps / Future Enhancements

Next Steps

• Continue to improve usability of the tools• Electronic Report Submission:

Shift to html versions to avoid technical difficulties due to advancing .pdf technology

• Analyst Workbench: Enhance tool to allow for adaptability to new domains Incorporate workflow editing

• DBOL: Use of javascript libraries to simplify handling of client

side code Leverage AJAX technology to enhance user

experience