European Civil Protection

Data Processing

Gisli Olafsson – Consultant

Sources: UNDAC MethodologyOCHA IM Strategy

Outline• Principles of data processing

• Plan of Action (PoA) and IM

• Internal information flow within team/OSOCC – practical examples

• IM strategies

• Group exercise

The IM chain

Input Throughput Output

Collect Analyse DisseminateProcess

Why we need information

noise becomes data when it has a cognitive pattern

data becomes information when its assembled into a coherent whole which can be related to other information

information becomes knowledge when its integrated with other information in a form that is useful for making decisions and determining actions,

knowledge becomes understanding when related to other knowledge in a manner useful in anticipating, judging and acting,

understanding becomes wisdom when its informed by purpose, ethics, principals, memory and projection

Dee Hock, 1996

Well organized information

Incoming data

Incoming phone

calls

Incoming emails

Data Processing

What are some issues you can foresee?

• Information coming from multiple sources

• Information coming through multiple paths

• Information coming in multiple formats

• Some information digital, other analog

• Reliability of information• Conflicting information• Information Gaps• Getting information to everyone is

hard

Sources of Information

New Sources?

Format

• Enter data• Reformat• Clean

Collate

• Compile• Collate

Verify quality

• Cross-reference

• Request information

• Triangulate

Storage

• Track• Store• Assign

metadata

Archiving

• Back-up• Retrieval

Data Processing Framework

Format

• Structured vs. Unstructured• Electronic vs. Paper vs. Word of

Mouth• Spelling• Number format• Date format• Currency format• Database vs. Spreadsheet• Geo-tagging

Group Exercise

• You have been assigned to the PAHO Emergency Operation Center

• Your task is to go through reports coming in of a Cholera outbreak in Haiti

• Your task is to clean up the data• Think about what data you would like

to capture about each report

Cholera OutbreakSOURCE of INFORMATION (LINK) SUMMARY OF EVENThttp://twitter.com/vidipblip de #cholera en puerto principe http://twitter.com/firesideint Death confirmed near Cite Soleilalertnet.org 1500 cases in Haiti with http://alertnet.org/thenews/newsdesk/N22160215.htm

Haiti's government and its aid partners fought on Friday to contain a cholera epidemic that has killed at least 140 people in the nation's worst medical emergency since the Jan. 12 earthquake.

http://twitter.com/mediahacker 6 suspected caseshttp://twitter.com/mediahacker Report of child dying

http://twitter.com/gaetantguevaraSant hospitalier lafiteau,gen posiblite pou gen yon moun ki gen #cholera ki ap pran swen.(Center Hospital lafiteauare likely to have a person with cholera # will take care.

http://twitter.com/mediahacker Just been told of another death in Lafito, not far from the huge Canaraan camp - man who came from St. Marc. #haiti

http://twitter.com/melindayitiJust witnessed a death from #cholera in Lafito/Lafiteau #Haiti only a few minutes from camps Kanaran and Corail, not far from Cite Soleil

http://biosurveillance.typepad.com/haiti_operational_biosurv/2010/10/confirmed-cholera-in-arcahaie-ouest-haiti.html

Confirmed Cholera in Arcahaie, Ouesthttp://twitter.com/MissionMANNA cholera reported in Archai and Lagonavehttp://twitter.com/dodof2 40 people infected in Akaye

http://twitter.com/renewal4haitiIn other news... 6 cows reported dead of Anthrax in #Leogane. Confirmed by several doctors @iceaxe5 @biosurveillance #whatsgoingon

http://twitter.com/realhope4haiti 6 cases reported--First heard it on radio and then a family member from that area came to our village http://twitter.com/biosurveillance Discovery of suspect cholera in Lafito/Lafiteau places the lead edge of the epidemic in direct proximity to Cite Soliel.

cholera reported in Archai and Lagonavehttp://twitter.com/Just_my2cents

15 in Lagonave are at the clinic, very sick, all say they ate fish yesterday! #Cholera International Medical Corps California-based aid group, International Medical Corps, said they had confirmed cases in Croix-des-Bouquet.

http://twitter.com/Shayners7842 now dead in Mirebalais! 28 more than yesterday. I heard from someone who works in our hospital here just up the street from our compound!

PAHOhttp://twitter.com/tonycece 17 dead in remote village of Dauphin. Tried to go further to Grande-Saline but couldn't get therehttp://twitter.com/newhopehaiti The cholera outbreak has claimed the lives of two of Jean-Claubert's cousins.Artibonite (Gonaives)http://community.ob.org/site/PageServer water purification system installed in Babou Laporte near St Marc on Oct 23

http://twitter.com/CharlesHBaker Petite Riviere is hard Hit. Just left St Michel,it has 2 communal sections w problems; Plantana & Another w 15 dead 26 sick(6 babies)

HealthMap 2 cases of cholera in Bel-Airhttp://twitter.com/pih 10 cases reported in Gonaiveshttp://twitter.com/biosurveillance Two Suspect Cases of #Cholera Seen at HUEH / PAP General Hospital From Artibonite Valley

http://twitter.com/biosurveillance

Confirmation of Cholera at Multiple Sites in Port au Prince, We now have confirmation of multiple cholera cases inside Port-au-Prince in the Petionville area

Cholera Outbreak (cont‘d)

Dr Marie-Carmel Charles First human death of Anthrax ("Charbon" in french") is confirmed in Leoganehttp://twitter.com/renewal4haiti Hep A, ten cases this week, 1 already treated, 9 at CAMEJO as of Oct 23http://twitter.com/CharlesHBaker We are Hopital de Secours des Gonaives, One Dead from #cholera #Haiti 70 infected http://twitpic.com/308as4Samaritans Purse First water filtration system is installed. Word is out&many people have come w/buckets to fill up

http://twitter.com/lionelstpierre Report NON CONFIRMED from source of mine!! Kolerat has been detected in "les nippes" South of Haiti From a journalist working for ABC

http://twitter.com/KarlJeanJeune @KarlJeanJeune Just witnessed the 10th death today at Petite Rivièrehttp://twitter.com/sovelaviorg we have had no deaths. many on IV's. (Sove Lavi Clinic)lcyarrington@email.wm.edu Justinian Hospitallcyarrington@email.wm.edu Sacre Coeur Hospital lcyarrington@email.wm.edu Planned Health Facilityhttp://twitter.com/stevehelling Four cases of cholera detected overnight in a tent city near Delmas 33http://twitter.com/louis_press Oxfam will provide clean h2o to 25k todayhttp://twitter.com/Renewal4haiti 2 suspect cases at #MSF in #Leogane. Docs working on confirmation now.

http://twitter.com/operationbless We've distributed LifeSaver Jerrycans in Dauphin and Luben, as well as water purification unit in St. Marc. Hope this helps! http://twitter.com/operationbless Jerrycans in Lubenhttp://twitter.com/operationbless Purification unit

Formatted data

Collation

• Identify different sources of data to compile from• Agree methodology with ALL team members• Identify Priorities EARLY• Capture information about people and

organizations into 3W• Collect information into charts, matrixes on walls

etc• Ensure enough human resources to process• Establish a reporting systems / forms• Enter ONLINE if possible• Document what you are collating• Try to compile information into one repository

Consumersocial

networks

E-mail Fax

Face-to-face

File shares Paper-based

Messaging

Phone

Different sources of data

OPERATIONALIMPORTANCE

TIME / EFFORT INVOLVED TO PROCESS DATA

High

Low

Easy Medium Hard

PRIORITY #1

PRIORITY #2

PRIORITY #3

Prioritization Graph

Group Exercise

• Identify a process to collate and compile information that comes from– Meetings– Phone calls– Reports via radio– Email

• How do you deal with important information

PoE

electronic

Hard Copy Phone Person

Data Capture

on Template

PhysicalInbox

-Email-Websites

-file Transfers (eg.USB FTP)

TO DO

Data Entry

Defined CategoriesAssessment Data Contacts 3W SITREPS

Resources

Quality Assurance

• Verification• Cross-reference• Request information• Triangulate

Verification

• Make sure to record source & other metadata

• Add a ranking to assess credibility (1-High; 5 low)

• Different quality control needed on different types of data

Cross-reference

• Requires baseline data• Are we all using the same common

baseline data?• Are we seing similar data from other

sources?

Feedback loop

• If something doesn‘t seem right – can you reach back to the source?

• If you publish the data to those who report it they will usually provide you feedback

• Social media has been shown to be very self-correcting due to its openess with the data

Triangulate

• Are there more people affected than live in that area?

• Do the numbers add up?

Storage

• Where does the data reside?• Do we share data with others?• Do we track what data we are

keeping?• Do we track information about the

data?• Technology options

Storage/Tracking• A good file structure

– If using Groove, there is a standard storage format suggested within UNDAC– If not using Groove, need to agree upon a structure to use

• Track and share information– Step 1: How many excel sheets, word docs, pdfs, etc– Step 2: Excel sheets can be pasted on google docs– Step 3: Word docs can be pasted on google docs

= Work on own computer as well as internet

• Need to back up original data as well as collated data base•  • How do we track the data and file/label data so that the right people get it?

– Protocol format for files received, priority, flagging for clusters/units– Creation of virtual mailboxes for each cluster/sector, multiple copies of each

document to tag it for the correct audience– Mark each item with: time of receipt, source, who is it for– Who stores the data – the recipient or the sender?

Archiving

• Do we keep backups of original data and processed data?

• How can we get back to the original data?

• Do we trace decisions back to information back to reports?

Archiving• Follow folder system set up by storage team.• Compile comprehensive directory structure / catalogue / map of

archive• Per day • Per cluster• Back-up of archive – e.g. secure remote server.• Changes to archive backed-up automatically to remote server.• During versus post-mission.• During mission back-up every document• After mission – archive original sources and final products.• During a two week mission – storing, tracking and

archiving/retrieving are the sane thing for a short-duration UNDAC mission.

• Simple logic important to facilitate handover (e.g. UNDAC mission software).

Lessons learned

When Processing/Collating…

• Focus on the value added

• Be aware of data volume

• Consider remote processing

• Create (or make use of) standard products and avoid ad hoc ones

• Consider speed over perfection

• Balance resources and time management

Convince partner to share

Add value to information

Disseminate quickly

Partner sees net benefit

IM as a Value Added Service

Cholera OutbreakSOURCE of INFORMATION (LINK) SUMMARY OF EVENThttp://twitter.com/vidipblip de #cholera en puerto principe http://twitter.com/firesideint Death confirmed near Cite Soleilalertnet.org 1500 cases in Haiti with http://alertnet.org/thenews/newsdesk/N22160215.htm

Haiti's government and its aid partners fought on Friday to contain a cholera epidemic that has killed at least 140 people in the nation's worst medical emergency since the Jan. 12 earthquake.

http://twitter.com/mediahacker 6 suspected caseshttp://twitter.com/mediahacker Report of child dying

http://twitter.com/gaetantguevaraSant hospitalier lafiteau,gen posiblite pou gen yon moun ki gen #cholera ki ap pran swen.(Center Hospital lafiteauare likely to have a person with cholera # will take care.

http://twitter.com/mediahacker Just been told of another death in Lafito, not far from the huge Canaraan camp - man who came from St. Marc. #haiti

http://twitter.com/melindayitiJust witnessed a death from #cholera in Lafito/Lafiteau #Haiti only a few minutes from camps Kanaran and Corail, not far from Cite Soleil

http://biosurveillance.typepad.com/haiti_operational_biosurv/2010/10/confirmed-cholera-in-arcahaie-ouest-haiti.html

Confirmed Cholera in Arcahaie, Ouesthttp://twitter.com/MissionMANNA cholera reported in Archai and Lagonavehttp://twitter.com/dodof2 40 people infected in Akaye

http://twitter.com/renewal4haitiIn other news... 6 cows reported dead of Anthrax in #Leogane. Confirmed by several doctors @iceaxe5 @biosurveillance #whatsgoingon

http://twitter.com/realhope4haiti 6 cases reported--First heard it on radio and then a family member from that area came to our village http://twitter.com/biosurveillance Discovery of suspect cholera in Lafito/Lafiteau places the lead edge of the epidemic in direct proximity to Cite Soliel.

cholera reported in Archai and Lagonavehttp://twitter.com/Just_my2cents

15 in Lagonave are at the clinic, very sick, all say they ate fish yesterday! #Cholera International Medical Corps California-based aid group, International Medical Corps, said they had confirmed cases in Croix-des-Bouquet.

http://twitter.com/Shayners7842 now dead in Mirebalais! 28 more than yesterday. I heard from someone who works in our hospital here just up the street from our compound!

PAHOhttp://twitter.com/tonycece 17 dead in remote village of Dauphin. Tried to go further to Grande-Saline but couldn't get therehttp://twitter.com/newhopehaiti The cholera outbreak has claimed the lives of two of Jean-Claubert's cousins.Artibonite (Gonaives)http://community.ob.org/site/PageServer water purification system installed in Babou Laporte near St Marc on Oct 23

http://twitter.com/CharlesHBaker Petite Riviere is hard Hit. Just left St Michel,it has 2 communal sections w problems; Plantana & Another w 15 dead 26 sick(6 babies)

HealthMap 2 cases of cholera in Bel-Airhttp://twitter.com/pih 10 cases reported in Gonaiveshttp://twitter.com/biosurveillance Two Suspect Cases of #Cholera Seen at HUEH / PAP General Hospital From Artibonite Valley

http://twitter.com/biosurveillance

Confirmation of Cholera at Multiple Sites in Port au Prince, We now have confirmation of multiple cholera cases inside Port-au-Prince in the Petionville area

Visualized Data

Consider the volume of data

DATA VOLUME = # of Forms Collected x Number/Type of Questions x Periodicity

(Frequency of data collection)

Remote Processing

Simple things…

Mission Log

Key events flipchart

Inbox/Outbox

Assigned ownership of certain tasks

Deliverables overview (when, how, to whom)

Meeting schedule overview (where, when, who)

Scrum (interrupt everyone for 5 minutes every few hours hours to run very quickly through newest status)

Situational overview map

Experience from missions

• Information can take hours to get from one end of an OSOCC tent to the other

• Crucial information often does not get processed

• Information Management people often get very overloaded

• Technology often becomes a hurdle – especially when there is no process in place

• …What is your experience?

Some key things to remember

• Stress the importance of good information management practices to everyone– Meeting minutes for all minutes– Geospatial coding of data collected whenever possible

• Assign ownership – divide and conquer• Use the tools available in an efficient manner• But above all

Put in place an Information Management Process!!!

Remember to keep it simple, stupid!!

Source: xkcd, http://xkcd.com/386/

And finally...

© European Commission

Thank you

gisli.olafsson@nethope.orgGisli Olafsson

Skype: disasterexpert