Multimedia rescue 161018

PowerPoint Presentation

Mengfan Tang,Siripen Pongpaichet, andRamesh Jain

Department of Computer ScienceUniversity of California, Irvine

ACMMM 2016, BNI session Tue 18 October 2016 @2PMMultimedia FOR Emergency Situations

Society exists only as a mental concept; in the real world there are only individuals. -- Oscar Wilde

Multimedia: Machines/Devices3

1993

2016

Technology: From Features to Machine Learning

1993

2016

Going From Advertisement and Entertainment

Going From Advertisement and Entertainment to Social Good

Shit Happens!7

Hurricanes - Typhoons

Forest Fires

Earthquakes - TsunamisDisaster Happens!Infectious diseases

Disease Epidemics -- Zika

Riots

7

Life Cycle of Disaster Management8

Focus of discussion today.

Community Computing: Merging 3 Big TrendsCommunity-based AppsReal-time CoordinationPower to Create and Share Ideas

Waze

Uber, Airbnb

Twitter, Instagram

Micro-events from M reportersSituation at Macro Level

Hurricane Matthew

ResponseReport

5-1-1 road traffic3-1-1 Red CrossNOAA SuperRes Radar US appDark Sky App

10

Fundamental Problem during a Disaster 11

Providing Resources and Information Effectively, Efficiently, and Promptly In evolving Emergency Situations.

Disaster Response12Physical World

And

Information SystemsEnvironment, Resources,InformationPersonal Situation, Needs,InformationMatchingAction Signals

Environment and ResourcesDisaster Related Evolving Situations TrafficPrecipitation MeteorologyPower outageFlood levelFoodDiseases

Data SourcesWebcams, Traffic systems (Google)Weather.com NOAAGovernment: Division of Emergency Management Social Media

Personal Situation and NeedsPersonal NeedsShelterFoodWaterFamily and FriendsInformation

Data SourcesVolunteer The-Emergency-Food-Assistance-ProgramDrinking-water, flooded-wellsCitizen reportsGovernment announcements

Shelter locationsFood assistance locations

Boil Water Notice

Timely Alerts and ResponseEvacuation ZoneEvacuation Route

Communication Methods(pull)NewsRadio

Alert Methods(push)Targeted messagesMobile apps

Disaster Report and Response App

Personal Situation RecognitionSituation Recognition and PredictionData Stream Ingestion and AggregationBuilding Emergency Situation ModelsDatabaseReal-Time Situation RecognitionOffline ProcessREPORTRESPONSE

MatchingResourcesDeterminationIdentify People NeedsEmergency Situations

Actionable Decisions

Current Explorations: Clear Potential of Emerging MultimediaEventShopStarted about 8 years agoSelect and combine diverse heterogeneous geo-spatial data streams to detect situationsDemonstrated for several applicationsInspired by PhotoshopMultimedia Micro-Reports (MMR)Started about 2 years ago.Create multimedia platform for citizen sensingDemonstrated using YFCC100M photo releaseInspired by Krumbs

Prototype: Thai Flood Emergency Response

EventShop: Thai Floods

Flood level - ShelterFlood LevelShelter

Twitter

Classify (Flood level - Shelter)

Taking personalized actions

FILTERLOC=CAAGGFUNC=AVGFILTERLOC=CAGROUPTHRESHOLD

Asthma HospitalizationGround Truth Asthma Risk Area without InterpolationAsthma Risk Area with interpolationAGGFUNC=AVGGROUPTHRESHOLDPM2.5 ConcentrationFrom StationsInterpolatedPM2.5 using SSGPPollenOzoneAQIEventShop: Asthma Risk Estimator Model and Result









FILTERLOC=CAAGGFILTERLOC=CA

Asthma Risk Area without InterpolationSparse PM 2.5 SensorsPollenOzone AQI

Asthma Risk Area with interpolationAGG Interpolated PM2.5

Physical model outputs

Satellite Images TrafficAssimilating Data Sources = Better Situation Recognition

correlation with asthma hospitalizations increased from 0.17 to 0.45

Is a Smartphone camera a camera?Smartphone collects all metadata related to the Event.Exposure TimeAperture DiameterFlashMetering ModeISO RatingsFocal LengthTimeLocationFace and emotionsActivity parametersContextual reasoningMyriad sensors through wearables

Camera captures photos.Smartphone camera captures events.

Media Data is about W5H{"micro_reports":[{ "where":{"geo_location":{ "latitude":32.90233332316081, "longitude":-117.2441166718801}, "when":{"start_time":"Jun 14, 2009 11:25:19 AM","end_time":"Jun 14, 2009 11:25:19 AM","time_zone":"America/Los_Angeles"}, "what":[{"concept_name":"people","confidence":0.9836078882217407,"visual_concept_provider":"CLARIFAI"}, {"concept_name":"food","confidence":0.8526291847229004,"visual_concept_provider":"CLARIFAI"}], "tag":#niceday #summer", "source":{"default_src":"https://.jpg"}}, "sub_event":[], "why":[]},]}

Photo, videoWhatWhereWhenWhoWhySound

How

Using Photo Concepts for Creating Worldwide Situation Map!

We can detect situations using bag of concepts. 5+ Billion phone photos = Situation Map of the WorldNew frontier for Machine learning.

Figure 5 shows the evolving concepts in Beijing in the year of 2008. As we know, Beijing Olympic Games was held in Beijing in the summer of 2008. This event is clearly reflected in Figure 5 where the number of these ten concepts basketball,court game,gymnastics,people,sport,stadium,swim and tennis reaches peak at the same time of Olympic Games.

24

Resource

ShelterMultimedia Micro-Reports

SituationNeeds

Rescue Project

Rescue Project

Rescue Project

Rescue Project

Rescue Project

Research ChallengesHumanReportingData ProcessingReal-time Recognition

Predictive AnalyticsDecision SupportWhat-If ScenarioSituation RecognitionCommunica-tion TechData IdentificationData Quality

Support fast data flowHandle heterogeneous types of data streamsEfficiently aggregate data at different granularitiesProvide storage system to archive both data input and system output.Create situation model and provide actionable informationContain predictive componentGeneric computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

28

Big 429

MultimediaParticipatoryReportingMultimedia Participatory Sensing.Real-timeSituation RecognitionSituation Recognition: Situation Models from past data Real time recognitionPredictive AnalyticsPredictive analyticsMultimediaSemanticsMultimedia Semantics

Support fast data flowHandle heterogeneous types of data streamsEfficiently aggregate data at different granularitiesProvide storage system to archive both data input and system output.Create situation model and provide actionable informationContain predictive componentGeneric computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface

29

Data StreamsMultimedia Semantics: From Different Data Streams to Abstracted Event StreamsReal World Event Streams30

sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, human motions, etcNot only heterogeneous, and unstructured, but time is a very important dimension.

They all generate EventsWhat about frameworks that consume events for knowledge discovery?

In particular, the accommodation of time into mining techniques provides a window into the temporal arrangement of events and, thus, introduce the ability to suggest cause and effect that are overlooked when the temporal component is ignored or treated as a simple numerical attribute. Moreover, temporal data mining has the ability to mine the behavioral aspects of a system as opposed to mining rules that describe their states at a point in time. Hence, temporal data mining helps understanding why rather than merely understanding what.

30

Multimedia Micro-ReportsTwitter is a good concept for opinions.

However, it is too noisy and too subjective for solving real problems.

Designing Multimedia Micro-ReportsObjective SpontaneousCompellingUniversal (Language Independent)Requirements of reportsOpportunityPhotos and VideosTextContextual dataPersonal dataSubjective and ObjectiveUniversal

Recognition ChallengeYesterday: Feature Based Recognition

Current: Machine Learning for Objects

Tomorrow: Situation Recognition

Real Time Situation RecognitionUse appropriate data sources.

Build Situation ModelsNew Frontier for Machine Learning.

Great Opportunity: Real Time Situation Recognition algorithms classify world wide situations.

Predictive Analytics: Spatial InterpolationObservations from sensors are sparse.We need spatial interpolation. Accurate interpolation usually requires combining multiple data sources.

Predictive Analytics: Cyber Space to Cybernetic SystemsPrediction of evolving situation at every point gives us better chances to control situations.Can we extend Wiener filtering and Kalman filtering into prediction methods for streaming data?

Kalman Filetering

4

1

3

1

1

1Our Partners using EventShop.

Our DreamBuild a vibrant Multimedia Rescue communityResearch: Ready Research ProblemsImplementationExperienceDeploy and LearnMultimedia is natural for humans. Lets make multimedia technology also natural in important situations.

Join us, please!