Graph Analysis & HPC Techniques for Realizing Urban OS

Graph Analysis & High-Performance Computing Techniques for Realizing Urban OS

Katsuki FujisawaHisato P. Matsuo

1

Kyushu Universityin Ｆｕｋｕｏｋａ

Katsuki Fujisawa

Hisato Peter Matsuo

Presenters

2014

Center of Innovation Project

1998

Received Ph.D.

Full Professor,Institute of Mathematics for Industry (IMI),Kyushu University

Joined IBM

Research Fellow,Center for Co-Evolutional Social Systems,Kyushu University

- Research Director of the JST CREST for Post-Peta HPC- Graph500 Winner / Green Graph500 3rd winner in 2014

- Memory system Architect for Storage subsystem- IaaS/PaaS product Consultant-> now Urban OS Designer

Joined Kyushu-U as Full Professor

Left IBM, Joined Kyushu-U

2022Now

Agenda

Urban OS that realizes next generation Smart City

Architecture and Infrastructure

Software architecture and Analytic system

Graph Analysis & HPC

Summary and our goal

Energy Supply-Demand BalanceTraffic Congestion

Urban Issues/Challenges

・ Energy issues・ Environment, water, sanitation & hygiene・ Disaster control

・ Population decline・ Low birth rate & rapid aging・ Urban concentration

・ Traffic problems・ Means of mobility・ Information access

・ Globalization・ Diversity・ Information divide

・ Finance・ Industrial promotion・ Gov. administration・ Education/Welfare・ Innovation

Urban OS provides three Mobility’sAnyone can access … anytime,

anywhere

Urban OS

Urban OSPeople/Materials mobility

on-demand and effective transportation

Energy mobilitysecured energy supply

Information mobilityappropriate information

Three Mobility’s lead sustainable society

People/MaterialMobility

InformationMobility

EnergyMobility

Efficient & optimized

Infrastructure

CreativeCommunity

Efficient & flexible Energy

Agenda






Urban OS Functions

Advanced future social services createdWith optimization/analytic engines

Event secu-rity plan

Complaintresponse

Trafficinformation

UrbanOS

Social services that utilizesThe platform data

Flexible energydemand response

Effective eva-cuation plan

Smart trafficcontrol

Trafficdata

Weather/Disaster data

Gov./Publicdata

Energydata

Persondata

Open Data

InformationFeedback

Co-evolutionalSociety

Urban O S

Cross-utilization of variousdata

Automatic optimization,control & bottleneck analysis

Open platform forsocial/public/commercial

applications

Big data / Open dataSensor Network

Application Service

Optimization/Analytic

Data Store

Data

Open platform for advanced urban services

Urban OS Architecture

Data for Person/Traffic/City plan optimization

Data Example : Public Open dataGovernment Open data in Fukuoka city

Map Mashup

Utilized ApplicationsData Catalogue

Dataset Search

• Open data– Census– Statistics– Facilities– Report– others

• Provided as:– CSV– PDF– …

then• Transform to

– RDF format– Linked data

Data Example : Sensor Poles

14 Poles in the campus

Sensor Network in Kyushu UniversityNetwork Camera

WiFi Access Point

Temp/Humid Sensor

IC card Reader

Laser Range Finder

Gateway

• Analyze– Campus people flow

• Connect to– smartphones– with ID badge

authentication traces

Data for Energy optimization

Data Example : Campus Energy MonitorHydrogen Society model case in Kyushu University

Hydrogen StationLarge scale Fuel Cell

• Research how we utilize hydrogen in our society.– using renewable energy– using vehicle energy

How data are processed in Urban OS

Agenda






Required Technology for Urban OS

Cyber Space

Urban OS Optimization Layer

Long term oriented analysis (Quarter / Year)Compute complex calculation in advance, Apply to plan / design Large computation

Mid-levelAnalysis

Layer

MicroAnalysis

Layer

Real World Real WorldModeling Real World Optimization / Simulation Feedback/Control Real World

MacroAnalysis

Layer

Mid term oriented analysis (Day / Week)Adaptive plan / design revision depending on events / condition changes

Short term oriented analysis (real-time)Compute “present” condition continuously, Respond to emergency situations Small computation

Implement individualized analysis algorithm for long/mid/short term analysis layers

Model various Real World facts, Analyze on Cyber Space, Feedback to Real World

Cyber Space

Urban OS supported Society -Traffic-

Real-time Calculation

On-Demand Calculation

Deep Calculation

MacroAnalysis

Layer

Mid-levelAnalysis

Layer

MicroAnalysis

Layer

Traffic network/facility distribution Apply to City Plan

Roads / Traffic /Pedestrian / Vehicles

Bottleneck analysisOptimization calculation

Quickest Flowcalculation

Congestion-adaptive real-time evacuation guidance

Real World Real WorldModeling Real World Optimization / Simulation Feedback/Control Real World

Long Term

Mid Term

Short Term

Adaptive traffic scheduling per events

“Present” crowd and facilities

City

Community

Vicinity


Urban OS supported Society -Energy-

Real-time Calculation

On-Demand Calculation

Deep Calculation

Energy infrafacility distribution

Apply to Smart Grid /City Energy Plan

Area energy statusfacility distribution

Hydrogen utilized area energy ecosystem

Demand Supply analysisoptimization

Flexible energy operation using mobile energy

objects for an emergency

“Present” energy status / distribution

MacroAnalysis

Layer

Mid-levelAnalysis

Layer

MicroAnalysis

Layer

Long Term

Mid Term

Short Term

City

Community

Vicinity

Cyber SpaceReal World Real WorldModeling Real World Optimization / Simulation Feedback/Control Real World



Agenda






Emerged Graph Analysis

• The extremely large-scale graphs that have recently emerged in various application fields　– US Road network : 58 million edges– Twitter fellow-ship : 1.47 billion edges– Neuronal network : 100 trillion edges

89 billion nodes & 100 trillion edgesNeuronal network @ Human Brain Project

Cyber-security

Twitter

US road network24 million nodes & 58 million edges 15 billion log entries / day

Social network

• Fast and scalable graph processing by using HPC

61.6 million nodes & 1.47 billion edges

The size of graphs

20

25

30

35

40

45

15 20 25 30 35 40 45

log 2

(m)

log2(n)

USA-road-d.NY.gr

USA-road-d.LKS.gr

USA-road-d.USA.gr

Human Brain Project

Graph500 (Toy)

Graph500 (Mini)

Graph500 (Small)

Graph500 (Medium)

Graph500 (Large)

Graph500 (Huge)

1 billion nodes

1 trillion nodes

1 billion edges

1 trillion edges

Symbolic Network

USA Road Network

Twitter (tweets/day)

No. of nodes

No. of edgesK computer: 65536nodesGraph500: 17977 GTEPS

Extremely Large-scale Graph Analysis System

‘03 ‘05 ‘07 ‘09 ‘11

Data Source

Data Source

Large Sensor• Monitoring Data• Smart Grid• Traffic

Transportation• SNS (Twitter)

Visualization

Indexing

Centrality

Clustering

ShortestPath

Connected Component

Page Rank

MathematicalOptimization

Multi-thread LibraryStreaming Processing System

Graph Processing Graph Analysis and Optimization Library

Post-petascale or Exascale Supercomputer

　　Hierarchical Graph Store

Protection againstdisasters

Traffic ・ Transportation Network

Large Scale Social NetworksSmart Grid

Our achievements : Graph500

×3.25

K computer

SGI UV2000

TSUBAME 2.5

#3

#4

#3FX10

TSUBAME-KFC

#1

#4 #4 #4

CPU only

GPU

CPU only4-way Xeon server

Our achievements : Graph500

Graph Analysis in Urban Society

A traffic infrastructure is represented as a graph Road network / Transportation network Person flow / Vehicle flow is superimposed on a network

An energy infrastructure are represented as a graph Power grid / gas pipeline / hydrogen Supply-Demand and environmental data are superimposed

on an energy network Urban graph data will be calculated.

Optimization with Graph Analysis City level : very large scaled Community : large scaled Local : realtime with contraction

Algorithm / Hardware resourceshould be appropriately selected

Technology used in Macro Analysis Layer

Technology used in Mid-level Analysis Layer

Betweenness Centrality

Highway

Bridge

• Definition

: # of (s,t)-shortest paths: # of (s,t)-shortest paths passing throw v

Osaka road network13,076 vertices and 40,528 edges

High score vertex/edge ＝ Important place

c.g.) Highway, Bridge

• BFS => one-to-all• <#vertices> times BFS => all-to-all

• BC requires the all-to-all shortest paths

• BC measures important vertices and edges without coordinates

=> 13,076 times BFS computations

Fukuoka road network

# of nodes: 314,571# of edgs 694,906

Graph

Computation time 2m 30s (180 CPU cores)

Betweenness centrality HP ProLiant m710 Server cartridge

Technology used in Micro Analysis Layer

Real-time Emergency Evacuation Planning • catastrophic disasters by massive earthquakes are increasing in the world, and

disaster management is required more than ever

Quickest Evacuationmaximizes the cumulative number of evacuees

Cum

ulat

ive

num

ber o

f eva

cuee

s (%

)

Universally Quickest Flow(UQF) Not simulation But Optimization Problem UQF simultaneously maximizes the cumulative number of evacuees at an arbitrary time.Evacuation planning can be reduced to UQF of a given dynamic network.

0% 100%

Utilization Ratio of Refuge (%)

Agenda






Where we are

Evaluation of regulatory policy for a new technology through science, technology and innovation policy perspective.

Creation of smart and multimodal mobility systems. Development of energy economics model for consumers taking

bounded rationality behavior in consideration.

Urban OS

Application

Device/Data Development of durable, efficient and high performance solid oxide /

polymer electrolyte fuel cells. Development of next generation display devices using OLED, which can

facilitate communication exchange for all people anytime, anywhere.

Development of CPS (Cyber Physical System)-based urban OS, which manages, controls, and optimizes mobility of people and materials.

Development of realistic analysis models for urban OS utilizing techniques developed by “math for industry”.

Our Goal

Urban OS as an open platform of data aggregation big data / open data / sensor data / linked data

Urban OS as an advanced optimization / analytic platform utilizing HPC based graph analysis experience

Urban OS as an application platform to delightedly support start-ups.

Thank you!

Data & Analytics

Graph Analysis & HPC Techniques for Realizing Urban OS