PTV Group_impact_camunda_bpm_20140122

www.ptvgroup.com

USING BUSINESS PROCESS MANAGEMENT IN SPATIAL DATA PROCESSING

Camunda Community Meeting

Siegfried Klausmann, Dominik EisenbergKarlsruhe, 22.01.2014

http://www.camunda.org/community/meetings/register.html?id=17



www.ptvag.com

AGENDA

1. Motivation and goals2. Main requirements3. System Architecture 4. Demo of current solution5. Process/Service concepts6. Experiences with BPM introduction7. Future plans8. Discussion / Best practices

www.ptvgroup.com I Page 4

MOTIVATION – PLANNING AND OPTIMISATION SOFTWARE

Planning and optimising the flow of goods

Planning and optimisingthe flow of people


MOTIVATION – PRODUCTS NEED OPTIMIZED DATA

Vision Suite - PTV Visum, PTV Vissim, PTV Viswalk:Traffic planning, traffic technology and simulation

PTV Map&Guide:Transport route planning

PTV Smartour:Transport planning and optimisation

PTV Map&Market:Geomarketing, sales, marketing and sales force planning

PTV xServer:Developer components with core algorithms

Provider raw data

Validation, Harmonization

Database

Conversion to binary formats

Qu

alit

y A

ssura

nce

ste

ps


MOTIVATION – CURRENT DATA PROCESSING

GDF MapBase

Mautdaten

TBLalt

Fährenlayer

GDF2MapMap2AGF AGF

StatistikFiles

MapInfoKontrollsch.

Greenzones

PremiumNetz MapInfo AND2AGF

Meerland

Meeressaum

Höhenschicht

MautdatenAND

ANDMapInfo

GeoDB

PremiumMap

ProDB-Integration

Tiler

TBLneu

Teleporterabgleich

ATF

Map2ATF

PNDAufbereitung

PND

Log-FilesAGF

Check

Meerlandbearbeiten

PremiumNetzbearbeiten

Meeressaumbearbeiten

Greenzonesbearbeiten

MautbearbeitenManueller Bearbeitungsschritt

Mit GIS-Systemen (MapInfo)

Automatischer Bearbeitungsschritt

ASCII-Daten (intern)

„Externe“ Daten

Globaltabellenbearbeiten

Teleporternachbearbeitung

Teleporterintegration

StatistikFiles

StatistiX

eKartenAufbereitung

eKarte

1

1

Wurmloch

feature_stat.pl

MapBasePremium

PremiumOrtebearbeiten

TeleporterAnalysesch.

finaleTeleporter

Fährbearbeitung

AGF2GeoDB

MapInfoKontrollsch.

AGF2MapInfo

Overviews

Typkonvertierung

GlobalTabellen

POI-IconAbleitungen.

SteuerungsDatenGlobaldateien

TINFO

Ableitungsvorschriften

TBL neuCreator

Map2MIF

ATFID2AGFID

ATFAGF-IDZuordnung

MapAGF2AGF

MapAGF

TruckPOITMC

PremiumOrte

MIF/MID-Daten (intern)tw. mit zusätzlicher *.ini)

Manueller Bearbeitungsschrittmit UltraEdit, Scripten, SQL,...

Relevante Zwischenformate

Datenbank

MapInfoKontrollsch.

Map2MIF

*.xls-Daten (intern)

xml-Daten (intern))

Projekt-/Fremddaten

Projekt Importer

2

2Maut

ProjektMid

Projektdatenbearbeiten


Personenfährenlöschen

TALogistic

Truck2agf

DEU LKWSperren

DiverseScripte

Regionennach-berarbeitung

Hamlet FilterLSN-Korrektur

Tausch-Namen/RoutNr

ON Priorisierung

AustralienVorverarbeitung

Groß/KleinWandlungLöschen env. Zonen




MAIN GOALS OF THE NEW SOLUTION

Business perspective Reduce overall data processing times by

increased productivity Enable world wide coverage incl. new

feature layers with the same data processing team

But do not invent everything again / Limit implementation costs


MAIN REQUIREMENTS

How to achieve the goals? Easy to use GUI (Graphical User

Interface) with transparent business workflows/processes

Homogeneous behavior of services / processes (e.g. Logging, Error handling)

Robust processing (high availability, monitoring, fault tolerance)

Increase automation Parallel processing (support scaling) Re-use of existing concepts and software

(preferred Open Source), already existing business logic


SYSTEM ARCHITECTURE


SYSTEM ARCHITECTURE

Loose coupling of process control and workers

Load distribution by using multiple workers and multiple data DBs

Separating data DB from process control DB

No map data in payload of BPM engine

Main spatial data processing in DB

Web GUI(JavaServer Pages/jQuery/Spring MVC)

Process Control(Tomcat container with BPM engine)

Messaging System(Apache Active MQ JMS Message Broker)

Worker 1(Apache Karaf OSGi container with Apache

Camel based Services)





ServiceA

ServiceB

ServiceC

ServiceB

ServiceA

ServiceB

CDM (PostGIS database)



BPM engine DB (PostgreSQL)


DEMO – MAP DATA PROCESSING

http://172.23.71.11:8181/ImpactGui/

http://172.23.71.11:8181/ImpactGui/

http://172.23.71.11:8181/ImpactGui/


PROCESS/SERVICE CONCEPTS

BPM – Workflow engine

We use an existing workflow engine which supports to• Persist process state• Store durations of process executions for later analysis and optimization• Profit of existing and new features (tasks, transitions, forks, joins)• Profit of tooling for process modeling, process monitoring, persistence• thus allowing an efficient and transparent data processing

2 kinds of BPM processes for spatial data processing at PTV• Data Production Processes• Technical Processes



BPM – Kinds of processes

Data Production Processes (DPPs)• Combining the Human Workflows with automatic data preparation• The only processes, directly started by users• Configured for the data preparation of multiple countries• Can trigger several technical processes as data preparation steps• Can be recovered in case of errors



BPM – Kinds of processes

Technical Processes (TPs)• Contain only service tasks for complete automatic data preparation• Not directly started by users• Started by DPPs• Cover a single step of the automatic data preparation • Configured to for data preparation of a single country• Can not be recovered in case of error



BPM – Extensions of Service Tasks MultiProcessExecutionTask in Data Production Processes



BPM – Why extending the ServiceTask for this? ServiceTask in Technical Processes


EXPERIENCES

Re-Use: BPM engine for process management is a powerful tool, has many features which must not be implemented by us

Lightweight: Using the BPM Engine does not have too much impact on architecture or application development

Transparency: Easy API for implementing visualization and monitoring of process information, allowing a central overview of all processes (historic, current, planned)

BPM Engine


EXPERIENCES

Pitfalls for developers using the BPM Engine

OptimisticLockingException• Querying variables with RuntimeService increments revision rev_• Exception occurs when quering the same variables in another thread

• E.g. for updating the process monitoring page in GUI • Solution:

• Usage of HistoryService where ever possible• Isolating concurrent variable updates by semaphor

HistoricVariableUpdateQuery• Results of type HistoricVariableUpdate can contain the exact same Date

returning by getTime()• Problem to determine the last variable update

• Solution:• Workaround: Using the id or revision, which are strongly incremented


EXPERIENCES

Co-Work with business users

If BPM is new for the business users, give introduction / benefits

First examples with small models (created by developers) Hide generic / recurring workflows (especially technical ones) Create an easy to use and attractive GUI Train users and let them work with it Ask for feedback / improvements In common sessions with users, add more and more steps to

workflows where users see an added value (do not oversize things)

If possible: Integrate business users with dev skills into dev team

Do not give up, it takes time to experience the benefits


FUTURE PLANS

Modeling Add higher level processes representing

the overall workflow (all phases of map production beginning from provider data checks and ending with data delivery)

Transparency Visualize process statistics to be able to

find improvements/speed up data production

Statistics depend on configuration

Technical Supporting different process versions „Real“ parallel processing


QUESTIONS/DISCUSSION

[email protected]@ptvgroup.com

mailto:[email protected]

mailto:dominik.eisenberg@

mailto:[email protected]

www.ptvgroup.comwww.ptvgroup.com

Technology

PTV Group_impact_camunda_bpm_20140122