Prof. Dr. Rudolf Kruse Otto-von-Guericke-University Magdeburg [email protected] PD. Dr. Jörg Gebhardt Volkswagen, Wolfsburg ICS, Celle Probabilistic

Prof. Dr. Rudolf Kruse

Otto-von-Guericke-University [email protected]

PD. Dr. Jörg Gebhardt

Volkswagen, Wolfsburg ICS, Celle

Probabilistic and Possibilistic

Graphical Models

in Complex Applications

2

Research Group „Computational Intelligence“ in Magdeburg

Main Research Topic: Intelligent Data Analysis

Intelligent Data Analysis with different methods such as Neuronal Networks, Fuzzy Systems und Bayes-Methods

Development of the Data Mining Platform „InformationMiner“

Current Industrial Projects Item Planning with Markov-Networks (Volkswagen) Information Mining (BMW, Daimler Chrysler) Bayes-Methods in Finance (Several German Banks) Implementation of new Data Analysis Methods (British Telecom)

http://www.fuzzy-clustering.de/fc-book-wiley.html

3

prefer individual vehicle specifications

by customers

bestseller-oriented vehicle specifications

by car makerMarketing strategy

Complexityvery large number of

possible variantslow number of

possible variants

STRATEGY OF VW GROUP

short back

2,8L 150 kW spark

Type alpha

4 leather, Type L3

yes ......Item

body variant engine radio

door layouts

seat covering

vanity mirror

......Item family

Vehicle specification

Marketing Strategies in Automotive Industry

4

Example: „Golf“ Class of Vehicles

approximately 200 item families (variables)

from 2 to 50 items in each family

i.e. more than possible vehicle specifications

choice of valid specifications is restricted by RULE SYSTEMS

(10.000 technical rules, even more marketing-, and production-oriented)

Example (technical rules that restrict validity of item combinations)

if then

if and

then

2002

1eengine 3tontransmissi

4eengine 1 hheaterauxiliary

},,{ 543 ggggenerator

5

Problem Representation

Rules for the validityof item combinations

(specified for a vehicle classand a planning interval)

Sample of producedvehicle specifications

(representative choice, context-dependent, f.e. Golf)

... (Golf, short back, 2.8 L 150 kW spark engine, radio alpha, ...) ...

If engine = e1 and auxiliary heater = h2

then generator in {g3,g4,g5}

predicted / assigned planning data(production program, demands, installation rates,

capacity restrictions,... bills of material, ...)

?

Historical Data

Prediction Planning

System of rules

6

Handling rules: Modelling Constraints

Handling historical data: Learning from Data

Combining the different sources: Fusion of Models

Supporting planning: Belief Change

These types of problems were treated in the three

big EC Projects: DRUMS 1, DRUMS 2 and FUSION

Our recommendation was to use

Probabilistic Graphical Models, see e.g.

Result of Problem Analysis

7

Software Environment?

1. Statistic Package 2. SPSS + Information Miner

3. Specialized Bayesian Networks Software +Consultants

Decision at VW: Version 3

8











?

Historical Data

Prediction Planning

System of rules

9

Basic Ideas: A Toy Example

Example World Relation

color

shape

smallmediumsmallmediummediumlargemediummediummedium large

size

3 variables, 36 item combinations

here: 10 simple geometric objects

10

Item Combinations

Relation

color

shape

smallmediumsmallmediummediumlargemediummediummedium large

size

Geometric Interpretation

Each cube represents one tuple

large

mediumsmall

11

Projections

largemedium

small

largemedium

small

largemedium

small

large

mediumsmall

12

1a

2a

3a

1b 2b 3b 4b 5b

: 1r

: 2r

},,{ then if 5431 bbbBaA

22 then if aAbB

Rule scheme : (A,B)

},,{)( 321 aaaADom

},,,,{)( 54321 bbbbbBDom

Rules :

Rule Systems: Use Material Implication

13

Cylindrical Extensions and Their Intersection

Intersecting the cylindrical extensions of the projection to the subspace formed by color and shape and of the projection to the subspace formed by shape and size yields the original three-dimensional relation.

large

mediumsmall

large

mediumsmall

largemedium

small

14

Focussing

Let it be known (e.g. from an observation) that the given object is green.

This information considerably reduces the space of possible valuecombinations.

From the prior knowledge it follows that the given object must be- either a triangle or a square and- either medium or large

large

mediumsmall

large

mediumsmall

15

Focussing with Projections

The same result can be obtained using only the projections to the subspaces without reconstructing the original three-dimensional space:

s m l color size

extend shape project

project extend

s m l

This justifies a network representation color shape size

16

Graphical Models

Relational Graphical Model

Decomposition + Local Model

Example

colour shape size

graph

colour shape size

hypergraph

Operations: Focussing ,…

17

Transformation into Hypertree Structure

A B D

E

FG

C

HJ

A B D

C E

FH G

J

},{ | ),( YXVYXYX E

Hypergraph Undirected Graph

Interpretation as a conditional independence graph:

18

Transformation into Hypertree Structure

A B D

C E

FH G

J

Triangulation A B C B D E

B C E

C E G

C G H J E F G

Sceleton of Tree of Cliques

(hypertree structure)},{ | ),( YXVYXYX E

Loss of some information:

19

Tree Of Cliques ( VW Bora )

186 variables 174 cliques max. 9 dimensions

20











?

Historical Data

Prediction Planning

System of rules

21

Planning Problem: Prediction of Parts Demand

Variants-related bill of material

root of vehicle class specification tree

installation point

variants of parts

Installation condition: disjunction of item combinations

...

Installation rates at installation point sum up to 1

...

... ...

intermediate structuring levels

EXAMPLE : > 100.000 item combinations needed in „Golf“ class

......

22

crisp

uncertain

single-valued set-valued

relational

probabilistic random sets

possibilistic

Approximation by aggregation

One-point-coverage

Choice of the Uncertainty Calculus

23

Probabilistic Graphical Model

Probabilistic Graphical Model :

Decomposition + Local Models

A B C

Decomposition : Hypergraph on Variables

Local Models: Marginal Distributions of A,B

and B,C that „fit together“

24

Bayes NetworksDirected dependency network

Rule conditional dependency

Hypergraph representation

Rule constraint

25

Graphical Model

context-dependent rules forthe validity of

item combinations

Modify representation

Relational Graphical Model

Fusion

System of rules

fused consistent Markov network

context-dependent sampleof produced

vehicle specifications

Learning

Probabilistic Graphical Model

Historical data

CompositionDecomposition

Graphical Model

context : vehicle class, planning interval

26

Learning Graphical Models

known structure unknown structure

complete data

incomplete data(missing values,hidden variables,...)

A<a4,

<a3,

B?,

b2,

Cc1>

?>

A<a4,

<a3,

Bb3,

b2,

Cc1>

c4>

A B

C

A B

C

Statistical Parametric Estimation (closed from eq.): statistical parameter fitting, ML Estimation, Bayesian Inference, ...

Discrete Optimization overStructures (discrete search): likelihood scores, MDL Problem: search complexity heuristics

Parametric Optimization: EM, gradient descent, ...

Combined Methods: structured EM only few approachesProblems: criterion for fit? new variables? local maxima?

fuzzy values?

27

Application at the DaimlerChrysler AG

Improvement of Product Quality by Finding WeaknessesLearn decision trees or inference network for vehicle properties

and faults.Look for unusual conditional fault frequencies.Find causes for these unusual frequencies. Improve construction of vehicle.

Improvement of Error Diagnosis in GaragesLearn decision trees or inference network for vehicle properties

and faults.Record properties of new faulty vehicle.Test for the most probable faults.

28

Analysis of Daimler/Chrysler Database

Database: ~ 18.500 passenger cars> 100 attributes per car

Analysis of dependencies between special equipment and faults.

Results used as a starting point for technical experts looking for causes.

29

Analysis of Daimler/Chrysler Database

electricalroof top

air con-ditioning

type ofengine

type oftyres

slippagecontrol

faultybattery

faultycompressor

faultybrakes

Fictitious example:There are significantly more faulty batteries, if bothair conditioning and electrical roof top are builtinto the car.

Fictitious example:There are significantly more faulty batteries, if bothair conditioning and electrical roof top are builtinto the car.

30

Example Subnet

Influence of special equipment on battery faults:

(fictitious) frequency of air conditioningbattery faults with withoutelectrical sliding roof with 8% 3%

without 3% 2%

significant deviation from independent distribution hints to possible causes and improvements here: larger battery may be required, if an air conditioning

system and an electrical sliding roof are built in

(The dependencies and frequencies of this example are fictitious)

31

Problems in Structure Learning of PGM

Complexity oflearning problem

Exhaustive graph searchin „poor“ classes

Greedy search (heuristics)in „richer“ classes

Dependency analysis(CI-Tests)

probability maximization(Bayesian-Dirichlet)

Unsufficient quality of results,need for controllable

search strategies

Handling „soft“ dependencies

Integrability ofstructure knowledge

32

Information Fusion


item combinations


Transformation into a relational network

with hypertree structure

Fusion

System of rules



Quantitative Learning

PGM (Markov network) having the structure of the relational network

Historical data

Use cond. independencies (Composition)

Estimate prior distribution of installation rates


33

Planning Models

Typical complexity:

• 200 item families

• 150 cliques

• 5 to 7 dimensions (typical)

• max. dimensions: 11 to 14

• 100 vehicle model groups

• 20 to 40 planning intervals

(i.e. 2000 to 4000 networks)

34

Planning Operation : Conditioning ( Focussing)

Input Data : item combination (set of variable instantiations)

Operation : Calculate the conditioned network distribution and

the probability of the given item combination

(propagation).

Application : Calculation of part demands

Compute the installation rate of item combination

.

Simulation

Analyze customers‘ preferences with respect to those

persons who buy a navigation system in a VW Polo.

),,( 543 gtm

35

Knowledge Propagation in Trees of Cliques

A B C B D E

B C E

C E G

C G H J E F G

2. Collect information

A B C B D E

B C E

C E G

C G H J E F G

3. Distribute information

1. Local computations w.r.t. cliques

Lauritzen, Spiegelhalter, 1988 Shafer, Shenoy, 1988

Local Operation: Conditioning

36

Planning Model based on Belief Change


item combinations




Fusion

System of rules

fused consistent Markov network for

item planning





Historical data

Use cond. independencies (Composition)

Estimate prior distribution of installation rates

Revision

Adaption of installation rates of item combinations

that change from valid to invalid

Updating

Find referential for item combinations that change from invalid to valid

Planning Model


37

Effiziency gain with HUGIN

Example: Markov Net for VW „Bora“

Installation Rates for 460.000 attribute combinations

Reduction of RAM from 600 MB to 16 MB(Divisor = 38)

Reduction of computing time from infeasible to 250 sec (Divisor = 80.000)

38

Gain with efficient operations

14

Example : Markov Net for Volkswagen Sharan

First Prototyping (HUGIN) ... today ...

Max. numberdimensions

Number tuples

7 11

500.000 20.000.000.000

Full NetworkPropagation(Pentium 1,5 GHz)

1 sec 30 ms

Valid Tuples100.000 1.000.000

39

Planning Operation : Updating

Input Data : Set of item combinations that will change from invalid

to valid; set of valid referential combinations

Operation : Copy dependency structure (cross-product ratios)

from referential combination to input combination

and initialize with -probabilties.

Application : Technical modifications

The combination of engine and transmission

changes from invalid to valid, and it adapts the

quantitative dependencies from .

1e

3t

),( 32 te

40

Planning Operation : Revision

Input Data : Family of marginal / conditional probability distributions

Operation : Calculate Markov network with same structure that

satisfies all input distributions and is conform to the

principle of minimal change.

Application : Marketing stipulations

Installation probability of item air condition increases

by 10 % in case of Golf all-wheel drive in France.

Logistic restrictions

The maximum availability of engine in week 32/05

is 1.000 .

1e

416

Specification of Planning Data

Engines

Name: Golf - No. 02/07/05 - 17

Revision scheme:

Vehicle class: Golf Market: Germany Planning interval: 36/05

Revision context:

Partitioning:

Group of 1,8L spark engines

Rest

Diesel engine X2 (single item)

Diesel engine X1 (single item)

5,79 9,00

2,13 3,00

21,07 [18,20]

71,01

Installation rates (%)

estimated assigned

Context scheme: Body Equipment

Short back Comfort

≤ 500

Restriction

42

• Client-Server System

(current state: software implementation

and test environment for users)

• Server on 6-8 Machines (16 GB each)

• 4-Processor AMD Opteron system

• Terabyte storage device

• Operating System: Linux

• up to 15 system developers

• Programming language: JAVA

• WebSphere Application Developer, Eclipse

• DB-System: Oracle

• Worldwide rollout: now

18

Current State of Software Development

43

Need for Theory / Efficient Algorithms

Efficient transformation of logical rule systems into a relational network, techniques for complexity reduction and inconsistency management

Consistent quantitative fusion of a prior Markov network with a dependencies modifying relational network to a new Markov network

Handling generalized constraints

Efficient algorithms for revision and updating




Fusion



Revision

Adaption of installation rates of item combinations

that change from valid to invalid

Updating

Find referential for item combinations that change from invalid to valid

Documents

Prof. Dr. Rudolf Kruse Otto-von-Guericke-University Magdeburg [email protected] PD. Dr. Jörg Gebhardt Volkswagen, Wolfsburg ICS, Celle Probabilistic