Towards Validating Social Network Simulations

Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 1

Towards Validating Social Network Simulations

SMA Abbas1, Shah Jamal Alam2 and Bruce Edmonds1

1Centre for Policy Modelling, Manchester Metropolitan University2School of Geosciences, University of Edinburgh


The Situation

A Simulation Social

“System”

GeneratesMeasured

Are these “essentially” the

same?

A Class of Networks Another Class of Networks?

Synthetic

Networks

TargetNetworks


Key questions

• What properties of the synthetic networks, one would expect to observe given how the model has been constructed

• Which of these properties are ‘significant’ in terms of the intended processes in the model

• Which class of target networks one might expect to observe if one could “re-run” reality under the same basic conditions as assumed in the model

• Do these classes match in important respects• How do we know they do given we only have

samples of synthetic and target networks


The Problem


Summary of Issue

• The space of possible networks is vast • But many networks will look similar to us, because

our brains can not deal with them but automatically simplifies them as part of perception

• We are not dealing with single networks but classes of networks…

• …though these classes are often implicit when a single network stands for that class (somehow)

• However, in principle, if synthetic and target networks do match (in some way) then this is potentially a strong validation


A Cautionary Tale – comparing two network modelsPapadopoulos et al. (2012) Popularity versus similarity in growing networks. Nature, 489:537-540.


But when compared in a different way…


Some Network Comparison Approaches

Different kinds of things to compare:• Network Measures• Network Distributions• Eigenvalue/Eigenvectors• Subgroup Identification• Functional Comparison• Likelihood of being described by an Exponential

Random Graph Model• Motif Prevalence


Examples that follow are from work of S.M.A. Abbas

(see papers at https://sites.google.com/site/maliabbas)

An Example of Validating Synthetic vs. Target Networks


An example of comparing measures


An example of comparing distributions

1 5 10 50 100 500 1000

5e

-04

5e

-03

5e

-02

5e

-01

Log-log Plot of Degree Distribution

Degree

Cu

mu

lative

Fre

qu

en

cy

ReferenceRandomFAOFPartyHybrid


Silo Index Comparison

Correlation0.83

-1.0 -0.9 -0.8 -0.7 -0.6 -0.5

-1.0

0-0

.85

-0.7

0

Dorm Silo Index

Reference Dorm

Hybrid D

orm Correlation

0.93

-1.00 -0.90 -0.80 -0.70

-1.0

0-0

.90

-0.8

0

Major Silo Index

Reference Major

Hybrid M

ajo

rCorrelation0.84

-1.0 -0.5 0.0 0.5

-1.0

-0.8

-0.6

Year Silo Index

Reference Year

Hybrid Y

ear

Correlation0.29

-1.0 -0.8 -0.6

-1.0

0-0

.90

High School Silo Index

Reference High School

Hybrid H

igh S

chool

Reference vs. Hybrid Mode Silo Indices


Assortativity Mixing


Problems in the Literature

Authors are often not clear about:• Precisely what the links in a synthetic network are

supposed to represent (in terms that would allow an in principle measurement of observed actors)

• Which aspects of the target network are subject to measurement error (or otherwise judged not to be significant) and which should be reproduced by a synthetic network

• Which aspects of the synthetic network are significant in terms of the generating process (and which are essentially accidental)

Readers often cannot judge the extent or meaning of the match/mismatch between synthetic and target networks


Conclusions

• Many social simulation models assume stereotypic networks (e.g. Watts-Strogatz)

• It is increasingly clear that the exact network structure matters (e.g. Holzhauer ESSA 2013)

• No single ‘Golden Bullet’ technique• More thought needed about what is significant about the

synthetic and target class networks• Multiple approaches needed to show that classes of

networks are similar – a few 1D measures is not enough to show this

• Validating networks could be quite a strong validation of our models…

• …but much more work is needed in this area!


Thanks!

SMA Abbas

https://sites.google.com/site/maliabbas

Shah Jamal Alam

https://sites.google.com/site/jamialam

Bruce Edmonds

http://bruce.edmonds.name

Slides at:

http://www.slideshare.net/BruceEdmonds

Education

Towards Validating Social Network Simulations