DIRECTORS OF METHODOLOGY/IT DIRECTORS · In social sciences, the measuring process requires: (1) a...

EUROSTAT

DIRECTORS OF METHODOLOGY/IT DIRECTORS

JOINT STEERING GROUP

Luxembourg

June 28, 2016

From indicators tosynthesis.

Methodological issues in the construction of the construction of complex indicators

Filomena Maggino(University of Florence – Italy)

Synthesis of indicators

We need to start by

Premise

- sketching out the main methodological steps aimed

at developing indicators

- clarifying the different issues to face in synthesizing indicators

Premise

• considered a “niche field” from a scientific

point of view

• never missed in any conference, workshop,

seminar on measuring socio-economic

dimensions during the last decades

Developing indicators

and managing the

complexitycomplexity

(1) a normative exercise

Why talking about “indicators”?

In order to start any measurement process, a

crucial guiding principle is identified …

Three approaches to measurement:

• fundamental process ⇒ not derived from other

measures (length, volume)

• deriving process ⇒ derived from other measures

(density, velocity)

• defining process ⇒ achieved as a consequence of a

definition (socio-economic status)

In social sciences, the measuring process

requires:

• a robust conceptual definition

• a consistent collection of observations

• a consequent analysis of the relationship between

observations and defined concepts.

Indicator

what relates

concepts to reality

through observation

but …… what actually is an “indicator”?

“index”

“any thing that is

useful to indicate”

from Latin

“indicator” “index”

“who or what

indicates”

“any thing that is

useful to indicate”

from Latin

Indicator

a simply crude statistical information

a measure organically connected to a

conceptual model

Indicator

purposeful statistics

(K. Land)

index � indicator

when its definition and measurement occur in the

ambit of a conceptual model and is connected to a

defined aim

Indicators should be developed and managed so that they …

... represent different aspects of the reality,

... picture the reality in an interpretable way, and

... allow meaningful stories to be told

lack of any logical cohesion and consistency

deforming reality through distorted results

(hidden – sometime - by using and applying

sophisticated procedures and methods)

normative nature of the selection of indicators

cannot be denied cannot be denied

the process contains a “subjective” component

(2) between objectivity and subjectivity

Some clarification

on the use of the term “subjective”

Some clarification

on the use of the term “subjective”

• are we talking about defining the phenomena?

• are we talking about components of the phenomenon?

• are we talking about defining the method of measurement and analysis?

“subjective”

in defining phenomena

Process of describing the reality (conceptual framework) is

always subjective

�related to the researchers’ view of the reality

The conceptual definition represents only a “small window” through which

only some facets of the reality can be seen (reductionism).

“subjective”

as one of the components of the reality

We can distinguish between:

• objective information, collected by observing reality

• subjective information, collected only from individuals and

their assertions

“subjective”

in the measuring process

“subjective”

in the measuring process�

“subjective”

in the measuring process�

Sometimes in this context the dichotomy "subjective-objective" is considered

equivalent to the dichotomy "qualitative-quantitative". However, the two

dichotomies should be kept distinct

(3) the hierarchical design

Process allowing indicators to be developed

hierarchical design � requires the definition of the different

subsequent components

Hierarchical

Components Questions Components’ definition

Conceptual model �

What is the

phenomenon to be

studied? � It defines the phenomenon, its domains and its general aspects.

Variables � What aspects define

the phenomenon? �

Each variable represents an aspect allowing the phenomenon to

be specified consistently with the conceptual model

Hierarchical

design Variables �

the phenomenon? �

Dimensions �

What factors define the

aspect should be

observed? �

Each dimension represents each factor defining the

corresponding variable.

Basic indicators �

In which way each

dimension should to be

measured? �

Each indicator represents what is actually measured in order to

investigate each variable and its dimensions.

Hierarchical

What is the

phenomenon to be

the phenomenon? �

Hierarchical

the phenomenon? �

Dimensions �

aspect should be

observed? �

In which way each

measured? �

a) the model aimed at data construction,

b) the spatial and temporal ambit of observation,

c) the aggregation levels (among indicators and/or among observation units),

d) the models allowing interpretation and evaluation.

Hierarchical

What is the

phenomenon to be

the phenomenon? �

Hierarchical

the phenomenon? �

Dimensions �

aspect should be

observed? �

In which way each

measured? �

Hierarchical

What is the

phenomenon to be

the phenomenon? �

Hierarchical

the phenomenon? �

Dimensions �

aspect should be

observed? �

In which way each

measured? �

“Dimensionality” � theoretical

Two different situations can be observed:

• uni-dimensional � variable assumes a unique, fundamental underlying dimension

• multidimensional � variable assumes two or more underlying factors

Hierarchical

What is the

phenomenon to be

the phenomenon? �

Hierarchical

the phenomenon? �

Dimensions �

aspect should be

observed? �

In which way each

measured? �

“Dimensionality” � theoretical

The correspondence between the defined dimensionality and the selected indicators has to be

demonstrated empirically by testing the selected model of measurement.

Hierarchical

What is the

phenomenon to be

studied? �

It defines the phenomenon, its domains and its general

aspects.

the phenomenon? �

Each variable represents an aspect allowing the phenomenon

to be specified consistently with the conceptual model

Hierarchical

the phenomenon? �

to be specified consistently with the conceptual model

Dimensions �

aspect should be

observed? �

In which way each

measured? �

Each indicator (K. Land)

- represents a component in a model

- can be measured and analysed to compare different situations/groups … to observe

evolutions along time)

- can be related and integrated (aggregated) to specify the model

How many indicators?

1° option

Each variable measured by one indicator

single-indicator approach

1° option

Each variable measured by one indicator

single-indicator approach

weak � low precision and low accuracy

2° option

Each variable measured by more than one

indicator

multi-indicator approach

2° option

Each variable measured by more than one

indicator

multi-indicator approach

Necessary with multidimensional variables

Defining domains

- each variable

- each dimension

refers to domains

Defining domains

- each variable

- each dimension

Other indicators are needed!

- each dimension

refers to domains

Defining domains

Segments of the reality in which the

relevant concepts and their relevant concepts and their

dimensions have to be observed and

assessed

(4) the model of measurement

relationship between

variable and indicators

model of measurement

Conceptualmodel

variable I

dimension1

dimension2

B.I. a B.I. b B.I. c

variable II

dimension1

Two different models:

reflective formative

indicators � functions of latent variable

reflective

explanatory

perspective � top-down

�changes in the latent variable are reflected in

changes in the observable indicators

• indicators are

- linearly related

Statistical assumptions and properties

reflective

- linearly related

- interchangeable (the removal of an indicator does not change the essential

nature of the underlying construct)

• evidence for assessing the model � internal consistency:

- correlations between indicators can be interpreted only by the presence of

latent variables

- two uncorrelated indicators cannot measure the same construct

- each indicator has an error term

• total variance of each indicator can be expressed as a function of

i. latent variable (� uni/multi-dimensional � factors � communality)

Statistical assumptions and propertiesreflective

i. latent variable (� uni/multi-dimensional � factors � communality)

ii. individual indicator’s characteristics (uniqueness)

• errors and disturbance factors are not interrelated and are not correlated with

latent variables

Total variance of each indicator

sum of threecomponents�

1. common variance

• explained by � latent variable (and its dimensionality)

• measured by � correlation between indicators

2. specific variance

• not correlated with the other indicators

3. error, portion of the total variance

• not correlated with the previous

Multidimensional Latent Variables

explorative confirmatory

indicators � causal in nature

formative

explanatory

perspective � bottom-up

�changes in the indicators determine changes

in the definition / value of the latent variable

• indicators are not interchangeable (omitting an indicator is omitting part of

the construct)

Statistical assumptions and properties

formative

• two uncorrelated indicators can serve as meaningful indicators of the same construct (internal consistency is not important)

• indicators have no error term

Proper and accurate application of the

hierarchical design

complex structure

From basic indicators to systems of indicators

complex structure

each indicator measures and represents a

distinct component of the phenomenon of

interest

Proper and accurate application of the

hierarchical design

complex structure

System of indicators

DOMAINS

1 2 3 ..

variable i

dimension 1

sub-dimension 1

indicator A indicator A indicator A

indicator B indicator B indicator B

sub-dimension 2

indicator C indicator C indicator C

… … …

dimension 2

indicator … indicator … indicator …

CONCEPTUAL

dimension 2 indicator … indicator … indicator …

… … …

dimension … indicator … indicator … indicator …

variable ii dimension …

… … …

variable iii dimension … indicator … indicator … indicator …

variable … dimension … indicator … indicator … indicator …

(1) functions

Descriptive and

explanatory

functions

� Monitoring

� Reporting

(1) functions

Evaluation

functions

� Forecasting

� Accounting

�Program management and performance

evaluation

� Assessment

They can be seen in cumulative terms (each one requires the

previous one)

• Monitoring � capacity of the system to monitor changes

over time and meet the need of improving knowledge

• Reporting

• Forecasting

(1) functions

• Forecasting

• Accounting

• Program/performance evaluation

• Assessment

• Monitoring

• Reporting � monitoring + analysis + interpretation

• Forecasting

(1) functions

• Forecasting

• Accounting

• Assessment

• Monitoring

• Reporting

• Forecasting � observed trends can help in supposing

(1) functions

• Forecasting � observed trends can help in supposing

future trends and planning ex-ante analyses

• Accounting

• Assessment

• Monitoring

• Reporting

• Forecasting

(1) functions

• Forecasting

• Accounting � supporting decision concerning the

allocation and the destination of resources

• Assessment

• Monitoring

• Reporting

• Forecasting

(1) functions

• Forecasting

• Accounting

• Program / performance evaluation � problem

definition, policy choice and evaluation of alternatives and

program monitoring

• Assessment

• Monitoring

• Reporting

• Forecasting

(1) functions

• Forecasting

• Accounting

• Assessment � certificate or judge subjects (individuals

or institutions) by discriminating their performances or infer

functioning of institutions, enterprises or systems

Indicators developed through the hierarchical process are

seen in relation to each other and show a meaningful and

(2) characteristics of indicators

seen in relation to each other and show a meaningful and

precise position in the system consistently with the

conceptual model

(i) the perspective through which the indicators are reporting the

phenomenon to be observed

(ii) the communication context in which the indicators are used

(iii) the interpretation attributed to the indicators in statistical

analyses

(iv) the criteria of their adoption

(v) their quality

(i) perspectives of observation

indicators can describe

- a status or a trend- a status or a trend

- an objective fact or a subjective expression

- a positive or a negative aspect

- a conglomerative or a deprivational perspective

- a benefit or a cost

- an input or an outcome

- an impact

(ii) communication context

indicators can be

- cold indicators ⇒ complex and difficult, for specialists- cold indicators ⇒ complex and difficult, for specialists

- hot indicators ⇒ simple and easy

- warm indicators ⇒ good balance between quality, comprehensibility and

resonance

(iii) interpretation

indicators can be

- descriptive (informative)- descriptive (informative)

- explicative (interpreting another indicator)

- predictive (delineating possible trends)

(iv) criteria

interpretation of the results according to a specific reference frame

including particular standard-values, defined a prioriincluding particular standard-values, defined a priori

(iv) criteria

• reference point (or critical value)

• signpost arrow (comparison with previous performances)• signpost arrow (comparison with previous performances)

• best practice (a model to be followed)

(iv) criteria

Indicator: blood pressure

(iv) criteria

“the pressure is lower than normal”

(iv) criteria

“the pressure is getting lower”

(iv) criteria

requires

a wide consensus (not easy to be reached)a wide consensus (not easy to be reached)

involves

cultural paradigms, normative demands, expert

groups’ pressure, shared wishful ideas

(v) quality

clear, meaningful, consistent in describing the conceptual model s and in

relating to the defined aims and objectives

ACCURACY AND VALIDITY METHODOLOGICAL

SOUNDNESS appropriate, exhaustive, pertinent

in meeting requirements underlying its

construction (knowing, monitoring, evaluating,

accounting, …)

repeatable, robust PRECISION

INDICATOR

SHOULD BE

repeatable, robust in measuring the underlying concept with a

degree of distortion as low as possible

PRECISION

reproducible, stable RELIABILITY

transparent, ethically correct in data collection and dissemination OBJECTIVITY INTEGRITY

relevant, credible in meeting users’ needs APPROPRIATENESS

SERVICEABILITY

practicable, up-to-datable, thrifty in observing through realistic efforts and costs in

terms of development and data collection

PARSIMONY

well-timed, timely, punctual In reporting the results with a short length of

time between observation and communication

AVAILABILITY periodic, regular In observing the phenomenon over time (for

example, short time between observation and

data availability)

discriminant, disagregable, in recording differences and disparities between

units, groups, geographical areas and so on

COMPARABILITY

accessible, interpretable,

comprehensible, simple,

manageable

in being findable, accessible, useable,

analyzable, and interpretable

USABILITY ACCESSIBILITY

Managing indicators: instructions for use

a riska challenge

a need

a challenge

��

complexity

(1) A challenge: complexity

-Multidimensionalitydifferent aspects to be identified, not necessarily consistent

among them

-Nature-Nature- objective vs. subjective

- quantity vs. quality

-Levels of observation- micro vs. macro

-Dynamics- internal levels vs. external conditions

- trends, not necessarily linear

- relationships between phenomena

a challenge

��

complexity

making relative�

a need

(2) A need: making relative

From the conceptual point of view

in terms of

•consistency with the reference concept•consistency with the reference concept

•adequacy with reference to territory

e.g., nr. of beds in hospital

Making relative has strong implications with reference

to comparability of indicators

over time across

territories /

between

groups

concepts

analysis

to statistical treatment of indicators

equivalenceinvariance

- Sampling design

- Questionnaire design

- Data collection method

to statistical treatment of indicators

normalization

Which should consider:

• data properties

• original meaning of indicators

• values to be emphasized or penalized

• whether or not absolute value are used

• whether or not cases are compared to each other or to a reference unit

• whether or not units are evaluated across time

a challenge

�a risk

��

complexity

making relative�

a need

reductionism

(3) A risk: reductionism

reductionism� ��

unavoidable dangerous

The complexity of the system of indicators may require approaches allowing more synthetic

views through more comprehensive measures ...

… by taking into account that

all elements included in the system are

organically integrated

�they are not chosen independently

Reducing data complexity

to be considered as integral part of the process

leading to indicators development

Reducing data complexity

Solutions

(a) reducing the number of indicators

��Need of a solid conceptual framework

�� Statistical rational � correlations

nr. of firemen vs. amount of damages of fires

(b) synthesizing indicators

e.g. composite indicators

(b) synthesizing indicators answer the call by 'policy makers' for condensed information

improve the chance to get into the media

allow to make multi-dimensional phenomena uni-dimensional

allow situations across time more to be easily compared

compare cases (e.g. countries) in a transitive way (ranking and benchmarking)

allow clear cut answers to defined questions (related to change across time, difference

between groups of population or comparison between cities, countries, and so on)

(Noll, 2009)

Dealing with syntheses

in a system of indicators

A synthesis should be meaningful

Introduction

A synthesis should be meaningful

Introduction

Meaningful � telling stories

This is a meaningful

synthesis

Introduction

synthesis

Introduction

synthesis

Introduction

Is this is a meaningful

synthesis?

Introduction

What makes a synthesis meaningful

Introduction

What makes a synthesis meaningful

Conceptual framework

Introduction

Conceptual framework

Introduction

producing a

Introduction

Statistics offer many instruments

Introduction

Statistics offer many instruments

aimed at synthesizing …

Introduction

Instruments

Introduction

synthesizing indicators is not

Introduction

synthesizing indicators is not only a matter of having analytical instruments

Aspects of the system that can be synthesized

Introduction

synthesis of units (cases, subjects, etc.)synthesis of units (cases, subjects, etc.)

synthesis of basic indicators

aggregating the individuals' value of indicators

observed at micro level

aggregating the values referring to several

indicators for each unit (micro or macro)

Aspects of the system that can be synthesized

Introduction

each synthetic columns � indicators

synthetic

aggregation

goes through

reference

obtain

rows � units each

indicator macro-unit

Synthesizing units

MACRO-UNITS DEFINITION

Generally, pre-existent / pre-defined partitions, such as Generally, pre-existent / pre-defined partitions, such as

- identified groups (social, generation, etc.)

- areas (geographical, administrative, etc.)

- time periods (years, decades, etc.)

Synthesizing units

Looking for a value synthesizing the distribution

Generally

synthesis obtained by averaging individuals’ values synthesis obtained by averaging individuals’ values

at the level of interest (country, region, social group, and so on)

Synthesizing units

Generally

synthesis obtained by averaging individuals’ values synthesis obtained by averaging individuals’ values

at the level of interest (country, region, social group, and so on)

Which value?

Synthesizing units

Nature of data Value Index

different indexes according to nature of data

Nature of data Value Index

qualitative

disjointed � label mode

ordinal � natural /

conventional order median

quantitative

(additive)

discrete � natural number mean

continuous � real number mean

Synthesizing units

Synthesizing indicators

Different perspectives to be taken into

account:

conceptual design that guided in defining the indicators

(variables, dimensions, domains)

theoretical definition of the indicators (reflective or

formative indicators)

technical issues of synthesis (weighting, aggregation

techniques)

a. Conceptual perspective

b. Model-of-measurement perspective

c. Technical perspective

Consistent application of the hierarchical

design

complex structurecomplex structure

- variables

- indicators

Synthesizing indicatorsa. Conceptual perspective

- indicators

- domains

- cases

- etc.

DOMAINS

1 2 3 ..

variable i

dimension 1

sub-dimension 1

sub-dimension 2

… … …

CONCEPTUAL

dimension 2

… … …

The synthesis can be achieved through different perspectives:

for each uni-dimensional variable

- for a single domain (e.g., satisfaction at work)

- across domains (e.g., life satisfaction � all domains)

perspectives

for each multidimensional variable

- for a single domain (e.g., subjective wellbeing at work)

- across domains (e.g., subjective wellbeing � all domains)

across variables

- for a single domain (e.g., work � all concepts)

perspectives

across variables

perspectives

DOMAINS

1 2 3 ..

variable i

dimension 1

sub-dimension 1

sub-dimension 2

… … …

CONCEPTUAL

dimension 2

… … …

perspectives

across variables

perspectives

DOMAINS

1 2 3 ..

variable i

dimension 1

sub-dimension 1

sub-dimension 2

… … …

CONCEPTUAL

dimension 2

… … …

perspectives

across variables

perspectives

DOMAINS

1 2 3 ..

variable i

dimension 1

sub-dimension 1

sub-dimension 2

… … …

CONCEPTUAL

dimension 2

… … …

perspectives

across variables

perspectives

DOMAINS

1 2 3 ..

variable i

dimension 1

sub-dimension 1

sub-dimension 2

… … …

CONCEPTUAL

dimension 2

… … …

perspectives

across variables

perspectives

DOMAINS

1 2 3 ..

variable i

dimension 1

sub-dimension 1

sub-dimension 2

… … …

CONCEPTUAL

dimension 2

… … …

Sometimes,

somebody wishes

to synthesize …

DOMAINS

1 2 3 ..

variable i

dimension 1

sub-dimension 1

sub-dimension 2

… … …

CONCEPTUAL

dimension 2

… … …

Sometimes,

somebody wishes

to synthesize …

… too much …

Sometimes,

somebody wishes

to synthesize …

… too much …

Synthesizing indicatorsb. Model-of-measurement perspective

Conceptualmodel

variable I

dimension1

dimension2

variable II

dimension1

indicators � functions of latent variable

reflective

explanatory

perspective � top-down

�changes in the latent variable are reflected in

changes in the observable indicators

indicators � causal in nature

formative

explanatory

perspective � bottom-up

�changes in the indicators determine changes

in the definition / value of the latent variable

Aggregation based upon latent variables

When basic indicators are

- Reflective � high correlations

- variables difficult to observe

- synthesis easily interpretable

- Formative � no correlation

- variables difficult to interpret

- variable with normative meaning

Aggregation based upon latent variables

The hypothesis can be tested through structural modelswhich

• can estimate correlations between latent variables

• can not produce individual scores (undetermined

scores)

Two different general approaches

- aggregative-compensative techniques

based on correlations (reflective approach)

based on weights (formative approach)

- non-aggregative synthetic techniques

based on discrete mathematics

aggregative-compensative techniques

Synthesizing indicatorsc. Technical perspective

techniques

1. level of aggregation (micro/macro level)

2. dimensionality (dimensional analysis)

3. importance of each indicator (weighting criteria)

4. aggregating approach (aggregation technique)

1. level of aggregation (micro/macro level)

3. importance of each indicator (weighting criteria)

4. aggregating approach (aggregation technique)

formative indicators

1. Level of aggregation

In many cases, indicators observed at micro (individual level)

can be synthesized at

micro level or macro level

1. Level of aggregation

Example

Variable � subjective wellbeing

Dimension � affective

Indicators � positive affects and negative affects

Synthesis � affect balance

Computation

micro version � (positive affects) – (negative affects)

macro version � (positive affects) – (negative affects)

2. Dimensionality

Indicators are developed through a conceptual

concept � variable � dimension � indicator

Indicators are developed through a conceptual

process

2. Dimensionality

Testing dimensionality

� testing model of measurement

indicator � dimension � variable � concept

� testing model of measurement

� testing level of complexity

in order to evaluate the synthesis approach

2. Dimensionality

Highly correlated indicators

Reflective � indicators refer to the same

conceptual dimension

Consistency

They can be synthesized

2. Dimensionality

Highly correlated indicators

Formative � redundant indicators

Redundancy

two indicators highly correlated are

considered redundant

Recommendation � select only one

2. Dimensionality

• Correlation Analysis

• Principal Component Analysis

• Multidimensional Scaling

• Cluster Analysis

• Factor Analysis

• Item Response Theory

reflective approach

3. Importance

weighting system

Weight � indicator’s importance in measuring the

3. Importance

weighting system

who / what defines the importance�

3. Importance

Decisions

proportional size of weights

equalequal or differentialdifferential weighting

3. Importance

Decisions

approach to obtain weights

objectiveobjective or subjectivesubjective

3. Importance

Decisions

level for obtaining and applying weights

individualindividual or groupgroup

4. Aggregating approach

Choosing the aggregation technique

criteria � does the aggregating technique …

� admit compensability among the indicators?

� require comparability among indicators? (direction and distribution)

� require homogeneity in indicators’ levels of

measurement?

1. testing the robustness: capacity of the synthesis to

produce correct and stable measures

• uncertainty analysis

More over …

• sensitivity analysis

Testing robustness

evaluating role and consequences of choices made with

reference to:

• procedure for data management (missing data imputation, data

standardization, ...)

• weights definition adopted

• aggregation technique applied

Testing robustness

evaluating role and consequences of made choices with

reference to:

• procedure for data management (missing data imputation, data

standardization, ...)

• weights definition adopted

• aggregation technique used

This process can be included in the wider field of “what-if” analysis

Testing robustness

�Two stages

1. uncertainty analysis

to analyze to what extent the

synthetic indicator depends on

information composing it.

each individual score � scenario

scenario � combination of choices

producing a certain synthetic value

Testing robustness

�Two stages

1. uncertainty analysis

to analyze to what extent the

synthetic indicator depends on

information composing it.

each individual score � scenario

scenario � combination of choices

producing a certain synthetic value

Testing robustness

�Two stages

2. sensitivity analysis

to evaluate the contribution of each

identified source of uncertainty by

decomposing the total variance of the

synthetic score obtained.

Testing robustness

�Two stages

composite indicator

formula (12%)

selection of sub-indicators

data selection (25%)

data editing (8%)

data normalisation

weighting scheme (15%)

weights' values (15%)

Testing robustness

�Two stages

to tests how much the synthetic score

is sensitive to different choices (small

differences reveal low sensitivity)

Testing robustness

�Two stages

to tests how much the synthetic score

is sensitive to different choices (small

differences reveal low sensitivity)

1. testing the robustness: capacity of the synthesis to

produce correct and stable measures

More over …

• sensitivity analysis

2. testing the discriminant capacity: capacity of the final

score to discriminate between cases (e.g., cut-points,

cut-offs, …)

Exploring the capacity of the synthetic score in

• discriminating between cases and/or groups (statistical tests)

• distributing all the cases without any polarization

• identifying particular values or reference scores

cut-point � continuous data

cut-off � discrete data

Conceptual, methodological and technical

criticisms

Synthesizing indicatorsd. Criticisms

Conceptual, methodological and technical

criticisms

�Aggregative –compensative approach is not able

- to reflect the complexity of a phenomenon

- to capture the complexity of variables’

relationships

Who is in favor of aggregative-compensative

approach

�- objectively built

- easy to manage

- easy to communicate

Methodology is far from being aseptic

Each stage introduces some degree of arbitrariness in

taking decisions concerning

a. Data metrics

b. Indicators selection

c. Weights definition

d. Indicators aggregation

e. Indicators assessment

a. Data metrics

Ordinal data

treated as

cardinal

a. Data metrics

Use of

multidimensional

analysis

a. Data metrics

Objective

subjective

a. Data metrics

• No homogeneity

• Compensation

• Numerical weights

a. Data metrics

e. Indicators assessmentAre the compared

combination really

comparable?

4. Aggregating approach

Question

… and with ordinal data?

Non-aggregative synthetic techniques

techniques

Main goal

Assessing variables in a multidimensional ordinal setting by:

setting by:

1) respecting the ordinal nature of the data

2) avoiding any aggregation among indicators (i.e., no

composite is computed)

3) producing a synthetic index

Main tool

Partial Order Theory

1. The focus is not on dimensions, but on «profiles» �

1. The focus is not on dimensions, but on «profiles» �combination of ordinal scores, describing the

«status» of an individual.

2. Profiles � mathematically described and analyzed

through Partially Ordered Set (Poset) Theory,

instead of using classical linear algebra tools

(variances, correlations,…)

Main tool

By considering all this, new challenges and perspectives can be identified in

By considering all this, new challenges and perspectives can be identified in order to improve the technical strategies allowing the social indicators to be constructed and managed, with reference to the challenging issues in managing the complexity in terms of both:• reducing data structure in order to aggregate and combining both

observational units (from micro-units to macro-units) and indicators• correctly and significantly communicating the “picture” obtained through the

indicators (correctly presenting results).

Let’s consider

-three indicators d , d , d

-three indicators d1, d2, d3

-their ordinal degrees 1, 2, 3, 4

(d1, d2, d3)

� individual profile

How many profiles with 3 four-degree

indicators?

4 x 4 x 4 = 64

profile combinations

We cannot

- sum up the degrees

- manipulate them using classical statistical tools

- compute aggregated indicators

We can say that:

• some profiles are better than others, for example

(4,3,3) is better than (2,3,2)

• some profiles are incomparable, for example (4,3,3)

and (3,3,4). In fact the first profile is better on the first component, but it is worse on

the third. So, which the best of the two? We cannot say!

So, some profiles can be ordered, others cannot. What we get

PARTIAL ORDER

(«partial» since not any pair of profiles can be ordered)

We can represent the partial order of the well-being profiles very

effectively, by means of a graph, called «Hasse diagram» (from the name

of the German mathematician Helmut Hasse)

Only profiles linked by a

descending path can be

ordered.

Two profiles not linked by

a descending path are

incomparable, even if

they belong to different

«layers»

The meaning of the colors will be explained later

Only profiles linked by a

descending path can be

ordered.

Incomparable profiles

Comparable profiles

Best profile

ordered.

Two profiles not linked by

a descending path are

incomparable, even if

they belong to different

«layers»

The meaning of the

colors will be explained

laterWorse profile

In order to make incomparable profiles, comparable, a

threshold should be defined.

(1,2,2) (2,1,2) (2,2,1)

How to establish the threshold?

�normative problem

Green: profile

above the

threshold.

Black: ambiguous

profile.

Red: profiles on or

below the

threshold.Threshold

Threshold

The idea is to quantify this «relational position» in the graph.

This idea can be formalized within partial order theory.

What is important to stress here is that this way we get a

synthetic measure of deprivation

WITHOUT AGGREGATING ORDINAL SCORES

Final remarksFinal remarks

all elements composing the process should be

organically integrated �

Final remarks

�they cannot be chosen independently

However, sometimes they appear unconnected

in particular

Final remarks

in particular

Final remarks

data and statistical approaches are not able to

rebuild the conceptual framework

in particular

Final remarks

data and statistical approaches are not able to

rebuild the conceptual framework

�This is obtained through subsequent

approximations of the whole process

� - - - - - - - - - - - - - - - - ?

Is this acceptable in a complex structure?

How can we build a correct process without

Final remarks

How can we build a correct process without

producing a chain of approximations?

In order to obtain a meaningful and

interpretable picture

Final remarks

interpretable picture

data should be managed in some way…

in other words …

Indicators obtained through the hierarchical

design are and remain a

Final remarks

design are and remain a

complex system

ManyMany thanksthanks forfor youryourattentionattention!!!!

filomena.maggino@unifi.it

attentionattention!!!!

DIRECTORS OF METHODOLOGY/IT DIRECTORS · In social sciences, the measuring process requires: (1) a...

Documents

Actele normative

UNESCO's normative instruments concerning Technicalunesdoc.unesco.org/images/0014/001406/140603e.pdf · UNESCO's normative instruments concerning Technical ... UNESCO’s normative

Normative Theory

Normative Accounting Theories - سعید نصیریnasiri.iauahvaz.ac.ir/Files/normative acounting theory1.pdf · Normative Accounting Theories ... normative accounting theorists

Normative Zidarii

Gjuha Normative

Normative Ethics

Normative Pardoseli

Yosemite Mariposa County Tourism Bureau Board of Directors ... · 3/19/2020 · •Trade Shows • Yosemite.com consistent visitation increases of over 200,000 visitors monthly,

Toward A Normative Theory of Normative Marketing Theoryimpgroup.org/uploads/papers/4650.pdf · Page 2 Toward A Normative Theory of Normative Marketing Theory Abstract We show how

Children's Normative Beliefs About Aggression and … · · 2006-09-25Children's Normative Beliefs About Aggression and Aggressive Behavior ... longitudinal relation between normative

Normative Personal Spitale

Normative Beliefs

Lista Normative 2007

Normative universo

Normative bosica

Emotion and Efficacy Pathways to Normative and Non-normative Collective Action

normative position

STOP - APLU of directors with the authority to make decisions consistent with a common mission and ... Through the eXtension Foundation Board of Directors,

Normative Uncertainty, with correctionscommonsenseatheism.com/.../2014/03/MacAskill-Normative-Uncertai… · normative uncertainty analogously with how they treat empirical uncertainty