View
37
Download
0
Embed Size (px)
DESCRIPTION
Integrates three perspectives of modelling and analytics: modeller, manager, and regulator.
Citation preview
Analytics in Context:
Modelling in a Regulatory
EnvironmentPresented to:
International Conference on Analytics Solutions Ottawa, Ontario, September 29-30, 2014
Presented to:
International Conference on Analytics Solutions Ottawa, Ontario, September 29-30, 2014
Albert SimardIntegrated Knowledge Services
2
OutlineOutline
Modelling Context
Human, Technical
Framework
Structure, Hierarchy
Modelling Context
Human, Technical
Framework
Structure, Hierarchy
3
Diverse PerspectivesDiverse Perspectives
What Analysts proposed
What managers funded
What stakeholders wanted
What users needed
Manager DiversityManager Diversity
• Accountable for actions• Situational pressure• Broader view than analysis• Organizational infrastructure • Depth of understanding • Confidence in results• Risk tolerance• Previous experience• Belief system
• Accountable for actions• Situational pressure• Broader view than analysis• Organizational infrastructure • Depth of understanding • Confidence in results• Risk tolerance• Previous experience• Belief system
Analyst DiversityAnalyst
Diversity
• Objective Criteria–Problem space–Type of problem–Available techniques
• Subjective Criteria–Awareness–Diversity –Skill–Experience–Expertise–Mental model
• Objective Criteria–Problem space–Type of problem–Available techniques
• Subjective Criteria–Awareness–Diversity –Skill–Experience–Expertise–Mental model
User Diversity
User Diversity
GoalsCompatible Conflicting
Interests
Mutual
Autonomous
Competitiondefence or victory aggressive approach no trust secretive, hostile
Collaborationjoint or peer production partnership approach high trust diverse, synergistic
Negotiationmutual agreement adversarial approach nominal trust structured, formal
Sharingleverage knowledge passive approach moderate trust benign, supportive
The Challenge
The Challenge
Integrating the diverse views to increase understanding and achieve a mutually agreed solution!
Integrating the diverse views to increase understanding and achieve a mutually agreed solution!
7
8
Modelling ProcessModelling Process
• Modelling combines science & computers; judgement & experience; insight & intuition.
• Principles: effort, simple, data, knowledge, transparent, understandable.
• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.
• Development: techniques are well-understood; management less understood and practiced.
• Use: Decision making under uncertainty, unknown elements, outcome probabilities.
• Modelling combines science & computers; judgement & experience; insight & intuition.
• Principles: effort, simple, data, knowledge, transparent, understandable.
• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.
• Development: techniques are well-understood; management less understood and practiced.
• Use: Decision making under uncertainty, unknown elements, outcome probabilities.
Spectrum of Evidence
Spectrum of Evidence
• mathematics, logic, proof
• science, engineering, technology
• statistics, data, facts, measurement
• collaboration, validation, management
• expertise, experience, judgement
• opinion, perception, bias
• belief, emotion, values
• mathematics, logic, proof
• science, engineering, technology
• statistics, data, facts, measurement
• collaboration, validation, management
• expertise, experience, judgement
• opinion, perception, bias
• belief, emotion, values
Quantitative (irrefutable evidence)
Qualitative (no evidence)
10
Data Attribute
s
Data Attribute
s A model and its data are inseparable;
they succeed or fail as one.• Data Needs: Situation may involve nature, the
system, and/or intervention.• Sampling: Statistics are essential to determine
how much data is needed.• Source: Ownership? Use rights? Privacy &
security concerns?• Scale: Time, space, and process scale must
match the situation.• Quality: What level of accuracy, detail, and
completeness are needed?
A model and its data are inseparable; they succeed or fail as one.
• Data Needs: Situation may involve nature, the system, and/or intervention.
• Sampling: Statistics are essential to determine how much data is needed.
• Source: Ownership? Use rights? Privacy & security concerns?
• Scale: Time, space, and process scale must match the situation.
• Quality: What level of accuracy, detail, and completeness are needed?
Knowledge Domains
Knowledge Domains
Adapted from Kurtz and Snowden (2003)
12
OutlineOutline
Modelling Context
Human, Technical
Framework
Structure, Hierarchy
Modelling Context
Human, Technical
Framework
Structure, Hierarchy
13
Framework ObjectivesFramework Objectives
• Support needs-driven and science-driven analysis.
• Promote dialogue among modelers, managers, & users.
• Reduce wasted time, effort, & money.
• Provide a basis for planning and action.
• Document and justify decisions.
• Support needs-driven and science-driven analysis.
• Promote dialogue among modelers, managers, & users.
• Reduce wasted time, effort, & money.
• Provide a basis for planning and action.
• Document and justify decisions.
14
Framework DesignFramework Design
• Reflect modelling, management, and user perspectives.
• Balance efficiency and effectiveness with cost and effort.
• Applicable to both demand and supply approaches to modelling.
• Applicable to both logical and computational models
• Reflect modelling, management, and user perspectives.
• Balance efficiency and effectiveness with cost and effort.
• Applicable to both demand and supply approaches to modelling.
• Applicable to both logical and computational models
15
Framework HierarchyFramework Hierarchy
• Phase: (4) demand, supply, project, use
• Stage: (9) approach, design, establish, develop, evaluate, identify, applicability, implement, conclude
• Step: (34) screening, problem definition, suitability, knowledge, data
• Consideration (132): classification, statements, decision
• Phase: (4) demand, supply, project, use
• Stage: (9) approach, design, establish, develop, evaluate, identify, applicability, implement, conclude
• Step: (34) screening, problem definition, suitability, knowledge, data
• Consideration (132): classification, statements, decision
16
Framework PhasesFramework Phases
Issue Model
Demand Supply
Project
End
Implement
Supply & DemandSupply & Demand
Supply: What can I do with an available model?
(Intelligence)
Demand: What model do I need to solve a problem?
(Regulation)
18
Start (methods)
Demand-drivenbackward chaining, closed question
Model
Supply-drivenforward chaining,open question
Start (model)
Uses
Supply & DemandSupply & Demand
19
Issue
Approach
Design
Development
Acquire Data
Generate Knowledge
Out
Demand PhaseDemand Phase
20
Design
Acquire Data
Generate Knowledge
Approach
Out
Evaluation
IssueStart: (Demand)
Implementation
Outputs
Conclusion
End
Development(Modeller)
(Manager)
(User)
Applicability
ModelStart (Supply) identification
Establishment
(Manager)
Framework Stages
21
Issue
Initial Screening
Problem Definition
Suitability
Knowledge Evaluation
Data Availability
Design
Recurrence ImportanceProblem spaceExistence
Business FunctionIntended use
Time available Situation
NeedsExistingGap
NeedsAttributesAccessibilityProcessing
Continue
Continue
Continue
Continue
Continue
Below threshold
Can’t define
Unsuitable
Excess gap
Inadequate
Generate?
Acquire ?
Yes
Yes
Out
No
No
Approach Stage - Steps
Considerations
22
ConsiderationsConsiderations
• Explain what is being considered• Classify a situation or write a short
description. (categories)• Complete a statement. (template)• Decide where to go next. (decision guide)• Not a cookbook to be followed without
interpretation.• Compliments experience & judgement;
doesn’t replace them.
• Explain what is being considered• Classify a situation or write a short
description. (categories)• Complete a statement. (template)• Decide where to go next. (decision guide)• Not a cookbook to be followed without
interpretation.• Compliments experience & judgement;
doesn’t replace them.
23
• [Specified data] are needed to [develop, run] a model of [function, process] because [explanation].
• The [model name] needs [specified knowledge] about [specified function, process] because [explanation].
• [Specified data] are needed to [develop, run] a model of [function, process] because [explanation].
• The [model name] needs [specified knowledge] about [specified function, process] because [explanation].
Consideration Template
Consideration Template
24
• If a substantial majority of the data needed by the model exist, go to Data Accessibility.
• If a majority of the data needed by the model exist, go to Data Accessibility, subject to limits or conditions; describe the limits.
• If existing data are adequate for a simpler or partial model, go to problem definition; redefine the problem.
• If existing data are not adequate for a simpler or partial model, go to Data Acquisition.
• If a substantial majority of the data needed by the model exist, go to Data Accessibility.
• If a majority of the data needed by the model exist, go to Data Accessibility, subject to limits or conditions; describe the limits.
• If existing data are adequate for a simpler or partial model, go to problem definition; redefine the problem.
• If existing data are not adequate for a simpler or partial model, go to Data Acquisition.
Decision Guide
Decision Guide
The Modelling Framework
• Supports business objectives• Facilitates horizontal integration• Minimizes waste & inefficiency • Maximizes likely success • Documents & justifies decisions
• Supports business objectives• Facilitates horizontal integration• Minimizes waste & inefficiency • Maximizes likely success • Documents & justifies decisions
“Using a clear blueprint first prevents chaos latter.” Carla O’Dell (1998)