Applied(Risk(Management Specialty(Group( …c(Supportfor(Prac&cal(Risk(Management Applied(Risk(Management Specialty(Group(Principles, Guidelines and Core Knowledge For Analytic Support

Analy&c Support for Prac&cal Risk Management

Applied Risk Management Specialty Group

Principles, Guidelines and Core Knowledge For Analytic Support of Risk Management

1

SRA Webinar, August 31, 2016




Value ProposiBon !  Risk Management needs a set of clearly wriEen principles

that is all in one place, and that apply to all domains of applicaBon.

!  That set of principles should provide the basis for a general climate of pracBce: -‐ all risk management analyses should comply with those principles -‐ provide a basis for Third Party Reviews, etc., re transparency, documentaBon.

!  Funding agencies should: -‐ insist that those principles be followed -‐ provide adequate funding and schedule for compliance.

!  All RFPs and equivalents should specify that those principles be followed.

!  But, we have ISO 31000, ISO/IEC 15288, and many other standards. !  But most are vague and/or don’t apply to the issues we’ve listed in this doc. !  And anyway, we don’t see those applied and encouraged, as above. !  Central QuesBon: How can we make that encouragement happen? !  Our Working Hypothesis: We can work toward that encouragement

happening if we embark on an ARMSG-‐SRA “Community Effort” to write the doc we describe here, and push it hard within ARMSG – SRA.

!  Are we delusional? 2


Overview How Do We Make This Work, Avoid Delusion? …………………………………………….. 4 Scope …………………………………………………………………………………………………………… 5 Goals of This Doc and This Project ………………………………………………………………… 6 Core Values …………………………………………………………………………………………………. 8 Ten Principles ……………………………………………………………………………………………… 9 21 Challenges ……………………………………………………………………………………………… 12 Guidelines ………………………………………………………………………………………………….. 14 Reality Check ……………………………………………………………………………………………… 15 Ideas we couldn’t fit into our SecBons: Appendices …………………………………... 16 A1. Features of a “Good” Analysis to Support Risk Management ………………. 16 A2. A List of Problems and Piealls to be Addressed, Based on Experience …. 16 A3. MisconcepBons of Risk ……………………………………………………………………….. 16 A4. Survey of Failures of Risk Assessment and Risk Governance ……………….. 16 A5. Other Examples Conveying the Importance of our Core Values, etc. ……. 16 A6. Examples of Issues of Trust and Acceptance ………………………………………... 16 A7. EvoluBon of Risk Assessment Models, and Future DirecBons …………….... 16 A8. Bold New Approaches to Risk Assessment and Management ………………. 16 A9. Leveraging Work of Other Groups, Tying into a Larger Framework ………. 17 A10. A Catalog of Risk Management Standards ………………………………............ 17 Paths Forward ……………………………………………………………………………………………. 18 Issues to Discuss ………………………………………………………………………………………… 19

3

Slide



How Do We Make This Work, Avoid Delusion? !  This project will be centered on a document, described here,

but not simply on that document, but rather how it is wriEen, how it is applied, how it is discussed and tested out at SRA conferences.

!  Embed our Principles within: -‐ Core Values -‐ Challenges Addressed -‐ Domain-‐Specific Guidelines to apply those principles

!  Base all of that on our pracBcal experience. No academic treaBses allowed.

!  Build this doc based on contribuBons from each applicaBon domain, building it up as an analysis community effort.

!  Hammer on this at every SRA conference: -‐ present papers evaluaBng projects based on this doc. -‐ present lessons learned from both successes and failures. -‐ some of those lessons including how to get this doc (principles, guidelines) complied with.

!  Figure out how to avoid the situaBon I was in at an SRA conference, where I presented a paper on how to improve an agency’s model, when an agency representaBve told me to get off the stage.

4

How can we de-‐delusional this?

Are we delusional, here?



Scope !  AnalyBc support of risk management, as in, decision aiding.

!  That is, risk management decision aiding, as opposed to simply risk analysis.

!  All this extends to whatever is called for for risk management: risk: idenBficaBon, research, analysis, assessment, evaluaBon, communicaBon, decision making and management.

!  That extends to whatever analyses are called for, perhaps beyond PRA. For example if PRA doesn’t address the need for robustness, accounBng for Black Swans, etc., then decision aiding analyses should extend beyond PRA.

! Our SecBon 10 lists 16 domains of applicaBon (this list is bound to cause argument): -‐ Health -‐ Asset Management -‐ Project Risk -‐ Environment -‐ Finance -‐ Any other domains -‐ Terrorism -‐ Governance contributors suggest -‐ Infrastructure -‐ Foreign Policy -‐ Engineered Systems -‐ Military -‐ Natural Hazards -‐ Crime -‐ Cyber Security -‐ Insurance: Life, Health, Property, etc. -‐ Cyber-‐Physical Systems (e.g. SCADA, drones, driverless cars)

5

Any edits or addiBons?



Goals of This Doc and This Project 1 of 2

G1. To bring into one place all Principles-‐Guidelines, logical structure, criBqued, agreed.

G2. To achieve the most effecBve risk management possible.

G3. To support/encourage analyses that make the best use of available data, subject maEer expert (SME) judgments, assumpBons and analyses.

G4. To establish a system of Principles-‐Guidelines in a single, universally applicable set.

6


(Each goal supports the goals above it.)

G5. To enable the shared understanding-‐communicaBon of those principles-‐guidelines, including the language necessary for meaningful and consistent applicaBon with, and tesBng for compliance of, those principles and guidelines.

G6. To create an environment and culture of “Analysis Quality” -‐ among analysts. G7. -‐ among risk managers.

InsisBng on compliance with these Principles-‐Guidelines, supported by reviews.

6



Goals of This Doc and This Project 2 of 2

7


G10. To enable system(s) by which analyses supporBng risk management decisions can be tested against those principles.

G11. … and that, in turn, calls for standards of transparency and documentaBon.

G12. That procedural framework designed to counter “check the box” acBvity.

G13. Establish a ~2-‐page knowledge base among the funders/commissioners/users that establishes a proper understanding of: -‐ what analyses supporBng risk management are about -‐ how analyses supporBng risk management are supposed to, can assist in decision making.

G8. To establish a procedural framework where funders/commissioners of risk management analyses insist that those analyses comply with those principles and guidelines, as determined by some tests/review.

G9. To establish a procedural framework where funders/commissioners of risk management analyses allocate adequate budget and schedule for the risk management analyses and the associated tests/review.

This could use examples: -‐ of especially good cases -‐ of egregiously bad cases.



Core Values Analyses in support of risk management:

! … should bring the power of analysis to bear to apply what is known, including all uncertainBes, and consider what is not known, to generate the most effecBve guidance for risk management possible.

! … should provide honest, transparent risk management advice independent of vested interests.

! … should be sufficiently comprehensive for its purpose.

8


Analyses in support of risk management and any associated models:

! … should be adequately verified and validated.

! … should be effecBvely peer reviewed.

! … should include a consideraBon of the risk of improper analysis.




Ten Principles 1 of 3 !  P1: The Single Overriding Principle: Best Use of available data and analyses.

That includes: -‐ the clarity of that risk management guidance to non-‐specialist risk managers -‐ appropriately couching (caveaBng) that guidance with the limitaBons of data, scope and analyses involved, and the implicaBons of those limitaBons for risk management decisions.

Principles 2 – 10 spell out parBcular aspects of this first principle:

9

!  P2: Analyses must be engaged effecBvely in the risk management decision process. That is ooen beyond the control of the analysts, but our work here can help analysts make that case.

Principles 3 – 10 spell out parBcular aspects of this second principle:

!  P3: An essenBal element of that effecBveness is Trust. That is, all users and stakeholders must trust that all acBviBes of the analyses have been conducted with full transparency, and intenBons as announced.

These could use examples: -‐ of especially good cases -‐ of egregiously bad cases.



Ten Principles 2 of 3 !  P4: Results should be formaEed and presented such that non-‐specialist

risk managers can apply those results validly to advise their decisions, including balancing all the risk consideraBons with all other decision factors.

!  P4a: Results should be formaEed in units that non-‐specialists can validly trade off against other aspects of their opBons. Example: “Confidence Factor” vs cost?

!  P4b: Results formaEed in units reflecBng actual metric level / precision/accuracy. Example: Bar charts are read as raBo-‐scale data, so … are they? Example: Inadequate, or inadequately labeled, error bars. (e.g. 90%? Correlated?) Example: Three sigfigs displayed, when results are only valid to one sigfig.

!  P5: Timeliness. If the analysis is too late to advise the decision, then it plays a different role than risk management advice, e.g. defense. So:

!  P6. Adjust the analysis to the actual, as opposed to announced, role it is to play.

!  P7. PracBce full disclosure re the actual role of the analysis, e.g. advise vs. defend.

!  P8. Be explicit about other roles for the analysis, less strategic than in P7. Examples: VisualizaBon, communicaBon, guide further research and analysis.

10



Ten Principles 3 of 3

11

!  P9: Clearly state assumpBons and caveats, and the implicaBons of those assumpBons and caveats for using the results for to advise decisions, -‐ in terms understandable to the decision makers / risk managers -‐ placed immediately next to results (text, numbers, graphics, tables), such that -‐ the decision makers understand the limitaBons of the analysis, including: -‐ scope and its implicaBons for interpreBng and applying the results -‐ assumpBons and their implicaBons for interpreBng and applying the results -‐ data limitaBons and their implicaBons for interpreBng and applying the results

!  P10: Full Disclosure (P7 – P9). Any analysis supporBng risk management is limited by budget, schedule and data limitaBons. No analysis can be ideal. But all analyses should include full disclosure of all shorealls and the implicaBons of those shorealls, all stated in terms such that risk managers can understand them and validly apply them to advise their risk management decisions.




21 Challenges That Guidelines Should Address 1 of 2

12

!  C1: Capturing the risk generaBon process re known, knowable, and unknowable. Then developing risk management advice that accounts for that.

!  C2: Characterizing the risk event space, scenario space, and its completeness. Then developing risk management advice that accounts for that.

!  C3: Reducing large amounts of data down to effecBve decision guidance in a way that is valid and reviewable by a third party.

!  C4: Assessing the uncertainBes. !  C5: Taking those uncertainBes into account in risk management. => C6 – C8:

!  C6: Preparedness for scenarios “not on the list.” That is, recognizing the possible occurrence of such scenarios, and preparing for them.


!  C7: Developing robust risk management strategies.

!  C8: Developing resilient risk management strategies.

!  C9: Setng an adequate budget to achieve consistency with the principles, guidelines and core knowledge specified here.

!  C10: Validly choosing among and applying the most appropriate analyses, among the analyses that could be applied.



13

!  C11: If the risk involves an adversary (terrorist, criminal, government, compeBtor), then modeling that adversary in a way that captures adapBve behavior.

!  C12: Data availability, collecBon.

!  C13: Data validaBon.

!  C14: Data management.

!  C15: Data Quality Assurance, Quality Control.


!  C16: Model validaBon.

!  C17: Model documentaBon.

!  C18: Model communicaBon.

!  C19: Decision process validaBon.

!  C20: Decision process documentaBon.

!  C21: Decision process communicaBon.

21 Challenges That Guidelines Should Address 2 of 2



Guidelines

14

Our logic:

! We have listed a first drao of Core Values, Principles and Challenges.

! We consider those as applying over all risk management domains of applicaBon.

!  But now we posit that Guidelines, more directly applicable guidance than Core Values, Principles and Challenges, should be specified: -‐ specific to each domain of applicaBon, -‐ by experts in each domain of applicaBon.

!  So here we simply repeat our first-‐drao list of domains of applicaBon from Slide 4:

-‐ Health -‐ Asset Management -‐ Project Risk -‐ Environment -‐ Finance -‐ Any other domains -‐ Terrorism -‐ Governance contributors suggest -‐ Infrastructure -‐ Foreign Policy -‐ Engineered Systems -‐ Military -‐ Natural Hazards -‐ Crime -‐ Cyber Security -‐ Insurance: Life, Health, Property, etc. -‐ Cyber-‐Physical Systems (e.g. SCADA, drones,

driverless cars)




Reality Check

15

!  All of this, even before you see the rest of the slides, may seem like an absurdly ambiBous undertaking.

!  But we are proposing this as a mulB-‐year effort, paced to whatever people feel like doing in any given year.

!  Though we are designing it to produce useful intermediate products at intermediate Bmes.

!  In fact, we could set up an orderly system of one or two special sessions at the SRA annual conference each year, to review work for that year and call for work in the coming year.

!  So in fact we are proposing an organizing framework, into which we can fit whatever anyone wants to develop, whenever they want to develop it.

!  Note in parBcular that we seriously doubt we will get contribuBons from domain pracBBoners for many of the 16 domains listed on the previous slide.

!  This is an effort coordinated by the Applied Risk Management Specialty Group, and we want to maintain the focus on risk management decision aiding, but other than that, we want this to be an all-‐SRA document.



Ideas we couldn’t fit into our SecBons 1 of 2

16

We admit it: What we couldn’t fit into SecBons, we list as appendices:

!  A1: Features of a “Good” Analysis to Support Risk Management, vs. a “Poor” One. This could include a user friendly framework/table/query system that a non-‐specialist could use, keyed to his/her domain of applicaBon, to: -‐ check to see if a given analysis is “State of Art,” “accepted,” “unfavored.” -‐ compare one analysis with another one for that domain, with pros and cons. -‐ relate those two things to the complexity of the system of risks involved. -‐ become aware of unmet challenges and future direcBons.

!  A2: A List of Problems and Piealls, Based on Experience, maybe vs. our Challenges

!  A3: MisconcepBons of Risk, perhaps with a summary of Terje Aven’s book.

!  A4: Survey of Failures of Risk Assessment and Risk Governance, with examples illustraBng the importance of our Core Values, Principles and Guidelines.

!  A5: Other Examples Conveying the Importance of our Core Values, Principles.

!  A6: Examples of Issues of Trust and Acceptance.

!  A7: EvoluBon of Risk Assessment Models, and Future DirecBons, for each domain.

!  A8: Bold New Approaches to Risk Assessment and Management.



More Appendices 2 of 2

17

!  A9: Leveraging Work of Other Groups, Tying Our Concepts into a Larger Framework -‐ INCOSE has developed principles of addressing complexity in systems. -‐ Formal “philosophy of systems.” See systemology.org/manifesto.html. -‐ “A Framework for the Next GeneraBon of Risk Science,” Krewski et al. -‐ The EPA Next Gen program. -‐ The IRGC Risk Management Escalator. -‐ Generally connecBng risk management with systems engineering/thinking. -‐ AlternaBve procedures for audiBng for compliance with our Principles, etc.

!  A10: A Catalog of Risk Management Standards. This table + ISO 31000, ISO/IECC 15288, etc.

From INCOSE 2006:



Paths Forward

18

!  Call for responses to the current drao, encouraging scathing criBques.

!  Respond to those responses to generate a second drao.

!  Some of those responses may call for more wriBng/development. Set that up.

! Work with SRA to develop a cooperaBve framework within which to pursue all of what we have discussed here.

!  Appeal to specialists to write Guidelines for each domain of applicaBon.

! More generally, prioriBze / set up to develop the “fill in later” secBons of the doc.

!  Review Risk Analysis for related arBcles, bring our doc into alignment with them, incorporate them, cite them.

!  Review the many related exisBng standards (see our Appendix 10), bring our doc into alignment with them, incorporate them, cite them.

!  Some of us have suggested insigheul relaBonal graphics (mappings, road maps, logical networks). Pursue those.



Issues to Discuss

19

! What should be the process for deciding what goes into the document? Full SRA-‐wide consensus for each decision would take too long. So …?

! We are concerned that we may be seen as infringing on the territories of other SRA Specialty Groups. How do we avoid infringing? By strongly encouraging parBcipaBon by all Specialty Groups.

!  The drao provides a framework within which to address 21 Challenges and 16 ApplicaBon Domains. Our hope is to invite each ApplicaBon Domain to specify Guidelines for its own domain, Guidelines which address the 21 Challenges. That might turn out to be simply unworkable. Any ideas for a beEer way?

!  The doc has evolved into a very byzanBne structure, what with its 13 Goals, 6 Core Values, 10 Principles, 21 Challenges, Guidelines for each of 16 domains of applicaBon and 10 appendices. Is that OK?

!  Do we seek some framework with which to encourage these Principles/Guidelines?

! One idea: Do we seek some organizaBonal framework for review of analyses? For example, an “ARMSG Seal of Approval”?

!  Do we include Post Mortems? Case Studies? Send ideas to: [email protected]

Documents

Applied(Risk(Management Specialty(Group( …c(Supportfor(Prac&cal(Risk(Management Applied(Risk(Management Specialty(Group(Principles, Guidelines and Core Knowledge For Analytic Support