v©2019 SAE International

Introduction xiii

C H A P T E R 1

Fault-Tolerant Ability Testing for Automotive Ethernet 1

Introduction 2

Physical Layer Analysis 3

Fault Tolerance Testing 5Wire Short or Open Testing 5Resistance Testing 6Capacitance Testing 6Ground Shift Testing 7

Result Analysis 7

Summary/Conclusions 10

Contact Information 10

Acknowledgments 11

Definitions/Abbreviations 11

References 11

C H A P T E R 2

An Analysis of ISO 26262: Machine Learning and Safety in Automotive Software 13

Introduction 14

Background 14ISO 26262 14Machine Learning 15

Analysis of ISO 26262 16Identifying Hazards 17Faults and Failure Modes 17

contents

vi Contents

Specification and Verification 18Level of ML Usage 19Required Software Techniques 20

Summary and Conclusion 21

Acknowledgment 22

References 22

C H A P T E R 3

The Development of Safety Cases for an Autonomous Vehicle: A Comparative Study on Different Methods 27

Introduction 28

Vehicle Layout and ISO26262 29Vehicle Control System and Propulsion System 29ISO26262 Road Vehicle Functional Safety Standard 30Failure Model and Effects Analysis Method 31Goal Structuring Notation Method 32Safety Case Development 33Case Study 34

Conclusions 36

Contact Information 37

Acknowledgments 37

Definitions/Abbreviations 37

References 37

C H A P T E R 4

Autonomous Vehicle Sensor Suite Data with Ground Truth Trajectories for Algorithm Development and Evaluation 39

Introduction 40Location 40

Experimental Design 40Autonomous Vehicle Sensors 43

Cameras 43

Contents vii

Radar 43

Lidar 45

Ground Truth Collection - AV Trajectories 45Ground Truth Collection - Pedestrian and Cyclist Tracks 45

Traffic Light Phase Data 46Aerial Observation 46A Note about Coordinate Frames 47

Summary 48

Contact Information 50

Acknowledgments 50

Definitions/Abbreviations 50

References 50

C H A P T E R 5

Integrating STPA into ISO 26262 Process for Requirement Development 53

Introduction 54

Process Map for STPA Integration 55ISO 26262 55STPA 56Process Map for Creating Functional Safety Requirement 56

Modeling and Tool Support 57Intro to SysML 57

Meta-Model for Hazard Analysis & Requirement Generation 58System Engineering Foundations Based on Item Definition 60Integration of STPA Step 1 for Evaluating Existing Safety Goals 61Integration of STPA Step 2 for Creating Functional Safety Requirements 63Consideration for Integration with Cyber Security Analysis 65

Summary/Conclusions 66

Definition/Abbreviations 67

References 67

viii Contents

C H A P T E R 6

Hazard Analysis and Risk Assessment beyond ISO 26262: Management of Complexity via Restructuring of Risk-Generating Process 69

Introduction 70

SOTIF HARA and State Space Explosion 71HARA Composition 71

Hazards 71

Use Cases 71

HARA and the Hidden Semi-Markov Chain 72

Restructuring of Risk-Generating Process 73

Automatic Emergency Braking (AEB) Example 74Markov Chain Solution 74Regions of the Transition Matrix 75Is There Another Way to Do It? 76

Summary 76

Outlook 77

Contact Information 77

Definitions/Abbreviations 77

References 77

C H A P T E R 7

Toward a Framework for Highly Automated Vehicle Safety Validation 79

Introduction 80Approach 80Terminology 81

The Role of Vehicle Test and Simulation 81Beyond ISO 26262 81System Test/Debug/Patch as a Baseline Strategy 82Limitations of Vehicle-Level Testing and Simulation 82Simulation Realism for Its Own Sake Is Inefficient 83

Clarifying the Goals of Testing 83HAV Requirements Will Be Incomplete 84Vehicle Testing for Debugging Can Be Ineffective 84Vehicle Testing as Requirements Discovery 85

Contents ix

Separating Requirements Discovery and Design Testing 86Vehicle Testing to Mitigate Residual Risks 86

A Layered Residual Risk Approach 87Validation According to Safety Requirements 87Basing Validation on Residual Risks 88Managing Residual Risks 88An Example of Residual Risks 89

Improving Observability 90Controllability and Observability 90Software Test Points 91Passing Tests for the Right Reason 91

Coping with Uncertainty 93Knowns and Unknowns 93Dealing with Unknown Defects 93HAV Maturity 94HAV Probation: Monitoring Assumptions 94Deploying with Residual Risks 95

Conclusions 95

Contact Information 96

Definitions/Abbreviations 97

References 97

C H A P T E R 8

Bayesian Test Design for Reliability Assessments of Safety-Relevant Environment Sensors Considering Dependent Failures 101

Introduction 102

Background: Reliability Assessment of Automotive Environment Perception 103

Null Hypothesis Significance Testing for Sensor Reliability Assessment 104

Performance Evaluation of NHST 105

Alternatives to NHST for Reliability Assessments 106

Bayesian Methodology for Empirical Perception Reliability Assessments of Environment Sensors 106

Statistical Model 106Mathematical Representation of Dependent Errors 107

x Contents

Considering a Non-Stationary Error Rate 109Bayesian Reliability Assessment and Test Effort Estimation 110Assessing the Reliability of a Multi-Sensor System 112

Case study: Empirically Demonstrating the Perception Reliability of Environment Sensors 114

Estimating the Necessary Test Drive Effort 115Evaluating Hypothetical Test Results 116Influence of Error Dependence on Multi-Sensor Based Machine Vision 116

Discussion 118

Conclusions 120

Contact Information 120

References 120

Appendix 123

C H A P T E R 9

Challenges in Autonomous Vehicle Testing and Validation 125

Introduction 126Infeasibility of Complete Testing 126The V Model as a Starting Point 127

Driver Out of the Loop 127Controllability Challenges 128Autonomy Architecture Approaches 128

Complex Requirements 129Requirements Challenges 130Operational Concept Approaches 130Safety Requirements and Invariants 131

Non-Deterministic and Statistical Algorithms 132Challenges of Stochastic Systems 132Non-Determinism in Testing 133

Machine Learning Systems 134Challenges of Validating Inductive Learning 134Solutions to Inductive Learning 135

Mission Critical Operational Requirements 136Challenges of Fail-Operational System Design 136Failover Missions 137

Contents xi

Non-Technical Factors 137

Fault Injection 138

Conclusions 138Phased Deployment 139Monitor/Actuator Architecture 139Fault Injection 139Future Work 140

Contact Information 140

Definitions/Abbreviations 140

References 140

C H A P T E R 1 0

RV-ECU: Maximum Assurance In-Vehicle Safety Monitoring 143

Introduction 144Limitations of Current Approaches 144Enabling Safety Standardization 145

Runtime Verification 145

RV-ECU: A Vehicle Safety Architecture 147Global and Local Monitoring 148

Certifiable Correctness 150

RV-ECU Compared: Other RV Efforts 151

Recalls and RV-ECU, a Case Study 152

A Practical Demonstration 154

Future Work and Applications 156

Technical Limitations and Drawbacks 157

Conclusion 158

Acknowledgements 158

References 159

Epilogue 163