Upload
letruc
View
225
Download
2
Embed Size (px)
Citation preview
introduction
Ensuring Continuous Monitoring Relevance
©2015 Ramaley Group, LLC
Using Control Charts
Ken Ramaley, CIA, CRMA Managing Director
Ramaley Group [email protected]
What is this all about?
©2015 Ramaley Group, LLC
• Continuous Monitoring (CM) programs have tremendous value to auditors • Early identification of potential control failures
• Dynamic re-allocation of resources • “Automated issues”
CM are High Potential Programs – But Only Relevant if they enable effective inferences
How to ensure Relevance?
©2015 Ramaley Group, LLC
Statistically-based continuous monitoring enables appropriate response to out-of-control conditions
• Observed control failures may not mean anything. Continuous monitoring is like performing an extended hypothesis test
• Plan to react when you see “signal” vs. “noise”
What does audit do next?
You have implemented a real-time continuous monitoring program. Last night’s sample came in this morning – yielding results on 100 units, divided equally between two sites. The control is expected to be functioning with 90% quality. Site A shows up with 10 exceptions (of the 50 tested). Site B has only 2 exceptions. Is the control functioning acceptably? Are your concerns limited to Site A, or are they across the board?
©2015 Ramaley Group, LLC
What does audit do next?
A regulatory agency reports that it has received double the volume of complaints about your firm this month that it received last month. How do you respond? Is this a problem?
©2015 Ramaley Group, LLC
In order to conclude on the relevance of an observation, you must understand context and statistical likelihood.
Testing for Significance
Statistical data allows you to perform tests of significance for exceptions Fundamentally: How unusual is the observation?
Solution: Hypothesis Testing Based on forming a “Null Hypothesis” – that the population has a particular property (e.g. mean of a certain value, etc.) - written as H0
Likelihood of what you have observed will allow you to reject H0 or NOT reject H0
©2015 Ramaley Group, LLC
Control Charts
A control chart is a visual display which focuses attention on process performance over time Key Features: • Center line – average performance • Data in time order • Upper/Lower Control Limits displayed (optionally also display Zones A,B,C at 1s,2s) • For variable data, top chart monitors
average (location), bottom chart monitors range (grouping)
©2015 Ramaley Group, LLC
Control Chart Example
©2015 Ramaley Group, LLC
Mean Perf.
Time
UCL
LCL
Spec Limits and Control Limits
Specification Limits Determined by customers of the process
(requirements!) Changed based on customer demands
Control Limits
Calculated based on historical process performance
Only changed when process has experienced a special-cause change – and new process has been in place for at least 20 data points
©2015 Ramaley Group, LLC
Control Charts as Hypothesis Tests
©2015 Ramaley Group, LLC
By mapping your CM data in a Control Chart, you can identify critical factors in ensuring good audit conclusions: Is the process capable of delivering to spec limits?
If capable, is the process continuing to function “in control”?
“Out of control” = “Would fail a hypothesis test”
Standard rules of thumb can help
Western Electric Rules for Interpreting Control Charts*
*Adapted from Western Electric Company (1956), Statistical Quality Control handbook. (1 ed.), Indianapolis, Indiana: Western Electric Co.
Limits of Simple Control Chart Rules
Rules exist as “guidance” not “gospel” for finding out-of-control signals
Judgment must be incorporated into control chart assessment
False Positives may occur (approx 1% due to chance if executing all 4 rules)
Control charts must be accompanied by control plans
A control chart is only as good as its underlying data ©2015 Ramaley Group, LLC
Additional Control Chart Considerations
Autocorrelation may be a factor – the data collection period should be long enough that each data point is truly independent (can test using a time series regression)
Control Limits should only be re-calculated when the process has experienced a known external change and some new data has been collected
Context for control chart data must be provided by people who are engaged in the process
©2015 Ramaley Group, LLC
±
How Do I Build a Control Chart?
1) Begin with a data set. For our example, lets assume it is in small batches.
2) Take the mean of each batch of data you have collected, then take the mean of the means (m).
©2015 Ramaley Group, LLC
DISCLAIMER: There are many different kinds of control charts – it may take assistance to determine the correct one for your situation. This is a simple illustrative example.
How Do I Build a Control Chart?
3) Calculate the standard deviation (s) of your data points
4) Define your Upper Control Limit (UCL) as m+3*s and your Lower Control Limit (LCL) as m-3*s
5) Draw your control chart with the time series data, drawing in the mean line, UCL, LCL, and lines showing m±s, m±2s
©2015 Ramaley Group, LLC
Practical Control Chart Implementation
©2015 Ramaley Group, LLC
Standard statistical functions can be executed through tools you already have: Excel SQL Minitab JMP
Practical Control Chart Implementation
©2015 Ramaley Group, LLC
As you establish your CM program, the outputs should populate control charts and automatically execute tests – triggering alerts based on Control Chart diagnostics!
Relevance flows from meaningful CM alerts. False alarms will cause results to be ignored. Integrate Control Chart diagnostics into your CM program to hit next-level audit efficiency and effectiveness
Control Chart ACTION
When your control chart suggests a concern: EXECUTE YOUR ACTION PLAN!!!
©2015 Ramaley Group, LLC
Consider responses to out-of-control conditions in advance of experiencing them. This drives audit consistency and relevance.
Questions?
©2015 Ramaley Group, LLC