Embed Size (px)
Citation preview
RADAR and Audit Analytics
Miklos A. Vasarhelyi
KPMG Distinguished Professor of AIS
Rutgers Business School
Florianopolis
4/75
Outline
1. The CarLab
2. The audit process and innovation
3. Full population testing (MADS)
4. Process mining
5. Visualization
THE CARLAB (CONTINUOUS
AUDIT AND REPORTING
LABORATORY)
RUTGERS BUSINESS SCHOOL
6
7
BRIGHAM YOUNG
UNIVERSITY
The Ranking of Rutgers in the Accounting Areas
Areas Ranking 2008-2013 Ranking 2002-2013
Ranking
1990-
2013
AIS #1 out of 179 #1 out of 207
#1 out
of 241
Audit #6 out of 320 #7 out of 370
#11 out
of 438
Financial #70 out of 356 #89 out of 406
#83 out
of 470
Managerial #120 out of 286 #80 out of 346
#66 out
of 413
Tax #53 out of 129 #76 out of 178
#79 out
of 246
Other #35 out of 171 #18 out of 248
#25 out
of 341
Usage
http://raw.rutgers.edu/RADL.html
1. THE AUDIT PROCESS AND
INNOVATION
Audit Data Analytics
Bob Dohrer, IAASB Member and Working Group Chair
Miklos Vasarhelyi
Phillip McCollough
IAASB Meeting
September 2015
Agenda Item 6-A
Risk
Assessment
Data Analytics
Audit Data Analytics – Technique, not a Tool
Internal
Control
Evaluation
Substantive
Analytical
Procedures
Substantive
Procedures
(tests of
detail)
Audit Data Analytics
Traditional Stages of the Audit
• Procedure: For every invoice, shipping document and sales order
received from customers, compare the invoiced customer, quantity,
and unit price to the quantity shipped per the shipping documents and
the quantity and unit price reflected in the sales order received from the
customer.
• Objective: Obtaining audit evidence over the existence and accuracy of
revenue. (ISA 500 paragraphs 6 and 9).
• Prior year approach: Tests of internal controls over the revenue
process, substantive analytical procedures and tests of details
(sampling).
Revenue- Three Way Match
Data Analytics
Entity ABC has revenue of €125 million generated by 725,000 transactions. The
three way match procedure is executed with the following results:
Note: Materiality for the audit of the financial statements as a whole is €1,000,000.
Illustration 1 – Revenue Three Way Match (cont.)
Amount
(€‘000) %
Number of
Transactio
ns
%
No differences 119,750 95.8 691,000 95.
3
Outliers:
Quantity
differences
3,125 2.5 16,700 2.3
Pricing
differences
2,125 1.7 17,300 2.4
19
PUBLIC AUDITING, ANALYTICS, AND BIG
DATA IN THE MODERN ECONOMY
Dissertation Defense by Deniz Appelbaum
Wednesday April 5, 2017
Committee Chair: Dr. Miklos A. Vasarhelyi
Evolution of the External Audit Analytics
Framework
Big Data• Big Data has become the new business currency
• 4 V’s of Big Data: Variety, Volume, and Velocity create a bigger issue
of VERACITY (quality & provenance)
• Gartner: In 2015, more than $2.5 Billion paid by U.S. businesses
for AML violations due to incomplete and inaccurate data
• Businesses using Big Data for decision making
• External Auditors:
– May need to verify or re-perform client applications of Big Data
– May want to access Big Data for industry and client assessment,
risk analysis, confirmations, reasonableness tests
• Internal Auditors:
– Access Big Data to perform efficiency and fraud examinations
– Verify the validity of the data underlying firm decisions
– Similar interests as the External Auditor
– More exposure to Big Data
BUT…… 22
Big Data Provenance! The elephant in the room
23
The audit standards specify that external
sources of evidence and information are
generally more reliable for verification.
Big Data potentially poses the opposite
situation!
Due to its possible lack of provenance
and veracity, it could be a
LESS RELIABLE source of
evidence for auditors!
Issue #2: Can the
Provenance of Big Data be
regarded as sufficient audit
evidence?
24
3. FULL POPULATION
TESTING (MADS)
Rutgers and AICPA Data Analytics Research Initiative
May 15, 2018
Traditional sampling
approach
New approach
• BUT, often generate large numbers of outliers.
• Impractical for auditors to investigate entire outliers
Advance in data processing ability & data
analytic techniques allows auditors to
evaluate the entire population instead of
examining just a chosen sample.
• Crucial to develop a method that can help auditors
effectively deal with large amounts of data, but also assist
them to efficiently handle a massive number of outliers.
Multidimensional Audit Data Selection (MADS) Analytic Framework
❖ To assist auditors identifying questionable transactions/data in performing substantive test of details
▪ Developed based on prior literature and professional guidelines.
▪ Modified based on comments from several panel discussions of scholars and auditing professionals.
▪ Consist of six components.
❖ The practice of these six components is guided by the overall objectives of audit, specifically audit risk
and materiality.
Data Collection
and
Understanding
Overall Audit Objectives (Risk & Materiality)
Objectives and Criteria Identification
Data
Preparation
MADS Model
Building
• Additional Filters
• Visualization Techniques
(e.g., scatter plots)
• Professional Judgement
(e.g., knowledge and experiences)
• Outlier Detection Techniques
(e.g., classification & clustering).
MADS Model Build Process
Whole Transaction Data
(Entire Population)
Step 1:Filters for Significant Potential
Risk Factors
Step 1 Outputs
Step 2:Data Analytic Techniques
Apply a set of filters to examine significant
risks (i.e., What Could Go Wrong)
(e.g., duplicate payment)
• Use professional judgement based
on the importance of each step 1
filter and step 2 filter.
• Use the step 1 and/or step 2 results.
• Use a reasonable factor
(e.g., dollar amount).
MADS Model Build Process
Whole Transaction Data
(Entire Population)
Step 1:Filters for Significant Potential
Risk Factors
Step 1 Outputs
Step 2:Data Analytic Techniques
Step 2 Outputs
Step 3:Prioritization
Prioritized
Notable Items
• Additional Filters
• Visualization Techniques
(e.g., scatter plots)
• Professional Judgement
(e.g., knowledge and experiences)
• Outlier Detection Techniques
(e.g., classification & clustering).
Apply a set of filters to examine significant
risks (i.e., What Could Go Wrong)
(e.g., duplicate payment)
Jimmy Chin
Skiing down the
Everest
5. PROCESS MINING
Tiffany Chiu, Abdulrahman Alrefai
and Miklos A. Vasarhelyi
Evaluating the Effectiveness of Internal
Control using Process Mining
What is Process Mining of Event Logs?
• Process mining technique refers to using event log to analyze
business process.
– Event Log is defined as “a chronological record of computer
systems activities which are saved to a file on the system. The file
can be reviewed by the system administrator to identify users’
actions on the system or processes which occurred on the system”
(FAS: Federation of American Scientists)
Characteristics of Event
(1) Activity The activity taking place during the event, e.g. sign
(2) Process Instance The process instance of the event, e.g. invoice
(3) Originator The originator, or party responsible for the event, e.g.
name of the action owner
(4) Timestamp The timestamp of the event,
e.g. date/time of the event (2006-11-07T10:00:36)
Dataset Overview
Event 181,845
Process Instance 26,185
Activity 7
Activity Detail (1) Create PO
(2) Sign
(3) Release
(4) GR
(5) IR
(6) Pay
(7) Change Line
Variant 980
Mean Process Instance
Duration
46.2 Days
Start 01/02/2007
End 01/25/2008
Essay Two
38
Actual
Process
Chart
40
Applying Process Mining to Evaluate the Effectiveness of Internal Control
Advantages of Process Mining
1.Gaining detailed and objective
information on the business
process
2.Obtaining high levels of
assurance
3. Gathering strong evidence
Computerization of Occupations
6/11/2019Public Sector Accounting & Data Analytics-Rio de Janeiro
42
Adapted from: “The Future of Employment: How Susceptible are Jobs to Computerisation?” (Frey and Osborne, 2013)
45
Data Visualization in Auditing
Qi Liu, Heejae Lee, and Lu Zhang
What is data visualization• Visual Representation of Data
– Tabular displays are most useful for looking up the precise value of a specific data item.
– Graphs are more useful for examining relative magnitudes, relationships, and anomalies.
• For exploration, discovery, insight, ….
• Interactive component provides more insight as compared to a static image.
They allow the user to:
– Select specific information
– Customize the visualization format
Types of data visualization
• Information Visualization
--(Interactive) visual representations of abstract data to
reinforce human cognition.
-- News, stock market, top grossing movies, facebook
connections
• Visual Analytics
--The science of analytical reasoning facilitated by interactive
visual interfaces.
-- audio, video, text, images, networks of people ..
• Scientific Visualization --Graphically illustrate scientific
data to enable scientists to
understand, illustrate, and
glean insight from their data.
-- Architectural, Meteorological,
biological, Medical, ..
Visual analytics process
Source: Keim et al., 2010
How to apply visualization to auditing
Understand the objectives of audit
tasks
• Comparison
• Distribution
• Composition
• Relationship
• Prediction
Understand characteristics of the data
• Data type of attributes
• Size: Do we need to select a
sample?/Do we need to
aggregate some information?
Do we need to filter out some
information?
• Scale: Do we need to
normalize/standardize the
data?
Choose appropriate visualization
• Static visualization or
Interactive visualization？• Are there any emerging
visualization techniques
that are suitable for this
task?
• What kind of graph to use?
(Next Slide)
Evaluation (Optional)
• How is the
effectiveness of the
chosen graph in terms
of user performance
and computer
performance?
• How is the users’
experience with the
chosen graph?
Use visualization to assess financial statement
risk – Walmart example Various graphs can be used to explore different risks. For examples, we can
• Use bar chart to visualize horizontal analysis on Balance Sheet items.
• Use scatter plot to check the relationship between sales and expenses.
• Use line chart to identify quarterly trend of sales.
Use visualization to assess financial statement
risk – Dynamic visualization• We can also create a
dynamic visualization to
show the change of
selected financial
statement items overtime.
• This video shows the
change of Walmart’s
sales, net income, and
cash balance from 2003 to
2015. It also displays all
the other retail companies
to assist us in comparing
Walmart and its commonly
recognized competitors,
such as Target and
Amazon.
Use visualization to assess financial statement risk
– Interactive dashboard• In last webinar, we clustered Walmart’s peers
based on four ratios. • Quick ratio (QUICK) =
(Current assets - Inventory) / Current
Liabilities
• Leverage (LEV) =
Total Debt / Total Assets
• Inventory Turnover (INV) =
Cost of Goods Sold / Average Inventory
• Gross Margin (GM) =
(Net Sales – Cost of Goods Sold) / Net Sales
• By using interactive dashboard to visualize
Walmart and its peer companies’ ratios, we can
easily see how Walmart’s ratios change
overtime and how it compares with its peers in
terms of each ratio. This can help us quickly
allocate the major risk area in Walmart.
Use visualization to facilitate substantive testing –
Interactive 3D visualization
• We have done a 3-way match using
Walmart’s order, shipping and invoice
data.
• Visualizing the order, shipping, and
invoice amount using interactive 3D
scatterplot can discover their
relationships, therefore identify the
problematic accounts.
Use visualization to facilitate substantive testing
– Correlation Analysis
• When we have
more than three
variables,
correlation analysis
can visualize the
relationships
between them.
• This graph
demonstrates the
relationships
between different
types of quantities
and unit costs. It
also shows their
relationships
between the
difference of invoice
and order amount.
Conclusion
• Visualization, especially interactive visualization, can be a powerful tool for auditors to perform various of audit tasks such as risk assessment and substantive testing.
• Understand your objectives and the dataset is the first step to perform visual analytics.
• Well designed graphs are essential for auditors to accurately identify notable items and make unbiased decisions. The 10 points listed on the right can provide you some basic guidance on how to design your visuals.
Source: Data Visualization 101: How to Design Charts and
Graphs
Artificial Intelligence prospective in audit
Helen Brown-Liburd
Ivy Munoko
Miklos Vasarhelyi
Rutgers Business School
Current implementations: All Big 4 accounting
firms are reporting the use of these AI blocks
61
Machine Learning
• Classifier ( tax transaction, contract)
• Fraud detection
• Review full population for outliers
Big AI / Smart
Analytics
• Large scale data analysis
• discover facts and relations that are difficult for the human mind
Natural Language Processing
• Synthesis of text
e.g. review of contracts, vendor invoices, emails, transcribed conversation
Machine Vision
• OCR + Machine learning to extract data from images
• Drones + IoT to perform inventory inspection
Intelligent RPA
• Test of transactions
• Document workpapers
• GL review
• Bank confirmation
Speech
Recognition
• Decode conversation
• Chat bots
• Digital Assistants
Current implementations: Small / medium size
accounting firms are using AI SaaS
62
Ensemble AI:
Combines Machine
learning with Domain
expertise / business
rules and statistical
methods to:
-gain actionable
insight into large data
-review 100% of data
-sample from outliers
-perform risk
assessments
Current Features and Applications of AI
63
64
Group 1: Assisted AI: Support lower level decisions
Examples: Chatbots
Group 2: Augmented AI: Support high risk decisions
Examples: Performing audit risk assessments, Fraud detection,
Going concern evaluations
Group 3: Autonomous AI: Assumes decision making
Examples: Expense compliance
Current Applications of AI - Assisted, Augmented
and Autonomous
As AI shifts
from assisted
towards
autonomous,
the tool
sophistication,
resultant
benefits as
well as the
risks increase
CONCLUSIONS
65
A conceptual framework of an audit plan
cognitive assistant
Rutgers, the State University of New Jersey
Qiao LiMiklos A. Vasarhelyi
QA
Proposed Audit Cognitive Assistant Luca
Automatic
Speech
Recognition
Query
Classifier
Question
or Action
text
Answer Show
Answer
Execute
Action
Action
Lucaindustry
Client
Position
Luca Luca
LucaRecommended Topics:
General understanding,
new events, business
risks…
Processing…
You may also
interested in:…
Query
Open an
application
Inte
rface
Arc
hitectu
re
Modules:
• Automatic Speech
Recognition (ASR)
• Language Understanding
• Dialogue Management
• Natural Language
Generation
• Text-to-Speech synthesis
Audit Related Applications It Can Access
Web
Search
Open
(ACL,
IDEA…)
Calculato
r
Open
standards
Open
templatesAudit work
paper
Calendar …….backsta
ge
s
upport
er
Knowledge Database
Knowledge
about users
Knowle
dge
Base
DBMS
Unstructu
red data
Domain
Knowledge
Ap
pCapability
1 Search engine such as google
2 ACL, IDEA
3 Ratios; others like Benford’s
law
4 Regulations related to the
audit area or account
5 Required procedures,
guidance or programs
6 Open prior audit
7 Audit plan schedule
Domain: judgement or experience
Unstructured: financial
statements, accounting policies,
analytical procedures, litigation,
claims, recent news information,
audits workpapers, prior year audit
deficiencies and adjustments…
User interaction: queries and
search
4/13
SUSTAINI
NG
POTENTIA
L
DISRUPTI
VE
Emerging Technology
Emerging Technology Sustaining Disruptive
Blockchain and Smart Contract
√
Continuous Auditing
√
Interactive Data Visualization √ √
Linguistic Analysis
√
Machine Learning and Big Data Analysis √ √
Process Mining
√
Robotic Process Automation (RPA) √
Unmanned Aircraft Systems (Drones) √
DISRUPTIVE
TECHNOLO
GIES
INDEPENDENT PROCESS
CLIENT FIRM USAGE
ADVISORY AUDIT
MARKET
MAINSTREAM
AUDITING
PRACTICE
AUDITOR TECHNOLOGY
ACCEPTANCE
(MODIFIED UTAUT)PERFORMA
NCE
EXPECTANC
YSOCIAL
INFLUENCE
EFFORT
EXPECTANC
YFACILITATIN
G
CONDITION
SPRICE
VALUE
INDIVIDUAL
DIFFERENC
E
REGULATI
ON
SUSTAINING
TECHNOLOGIE
S
AUDIT VALUE NETWORK
Big Data Based
Government Economic Monitoring (GEM)
and Targeted Action
Miklos A. VasarhelyiKPMG Distinguished Professor of Accounting Information Systems
and Director of Rutgers Accounting Research Center and
Continuous Auditing & Reporting Lab (CAR Lab)
Arion Cheong
Xinxin Wang
GEM Overview
• The US Government 6 major social welfare programs
– Temporary Assistance for Needy Families program
– Medicaid
– Child's Health Insurance Program
– Food Stamps
– Supplemental Security Program
– Earned Income Tax Credit, and Housing Assistant program
• Using modern information technology to directly guide
government provide target actions.
• Big Data based GEM will enable the close-to-timely-reality
tracking of microeconomic status in real-time.
• Surveillance capitalism VS Your Big HUG government
• Transparency of every tax dollar.
Telephone
Drug Usage
Energy Usage
Government Agency
Online Retailers
Police Report
Supermarkets
Transportations
Stock Market
Darkweb
Exogenous data Collection Tree
15%
Governmental Institutions Blockchain
A wants to
vote
B need to fill her tax
return
C is a senior citizen who
needs to receive social
welfare payment
Government Blockchain
Privacy-preserved Blockchain Database
Privacy Data
BlockDB
by
CARLab
Untraceable/ Secured
Form
• Every Privacy data are securely stored in a Blockchain Database BlockDB
• BlockDB preserves privacy by transforming all the data in an untraceable format
- Name, SSN, etc. all be replaced to an index where outsiders would never be able to
trace back to individuals
- Blocks (data) are only retrievable for analysis
15%
Security
and
Protect
Privacy
Processed Information
Social Alert
Family/Individual Index
PatternIdentification
Traditional
Field CaseIdentification
Supervised DataLearning
Privacy-preserved Blockchain Database
UniqueIdentification
GEM Dynamic Process
Case – Food Stamp
• Better allocation of Food Stamp (Efficient Budget Allocation)
• Timely Feedback for policy adoption
79
