RADAR and Audit Analytics - Minas...

RADAR and Audit Analytics

Miklos A. Vasarhelyi

KPMG Distinguished Professor of AIS

Rutgers Business School

Florianopolis

4/75

Outline

1. The CarLab

2. The audit process and innovation

3. Full population testing (MADS)

4. Process mining

5. Visualization

THE CARLAB (CONTINUOUS

AUDIT AND REPORTING

LABORATORY)

RUTGERS BUSINESS SCHOOL

6

7

BRIGHAM YOUNG

UNIVERSITY

The Ranking of Rutgers in the Accounting Areas

Areas Ranking 2008-2013 Ranking 2002-2013

Ranking

1990-

2013

AIS #1 out of 179 #1 out of 207

#1 out

of 241

Audit #6 out of 320 #7 out of 370

#11 out

of 438

Financial #70 out of 356 #89 out of 406

#83 out

of 470

Managerial #120 out of 286 #80 out of 346

#66 out

of 413

Tax #53 out of 129 #76 out of 178

#79 out

of 246

Other #35 out of 171 #18 out of 248

#25 out

of 341

Usage

http://raw.rutgers.edu/RADL.html

1. THE AUDIT PROCESS AND

INNOVATION

Audit Data Analytics

Bob Dohrer, IAASB Member and Working Group Chair

Miklos Vasarhelyi

Phillip McCollough

IAASB Meeting

September 2015

Agenda Item 6-A

Risk

Assessment

Data Analytics

Audit Data Analytics – Technique, not a Tool

Internal

Control

Evaluation

Substantive

Analytical

Procedures

Substantive

Procedures

(tests of

detail)

Audit Data Analytics

Traditional Stages of the Audit

• Procedure: For every invoice, shipping document and sales order

received from customers, compare the invoiced customer, quantity,

and unit price to the quantity shipped per the shipping documents and

the quantity and unit price reflected in the sales order received from the

customer.

• Objective: Obtaining audit evidence over the existence and accuracy of

revenue. (ISA 500 paragraphs 6 and 9).

• Prior year approach: Tests of internal controls over the revenue

process, substantive analytical procedures and tests of details

(sampling).

Revenue- Three Way Match

Data Analytics

Entity ABC has revenue of €125 million generated by 725,000 transactions. The

three way match procedure is executed with the following results:

Note: Materiality for the audit of the financial statements as a whole is €1,000,000.

Illustration 1 – Revenue Three Way Match (cont.)

Amount

(€‘000) %

Number of

Transactio

ns

%

No differences 119,750 95.8 691,000 95.

3

Outliers:

Quantity

differences

3,125 2.5 16,700 2.3

Pricing

differences

2,125 1.7 17,300 2.4

19

PUBLIC AUDITING, ANALYTICS, AND BIG

DATA IN THE MODERN ECONOMY

Dissertation Defense by Deniz Appelbaum

Wednesday April 5, 2017

Committee Chair: Dr. Miklos A. Vasarhelyi

Evolution of the External Audit Analytics

Framework

Big Data• Big Data has become the new business currency

• 4 V’s of Big Data: Variety, Volume, and Velocity create a bigger issue

of VERACITY (quality & provenance)

• Gartner: In 2015, more than $2.5 Billion paid by U.S. businesses

for AML violations due to incomplete and inaccurate data

• Businesses using Big Data for decision making

• External Auditors:

– May need to verify or re-perform client applications of Big Data

– May want to access Big Data for industry and client assessment,

risk analysis, confirmations, reasonableness tests

• Internal Auditors:

– Access Big Data to perform efficiency and fraud examinations

– Verify the validity of the data underlying firm decisions

– Similar interests as the External Auditor

– More exposure to Big Data

BUT…… 22

Big Data Provenance! The elephant in the room

23

The audit standards specify that external

sources of evidence and information are

generally more reliable for verification.

Big Data potentially poses the opposite

situation!

Due to its possible lack of provenance

and veracity, it could be a

LESS RELIABLE source of

evidence for auditors!

Issue #2: Can the

Provenance of Big Data be

regarded as sufficient audit

evidence?

24

3. FULL POPULATION

TESTING (MADS)

Rutgers and AICPA Data Analytics Research Initiative

May 15, 2018

Traditional sampling

approach

New approach

• BUT, often generate large numbers of outliers.

• Impractical for auditors to investigate entire outliers

Advance in data processing ability & data

analytic techniques allows auditors to

evaluate the entire population instead of

examining just a chosen sample.

• Crucial to develop a method that can help auditors

effectively deal with large amounts of data, but also assist

them to efficiently handle a massive number of outliers.

Multidimensional Audit Data Selection (MADS) Analytic Framework

❖ To assist auditors identifying questionable transactions/data in performing substantive test of details

▪ Developed based on prior literature and professional guidelines.

▪ Modified based on comments from several panel discussions of scholars and auditing professionals.

▪ Consist of six components.

❖ The practice of these six components is guided by the overall objectives of audit, specifically audit risk

and materiality.

Data Collection

and

Understanding

Overall Audit Objectives (Risk & Materiality)

Objectives and Criteria Identification

Data

Preparation

MADS Model

Building

• Additional Filters

• Visualization Techniques

(e.g., scatter plots)

• Professional Judgement

(e.g., knowledge and experiences)

• Outlier Detection Techniques

(e.g., classification & clustering).

MADS Model Build Process

Whole Transaction Data

(Entire Population)

Step 1:Filters for Significant Potential

Risk Factors

Step 1 Outputs

Step 2:Data Analytic Techniques

Apply a set of filters to examine significant

risks (i.e., What Could Go Wrong)

(e.g., duplicate payment)

• Use professional judgement based

on the importance of each step 1

filter and step 2 filter.

• Use the step 1 and/or step 2 results.

• Use a reasonable factor

(e.g., dollar amount).

MADS Model Build Process

Whole Transaction Data

(Entire Population)

Step 1:Filters for Significant Potential

Risk Factors

Step 1 Outputs

Step 2:Data Analytic Techniques

Step 2 Outputs

Step 3:Prioritization

Prioritized

Notable Items

• Additional Filters

• Visualization Techniques

(e.g., scatter plots)

• Professional Judgement

(e.g., knowledge and experiences)

• Outlier Detection Techniques

(e.g., classification & clustering).

Apply a set of filters to examine significant

risks (i.e., What Could Go Wrong)

(e.g., duplicate payment)

Jimmy Chin

Skiing down the

Everest

5. PROCESS MINING

Tiffany Chiu, Abdulrahman Alrefai

and Miklos A. Vasarhelyi

Evaluating the Effectiveness of Internal

Control using Process Mining

What is Process Mining of Event Logs?

• Process mining technique refers to using event log to analyze

business process.

– Event Log is defined as “a chronological record of computer

systems activities which are saved to a file on the system. The file

can be reviewed by the system administrator to identify users’

actions on the system or processes which occurred on the system”

(FAS: Federation of American Scientists)

Characteristics of Event

(1) Activity The activity taking place during the event, e.g. sign

(2) Process Instance The process instance of the event, e.g. invoice

(3) Originator The originator, or party responsible for the event, e.g.

name of the action owner

(4) Timestamp The timestamp of the event,

e.g. date/time of the event (2006-11-07T10:00:36)

Dataset Overview

Event 181,845

Process Instance 26,185

Activity 7

Activity Detail (1) Create PO

(2) Sign

(3) Release

(4) GR

(5) IR

(6) Pay

(7) Change Line

Variant 980

Mean Process Instance

Duration

46.2 Days

Start 01/02/2007

End 01/25/2008

Essay Two

38

Actual

Process

Chart

40

Applying Process Mining to Evaluate the Effectiveness of Internal Control

Advantages of Process Mining

1.Gaining detailed and objective

information on the business

process

2.Obtaining high levels of

assurance

3. Gathering strong evidence

Computerization of Occupations

6/11/2019Public Sector Accounting & Data Analytics-Rio de Janeiro

42

Adapted from: “The Future of Employment: How Susceptible are Jobs to Computerisation?” (Frey and Osborne, 2013)

45

Data Visualization in Auditing

Qi Liu, Heejae Lee, and Lu Zhang

What is data visualization• Visual Representation of Data

– Tabular displays are most useful for looking up the precise value of a specific data item.

– Graphs are more useful for examining relative magnitudes, relationships, and anomalies.

• For exploration, discovery, insight, ….

• Interactive component provides more insight as compared to a static image.

They allow the user to:

– Select specific information

– Customize the visualization format

Types of data visualization

• Information Visualization

--(Interactive) visual representations of abstract data to

reinforce human cognition.

-- News, stock market, top grossing movies, facebook

connections

• Visual Analytics

--The science of analytical reasoning facilitated by interactive

visual interfaces.

-- audio, video, text, images, networks of people ..

• Scientific Visualization --Graphically illustrate scientific

data to enable scientists to

understand, illustrate, and

glean insight from their data.

-- Architectural, Meteorological,

biological, Medical, ..

Visual analytics process

Source: Keim et al., 2010

How to apply visualization to auditing

Understand the objectives of audit

tasks

• Comparison

• Distribution

• Composition

• Relationship

• Prediction

Understand characteristics of the data

• Data type of attributes

• Size: Do we need to select a

sample?/Do we need to

aggregate some information?

Do we need to filter out some

information?

• Scale: Do we need to

normalize/standardize the

data?

Choose appropriate visualization

• Static visualization or

Interactive visualization？• Are there any emerging

visualization techniques

that are suitable for this

task?

• What kind of graph to use?

(Next Slide)

Evaluation (Optional)

• How is the

effectiveness of the

chosen graph in terms

of user performance

and computer

performance?

• How is the users’

experience with the

chosen graph?

Source: The Extreme Presentation

Method

Use visualization to assess financial statement

risk – Walmart example Various graphs can be used to explore different risks. For examples, we can

• Use bar chart to visualize horizontal analysis on Balance Sheet items.

• Use scatter plot to check the relationship between sales and expenses.

• Use line chart to identify quarterly trend of sales.

Use visualization to assess financial statement

risk – Dynamic visualization• We can also create a

dynamic visualization to

show the change of

selected financial

statement items overtime.

• This video shows the

change of Walmart’s

sales, net income, and

cash balance from 2003 to

2015. It also displays all

the other retail companies

to assist us in comparing

Walmart and its commonly

recognized competitors,

such as Target and

Amazon.

Use visualization to assess financial statement risk

– Interactive dashboard• In last webinar, we clustered Walmart’s peers

based on four ratios. • Quick ratio (QUICK) =

(Current assets - Inventory) / Current

Liabilities

• Leverage (LEV) =

Total Debt / Total Assets

• Inventory Turnover (INV) =

Cost of Goods Sold / Average Inventory

• Gross Margin (GM) =

(Net Sales – Cost of Goods Sold) / Net Sales

• By using interactive dashboard to visualize

Walmart and its peer companies’ ratios, we can

easily see how Walmart’s ratios change

overtime and how it compares with its peers in

terms of each ratio. This can help us quickly

allocate the major risk area in Walmart.

Use visualization to facilitate substantive testing –

Interactive 3D visualization

• We have done a 3-way match using

Walmart’s order, shipping and invoice

data.

• Visualizing the order, shipping, and

invoice amount using interactive 3D

scatterplot can discover their

relationships, therefore identify the

problematic accounts.

Use visualization to facilitate substantive testing

– Correlation Analysis

• When we have

more than three

variables,

correlation analysis

can visualize the

relationships

between them.

• This graph

demonstrates the

relationships

between different

types of quantities

and unit costs. It

also shows their

relationships

between the

difference of invoice

and order amount.

Conclusion

• Visualization, especially interactive visualization, can be a powerful tool for auditors to perform various of audit tasks such as risk assessment and substantive testing.

• Understand your objectives and the dataset is the first step to perform visual analytics.

• Well designed graphs are essential for auditors to accurately identify notable items and make unbiased decisions. The 10 points listed on the right can provide you some basic guidance on how to design your visuals.

Source: Data Visualization 101: How to Design Charts and

Graphs

Artificial Intelligence prospective in audit

Helen Brown-Liburd

Ivy Munoko

Miklos Vasarhelyi

Rutgers Business School

Current implementations: All Big 4 accounting

firms are reporting the use of these AI blocks

61

Machine Learning

• Classifier ( tax transaction, contract)

• Fraud detection

• Review full population for outliers

Big AI / Smart

Analytics

• Large scale data analysis

• discover facts and relations that are difficult for the human mind

Natural Language Processing

• Synthesis of text

e.g. review of contracts, vendor invoices, emails, transcribed conversation

Machine Vision

• OCR + Machine learning to extract data from images

• Drones + IoT to perform inventory inspection

Intelligent RPA

• Test of transactions

• Document workpapers

• GL review

• Bank confirmation

Speech

Recognition

• Decode conversation

• Chat bots

• Digital Assistants

Current implementations: Small / medium size

accounting firms are using AI SaaS

62

Ensemble AI:

Combines Machine

learning with Domain

expertise / business

rules and statistical

methods to:

-gain actionable

insight into large data

-review 100% of data

-sample from outliers

-perform risk

assessments

Current Features and Applications of AI

63

64

Group 1: Assisted AI: Support lower level decisions

Examples: Chatbots

Group 2: Augmented AI: Support high risk decisions

Examples: Performing audit risk assessments, Fraud detection,

Going concern evaluations

Group 3: Autonomous AI: Assumes decision making

Examples: Expense compliance

Current Applications of AI - Assisted, Augmented

and Autonomous

As AI shifts

from assisted

towards

autonomous,

the tool

sophistication,

resultant

benefits as

well as the

risks increase

CONCLUSIONS

65

A conceptual framework of an audit plan

cognitive assistant

Rutgers, the State University of New Jersey

Qiao LiMiklos A. Vasarhelyi

QA

Proposed Audit Cognitive Assistant Luca

Automatic

Speech

Recognition

Query

Classifier

Question

or Action

text

Answer Show

Answer

Execute

Action

Action

Lucaindustry

Client

Position

Luca Luca

LucaRecommended Topics:

General understanding,

new events, business

risks…

Processing…

You may also

interested in:…

Query

Open an

application

Inte

rface

Arc

hitectu

re

Modules:

• Automatic Speech

Recognition (ASR)

• Language Understanding

• Dialogue Management

• Natural Language

Generation

• Text-to-Speech synthesis

Audit Related Applications It Can Access

Web

Search

Open

(ACL,

IDEA…)

Calculato

r

Open

standards

Open

templatesAudit work

paper

Calendar …….backsta

ge

s

upport

er

Knowledge Database

Knowledge

about users

Knowle

dge

Base

DBMS

Unstructu

red data

Domain

Knowledge

Ap

pCapability

1 Search engine such as google

2 ACL, IDEA

3 Ratios; others like Benford’s

law

4 Regulations related to the

audit area or account

5 Required procedures,

guidance or programs

6 Open prior audit

7 Audit plan schedule

Domain: judgement or experience

Unstructured: financial

statements, accounting policies,

analytical procedures, litigation,

claims, recent news information,

audits workpapers, prior year audit

deficiencies and adjustments…

User interaction: queries and

search

4/13

SUSTAINI

NG

POTENTIA

L

DISRUPTI

VE

Emerging Technology

Emerging Technology Sustaining Disruptive

Blockchain and Smart Contract

√

Continuous Auditing

√

Interactive Data Visualization √ √

Linguistic Analysis

√

Machine Learning and Big Data Analysis √ √

Process Mining

√

Robotic Process Automation (RPA) √

Unmanned Aircraft Systems (Drones) √

DISRUPTIVE

TECHNOLO

GIES

INDEPENDENT PROCESS

CLIENT FIRM USAGE

ADVISORY AUDIT

MARKET

MAINSTREAM

AUDITING

PRACTICE

AUDITOR TECHNOLOGY

ACCEPTANCE

(MODIFIED UTAUT)PERFORMA

NCE

EXPECTANC

YSOCIAL

INFLUENCE

EFFORT

EXPECTANC

YFACILITATIN

G

CONDITION

SPRICE

VALUE

INDIVIDUAL

DIFFERENC

E

REGULATI

ON

SUSTAINING

TECHNOLOGIE

S

AUDIT VALUE NETWORK

Big Data Based

Government Economic Monitoring (GEM)

and Targeted Action

Miklos A. VasarhelyiKPMG Distinguished Professor of Accounting Information Systems

and Director of Rutgers Accounting Research Center and

Continuous Auditing & Reporting Lab (CAR Lab)

Arion Cheong

Xinxin Wang

GEM Overview

• The US Government 6 major social welfare programs

– Temporary Assistance for Needy Families program

– Medicaid

– Child's Health Insurance Program

– Food Stamps

– Supplemental Security Program

– Earned Income Tax Credit, and Housing Assistant program

• Using modern information technology to directly guide

government provide target actions.

• Big Data based GEM will enable the close-to-timely-reality

tracking of microeconomic status in real-time.

• Surveillance capitalism VS Your Big HUG government

• Transparency of every tax dollar.

Telephone

Drug Usage

Energy Usage

Government Agency

Online Retailers

Police Report

Supermarkets

Transportations

Stock Market

Darkweb

Exogenous data Collection Tree

15%

Governmental Institutions Blockchain

A wants to

vote

B need to fill her tax

return

C is a senior citizen who

needs to receive social

welfare payment

Government Blockchain

Privacy-preserved Blockchain Database

Privacy Data

BlockDB

by

CARLab

Untraceable/ Secured

Form

• Every Privacy data are securely stored in a Blockchain Database BlockDB

• BlockDB preserves privacy by transforming all the data in an untraceable format

- Name, SSN, etc. all be replaced to an index where outsiders would never be able to

trace back to individuals

- Blocks (data) are only retrievable for analysis

15%

Security

and

Protect

Privacy

Processed Information

Social Alert

Family/Individual Index

PatternIdentification

Traditional

Field CaseIdentification

Supervised DataLearning

Privacy-preserved Blockchain Database

UniqueIdentification

GEM Dynamic Process

Case – Food Stamp

• Better allocation of Food Stamp (Efficient Budget Allocation)

• Timely Feedback for policy adoption

Visit Our Website

• http://raw.rutgers.edu/GEM.html

78

79