PowerPoint Presentation

Get Results From Your Performance AuditsSession ObjectivesThis session is intended to help you:Identify and select high impact performance audits.Perform preliminary reviews that enhance audit effectiveness and efficiency with:The audit team having an understanding of the program or entity being audited.A prioritized assessment of audit risk and audit opportunity relative to the program or entity.A basis for developing audit objectives or terminating the audit.

Session ObjectivesThis session is intended to help you:Prepare preliminary review documents that present a persuasive rationale for proceeding with or terminating a performance audit.Develop well-defined audit objectives to achieve a focused audit effort for maximum audit efficiency.Understand how group activity techniques add value to performance auditing.

What Is a Performance Audit?A Performance Audit is an analysis that answers a question.The question is: What Is?

Which is then compared to:What Should Be?

4Yellow Book DescriptionPerformance audits are defined as audits that provide findings or conclusions based on an evaluation of sufficient, appropriate evidence against criteria.

Performance audits provide objective analysis to assist management and those charged with governance and oversight in using the information to improve program performance and operations, reduce costs, facilitate decision making by parties with responsibility to oversee or initiate corrective action, and contribute to public accountability. (2.10)5Examples of High Impact Performance Audit FindingsUnemployment Insurance AgencyBusiness Application Modernization (BAM) ProjectPharmaceutical CostsUnemployment Insurance Benefit Overpayments and Nonmonetary Eligibility DeterminationsWe estimated that UIA failed to identify and pursue recovery of overpayments of up to $72.5 million:Overpayments -Claimant wage and benefit cross match process = $17.9 millionCross-program benefit offset process = $12.1 millionVerification of claimant identity = $8.2 millionDeceased claimants = $0.4 millionUntimely nonmonetary eligibility determinations = $26.0 millionIneligible alien claimants = $7.9 millionTotal = $72.5 million

Unemployment Insurance Benefit Overpayments and Nonmonetary Eligibility DeterminationsWe estimated that UIA did not assess fraud-related penalties of between $120.0 million and $236.6 million:

Unassessed Penalties -Classification of claimants misrepresentations = $81.5 - $191.8 millionClaimant wage and benefit cross match process = $37.8 $43.4 millionDeceased claimants = $0.7 $1.4 millionTotal = $120.0 - $236.6 millionBusiness Application Modernization (BAM) ProjectEnforcement of Contract Provisions -

The Department should have collected liquidated damages resulting from the development contractor failing to meet the agreed-upon implementation dates for BAM. The development contract specified that liquidated damages would be assessed at $200,000 per month if the contractor failed to meet the agreed-upon implementation dates for Phases 3A, 3B, and 3C.

The development contractor missed the initial implementation date of August 2007 for Phase 3A (renamed as Release 1).

As a result, we calculated that the development contractor owed the State liquidated damages of $12.5 million.Prisoner Pharmaceutical CostsPotential Savings Based on Seroquel and Risperdal Prescription DataJanuary through July 2010Percentage reduction in Seroquel10%25%33%50%75%100%Average monthly number of Seroquel scripts2,3862,3862,3862,3862,3862,386Potential reduction in number of Seroquel scripts2395977871,1931,7902,386Average monthly cost of Seroquel script $ 335 $ 335 $ 335 $ 335 $ 335 $ 335 Projected reduction in Seroquel cost $ 80,065 $ 199,995 $ 263,645 $ 399,655 $ 599,650 $ 799,310 Average monthly cost of Risperdal script $ 38 $ 38 $ 38 $ 38 $ 38 $ 38 Projected increase in Risperdal cost $ 9,082 $ 22,686 $ 29,906 $ 45,334 $ 68,020 $ 90,668 Average monthly reduction in cost $ 70,983 $ 177,309 $ 233,739 $ 354,321 $ 531,630 $ 708,642 Annualized potential savings $851,796 $2,127,708 $2,804,868 $4,251,852 $6,379,560 $8,503,704 Prisoner Pharmaceutical CostsPHARMACEUTICAL COSTSDepartment of Corrections (DOC)Comparison of Michigan's and Other States' Average Atypical Antipsychotic Pharmaceutical Costs Per Prisoner Per Month For the Period February 2010 through July 2010Note: The chart includes 10 states whose departments of corrections contract with PharmaCorr, LLC, for statewide pharmacy services. As noted in Finding 1, about 80% of Michgan's atypical antipsychotic medications prescribed for DOC prisoners are written by the Department of Community Health and contracted psychiatrists.

How to Identify and Select High Impact Performance Audits? Start with a Risk RankingQualitative QuantitativeProgram/SystemQualitative Risk Ranking Is there a risk to:Public Health, Safety, ProtectionService DeliveryState SecurityInformation Security & PrivacyEconomic GrowthEducationConservation & Protection of Natural ResourcesCommercial or Public TransportationQualitative Risk RankingThings to consider:Public Health, Safety, &ProtectionInjury or loss to life, care & protection of vulnerable citizens, privacy or citizen rights, food safety, agricultureService DeliveryServices not factored into other categories (examples, state park reservation system, hunting permits, licensing), Medicaid payments provided to doctors, eligibility determination, family reunification services, mental health services, etc. State SecurityHomeland security, emergency preparedness (state & citizen), customs & borders, national guard, escape, recurring offender, parole violations, and MSP. Information Security & PrivacyProtection of IT resources, data securityQualitative Risk RankingThings to consider:Economic GrowthMichigan economy growth, job increase, business increase

EducationK - 12, higher-ed, adult ed, early childhood, special ed, laws & regulations, school aid, school improvement grantsConservation and Protection of Natural ResourcesAir, land, waste, water, forests, wildlifeCommercial or Public TransportationRoads, rail, public transportation, bridges, ferries, aeronauticsQualitative Risk Ranking0 = Not Relevant1 = Low Relevance2 = Moderate Relevance3 = High RelevanceQuantitative Risk Ranking Is there a risk of:State Revenue Sources Not Being Realized or CollectedIneffective and/or Inefficient Use of State ResourcesImproper Payments (State or Federal)Contingencies of Potential LiabilitiesValue/Condition of Major State Assets

Quantitative Risk RankingThings to consider:Value of Major AssetsNeed for replacement, need for repairs, and need for maintenance using State resources Examples of Assets: infrastructure, bridges, roads, property, IT equipmentState Revenue Sources Not Being Realized or CollectedFailure to collect tax revenues or fees, incorrect federal match, underreporting of accounts receivableContingencies or Potential Liabilities Financial obligation, debt, claimQuantitative Risk Ranking Things to consider:Ineffective and/or Inefficient Use of State ResourcesNot seeking federal grants, higher priced contracts, use of overtime, staffing allocations, potential privatization, combined departmental operations, program does not function as intended, program provides no value to State or stakeholders, ineffective programImproper Payments (State and Federal)Payments to ineligible recipient, payments for ineligible service, duplicate payments, payments for services not received, wasted payments that should not have been made, payments of incorrect amount (overpayment & underpayment)Quantitative Risk Ranking0 = Not Relevant1 = Less than $5 million2 = $5 million - $10 million3 = $10 million - $50 million4 = Over $50 millionProgram/System Level Risk Ranking Is there a risk of:Unauthorized Access of Loss of Sensitive InformationPersons/Clients at Risk if Program Fails to Provide ServicesInternal Monitoring/OversightReduced Confidence in GovernmentLegislative/Media InterestMaterial Control WeaknessesReliability of External Data

Program/System Level Risk RankingThings to consider:Reliability of External DataFor example - daycare billings from providers, data from school districts, Medicaid provider billings. Lack of oversight of third party administrators. Program EnvironmentUnreliable decision-making data, competent staff, management oversight, tone at the top, high staff turnover, limited resources, segregation of duties, limited evaluation of program dataInternal Monitoring/OversightFunctioning internal auditProgram/System Level Risk RankingThings to consider:Persons/Clients at risk of program fails to provide servicesClients not receiving assistance, medical treatment, or needed services; persons harmed, neglected, or killed because of failure to keep safe or protectUnauthorized access or loss of sensitive informationConfidentiality of information, potential impact to organizational operations, assets, or individualsReduced confidence in governmentPast program failureProgram/System Level Risk RankingThings to consider:Legislative/Media InterestRecent or current legislative interestMaterial Control WeaknessesLack of prior audit finding correction, demonstration to resolve material control weaknesses, progress to strengthen controls to address material control weaknessesOtherUnusual risks such as fraud, significant safety risks, other extreme high risks not fully measured elsewhere in modelProgram/System Level Risk Ranking0 = Not Applicable1 = Low Risk2 = Medium Risk3 = High RiskAudit Risk RangesHigh Risk Audit = 77 and UpModerate Risk Audit = 39-76Low Risk Audit = 0-38 How do I begin a Performance Audit?To determine the SCOPE of your performance audit, start by gathering INFORMATION and obtaining an understanding of the program or entity.

27INFORMATION How Do You Get It?

28

NASACTNLPESSURVEY29Exercise 1As auditors, we are used to being observant. But just How Observant Are You?

On a standard traffic light, is green on the top or bottom?When you walk, does your left arm swing with your right or left leg?On the U.S. flag, is the top stripe red or white?Which way does water go down the drain: clockwise or counter-clockwise?Which way do fans rotate?Whose face is on a dime?Do books have even numbered pages on the right or left side?How many curves are there in a standard paper clip?How many lug nuts are on a standard car wheel?Great Job!!As auditors, we are used to being observant. But just

30Determine the Scope and Gather Information During the Preliminary ReviewThe primary goal during the preliminary review is to gain the best understanding of the agency/program and its processes. This will allow the audit team to identify deficiencies and risks, and plan the audit scope and methodology.

31Preliminary Review ProcessObtain an overall understanding of the audited activity.Quickly gather information without detailed verification.Gain an understanding of internal control.Identify audit risk and audit opportunity.Preliminary Review ProcessAudit Risk

The degree to which activities are exposed to the potential for ineffective or inefficient use of resources; the inappropriate disclosure of data; or to forms of fraud, waste, and abuse which could result from inadequate controls.Preliminary Review ProcessAudit Opportunity

The potential for developing audit findings related to effectiveness and efficiency of a program.Preliminary Review ProcessPreliminary Review ConsiderationsWhat are the programs mandates, responsibilities, functions and activities?Which functions are most critical to the overall operation of the program?What are the risks related to the program?What responsibilities involve/create the most problems?What changes would make your program more effective in accomplishing its designated responsibilities?Who are the programs clients/customers?Does program management have any concerns that should be considered?Have recent reorganizations occurred (downsizing)?Preliminary Review ProcessPreliminary Review Procedures/MethodologiesInterview program personnel.Perform analytical reviews of expenditures, revenue, and data pertinent to the programs mission.Analyze prior audit results of audits of similar programs in other states.Review other external or internal audits.Review program mission, goals, objectives, policies, & procedures.Obtain organization charts and review staffing levels, including the number and type of employees.Research professional literature.Tour facilities, including off-site locations.Review other states similar programs and their mission, goals, objectives, and requirements.Obtain norms and standards pertinent to the programs operations.

Preliminary Review SummaryPreliminary Reviews enhance audit effectiveness and efficiency with:The audit team having an understanding of the program or entity being audited. A prioritized assessment of audit risk and audit opportunity relative to the program being audited.A basis for developing audit objectives.Achievable audit budgets.

Developing Audit ObjectivesWhat is the audit going to accomplish?

Specific audit objectives are critical to the process of auditing smarter and focusing our audit efforts. Without precisely stated objectives, the risk is that the audit work will not produce the desired results.

Therefore, it is important to define the issue, problem, or concern that the audit is to examine.38Types of ObjectivesEffectivenessSuccess in achieving mission and goals

EfficiencyAchieving the most outputs and outcomes practical with the minimum amount of resources.

39Audit Objective Characteristics Clarity Objectives should be stated clearly to inform the reader what aspects of the program, activity, or function are being assessed.NeutralityObjectives should be stated in neutral terms so that the reader understands that we gathered and analyzed data without bias. 40Writing Solid Audit ObjectivesObjectives should be:Specific Focused on one topicObjectives should not be:BroadCompound

41Writing Solid Audit ObjectivesAsk Yourself:Will I be able to conclude on this objective?Will I have multiple conclusions ? Effective at one thing; not effective at another thing.Will my conclusion make sense?Is this something we even want to conclude on?We may not want to conclude that tax collections were accurate, but we would conclude that their efforts to collect taxes were effective.42Audit Objectives - Example 1To assess the effectiveness of the Department's efforts to establish performance measures, identify best practices, and evaluate services provided to adults transitioning from foster care to self-sufficiency.

43Audit Objectives - Example 1Final Report VersionTo assess the effectiveness of DHS's efforts to evaluate services provided to youth transitioning from foster care to self-sufficiency.

To assess the effectiveness of DHS's efforts to ensure the propriety and reasonableness of discretionary payments and post-secondary educational assistance payments made on behalf of youth transitioning from foster care to self-sufficiency.

To assess the effectiveness of DHS's efforts to ensure that training for caseworkers is tailored to address youth transitioning from foster care to self-sufficiency.

44Audit Objectives - Example 2To assess the effectiveness of the Correctional Facility's efforts to comply with policies and procedures.

45Audit Objectives - Example 2 Final Report VersionTo assess the effectiveness of DOC's efforts to comply with selected policies and procedures related to safety and security at the St. Louis Correctional Facility.

46Audit Objectives Example 3

To assess the effectiveness of the Bridges information system.47Audit Objectives Example 3Final Report Version

To assess the effectiveness of DHS and DTMB's efforts to implement change controls over the Bridges application and data.

To assess the effectiveness of DHS and DTMB's efforts to implement controls to ensure the accuracy, completeness, and timeliness of Bridges interfaces.48Audit Objective SummaryWell-defined audit objectives achieve a focused audit effort and maximum audit efficiency.

Referent Group MeetingsObjectivesDocuments to BringTopics Discussed

Referent Group MeetingsHelp decide the scope of the audit. Objectives of a referent group meeting:Use a risk-based approach to select activities & programs for auditAgree upon audit focus, scope, and methodologyBring people togetherMaximize the use of limited resourcesProduce significant savings in audit hoursReferent Group MeetingWho Attends:Audit Supervisor Audit TeamPrimary Audit Division Administrator and/or Audit ManagerConcurring Audit Division Administrator and/or Audit ManagerPerformance Audit CoordinatorDeputy Auditor GeneralState Relations Officer

Referent Group MeetingWhat happens during the meeting?

Bring several people together (brainstorming).

Decide on focus and scope of audit.Discuss audit methodology.Discuss identified risks and probable audit findings.Review the wording of the audit objectives.Strive to include only high impact areas.Review budget proposals.

Alleviate fears of missing an issue.

Discuss fraud risk factors.

Produce significant savings in terms of audit hours.

Documents Used at the Referent Group MeetingNarrative document that includesDescription of Agency/ProgramPreliminary ObjectivesPrimary Questions that Objectives Will AnswerAudit BudgetProposed Budget HoursEstimated Fieldwork Completion DateEstimated Staff Release DatesDocuments Used at the Referent Group MeetingAudit Approach MatrixPreliminary Audit ObjectivesKnown FindingsPotential FindingsPreliminary Testing PerformedProposed Audit MethodologyAudit Objectives MemoOther Topics Discussed at the Referent MeetingEfficiency Aspects of the Audit Identify efficiency related issues (dollar savings)Develop efficiency objective (if warranted)Legislative Interests/ConcernsAudit-related interests/concerns communicated by legislators and/or their staffOther Topics Discussed at the Referent MeetingIT SystemsAre there any significant information systems related to the audit objectives? Do we need an objective regarding the reliability and integrity of data within significant systems?Need for IT auditor assistance?Potential for Data Mining (downloads, extractions, and analyses)?Fraud Risk Factors (SAS 99 Compliance)Discuss significant results from Fraud BrainstormingYellow Book Changes Related to Performance Audits(from 2011 Government Auditing Standards)The discussion of validity as an aspect of the quality of evidence has been revised to indicate that it is the extent to which evidence is a meaningful or reasonable basis for measuring what is being evaluated. In other words, validity refers to the extent to which evidence represents what it is purported to represent. (6.60b)

The discussion of the sufficiency and appropriateness of computer-processed information now indicates that the assessment of the sufficiency and appropriateness of computer-processed information includes considerations regarding the completeness and accuracy of the data for the intended purposes. (6.66)

Yellow Book Changes Related to Performance Audits(from 2011 Government Auditing Standards)The auditors responsibilities for communicating identified internal control deficiencies that are not significant to the audit have been clarified. Related documentation requirements and those related to noncompliance with provisions of contracts or grant agreements or abuse that are not significant to the audit have been removed. (7.19, 7.22)

The fraud reporting requirement is now limited to occurrences that are significant within the context of the audit objectives (7.21), with a requirement to communicate in writing other instances of fraud that warrant the attention of those charged with governance. (7.22)

Yellow Book Changes Related to Performance Audits(from 2011 Government Auditing Standards)The requirement that audit organizations develop policies to address requests by outside parties to obtain access to audit documentation has been removed. (2007 GAGAS, 7.84)

Early communication of deficiencies has been added as a consideration auditors may follow in the course of the performance audit. (6.78)

You Too Can Get Results From Your Performance Audits by

Conducting a risk assessment to select high impact audits

Conducting a preliminary review to establish scope

Developing clear audit objectives

Brainstorming with a team