Inside the NEWS

Chair’s Corner .........................................................................1 Getting the Word Out...............................................................................5 GIS in the Evaluation Process ................................................2 Enterprise Mississippi..............................................................................6 State Profile: Montana LAD-Performance Audits .................3 Office Happenings ...................................................................................7 Battle Hymn of Evaluation ......................................................4 From the Editor ........................................................................................8

1

National Legislative Program Evaluation Society A Staff Section of the National Conference of State Legislatures APRIL 2009

Chair’s Corner Jan Yamane (Hawaii)

Aloha kakou (greetings to all). So much has happened in the four short months since the last newsletter. Led by the downfall of US financial markets, the world has slipped into

economic chaos and we face what may be the deepest recession of our lifetime. State Budget Update President Obama has implemented a controversial economic stimulus package that promises billions of federal dollars to the states. However, it will be months, perhaps even years, before states feel the impact of this spending. In the meantime, our states’ economies are suffering. During informational briefings at the Hawaii Legislature last month, NCSL Executive Director Bill Pound reported that state revenue performance suffered in a number of categories. Sales taxes continue to fall as purchases decline and consumer confidence plummets. Personal income taxes are reeling from rising unemployment and stock market declines. Corporate income taxes are faltering as business profits dropped and severance taxes are weakening as energy prices drop. In order to balance budgets, states have responded with targeted program cuts, including higher education, K-12 education, Medicaid, corrections, and local government aid. Other cost cutting measures include salary and hiring freezes, travel bans, layoffs, furloughs, IOUs, tax and fee increases, fund sweeps, and tapping of rainy day funds.

Fall Training Conference These are trying times for NCSL staff sections, including NLPES. Travel and training bans have curtailed planning for staff section professional development seminars, including the NLPES 2009 Fall Training Conference in Hawaii. With member states unable to travel, the NLPES Executive Committee will be making the difficult decision of whether to continue, cancel, or postpone planning for its Hawaii conference. But whatever the outcome, NLPES will continue to provide training to its membership via alternative means, including podcasts, webinars, and audio conferences, among others. We will keep you abreast of these upcoming decisions on training and professional development. NLPES Executive Committee Elections Despite these trying times, NLPES will continue to thrive with your support and participation. Now, more than ever, we ask you to step up and get involved. We are in the process of conducting elections to the NLPES Executive Committee. Even if your state is currently under a travel ban, you can still be involved as an Executive Committee member and join in our efforts to bring training and other professional development opportunities to the NLPES membership. We encourage your involvement even if it means only being able to participate via teleconference or email. NLPES Awards Program NLPES is in the process of seeking nominations for the NLPES Awards Program, which will be presented at the NCSL Legislative Summit in Philadelphia, Pennsylvania, this July. The four award categories are: Excellence in Evaluation; Excellence in Research Methods; Impact; and Outstanding Achievement. Criteria and deadlines have been posted on the NLPES website and mailed to your offices.

NLPES News April 2009

2

Award winners will be notified in the early summer. We hope to see all offices participating in this year’s awards program.

Good luck to your legislatures as they work to balance their budgets. We hope that all states benefit from federal stimulus dollars.

Geographic Information Systems in the Evaluation Process Angus Maciver (Montana Legislative Audit Division)

The Montana Legislative Audit Division has been using geographic information systems (GIS) during performance audits for several years now. GIS is commonly defined as a system for capturing, storing, analyzing and managing data that are spatially referenced to the earth’s surface. The GIS tools currently available are really sophisticated database applications with graphics processing capabilities for producing maps. GIS applications have become an increasingly important tool in our work and have been used in a variety of audit projects. What follows is a brief discussion of how our use of GIS has developed over the years and where it may go in the future. We also include some discussion of the practical aspects of developing GIS capability within an audit/evaluation function. Graphical Representation (making maps) The most basic functionality offered by GIS is map production. A well-designed map will often have more impact than pages of narrative or tabular data. We have been using GIS mapping in many of our recent reports and it rarely fails to make an impact. Legislators like maps because they illustrate locality and, as the old mantra goes, all politics is local. Even if your primary audience is not exclusively legislative, maps are a powerful means for conveying a message and presenting large quantities of data in visual format. Spatial Relationships and Geoprocessing (doing data analysis) While maps are often the end result, the real power of GIS as an evaluation methodology lies in the ability to compile, categorize, format, and manipulate data. The map is just the visual output from the underlying data and the GIS allows you to use spatial references to sort, filter, query, and select data. GIS people call this geoprocessing, but it is not that different from what we do every day with more conventional data sets. However, using GIS provides a unique ability to analyze data and understand spatial relationships--i.e., where things are and how different things are linked by location. We have used GIS to analyze data on high school dropouts, emergency medical response, impacts from mining operations, land use and planning, wildlife habitat management, and many other issues.

Geostatistical Analysis (statistics and other things) As you gain more experience in making maps and conducting analysis, you will likely become aware of the more advanced tools available in full-function GIS applications. Depending on what application you use (see below), there are a large variety of tools available for conducting geostatistical analysis (applying statistical tests and routines to spatial data). There are also a lot of other exotic functions available for the advanced user that may be useful for audit/evaluation purposes. Some examples include investigating correlation in spatial data, identifying geographic outliers, predictive modeling, 3-D visualization, and mapping and analyzing complex networks. We are just starting to explore the potential for applying these more advanced features. Practical Considerations Developing GIS capabilities does require some effort and expense. While there are freeware applications available, advanced cartographic and analytical features are generally only found in proprietary software. The following points summarize a few practical considerations regarding buying and using GIS applications: There are quite a few vendors out there, but some

examples of general GIS applications are ESRI ArcGIS and Pitney Bowes/MapInfo. We use ArcGIS from ESRI (no endorsement implied) and have found it meets most of our needs. MapInfo appears to offer much of the same functionality.

State agencies have been investing heavily in GIS capabilities and there may be enterprise licensing deals and other resources available. We participate in an enterprise licensing arrangement that gives us access to free training and reduced-price software.

GIS people tend to be fairly tribal and can often be found in groups. Accessing their informal networks can provide useful support, advice on solving problems, and data and other resources.

Training staff is important, as is identifying people who can think spatially. We all have different talents and shouldn’t assume that the ability to think in terms of spatial relationships is universal. If you have some people on staff who show a willingness to think creatively about how to use GIS, invest in some training for them and make sure they keep their skills current.

NLPES News April 2009

3

STATE PROFILE: Montana Legislative Audit Division—Performance Audits

Responsibilities: Performance audits assess the effectiveness and efficiency of the operations of state government. The Legislative Audit Committee reviews the audit reports submitted by the Legislative Auditor, releases the audit reports to the public, and serves as the conduit between the Legislative Auditor and the Legislature. Staff Background: Accounting, Business Administration, Journalism, Chemistry, Economics, Sociology, Political Science, International Security, Anthropology, Finance, English, Public Administration, and CIS. Seven have master’s degrees with an eighth currently working on hers. After twenty-three years, we have a new Legislative Auditor In-Charge. Tori Hunthausen took over the reins on July 1, 2008. The Allure of Big Sky Country: Eight of our thirteen staff members are not born-and-raised Montanans. One is even from Scotland! Getting to Know Us Better: • One speaks three foreign languages (Farsi, Arabic and

German) and also knows sign language • One participated in the Olympic track and field trials

for the Steeplechase event while in college • For a $5 bet, one ate five Whoppers during lunch Some Highlights from Road Trips: • Ate a one pound cheeseburger in Livingston, felt sick

for days • Getting a “warning” for speeding in a state vehicle

(State employees are fired after three speeding tickets in a state vehicle)

• Conducting a two-hour interview with your fly open. • Slamming your head in the door in front of agency

staff while walking into a room to conduct an interview

• Listening to tall tales and drinking beers after a 15-hour day watching operations at a wildfire camp.

Best Things About Where We Work: • Flex and comp time and happy hours at the Blackfoot

River Brewery and Chili’s Dream Audit: • Anything to do with Fish, Wildlife & Parks • …is an oxymoron Most Dreaded Audit: • Department of Revenue taxpayer compliance • Child protective services • Anything to do with livestock and Medicaid

Scariest/Funniest/Weirdest Moment During an Audit: • Scariest - Riding across the state with a fellow

auditor • Scariest - Sitting in on a counseling session at the

WATCH program (4th time DUI offenders) • Scariest - Walking without escorts across a prison

yard • Funniest - Department director taped a fake soul

patch made from cat hair on his chin to mock Audit Project Lead.

• Funniest - Helicopter rides during a big game count audit. Need I say more?

• Weirdest – During a fire suppression audit, as the two firefighters returned from their assigned duty, they were asked by another firefighter how it went. Not knowing auditors were standing nearby, one of the firefighters responded “What a flipping waste of money…but at least I got some of it.” The auditors smiled, introduced themselves, and asked the speechless fire fighter for a few moments of his time.

Awkward Moments While Interviewing Agency Personnel: • During a meeting to tell stakeholders that we were

conducting an audit, one of the auditees stated, “I hope this isn’t inappropriate, but I love you. Will you marry me?” His boss replied that the question was indeed inappropriate.

Common Office Phrases: • We’re gonna end up in training • What happens on the road stays on the road Interesting Agency Responses to Audits: • “I didn’t know whether to sit down and have a

double martini or cry.” • “You’re a liar.” Oddest Question from a Legislator: • Don’t you think that was a stupid recommendation? Unofficial Motto: • Auditors watch the battle from the safety of the hill

and afterwards come down and bayonet the wounded!

Biggest Misconceptions: • We are a part of the State Auditor’s Office • All auditors are financial compliance auditors • All auditors are geeks, all geeks are auditors What We Want You to Know About Montana: • We now have the internet and cell phones; • We now use microwave ovens; and • We are now Yooper-friendly

NLPES News April 2009

4

Battle Hymn of Evaluation Almost anybody can have an association or society, but as someone put it, you’ve really arrived when you have a

song. A member of the American Evaluation Association composed the following in 1989 (we made one minor variation—changing AEA to NLPES in the third stanza). Our apologies to Mrs. Julia Ward Howe, lyricist of the Battle Hymn of the Republic.

Mine eyes have seen the glory

Of evaluation’s rise, We are trampling out poor programs

With our findings ever wise We have loosed the fateful lightning

By uprooting all the lies Our truth is marching on.

CHORUS: Useful! Useful! Hal-le-lu-jah

Our u-til-ity may fool ya Useful! Useful! Hal-le-lu-jah Our truth is marching on.

We work among the watchfires Of a hundred circling camps, Doing fieldwork, getting data

In the evening dews and damps, We can find program improvements

By the dim and flaring lamps Our day is marching on.

(Repeat Chorus)

We have written fiery findings, Writ in burnished prose of steel,

As we work with our stakeholders To earn bread for our next meal

Yet we never stray from standards, Earning NLPES’s good seal, Since truth is marching on

(Repeat Chorus)

We have sounded forth the trumpet,

The report is finally done, We have sifted out a judgment

From the data’s final run All the recs have now been given—

Oh what glory! Oh, what fun! When truth is marching on.

(Repeat Chorus)

In the beauty of the lilies,

What we do is seldom seen, Yet there’s glory in our calling

Keeping programs lean and clean, As we work to make them holy Let the findings reign supreme

While truth is marching on.

NLPES News April 2009

5

Getting the Word Out

As reported in the 2008 Ensuring the Public Trust survey, legislative program evaluation offices typically take several steps to make their legislatures and the public aware of their reviews. Communication strategies range from providing hardcopies of reports to posting reports and/or summaries on websites to making formal presentations. NLPES News recently surveyed legislative program evaluation offices regarding strategies that they use to get the word out. Listed below is a cross-section of communication strategies provided by respondents to the survey. Florida Office of Program Policy Analysis and Government Accountability

• The OPPAGA website, the Florida Monitor, enables legislators, staff and the public to download summaries and copies of the office’s formal reports and legislative presentations.

• OPPAGA maintains the Florida Government Accountability Report (FGAR), an on-line encyclopedia of all major state government programs. OPPAGA also distributes printed versions of FGAR to all legislators and committees.

• OPPAGA sends out The Florida Monitor Weekly, an electronic newsletter that profiles OPPAGA reports and other timely policy issues.

• Once a month, OPPAGA e-mails all members a listing of recently released reports, reports about to be released, and projects in progress.

• Before issuing every report, OPPAGA e-mails key legislators and staff of the upcoming findings and whether they are expected to be controversial.

• OPPAGA’s newest product is PolicyCasts, which are narrated PowerPoint presentations on selected reports that run three to five minutes and describe the most significant aspects of a report’s findings and recommendations. The PolicyCasts have proven to be very popular.

Hawaii Office of the State Auditor

• For one project that included information from the fifty states, the office produced a DVD. The office is in the preliminary stages of considering video and audio format products for dissemination of the office’s work.

Iowa Fiscal Services Division, Legislative Services Agency

• For large projects, the division announces the availability of a report by notifying by e-mail all

members, caucus staff and subscribers to the agency’s website. The notification provides a brief overview of the content and includes an Internet link to the document. The agency also has a section on its home page titled “Popular Links” under which large reports and documents are listed. The home page also has a subscription link for anyone who would like to be notified of new Legislative Services Agency publications.

Kansas Legislative Division of Post Audit

• The division has always tried to make its reports as short as possible, given the subject matter, with lots of graphics. Recently, with the availability of cheaper color copying, the division has increased its use of color graphics where it makes sense to do so and can better convey the information.

• The division develops a colorful highlights sheet that can be used as both a mailer to legislators and the media and as a summary inserted into the full report.

• The division occasionally uses CDs as a cost-effective way to avoid major printing costs for larger or complex reports. For example, the division recently completed an audit of low-priority state programs. By releasing the 200+ page report on CD, readers could read or print the entire report, the 16-page report text, or the tables for any of the audited agencies. The CD has an auto-run table of contents page that pops up automatically to make it easy for busy legislative leaders to locate the information in which they are interested.

• The division’s website provides a searchable database of thirty years’ worth of work. All reports since 2000 are available online as PDF files.

• Each year, the division issues an annual report that highlights the work performed in the prior year. The division also issues a follow-up report that describes agencies’ efforts to implement the reports’ recommendations.

• The division uses a savings database to track the potential or realized savings resulting from the division’s recommendations.

Maine Office of Program Evaluation and Government Accountability

• The office has plans to produce CDs and/or audio files of reports that can be downloaded.

Maryland Office of Legislative Audits

• The office lists reports issued in the quarterly “Legislative Audit Bulletin” that provides capsule summaries of audits with significant findings.

NLPES News April 2009

6

New Hampshire Legislative Budget Assistant Office, Audit Division

• The division presents its performance and financial audit reports (including contracted audits) to the Joint Legislative Fiscal Committee. House rules require the Committee on Executive Departments and Administration to review all performance audits. House rules also require all standing committees to review audits in their subject area, consider suggestions for recommended legislation, and monitor compliance with audit recommendations.

Texas State Auditor’s Office

• The office provides two reports in hard copy form at the beginning of each regular legislative session to legislative leadership, key staff, and all members and staff of the Senate and House budget-writing committees. The Audit Highlights report is a composite summary of all audits, reviews, and investigations released in the prior two years. It is organized by budget “article” and prioritized into four categories: potential testimony, potential resource, other audit reports of interest, and work in progress. The Workforce Summaries report provides a two-page summary or “snapshot” of employee demographic and salary information for each agency and higher education institution.

• The office staff, upon request, testifies on the results

of audits before the House and Senate budget and policy committees. Prior to testifying, office staff provide copies of the report, along with a handout that summarizes the key points, overall conclusion, findings, and recommendations.

West Virginia Performance Evaluation and Research Division

• The division sends a summary paragraph of each audit report to the Senate President and Speaker of the House several days prior to the release of the report.

Wisconsin Legislative Audit Bureau, Program Evaluation Division

• For each formal report, the bureau prepares a four-page “highlights” section. The bureau also prepares a ‘tabloid’ version of the highlights so that a legislator could have the tabloid for quick reference while on the road or in the district.

• The bureau prepares and posts on its website podcasts of the highlights of released reports.

• RSS feeds enable legislators and all interested persons to access the bureau’s reports at the time of release and posting on the website.

Enterprise Mississippi

It goes without saying that “things are tough all over,” at least regarding the country’s and states’ economic outlook. Daily news reports describe in excruciating detail the latest job layoffs and company bankruptcies or closures. State legislatures are scrambling to plug budget gaps that seem to grow larger by the day. In its February 20, 2009, Update on State Budget Gaps: FY 2009 & 2010 report, NCSL stated that the states faced a cumulative FY 2009 budget shortfall of approximately $47.4 billion. NCSL’s report also projected a cumulative budget shortfall among the states of approximately $84.3 billion for FY 2010. Given the trying economic times, state legislatures are searching for creative ways to reduce expenditures and/or increase revenues. Over the last decade or so (even before the current economic crisis), some states undertook statewide performance reviews. The wave of statewide performance reviews began in Texas in 1991. Attempting to address financial problems created by economic recession and an oil-industry slump, the Texas Legislature created the Texas Performance Review to “identify and recommend changes in state government that could save money while improving services.” The

Texas Comptroller of Public Accounts took the lead on the review and assembled a staff of more than 100 auditors, research analysts, and other specialists from sixteen state agencies and the private sector. Since the initial Texas review, at least twenty states have undertaken some form of statewide efficiency and effectiveness reviews. Mississippi’s Performance Review During its 2008 Regular Session, the Mississippi Legislature gave the PEER Committee responsibility for planning and conducting a performance review of Mississippi state government. Building on previous work done in Mississippi and in other states, on previous work done by the private sector to help improve state governmental efficiency and effectiveness, and on ideas coming directly from state citizens, the PEER Committee set for itself the goal of creating a comprehensive vision for change that, if implemented, would provide the greatest impact on the efficiency and effectiveness of Mississippi state government. Lessons Learned The PEER Committee released its performance review report on December 9, 2008 (see report # 518 at

NLPES News April 2009

7

www.peer.state.ms.us). Rather than focusing on particular agencies or programs, the PEER Committee chose to focus its statewide performance review report on possibilities for improving the administrative areas of government activities that undergird the work of all state agencies: information technology; budgeting and accountability; human resources management; and, procurement and asset management. In finalizing the report, Committee members and PEER staff contemplated what they had learned from the review. They learned that Mississippi’s history and the histories of other states are rife with good ideas for making government more efficient and effective and that the ideas sound similar across the years and across the nation. It seems that no one has yet found and fully implemented a system that will be self-correcting and self-perfecting. Mississippi public officials seem to be able to create “new beginnings,” but are much less efficient at completing the things started. Committee members and PEER staff concluded that Mississippi’s performance review was no different from previous performance improvement efforts in that until the state fully and unequivocally commits to a logical and broadly supported course for change, the state will never alter the primary efficiency and effectiveness of government. We will simply continue to operate at its fringes. In an effort to assist the Mississippi Legislature in understanding the concept of enterprise government and transformational change, the Committee developed the following top ten critical elements needed for improving government performance and accountability. 1. Leadership is critical to any performance

improvement effort. In order for any effort to improve the core institutions of government to be successful, bipartisan leadership of both the legislative and executive branches must agree on the form and content of the proposed change and support its implementation.

2. A state-level strategic planning process should be in

place to allow governmental leaders to view government as a whole and to set the agenda for interrelated strategic and operational planning at the agency level.

3. States should implement and take full advantage of the benefits of a well-planned and responsive information technology effort, integrating automated government operations where possible.

4. Government should efficiently manage financial and program performance information, since such information provides the basis for identifying needed performance improvement efforts and for measuring change.

5. The budget process can and should be a foundation element in any move toward improving efficiency and accountability in government.

6. “Right-sizing” of government can be a valid concept

when approached with a careful eye toward customer service, an efficient shared services system, and appropriately vetted privatization efforts.

7. “Right-sizing” of government should include not only

the efficient offering of programs and services, but also the assurance of appropriate levels of state and local political representation in the policymaking process. This may include periodic review of the size and type of representative bodies and the number of elected offices required to address the business of the state efficiently.

8. Asset management (e. g., equipment, buildings) should be

a foundation element in any governmental accountability process.

9. Two important questions to be answered relative to any

government improvement effort are: Does this make government more transparent to the public? Does this improve service to our citizens?

10. Performance improvement requires follow-through.

States should provide for ongoing review efforts in order to generate lasting economy and efficiency measures.

Office Happenings

Arizona

Bill Thomson, Deputy Auditor General, Office of the Auditor General, retired on March 12th after serving in various positions within the office for thirty-one years. Bill previously served as the director of the office’s Performance Audit Division from 1982 until his appointment as Deputy Auditor General in 2000. Prior to joining the Arizona office in 1978, he worked in performance auditing for the Utah Legislative Auditor General. Bill has appeared as a panelist or presenter on performance auditing at national and international conferences of the: American Society for Public Administration; Institute of Internal Auditors; National

Conference of State Legislatures; National Legislative Program Evaluation Society; National Association of State Auditors, Controllers and Treasurers; National State Auditors Association; and, Western Inter-Governmental Audit Forum. Bill is returning to work on a part-time basis with the office. Maine

The Office of Program Evaluation and Government Accountability recently welcomed Matt Kruk to the staff. Matt previously worked for the State of Maine as an economic research analyst for the Department of Labor. He is also trained in “lean manufacturing” techniques and participated in

NLPES News April 2009

8

the department’s Bend the Curve efficiency efforts as a Continuous Improvement Practitioner. Mississippi

The Joint Legislative PEER Committee recently welcomed Angela Norwood as an evaluator. Texas Sunset

The Texas Sunset Commission recently honored Ginny Mckay and Steve Hopson for twenty-five years’ and fifteen years’, respectively, of service with the Commission. Washington

Washington State’s Legislative Auditor was pleased to recently host a professional affiliation visit with Lech Marcinkowski from the nation of Poland. Mr. Marcinkowski is the Deputy Director for the Polish Prime Minister’s Audit and Supervision Department. He

is spending the year as a Hubert Humphrey Fellow with the University of Washington’s Evans School of Public Affairs. Our office was pleased to receive a request by Mr. Marcinkowski to learn about the performance auditing approache used at the Washington State Legislature. Mr. Marcinkowski spent six weeks at our office asking great questions about our work and procedures and offering helpful insights and advice to us as well from his experiences in Poland. Other evaluation and audit offices should seek out opportunities to link up with foreign performance auditors, as this experience provided meaningful professional development to both parties. Wisconsin

The Legislative Audit Bureau recently welcomed Jennie Mauer to the Program Evaluation Division as a Legislative Program Analyst.

From the Editor

According to Henry David Thoreau, “it is not enough to be busy; so are the ants. The question is: what are we busy about?” I think the answer to Thoreau’s question is fairly easy for all of us this time of the year with most of our state legislatures in full swing. Session days can be exciting, especially when we see the fruits of our labor being used during the public policymaking process. As Jan said in her Chair’s Corner on page 1, NLPES is wrestling with the challenges brought on by the country’s economic situation and associated state budget reductions and travel restrictions. Rest assured, the Executive Committee will do its best to keep NLPES as vibrant as possible. Having said that, the Executive Committee would certainly be interested in hearing your ideas as to how NLPES could provide services or assistance to you in an efficient and cost-effective manner. James Barber

NLPES-NEWS is published three times a year by the National Legislative Program Evaluation Society, a staff section of the National Conference of State Legislatures. NLPES serves the professionals of state legislative agencies engaged in government program evaluation. The purposes of NLPES are: To promote the art and science of legislative To enhance professionalism and training in To promote the exchange of ideas and infor- program evaluation. legislative program evaluation. mation about legislative program evaluation.

NLPES News Editor: James Barber (MS) 601-359-1226

james.barber@peer.state.ms.us www.ncsl.org/programs/nlpes/

National Conference of State Legislatures 7700 East First Place Denver, Colorado 80230 303-364-7700