F e a t u r e

M O RTG AG E B A N KI N G / F E B R U A RY 2 0 0 8

Decisioningthe Right Way

utomated decision technology (ADT) has become pervasive in themortgage lending industry. From back-office automated underwrit-

ing to loan pricing to point-of-sale product best fit, the automa-tion of key decisions has delivered a number of benefits. Theseinclude enabling companies to reduce loan origination and pro-

cessing costs, decrease processing time, increase the consistency of decisions,improve the customer experience, and make companies more agile in their ability tomodify policies and pricing in response to market changes.

The authors of this article have been at the cutting edge of mission-critical auto-mated decision technology in the mortgage industry over the past decade. We havehelped mortgage lenders, banks, credit unions and Wall Street firms plan, design,build, test, deploy and maintain a variety of automated decision systems.

Over the years, we have seen the pain and joy that clients have experiencedthrough the software development life cycle. Much of the pain can be attributed tothe inexperience that companies have with ADT and ill-advised decisions that weremade during the planning process. The design and implementation process for ADTsystems differs enough from conventional software projects that companies cannotsafely assume that their past successes in planning and managing information tech-nology (IT) projects will be transferable to ADT projects.

The good news is that with proper training and forethought, companies can signifi-cantly improve their ability to plan and execute ADT projects. This article describesmany of the pitfalls that companies commonly face and identifies best practices forpreventing these problems from derailing ADT initiatives. The article is segmented intosix major topic areas describing pitfalls common in each. The topics range from execu-tive strategy and individual project management to technology and knowledge engi-neering. The last two sections address the question of whether or not to “go it alone”(or seek outside help), and how to place an ADT project within a broader scope.

Although the examples provided stem from deep experience with mortgagetechnology, the lessons learned are applicable to other data- and knowledge-richenvironments implementing a strategy of automating decisions and processes withthe vision of an agile business.

Organizat ional planningA majority of software projects that fail were doomed prior to kick-off due toimproper planning and resource misallocation. Identifying the correct set ofstakeholders and participants is crucial to successful planning. ADT projects require theknowledge and experience of subject-matter experts (SMEs), IT, management and endusers. The needs and objectives of these diverse groups must be coordinated into a setof requirements that will lay the foundation for the development effort.

Consider, for example, the planning for a new best-fit product and pricing point-of-sale system to be deployed for a retail and wholesale lending operation. The coreteam for this effort includes subject-matter experts (product development andunderwriting, secondary marketing, sales experts and management), corporateresources (project management, project sponsor, legal and compliance), IT (internal

Here are some tips

for avoiding common

pitfalls that can

derail decisioning-

technology projects.

AB Y G I L R O N E N , J E F F R E Y A D L E R A N D C H U C K P E P E

and external technology partners) and end users (loan officers,brokers, processors and underwriters).

The lack of organizational planning will result in one or morewarning signs that can forecast increased project risk.

Business units are not aligned with corporate strategy andproject vision/scope: As notedearlier, ADT initiatives requireparticipation from several busi-ness units. New technology isexpected to help execute abusiness strategy, and workerswill be expected to adjust theiractivities to align themselveswith this new approach. Theimplementation of decisiontechnology has profoundimpacts on business workflowand the activities of key peoplein the value chain. People areinherently wary of ADT because of fears that their jobs willchange or be lost. All participants need to understand the objec-tives, scope, timelines and interdependencies involved.

Lack of project champion: Each project requires at least onechampion who represents the vision of the company and willdrive the project forward. The champion must have the ability tounite all of the business units behind the vision and the authorityto make strategic changes to the project team as needed to man-age the project budget and timeline.

Lack of management sponsor: The sponsor serves as the drivingforce behind the company strategic vision, and must have powerto impel project members to action. With tight budgets and lim-ited resources, the management sponsor will help ensure thatcost-effective and timely decisions are made.

Fail to involve end users in the planning stages: Technology thatis developed and implemented independent of end users is likelyto be poorly scoped and have inaccurate requirements. End userscan be hostile if new technology is thrust upon them withouttheir input.

Lack of resource commitment: Projects fail when stakeholdersdo not have enough time committed to be fully involved in deci-sion-making. In particular, when key project members—such asdevelopers and subject-matter experts—are not allocatedenough time to work on the project, the sense of urgency will bediminished, and the schedule will lag.

Budget and t imel ine planningThere is a need to balance a global vision with the reality of sys-tems development constraints. Most ADT systems cannot bedesigned and implemented in a single development cycle. Successis achieved in small increments—aiming to satisfy all of the objec-tives in a single go-round is inherently risky. Common pitfalls dur-ing this phase include:

Planning biased by fiscal reality: Often projects are driven notby prioritized requirements, but by available resources—howmuch can we get for $X. The concern about not spending theentire budget or not having money in next year’s budget canadversely bias the scoping and requirements phase of a project.

Overwhelmed by integration: Decisioning systems becomemore powerful as they have access to more data—therefore, inte-

gration with a variety of systems within the enterprise can quicklyoverwhelm the implementation of a decisioning strategy. Thescope of decision and integration needs must be addressed earlyand often as a separate project with dedicated staff.

Perfection versus time-to-market: It is more important to getgood technology to market quickly, even if it covers less than 100percent of the decision-making domain, than it is to delay launchuntil it is comprehensive. Scope the project to get 85 percentaccuracy/fidelity, and develop workflow processes to deal withany issues not covered by the new system. It is generally unrea-sonable to expect 100 percent functionality upon initial rollout.

Failure to prioritize the depth and breadth of decisioning: Inlarge domains, there are many decisioning scenarios for the tech-nology to cover. Companies often are far too ambitious in tryingto automate the entire domain at once rather than focusing on thecore business decisions. Furthermore, more complex decisioningscenarios absorb a greater percentage of resources to scope andimplement. It is critical to prioritize scenarios and seek to maxi-mize the return on investment (ROI) that automation will yield.

For example, when translating guidelines to business rules foran underwriting engine, cases that are rarely encountered by thebusiness (e.g., loans for foreign nationals or rural properties)should be assigned lower priority. Similarly, cases that requireexcess resources to interpret or model, such as deciphering creditcounseling within credit reports, can be initially referred for man-ual review and automated in a future phase.

Long development cycles: Strive for iterative software develop-ment with short-term success. Longer cycles drain project teamsand limit the sense of urgency. Keep everyone focused on short-term successes. Break down projects into small deliverables andemphasize good requirement management.

Planning for maintenance: The term “maintenance,” so funda-mental for traditional data-driven systems, is a misnomer for ADTsystems—these are evolving systems often with exponentialimpact, their impact quickly creating additional and unforeseenrequirements. It isn’t maintenance that needs to be planned, buthow to evolve the business logic and how to integrate with addi-tional data sources to make possible more complex decisions.

Knowledge engineer ing Automated decision technology automates the manual, labor-intensive process of reviewing lots of data and ensuring consis-tent decisions are derived from these data. For example, auto-mated underwriting systems relieve human underwriters fromporing over loan applications and credit reports while ensuringthat credit policies are applied equally across all borrowers.

Designing ADT systems to accurately process multiple, complexdata sources and correctly mimic the company’s best decision-makers is far from a trivial process. The level of effort required tocodify and implement this decision-making knowledge is oftenunderestimated and leads to systems that fail to produce accurateresults. The following pitfalls should be avoided:

Insufficient SME allocation: Subject-matter experts representthe best practices of your company, and their expertise in pro-cessing data and making decisions must be represented. Workingon an ADT project must be a primary focus of an SME. Far toooften, we have seen companies assign SMEs to projects part-time,almost as an afterthought. SME involvement is critical across theproject life cycle, from planning to testing and deployment.

M O RTG AG E B A N KI N G / F E B R U A RY 2 0 0 8

Technology that is

developed and imple-

mented independent

of end users is likely

to be poorly scoped

and have inaccurate

requirements.

Wrong type of person serving as SME: Companies often assumethat the most knowledgeable decision-maker is best-suited toserve as the SME. In reality, an SME needs to have several skillsthat are equally important: understanding the business strategyand how the technology will help align to this strategy; computerskills to use and evaluate the system as it is built; and strong com-munication skills that enable him or her to describe how decisionsshould be made and to work with IT staff to implement.

Decisioning ambiguity: Undertaking the development of ADTsystems forces companies to think more critically about their poli-cies and workflow. When a single system is designed to replace mul-tiple human decision-makers, a consistent interpretation of policy issought. It is critical for these issues to be settled by the businessprior to development. We have seen projects delayed because dur-ing the implementation phase companies are tied up discussing pol-icy and determining how the system should make decisions.

For example, when translating underwriting guidelines to busi-ness rules and calculations, it is common to encounter ambiguitywhere the subject-matter experts disagree on how to performcalculations and/or apply a guideline (e.g., aggregating liabilities tocalculate debt ratio, handling missing data, determining key dates).The subject-matter experts need to proactively identify theseinstances of ambiguity and converge on a single best practice.

Inability to handle incomplete data: ADT systems are driven byinput data. In many cases, a complete set of data will be available,and the system will be able to return a fully informed decision. How-ever, systems are often called upon to reason over missing, incom-plete or conflicting data. Different levels of data impact the qualityof the decision and need to be accounted for in the design process.

Present ing the decis ion (user inter faces)Automated decision technology systems often portray complexinformation back to the user. In the case of a mortgage productbest-fit application, this can include many mortgage productoptions for the borrower, possibly bucketed by how closely theymatch the borrower needs. These can often include verbiageexplaining the benefits of each particular option and potentiallyincluding all of the calculations, assessments and explanationsthat together provide the appropriate level of information forunderstanding this complex financial instrument.

The full set of information requiring quick digestion and actioncan be many pages deep, and more akin to the amount of infor-mation presented to a fighter pilot on his helmet than to a tradi-tional IT system end user. The underlying complexity of the busi-ness logic (full mortgage underwriting coupled with point-of-salecustomer engagement technique) interacts with the complexityof the possible ways of displaying the information back to thevarious consumers of this information—they feed off of eachother. The ADT system’s ability to explain a decision is as impor-tant as being able to arrive at one.

For example, when an underwriting engine returns a “refer” deci-sion, it must provide detailed explanations not only on what policiesare violated and in what way, but also how the supporting values arederived. Additional messaging can help the user determine if theloan can be reworked with alternative product/pricing options.

When designing the interface and documentation to reportthe decision, the following issues should be considered:

Strong decision, weak display: Mortgage ADT systems can pro-duce complex decisions—a product best-fit application allows

originators to consider all loan programs being offered by thecompany, and makes sure that the borrowers are getting the bestloan. It is therefore critical to create a decision output with theright balance of information presented in a clear format. Toomuch information can overwhelm the user; too little informationwill devalue the decision.

Failure to account for different points of view: ADT may beleveraged across multiple user groups, each requiring differentinputs and outputs. For example, customers on the Web, mort-gage brokers and underwriters all want to view appropriate infor-mation about deal configuration, yet have very different goals,process flows and levels of sophistication. Objections may rangefrom the user interface being more of a burden to use than adecision aid, the new workflow not making sense within theframework of other existing systems or too many required inputs.While designing a single interface or process flow may appear tobe cost-effective, the failure to customize the interface for eachuser group will make the system less useful and impede adoption.

Engage usability experts to design the interface: Although thedecisioning system and the user interface must work in sync, theskills needed to design each component are quite different. Failingto focus on the user interface and the context of the outputincreases the risk that the system will not be readily accepted by theuser community. Skimping on the design of the user interface andthe user experience will greatly diminish the usability of the system.

Use visualization to bridge gaps in understanding: Often, busi-ness users do not fully realize the potential touch points on othersystems and business processes until they see results in the userinterface. When there is parallel and iterative development of theuser interface and the decisioning components, it allows the

stakeholders to better compre-hend the changes to workflowand the level of information pre-sented at each decision point.

Val idat ion and test ingThe testing of automated decisiontechnology systems is often mis-understood and poorly executed.Testing and validation are more sig-nificant for decision systems thanfor more traditional IT applica-tions, because often the decisionsare binding and have significant

financial and legal implications. In addition to traditional useracceptance testing of the technology, ADT requires in-depth testingof the knowledge and decisioning. Validating the decision is a verylabor-intensive process requiring the skills of subject-matter experts.Improper testing of an ADT system will seriously jeopardize itseffectiveness when deployed. Watch out for the following pitfalls:

Delaying the creation of the test plan: Countless times we haveworked with clients who have experience in testing conventionalsoftware but still do not take seriously the need to begin planningfor ADT testing at the start of the project. Without early planning,project managers are inevitably overwhelmed by the need for rigor-ous decision validation and testing of thousands of business rules.

Failure to prioritize test-case features: When resources are lim-ited, it is an imperative to prioritize testing, initially focusing on high-risk cases. This will ensure that the most critical features are fully

M O RTG AG E B A N KI N G / F E B R U A RY 2 0 0 8

Testing and

validation are more

significant for decision

systems than for

more traditional

IT applications.

M O RTG AG E B A N KI N G / F E B R U A RY 2 0 0 8

tested and pinpointed. Testing should also start with the most com-mon types of requests. For example, start testing an automatedunderwriting system with the most commonly offered loan prod-ucts and features, and work toward the least common scenarios.

Limited involvement of SMEs: Validation of ADT systemsrequires deep domain knowledge. Without adequate time for test-ing by the subject-matter experts, the likelihood of problems sur-facing after the system is deployed into production is significantlyincreased. If subject-matter experts have limited time to reviewresults during the normal workday, provide incentives for them todo the work during off-hours. In addition, rather than waiting untilthe end of the full development process, some domain-specifictesting can be done during an iterative implementation process.After each incremental release during development, it is possible todo feature-based testing of the business logic.

Underestimating the complexity of creating good test cases:Test cases can include thousands of combinations of data points,and there must be a process and an organization to support thebuilding of test cases. Often, there are interdependencies of dataelements within a test case that need to be maintained (e.g., realestate–owned [REO] records in a mortgage application and in acredit report). If the data are inconsistent or incomplete, the testcase will be invalid.

Automated testing is no panacea: Regression testing is neces-sary as new releases are deployed, and some of the regressiontesting can be automated by using tools to submit test cases andcompare results. The problems occur when organizations rely tooheavily on automated testing tools.

Understanding the output of an automated tool can be cumber-some and time-consuming. Test cases may contain time-sensitiveinformation (e.g., credit reports), and therefore results may changeover time (e.g., tax-lien records may recede further in time the longerthe test case using this credit report is used). Finding the right mix ofautomated and manual testing is key for a successful test plan.

Bui ld or buyDetermining whether to outsource part or all of the planning anddevelopment effort is the most strategic decision a company canmake when investing in ADT. ADT components are expensive, andcompanies with successful IT departments are often hesitant tobuy when they think they can build their ADT systems using in-house technology.

There are several common arguments proposed for buildingADT systems internally:

Building it will ensure that the specifications are met: While itis easy to envision at a high-level how ADT will help your business,given the complexity of the undertaking as described here, it is arisky proposition to hope staff inexperienced with the technology

will be able to take full advantage of it.IT staff will be better equipped to support and expand the sys-

tem if they build it: Most successful ADT systems are custom-builtfor clients with the intent that they take ownership once the sys-tem is deployed. Having experts train your IT and business staffwill enable you to take ownership early and collaboratively imple-ment your ADT strategy.

Buying is more expensive than building: ADT systems are builtusing special methodologies and tools that the business needs topurchase and learn to utilize. Expecting short-term turnaround onthese requirements is a tall order, and the risk to timeline andbudget is significantly increased.

F inal wordsUtilizing automated decision technology can provide significantreturn on the investment. When ADT is applied within a mortgageIT infrastructure, it can reduce the cost and shorten the time ittakes to process a loan, reduce mistakes, provide more consistentand accurate decisions, improve the customer experience, providebetter control and management of the policies, and support over-all innovation and agility for the business.

A vision for utilizing ADT for automating decision points through-out the enterprise is a commitment to transforming the enterprisefrom a traditional business using data-driven systems to a forward-looking, agile business utilizing knowledge-driven automation. Deci-sioning projects may fail if viewed as a one-time implementation ofa discrete system instead of a commitment to process transforma-tion and ongoing knowledge management. Use of ADT is a strategytoward implementing a vision—and not an end in itself.

When done right, an ADT strategy will prepare the business forthe future. However, the warning that we hope to have conveyedhere is that the planning, management, implementation and testingof ADT require resources that understand ADT and are experiencedenough to know how to avoid the pitfalls reviewed herein. MIBt

Gil Ronen is former vice president of consulting for Greenbrae, California–based

MDA Mindbox. He has 20 years of experience in building award-winning published

and patented systems utilizing rule-base and case-base technologies. Jeffrey Adler

is senior director, management consulting group, for MDA Mindbox. He has more

than 20 years of experience with business rules technology and automated deci-

sioning, and for the last seven years has guided the planning and implementation

of automated decision technology for leading residential mortgage lenders and

financial investment companies. Chuck Pepe is vice president of consulting and

managing director of the management consulting group for MDA MindBox. He has

22 years of experience in applied artificial intelligence, and for the last 13 years has

built automated decisioning applications and expert systems for financial services

companies. The authors can be reached at revvision@gmail.com, jeff.adler@

mindbox.com and chuck.pepe@mindbox.com, respectively.

REPRINTED WITH PERMISSION FROM

THE MORTGAGE BANKERS ASSOCIATION (MBA)