Measuring the Success of Strategic Initiatives

Chapter Four: Measuring the Success of Strategic Initiatives

LEARNING OUTCOMES

4.1. Compare efficiency IT metrics and effectiveness ITmetrics.

4.2. List and describe five common types of efficiencyIT metrics.

4.3. List and describe four types of effectiveness ITmetrics.

4.4. Explain customer metrics and their importance toan organization.

n an effort to offer detailed information to all layers of management, GeneralElectric Co. (GE) invested $1.5 billion in employee time, hardware, software,and other technologies to implement a real-time operations monitoring sys-

tem. GE’s executives use the new system to monitor sales, inventory, and savingsacross its 13 different global business operations every 15 minutes. This allows GEto respond to changes, reduce cycle times, and improve risk management on anhourly basis instead of waiting for month-end or quarter-end reports. GE estimatesthat the $1.5 billion investment will provide a 33 percent return on investment overthe five years of the project.13

Organizations spend enormous sums of money on IT to compete in today’s fast-paced business environment. Some organizations spend up to 50 percent of theirtotal capital expenditures on IT. To justify expenditures on IT, an organization mustmeasure the payoff of these investments, their impact on business performance,and the overall business value gained.

Efficiency and effectiveness metrics are two primary types of IT metrics. Effi-ciency IT metrics measure the performance of the IT system itself includingthroughput, speed, availability, etc. Effectiveness IT metrics measure the impact IThas on business processes and activities including customer satisfaction, conver-sion rates, sell-through increases, etc. Peter Drucker offers a helpful distinction be-tween efficiency and effectiveness. Drucker states that managers “Do things right”and/or “Do the right things.” Doing things right addresses efficiency—getting themost from each resource. Doing the right things addresses effectiveness—settingthe right goals and objectives and ensuring they are accomplished.14

Effectiveness focuses on how well an organization is achieving its goals and ob-jectives, while efficiency focuses on the extent to which an organization is using itsresources in an optimal way. The two—efficiency and effectiveness—are definitelyinterrelated. However, success in one area does not necessarily imply success in theother.

BENCHMARKING—BASELINING METRICS

Regardless of what is measured, how it is measured, and whether it is for the sakeof efficiency or effectiveness, there must be benchmarks, or baseline values the sys-tem seeks to attain. Benchmarking is a process of continuously measuring systemresults, comparing those results to optimal system performance (benchmark val-ues), and identifying steps and procedures to improve system performance.

Unit 1 Achieving Business Success through Information Technology 29*

I

haa83019_unit01 10/22/04 1:29 PM Page 29

Consider e-government worldwide as an illustra-tion of benchmarking efficiency IT metrics and effec-tiveness IT metrics (see survey results in Figure 1.19).From an effectiveness point of view, Canada ranksnumber one in terms of e-government satisfaction ofits citizens. (The United States ranks third.) The sur-vey, sponsored by Accenture, also included such attri-butes as CRM practices, customer-service vision, ap-proaches to offering e-government services throughmultiple-service delivery channels, and initiatives foridentifying services for individual citizen segments.These are all benchmarks at which Canada’s govern-ment excels.15

In contrast, the United Nations Division for PublicEconomics and Public Administration ranks Canada

sixth in terms of efficiency IT metrics. (It ranked the United States first.) This par-ticular ranking based purely on efficiency IT metrics includes benchmarks such asthe number of computers per 100 citizens, the number of Internet hosts per 10,000citizens, the percentage of the citizen population online, and several other factors.Therefore, while Canada lags behind in IT efficiency, it is the premier e-governmentprovider in terms of effectiveness.16

Governments hoping to increase their e-government presence would bench-mark themselves against these sorts of efficiency and effectiveness metrics. Thereis a high degree of correlation between e-government efficiency and effectiveness,although it is not absolute.

THE INTERRELATIONSHIPS OF EFFICIENCY AND EFFECTIVENESS IT METRICS

Efficiency IT metrics focus on the technology itself. The most common types of efficiency IT metrics include:

n Throughput—the amount of information that can travel through a system atany point in time.

n Speed—the amount of time a system takes to perform a transaction.

n Availability—the number of hours a system is available for use by customersand employees.

n Accuracy—the extent to which a system generates the correct results whenexecuting the same transaction numerous times.

n Web traffic—including a host of benchmarks such as the number of page-views, the number of unique visitors, and the average time spent viewing aWeb page.

n Response time—the time it takes to respond to user interactions such as amouse click.

While these efficiency metrics are important to monitor, they do not alwaysguarantee effectiveness. Effectiveness IT metrics are determined according to anorganization’s goals, strategies, and objectives. Here, it becomes important to con-sider the strategy an organization is using, such as a broad cost leadership strategy(Wal-Mart for example), as well as specific goals and objectives such as increasingnew customers by 10 percent or reducing new product development cycle times tosix months. Broad, general effectiveness metrics include:

n Usability—the ease with which people perform transactions and/or find in-formation. A popular usability metric on the Internet is degrees of freedom,which measures the number of clicks to get to desired information or process-ing capabilities.

30 Unit 1 Achieving Business Success through Information Technology*

1. United States (3.11)

2. Australia (2.60)

3. New Zealand (2.59)

4. Singapore (2.58)

5. Norway (2.55)

6. Canada (2.52)

7. United Kingdom (2.52)

8. Netherlands (2.51)

9. Denmark (2.47)

10. Germany (2.46)

Efficiency Effectiveness

1. Canada

2. Singapore

3. United States

4. Denmark

5. Australia

6. Finland

7. Hong Kong

8. United Kingdom

9. Germany

10. Ireland

FIGURE 1.19

Comparing IT Efficiencyand EffectivenessMetrics for E-GovernmentInitiatives

haa83019_unit01 10/22/04 1:29 PM Page 30

n Customer satisfaction—as measured by such benchmarks as satisfaction sur-veys, percentage of existing customers retained, and increases in revenue dol-lars per customer.

n Conversion rates—the number of customers an organization “touches” forthe first time and convinces to purchase its products or services. This is a pop-ular metric for evaluating the effectiveness of banner, pop-up, and pop-underads on the Internet.

n Financial—such as return on investment (the earning power of an organiza-tion’s assets), cost-benefit analysis (the comparison of projected revenues andcosts including development, maintenance, fixed, and variable), and break-even analysis (the point at which constant revenues equal ongoing costs).

In the private sector, eBay is an organization that constantly benchmarks its information technology efficiency and effectiveness. In 2003, eBay posted impres-sive year-end results with revenues increasing 74 percent while earnings grew 135percent. Maintaining constant Web site availability and optimal throughput per-formance is critical to eBay’s success.17

Jupiter Media Metrix ranked eBay as the Website with the highest visitor volume (efficiency) in2003 for the second year in a row, with an 80 per-cent growth from the previous year. eBay averaged4.5 million unique visitors during each week of the holiday season that year with daily peaks ex-ceeding 5 million visitors. To ensure constantavailability and reliability of its systems, eBay im-plemented ProactiveNet, a performance measure-ment and management-tracking tool. The tool al-lows eBay to monitor its environment againstbaseline benchmarks, which helps the eBay teamkeep tight control of its systems. The new systemhas resulted in improved system availability with a150 percent increase in productivity as measuredby system uptime.18

Do not forget to consider the issue of securitywhile determining efficiency and effectiveness ITmetrics. When an organization offers its cus-tomers the ability to purchase products over theInternet it must implement the appropriate secu-rity—such as encryption and Secure Sockets Lay-ers (SSLs; denoted by the lock symbol in the lower right corner of a browser windowand/or the “s” in https). It is actually inefficient for an organization to implementsecurity measures for Internet-based transactions as compared to processing non-secure transactions. However, an organization will probably have a difficult time at-tracting new customers and increasing Web-based revenue if it does not implementthe necessary security measures. Purely from an efficiency IT metric point of view,security generates some inefficiency. From an organization’s business strategypoint of view, however, security should lead to increases in effectiveness metrics.

Figure 1.20 provides a graph depicting the interrelationships between efficiencyand effectiveness. Ideally, an organization should operate in the upper right-handcorner of the graph, realizing both significant increases in efficiency and effective-ness. However, operating in the upper left-hand corner (minimal efficiency with in-creased effectiveness) or the lower right-hand corner (significant efficiency withminimal effectiveness) may be in line with an organization’s particular strategies. Ingeneral, operating in the lower left-hand corner (minimal efficiency and minimaleffectiveness) is not ideal for the operation of any organization.


eBay’s Web site

haa83019_unit01 10/22/04 1:29 PM Page 31

FIGURE 1.20

The Interrelationshipsbetween Efficiency andEffectiveness

DETERMINING IT EFFICIENCY AND EFFECTIVENESS

Stock market analysts tend to have a bias toward financial metrics. Clearly this em-phasis is important—however, focusing only on financial measurements is limitingsince financial metrics are only one factor that an organization can use to deter-mine IT effectiveness. Customer metrics assess the management of customer rela-tionships by the organization. These effectiveness metrics typically focus on a set ofcore measurements including market share, customer acquisition, customer satis-faction, and customer profitability.

Traffic on the Internet retail site for Wal-Mart has grown 66 percent in the lastyear. The site has over 500,000 visitors daily, 2 million Web pages downloaded daily,6.5 million visitors per week, and over 60,000 users logged on simultaneously. Wal-Mart’s primary concern is maintaining optimal performance for online transac-tions. A disruption to the Web site directly affects the company’s bottom line andcustomer loyalty. The company monitors and tracks the hardware, software, andnetwork running the company’s Web site to ensure high quality of service, whichsaved the company $1.1 million dollars in 2003.19

Customers are primarily concerned with the quality of service they receive froman organization. Anyone using the Internet knows that it is far from perfect and often slow. One of the biggest problems facing Internet users is congestion causedby capacity too small to handle large amounts of traffic. Corporations are continu-ally benchmarking and monitoring their systems in order to ensure high quality ofservice. The most common quality of service metrics that are benchmarked andmonitored include throughput, speed, and availability.

A company must continually monitor these behaviors to determine if the systemis operating above or below expectations. If the system begins to operate below ex-pectations, system administrators must take immediate action to bring the systemback up to acceptable operating levels. For example, if a company suddenly begantaking two minutes to deliver a Web page to a customer, the company would needto fix this problem as soon as possible to keep from losing customers and ultimatelyrevenue.

Web Traffic Analysis

Most companies measure the traffic on a Web site as the primary determinant ofthe Web site’s success. However, a lot of Web site traffic does not necessarily indi-cate large sales. Many organizations with lots of Web site traffic have minimal sales.A company can go further and use Web traffic analysis to determine the revenuegenerated by Web traffic, the number of new customers acquired by Web traffic, anyreductions in customer service calls resulting from Web traffic, and so on. The Yan-kee Group reports that 66 percent of companies determine Web site success solelyby measuring the amount of traffic. New customer acquisition ranked second onthe list at 34 percent, and revenue generation ranked third at 23 percent.20


High

Effic

ienc

y

LowLow HighEffectiveness

Optimal areain which to

operate

haa83019_unit01 10/22/04 1:29 PM Page 32


Analyzing Web site traffic is one way organizations can understand the effective-ness of Web advertising. A cookie is a small file deposited on a hard drive by a Website containing information about customers and their Web activities. Cookies al-low Web sites to record the comings and goings of customers, usually without theirknowledge or consent. A click-through is a count of the number of people who visitone site and click on an advertisement that takes them to the site of the advertiser.A banner ad is a small ad on one Web site that advertises the products and servicesof another business, usually another dot-com business. Advertisers can track howoften customers click on banner ads resulting in a click-through to their Web site.Often the cost of the banner ad depends on the number of customers who click onthe banner ad. Tracking the number of banner ad clicks is a great way to begin tounderstand the effectiveness of the ad on its target audience.

Tracking effectiveness based on click-throughs guarantees exposure to targetads; however, it does not guarantee that the visitor liked the ad, spent any substan-tial time viewing the ad, or was satisfied with the information contained in the ad.In order to help understand advertising effectiveness, interactivity measures aretracked and monitored. Interactivity measures the visitor interactions with the tar-get ad. Such interaction measures include the duration of time the visitor spendsviewing the ad, the number of pages viewed, and even the number of repeat visitsto the target ad. Interactivity measures are a giant step forward for advertisers,since traditional methods of advertising through newspapers, magazines, outdoorsuch as billboards and buses, and radio and television provide no way to track effectiveness metrics. Interactivity metrics measure actual consumer activities,something that was impossible to do in the past and provides advertisers withtremendous amounts of useful information.

The ultimate outcome of any advertisement is a purchase. Knowing how manyvisitors make purchases on a Web site and the dollar amount of the purchases cre-ates critical business information. It is easy to communicate the business value ofa Web site or Web application when an organization can tie revenue amounts andnew customer creation numbers directly back to the Web site, banner ad, or Webapplication.

Behavioral Metrics

Firms can observe through click-stream data the exact pattern of a consumer’s nav-igation through a site. Click-stream data can reveal a number of basic data pointson how consumers interact with Web sites. Metrics based on click-stream data in-clude:

n The number of page views (i.e., the number of times a particular page hasbeen presented to a visitor)

n The pattern of Web sites visited, including most frequent exit page and mostfrequent prior Web sites

n Length of stay on the Web site

n Dates and times of visits

n Number of registrations filled out per 100 visitors

n Number of abandoned registrations

n Demographics of registered visitors

n Number of customers with shopping carts

n Number of abandoned shopping carts

Figure 1.21 provides definitions of common metrics based on click-stream data. Tointerpret such data properly, managers try to benchmark against other companies.For instance, consumers seem to visit their preferred Web sites regularly, evenchecking back to the Web site multiple times during a given session. Consumerstend to become loyal to a small number of Web sites, and they tend to revisit thoseWeb sites a number of times during a particular session.

haa83019_unit01 10/22/04 1:29 PM Page 33

Evolving technologies are continually changing the speed and form of businessin almost every industry. Organizations spend an enormous amount of money onIT in order to remain competitive. Some organizations spend up to 50 percent oftheir total capital expenditures on IT investments. More than ever, there is a needto understand the payoff of these large IT investments, the impact on business per-formance, and the overall business value gained with the use of technology. One ofthe best ways to demonstrate business value is by analyzing an IT project’s effi-ciency and effectiveness metrics.


FIGURE 1.21

Web Site Metrics

Hits

Qualified hits

When visitors reach a Web site, their computer sends a request to thesite’s computer server to begin displaying pages. Each element of arequested page (including graphics, text, interactive items) is recorded bythe Web site’s server log file as a “hit.”

Exclude less important information recorded in a log file (such as errormessages, etc.).

Visitor Visitor Metrics

Exposure Exposure Metrics

Hit Hit Metrics

A visitor is an individual who visits a Web site. An “unidentified visitor”means that no information about that visitor is available.

A unique visitor is one who can be recognized and counted only oncewithin a given period of time. An accurate count of unique visitors is notpossible without some form of identification, registration, orauthentication.

A session ID is available (e.g., cookie) or inferred by incoming addressplus browser type, which allows a visitor’s responses to be tracked withina given visit to a Web site.

An ID (e.g., cookie) is available which allows a user to be tracked acrossmultiple visits to a Web site. No information, other than a uniqueidentifier, is available for a tracked visitor.

An ID is available (e.g., cookie or voluntary registration), which allows auser to be tracked across multiple visits to a Web site. Other information(name, demographics, possibly supplied voluntarily by the visitor) can belinked to this ID.

Unidentified visitor

Unique visitor

Session visitor

Tracked visitor

Identified visitor

The number of times a particular Web page has been viewed by visitorsin a given time period, without regard to duplication.

The number of visitor sessions at a Web site in a given time period,without regard to visitor duplication.

Page exposures(page-views)

Site exposures

The length of time a visitor spends on a Web site. Can be reported as anaverage in a given time period, without regard to visitor duplication.

The total number of pages a visitor is exposed to during a single visit to aWeb site. Can be reported as an average or distribution in a given timeperiod, without regard to visitor duplication.

The total number of unique pages a visitor is exposed to during a singlevisit to a Web site. Can be reported as an average or distribution in agiven time period, without regard to visitor duplication.

Visit Visit Metrics

Stickiness (visitduration time)

Raw visit depth (totalWeb pages exposureper session)

Visit depth (totalunique Web pagesexposure per session)

haa83019_unit01 10/22/04 1:29 PM Page 34


Jonathan Abrams is keeping quiet about how he is going to generate revenue from his Website, Friendster, which specializes in social networking. Abrams is a 33-year-old Canadiansoftware developer whose experiences include being laid off by Netscape and then movingfrom one start-up to another. In 2002, Abrams was unemployed, not doing well financially, andcertainly not looking to start another business when he developed the idea for Friendster. Hequickly coded a working prototype and watched in amazement as his Web site took off.

The buzz around social networking start-ups has recently been on the rise. A number ofhigh-end venture capital (VC) firms, including Sequoia and Mayfield, have invested more than$40 million into social networking start-ups such as LinkedIn, Spoke, and Tribe Networks.Friendster received over $13 million in VC capital from Kleiner, Perkins, Caufield, Byers, andBenchmark Capital, which reportedly valued the company at $53 million—a startling figure fora company that had yet to generate even a single dime in revenue.

A year after making its public debut, Friendster is one of the largest social networking Websites, attracting over 5 million users and receiving over 50,000 page-views per day. The ques-tion is how do efficiency metrics, such as Web traffic and page-views, turn into cash flow?Everyone is wondering how Friendster is going to begin generating revenue.

The majority of Abrams’s competitors make their money by extracting fees from their sub-scribers. Friendster is going to continue to let its subscribers meet for free but plans to chargethem for premium services such as the ability to customize their profile page. The companyalso has plans to extend beyond social networking to an array of value-added services suchas friend-based job referrals and classmate searches. Abrams is also looking into using hishigh-traffic Web site to tap into the growing Internet advertising market.

Abrams does not appear concerned about generating revenue or about potential competi-tion. He states, “Match.com has been around eight years, has 12 million users, and has spentmany millions of dollars on advertising to get them. We’re a year old, we’ve spent zero dollarson advertising, and in a year or less, we’ll be bigger than them—it’s a given.”

The future of Friendster is uncertain. Google recently offered to buy Friendster for $30 mil-lion even though there are signs, both statistical and anecdotal, that Friendster’s popularitymay have peaked.21

Questions1. How could you use efficiency IT metrics to help place a value on Friendster?2. How could you use effectiveness IT metrics to help place a value on Friendster?3. Explain how a venture capital company can value Friendster at $53 million when the

company has yet to generate any revenue. 4. Explain why Google would be interested in buying Friendster for $30 million when the

company has yet to generate any revenue.

Chapter Four Case: How Do You Value Friendster?

1. Formulate a strategy for how Levi’s can use efficiency IT metrics to improve its business.

2. Formulate a strategy for how Levi’s can use effectiveness IT metrics to improve its business.

O P E N I N G C A S E S T U D Y Q U E S T I O N S

haa83019_unit01 10/22/04 1:29 PM Page 35

Documents

Measuring the Success of Strategic Initiatives