L643: Evaluation of Information Systems

Week 9: March 3, 2008

2

Announcement Guest lecture by Dr. Jonathan Plucker, Director of

the Center for Evaluation and Education Policy on April 14th

In Week 11, choose one of the following articles: Hernon, P., & Calvert, P. (2005). E-serve quality in

libraries: Exploring its features and dimensions. Library & Information Science Research, 27(3), 377-404.

Negash, S., Ryan, T., & Igbaria, M. (2003). Quality and effectiveness in web-based customer support systems. Information & Management, 40(8), 757-768

3

Exhibit 1: Hilton’s First Kiosk

4

Exhibit 2: Hilton’s Improved Kiosk

5

Hilton’s Self-Service Kiosk Case Study Bill, the VP for Customer-Facing Technology,

suggested to conduct a pilot study of the kiosk and ordered to deploy 14 kiosks in New York & Chicago

Bill asked you to come up with a proposal for evaluation studies. What would you propose?

6

Updated D&M IS Success Model (2002, 2003)

InformationQuality

System Quality

ServiceQuality

IntentionTo Use

Use

UserSatisfaction

NetBenefits

Creation Use Consequences

7

Updated D&M IS Success Model (2002, 2003)

InformationQuality

System Quality

ServiceQuality

IntentionTo Use

Use

UserSatisfaction

NetBenefits

Creation Use Consequences

8

IS Success Model (DeLone & McLean, 1992)

9

Technology-to-Performance Chain(Goodhue & Thompson, 1995)

Precursors ofutilization

Taskcharacteristics

Technologycharacteristics

Task-technology

fit

Utilization

Performanceimpacts

Individualcharacteristics

Expected consequences

Affect toward use

Social norms

Habits

Facilitating conditions

Feedback

Feedback

10

Summary Productivity concerns: Is the worker more

productive because s/he is using the system? Easy to determine for factory-like information work,

e.g., data entry More is better, so more items processed = more

productivity Generally not true of professionals and managers

Quality is of greater concern than quantity Measurement of productivity of knowledge workers

Generally based on their own perceptions of their productivity

11

IS Success Model (DeLone & McLean, 1992)

12

Measuring Payoffs of IT Investment (Kohli & Sherer, 2002)

IT Expenditure IT Assets IT Impacts OrganizationalPerformance

The IT ConversionProcess

The IT UseProcess

The competitiveProcess

Figure 1, p. 247

Technical risk Project risk

Organizational/Political risk

Competitive riskDisaster/security riskCollaborative risk

13

Measuring Payoffs of IT Investment (Kohli & Sherer, 2002)

Operational IT Managerial IT Strategic ITInvestment Financial

investment inFTE’s (employees)Equipmentconsulting

Financial investment & budgeting forApplicationsTrainingEducation

Financial investment & budgeting forCollaborative technologiesElectronic data interchangeERP

IT Assets # ofWorkstationsModemsInformation kiosksTrainers

Process redesign projectsPerson hours investedChange management initiatives

Hubs and routersKnowledge-based applicationsTeams working on strategic systemsIndustry & vendor partnerships

14

Measuring Payoffs of IT Investment (Kohli & Sherer, 2002)

Operational IT Managerial IT Strategic ITIT Impacts # of

Customers servicedHits on the websiteReturning customers

# ofMissed deadlinesReporting errorsMid-project process redesignsProduct recalls

# ofActual usage by period by userExtend of integration of IT into corporate decision making such as reports requested

Organiza-tional impacts

ProfitabilityROIROA

Employee turnoverMaintenance expensedowntime

Market shareRankingIndustry awardsCustomer service ratingStock price

15

Performance Impacts of IT (Devaraj & Kohli, 2003)

Independent Variablesof Technology Usage•# of report executed•Processing (CPU) time•# of records accessed

Control Variables•Midicare•Medicaid•Casemix•Patient income•# of employees•Age of hospital•Outpatients

Dependent Variables of Hospital Performance•Mortality•Revenue per admission•Revenue per day

16

Proving Your Project’s Worth (Tynan, 2005) IT = utility/infrastructure

Invisible (Star & Ruhleder, 1996)

What’s your take-aways from this article?

Star, S. L., & Ruhleder, K. (1996). Steps towards an ecology of infrastructure: Design and access for large information spaces. Information Systems Research, 7(1), 111-134

17

Intranet Model and Metrics (Jacoby, 2007) Intranet Efficiency and Effectiveness Model

(IEEM) Front-end: user-based factors, e.g., accessibility

and site navigation Back-end: site-based factors, e.g.,

personalization, information search, etc. People, processes, and technology: knowledge-

worker-based factors, e.g., their vision, purpose, and products or service

18

Intranet Model and Metrics (Jacoby, 2007)

19

20

IT and the Modalisation of Organisational Behaviour (Corea, 2006)

Modalisation = the modification of an organization’s capacity to carry out desired actions

IT enables action as well as constraints action

21

IT and the Modalisation of Organisational Behaviour (Corea, 2006)

Being a desired stateBeing the reverse/opposite (of the desired) state

Not being a desired stateNot being the reverse/opposite (of the desired) state

22

IT and the Modalisation of Organisational Behaviour (Corea, 2006)

Having to Act Having Not to Act

Not Having to ActNot Having Not to Act

Causing to Act Causing Not to Act

Not Causing to Act

Not Causing Not to Act

Legitimization Power

MotivationWanting to Act Wanting Not to Act

Not Wanting to ActNot Wanting Not to Act

23

Group Activity Work individually to fill out the worksheet Form teams based on the Group Project Propose a new model to measure

organizational impacts of IS Tell us how you can apply the model to

evaluate organization impacts of a real-world IS (your case)

Give a name to the model