Operating systems, compilers, assemblers and application programs: audit trails of user satisfaction

Operating systems, compilers, assemblers and application programs: audit trails of user satisfaction

In their bid to outdevelop and outsell competitors, do computer manufacturers really know what their customers want? Avi Rushinek and Sara F Rushinek's rigorous statistical

survey could hold some surprises for so-called market leaders

The influence of manufacturer's software on overall computer user satisfaction, as determined by multiple regression, is analysed. Operating systems, number of systems and number of users are found to be the most significant variables affecting overall user satisfaction. The average life of a computer system (in months) and the microcomputer (as the type of installation) have the least significant effect in the user satisfaction equation.

An interactive expert system for manufacturer's software is designed, developed and demonstrated. The application of such an expert system is explained along with the use of artificial intelligence techniques. The implications for improvements in manufacturer's software by updates to an artificial-intelligence- based expert system database are discussed and analysed.

microsystems multiple-regression analysis expert systems

The term 'manufacturer's software' (MS) encompasses operating systems, compilers and assemblers, and application programs 1. Users, to whom MS is of particular concern 2, generally complains that the computer industry does not provide adequate software for them. These complaints, however, have been challenged by industry 1. The controversy intensifies as systems become increasingly more complicated and MS variables become increasingly important. Despite its growing importance, the contributions of MS to overall user satisfaction have not yet been determined. This study quantifies these contributions and summarizes the MS factors of various computer systems.

Various authors have investigated the role of performance measurement in relation to operating systems as one way of enhancing MS to users 3. Several advantages of performance management and MS

University of MiamL Coral Gables, FL 33124, USA

have been cited in the literature, but their exact impact on users' overall satisfaction is still in question 2.

Progress has been made in assessing users' perceptions in the quality of interactive systems 4. Applications of artificial intelligence (AI), expert systems (ESs) and MS have been investigated s' 6. However, these studies have not been fully integrated and do not influence user satisfaction.

The principal objective of this study is to build prior theoretical work by integrating AI, ES and MS and measuring the impact of MS on user satisfaction. The results of a system rating study are presented, in which users were asked to respond to questions on MS. These questions (or independent variables) were based on the literature, and considered the primary determinants of overall user satisfaction (the dependent variable).

Multiple-regression analysis was used to relate user satisfaction to the MS variables. On the basis of this analysis, an ES was designed for

Avi Rushinek is an associate professor of accounting and information systems at the Uni- versity of Miami, USA. He holds a

PhD from the University of Texas at Austin, USA. His interests include accounting information systems, managerial~cost accounting, EDP auditing, and business applications for mainframe, mini and microcomputers.

Sara Rushinek is currently an associate professor of computer information systems in the Department of Management

Science at the University of Miami, USA. She received her PhD from the University of Texas at Austin. Her current interests are in the area of computer- assisted instruction, computerized management information systems, database management systems, programming languages, research methods and statistics.

forecasting the level of user satisfaction in a specific computer installation, and then comparing this level with industry standards. Using ES techniques and AI, the system can help users isolate problem areas and suggest solu- tions, while constantly updating user satisfaction files through tele- communication networks.

0141-9331/85/05241-09 $03.00 © 1985 Butterworth & Co. (Publishers) Ltd

vol 9 no 5 june 1985 241

Main objectives

The major objectives of this study were

• data collection and evaluation of variables

• statistical quantif ication of the contr ibut ion of MS (along with other variables) to user satisfaction

• design of the ES to evaluate MS • database update and retrieval

through telecommunicat ion networks

It is important to discover which MS features are most desired by users. A survey of the literature reveals that overlooking of users' desires by uninformed managers leads to many problems (under- ut i l ization of systems, disputes over program control and ownership, and problems with data security). The lack of understanding of the desirable features is a barrier to a computer system's effective operat ion' . Manufacturers could alleviate some of these problems with the use of ESs.

ESs are specialized decision aids, which provide quantitat ive and qualitative analysis, probabil ist ic estimates and il lustrative explan- ations 8. AI facilitates the use of ESs through computerized, frequent ly updated databases 9' 10

The present study applies ES and AI techniques to the impact of MS on user satisfaction. Accordingly, an ES based on artificial intel l igence (ESAI) is introduced to evaluate MS.

Design of the ESAI required several steps. First, the theories of consumer (user) satisfaction were reviewed to define the determinants of user satisfaction. A questionnaire was sent to users to score these determinants (Appendix I) . The determinant scores were regressed against the satisfaction scores to construct the model and to com- pute the industry standards. Then the model was tested statistically to decide whether it was applicable to the populat ion of users. Finally, if it proved applicable, it was incor- porated into an interactive onl ine ESAI to aid users and staff in terms of their MS services.

Factors affecting user satisfaction

Several confounding variables affect user satisfaction. These include expectations, type of system used and system popularity.

The effects of consumer expectations on product satisfaction have been discussed extensively in consumer-behaviour literature 11-14. Postulates from consumer-behaviour theory have been extended to the computer industry where expectations inf luence the att i tudes of users. Unrealistic expectations about what computers can do contr ibute to potential diff iculties and dissatisfaction of users 7'1s-17. The degree of meeting the users' needs and expectations def ini tely affects user satisfaction with various systems TM. Thus the degree of meeting user expectations is incor- porated into the present ESAI system.

A more controversial variable attr ibuted to user satisfaction is the type of computer used. Some previous studies have failed to make a clear dist inction among micro, mini and mainframe computers 19. More recent articles have made a distinction among computer types, especially focusing on mini /microcomputers 2°-23. They have inferred that users favour microcomputers over mini and mainframe computers, because of their greater perceived sense of control, simplicity, afford- ability, portabi l i ty and privacy. On the basis of these previous studies, the type of computer used should affect user satisfaction.

In general, the power and capabilit ies of computer systems have improved; memory costs have gone down and performance dis- t inctions among systems have blurred 24. This has made it even more diff icult to differentiate the effects of micros, minis and mainframes on user satisfaction. Accor- dingly, this study's ESAI model distinguishes among system types in an attempt to determine whether this dist inction is significant.

Setting generally accepted computer standards still remains a major controversy wi th in the computer industry. For example, few

standards are available in the microcomputer market 2s Under such circumstances, the most popular can become a de facto standard. Therefore the number of users, the number of computer systems, and their average system life (measures of popularity) are considered important determinants of user satisfaction; the assumption is that the more popular the system is, the better the MS is, and therefore the greater ts the increase in user satisfaction.

Computer system buyers ought to look at tradeoffs in cost, storage, reliabil i ty etc. Another approach to this problem is simply to go by the populari ty and the size of the vendors (well established and larger firms versus newer and smaller firms) 26. In other words, buyers may expect the most popular system (especially within a certain application or profession) to be indica- tive of the qual i ty or usefulness of that system. The sayings 'go with the flow' or ' fol low the crowd' have frequently been applied to the selection of computers. The proml- nence of IBM in the computer world supports this notion 22. Therefore this study includes several measures of system popularity, number of systems, number of users and system life.

Finally, system rating information can be a useful tool for managers evaluating MS for their computer systems, as well as for vendors who have to decide what kind of MS to create. Such ratings are provided here, and an approach to updating information and sharing ~t throughout an organization or an entire industry (by using a database management system and telecommunicat ion network) is pro- posed.

Survey methodology and data collection

The 15 218 questionnaires on which this survey is based were mailed to a carefully control led nth sampling from specific subsets of computer users' lists. To improve the response rate (and thereby increase the statistical validity)

242 microprocessors and microsystems

users were contacted twice; the first request was followed three weeks later by a second request. The response rate was 32%, rep- resenting 4 597 users, who respon- ded to 4 870 questionnaires. (Some users evaluated more than one computer model and filled out more than one questionnaire.)

Judges invalidated 379 responses (I 78 users who rated two different computers at the same time, and another43 users who rated three or more different systems at the same time). Datapro 27 batched the remaining 4 448 valid returns by vendor, model, users and computer types (mainframes of plug- compatible mainframe computers (maxis), minicomputers and small business computers (minis), and desktop personal and microcomputers (micros)) as shown in Table I.

Each questionnaire allowed the user to rate only one system. Recipients were told to reproduce the form to rate more than one system. Responses were averaged and recorded for each system. Labels were used as initial valida- tion vehicles and for identification and elimination of duplicate returns. Participants were asked to summarize their experiences with computer systems currently being used and to answer questions about their systems.

Methods and procedures

Evaluations of 179 computer systems resulted from the manual survey (Appendix 1). The responses to the questions (variables) were coded and stored on the computer

Table 1. Breakdown of survey samples

Subjects Maxis Minis Micros Total

Users* 1 919 2 192 337 4 448 Computers t 67 93 119 179 Vendors t 10 28 17 55

*Number of users was planned and controlled. tBreakdown of computers and vendors was a side product.

Table 2. Variable legends

Number of computer users sharing the system Number of computer systems at your site Average life of computer systems in months Microcomputer-based systems Minicomputer-based systems Mainframe-computer-based systems Manufacturer's software - -operat ing system Manufacturer's so f twa re - compilers and assemblers Manufacturer's software - - applications programs Systems meeting user expectations

No. Users No. Systems Sys. Life Micro Mini Mainframe OS C&A AP User Expectation

Table 3. Multiple regression - - overall significance test for goodness of fit and analysis of variance

Analysis of Mult iple (R) 0.77 variance R 2 0.59 Regression

Adjusted R 2 0.56 Residual

Standard error 6.16 Critical F

DF Sumofsquares Mean square

9 9080.80 1008.98 169 6403.82 37.89

1.88 < F value 26.63*

*R 2 is significant at the 0.05 level.

(see Table 2). Data was tested for validity and consistency; eg the percentage values were checked for the range between 0 and 100. (Nonresponse bias was evaluated with an F test and found to be insignificant.)

Forward stepwise multiple- regression analysis was used to determine the relationship between the overall satisfaction (dependent) variable and the MS (independent) confounding variables.

The assumptions made for multiple regression were

• the relationships among the variables are linear and additive

• the variables have a multivariate normal distribution, equal variance, no multicollinearity, and no autocorrelation 28' 29

These assumptions have been met by the present sample. The distribution of the residuals (Figure 1) confirms some of the assumptions; the plot shows that the residuals (the deviations of the observed satisfaction scores from the estimated satisfaction scores) are randomly distributed without form- ing any specific abnormal patterns. Residuals also measure the error component and the possible viola- tion of the assumptions; examina- tion of the residuals demonstrates the absence of such violations (Figure 1).

Subsequent data collection and database updates were performed using an interactive online questionnaire (IOQ) (see the sample screen dump in Appendix 2). This method of data collection avoids the pitfalls of traditional manual questionnaires, such as: incomplete, illegible responses; nonresponse and sampling bias; low response rate; delays from data collection to data analysis; time-consuming error-prone data transcription and key-punch operation; disruption, resentment and anxiety produced in the respondent; and ambiguity in questionnaire items. The IOQ con- trols the above problems by valida- tion procedures. It also clarifies ambiguities through help files, whereby a respondent enters a '?' instead of an answer to obtain clarification.

vol 9 no 5 june 1985 243

R e s u l t s a n d d i s c u s s i o n

The dependent variable (overall satisfaction) was regressed over the criterion ( independent) variables by a stepwise procedure 3°-32. Table 3 presents the statistics used for the overall test for goodness of fit for step 9 of the regression model. This step was selected because each variable added to the model increased the mult iple (R) of the model, whi le having an overall F value statistically significant at the 0.01 level. This table shows R, R 2, the standard error and an analysis of variance for the regression model.

The tests indicate that 77% of the variation in overall satisfaction is explained by the independent variables examined here. The standard error of the estimate at the final step is 6.16; this means that, on average, predicted overall satisfaction wil l deviate from the actual scores by 6.16 units on the overall satisfaction scale.

The relative importance of each of the predictor (or independent) variables on the predicted (or dependent) variable is indicated in Table 4. This relative importance is described by the change (18) in satisfaction due to a change of one standard deviation in the predictor (criterion) variable value. These variables and their coefficients are the basis for the ESAI model used in the IOQ (Appendix 2).

According to Table 4, the fol low- ing model describes the overall satisfaction as a function of the criterion predictor variables, in descending order of their 18 values

Overall satisfaction = 0.47(05) - 0.22(No. Users) + 0.19(User Expectation) - 0.14(Mainframe) + 0.08(Micro) + 0.28(No. Sys- tems) + 0.20(C&A) + 0.18(AP) - - 0.10(Sys. Life)

The ranking of the independent variables affecting the overall satisfaction of this ES for MS reveals some interesting results. First, it appears that the variables OS, No. Systems, and number No. Users (ranked 1, 2 and 3) have the strongest effects (the largest values) on the dependent variable.

-2 .0 -1 .0 C.O 1.0 2.0

• ~ . . . . . . I . . . . . . XY. Y I Y X* * * I * X

2.0 + I * +

I I I

I I I

I I I

I I I

I * I I

I I* * I

I I * I

I 2 I * I

I * ** * I ** ** I

1.o + I * * +

I 2 I * * I

I * * I * * I

I * *I I

I * * * I * ** ** I

I * *I* * * * *I

I * * * I* * * ** * * I

I * 3 * 2 * * ** I

I 2 I* * * ** I

I ** *I* 2 * I .0 t * . . . . * - - * 2 * ' I

I 2 . . 2 1 2 2 ** 2 I

I ** ,2 • ** .4, . I

I * * I 2 * * I

I * * * * I*'3"2 2 * I

I * * *I * * I

I ** * I * I

I I ** * I

I * I I

I I* I

--1.0 + * * *I +

I * I* I

I I * I

I * * I * * I

I I I

I * I I

I * I * I

I * *I * I

I I * I

I I I

--2.0 + I +

X* * * I X y * • . y

.YX! ) i , i " t ~ ' .

-2 .0 -1 .0 C.O 1.0 2.0

Figure 7. Residuals scatterplot diagram: standard residuals (downward) on the vertical axis; predicted standardized dependent variable, system user's satisfaction, (across) on the horizontal axis; rows, columns Y - - values out- side (-3.0, 3.0); rows, columns X - - values in (-3.0, -2.05)


Contrary to the literature, the factor indicating computer usage (Micro) is less important.

The variables OS, User Expecta- tion, C&A, No. Systems and AP were found to have positive effects on overall satisfaction; while Sys. Life, No. Users and Mainframes have a negative effect. The negative coefficient of the Sys. Life variable may be explained by the rapid techno- logical changes that occur. The longer the system life, then the greater the technical obsolescence, and this obsolescence may account for the decrease in user satisfaction. The negative coefficient for mainframes may be explained by the increasing popularity and capabilities of minis and micros.

As can be seen in Table 4, inclusion of all nine variables has an influence on R 2, indicating that these variables should be incor- porated in the model. The variable OS has the greatest relative importance to the MS satisfaction model.

The only variable not included in the regression was Mini. This variable did not contribute to the R and the overall F value of the regression model. This implies that the system's popularity is not such an important factor in determining user satisfaction as the literature has suggested; ie although a system is popular it may have a relatively low level of user satisfaction (eg where the vendor has a monopoly).

Table 5 shows the significance test for specific coefficients of the model. The coefficients in Table 5 show the R 2 contribution to the model due to the addition of each variable. The R 2 change determines the inclusion sequence.

Expert systems diagnostics for evaluation of manufacturer's software

Traditionally, experts use survey questionnaires to evaluate MS. Such questionnaires are usually administered through interviews. These manual software evaluation methods have numerous disadvantages. In addition to being disrup- tive, manual data collection is

frequently error prone, incomplete, Diagnostic audit trails reports illegible, sampling biased, ambiguous, expensive, time consuming The ESAI interactively interrogates and very inefficient, the user concerning the MS system.

In contrast, online interactive User responses are underlined and data collection offers many advan- recorded anonymously in a data- tages. These include time saving by base. Subsequently, the ESAI both the expert and the users, generates the MS diagnostics audit improved accuracy reliability, trail. This report trails after the higher user response, complete interactive questionnaire, providing and legible data, controlled the user with immediate feedback. sampling, reduced error rate, and Later, it may also be used by an reduced transcription and paper internal or external auditor, manage- work. Moreover, the ambiguity can ment or staff members for system be reduced by providing online development. This audit trail com- help through the use of a '?' pares the user's installation scores to clarify ambiguous questions with ' industry standards' based on (Appendix 1). information from a frequently

Table 4. Significance for specific coefficients in the regression equation

Variable* B /~ Rank ~ Standard F error in B

OS 0.447 -t- 0 0.468 1 0.058 59.782 § AP 0.122 -t- 0 0.175 6 0.038 10.239 § User Expectation 0.113 -t- 0 0.194 5 0.031 12.955 § Mainframe -0.263 -I- 1 -0.135 7 1.056 6.218 § C&A 0.149 -t- 0 0.199 4 0.045 11.226 § Sys. Life -0.155 + 0 -0.102 8 0.076 4.122 ~ Micro 0.242 + 1 0.078 9 1.674 2.089 § No. Systems 0.527 + 1 0.279 2 6.404 0.678 No. Users - 0 . 4 6 0 + 1 -0.217 3 7.186 0.410 (Constant) 0.130 + 2

"Ranked in descending order of contribution to the explained variance (R 2 change) f ranked according to/~, which indicates the change in satisfaction due to a change of one standard deviation in the respective variable t'rhis F value is significant at the 0.05 level.

Table 5. Multiple-regression summary table -- significance test for subsets of the model

Variable Mult iple R 2 R 2 Simpl. B /~ (R) change* R

OS 0.666 0.444 0.444 0.666 0.447 + 0 0.468

AP 0.698 0.487 0.044 0.423 0.122 -I- 0 0.175

User Expectations 0.718 0.515 0.028 0.361 0.113 + 0 0.194

Mainframe 0.734 0.538 0.023 -0.061 -0.263 + 1 -0.135

C&A 0.752 0.565 0.027 0.495 0.149 -t- 0 0.199

Sys. Life 0.759 0.575 0.010 -0.164 -0.155 -t- 0 -0.102 Micro 0.762 0.581 0.006 -0.003 0.242 + 1 0.078 No. Systems 0.765 0.585 0.004 0.103 0.527 -t- 1 0.279 No. Users 0.766 0.586 0.001 0.099 -0.460 + 1 -0.217

(Constant) 0.130 + 2

*Primary key for forward stepw=se inclusion of criterion variables

vol 9 no 5 june 7985 245

updated database. (A sample diagnostic audit trail is given in Appendix 3).

The ESAI sorts the report items to reflect the weaknesses ( - ) or strengths (+) of a specific installation relative to the industry. It generates a current overall user satisfaction score, compares it with the prior score and computes the gain or loss in overall computer user satisfaction.

The ESAI distinguishes the change in overall satisfaction, and it identifies the sources of the change. On this basis, it also generates prioritized recommendations for further improvements (Sample recommendations are shown in Appendix 4.) The responses of the user, along with the diagnostics audit trail, are stored in a trans- action file and are eventually merged with the old database master file to form the updated master file. This information is updated and shared via telecom- munication networks, without dis- closing confidential information such as installation and/or user identification.

Summary, conclusions and implications

Multiple regression has been used to study the dependence of overall user satisfaction with computer systems as it relates to several MS independent variables. Overall significance tests of the goodness of fit of the model have been conducted. The multiple correlation coefficient was found to be 0.77, and the null hypothesis stating that the correlation equals 0 has been rejected.

The variables were rank ordered according to their/8 values. OS, No. Systems and No. Users were ranked the highest, while Mainframe, Sys. Life and Micro ranked lower. In fact, mainframes had a relatively large negative effect on the dependent variable compared with microcomputers. This may be explained by users attributing negative feelings to their mainframes in terms of their frustration with the down time, lack of versatility or portability, lack of

privacy, lack of control over their system and insufficient MS.

In contrast, micros had a positive effect, which may be explained by relatively simple MS requirements. This supports some past research which states that the users may feel micros are becoming more power- ful in their byte capacity, therefore enhancing the user satisfaction level.

In conclusion, it appears that overall satisfaction with computer systems is greatly affected by operating systems, number of systems, and number of users. System life, and the fact that the system is a micro, have a smaller effect on the overall satisfaction. This means that the additional variables do not contribute significantly to the overall satisfaction of the computer system.

The implications of the present study are many. Overall satisfaction of computer systems can be measured by answering certain questions. Resulting information can be very useful to computer users, potential buyers and vendors. Buyers could compare different computers (cross-sectional), calcu- late overall satisfaction and estimate the satisfaction that they would derive by buying a given system.

The vendors of the computer systems could maximize sales levels by incorporating and promo- ting the MS variables which increase the overall satisfaction of their products. This could lead to new and better MS products. For example, vendors might provide their users with toll-free telephone numbers for operating systems, live hands-on demonstrations of software to raise realistic expectations, and information about the performance measurement potential power of the operating system in question.

Vendors would be able to use data from online users as a market- ing tool for their products. They could advertise their high standing in user satisfaction studies to give them an advantage over their competitors.

This ESAI for MS also provides corporate users, computer staff

members and management with an effective tool for various MS policies and procedures. Data processing managers and computer centre directors could monitor, evaluate and upgrade the MS they provide to end users and thereby improve user satisfaction. Users and management could evaluate the levels of satisfaction at their installation, compare it with market standards, and identify weaknesses and strengths. Moreover, management could apply remedial action to improve the satisfaction of their users, and gauge their progress by running the ESAI on a regular basis.

A computerized ES can analyse data immediately after it has been entered, thus providing immediate feedback and diagnostics to the user. Research and development costs, time and effort are major disadvantages of such an ESAI. Frequent use and updating could, however, make such an ESAI cost effective.

References

1 Immel, R F'Operating systems' Popular Comput. Vol 12 No 3 (December 1983) pp 49-54

2 Keefe, P 'Micro operating systems: a crowded stage' Computerworld Vol 18 No 6 (February 1984) pp 49-54

3 Kovach, R and Iselbers, A 'Per- formance-measurement software pinpoints system bottle- necks' Mini-Micro Syst. Vol 18 No 2 (April 1984) pp 157-164

4 Dzida, W, Herda, S and Itzefeidt, D 'User-perceived quality of interactive systems' IEEE Trans. Software Eng. Vol 4 No 4 (July 1978)

5 Kearsley, G 'The relevance of AI research to CAI' Res. Rep. DERS-O6-O4-RIR-77-2 (August 1978) pp 1-25

6 Hansen, l V and Messier, W 'Expert systems for decision support in EDP auditing' Int. J. Comput. Inform. Sci. Vol 11 No 5 (October 1982) pp 357-379

7 Stair, R M 'Using in-house computers: some basic tips' Assoc. Manage. Vol 33 No 10 (October 1981) pp 143-144


8 Stefik, M 'The organization of expert systems' Artif. Intell. Vol 18 (March 1982) pp 135-173

9 Duda, R O and Gasching~ J G 'Knowledge-based expert systems come of age' Byte Vol 6 No 9 (1981) pp 238-281

10 Wong, H K and Mylopoulos, J 'Two views of data semantics: data models in artificial intelligence and database management' INFOR J. (Can. J. Oper. Res. Inf. Process.) Vo115 No 3 (1977)

11 Resnick, A J and Harmon, R R 'Consumer complaints and managerial response' J. Market. Vol 47 No 1 (1983) pp 86-97

12 luster, T F 'An expectational view of consumer spending prospects' J. Econ. Psychol. Vol 1 No 2 (June 1981) pp 87-103

13 Taylor, J L and Durand, R M 'Effect of expectation and dis- confirmation on postexposure product evaluations' Psychol. Rep. Vol 45 No 3 (December 1979) pp 803-810

14 Olshavsky, R W and Jaffe, B L 'Responsiveness of consumer expectations and intentions to economic forecasts: an experi- mental approach' Rev. Econ. Slat. Vol 63 No 2 (May 1981) pp 98-102

15 Good, R E and Jenkins, K M 'Managing with microcomputers' Business Vol 33 No 1 (March 1983) pp 37-43

16 Scannel, T 'Singer rated tops in mainframe survey/Users find software support problem areas' Computerworld Vol 16 No 23 (June 1982) pp 56-63

17 Faerber, L G and Ratliff, R L 'People problems behind MIS failures' Financial Executive Vol 84 No 4 (April 1980) pp 18-24

18 Grilz, A F 'Designing a success- ful user computer dialogue' Computerworld Vol 15 No 5 (9 March 1981) pp 13-16

19 Gates, P O 'How can account- ants and auditors help prevent computer fraud?' Government's Accountant J. Vol 27 (Summer 1978) pp 10-15

20 Claret, J'Microtechnology: friend or foe?' Manage. Account. (May 1982) pp 18-19

21 Bales, A 'Choosing a micro? Here's a cautionary tale' Accountancy (February 1982) pp 98-101

22 Farmer, D F 'Comparing the 4341 and M80/42' Computer- world (February 1981) pp 9-20

23 McGrath, ME'Howto cope with the microcomputer revolution' Pract. Accountant (January 1982) pp 46-49

24 Sample, R L 'Minis--moving beyond the small business user' Admin. Manage. (September 1981) pp 58-64

25 Canion, R'Few standards available in micro market' Computer- world Vol 17 No 13 (28 March 1983) pp 9-14

26 Cheney, P H'Selecting, acquiring and coping with your first computer' J. Small Business Manage. (January 1979) pp 43-50

27 Datapro Research Corp. User ratings of computer systems (1984)

28 Kerlinger, F N and Pedhazer, E Multiple regression in behavioural research Holt, Rinehart and Winston, New York, USA (1973)

29 Overall, J E and Klett, CApplied multivariate analysis McGraw- Hill, New York, USA (1973)

30 Theil, H "Principles of econo-

metrics Wiley, New York, USA (1971)

31 Nie, H N, Hull, C H, Jenkins, J F, Steinbrenner, K and Bentl, D H Statistical package for the social sciences McGraw-Hill, New York, USA (1975) pp 468-514

32 Gordon, R A'lssues in multiple regression'Am. J. Sociol. Vo173 (1968) pp 592-616

Bibliography

Adelson, S 'Computerized market support systems are the way of the future' Sales Market. Manage. Can. Vol 24 No 12 (December 1983) pp 17-18 Barcus, S W and Boer, G B 'How a small company evaluates acquisi- tion of a mini-computer' M~nage. Account. (March 1981) pp 13-23 Barr, A et al. 'The computer as a tutorial laboratory' Int. ]. Man- Mach. Stud. Vol 8 No 5 (1976) pp 576-596 Bilbrey, C P and House, W C'Mini- computer selection' J. Syst. Manage. (July 1981) pp 36-39 Kleinrock, L 'A decade of network development'J. Telecommun. Net- works Vol 1 No 5 (Spring 1982) pp 1-11 McDermott, J 'A rule-based con- figurer of computer systems' Artif. IntelL Vol 19 (September 1982) pp 39-88 Pournelle, J 'The operating systems jungle' Popular Comput. Vol 18 No 2 (June 1984) pp 81-86 Pournelle, J 'Clearing a path' Popular Comput. Vol 19 No 2 (July 1984) pp 81-86 Turney, P E and Laitala, P H 'A strategy for computer selection by small companies' Manag. Plan. (November-December 1976) pp 24-29

Appendix 1: Manual questionnaire

The objective of this questionnaire is to identify and quantify the weaknesses and the strengths of your MS.

Your computer vendor is: Model: Others (Y/N):

ENTER YOUR RESPONSES:

I. Number of computer users sharing this system? . . . . . . . . . . . . . . . 2. Number of computer systems at your site? . . . . . . . . . . . . . . . . . . . . 3. Average life of these computer systems in months? . . . . . . . . . . . .

vol 9 no 5 june 1985 247

IS YOUR CURRENT MS:

4. Micro computer based (Enter Y/N)? . . . . . . . . . . . . . . . . . . . . . . . . . . . 5. Mini computer based? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6. Mainframe computer based? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CONCERNING MICROCOMPUTERS (EXCLUDING MINISAND MAINFRAMES), PLEASE RATE 1% FOR VERY B A D . . . T H R U . . . 1 0 0 % FOR VERY GOOD

7. M.S. Operating Systems? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8. M.S. Compilers and Assemblers? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9. M.S. Application Programs? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10. Systems meeting user expectations? . . . . . . . . . . . . . . . . . . . . . . . . . .

Please enter additional comments

Appendix 2: Interactive online questionnaire sample screen dump (user entries are underlined)

WELCOME TO THE INTERACTIVE ON-LINE EXPERT SYSTEM FOR MANUFACTURER'S SOFTWARE (MS) DIAGNOSTIC PROGRAM -- AS OF 06/30/84

The objective of this questionnaire is to identify and quantify the weaknesses and the strengths of your MS. It will print out a prioritized deviations list of this installation from industry standards, and diagnostics AUDIT-TRAIL recommendations.

Your computer vendor is: IBM Model: PC XT Others (Y/N): N

ENTER YOUR RESPONSES CONCERNING EXCLUSIVELY MS ON THE IBM PC XT: (Enter a '?' when additional information is needed)

1. Number of computer users sharing this system? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1__55 2. Number of computer systems at your site? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3. Average life of these computer systems in months~ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.80

IS YOUR CURRENT MS:

4. Micro computer based (Enter Y/N)~ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Y 5. Mini computer based (if ~ 0 will rerun for Minis)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . N 6. Mainframe computer based (if ~ 0 will rerun Maxis)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . N

CONCERNING MICROCOMPUTERS (EXCLUDING MINIS AND MAINFRAMES), PLEASE RATE 1% FOR VERY B A D . . . T H R U . . . 1 0 0 % FOR VERY GOOD

7. M.S. Operating Systems? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61% 8. M.S. Compilers and Assemblers? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60% 9. M.S. Application Programs? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43%

10. Systems meeting user expectations? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55%

Please enter additional comments, and press two carriage returns:

I need help. The education/traininp~ is inadequate. Learnin~ on my own is very frustratin~ and I wish I could attend some workshops!

Would you like a MS Printout, Display, or Both (P/S/B)? P O.K., is your printer online and ready to print (Y/N)? Y_

** * * THE MS DIAGNOSTICS AUDIT TRAIL IS NOW BEING PRINTED ** * *

Appendix 3: Sample diagnostic audit trai l

1 EXPERT SYSTEM FOR MS DIAGNOSTIC AUDIT TRAIL FOR IBM PC XT**

2 06/30/84 INDUSTRY STANDARDS USER INSTALLATION 3 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ~ :It * :11 ~ :~ ~ ~ :* ~ ~ :11 :ll~lc:~ ~ ~ ~ :11 ~ ~ ~11 ~ ~ I I ~ ~ ~ :I1:1I ~ ~ :~ ~ ~ ~ :~ ~ ~ ~ ~ . ~ ~ ~ ~ ~ ~


4 COLUMN: A B C D=B*C E F=B*E G = F - D 5 VARIABLE B- AVER- AVG RES- MS CURRENT 6 NAME VAL AGE SCORE PONSE SCORE DEVIATE

7 0 S . . . . . . . . . . . . . 447 81 36.207 61 27.267 -8 .940 B AP . . . . . . . . . . . . . . 122 68 8.296 43 5.246 --3.050 9 C&A . . . . . . . . . . . . 149 78 11.622 60 8.940 -2.682

10 Expectations . . . . 113 85 9.605 71 8.023 -1.582 11 No. Systems . . . . 5.273 .27 1.424 .24 1.265 - .159 12 Micros . . . . . . . . . 2.420 .10 .248 .07 .169 - . 0 7 9 13 No. Users . . . . . . -4 .602 .22 -1.012 .15 - .690 .322 14 System Life . . . . - .155 3.90 - .605 1.80 - .279 .326 15 Mainframes . . . . -2 .633 .35 - . 9 2 2 .09 - . 2 3 7 .685 16 Constant . . . . . . . 12.964 . . . . . . . . . . . . 12.964 . . . . . . . . . . . . . . . . 14.237 1.273 31 Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77.B21)--k 63.941 -13.886

\ 32 INDUSTRY STANDARD FROM THE DATA BASE . . . . . . . . . . . . . . \ > 77.821 33 USER INSTALLATION OVERALL SATISFACTION SCORE . . . . . . . . . . . . . . . . . . . 63.941 34 LESS USER PAST SATISFACTION SCORE from 12/30/83 . . . . . . . . . . . . . . . . . . . -30.250 35 SATISFACTION SCORE GAIN SINCE 12/30/83 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33.691 36 Notes: 37 > Improvement from the last ESAI survey 38 < Degradation from the last ESAI survey 39 = No change since the last ESAI survey 40 * 41 G 42 # 43 **

Statistically significant deviate is greater than industry standard deviation Column G, the current deviate is sorted in ascending order Computers divided by the number of users

H PAST DEVIATE

~ :k :k 3k >k ~ :k3k :k:k ~ ~<

>--10.56" >-- 7.34* >-- 8.39 >-- 6.45 >-- 4.35 >-- 1.35 > .90 > .13 > .35 > .00 >-37.07

-->77.82

<--30.25

This program was developed on IBM PC XT interfacing with the Lotus 123 program (c) by Lotus Development Corporation

Appendix 4: Sample recommendations

ESAI FOR MS DIAGNOSTICS AUDIT TRAIL RECOMMENDATIONS AS OF 05/30/84

RANK PRIORITY DESCRIPTION AND SUGGESTED REMEDIAL ACTION (IBM PC XT)

1. Operating systems should be improved through the development of online help files that indicate performance measurements.

2. Application programs should be placed on line and used interactively.

3. More popular compilers and assemblers should be maximized to increase performance measurement.

4. Expectations should be brought down to a more realistic level, by quality-circle discussions and staff meetings.

5. An increase in the number of systems will increase user satisfaction.

6. Users need to be trained to retrieve data from the mainframes and transfer data to their microcomputer software, increasing microcomputer utilization.

7. User meetings should be organized frequently.

8. System life in mainframes should be improved to be more satisfactory.

9. More programs should be developed to download information from the mainframe onto the microcomputer, reducing the mainframe workload.

CLOSING REMARK BY ESAh

Keep up the good work folks!l!l Since 12/30/83, there have been major improvements in virtually every area, especially in the area of compiler and assemblers and user expectations.

vol 9 no 5 june 1985 249

Documents

Operating systems, compilers, assemblers and application programs: audit trails of user satisfaction