The Four Faces of Institutional Research

This chapter describes the purposes and roles ofinstitutional researchers–to describe the institution, to analyze alternatives, to present the best case, and tosupply impartial evidence—and summarizes the conflictsresearchers face as they attempt to fulfill them.

The Four Faces of Institutional Research

J. Fredericks Volkwein

The ancient Roman god Janus was the god of doors and gateways. Like thetwo sides of a door, Janus has two faces—one looking outward and onelooking inward. While Janus encourages us to consider both the externaland the internal aspects of our endeavors, Janusian thinking also remindsus that when we pass through a door we are simultaneously entering andleaving. Thus one action viewed in opposite ways recognizes the dual natureof almost everything we do. Commencement is both an ending and a begin-ning. For every action there is a reaction.

All individuals in colleges and universities experience such dualities.This chapter gives some prominent examples and suggests some conflict-ing roles that institutional researchers confront.

Dualities in Colleges and Universities

There are three kinds of opposing pressures in postsecondary institutionsthat are particularly relevant to institutional researchers: internal demandsmay contrast with external demands, the academic culture differs from theadministrative culture, and the institutional needs may vary from profes-sional needs.

Duality 1: The Internal Versus the External. Presidents and vice pres-idents direct much time and energy toward internal management. But they also interact with key people in the external environment and indeedattempt to shape that environment in order to maximize institutionalresources. When interacting with key external stakeholders (like trustees andlegislators), presidents must appear to be in strong control of the educational

NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH, no. 104, Winter 1999 © Jossey-Bass Publishers 9

1

10 WHAT IS INSTITUTIONAL RESEARCH ALL ABOUT?

organization and able to deliver on promised programs and initiatives. Backat home, however, presidents and provosts face major internal impedimentsto change: consensus governance, faculty control over the major goal activ-ities, an organizational culture that supports change by adding resourcesrather than by reallocating resources, and a curriculum structure that makesfalse assumptions about learner homogeneity. University presidents shouldbe forceful leaders, but they should never interfere with faculty democracy.Decisions should be made swiftly, but everyone should be consulted aheadof time. Clark Kerr complained that his faculty at the University of Califor-nia at Berkeley expected him to be a mouse at home but a lion abroad.

Duality 2: Academic Versus Administrative Cultures. Universitiesare fascinating organizations because they house at least two strong cultureswithin the same organizational structure. One is the administrative culture,the bureaucracy, where authority and responsibility are based largely onone’s position or rank. Among the administrative offices that compose thebureaucracy, communication and decision making are largely hierarchicaland based on division of labor and delegated responsibility. The other cul-ture, of course, is the faculty’s academic culture, the professional culture,where authority is based not so much on one’s position as on one’s knowl-edge or expertise. In the academic department, communication and deci-sion making are more collegial.

Moreover, the organization’s goal activities (teaching, research, service)are carried out by the academic culture, which places a high value on qual-ity and effectiveness. The administrative bureaucracy supplies the supportservices; this culture is much more cost conscious and places a high valueon efficiency. Thus faculty and administrators live in organizational culturesthat are importantly different. Their varied approaches and values often pro-duce organizational tensions for institutional researchers and others whomust operate in both cultures.

Because institutional research (IR) operates in both of these contrastingcultures, it may be thought of as a halfway house. Our efforts and analyticalwork are administratively directed because our data inform managerial deci-sion making—and we know who signs our paychecks. But some of our work(like faculty workload analysis and assessment) takes us into the academicculture. The IR office serves at times as a home for theory-driven social sci-ence research but more often as a practice-oriented detective agency. We aretrained as researchers, and some of us hire graduate assistants or draw uponfaculty expertise to help us with our research, in the same way that an acad-emic department does. But unlike the academic department’s work, ourresearch sometimes is for the president’s inner circle only.

Duality 3: Institutional Versus Professional Roles. Faculty in mostuniversities experience tension between their institutional role (teaching)and their professional role (scholarship). For the most part, faculty are hiredto teach particular subjects, but they are trained and rewarded, at least inuniversities, for their research and scholarship. Only occasionally do uni-

11THE FOUR FACES OF INSTITUTIONAL RESEARCH

versity faculty receive formal training for the role they have been hired for—teaching students. Moreover, the products of teaching are generally less vis-ible than the products of research, so most faculty are given promotion andtenure (and professional mobility) based largely on their research and schol-arship.

Institutional researchers may face a similar role tension when they arehired to produce accurate numbers and descriptive statistics about the cam-pus (the institutional role) but are trained for and find fulfillment in thechallenges of research and analysis (the professional role).

These three dualities constantly rub against each other, and the ten-sions they produce are greatly exacerbated by the conflicts among five pub-lic policy concerns.

Collision of Five Policy Concerns

An array of public policy issues intersect on the campus and challenge allof us in higher education. The first is a concern about the high cost of a col-lege education—a concern expressed by parents, college students, and tax-payers alike. Since 1970, tuition and fees on both public and privatecampuses have risen on average at a rate that is double the increases in theConsumer Price Index.

A second policy concern is the almost universal need for managementefficiency and increased productivity. Public concern about high cost mag-nifies concern about low productivity. Virtually every sector of the nation’seconomy has made substantial gains in productivity in the past thirty years;yet in higher education, salaries and cost per student have been rising atrates well above inflation. Colleges and universities use more resources inrelation to outputs than before, and economists call that declining produc-tivity. Many trustees and state legislators have business backgrounds, andfor them, the idea of restructuring and becoming more productive is a famil-iar necessity. However, few faculty see it this way.

The third public concern focuses on universities’ and colleges’ effec-tiveness. Most customers are willing to pay more for higher quality and bet-ter service, but it is not clear that higher tuition prices translate into higherquality. In fact there is ample evidence from employers and researchers alikethat many college graduates are not as well educated, nor as employable, asthey were in the past and as they need to be in the future. For example, theNational Adult Literacy Survey found that large numbers of two-year andfour-year college graduates are unable to use basic skills in everyday situa-tions involving reading, writing, arithmetic computation, and elementaryproblem solving. The Wingspread Group on Higher Education (1993) con-cluded that a bachelor’s degree seems to be a credential without content.

A fourth policy issue is access. The national value system in the UnitedStates demands that access to higher education should be provided to all.Most families see higher education as the vehicle for upward mobility and


a ticket to the “good life” for their children. Certainly in public higher edu-cation, student access is a significant if not controlling mission, but publicand private institutions alike have expanded their commitments to educa-tional opportunity and have used this commitment to justify additionalfunding.

The fifth policy issue is accountability. Both public and private institu-tions have stakeholders who do not necessarily agree that campus auton-omy is a good thing. At the very least, they believe that campuses need tobe held accountable for fulfilling their educational missions (Bowen, 1974).At the extreme, they believe that campuses need to be micro-managed inorder to ensure efficient operation and the attainment of strategic goals.Such views are in plentiful supply, not only at the trustee and local level butalso at the state and national levels.

When these five competing concerns about cost, productivity, access,effectiveness, and accountability all collide at the campus level, they createan array of campus tensions for planners, institutional researchers, andadministrators at all levels.

Tension 1: Campus Autonomy, Public Accountability, and Govern-ment Regulation. There has been much public discussion about the virtuesof deregulation and decentralized, customer-based management. The busi-ness literature is filled with the advantages of hierarchically flat organiza-tions that can move quickly and adapt to changing circumstances in themarket. Even government bureaucracies are being encouraged to reinventthemselves. Most elected officials now recognize the value of academicautonomy, or at least give lip service to the ideal of government deregula-tion in public higher education. The recent atmosphere in Washington,D.C., and most state capitals reflects a growing consensus that regulationand control can be both costly and unproductive.

Nevertheless, these impulses conflict with the philosophy of publicaccountability. The concept of accountability in a bureaucratic democracyjustifies the government’s attempts to enforce sound management practices(Yates, 1982). The government acts in the public interest in part becausehigher education is now viewed as a social and political right rather than aprivilege (Kerr, 1963). Access to higher education, facilitated by publicfunding, should be provided to every child and adult who seeks it. Thus,postsecondary institutions, as agents of this national policy, becomeaccountable to the general public interest (Mortimer, 1972). This perspec-tive creates a climate that justifies intrusive state and federal policies. Insome states, financial and personnel transactions at public colleges and uni-versities receive redundant scrutiny by central system offices, legislativecommittees, and state executive staff (Volkwein, 1987). Most politicians,not wanting to appear unresponsive, are quick to jump on the latest media-hyped story of some campus incident or indiscretion, and use it as anexcuse to pass yet another piece of legislation or reporting requirement. Sothere are now myriad state and federal regulations and reporting require-


ments related to affirmative action, Americans with disabilities, athletics,freedom of information, clean air, campus crime, financial accountingrequirements, fuel use and contamination, occupational health and safety,purchasing practices, personnel benefits, payroll transactions, researchinvolving human subjects, research involving warm-blooded animals, radi-ation safety, student financial aid, waste disposal, and student privacy rights,the list goes on. “Taken by itself, any single action may not be unbearablyintrusive, but the combined impact of many actions can nearly suffocate aninstitution” (Carnegie Foundation, 1982, p. 65). To make matters worse,the public and our elected officials then act surprised, perhaps even dis-mayed, when discovering that two-thirds of the postsecondary educationworkforce are engaged in activities outside the classroom.

Tension 2: Access, Excellence, and Accountability. Simultaneouspressures for access, excellence, and accountability create an enrollmentmanagement dilemma: colleges and universities experience opposite enroll-ment pressures—they are asked to simultaneously admit more students (forfinancial health and access) and become more selective (to bolster acade-mic standards and performance measures). Janus is alive and well.

On the one hand campuses need student enrollments (and the revenuethat comes with them) in the current competitive enrollment-driven econ-omy in which 90 percent of accredited institutions of higher education inNorth America admit over 90 percent of their applicants (Volkwein, 1995).Most, though not all, campuses have a self-interest in admitting and retain-ing a huge proportion of their applicants as a matter of economic survival.

On the other hand many applicants (though I hope not many at yourcampus) have poor records of prior achievement and are not good risks foracceptable academic performance and graduation. Moreover, faculty haveconcerns about academic standards—concerns that are usually reflected inpressures for selective admissions. And here is the rub: we would all like toignore the old elitist resource and reputation model with its overemphasison maximizing the inputs in order to guarantee the quality of the outputs.However, the truth is that a number of extremely important outcomes arehighly predictable from the inputs. We can predict about one-third of thevariance in student academic performance and two-thirds of the variance inpersistence to graduation by knowing the high school rank in class and SATscores of freshmen (Volkwein, 1995). Poor academic performance is thenumber one reason for student departure, and departure before degree com-pletion is the number one reason for loan default (Volkwein, 1995). Main-taining selective admissions standards, therefore, is a viable enrollmentmanagement and loan default reduction strategy because it improves cam-pus performance on three highly visible accountability measures—rates ofnew student persistence, rates of graduation, and rates of loan default.

The good news is that U.S. college students have more variety, choice, andaccess than at any time in history—and more than students anywhere else inthe world. The bad news is that U.S. institutions face severe competition,


not only for students but for faculty and other resources as well. Theydevote considerable campus resources to the processes of recruiting, financ-ing, and retaining diverse populations of both faculty and students in thefiercely competitive environment. These competitive pressures, by the way,lower institutional productivity, because they prevent colleges and univer-sities from investing more directly and heavily in their primary goal activi-ties of teaching, research, and service. Instead, institutions hire more staffto help in necessary support areas like admissions, financial aid, govern-ment relations, fundraising, student advisement, and an array of faculty andstudent support services. Why? Because in an atmosphere of accountabil-ity, competition, and enrollment-driven budgets, they cannot afford toignore either the quantity or the quality of inputs.

Even so, throughout higher education we now see much greater inter-est among institutions and their stakeholders in measuring outcomes—judging the results. Although most of us are more comfortable with thisapproach, it runs the danger of providing information too late in the processto render anything but a summative, or acceptable versus unacceptable,judgement. An overfocus on outcomes may not provide the informationneeded for internal development and educational enhancement.

Therefore there is now a renewed interest in process measures, rootedin the theory that good outcomes will not result from flawed educationalprocesses. Measurement at critical process points enables institutions todetermine which student experiences are having the greatest (and the least)impact and to make corrective interventions. Moreover, the research evi-dence indicates that many outcomes such as student growth and satisfac-tion are most heavily and favorably influenced by those campus experiencesthat produce student academic and social integration (Pascarella and Teren-zini, 1991; Volkwein, 1992, 1993).

Thus we have for institutional researchers the ideal jobs bill:

The need to measure and improve inputs, because of the strong empiricalconnection between input measures and important outcomes like acade-mic achievement, graduation rates, and loan default

The need to measure critical processes, both because of the role of theseprocesses in student integration and growth and because such measure-ment facilitates corrective intervention

The need to measure a variety of outputs and outcomes, because resultsmatter most

In short, we need to measure everything that moves.Tension 3: Productivity Versus Outcomes. Parents believe and

research suggests that the most effective educational experiences are usu-ally found in academically oriented living and learning communities inwhich full-time students receive a good deal of faculty contact and many


academic support services in the residential setting (Pascarella and Teren-zini, 1991).

However, we also know that being educated within these highly effec-tive living and learning communities is expensive when compared to beingeducated via the new information technologies. The dilemma is that par-ents and students desire the more personal learning experience that comeswith traditional settings, but the public demand for access and productiv-ity encourages legislators to give a high priority to distance learning and toencourage the use of technology to instruct larger numbers of students withfewer faculty.

The rub is that it’s easier and cheaper to transmit information usingcomputer assisted instruction than it is to teach students to think. The dan-ger is that technology will provide greater access but diminish educationalquality. The challenge is to use information technology in ways that assistindividualized learning. This will probably not translate into a need forfewer faculty. And it may not even be cheaper! There is an emerging rule ofthumb that in order to stay current, an institution should spend at least 5 percent of its annual budget on information technology.

Tension 4: Cost and Efficiency Versus Quality and Effectiveness. In1995, the State University of New York (SUNY) trustees issued a documentcalled Rethinking SUNY. This report, aimed at restructuring the SUNY sys-tem, lowering its cost, and increasing its productivity, struck many of us asoveremphasizing indicators of cost and efficiency, and underemphasizingindicators of quality and effectiveness. It is easier for centralized authorityto count the number of students in the classroom than to measure whatthey are learning. It is easier to count the square feet of campus space in usethan to discover how well the space is being used.

Another interesting development occurred in that same year when theNew York State Comptroller’s Division of Management Audit issued itsreport on the state university’s performance indicators. The audit assessednot only the extent to which SUNY’s measures reflected progress toward thegoals described in the planning document SUNY 2000 but also the extentto which these measures were consistent with those contained in the Per-formance Reporting Model of the national Government Accounting Stan-dards Board.

Surprisingly, this model developed by government accountants con-tains measures of performance that we might have designed ourselves,including ratings by students and alumni, retention and graduation rates,student performance on academic tests and professional examinations, ratesof employment and graduate school attendance, and employer satisfaction.This is a refreshing change from legislative and trustee attention to thingslike class size, faculty contact hours, and student time to degree completion,measures that focus on means rather than ends—on the cost of the educa-tional process rather than on the results.


At any rate, I believe that the planning environment for the foreseeablefuture means that higher education will be viewed with increasingly criticaleyes. And these eyes will focus with equal acuity on the education offeredby traditional campuses and by the newer technologies. Outcomes assess-ment and performance measures are in danger of becoming a relatively per-manent part of the educational enterprise.

Tension 5: Assessing for Internal Improvement Versus ExternalAccountability. The classic Janusian challenge for most of us is resolvingthe tension between the internal and the external uses of assessment andperformance. In public and private institutions alike, all individuals face theneed to improve themselves and to become better teachers, learners, schol-ars, and administrators. To accomplish this, they need to expose their weak-nesses and identify what needs to be changed. However, such openness runsthe danger of reducing an institution’s market appeal and its resources, espe-cially in an atmosphere of fierce competition and performance funding. Asone writer puts it, “The spirit of assessment requires a diligent search forbad news, but accountability encourages the opposite” (Peters, 1994, p. 17).

Various regional accrediting associations (most prominently the Mid-dle States, Southern [SACS], and North Central [NCA]) attempt to resolvethis tension by requiring each campus to present evidence of student learn-ing and growth as a key component in demonstrating the institution’s effec-tiveness. To be accredited each institution is expected to gather and presentevidence that it is accomplishing its educational goals. The new MiddleStates monograph, Framework for Outcomes Assessment (Commission onHigher Education, 1996), is extremely helpful in that it provides guidanceto campuses in assessment areas such as basic skills, general education, per-sonal growth, and the major field of study.

Simply put, the institutional goal should be the improvement of stu-dent learning and growth. Campuses need to carry out assessment and self-evaluation not for external accountability but for internal enhancement.Outcomes assessment does not judge undergraduate education but improvesit. Faculty, especially, identify with this emphasis.

Now for some realism: in an atmosphere of resource scarcity, campusesthat can measure their effectiveness will do better in the competition forexternal resources than will campuses that cannot. And academic depart-ments that are able to provide presidents and provosts with evidence aboutthe impacts they are having on their students will be more successful in thecompetition for campus resources than will academic units not able to pro-vide such evidence (Volkwein, 1995).

Thus the action by Middle States, NCA, and SACS properly calls ourattention to two principal uses of assessment improvement and account-ability. These dual emphases, these twin purposes, seem to offer a con-structive path. They provide the motivation and foundation for internaldevelopment and at the same time recognize the need to demonstrate effec-tiveness to stakeholders.


The Four Faces of Institutional Research

I have discussed a variety of campus dualities, tensions, and policy colli-sions—internal versus external, academic versus administrative, professionalversus institutional, access versus excellence, efficiency versus effectiveness,and assessing for improvement versus assessing for accountability. Thesecontradictory pressures produce a variety of challenges for institutionalresearchers and force us to play a medley of roles. In our campus endeav-ors, we pursue a number of different purposes, summarized by the frame-work in Figure 1.1, which identifies four IR types.

On the horizontal axis of the matrix, this typology distinguishesbetween those IR purposes and audiences that are more formative, internal,and improvement oriented and those that are more summative, external,and accountability oriented. The vertical axis characterizes organizationalroles and cultures in two ways: first, administrative and institutional, andsecond, academic and professional. As discussed earlier, the administrativeculture is more consistent with bureaucratic values like efficiency, and theacademic culture is more consistent with professional values like effective-ness. Combining these categories produces a typology of four overlappingyet distinguishable types of IR purposes and roles. These are not pure types,but they reflect dominant tendencies and can be applied either to the IRoffice as a whole or to the separate individuals and functions within it.

IR as Information Authority. The internal and administrative pur-pose and support role calls upon institutional research to describe the insti-tution’s shape and size, its students and staff, and its activities. Here theinstitutional researcher educates the campus community about itself interms of data on admissions, enrollment, faculty, and degrees awarded. Gen-erating most of the information in a factbook falls into this category. In thisrole the institutional researcher compiles and packages descriptive statisticsfor campus audiences. Of the many challenging IR tasks, this one probablyrequires the least preparation in the form of education and experience. The

Figure 1.1. Four Purposes and Roles of Institutional Research

PURPOSES AND AUDIENCES

Formative and Internal—for Improvement

Administrativeand Institutional

Academic andProfessional

To describe the institution—IR as information authority

To present the best case—IR as spin doctor

To supply impartial evidence of effectiveness—IR as scholar and researcher

To analyze alternatives—IR as policy analyst

Summative and External—for Accountability

ORGANIZATIONALROLE AND CULTURE


role requirements correspond roughly to Terenzini’s description of techni-cal intelligence (see Chapter Two).

IR as Policy Analyst. The internal and professional purpose callsupon IR to study and analyze the institution and its policies. In this role theinstitutional researcher works with top management as an analyst or con-sultant, providing support for planning and budget allocation decisions, pol-icy revision, administrative restructuring, or other needed change. Here theinstitutional researcher is the policy analyst who educates the managementteam. Many of us are especially likely to assume this role when we conductstudies for our colleagues in academic affairs, budgeting, and student ser-vices. Falling into this category are studies that give alternative enrollmentscenarios and revenue projections based upon different assumptions aboutinputs and attrition. Comparative cost analysis, student opinion research,and studies of salary equity are other examples. This role requires relativelyhigh levels of education and training, as well as both analytical and issuesintelligence (also described in Chapter Two).

IR as Spin Doctor. Of the two external types of IR, the administrativestyle is visible when IR assembles descriptive statistics that reflect favorablyupon the institution. Many of us are called upon to play this advocate rolefrequently, and we need to protect against carrying this style to an unethi-cal extreme. Here the IR staff present the best case for the campus, describ-ing the glass as half full rather than half empty. Some would put ouradmissions material for students and parents in this category, but we cer-tainly perform this role when we assist campus fundraisers and governmentrelations staff in presenting a positive image. Some experience on the joband knowledge of the institution is usually needed for success in this role.

IR as Scholar and Researcher. The professionally oriented and ana-lytical version of the external accountability role is that of the impartialresearcher and scholar who investigates and produces evidence so that insti-tutional effectiveness, legal compliance, and goal attainment can be judged.Conducting outcomes studies and performance reports when the primaryaudience is external to the campus falls into this category. Support for theaccreditation of self-study might be another example. This is a sophisticatedrole that requires advanced training and years of experience.

Some IR activities are difficult to classify because they overlap several cate-gories. A factbook has both internal and external audiences. Compliancereporting has both descriptive and analytical aspects. When we score andreport student ratings, we act as the information authority, but we becomethe research analyst when we carry out studies based on student ratingsdata. Faculty workload and instructional analysis may appear in any of thefour quadrants, depending on the audience and the complexity of the task.

Nevertheless, most of what we do forces us to play one or another ofthese roles, and sometimes two or more simultaneously. Although the


boundaries around these four faces of institutional research may blur fromtime to time and the transition from one to the other may be as rapid as atelephone call, I’m convinced that my institution needs IR to play all fourroles effectively. I suspect that yours does too.

J. FREDERICKS VOLKWEIN is professor, senior scientist, and director of the Cen-ter for the Study of Higher Education at Penn State University.

Documents

The Four Faces of Institutional Research