1

Assessing & Evaluating Impact

NSF ADVANCE National Conference

April 20, 2004

Dr. Cathy A. Trower

2

Overview What is transformation?

What is culture?

What about data?

So now what?

3

“The progress of this institution…will be directly proportional to the death rate of the faculty.”

4

William T. Foster (1879-1950)President, Reed College – 1911

There were 46 students and 5 faculty members.

5

Law #1: Inertia“The status quo is

the only solution that cannot be vetoed.”

Clark Kerr (1911-2003) Chancellor, UC Berkeley President, UC

Comment made 1982

6

Academic Transformation: An Oxymoron?

Alters the culture of the institution by changing select underlying assumptions and institutional behaviors, processes, and products;

Is deep and pervasive, affecting the entire institution;

Is intentional; and, Occurs over time.

7

Organization Culture

“A pattern of shared basic assumptions that the group learned as it solved problems…that has worked well enough to be considered valid and, therefore, to be taught to new members as the correct way to perceive, think, and feel in relation to those problems” (p. 12).

(Schein, 1992, Organizational Culture and Leadership, Jossey-Bass).

8

Organization CultureThe ethos of a place – the behavioral norms,

espoused values, language, customs, rituals, and the modus operandi of an organization (Bolman and Deal, 1997; Schein, 1992).

Three layers of culture Artifacts: visible structures and practices Espoused values: what people say they believe Underlying assumptions: unconscious, taken-for-

granted beliefs, thoughts, and feelings – ultimate source of values and actions

9

Why we don’t change We are working primarily at the artifact level. Academics have the lingo down pat – the

espoused values are not a problem.________________________________________A. We don’t get to the underlying assumptions.

--AND--B. We don’t change the reward structure; we

don’t reward what we say we value.

10

We don’t examine our underlying assumptions The academy is a meritocracy Those who perform get rewarded There is an “ideal worker”* who…

Receives terminal degree in late 20s Completes postdoc training Takes a full-time tenure-track position Performs a modicum of service, builds a teaching

dossier, and generates research resulting in publications.

* Penn State/Sloan “Faculty & Families” Drago, Crouter, Wardell, and Willits

11

We don’t examine our underlying assumptions

If he performs as an “ideal” worker, he gets tenure and promotion at the end of the 6th year.

He will then strive to generate a superior record to be promoted to full professor.

T&P are the most obvious rewards, but there is also: pay, course scheduling, resource allocation, accolades, citations, awards, and leadership positions.

12

Discrimination avoidance Caregiving signals that you are not an “ideal”

worker and are, therefore, a substandard academic.

Policy reform No meetings before 8 or after 5 Quality, onsite childcare Part-time tenure tracks Emergency leave Stop-the-tenure-clock Flexible schedules

13

Espoused values v. reward structures

We say we value… Teaching and service, but research earns you tenure. Interdisciplinary work, but discipline-specific work brings in

grants/prestige. Diversity (we need you on committees), but it won’t earn you tenure. Students, but spending time with them will hurt your tenure bid. Innovation, but reward the status quo. Collaboration, but reward competition (solo work). Academic freedom, but your work better reinforce the prevailing norms

rather than rock the boat. New thinking/entrepreneurialism, but uniformity and conformity win. Liberalism, but reward conservatism.

14

Multiple cultures

National System of Higher Education Academic Profession Discipline Institution Department

15

Roles that Data Play

Catalyze Compare Identify/Signal Illuminate/

Enlighten Influence

Inform Monitor Socialize Substantiate Symbolize

16

Why No Direct Link?

• Assuming rationality may be erroneous.– Data are used selectively for political and symbolic purposes

that may or may not be tied directly to decisions. Links between data and decisions get lost in chaos and “organized anarchy.”

• We tend to assume that all participants have all the data at the same time.

– But this rarely happens. Decisions are, in part, a function of the availability of data at any given decision moment.

17

Why No Direct Link?

The amount of data that people can or choose to consume differs.

Data consumption depends on the person and the issue.

Sometimes a sample of one is all people need to draw a conclusion or make a case.

e.g., A senior woman in chemistry; a woman dean…becomes iconic

18

Why No Direct Link?

Data appear in different places at different times and people carry data from one arena to another and use it in ways not originally intended.

– Difficult to get the “right” data into play in the “right” venue.

19

Why No Direct Link?

Data “use” is an ambiguous concept. Information is sought from numerous sources in a non-linear way. Often, decision-makers are not sure what data, if any, they are

using and how they are using it. People gather data but use other means to make decisions. Decisions are often made before seeing data and the data are then

used to justify the decision. Decision-makers rarely admit to “gut-feel” decisions so pretend to

use data.

20

Academe Even More Complicated

Faculty work, the workplace, and the culture are complex and not easily quantified.

Faculty autonomy is high at the best institutions -- a weak internal market for performance measures.

Even when quantifiable, the data are subject to multiple interpretations.

Data do not address visceral concerns. How do you place academic freedom, peer review,

tenure and promotion, into the realm of data?

21

Academe Even More Complicated

When the issue concerns beliefs and values, data have less sway.

Difficult, if not impossible, the locate the locus for many decisions in academe given shared governance processes. “Decisions happen…”

22

Henry Rosovsky (1990)

“Never underestimate the difficulty of changing false beliefs with facts.”

“When given the opportunity--in the absence of incontrovertible scientific proof, and sometimes even then--people believe what they wish, and empirical evidence does not lead to quick altering of cherished positions.”

The University: An Owner’s Manual

23

Framework for Assessing Change

Clarity about intended outcomes Consideration of unintended outcome Comparisons to baseline data, measured over

time What activities, processes, practices, outcomes,

expectations, structures, experiences, language, and symbols are different as a result of the intervention?

Are we working at all three levels? Are we working on the multiple cultures?

24

“The Master’s tools will never dismantle the Master’s house.”

--Audre Lourde