BUILDING A CULTURE OF EVIDENCE IN COMMUNITY COLLEGES · Building a Culture of Evidence in Community Colleges i Acrosstheeducationsector,callsforaccounta - bilityandresults—andforgreatertransparency

Lessons from Exemplary Institutions

BUILDING A CULTUREOF EVIDENCE INCOMMUNITY COLLEGES:

P R E PA R E D B Y

F O R

MetLife Foundation

By Lili Allen and Richard KazisOctober 2007

Table of Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

How Colleges Can Build Demand for StrongerCapacity to Collect, Analyze, and Use Data for Improvement. . . . . . . . . . . . . . . . . . . . . . . . . 3

Set the Tone and Insist on a Culture that Elevates Evidence and Inquiry . . . . . . . . . . . . . . . . 4

Put Data in the Hands of the Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Conduct Specific Analyses that Answer Important Questions AboutHow Particular Student Groups Are Progressing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Tie Data to Strategic Planning and Budgeting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

How More Colleges Can Be Encouraged to StrengthenInstitutional Research Capacity and Effectiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Building a Culture of Evidence in Community Colleges iii

Across the education sector, calls for accounta-bility and results—and for greater transparencyin the reporting of student outcomes—havebeen increasing. While this pressure began twodecades ago in K-12 education, communitycolleges are now also paying closer attention tohow they can and should use data on studentoutcomes to drive better results. Historically,colleges have generated data primarily forcompliance purposes—to meet state or federalreporting requirements. But this has begun tochange: some dynamic, entrepreneurial commu-nity colleges are taking a hard look at how theycan create and sustain an internal culture ofevidence-based practice. These colleges areengaging staff throughout the institution inlooking at data to identify areas of weakness,progress, and potential improvement.

Building a Culture of Evidence in CommunityColleges looks at four colleges that are nationalleaders in using institutional research strategi-cally and for improvement: City College of SanFrancisco; Community College of BaltimoreCounty; Indian River Community College; andLaGuardia Community College. All four ofthese leading colleges have received or beenfinalists for the MetLife FoundationCommunity College Excellence Award.

The pioneering efforts of these schools in theuse of student data for identifying problems andpotential solutions provide a rich source ofexpertise on what it takes to build an institu-tion-wide culture of evidence-based decisionmaking. They also illustrate how communitycolleges can make routine the use of data toidentify strengths and weaknesses, pinpointareas for improvement, and assess the impact onstudents of new programs and innovations. The

experiences of these institutions also suggestchanges in state and community college systempolicies that can enable significantly more insti-tutions to follow their lead.

The ways in which these four institutionsorganize and use institutional research suggestsconcrete, specific strategies that college leadersand their research teams can use to weave datause deep into the fabric of an institution.

These lessons cluster into four categories:

Set the tone and insist on a culture that elevatesevidence and inquiry: Leaders can drive theirsystems and messages toward the use of dataacross the college.

Put data in the hands of the users: Institutionalresearch offices can stimulate demand for databy making sure that the data are both reliableand accessible.

Some dynamic,

entrepreneurial

community colleges

are taking a hard look

at how they can create

and sustain an internal

culture of evidence-

based practice.



E X E C U T I V E S U M M A R Y

iv Jobs for the Future | MetLife Foundation

Conduct specific analyses that answer impor-tant questions about how particular studentgroups are progressing: Colleges can start on thepath of creating a culture of evidence byconducting longitudinal analyses to identifydifficulties for specific populations at specificpoints along the pipeline to graduation.

Tie data analysis and use tightly to planning andbudgeting processes: Instituting strategic plan-ning processes that explicitly tie studentoutcome data to budget allocations makes clear

to stakeholders throughout the college that datamatters to institutional priorities and resourceallocations.

The strategies pursued by the communitycolleges profiled in this report provide impor-tant and very practical lessons for other collegesthat want to embrace evidence-based decisionmaking. Their approaches to building capacityand trust; integrating data use into planning,budgeting, and teaching and learning; andchanging institutional culture have powerfullessons for other colleges around the country.

In the end, though, the kind of transformationthat these colleges have undergone and arecontinuing to pursue widely and in a sustainedway will not spread without state and systempolicies that provide additional resources,support more sophisticated data analyses andcapacity, and create incentives for institutionalleaders to drive change. The two must go handin hand: institutional excellence and supportivepolicies. When they do, the success of theseMetLife Foundation Award winners and final-ists will be more easily replicable by othercommunity colleges across the nation.

In the end, the kind of

transformation that

these colleges have

undergone and are

continuing to pursue

will not spread widely

and in a sustained way

without supportive

state and system

policies.

Building a Culture of Evidence in Community Colleges 1

Community colleges

are paying closer

attention to how they

can and should use

data on student

outcomes to identify

opportunities and

strengthen strategies

for improving the

delivery and support

of learning.



I N T R O D U C T I O N

Across the education sector, calls for accounta-bility and results—and for greater transparencyin the reporting of student outcomes—havebeen increasing. While this pressure began twodecades ago in K-12 education, communitycolleges are now also paying closer attention tohow they can and should use data on studentoutcomes to identify opportunities andstrengthen strategies for improving the deliveryand support of learning. Historically, collegeshave generated data primarily for compliancepurposes—to meet state or federal reportingrequirements. This has begun to change. Somedynamic, entrepreneurial community collegesare taking a hard look at how they can createand sustain an internal culture of evidence-based practice, and they are engaging staffthroughout the institution in looking at data toidentify areas of weakness, progress, and poten-tial improvement.

This shift to a culture of evidence and inquiry isnot easy—and it is not yet common. (See thesidebar, “Findings on Institutional ResearchCapacity.”) Without such critical conditions asresources for staff and studies, systems forassuring quality data collection and mainte-nance, or institutional leadership that makesresearch and the culture of evidence a highpriority, community colleges can struggle to usedata on student outcomes to assess areas ofweakness and strategies for improvement.

Without these conditions, colleges fall back onplanning, budgeting, and professional develop-ment by anecdote or politics rather than data.They neither make the necessary technicalimprovements in how data is collected, stored,and reported, nor the required organizationalchanges in how faculty and staff engage witheach other and with information about studentoutcomes.

This report looks at four colleges that areexceptions when it comes to using institutionalresearch strategically and for improvement:

• City College of San Francisco;

• Community College of Baltimore County;

• Indian River Community College; and

• LaGuardia Community College.

All four of these leading colleges have receivedor been finalists for the MetLife FoundationCommunity College Excellence Award.

2 Jobs for the Future | MetLife Foundation

The pioneering efforts of these schools in theuse of student data for identifying problemsand potential solutions provide a rich sourceof expertise on what it takes to build an institu-tion-wide culture of evidence-based decisionmaking. They also illustrate how communitycolleges can make routine the use of data toidentify strengths and weaknesses, pinpointareas for improvement, and assess the impacton students of new programs and innovations.The experiences of these institutions also

suggest changes in state and community collegesystem policies that can enable significantlymore institutions to follow their lead.

Our focus is primarily on offices of institutionalresearch. These key units within communitycolleges, typically understaffed and underuti-lized, are critical to efforts to strengthen institu-tional use of data for improvement, not just forcompliance (Morest et al. 2006). The experienceof these four institutions and how they organizeand use institutional research suggests concrete,specific strategies that college leaders and theirresearch teams can use to weave data use deepinto the fabric of an institution.

At the same time, this is not solely a study of IRoffices. The success of community colleges alsodepends upon strong, steady, committed institu-tional leadership that supports these offices andplaces IR at the center of strategy and planning,especially in relation to budget decisions. At theheart of the story of each college in this report isa commitment to changing institutional cultureas well as practice—to changing expectationsabout how faculty, administrators, and leadersrelate to, use, and promote data about studentsand their success. This cultural shift requires thestimulation of demand for data across the insti-tution, so that faculty and staff want to getmore, and more useful, information about howtheir students are progressing. At the same time,changing institutional culture must also beaccompanied by real changes in the organiza-tion of data collection, analysis, and use.

New Research on Institutional Research Capacity

How well prepared are today’s community colleges to move toward the greateruse of data and research to improve student success? Less well than is neededfor developing and routinizing the “culture of evidence,” according to recentresearch by the Community College Research Center at Teachers College,Columbia University. To assess the capacity of institutional research (IR) offices incommunity colleges and the obstacles to more effective data use, CCRC fielded anational survey of 189 college administrators responsible for institutionalresearch at their schools and augmented it with case studies of 28 communitycolleges in 15 states. The study aimed to learn how much IR capacity communitycolleges have in terms of IR staff size and facility with research methods, how IRis utilized by different actors within colleges, and what barriers may exist thatimpede the development of IR analysis that would benefit college decisionmaking.

The researchers identified three common challenges in the areas of resources,data, and leadership: institutional research offices are typically small and under-funded; difficulty “cleaning” student data entered by different departments andat different times and not designed primarily for research use makes it difficult toconduct research that can inform improvements to program and institutionalperformance; and leadership commitment to and investment in the capacity forrigorous research is typically limited.

The study noted that a small number of colleges have demonstrated the capacityand the commitment to use data to manage and improve programs and services.Such colleges typically combine institutional research, planning, institutionaleffectiveness, and assessment into one department, led by an individual withexperience and advanced training who is a full member of the college’s leader-ship team. They tend to employ sufficient staff to conduct research above what isrequired for the purposes of compliance and accreditation. The critical challengefacing community colleges and college systems is to develop strategies to movethe resource, data, and leadership “best practices” of these institutions intomany more institutions.

See Morest, Vanessa Smith, & Davis Jenkins. 2007. Institutional Research and the Culture

of Evidence at Community Colleges Report #1 in the Culture of Evidence Series of

Achieving the Dream. New York, NY: Community College Research Center. To download

this report, go to: http://ccrc.tc.columbia.edu/Publication.asp?UID=515.


The experience of these four schools providesimportant lessons for others about how toincrease demand for, and the centrality of, insti-tutional research in the life of a communitycollege. These lessons cluster into four cate-gories:

Set the tone and insist on a culture that elevatesevidence and inquiry: Leaders can drive theirsystems and messages toward the use of dataacross the college.

Put data in the hands of the users: Institutionalresearch offices can stimulate demand for databy making sure that the data are both reliableand accessible.

Conduct specific analyses that answer impor-tant questions about how particular studentgroups are progressing: Colleges can start on the

path of creating a culture of evidence byconducting longitudinal analyses to identifydifficulties for specific populations at specificpoints along the pipeline to graduation.

Tie data analysis and use tightly to planning andbudgeting processes: Instituting strategic plan-ning processes that explicitly tie studentoutcome data to budget allocations makes clearto stakeholders throughout the college that datamatters to institutional priorities and resourceallocations.

While all four colleges studied here are largeinstitutions with higher capacity in their IR thanin the typical community college, there arelessons for smaller institutions as well (seesidebar, “Advice for Small Colleges,” page 8).

H O W C O L L E G E S C A N B U I L D D E M A N D F O R S T R O N G E R

C A P A C I T Y T O C O L L E C T, A N A LY Z E , A N D U S E D A T A F O R

I M P R O V E M E N T

Accountability Pressures Are Changing the Context for Institutional Research

Historically, the collection and analysis of data in communitycolleges have been driven by compliance with state reportingrequirements, which frequently determine state funding alloca-tions. Today, accountability pressures emerging at the state,national, and accreditation agency levels are changing thecontext within which colleges report and use data on theirstudents. In addition, improved data collection and analysis toolsand technologies are making it possible to do far more sophisti-cated data work more quickly and cost effectively. As a result,colleges are increasingly expected—and increasingly able—to usedata more systematically and creatively to answer questionsabout student performance and how it might be improved.

Making this shift is slow, given the complex environment withinwhich colleges operate. Community colleges serve the mostdiverse population in higher education, with students rangingfrom new immigrants seeking English skills and short-term certifi-cates to incumbent workers seeking career advancement, highschool graduates seeking a first step toward a college degree,and high school dropouts seeking reentry into the educationpipeline. Given this diversity of students and their goals, there is

some debate over which outcomes and metrics should be used tojudge community college effectiveness (see, for example, Bailey,Jenkins, & Leinbach 2007).

However, there are signs of change. While the primary federallyrequired community college quality measure tracks graduationrates of first-time, full-time degree-seeking students, many stateswant to know more. They are asking institutions for the informa-tion that can answer important questions about quality andperformance. How well are developmental education studentstransitioning into credit-bearing courses and degree programs?How do part-time students fare, and what factors increase theirsuccess? What progress are various populations making through“gatekeeper” courses? What is the effect of different high schoolcurricula on later success in non-remedial courses? What are thelabor market outcomes for students who complete differentdegree programs? As states experiment with new and morevaried measures in their own accountability systems, and asaccreditation agencies demand more attention to outcomes andlearning, the importance of IR office capacity and quality isgrowing. This trend is likely to continue and accelerate.


Set the Tone and Insist on a Culture thatElevates Evidence and Inquiry.

The four colleges studied tell us much aboutwhat steps leaders can take: they can determinewhere institutional research sits in their institu-tion; they can signal the importance of data andprovide a clear picture of what a culture ofinquiry and evidence looks like at differentlevels of the institution; and they can createsystems of incentives to reward the honestappraisal of data, even if it is “bad news.”

Where to situate institutional research. Each ofthese institutions has a strategic rationaleregarding where it situates its Office ofInstitutional Research. IR offices need to beaccessible to the president, especially forstrategic planning. At the same time, accordingto many IR staff interviewed, deans, departmentchairs, and faculty need to view the IR office as aneutral player. The four colleges here have takena variety of approaches to meeting these criteria.

In one variant, a vice president with responsi-bility for both strategic planning and institu-tional research reports directly to the president,ensuring that data is used strategically to guidethe institution. At City College of San Francisco,for example, an IR staff of three researchersresponds to data requests from the faculty,while the vice chancellor for institutionaladvancement ensures that those requests alignwith the college’s overall strategic goals. Thisvice president also directs the involvement of theIR staff in the college’s strategic planningprocess.

In other colleges, the IR director is housed inadministration and finance or closely tied to theOffice of Information Technology; bothapproaches signal neutrality and ensure that thedata and technology systems are aligned. IndianRiver Community College put its IR departmentin administration and finance, but the Office ofInstitutional Effectiveness oversees strategic plan-ning and makes sure that all departments areeffectively using the IR department. AtLaGuardia, institutional research is housed in thedivision of information technology, but the officeis physically next door to that of the president.

Signaling the importance of data. The presidentsof the four MetLife Foundation Award finalistsemphasize that they consciously use keymoments of opportunity to convey a fullcommitment to open, honest appraisals of dataon institutional performance. At LaGuardia andIndian River, this means embracing negativedata as an opportunity to change. In the first sixmonths of Gail Mellow’s tenure as president, forexample, LaGuardia Community Collegeconducted a student survey in preparation foraccreditation, and the feedback on the quality ofstudent services was largely negative. Accordingto Dr. Mellow, the impulse of key staff at thetime was “How can we hide this?” Instead, shehelped faculty and staff embrace the news anduse it to fundamentally redesign enrollmentmanagement, resulting in shorter wait times andmore efficient delivery of services. As the VicePresident for Enrollment Management put it,“Our anecdotal impressions of the volume ofstudents served misled us into thinking we weremeeting demand. We had to rethink our servicesand significantly streamline the enrollmentprocess.” The resulting “high-tech/high-touch”model of enrollment paid off: in fall 2005-06,the college hit 103 percent of its enrollmenttarget three days before the semester started.

Following a climate survey that showed whatDr. Edwin R. Massey of Indian RiverCommunity College describes as a “culture ofcomplacency,” he convened a series of town hallmeetings. He answered every question—nomatter how challenging or seemingly trivial—and personally conveyed a commitment to “facethe brutal facts,” as IRCC’s guiding principlesput it. As Dr. Massey looks back on it, thesemeetings marked a significant turning point inthe institution’s transition to a culture ofinquiry.

Chancellor Philip R. Day, Jr., of City College ofSan Francisco launched his administration withthe publication of baseline data on all aspects ofthe college’s performance—a more open sharingof data than had previously been seen in thecollege. The college followed that publicationby conducting an extensive series of listeningsessions with internal and external stakeholders,including high school principals and college

Leaders can create

systems of incentives

to reward the honest

appraisal of data, even

if it is “bad news.”


faculty, to “put a face on the data.” Both datasets informed the priorities contained in thecollege’s strategic plan.

The presidents of these colleges strategically usedata in their messaging to internal audiences asa way to signal and build support for a cultureof evidence. Several intentionally use data everytime they speak to college audiences—bothnegative data (“this is not good enough”) andpositive (“here’s what the data can tell us aboutwhat’s working for our students”). Onemessaging strategy is for the president to modelthe use of inquiry-based discussion in grapplingwith the data. LaGuardia held a cabinet sessionto enable small group discussions about keydata findings, allowing cabinet members todraw their own conclusions about what the datadid and did not imply about college effective-ness.

Presidents interviewed by Jobs for the Futurestressed the importance of building in incentivesand rewards for faculty and administrators touse data. LaGuardia and San Francisco bothrequire all budget requests to be justified bystudent data. In addition, when designing aproject, LaGuardia builds in time for researchinto best practices across the country, and itincludes this in grant proposals. For example, arecent grant proposal on the use of electronicportfolios included a “research year” for facultyto review literature and visit other colleges tolearn about the use of e-portfolios and deter-mine how best to implement them back home.

Put Data in the Hands of the Users.

The four colleges have taken specific steps tomake data as accessible as possible to faculty.The strategy is to stimulate demand for more,and more fine-grained and useful, data: acces-sible data leads faculty to understand its utility,and they begin to seek more. As one IR stafferat City College of San Francisco has put it,“You dig the channel; you let the data flow.”Strategies include investing in technology,

creating mechanisms for faculty to use data toassess their own practice, taking steps to ensurethat the data is transparent and consistent, andhiring and training IR staff to be brokers andfacilitators of data.

Investing in technology. Finding the resources tobeef up technological systems and humanresources capacity is a major challenge, espe-cially for smaller colleges. One possibility is touse large, one-time grants to upgrade a college’stechnology infrastructure so that it is more flex-ible and usable at the college, department, andfaculty levels. Indian River took a gradualapproach: it developed its initial, streamlineddata system over a decade ago, and has modi-fied it dramatically ever since by responding touser requests. For example, the Indian Riverinstitutional research office has added “tabs” tothe computer desktop screens of faculty andstaff in response to multiple requests for aspecific type of data. Baltimore County andLaGuardia have built on-line communities ofpractice that allow faculty to share what theylearn about pedagogy and practice. Baltimorehas developed a portal attached to the collegeWeb site for posting executive summaries ofaction research that documents studentlearning.

Strategies for making

data accessible include

investing in

technology, creating

mechanisms for

faculty to use data to

assess their own

practice, taking steps

to ensure that the data

is transparent and

consistent, and hiring

and training IR staff to

be brokers and

facilitators of data.


At City College of San Francisco, the institu-tional research office created the Web-basedDecision Support System, geared towardproviding deans, department chairs, and otherprimary users with a sense of the kind of infor-mation and analyses they might be able to do—or to ask for the IR office to undertake. Whilethe DSS is still “pretty basic,” according to IRstaffers, its designers use it to help change insti-tutional culture: they have made it accessible toall faculty and administrators, not just to thedepartment heads and deans most likely to useit, in an effort to break down the barriersbetween data gurus and data users.

Conveying the value and reliability of data.Trust is critical. The IR staff at these institutionsconsistently strive to garner the trust of faculty,deans, and department chairs. First and fore-most, they concentrate on making themselvesuseful; their orientation is toward customerservice. San Francisco and LaGuardia IR staffhave adopted a strategy of offering more datathan they are asked for; in both colleges, IRstaff will sit with the person requesting the dataand help determine if additional data, or adifferent query, would meet their needs better.At San Francisco, when a dean in the English asa Second Language department wanted to knowhow ESL students do after transitioning intocredit-bearing courses, he expected to receiveaggregated data, in hard copy, from the IR

office. Instead, he received an extensive, interac-tive database that allows him to see how formerESL students do in any credit-bearing class,along with a tutorial from IR staff on how touse the database to answer any questions hemight have about student performance.

To ensure that the college serves all audiences,Baltimore County has organized its IR depart-ment by stakeholder groups, including faculty,college leaders, and state and federal stake-holders. While some staff focus on state andfederal reporting, others work primarily withfaculty on defining learning goals and analyzingdata as part of action research. At the sametime, the college has a relational database thatallows staff to look at student outcomes acrossdepartments and to compare the success ofparticular populations—such as African-American males—in various courses. BaltimoreCounty’s mantra “do it once and use it manytimes” reflects a commitment to drilling eachtable and report down from the college level tothe program and department levels. All datarequests lead to both a research report—with aneasy-to-read executive summary—and the devel-opment of other reports or files that enable eachtype of user to use similar reports for their ownneeds.

At City College of San Francisco, where thedemographics of the student body—race,ethnicity, age, gender—are constantly evolving,IR staff are valued for helping faculty under-stand the changing mix and characteristics ofstudents in the classroom and the implicationsfor classroom practice. For example, the IRoffice produced a report indicating that thecollege’s student body is increasingly computer-savvy, resulting in an increase in courses with anon-line component.

San Francisco illustrates a second strategy forpromoting data use: to increase the trans-parency of the data. CCSF revitalized its fledg-ling institutional research department with awidely circulated document that gave defini-tions and protocols for using data; this reas-sured faculty and department chairs who hadnot only lacked access to data but had perceivedthat it would be used against them. IR staff alsopresent data in multiple ways so that a variety

First and foremost,

institutional

researchers concentrate

on making themselves

useful; their

orientation is toward

customer service.


of stakeholders can understand and use them.The template for CCSF’s reports invites useand includes a description of how to use thedata report. Data are presented in three levelsof specificity and detail: a one-page summary;a supporting fact sheet; and full data tables.This template allows users to access the data atmultiple levels, depending on their degree ofcomfort, need, and interest.

Another strategy to build trust in the reliabilityof data is to vet it with key audiences. This canhelp defuse anxiety about and distrust of thedata and secure stakeholders as allies inaddressing what might be interpreted as badnews. San Francisco has developed a processthe IR staff call “walk-arounds”: in small groupmeetings, they share emerging data with rele-vant staff and faculty, then conduct additionalqueries or aggregate the data differently inresponse to questions or suggestions.

CCSF Chancellor Day emphasizes the value of“sunshining” data with multiple audiences—tomaximize dissemination and value, and also tominimize the potential for political pushback.For example, when data indicated that studentsof color were faring worse than their whitepeers in basic skills math courses, Dr. Daybrought the data to the college’s DiversityCommittee, which he chairs, for its reviewand input into potential intervention strategies.Now the college is experimenting with differentformats for that course and tracking outcomesto determine the most effective approach.

Organizing opportunities for faculty to usedata. These colleges have created varied mecha-nisms to facilitate and promote faculty use ofstudent performance data. Indian Riverprovides multiple opportunities for staff toreview data and discuss implications for prac-tice. Work groups on specific topics conveneregularly and explore data related to their keyquestions. For example, a work group onstudent performance that was comparingcollege with state data had hypothesized thatthere would be an achievement gap betweenwhites and blacks on campus; however, a closelook at the data revealed no gap.

CCSF has conducted focus groups with facultyto review data on the progress of students inremedial courses. Faculty review data onstudent performance in classes with specificinterventions and consider the implications fortheir teaching practice. For CCSF researchers,their role is to support those who want to usedata effectively. According to one staffer, “Wedon’t want to be data providers to ‘consumers’but rather to ‘investigators.’”

Some colleges have made strides in promotingaction research, which engages faculty inanalyzing their own practice for its impact onstudent outcomes. Through action research,faculty become data “producers” and, in turn,can become more accomplished and routineusers of data. Community College of BaltimoreCounty has developed a Learning OutcomesAssessment process that requires faculty acrossthe institution—especially those teaching “highimpact” courses with high enrollment—toengage in action research. Faculty receive astipend for participating. These assessmentsbegin with a group of faculty—all of whomteach a specific course—developing a measur-able statement of what students are expected toknow and be able to do upon completion of thecourse. Faculty collaborate to identify a methodto collect data that measures the identifiedoutcomes, collect data across at least threesemesters, and analyze the data and planimprovements in curriculum or pedagogy.Following one or two semesters of implementa-tion of the new approach, faculty conduct areassessment. The IR department supports theLearning Outcomes Assessment process byguiding faculty in research design and the selec-tion of an assessment methodology, which mayinclude portfolios, standardized tests, externalgraders, and surveys. They also provide supportfor data analysis. The college has expandedLearning Outcomes Assessments to the programlevel, using them regularly in program review.

Staff at Baltimore County credit the LearningOutcomes Assessment process with driving apowerful culture of inquiry throughout theinstitution. For example, in years past, therewas little discussion among faculty of the find-ings of the Community College Survey on

Through action

research, faculty

become data

“producers” and, in

turn, can become more

accomplished and

routine users of data.


Student Engagement for the college, which isone of many institutions that use the survey asan external assessment of classroom and studentsupport practices. This year, though, manyfaculty members are highly engaged in consid-ering the implications of the college’s scores asbenchmarked against national averages.

Like Baltimore County, LaGuardia CommunityCollege has created an innovative way to engagefaculty in data-based inquiry into their ownpractice as a first step in promoting data useacross the institution. LaGuardia promotes

action research through its Center for Teachingand Learning, which provides opportunities forfaculty to explore issues of pedagogy to improvestudent learning. For example, through Writingin the Disciplines, a professional developmentseminar, faculty prepare a writing-intensivecurriculum and then build a portfolio thatreflects their work throughout the year.Portfolios include writing assignments, samplestudent work, and excerpts from a teachingjournal on the effectiveness of their instructionand changes they plan to make. LaGuardia isseeking to engage faculty across the college inthe Writing in the Disciplines initiative,including part-time and adjunct faculty: morethan 200 faculty have participated to date.

Hiring and training IR staff to be “brokers” ofdata. Making data transparent and usablerequires a set of skills not usually found among“data geeks,” as more than one IR staffpersonput it. More often than not, IR staff at theseinstitutions emphasized the critical role theyplay in supporting and training faculty, inter-acting with them, and motivating them to takerisks and change their patterns around data use.They talk of “brokering” the data, “facili-tating” data use, “teaching” faculty and admin-istrative staff how to use the data, and “encour-aging” them to consider how they might use thedata. At Indian River, several interviewees spokeof the value of having their IR director serve ontheir committees, since he is valued as astrategic thinker as well as a technical expert. AtCCSF, as noted, the institutional research staffmake it a habit to walk deans and departmentchairs through data generated in response tospecific queries, teaching them how to manipu-late the tables, pull-down menus, and otheranalytical tools. These institutions hire IR stafffor their communication and negotiation skills,not just their research training and capabilities.

Advice for Small Colleges

The four institutions highlighted here are all fairly large, with well-funded andwell-staffed institutional research offices and functions. Yet the leaders of theseinstitutions and offices have advice for smaller colleges beginning to travel downthe road of building a culture of data use. Several college presidents who hadpreviously led smaller institutions offer suggestions based on their own experi-ence. In addition to suggesting that colleges begin by identifying key subgroupsto follow and key transition points to monitor across credit and non-creditbearing courses, presidents made a number of suggestions related to staffingand capacity.

Colleges with minimal IR capacity might want to explore any of three options toimprove their data collection and use:

• Free up faculty time, or give faculty stipends, to look at specific data related tosubpopulations. Several presidents note that sociology faculty, in particular,often have backgrounds in the statistical programming required for reliable anddefensible data use.

• Reshape job descriptions in the office of information technology, so that staffare not only collecting data but also conducting preliminary analyses aroundkey indicators or a specific cohort.

• Form a consortium with other colleges in the region, and share the costs of aninstitutional researcher who can look at common data across collaboratingcolleges.

In all cases, presidents say, the work of individuals must be leveraged for broaderinstitutional impact. Leverage strategies include communication and resourceallocation. For example, a college leader can play a key role by highlighting find-ings and ensuring that faculty and staff across an institution understand thepower and implications of the data analyses being conducted. Some colleges usea scorecard—red for poor outcomes, yellow for troublesome outcomes, andgreen for good outcomes—to give faculty and staff a quick picture of studentperformance in key areas. In terms of resource allocation, presidents can investscarce institutional resources: the recommendation about using grant funding fora one-time technical boost to build a data system that encompasses data onprogress in both credit and non-credit courses was mentioned as especiallyimportant for small colleges.


Conduct Specific Analyses thatAnswer Important Questions AboutHow Particular Student Groups AreProgressing.

These four colleges began to change theircultures of data use by looking at cohorts ofstudents who start courses at the same time, andfollowing them longitudinally to ascertain theirprogress. Longitudinal cohort analysis allowscolleges to identify danger spots—such as thetransition from developmental into credit-bearing courses—and to identify gaps inachievement as students progress from whereverthey start in the institution. All four collegesidentified key transition points and focusedtheir data analyses on those moments ofelevated risk: transitions from developmentalinto credit-bearing courses; transitions fromnon-credit courses, such as ESL, into credit-bearing courses; gateway courses (first-yearcredit courses in a major); and capstone courses(final courses in a major). Each of these transi-tion points can be a serious obstacle on the wayto degree completion, especially for studentsfrom groups that traditionally underperform inhigher education.

While cohort analysis is increasingly the waythat community colleges organize and reportstudent outcome data, identifying whichsubpopulations to track is not a simple matter.These four colleges all started with narrow orparticular analyses that were critical for theirinstitutions; the initial findings are informingsubsequent decisions about the broader set ofstudent data they should be tracking.

Baltimore County, for example, embarked on anAchievement Gap initiative several years agoand trained its data eye on outcomes for AfricanAmericans because of widespread concernsabout their progress. That initiative has resultedin interventions that are paying off in improvedoutcomes: the achievement gap betweenAfrican-American and white students hasnarrowed on pass rates, retention, completion,and graduation, even as overall success rateshave increased. At the same time, the initiativehas spurred the use of data across the institu-tion, beyond the initial subgroup analyses.Today, under the leadership of President Sandra

Kurtinitis, the institution is assessing how toexpand its analyses further. CCBC is consid-ering which interventions, such as learningcommunities, it can expand in order to servewhat administrators recognize is an increasinglyheterogeneous underserved population.

LaGuardia has put special effort into analyzingthe trajectories of students entering from non-credit ESL courses, because a large proportionof its population on the credit-bearing side isimmigrant. Their findings have provensurprising: it turns out that simultaneous enroll-ment in ESL and credit-bearing courses (bylinking these courses in learning communities)results in better outcomes for ESL students,compared to enrolling students in consecutivenon-credit and then credit-bearing courses.Starting with this population was especially crit-ical for LaGuardia, where it is yielding higherenrollment numbers and better outcomes.

While cohort analysis

is increasingly the way

that community

colleges organize and

report student

outcome data,

identifying which

subpopulations to

track is not a simple

matter.


City College of San Francisco rebuilt its institu-tional research office and capacity in the 1990s.An early project involved an analysis of studentswho were on academic probation, whichrevealed a high proportion of Pacific Islandersamong those on academic probation. Theresearch office identified the trend and broughtit to the Chancellor’s Cabinet for discussion at amoment when Asian-American faculty andstudents were open to hearing the “bad news.”The college held grassroots meetings withfaculty, students, and others, with a focus onsoliciting solutions. Ideas for targeted supportswere brought to the board of trustees. Thecollege created an Asian Pacific SuccessProgram and offered more Learning Successcourses. Over time, the proportion of AsianPacific Islanders on probation has decreasedsignificantly.

Tie Data to Strategic Planningand Budgeting.

In the end, the real indicator of priorities is howinstitutional leaders incorporate various strate-gies into planning and budgeting processes, sothat those priorities are reflected in and shaperesource allocation. Colleges that are seriousabout building and sustaining a culture ofevidence and inquiry use analyses of studentoutcome data to shape planning and budgetpriorities.

Baltimore County houses its strategic planningefforts in the IR department, indicating the highvalue the college puts on data in that process. IRstaff organize forums to ask stakeholders to“vision” their priorities for the college and toconsider the results of external scans of demo-graphic changes, revenue projections, and laborforce and economic trends. These scans informthe college’s mission, vision, and values state-ments that guide strategic planning and, ulti-mately, resource allocation.

Indian River Community College has gone onestep further: it has developed a StrategicPlanning On-Line tool that puts data on institu-tional effectiveness in the hands of all users sothat the college’s strategic planning is morevisible and “alive” in the institution. Usersdevelop their objectives based on the college’soverall goals, identify assessment measures,make budget requests aligned with collegepriorities, and track progress—all on line. Theirbudget requests then “roll up” to influence insti-tutional priorities. The college is currently usinga U.S. Department of Education Fund for theImprovement of Postsecondary Education grantto pilot this on-line tool and process in an addi-tional ten schools across the country.

When Phillip Day became chancellor of CityCollege of San Francisco in 1998, he launchedan extensive strategic planning process thatculminated in a comprehensive documentreflecting an analysis of college data and inputfrom faculty, students, and community residents.The 2003 plan has guided the college’s prioritiesand influenced annual budgeting in recent years.To ground the plan and its priorities in data, theOffice of Research, Planning and Grantsproduces an annual College PerformanceIndicators Report tracking 29 indicators ofCCSF performance within the priorities delin-eated in the strategic plan. The report tracksannual changes and also long-term trends onindicators such as degree production, successrates in credit courses, and transfers. The chan-cellor hopes to increase the disaggregation ofdata by population subgroups in future years.For now, the indicators help college leaders andfaculty focus on both day-to-day operationalobjectives and long-term developmental objec-tives, the priorities that are critical to “movingforward,” according to Chancellor Day. Howthe college is doing on its key indicators influ-ences the chancellor’s annual priorities that, inturn, guide budget decisions for the year, particu-larly in regard to developmental objectives.

Colleges that are

serious about building

and sustaining a

culture of evidence

and inquiry use

analyses of student

outcome data to shape

planning and budget

priorities.


The MetLife Foundation Community CollegeExcellence Award rewards colleges that candemonstrate institution-wide innovation strate-gies that help more low-income, minority, andfirst-generation college students succeed. At theheart of the efforts of the award recipients andfinalists is a commitment to using data effec-tively and creatively to support institutionalchange strategies and their implementation.

The four colleges highlighted in this brief haveused a variety of strategies to increase demandfor better data and more useful analyses and togenerate data for improvement in ways that canmove the college forward. These colleges havefound creative ways to build demand for data sothat requests come from across the institution—from top leadership and from faculty and staff.They have created systems and infrastructuresthat continually drive the data into the hands ofthe users. In so doing, they have considered the“brass tacks” of culture change: What techno-logical support do people need? What level ofeducating—and even hand holding—do facultyand staff need in order to consider the implica-tions of the data? What messaging, and whataccountability measures, can the college presi-dent use to promote the use of data to drivebetter outcomes? And how can a college weavethe analysis of student outcomes in the name ofimprovement into the very fabric of the institu-tion, its priorities, and its investment for thefuture?

A common theme in our research is that thenext challenge for these lead institutions is forfaculty and staff to make the transition fromdata users to data producers. “There is a lot ofinformation in this college that we don’tleverage—like how to succeed with specificpopulations,” said one college president. “The

next frontier is to figure out how to access whatour people know so we can keep improvingoutcomes for our students.”

These lead institutions see clearly where theystill need to strengthen and deepen their efforts.But they are exceptional, not typical. What willit take to help more community colleges institu-tionalize the kind of commitment to using datafor improvement that characterizes these MetLifeFoundation Award winners and finalists?

Ultimately, good intentions are not enough.Even strong leadership is likely to fall short inthe absence of external support and pressure,particularly at the state level. If more commu-nity colleges are to follow the lead of the fourcolleges profiled here, they will need specific,often technical, support for institutional use ofdata, combined with a political mobilizationeffort that will help states and state systemsplace a higher priority on data-informed deci-sion making and the diffusion of a culture ofevidence and inquiry.

H O W M O R E C O L L E G E S C A N B E E N C O U R A G E D T O

S T R E N G T H E N I N S T I T U T I O N A L R E S E A R C H C A P A C I T Y

A N D E F F E C T I V E N E S S

The next challenge for

these lead institutions is

for faculty and staff to

make the transition from

data users to data

producers.


At the technical level, states and state commu-nity college systems can take a number ofactions to help their colleges collect and usedata more easily and strategically, including:

• Create and support performance measurementsystems that define system goals clearly, iden-tify precise indicators of progress towardthose goals, and focus institutional effort onboosting the success rate of students whotraditionally have not fared well.

• Support the development of a fine-grained,state-level data system that enables states andinstitutions to probe the factors most highlyassociated with student success and to adjustbenchmarks and goals in light of these find-ings.

• Create a “virtual data warehouse” thatenables data from different education andworkforce systems to be linked for longitu-dinal analysis.

• Provide institutions with user-friendly accessto longitudinal data, data programming andresearch support, and training for institutionalresearch staff.

• Conduct state-level analyses and researchprojects focused on policies and practices thatmight improve student outcomes, using longi-tudinal student-level data that the statecollects and manages.

For most states to implement these kinds ofchanges will take political will—and that mostlikely means mobilizing key constituencies. Atleast three key interests have important leveragethat can be brought to bear:

• Accreditation agencies have the ability toratchet up expectations regarding the use ofevidence to guide institutional improvementplans and to measure institutional perform-ance. For colleges undergoing re-accreditation,some accreditors, such as the SouthernAssociation of Colleges and Schools and theNorth Central Association of Colleges andSchools, have made significant progress in thisdirection in the past decade. Given thenational discussion about accreditationsparked by the Secretary’s Commission on theFuture of Higher Education, this is a goodtime for accreditors to be proactive, providingan external stimulus to institutional efforts tostrengthen institutional research.

• Regional and national foundations can helppromote and support cross-state learning andpolicy benchmarking. Funders such as theMetLife Foundation, Lumina Foundation forEducation, and the Ford Foundation havedemonstrated significant leadership on strate-gies to build broad public and policymakerawareness and motivation to strengthen datasystems and strategic use of data. They havefunded reports, convenings, multi-statelearning and benchmarking efforts, and publicengagement campaigns that can help mobilizesupport for increased investment in institu-tional research.

• State community college systems, associations,or advocates can take steps to build support atthe institutional and state levels for greaterattention to evidence and data in improvementstrategies. A statewide organization ofresearch and institutional planners, as hasdeveloped in California, can be an importantprofessional development vehicle for facultyand staff, and it can help build a powerfuladvocacy voice from within communitycolleges. Support for involvement in multi-state learning and benchmarking networkscan also be a powerful way to strengthen

Strengthening

institutional research

and the routine use of

data for improvement—

so that the capacity

and sophistication of

research offices in these

four colleges is the

norm rather than the

exception—will require

both institutional and

policy change.


arguments and support for state innovationaround data and performance systems andsupport for a culture of evidence.

Strengthening institutional research and theroutine use of data for improvement—so thatthe capacity and sophistication of researchoffices in these four colleges is the norm ratherthan the exception—will require both institu-tional and policy change. The strategies pursuedby these four MetLife Foundation CommunityCollege Excellence Award winners and finalistsprovide important and very practical lessons forother colleges that want to embrace evidence-based decision making. The approaches of theseinstitutions to building capacity and trust, tointegrating data use into planning, budgeting,and teaching and learning, and to changinginstitutional culture have powerful lessons forother colleges around the country. In the end,though, the kind of transformation that thesecolleges have undergone and are continuing topursue widely and in a sustained way will notspread without policies that provide additionalresources, support more sophisticated dataanalyses and capacity, and give incentives toinstitutional leaders to drive change across theirschool. The two must go hand in hand: institu-tional excellence and supportive policies. Whenthey do, the success of these MetLifeFoundation Award winners and finalists will bemore easily replicable by other communitycolleges across the nation.

References

Morest, Vanessa, et al. 2006. Opening the BlackBox: Institutional Research at CommunityColleges. New York, NY: Community CollegeResearch Center.

Bailey, Thomas, Davis Jenkins, and D. TimothyLeinbach. 2007. The Effect of Student Goals onCommunity College Performance Measures.CCRC Brief Number 33. New York, NY:Community College Research Center.

MetLife Foundation Community College Excellence Awards

The MetLife Foundation Community College Excellence Award honors institu-tions that are effective in helping students from underrepresented populations tosucceed in postsecondary education. Every two years, the award, administeredby Jobs for the Future, goes to two colleges that make significant institutionalcommitments to helping first-time college-goers, new immigrants, workingadults, welfare recipients, high school dropouts, and other populations withlimited college experience and success prepare for further education or for afamily-supporting career.

Award winners demonstrate effective strategies and promising outcomes in:

• High-quality and flexible instructional programs;

• Academic, social, and financial supports;

• Strategies to create smooth transitions from secondary education; and/or

• Seamless transitions into and through credit-bearing courses (e.g., from devel-opmental or ESL courses).

2006 Award Winners: Indian River Community College in Fort Pierce, Florida,and LaGuardia Community College, in Long Island City, New York

2004 Award Winners: City College of San Francisco and Community College ofDenver

2002 Award Winners: West Hills Community College in California’s San JoaquinValley and Sinclair Community College in Dayton, Ohio


Acknowledgements

The authors would like to thank the manywonderful educators we interviewed at thecommunity colleges highlighted in this report.They took valuable time from their schedulesto explain their strategies and systems for usingdata to drive improvement in studentoutcomes, narrowing achievement gaps, andchanging institutional culture. We are in aweof their commitment and creativity. We wouldalso like to thank a few individuals who gaveearly drafts of this report a careful read: PaulArcario, Christopher Coogan, RobertGabriner, Christina Hart, Dan McConochie,and Vanessa Morest. Their insights havestrengthened this paper; any errors or mischar-acterizations, though, are solely our responsi-bility. We greatly appreciate the generosity ofMetLife Foundation for its ongoing support ofthe MetLife Foundation Community CollegeExcellence Award and of the research.

About the Authors

LILI ALLEN is a program director at Jobs forthe Future and manages several projectsrelated to youth education and transitions.She also is conducting research on systemicand programmatic approaches to addressingthe learning needs of out-of-school youth anddocumenting district redesign in cities acrossthe country. Her publications includeBuilding a Portfolio of High Schools: AStrategic Investment Toolkit (with Cheryl A.Almeida, Lucretia Murphy, and AdriaSteinberg) and Big Buildings, Small Schools:Using a Small Schools Strategy for HighSchool Reform (with Adria Steinberg).

RICHARD KAZIS, senior vice president at JFF,leads the organization’s policy and researchefforts. Recent publications include AdultLearners in Higher Education, RemakingCareer and Technical Education for the 21stCentury: What Role for High SchoolPrograms? and Double the Numbers:Increasing Postsecondary Credentials forUnderrepresented Youth.

JOBS FOR THE FUTURE seeks to accelerate theeducational and economic advancement ofyouth and adults struggling in today’seconomy. JFF partners with leaders in educa-tion, business, government, and communitiesaround the nation to: strengthen opportuni-ties for youth to succeed in postsecondarylearning and high-skill careers; increaseopportunities for low-income individuals tomove into family-supporting careers; andmeet the growing economic demand forknowledgeable and skilled workers.

Jobs for the Future88 Broad StreetBoston, MA 02110617.728.4446www.jff.org

Documents

BUILDING A CULTURE OF EVIDENCE IN COMMUNITY COLLEGES · Building a Culture of Evidence in Community Colleges i Acrosstheeducationsector,callsforaccounta - bilityandresults—andforgreatertransparency