Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20061

October 27, 2006 1

Columbia University :: Office of the Provost :: Planning and Institutional Research2006-7 NRC Assessment of Research-Doctoral Programs

The National Research Council The National Research Council Assessment of Research Doctorate Programs, 2006-07Assessment of Research Doctorate Programs, 2006-07

Assessment Overview and Implications for Columbia

Lucy DrotningAssociate Provost for Planning and Institutional Research

John ScanlonData Manager, Planning and Institutional Research

October 27, 2006 2


BackgroundBackground

1. Collection of quantitative data through questionnaires

2. Data on faculty publications, citations, and research activity and student dissertation keywords

3. Faculty opinion on the relative importance of measures of program quality

Who is doing what?

• NRC• Mathematica Policy Research (MPR)• Columbia

Goal

The stated goal of the study is to “help universities improve the quality of these programs through benchmarking; provide potential students and the public with accessible, readily available information on doctoral programs nationwide; and enhance the nation's overall research capacity.”

The study will consist of:

October 27, 2006 3


End results: what will the study produce?End results: what will the study produce?

• A quantitatively based methodology for rating graduate programs

• Online database of graduate programs available to public

• Expert analyses and reports

October 27, 2006 4


Process and TimelineProcess and Timeline

NRC Stages Schedule Status @ Columbia

1. Collection of Program Data December 15, 2006* In process

1.a. Collection of the program lists August 2006 Submitted

1.b. Collection of Faculty names by program December 1, 2006 In process

1.c. Collection of the names of admitted to candidacy students in 5 fields.

December 1, 2006 In process

2. Collection of Institutional Data December 15, 2006 In process

3. Collection of Faculty Data

Questionnaire sent to all faculty identified in 1.b.

December-February 2006

To be administered by MPR

4. Collection of Student Data

Questionnaire sent to all students identified in 1.c.

December-February 2006

To be administered by MPR

5. Rating Questionnaire

Questionnaire administered to a random sample of faculty in each field

March-April 2007 To be administered by MPR

October 27, 2006 5


ProgramsPrograms

Program Questionnaires

What’s being collected?

•Faculty Lists•List of all doctoral graduates from the past three years.•General program characteristics and policies.•Admissions, enrollment and degree completion data.•Detailed financial support data for full-time doctoral students in fall 2005.

How can programs help with the process?

•Confirm/correct data that can be collected centrally•Provide answers to questions that cannot be answered centrally

Program List:

Submitted in September to MPR; 56 total programs submitted. See website for all programs and field designations.

October 27, 2006 6


Programs- Faculty ListsPrograms- Faculty Lists

Collection of faculty names by program

For each participating program, we have to submit a list of faculty members. Faculty members must be classified in one of 3 categories:

CORE•has served on a dissertation committee in the past 5 years OR is a member of admissions or curriculum committee•is currently and formally designated as faculty in the program

NEW•is currently and formally designated as faculty in the program, but does not meet the Core requirements•has been hired in a tenured or tenure-track position in the past 3 academic years.

ASSOCIATED•has served on a dissertation committee in the past 5 years•is not currently and formally designated as faculty in the program

Note: Faculty members can be reported in more than one program

October 27, 2006 7


Programs-Faculty Lists, cont’dPrograms-Faculty Lists, cont’dWhat’s being collected?

•Name•Street address and email address•Category (i.e. Core, New, Associated)•Highest degree awarded•Rank and tenure status•Number of dissertation committees “chaired” (i.e. “Sponsored”) in the past 5 academic years•Total number of dissertation committees served on in the past 5 academic years.

How will the data be used?

•NRC will use email addresses to contact faculty members directly for faculty questionnaire.•NRC will collect citation and publication data for faculty members.•NRC will use a formula to allocate faculty headcount and productivity (publications and citations) for faculty serving in more than one graduate program.


•Confirm/correct list: are all the appropriate faculty members listed? Is anyone missing? Is anyone on the list who shouldn’t be there? Is anyone categorized incorrectly?•Confirm/correct faculty name, contact, rank, tenure, and degree information, especially email address

For each program, we will indicate all other programs in which the faculty are listed; we will also list anyone not included at all but who is appointed in the program.

October 27, 2006 8


Programs-Faculty ProductivityPrograms-Faculty Productivity

Faculty productivity will be calculated based on status in program and nature of dissertation service.

Example: Professor Smith is considered Core in Program A and Program B. Between 2001-02 and 2005-06, she served as principal advisor on 3 dissertations and participated in 4 others in Program A and served as principal advisor on 1 dissertation and participated in 2 others in Program B.

= 69% allocated to Program A.And therefore, 31% allocated to Program B.

October 27, 2006 9


Faculty QuestionnaireFaculty Questionnaire

Who receives the survey?

All Core and New faculty members.


•Name and contact information.•Demographic data.•Specific information about faculty’s research field(s).•Education and work experience.•Doctoral graduates for the past 5 years.•Information about scholarship/research activity (with opportunity to upload CV).•Section on “relative importance of program characteristics to program quality.”


•Information on names used to aid in NRC’s productivity data collection•Information on doctoral graduates will be used to check NRC’s effort to gather placement data•“Program quality” responses will be used to identify and weigh key study variables in conjunction with “Anchoring” study.

October 27, 2006 10


Anchoring StudyAnchoring Study

Who receives the survey?

Random sample of Core faculty members • who completed faculty questionnaire• who indicated willingness to participate.


•The faculty will receive a list of program faculty for a sample of 15 to 20 programs as well as additional program data and information (e.g. faculty diversity, student completion & placement)

•Respondents will:• indicate familiarity with program• rate the quality of the programs on a 5-point scale.


Respondent ratings regressed on quantitative variables to derive weights of the key study variables.

October 27, 2006 11


SummarySummary


•Recognize that there still is uncertainty in the process

•Think through faculty lists carefully, and

•Encourage faculty participation!

For more information:

Website: http://www.columbia.edu/cu/opir/nrc/index.html

Email: [email protected]

Documents

Columbia University :: Office of the Provost :: Planning and Institutional Research 2006-7 NRC Assessment of Research-Doctoral Programs October 27, 20061