Upload
texas-association-of-graduate-admissions-professionals
View
54
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Can we use effectively use predictors of success to allocate funding?
A study on the correlation between GRE scores and other factors of success in engineering
Presented by:Melissa Lee
Department of Civil Engineering
Presentation outline
• Introduction• Literature review• Methodology, results and discussion• Conclusions
Introduction
• Limited funding opportunities for students• Funding in the form of assistantships and
fellowships helps attract the best potential students
• How do we recruit and identify students?• How do we identify those most likely to succeed
through their graduate school applications?
Introduction
• Helps us better effectively manage enrolment and offers of funding
• How do we know what we know about graduate school admission?
• What do we know about how we admit students, our admission profile, and funding profile?
Existing LiteratureLittle to No correlation
Very accurate predictor
Valid but with other factors
• Cooksey and Stenning (1981)
• Morrison and Morrison (1995)
• Powers (2004) found that there is some validity, though it is clearly accompanied by other corresponding factors
• Holt, Bleckmann et al. (2006), found that GRE scores were a fairly accurate predictor of graduate school success
• Nathan Kuncel
Cooksey, L., & Stenning, W. F. (1981). The Empirical Impact of the Graduate Record Examination and Grade Point Average on Entry and Success in Graduate School at Texas A&M University. ERIC. Holt, D. T., Bleckmann, C. A., & Zitzmann, C. C. (2006). The Graduate Record Examination and Success in an Engineering Management Program: A Case Study. Engineering Management Journal, 18(1), 10-16. Morrison, T., & Morrison, M. (1995). A Meta-Analytic Assessment of the Predictive Validity of the Quantitative and Verbal Components of the Graduate Record Examination with Graduate Grade Point Average Representing the Criterion of Graduate Success. Educational and Psychological Measurement, 55(2), 309-316. doi: 10.1177/0013164495055002015Powers, D. E. (2004). Validity of Graduate Record Examinations (GRE) General Test Scores for Admissions to Colleges of Veterinary Medicine. Journal of Applied Psychology, 89(2), 208-219.
Existing literature
• Orlando (2005) – Reviewed a range of studies between 1977 and 2001– Different findings regarding validity– Some correlation, though there is disagreement to the extent of
correlation– Difference in disciplines, skills demanded, type of admission,
age
Orlando, J. (2005). The reliability of GRE scores in predicting graduate school success: a meta-analytic, cross-functional, regressive, unilateral, post-kantian, hyper-empirical, quadruple blind, verbiage-intensive and hemorrhoid-inducing study. Ubiquity, 2005, 1-1. doi: 10.1145/1071916.1071921
Existing Literature
• Nathan Kuncel – extensive research on GRE scores as predictor of success– Meta-analysis of different factors influencing graduate school success
(2001)– Synthesized studies done on all standardized tests (2007)– Another meta-analytic study on GRE scores for Master’s and Doctoral
programs (2010)
• Existing studies discussed the validity of GRE scores, but what can we do with this knowledge?
Kuncel, N. R., & Hezlett, S. A. (2007). Standardized Tests Predict Graduate Students' Success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162-181. doi: http://dx.doi.org/10.1037/0033-2909.127.1.162Kuncel, N. R., Wee, S., Serafin, L., & Hezlett, S. A. (2010). The Validity of the Graduate Record Examination for Master’s and Doctoral Programs: A Meta-Analytic Investigation. Educational and Psychological Measurement, 70(2), 340-352. doi: 10.1177/0013164409344508
Questions• Is there a relationship between GRE scores and GPA?
Are GRE scores an accurate predictor of success?• If GRE scores could predict success, how do we use this
knowledge to allocate funding?
Can GRE scores be used as an accurate determinant of success in graduate school?
Profile
• Civil engineering graduate students at a large, research university
• Enrolled each Fall from 2003-2013• Enrolments at three different degree
programs, Master of Engineering, Master of Science, and Doctoral program
Citizenship status
71
28
Citizenship status (% of total enrolment)
InternationalDomestic
Degree program
24
31
43 MENMSPhD
Percentage of students enrolled in each degree program
Is there a correlation between GRE scores and graduate school GPA?
• GRE scores and graduate school GPA were taken from the Fall enrolments between 2003-2013
• Total of 1,083 students sampled over this period• Multiple regression was applied across
enrollments during the 10 year period• GRE scores as the independent variable• Graduate school GPA as dependent variable
Correlation between GRE scores and GPA
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
Correlation 0.379 0.108 0.114 0.246 0.403 0.182 0.196 0.190 0.262 0.221 0.209
Coefficient 0.143 0.012 0.013 0.061 0.162 0.033 0.038 0.036 0.069 0.049 0.044
Significance -F 0.0065 0.732 0.634 0.099 0.0002 0.131 0.102 0.129 0.033 0.069 0.043
Sample size 68 56 72 77 100 123 120 114 99 110 144
Correlation between GRE scores and GPA (international students)
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
Correlation 0.144 0.216 0.257 0.611 0.117 0.156 0.147 0.270 0.243 0.288
Coefficient 0.021 0.047 0.066 0.374 0.013 0.024 0.022 0.073 0.059 0.083
Significance -F 0.632 0.376 0.201 0.003 0.562 0.540 0.485 0.130 0.177 0.051
Sample size 47 44 50 28 86 53 69 57 60 72
Correlation between GRE scores and GPA (domestic)
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
Correlation 0.174 0.083 0.347 0.611 0.343 0.156 0.174 0.317 0.256 0.166
Coefficient 0.030 0.007 0.120 0.374 0.118 0.024 0.030 0.100 0.065 0.028
Significance -F 0.912 0.918 0.215 0.003 0.119 0.540 0.522 0.127 0.204 0.495
Sample size 9 28 27 28 37 53 45 42 50 53
Is there a correlation between GRE scores and graduate school GPA?
• Positive correlation, but fairly weak• Corresponds to earlier studies between
GRE scores and graduate school GPA• Kuncel’s studies support that GRE scores
have some validity with CGPA
Kuncel, N. R., & Hezlett, S. A. (2007). Standardized Tests Predict Graduate Students' Success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162-181. doi: http://dx.doi.org/10.1037/0033-2909.127.1.162Kuncel, N. R., Wee, S., Serafin, L., & Hezlett, S. A. (2010). The Validity of the Graduate Record Examination for Master’s and Doctoral Programs: A Meta-Analytic Investigation. Educational and Psychological Measurement, 70(2), 340-352. doi: 10.1177/0013164409344508
Can GRE scores therefore be used to help allocate funding?
1.Can GRE scores be used as an accurate determinant for success in graduate school?
2.Can we identify and allocate funding to those most likely to succeed?
Questions
1. Are we using GRE scores in the selection of our graduate assistants?
2. How does our graduate assistant scores match up against those who don’t have assistantships?
3. Are we directing funding to the most successful students? (how does their GPA look against others?)
4. If we are not awarding assistantships to those who are not most likely to succeed based off GPA, then is the assistantship helping them or hurting them?
Does having an assistantship make a difference?
• Assistantship data from 2009 to 2013 is used for this determination
• Students offered either a teaching assistantship or graduate assistantship in their first year of enrolment in the program• GRE scores and Graduate school GPA• Undergraduate GPA and Graduate school
GPA
Assistantships - Correlation between GRE scores and GPA
• Total of 96 students were sampled between 2009-2013
• Only students who were offered assistantships in their first semester
• Correlation of 0.141. Positive, but a weak correlation between GRE scores and the GPAs of those given assistantships
Relationship between U-GPA and G-GPA
• Correlation of 0.04• Extremely weak correlation between
undergraduate GPA and graduate GPA for those who are on assistantships
• How else would we determine who is best suited to be teaching and research assistants?
Questions
1. Are we using GRE scores in the selection of our graduate assistants?
2. How does our graduate assistant scores match up against those who don’t have assistantships?
3. Are we directing funding to the most successful students? (how does their GPA look against others?)
4. If we are not awarding assistantships to those who are not most likely to succeed based off GPA, then is the assistantship helping them or hurting them?
Average GRE Verbal scores for those on Assistantships
• For each of the years analyzed, it was found that:
Year Median GRE - Verbal
Total number of GAs
Above Below
2009 470 32 13 17
2010 480 20 12 6
2011 490 24 12 11
2012 530 10 4 5
2013 490 12 6 4
Average GRE Quantitative scores for those on Assistantships
• For each of the years analyzed, it was found that:
Year Median GRE - Quantitative
Total number of GAs
Above Below
2009 750 32 17 11
2010 760 20 8 8
2011 760 24 12 11
2012 780 10 4 5
2013 770 12 6 4
Questions
1. Are we using GRE scores in the selection of our graduate assistants?
2. How does our graduate assistant scores match up against those who don’t have assistantships?
3. Are we directing funding to the most successful students? (how does their GPA look against others?)
4. If we are not awarding assistantships to those who are not most likely to succeed based off GPA, then is the assistantship helping them or hurting them?
Average GPA
Average GPA With assistantships No assistantships
MEN 3.69 3.58
MS 3.7 3.7
PhD 3.75 3.68
Questions
1. Are we using GRE scores in the selection of our graduate assistants?
2. How does our graduate assistant scores match up against those who don’t have assistantships?
3. Are we directing funding to the most successful students? (how does their GPA look against others?)
4. If we are not awarding assistantships to those who are not most likely to succeed based off GPA, then is the assistantship helping them or hurting them?
Average time to degree (semesters)
Average time to degree With Assistantships No Assistantships
MEN 4.5 4.37
MS 6.58 6.05
PhD 11.36 11.82
Conclusions
• We cannot rely on GRE alone in predicting future success in graduate school (determined by graduate school GPA and time to degree)
• Students on assistantships tend to do better in terms of overall grades and time taken to complete their degrees
Further research
• More study is required on how we admit students and distribute funding to students
• Bigger sample size will help give a more accurate picture of how and why students are admitted
• Indicators of success for graduate students• Can we create a matrix of sorts to help better allocate funding?
• Best practices in admission and funding• Eliman (1991) and Suhayda, Hicks and Fogg (2008) have also explored this
possibility in other disciplines
• Is graduate school admission an art or a science?• Considering the different aspects of evaluating applicants, particularly the
non-measurable factors, can we reduce each and every applicant in graduate school to a data point?
Eliman, A. A. (1991). A decision support system for Univeristy admission policies. European Journal of Operational Research, 50(2), 140-156. doi: http://dx.doi.org/10.1016/0377-2217(91)90237-PSuhayda, R., Hicks, F., & Fogg, L. (2008). A Decision Algorithm for Admitting Students to Advanced Practice Programs in Nursing. Journal of Professional Nursing, 24(5), 281-284. doi: http://dx.doi.org/10.1016/j.profnurs.2007.10.002
ReferencesAttiyeh, G., & Attiyeh, R. (1997). Testing for Bias in Graduate School Admissions. Journal of Human Resources, 32(3), 524-548.
Cieboter, F. J. (1969). Factors Related to the Performance of Foreign Graduate Students. The Journal of Educational Research, 62(8), 360-365. doi: 10.2307/27532232
Cooksey, L., & Stenning, W. F. (1981). The Empirical Impact of the Graduate Record Examination and Grade Point Average on Entry and Success in Graduate School at Texas A&M University. ERIC.
Eliman, A. A. (1991). A decision support system for Univeristy admission policies. European Journal of Operational Research, 50(2), 140-156. doi: http://dx.doi.org/10.1016/0377-2217(91)90237-P
Feeley, T. H., Williams, V., & Wise, T. (2005). Testing the Predictive Validity of the GRE Exam on Communication Graduate Student Success: A Case Study at University at Buffalo. Communication Quarterly, 53(2), 229-245. doi: 10.1080/01463370500090209
Freeman, R. B. (2005). Fellowship Stipend Support and the Supply of Science and Engineering Students: NSF Graduate Research Fellowships. American Economic Review, 95(2), 61-65.
Holt, D. T., Bleckmann, C. A., & Zitzmann, C. C. (2006). The Graduate Record Examination and Success in an Engineering Management Program: A Case Study. Engineering Management Journal, 18(1), 10-16.
House, J. D., & Johnson, J. J. (1993). Graduate Record Examination Scores and Academic Background Variables as Predictors of Graduate Degree Completion. Educational and Psychological Measurement, 53(2), 551-556. doi: 10.1177/0013164493053002025
Ingram, R. E. (1983). The GRE in the graduate admissions process: Is how it is used justified by the evidence of its validity? Professional Psychology: Research and Practice, 14(6), 711-714. doi: http://dx.doi.org/10.1037/0735-7028.14.6.711
Kuncel, N. R., & Hezlett, S. A. (2007). Standardized Tests Predict Graduate Students' Success. Science, 315(5815), 1080-1081. doi: 10.1126/science.1136618
Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162-181. doi: http://dx.doi.org/10.1037/0033-2909.127.1.162
Marston, A. R. (1971). It is time to reconsider the Graduate Record Examination. American Psychologist, 26(7), 653-655. doi: http://dx.doi.org/10.1037/h0032124
Morrison, T., & Morrison, M. (1995). A Meta-Analytic Assessment of the Predictive Validity of the Quantitative and Verbal Components of the Graduate Record Examination with Graduate Grade Point Average Representing the Criterion of Graduate Success. Educational and Psychological Measurement, 55(2), 309-316. doi: 10.1177/0013164495055002015
ReferencesOrlando, J. (2005). The reliability of GRE scores in predicting graduate school success: a meta-analytic, cross-functional, regressive, unilateral, post-kantian, hyper-empirical, quadruple blind, verbiage-intensive and hemorrhoid-inducing study. Ubiquity, 2005, 1-1. doi: 10.1145/1071916.1071921
Powers, D. E. (2004). Validity of Graduate Record Examinations (GRE) General Test Scores for Admissions to Colleges of Veterinary Medicine. Journal of Applied Psychology, 89(2), 208-219.
Robertson, M., & Nielsen, W. (1961). The graduate record examination and selection of graduate students. American Psychologist, 16(10), 648-650. doi: http://dx.doi.org/10.1037/h0043209
Rosser, M. (1970). Criteria used for admission by graduate departments. American Psychologist, 25(6), 558-560. doi: http://dx.doi.org/10.1037/h0037862
Rubio, D. M., Rubin, R. S., & Brennan, D. G. (2003). How Well Does the GRE Work for Your University? An Empirical Institutional Case Study of the Graduate Record Examination across Multiple Disciplines. College and University, 78(4), 11-17.
Saaty, T. L., France, J. W., & Valentine, K. R. (1991). Modeling the graduate business school admissions process. Socio-Economic Planning Sciences, 25(2), 155-162. doi: http://dx.doi.org/10.1016/0038-0121(91)90013-H
Sternberg, R. J., & Williams, W. M. (1997). Does the Graduate Record Examination predict meaningful success in the graduate training of psychology? A case study. American Psychologist, 52(6), 630-641. doi: http://dx.doi.org/10.1037/0003-066X.52.6.630
Suhayda, R., Hicks, F., & Fogg, L. (2008). A Decision Algorithm for Admitting Students to Advanced Practice Programs in Nursing. Journal of Professional Nursing, 24(5), 281-284. doi: http://dx.doi.org/10.1016/j.profnurs.2007.10.002
Thornell, J. G., & Mccoy, A. (1985). The Predictive Validity of the Graduate Record Examinations for Subgroups of Students in Different Academic Disciplines. Educational and Psychological Measurement, 45(2), 415-419. doi: 10.1177/001316448504500229
Thank you!