A Delphi Study of Critical School Counseling Research Questions ASCA-ACES Research Summit St. Louis, Missouri June 28, 2003

A Delphi Study of A Delphi Study of Critical School Critical School

Counseling Research Counseling Research QuestionsQuestions

ASCA-ACES Research SummitASCA-ACES Research Summit

St. Louis, MissouriSt. Louis, Missouri

June 28, 2003June 28, 2003

National Center for School National Center for School Counseling Outcome ResearchCounseling Outcome Research

John CareyJohn CareyCarey DimmittCarey Dimmitt

http://www.umass.edu/http://www.umass.edu/schoolcounseling/schoolcounseling/

Center MissionCenter Mission

►The National Center for School The National Center for School Counseling Outcome Research is Counseling Outcome Research is dedicated to improving the practice of dedicated to improving the practice of School Counseling by developing the School Counseling by developing the research base that is necessary for research base that is necessary for responsible and effective practice. responsible and effective practice.

Center MissionCenter Mission

► The Center provides national leadership in The Center provides national leadership in the measurement and evaluation of the the measurement and evaluation of the outcomes of school counseling interventions outcomes of school counseling interventions and programs. The Center helps K-12 and programs. The Center helps K-12 leaders and practitioners use research leaders and practitioners use research findings and methods to improve school findings and methods to improve school counseling practice by enabling them to set counseling practice by enabling them to set standards, establish benchmarks, standards, establish benchmarks, implement evidence-based interventions, implement evidence-based interventions, measure student learning outcomes, and measure student learning outcomes, and evaluate program effectiveness.evaluate program effectiveness.

Why the Need for Empirical Why the Need for Empirical Studies of School Counseling Studies of School Counseling

Outcomes?Outcomes?►Need to Identify Best PracticesNeed to Identify Best Practices►Need to Identify Evidence-based Need to Identify Evidence-based

Interventions (NCLB)Interventions (NCLB)►Need to be AccountableNeed to be Accountable►Need to Demonstrate Impact on Need to Demonstrate Impact on

Student AchievementStudent Achievement►To Develop as a ProfessionTo Develop as a Profession

Leadership NeededLeadership Needed

► To Evaluate Existing EvidenceTo Evaluate Existing Evidence► To Identify Critical QuestionsTo Identify Critical Questions► To Identify the Most Important OutcomesTo Identify the Most Important Outcomes► To Identify or Develop Good Outcome MeasuresTo Identify or Develop Good Outcome Measures► To Develop New Research ApproachesTo Develop New Research Approaches► To Coordinate Multi-Site EffortsTo Coordinate Multi-Site Efforts► To Communicate the Implications of Research To Communicate the Implications of Research

Findings to PractitionersFindings to Practitioners

Center ActivitiesCenter ActivitiesNational FocusNational Focus

►Research Brief Listserve, 10,000 Research Brief Listserve, 10,000 subscriberssubscribers

►Summer Institute, “Using Data to Help Summer Institute, “Using Data to Help All Children Succeed” (July 14 thru 18)All Children Succeed” (July 14 thru 18)

►Review of Outcome Studies and Review of Outcome Studies and MeasuresMeasures

►Delphi Study of Critical School Delphi Study of Critical School Counseling Research QuestionsCounseling Research Questions

Center ActivitiesCenter ActivitiesRegional FocusRegional Focus

►Urban Guidance Directors Leadership Urban Guidance Directors Leadership GroupGroup

►Task Force on a Massachusetts School Task Force on a Massachusetts School Counseling ModelCounseling Model

►Program Evaluation and Leadership Program Evaluation and Leadership ConsultationConsultation

►Curriculum Development ConsultationCurriculum Development Consultation►Research Partnerships with School DistrictsResearch Partnerships with School Districts

Center New InitiativesCenter New Initiatives

►National Board for Evidence-Based National Board for Evidence-Based Practice in School CounselingPractice in School Counseling Establish CriteriaEstablish Criteria Select Critical OutcomesSelect Critical Outcomes Evaluate EvidenceEvaluate Evidence Identify Needed ResearchIdentify Needed Research Make Recommendations for Counselor Make Recommendations for Counselor

Education Education

Center New InitiativesCenter New Initiatives► Electronic Journal “School Counseling Electronic Journal “School Counseling

Outcomes”Outcomes” Promote empirical identification of best Promote empirical identification of best

practicespractices Promote the development of effective Promote the development of effective

research approachesresearch approaches Promote the development of quality Promote the development of quality

outcome measuresoutcome measures Disseminate findings broadly to multiple Disseminate findings broadly to multiple

constituenciesconstituencies Establish credibility for school counseling Establish credibility for school counseling

practicepractice

Delphi Study of Critical School Delphi Study of Critical School Counseling Research QuestionsCounseling Research Questions►Delphi Research Uses Expert Opinion toDelphi Research Uses Expert Opinion to

Forecast Future Trends and DirectionsForecast Future Trends and Directions Identify Critical IssuesIdentify Critical Issues Achieve Consensus and/or Map Points of Achieve Consensus and/or Map Points of

DisagreementDisagreement First Phase in Evidence-Based Professional BoardsFirst Phase in Evidence-Based Professional Boards

►Delphi Research is Especially Appropriate for Delphi Research is Especially Appropriate for Identifying Approaches to Complex Problems Identifying Approaches to Complex Problems in Areas with Multiple Constituenciesin Areas with Multiple Constituencies

►Delphi Research Has Been Used Successfully Delphi Research Has Been Used Successfully by National Agencies to Develop Priorities for by National Agencies to Develop Priorities for Research FundingResearch Funding

Delphi Study of Critical School Delphi Study of Critical School Counseling Research QuestionsCounseling Research Questions►Panel SelectionPanel Selection►Phase I: Identify the Purposes of Phase I: Identify the Purposes of

School Counseling ResearchSchool Counseling Research►Phase II: Identify Specific Research Phase II: Identify Specific Research

Questions For Each PurposeQuestions For Each Purpose►Phase III: Prioritize Research Phase III: Prioritize Research

Questions and Identify Criteria for Questions and Identify Criteria for Judging ImportanceJudging Importance

Expert PanelExpert Panel

► Judy BowersJudy Bowers► Carol DahirCarol Dahir► Teesue FieldsTeesue Fields► Norman GysbersNorman Gysbers► Trish HatchTrish Hatch► Richard HayesRichard Hayes► Peggy HinesPeggy Hines► Reese HouseReese House► Curly JohnsonCurly Johnson► Dawn KayDawn Kay► Judy MaddenJudy Madden

► Brenda MeltonBrenda Melton► Linda MillerLinda Miller► Rachelle PerusseRachelle Perusse► Lonnie RowellLonnie Rowell► Russell SabellaRussell Sabella► Susan SearsSusan Sears► Toni TollerudToni Tollerud► Jose VillabaJose Villaba► Susan WhistonSusan Whiston► Richard WongRichard Wong

Purposes of School Purposes of School Counseling Research: Phase ICounseling Research: Phase I

►Delphi Study Question 1: Given the Delphi Study Question 1: Given the history and current political context of history and current political context of school counseling, what ends should school counseling, what ends should research achieve?research achieve?

Purposes of School Purposes of School Counseling Research: Phase I Counseling Research: Phase I

ResponsesResponses►To identify evidence-based To identify evidence-based

interventions and best practices in interventions and best practices in terms of school counseling activities, terms of school counseling activities, interventions, programs, and models.interventions, programs, and models.

““Research can provide practitioners Research can provide practitioners with information on which with information on which interventions really help students.”interventions really help students.”


ResponsesResponses►To document effectiveness in order to be To document effectiveness in order to be

accountable to external constituencies accountable to external constituencies for school counseling activities, for school counseling activities, interventions, programs, and models.interventions, programs, and models.

Research needs “to show the results of a Research needs “to show the results of a comprehensive school counseling comprehensive school counseling program and demonstrate to program and demonstrate to stakeholders the effectiveness of the stakeholders the effectiveness of the program.”program.”


ResponsesResponses► To understand how to effect change in To understand how to effect change in

school counselor practice, school counseling school counselor practice, school counseling programs, and school counselor roles in programs, and school counselor roles in schools.schools.

““More research is needed to discover what More research is needed to discover what factors are contributing to or influencing the factors are contributing to or influencing the changes in school counselors’ beliefs and changes in school counselors’ beliefs and behaviors regarding the performance of behaviors regarding the performance of counseling and non-counseling activities.”counseling and non-counseling activities.”


ResponsesResponses►To identify the most effective approaches To identify the most effective approaches

to educating and supervising school to educating and supervising school counselors in training.counselors in training.

““If best practices could be determined, If best practices could be determined, they might help provide some kind of they might help provide some kind of standard by which all school counselors standard by which all school counselors could be prepared consistently.”could be prepared consistently.”

Specific Research Questions: Specific Research Questions: Phase II QuestionsPhase II Questions

►““1. Do your ideas about the purposes 1. Do your ideas about the purposes of school counseling research fit into of school counseling research fit into one or more of these categories?one or more of these categories?

►2. Would you add to or redefine the 2. Would you add to or redefine the definitions of any of these purposes?definitions of any of these purposes?

►3. Please identify what you consider 3. Please identify what you consider to be an important, specific research to be an important, specific research question for each purpose, if possible.”question for each purpose, if possible.”

Specific Research Questions: Specific Research Questions: Phase II ResponsesPhase II Responses

►Responses were collapsed into 42 Responses were collapsed into 42 distinct Research Questions.distinct Research Questions.

Specific Research Questions: Specific Research Questions: Phase III RatingsPhase III Ratings

► Participants were asked to rate the Participants were asked to rate the importance of each Research Questionimportance of each Research Question 1=Not Important1=Not Important 2=Important2=Important 3=Very Important3=Very Important 4=Critically Important4=Critically Important

► Participants were asked to identify criteria Participants were asked to identify criteria used in judging importanceused in judging importance

► Participants were asked to supply additional Participants were asked to supply additional questionsquestions

SurveySurvey

►Please CompletePlease Complete►We would like your completed surveyWe would like your completed survey

Phase III ResultsPhase III Results

► 23. Which specific School Counseling 23. Which specific School Counseling interventions result in the greatest gains in interventions result in the greatest gains in student’s academic development and student’s academic development and achievement? achievement? 3.83.8

► 27. What specific School Counseling 27. What specific School Counseling interventions are most effective at preventing interventions are most effective at preventing school dropout? school dropout? 3.73.7

► 30. What is the impact of enhancing School 30. What is the impact of enhancing School Counselors’ ability to use data in planning Counselors’ ability to use data in planning and accountability on student outcomes? and accountability on student outcomes? 3.73.7

► 26. What specific school counseling 26. What specific school counseling interventions are most successful at interventions are most successful at preventing school failure? preventing school failure? 3.63.6

Phase III ResultsPhase III Results► 37. What are the most effective approaches to 37. What are the most effective approaches to

changing school leaders’ opinions about the value changing school leaders’ opinions about the value and legitimacy of school counseling? and legitimacy of school counseling? 3.6 3.6

► 2. What factors contribute to the success of a 2. What factors contribute to the success of a school counseling program? school counseling program? 3.63.6

► 8. What’s the impact of implementing a results-8. What’s the impact of implementing a results-based school counseling program on students’ based school counseling program on students’ academic development and achievement? academic development and achievement? 3.63.6

► 4. What’s the impact of implementing a 4. What’s the impact of implementing a Comprehensive Developmental School Counseling Comprehensive Developmental School Counseling Program on students’ academic development and Program on students’ academic development and achievement? achievement? 3.63.6

► 36. What are the most effective approaches to 36. What are the most effective approaches to changing school leaders’ opinions about school changing school leaders’ opinions about school counselor role and function? counselor role and function? 3.53.5


► 24. Which specific School Counseling interventions 24. Which specific School Counseling interventions result in the greatest gains in students’ career result in the greatest gains in students’ career development? development? 3.53.5

► 25. Which specific School Counseling interventions 25. Which specific School Counseling interventions result in the greatest gains in students’ result in the greatest gains in students’ personal/social development? personal/social development? 3.53.5

► 6. What is the impact of implementing a 6. What is the impact of implementing a Comprehensive Developmental School Counseling Comprehensive Developmental School Counseling Program on students’ personal/social development? Program on students’ personal/social development? 3.53.5

► 9. What is the impact of implementing a Results-9. What is the impact of implementing a Results-based School Counseling Program on students’ based School Counseling Program on students’ career development? career development? 3.53.5

► 10. What is the impact of implementing a Results-10. What is the impact of implementing a Results-based School Counseling Program on students’ based School Counseling Program on students’ personal/social development? personal/social development? 3.53.5


► 12. What is the impact of implementing an ASCA 12. What is the impact of implementing an ASCA National Model School Counseling Program on students’ National Model School Counseling Program on students’ academic development and achievement? academic development and achievement? 3.53.5

► 35. What is the impact of enhancing school counselors’ 35. What is the impact of enhancing school counselors’ abilities to use data in planning and accountability on abilities to use data in planning and accountability on school leaders’ perceptions of the value and legitimacy school leaders’ perceptions of the value and legitimacy of school counseling? of school counseling? 3.53.5

► 1. What is a successful School Counseling program? 1. What is a successful School Counseling program? 3.43.4

► 11. What is the impact of implementing a Results-11. What is the impact of implementing a Results-based School Counseling Program on school leaders’ based School Counseling Program on school leaders’ perceptions of the value and legitimacy of school perceptions of the value and legitimacy of school counseling? counseling? 3.43.4

► 3. What are the factors that contribute to the efficacy of 3. What are the factors that contribute to the efficacy of a school counseling program’s implementation? a school counseling program’s implementation? 3.43.4


► 13. What is the impact of implementing an ASCA 13. What is the impact of implementing an ASCA National Model School Counseling Program on National Model School Counseling Program on students’ career development? students’ career development? 3.43.4

► 28. Under what circumstances are whole classroom, 28. Under what circumstances are whole classroom, small group, and individual counseling maximally small group, and individual counseling maximally effective? effective? 3.43.4

► 5. What is the impact of implementing a 5. What is the impact of implementing a Comprehensive Developmental School Counseling Comprehensive Developmental School Counseling Program on students’ career development? Program on students’ career development? 3.33.3

► 15. What is the impact of implementing an ASCA 15. What is the impact of implementing an ASCA National Model School Counseling Program on school National Model School Counseling Program on school leaders’ perceptions of the value and legitimacy of leaders’ perceptions of the value and legitimacy of school counseling? school counseling? 3.33.3

► 16. What are the best ways to measure the extent 16. What are the best ways to measure the extent to which a school counseling program model is fully to which a school counseling program model is fully implemented? implemented? 3.33.3


► 19. What is the impact of reducing 19. What is the impact of reducing student/counselor ratios on program outcomes? student/counselor ratios on program outcomes? 3.33.3

► 31. What are the most effective approaches to 31. What are the most effective approaches to enhancing the leadership skills of School Counselors? enhancing the leadership skills of School Counselors? 3.33.3

► 41. What are the most effective professional 41. What are the most effective professional development approaches to enhance School development approaches to enhance School Counselors’ competence? Counselors’ competence? 3.33.3

► 7. What is the impact of implementing a 7. What is the impact of implementing a Comprehensive Developmental School Counseling Comprehensive Developmental School Counseling Program on school leaders’ perceptions of the value Program on school leaders’ perceptions of the value and legitimacy of school counseling? and legitimacy of school counseling? 3.23.2

► 14. What is the impact of implementing an ASCA 14. What is the impact of implementing an ASCA National Model School Counseling Program on National Model School Counseling Program on students’ personal/social development? students’ personal/social development? 3.23.2

Phase III ResultsPhase III Results► 22. What research and evaluation skills are necessary 22. What research and evaluation skills are necessary

for effective School Counseling practice? for effective School Counseling practice? 3.23.2► 38. What is the impact of collaboration between 38. What is the impact of collaboration between

guidance leaders and school counselor educators on guidance leaders and school counselor educators on the quality of school counseling training and the the quality of school counseling training and the competence of school counseling trainees? competence of school counseling trainees? 3.23.2

► 39. What is the impact of aligning a school counselor 39. What is the impact of aligning a school counselor education program with a school reform agenda on education program with a school reform agenda on the quality of school counseling training and the the quality of school counseling training and the competence of school counseling trainees? competence of school counseling trainees? 3.23.2

► 40. What is the impact of aligning a school counselor 40. What is the impact of aligning a school counselor education program with the ASCA National Model on education program with the ASCA National Model on the quality of school counseling training and the the quality of school counseling training and the competence of school counseling trainees? competence of school counseling trainees? 3.13.1

Phase III ResultsPhase III Results► 17. What is the impact of aligning a school counseling 17. What is the impact of aligning a school counseling

program with the school-reform agenda on students’ program with the school-reform agenda on students’ academic development and achievement? academic development and achievement? 3.03.0

► 33. To what extent do School Counselors trained in 33. To what extent do School Counselors trained in leadership and systemic change models attempt to leadership and systemic change models attempt to effect change in their schools? effect change in their schools? 3.03.0

► 42. What is the impact of school counselor educators’ 42. What is the impact of school counselor educators’ work experience in schools on the orientation and work experience in schools on the orientation and outcomes of counselor education programs? outcomes of counselor education programs? 3.03.0

► 18. What is the impact of implementing the use of 18. What is the impact of implementing the use of specific competency-based student learning specific competency-based student learning objectives in a School Counseling Program on student objectives in a School Counseling Program on student outcomes? outcomes? 2.92.9

► 32. What are the most effective approaches to 32. What are the most effective approaches to developing and enhancing the professional identity of developing and enhancing the professional identity of school counselors? school counselors? 2.82.8


► 34. What are the most effective approaches 34. What are the most effective approaches to enhancing the program organization and to enhancing the program organization and management skills of school counselors? management skills of school counselors? 2.72.7

► 21. Do school counselors who use research 21. Do school counselors who use research to guide practice have a greater impact on to guide practice have a greater impact on student outcomes? student outcomes? 2.62.6

► 29. What is the impact of peer helper 29. What is the impact of peer helper programs on students’ attendance, grades programs on students’ attendance, grades and school behavior? and school behavior? 2.32.3

► 20. What is the impact of different methods 20. What is the impact of different methods of assigning students to counselors on of assigning students to counselors on different outcomes? different outcomes? 2.12.1

Criteria for Judging Criteria for Judging ImportanceImportance

► Research that demonstrates impact on Research that demonstrates impact on outcomes is needed because school counseling outcomes is needed because school counseling is “at risk”is “at risk”

► Research that would significantly add to our Research that would significantly add to our understanding of “best practices”understanding of “best practices”

► Research that is intrinsically interesting “to Research that is intrinsically interesting “to me”me”

► Research that advances the efficacy and Research that advances the efficacy and effectiveness of school counselor trainingeffectiveness of school counselor training

► Research that helps the profession meet the Research that helps the profession meet the challenges associated with NCLB and challenges associated with NCLB and implementing Transforming School Counseling implementing Transforming School Counseling Initiative and ASCA National Model reforms.Initiative and ASCA National Model reforms.

Additional QuestionsAdditional Questions

►What is the technological literacy of school What is the technological literacy of school counselors? How do school counselors counselors? How do school counselors implement technology? How does implement technology? How does implementing technology impact implementing technology impact outcomes?outcomes?

►What is the process through which school What is the process through which school counselors make decisions and how can counselors make decisions and how can decision-making be improved?decision-making be improved?

►What is the impact of school counselor self-What is the impact of school counselor self-reflection on efficacy and effectiveness?reflection on efficacy and effectiveness?

National Center for School National Center for School Counseling Outcome ResearchCounseling Outcome Research

Thank YouThank Youhttp://www.umass.edu/schoolcounseling/

Documents

A Delphi Study of Critical School Counseling Research Questions ASCA-ACES Research Summit St. Louis, Missouri June 28, 2003