Hill and Serra, 2015 AACU, gen ed assessment ... critical thinking, and reflective judgment, and (2)

  • View
    0

  • Download
    0

Embed Size (px)

Text of Hill and Serra, 2015 AACU, gen ed assessment ... critical thinking, and reflective judgment, and (2)

  • Revisiting e-Assessment: Revisiting e Assessment: Leveraging online platforms for outcomes assessment in the general education

    Jennifer Hill, Ed.D.

    Matt Serra Ph DMatt Serra, Ph.D.

    Office of Assessment, Trinity College

    Duke University

    assessment.aas.duke.edu

    1

  • We argue that assessment occurs most We argue that assessment occurs most authentically in the context of a course, degree program, or co-curricular learning

    i C ll h ld i t i experience. Colleges should persist in promoting authentic, embedded assessment in their courses and programs. p g

    [ and ]

    Given that stakeholder groups also demand generalizable evidence of learning across the gen ed curriculum learning across the gen. ed. curriculum and co-curriculum, we suggest a parallel effort at the college level, to enrich and

    l i l l b ti f l iexplain local observations of learning.

    4

  • We argue that assessment occurs most We argue that assessment occurs most authentically in the context of a course, degree program, or co-curricular learning

    i C ll h ld i t i experience. Colleges should persist in promoting authentic, embedded assessment in their courses and programs. p g

    [ and ]

    Given that stakeholder groups also demand generalizable evidence of learning across the gen ed curriculum learning across the gen. ed. curriculum and co-curriculum, we suggest a parallel effort at the college level, to enrich and

    l i l l b ti f l i

    How do we do this at Duke?explain local observations of learning. Duke?

    5

  • Gen Ed assessment initiative

    Our purposes are

    ( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and

    (2) to follow the development of student competencies, broadly defined, over time.

    6

  • Gen Ed assessment initiative

    Our purposes are

    ( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and

    (2) to follow the development of student competencies, broadly defined, over time.

    Project design allows (a) wide engagement among incoming and exiting students

    7

  • Gen Ed assessment initiative

    Our purposes are

    ( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and

    (2) to follow the development of student competencies, broadly defined, over time.

    Project design allows (a) wide engagement among incoming and exiting students, (b) collection of data from three separate nationally-normed instruments, three separate nationally normed instruments,

    8

  • Gen Ed assessment initiative

    Our purposes are

    ( ) t th b li d t t i i t i i (1) to gather baseline data pertaining to incoming students’ levels of ethical reasoning, global perspectives, critical thinking, and reflective judgment, and

    (2) to follow the development of student competencies, broadly defined, over time.

    Project design allows (a) wide engagement among incoming and exiting students, (b) collection of data from three separate nationally-normed instruments, and (c) three separate nationally normed instruments, and (c) efficiencies of administration, data analysis, and reporting.

    9

  • Gen Ed assessment initiative (the front end)

     What

     To whom

     When

     How

    10

  • Gen Ed assessment initiative (the front end)

     What: Administration of several assessment instruments i t t d t ’ t i l ti t D k E h t t prior to students’ matriculation at Duke. Each targets a

    general education student learning outcome.

     To whom To whom

     When

     H How

    11

  • Gen Ed assessment initiative (the front end)

     What

     To whom: Voluntary sample of incoming first year (non- transfer) students, A&S and Engineering

     When

     How

    12

  • Gen Ed assessment initiative (the front end)

     What

     To whom

     When: Project started in summer 2010 and continues j annually. Early communications start in April, with periodic reminders in July and August. Data collection ends at the start of first year student orientation.

     How

    13

  • Gen Ed assessment initiative (the front end)

     What

     To whom

     When

     How: Deploy three online, externally hosted instruments via a central project homepage. Data are returned to us b tby our partners.

    14

  • Gen Ed assessment initiative (the front end)

    15

  • Gen Ed assessment initiative (the front end)

    16

  • Gen Ed assessment initiative

     Defining Issues Test (DIT-2) ethicaldevelopment.ua.edu

    Global Perspectives Inventory (GPI) gpi.central.edu

     Reasoning about Current Issues Test (RCI) reflectivejudgment.org

    Critical thinking Assessment Test (CAT) tntech.edu/cat/home

    17

  • Gen Ed assessment initiative

     Publicity: Blurb in “Blue Book”. Email from Director of Academic Advising Center Class Facebook page Academic Advising Center. Class Facebook page posting. Direct email from our office.

     Incentives

     Consent

     Return rates

     Challenges

     Findings Findings

     Next steps 18

  • Gen Ed assessment initiative

     Publicity

     Incentives: Drawing among eligible students. Awards and award values vary. “Hierarchy” of awards.

     Consent

     Return rates

     Challenges

     Findingsg

     Next steps 19

  • Gen Ed assessment initiative

     Publicity

     Incentives

     Consent: Consent statement is embedded on initial project homepage, and on each instrument’s website.

     Return rates

     Challenges

     Findingsg

     Next steps 20

  • Gen Ed assessment initiative

     Publicity

     Incentives

     Consent

     Return rates: Surprisingly high in year 1, fluctuating slightly over time. Lower among graduating cohort.

     Challenges

     Findingsg

     Next steps 21

  • Gen Ed assessment initiative

     Publicity

     Incentives

     Consent

     Return rates

     Challenges: Monitoring completions on externally hosted b R t d i i t ti t i di t web apps. Remote administration prevents immediate

    distribution of incentives. Maintaining high return rates.

     Findings

     Next steps

    22

  • Gen Ed assessment initiative

     Selected findings from pre-tests (matriculates 2010-2014):

    Our voluntary samples of study participants are large and tend to be representative of their respective incoming classes classes.

    Results at the pre-test tend to be stable across incoming classes. classes.

    Individual students’ results and Duke’s overall results are emailed to them as early as possible. Content of the mailing is confidential, secure, contextualized, and written in accessible language.

    23

  • Gen Ed assessment initiative Response rates by instrument, and by matriculating cohort

    Overall 59% Overall70%

    80% 90%

    100%

    d st

    ud en

    ts

    Overall 48% Overall

    42% Overall40% Overall36%

    30% 40% 50% 60%

    nt o

    f i nv

    ite d

    DIT-2 60%

    DIT-2 46%

    DIT-2 43%

    DIT-2 41%

    DIT-2 38%

    GPI 66%

    GPI 54%

    GPI 47%

    GPI 42%

    GPI 33%

    RCI 52%

    RCI 45%

    RCI 37%

    RCI 38%

    RCI 36%

    0% 10% 20%

    Incoming 2010 Incoming 2011 Incoming 2012 Incoming 2013 Incoming 2014

    Pe rc

    en

    24

    g g g g g

    DIT-2 GPI RCI Overall

  • GPIGPI Global Perspectives InventoryGlobal Perspectives Inventory

    25

  • Gen Ed assessment initiative: GPI

    1.0

    Observed movement in GPI results (pre-test to post-test) Repeat measures cohort (N=119), graduating class of 2014

    0.5 0.6 0.7 0.8 0.9 1.0

    0.0 0.1 0.2 0.3 0.4

    * * * * * al al al y n

    O ve

    ra ll G

    PI

    C og

    ni tiv

    e

    og . K

    no w

    in g

    . K no

    w le

    d ge

    nt ra

    pe rs

    on al

    In tra

    pe rs

    on a

    Id en

    tit y

    In tra

    pe rs

    on a

    A ffe

    ct *

    In te

    rp er

    so n a

    er pe

    rs on

    al Re

    sp on

    sib ilit

    y

    te rp

    er so

    na l

    al In

    te ra

    ct io

    n

    C o

    C og

    . In

    In te

    So ci

    al

    In t

    So ci

    a