Assessments Live.

Scoring the Natural Activities of Students and

Institutions

David K. Dirlam, Ph.D.Savannah Technical Collegeddirlam@savannahtech.edu

A Realistic Competition Two naturalist take a hike in the Smokey Mountains – home

of more different living species than all of Northern Europe. They have a friendly competition, each one pointing to a

new species that neither had named so far that day and describing when it grows, how fast, what resources it needs and what other species compete with it.

As the day wears on, new species get harder and harder to find and about 10% of the time a brief debate occurs over correct identification and description.

At the end of the 8-hour hike, they estimate that together they had averaged a species every three minutes for the day – over 150 species in all.

In the process they kept a few notes for the Park Ranger on ecological challenges, suggesting resources that need replenishing and competing species imbalances.

A Dream Competition Two institutional researchers walk around a campus,

visiting classrooms, labs, offices and auxiliary buildings. They have a friendly competition, each one pointing to a

new activity that neither had named so far that day and describing when it develops, how fast, what resources it needs and what other activities compete with it.

As the day wears on, new activities get harder and harder to find and about 10% of the time a brief debate occurs over correct identification and description.

At the end of the 8-hour tour, they estimate that together they had averaged an activity every five minutes for the day – almost 100 activities in all.

In the process they kept a few notes for the campus President on educational and ergological* challenges.* The study of the psychological effects of work, or of work patterns

This Presentation Will Cover…1. Why developmental rubrics are interesting and

useful.2. What the differences are between developmental

rubrics and scoring rubrics.3. When during development each of the four

common levels of developmental rubrics appears.4. How you can create a table of developmental

rubrics with someone else.5. Where there is a community of rubrics users.6. Who will help build the community of

developmental rubric users.

Why…

Are Developmental Rubrics Interesting and Useful?

Consider the benefits of… Giving instant feedback,

To naturally occurring activities,

Using open criteria,

And sequences that obey universal natural laws.

Developmental Rubrics (DRs)… Give instant feedback.

Stimulate quicker learning Reduce anxiety among learners

Assess naturally occurring activities. Have instantly recognizable validity Are reliable and easy to manage

Use open criteria. Are easily adaptable to local experience Stimulate growth in expertise and creativity

Use sequences that obey universal natural laws Provide a check on local or conventional opinions Stimulate growth in expertise and creativity Allow testable generalizations about remediation

Do you find the benefits of DRs interesting and useful?

What…

Is the Difference Between a Developmental Rubric and a

Scoring Rubric?

Developmental Rubrics (DRs)

Define unique performance levels across each of several criteria.

Characteristics of DRs Create reliability by classifying what people are doing,

rather than limiting it. You can efficiently use them in natural situations.

Evaluators who are experienced with a DR table can choose one level from a 4-level criterion in roughly 5 seconds

4 criteria in 20 sec, 8 criteria in 40 sec, 12 criteria in one minute.

They are so powerful you can uniquely categorize an enormous variety of activities using a few terms.

4, 4-level criteria categorize 256 unique actions. 8, 4-level criteria categorize 65,536 unique actions. 12, 4-level criteria categorize over 16 million actions (a person

experienced with the criteria would take one minute to uniquely categorize any volume in the Harvard University Library).

A Developmental Rubrics Table for Student Evaluation of Instruction

Continuation of DR Table

Notice The successor to each rubric is a goal that

the instructor could strive for next time. The student-assessors learn something

about what to expect of good teaching. Student-assessor reliability can be

determined. The teaching techniques described in the

table can be tested for effectiveness against ultimate student outcomes (later courses, jobs, etc.).

Multiple Choice Tests A means of increasing objectivity by using

artificial tasks and severely limiting the options of participants.

Delayed, but easy and reliable machine scoring Give summary judgment that is statistically

dubious Rarely hint at performance cause or remediation

potential. Allow normative research with correlational and

difference statistics.

Scoring Rubrics Use a variant of the PAGE paradigm (Poor-Average-Good-

Excellent) for every criterion. Applicable to free participant actions and real-world tasks. Judge rather than contribute to participant knowledge. Provide instant scoring but with low reliability dependent on

scorer authority. Provide statistically dubious summed scores with no

attribution of cause or remediation potential. Have limited research potential.

A Scoring Rubric Found on the Web

4 = Excellent: 3 = Good: 2 = Fair: 1 = Poor: 0 = Unacceptable

 Dimension  Descriptor  3  2  1 0

 visibility: does not hide work with body any more  than necessary       

 writing: writing is large and clearly written; level       

 speech: is audible and enunciated clearly; easy to  follow       

 attention to students: talks to students, not board; frequently  turns to face students       

 interaction: keeps students actively involved in process  by asking questions       

 modeling: portrays logical thinking by using a  efficient "think aloud" protocol       

student needs: responds to student needs to know; asks if  there are any follow-up questions       

A Multiple Choice Test Format for DRs

Which answer below best describes yourself? I do not expect to make much use of the information in this course in my work. I am learning the subject of this course, but don’t use it much outside of class and homework. I will be using the information in this course every day in my work within a year. I use the information and skills learned in this course everyday in my work or daily life.Which answer below that best describes your Instructor’s knowledge of the subject? Instructor reads from the text or their notes. Instructor compares views of experts within the field. Instructor compares views of experts in and outside of field and tells how to improve on their answers. Instructor develops unconventional sources about the field or contributes to knowledge in it.Which answer below best describes your Instructor’s knowledge of student development? Instructor often presents activities or skills that are too hard or too easy. Instructor presents a few learning strategies for each activity and emphasizes repetition of the strategy. Instructor presents a variety of new learning strategies that you did not think of before. Instructor describes several whole sequences of strategies needed to master various aspects of the activity being taughtWhich answer below that best describes your Instructor’s use of your time? Instructor meets every class but spends too much time on irrelevant issues. Instructor focuses well when class meets, but often cancels class. Instructor fills scheduled classes with effective learning activities. Instructor prepares for every scheduled class and often creates surprisingly effective learning activities.Which answer below that best describes your Instructor’s approach to course goals? Instructor rarely mentions how class activities relate to course goals. Instructor focuses on specific content for each daily lesson and mentions course goals occasionally. Instructor connects each daily lesson with other lessons, other fields, personal applications and the course goals. Instructor seeks to inspire, motivate and inform beyond the daily lesson. Instructor connects each daily lesson with teaching how to make

creative use of the subject to benefit others. Which answer below that best describes your Instructor’s approach to using college resources? Instructor relies solely on texts plus the materials and equipment in the classroom, shop or lab. Instructor uses college resources outside of the class, such as the library, but does not insure that students consult the college experts in those

resources. Instructor helps students draw from various college resources and the experts in them, such as the library, other departments and

administrative units. Instructor contributes to and helps students use societal sources outside of the college, such as the community or professional organizations.Which answer below that best describes your Instructor’s approach to student assessment? Instructor bases assessments on ease of data collection rather than course goals. Instructor rates students using a common type of assessment, such as the number of facts learned or the number of problems solved. Instructor rates students using a variety of assessments, including active student performance in natural situations. Instructor uses assessments as ways to learn and to discover and expand student strategies for achieving lesson goals in settings beyond the

college.

Tests vs. Scoring Rubrics vs. Developmental Rubrics  Multiple Choice Tests Scoring Rubrics Developmental Rubrics

Options Restrict participant and assessor debasing both

Restrict participant but free assessor leaving participant anonymous

Free and respect participant while restricting assessor to testable professional standards

Task Reality Artificial task Real-world tasks Real-world tasks

Judgment Judge Judge Lead

Scoring EaseMachine scoring possible Expert scoring with no

special trainingExperts with special training

Contribution to expertise None None Substantial

ValidityLimited but testable Limited by undefined

levelsAchievable by study of classes used

Reliability (scorer agreement) Nearly perfectible Low 90-95%

Source of reliabilityTest research Scorer authority Rubric research and trained

scorer agreement

SummaryStatistically dubious summed scores

Statistically dubious average scores

Result patterns with performance options

Creativity Allow none Depressed Encouraged

Timing Delayed days Immediate Immediate

Attributions of performance cause and potential remediation

None None Implied by score

Research potential

Normative information useful for correlational studies and difference tests

Limited Frequency distributions for creating and testing mathematical models of developmental processes

Can you reliably distinguish multiple choice tests and

scoring rubrics from developmental rubrics?

When…

During Development Does Each of the Four Common Levels of Developmental Rubrics

Appear?

Background of DR Research Standardized Developmental Ratings (SDRs) were

constructed from a mathematical model of the organization of human behavior created from 1970-1980 (Dirlam, 1972, 1980, Dirlam and Byrne, 1978).

In the late 1970s SDRs were used to train scorers for the New York State Regent’s Competency Exam in Writing.

In the early 1980s, SDRs were combined with the National Assessment’s Primary Trait Scoring to create scoring rubrics.

Analyses of over 1,000 drawings, over 300 writing samples and nearly 1,000 developmental research articles revealed that in all cases, ecology’s Lotka-Volterra model of succession in an ecosystem applied. This led to the four the patterns in Table 1. (see Dirlam, 1999, 2003)

Since the 1990s, Portfolio Management software companies have created Rubrics Wizards for scoring first writing and Teacher Education portfolios, but now portfolios extending across the curriculum to include even institutional planning.

Default vs. Pioneering Data Analyses in Developmental Research 1930-1992

Default, Pioneering and Dominant Distance Cues in Children Drawings Aged 5-19

Four Growth Patterns    

Default lichen/scribble

Pioneering weed/stickman

Transitional bush/curves

Dominant tree/realistic

Growth Factors Examples

The first to appear in development but collapse with any competition

The next to appear in development but destroy selves by exhausting resources with no competition

Slow growing but endure for a long time

Moderate growth with high competitive strength leads to eventual dominance

Initial Strength

Native human characteristics / Prior experience high low low low

Growth Rate

Reinforcement (utility, stimulation, affirmation, support, security) and punishment / Ease of performance / Instant gratification low high moderate moderate

Competitive Strength

Automaticity / Conformity / Social penetration (depth and breadth of relationship) low low high very high

ResourcesTime and effort available or social profit expectations

The effects of growth factors differ depending on the percent of resources an activity takes.

The Four Parameters Create the Different Growth Patterns Easiness (initial strength) without gratification,

dies off with any competition. Quick gratification (growth potential) without

abiding social usefulness, bursts on the scene but without competition it burns out rapidly (the “Moose” strategy).

Abiding social usefulness (competitive strength) with delayed or artificial gratification, endures but stagnates.

Creativity that produces gratification through social contribution (raises resource levels).

Examples Beginners agree to try an activity but make no

long-term commitment to it. Explorers have commitment but are deficient in

training or experience and may not yet be aware of what they don’t know.

Workers are usually deriving at least part of their income from the activity (alternative sources of income sometimes preclude an interest in remuneration on the part of people who have become quite adept at an activity).

Masters contribute to the activity by writing about it, leaving behind collectable artifacts, or supervising, educating or acting as models for others.

Expectations and Remediation Beginners can be easily disrupted by another

activity. Encouragement helps. Explorers exemplify that “a little knowledge is a

dangerous thing.” Discovery of alternative approaches is necessary to prevent burn-out or catastrophe.

Workers endure, but without inspiration. They stagnate.

Masters are the dominate approach not due to power so much as the creativity that allows them to prosper in changing circumstances.

Only 4 Levels? Often, each criterion has one of each of

the four growth patterns. This creates the four levels.

Some criteria for some activities or groups of people display either none or more than one of some of the four growth patterns resulting in 2 to 7 or more levels for a particular criterion.

These are Stages of Activities, not Stages of Life Every new activity brings about the four

levels. Persons who have passed the beginner

stage, but are still inexperienced are dangerous to themselves and others in that activity regardless of whether they aged 2, 42 or 82.

When you see unusual behavior from a person, always ask about their experience with the activity they are engaged in.

Testable Remediation for Each Level Beginner – Suggest or give opportunity to

imitate an explorer. Explorer – Suggest worker approaches to

each criteria separately and give time to acquire each.

Worker – Encourage creativity.

Can you describe the 4 growth patterns?

How…

Can You Create a Developmental Rubrics Table

with Someone Else?

Find a master Usually 10 or more years of experience. Has worked with several people of each

level of lesser experience (beginners, explorers and workers).

Has helped at least one other person to be able to make their living at the activity.

Interview the Volunteer Master Ask the respondent to think of as many

differences between the people representing the four levels of experience as they can.

The idea is not to create the definitive last word on the topic, but to uncover the concepts or impressions that have been most practiced or most vividly remembered by the respondent.

Use minimal questions and only try to stimulate the respondent’s memory without biasing it. Try using ideas related to the growth parameters. For example, “Do you remember an approach that some previously committed novices used a lot just before they quit?”

Keep notes.

Create a Profile Interview notes are entered into a rubrics

wizard or similar table. When a growth pattern is missing from a

criterion, offer a suggestion. Send the table based on your notes and

suggestions to the volunteer to correct any misconceptions.

Overview1. The entire process is somewhere between

the open-endedness of a Studs Terkel interview and the content-structured interview of psychological research.

2. Applying a powerful, but content-free structuring tool (developmental rubrics) after the interview reduces interviewer bias and creates a deeper conception of the activity than researchers could otherwise create.

3. The end-result should reveal what the masters think most useful to tell others about their activities.

Example 1: Singer Elizabeth Everett, 23-Apr-03

Criterion Beginning Exploring Working Creating

Mental focus Trapped by fear

Copying; no confidence Technically perfect Spiritually expressive

Physical focus

Unfocused Voice, a little breath Whole bodyBody + audience + environment

Repertoire Learning first song A few good songs Huge repertoire Improvises

Rehearsal

Once per week or per month

A few times per week

Much singing every day

Integrates singing into every day actions

Audience relationship

Fearful

Tries to “be” somebody (pretty, an instrument, etc) Focuses on self

Creating a religious experience for the other person

Ensemble relationship Feels not

worthy Cat fight

Delight in exchange; comfortable competition

Union into a whole ensemble; no ego

Example 2: Health Researcher Carl V. Philips, June 24 – October 7, 2003

Criterion Beginning Exploraing Working Creating

Societal needs for research

Aware of need for field, often have specific research or policy questions in mind, but unaware of how to answer them.

Mechanically apply methods of the field while often forgetting the questions that motivated their interest as beginners.

Become immersed in problems of a narrow specialty and lose perspective on how much of society’s resources should be devoted to it.

Seek solutions to societal needs with awareness of the costs and benefits.

Scientific Literature

Unaware of the massiveness of the literature related to their data.

Read widely but sporadically. Do a mechanical or ritualistic search through a topic, but don’t think through the implications.

Read within their specialty and know a variety of tools for dealing with it, but fail to see repeated mistakes and thus, often confuse the existence of many papers with substantial knowledge.

Seek to improve the imperfect tools of the field by reading for methodology rather than content. Read in various disciplines and problem areas to find methods that apply to analogous problems.

Funding sources

Still open to a variety of sources for their own funding.

Deal with the problem of having to sign on to a source of funding to keep going. Realize their dependence on a principal investigator.

Are adept at getting funding, but do not challenge the source of their funds.

Seek to improve knowledge regardless of funding source. Have found novel ways to deal with the lack of funding sources for basic questions.

Prior results Have no prior resultsContinue to use methods already learned. Do not challenge their own prior results.

Seek to improve upon their own prior understandings even if it means rejecting a conclusion that they previously published.

MethodsWant to answer questions from life experiences.

Apply methods learned in classes to problems haphazardly, often using methods inappropriate for the problem.

Apply established methods to new problems.

Try to develop new methods to improve knowledge.

Data treatmentGather data and follow instructions.

Take care in making charts and counts. Apply a limited variety of tests by rote.

Believe that data speaks for itself. Run the data through statistical packages without questioning conclusions.

Return to the data many times to improve analysis

Responses to study limitations Unaware of limitations

Have a global understanding that limitations exist.

Identify limitations but do little to overcome them.

Recognize limitations and seek to improve studies.

Scientific audience Write for advisors.

Write to the “world” without understand the individual interests within their audience. Writes to policy makers within their field.

Write to present and future scientists.

Phillips/Dirlam Model of Developmental Change

People try to maximize values related to an activity. For example eating involves maximizing healthfulness, accessibility (cost and availability), simple taste, and habitualness.

Practice enables them to be experts at a local minimum solution (the American high-fat/sugar, fast-food diet).

Change requires leaping to a radically new solution (vegan) and then learning to maximize it.

Model applies to work and recreational activities as well as personal maintenance.

Other Rubric Profiles Other types of work

Student Success Specialist, Early Childhood Teacher, Administrative Assistant, Car Salesperson, more than a dozen others.

Other life activities Loving relationships

Projects SACS project

Communities of Practice Developmental research

See www.ChangingWisdoms.com

Will you try to create a developmental rubrics table

after you get home?

Where…

Is There a Community of Rubric Users?

Sources Portfolio Management Software with Rubrics

Assessment Capability TaskStream LiveText Open Source Portfolio

Web Rubrics Sites Googling “rubrics” returns 3.6 million pages Googling “developmental rubrics” gets 73

including this paper and few of these use development rigorously or as a key concept.

Who…

Will Help Build a Community of Developmental Rubric Users?

You Can Help Build a DR Users Community Add your email to the sign up list. Add the email of anyone you think might be interested. Check out www.TaskStream.com Send comments, suggestions, developmental rubrics tables

or data to ddirlam@changingwisdoms.com and they will be published to the www.changingwisdoms.com website.

www.ChangingWisdoms.com will publish with authors names… Critiques of any rubrics tables on the

website (added to the comments on the appropriate table).

Frequency distributions of scorings for any developmental criterion (whether defined on the site or by the researcher).

All suggestions for building the DR Users Community.

www.ChangingWisdoms.com will provide free information from the DR Users Community for all who work on behalf of learners. May they come to know dispersion, growth

patterns, resources, and competition as well as naturalists.