The AERO CRCD “Smart Materials” Project 1 Preparation for Evaluation Adaptation, Evolution and Application of The TAMU-TIDEE Design Knowledge Test Rita

The AERO CRCD “Smart Materials” Project 1

Preparation for Evaluation

Adaptation, Evolution and Application

of

The TAMU-TIDEE Design Knowledge Test

Rita Caso, Ripla Kohli, Jong-Hwan Lee,

Texas A&M University Developed from Work In Progress presentation made at FIE Conference, Session

1199, 6-9 November , Boston, MA.

Acknowledgements

This material is based upon work supported by the National Science Foundation under the following grants:

EEC-9802942

EEC-0088118

Adaptation of the Transferable Integrated Design Engineering Education (TIDEE), Mid Program Assessment: A Three Part Assessment of Team Based Design for Entering Juniors 2, Part 1, and its evolution as the TAMU-TIDEE Design Knowledge Test

Part I

Legend for.. Diagrams of Instrument & Rubric Evolution,

and Data Collection

• Instrument Version

• Informed Judgment to Change Instrument or Rubric

• Instrument Scored with Indicated Program Rubric

• Baseline Data Collected for Program Evaluation

• Data Collected for Design Assessment Instrument Development

Instrument & Rubric EvolutionTIDEE Instrument Prototype TIDEE Rubric Prototype

V.2 TAMU-TIDEE Rubric

TAMU-TIDEE Test V.3


TAMU-TIDEE Test V.2


FutureTAMU-TIDEE Test V.4

TAMU-TIDEE Test V.1

Data CollectionTIDEE Instrument Prototype TIDEE Rubric Prototype


Fall 2002V.3 TAMU-TIDEE TestEE Seniors*


Spring 2002V.2 TAMU-TIDEE TestFreshmenAero SeniorsProfessors


V.4 TAMU-TIDEE Test

Fall 2001 V.1 TAMU-TIDEE TestFreshmenAero Seniors

TIDEE Definitions for Levels of Achievement in Design Education Outcomes

Level Description

Basic Knowledge Recognition and understanding of facts, terms, definitions, descriptions, relationships, processes.

Application of Use of knowledge in ways that Knowledge demonstrate understanding of concepts or

terms, their proper use, and limitations of their applicability

Critical Analysis Examination and evaluation of informative required to judge its value to a design solution and to make decision.

Extension of Extensive knowledge beyond what was received, creating new knowledge, making inferences, transferring knowledge to usefulness in new area of application,

making improvements.

Key to Changes Made in Next Version of Instrument

• Text in red was identified as needing change in NEXT version– Example: Respond to the following questions/statements. You

have 15 mins.

• Text in red, set apart by “* *” indicates that something additional is needed in NEXT version

– Example: *( Lacks specific instructions for response) *

• Text in italics indicate new wording to that version– Example: In the space provided, describe your

understanding of the engineering design process, in diagram or flowchart form, in complete sentences, or in list form.

• Text which is both italicised and in red indicates that the wording is new to that version, but that it was identified as needing change in NEXT version

– Example: List the qualities that describe effective teamwork

Prototype: The TIDEE Design Knowledge Assessment Instrument 3

(TIDEE Website, June 2001)

Objective: Demonstrate your knowledge of key elements in the engineering design process, teamwork, and communication associated with team-based engineering design.

Assignment: Respond to the following questions/statements. You have 15 mins.

A. In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe and/or diagram your understanding of the engineering design process. *( Lacks specific instructions for response) *

B. List the elements required for effective teamwork in a design project assignment, and explain why each element is important.

C. In team-based design, documentation and exchange of design information are important. List features of effective communication. Consider a wide range of oral and written communication situations.

V.1--TAMU TIDEE Test: 1st Adaptation Given, as Pre-test, to Aero Senior Design Class and “Smart

Material” Freshman Class in Fall 2001


Assignment: Respond to the following questions/statements as instructed (Time allowed 20-30 mins.)

A. In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe and/or diagram your understanding of the engineering process. (Express yourself in complete sentences or in acombination of diagrams and clear phases.)

B. What kinds of things should teams do to prepare themselves beforehand for effective teamwork in a design project assignment? Respond in the form of a list. (Separate your list into different categories of activities)

C. In team-based design, documentation and exchange of design information are important. List the characteristics of written and oral communication that will be essential in a team design project.

V.2--TAMU TIDEE Test: 2nd Adaptation Given, as Post-test, to Aero Senior Design Class and a

Different Freshman Class in Spring 2002



A. In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe your understanding of the engineering design process, in diagram or flowchart form, in complete sentences, or in list form.

B. What kinds of things should teams do to prepare themselves for effective teamwork in a design project assignment? Respond in the form of a list or in sentences.

C. In team-based design, documentation and exchange of design information are important. Describe communication qualities and how communication should occur among team members in a design project.

V.3--TAMU TIDEE Test: 3rd Adaptation Given, as Pre-test, to Electrical Engineering Design Class in

Fall 2002



A. In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe your understanding of the engineering design process, in diagram of flowchart form, in complete sentences, or in list form.

B. Part A: For a Team-Based Engineering Design Project, what are the processes that contribute to good teamwork? List as many as you can.

Part B: List the qualities that describe effective teamwork. List as many as you can.

C. What are the component characteristics of good communication with regards to the quality of information communicated and with regard to receiving and presenting information? List as many qualities as possible.

V.4--TAMU TIDEE Design Knowledge Test: 4th Adaptation Based on WMU Faculty Feedback – For

future useObjective: Demonstrate your knowledge of key elements in the engineering design

process, teamwork, and communication associated with team-based engineering design.


A. In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe your understanding of the engineering design process, in diagram of flowchart form, in complete sentences, or in list form.

B. For a Team-Based Engineering Design Project, what are the processes that contribute to good teamwork? What are the qualities that describe effective teamwork. List and describe as many as you can.

C. What are the characteristics of good communication? (Include information about quality of communication with people external to the design team, as well as internal to tam.) Include as many characteristics as possible.

Our Prototype: The TIDEE Design Knowledge Assessment Scoring Rubric


Category: Design ProcessSubcategory: Problem Definition

1 5No design requirements stated, or few but Goal and requirements defined fully,

revisedambiguous at best. over time, address technical and non-

technical issues such as performance, cost, reliability, manufacturability,

and safety.

Subcategory: Evaluation and Decision Making1 5

Only cursory analysis of ideas; decision made arbitrarily Quantitative and qualitative issues analyzed; appropriate

analytical and experimental methods, tools, and information used; decisions based on established

criteria

Subcategory: Process Development1 5

Design steps performed linearly without steps towards Process elements repeated, used iteratively to

improvement. improve results; design process planned, recorded, and revised for improvement.

Category: TeamworkSubcategory:Purpose and Goals

1 5Team goals non existent or poorly defined; members Team driven by clear goals; members

know goals clueless about team goals. and how will be assessed; members committed

to common purpose.

Subcategory: Roles and Responsibilities1 5

Team members have no clear role assigned or role Members understand responsibility; team defines,

ownership; important roles not performed. enforces expectations; members own and fulfill their roles to support team; all important roles covered well.

Subcategory: Resource Management1 5

Team without adequate skills and resources to Team has proper mix, levels of skills needed for

Complete project; no effort to obtain needed resources. Project; team uses talents, knowledge, skills of all members; other resources accessed and used effectively.



Category: CommunicationSubcategory:Content

1 5Information incomplete or has obvious errors; Information stated clearly, completely, and overall Reliability questioned. accurately; measures of

reliability stated and justified.

Subcategory:Relevance to Audience1 5

Information not relevant to audience as presented.Information in form, appearance, language, and level attractive and understandable

to intended audience; understanding communicated well.

Subcategory: Value1 5

Information recorded or communicated has little Information is timely and of great value to team

value to design team or others. and to others; support quality design decisions and value-added actions by others.



V.1--TAMU TIDEE Rubric: 1st Adaptation (Fall 2001) -- Used on Fall 2001 Pre-tests , Spring 2002 Post-tests, and Tests completed by TAMU Freshman and Design Profs.

Part A: Knowledge of the Engineering Design Process

Six specific elements articulated by students.

1. Information gathering (i.e. looking for resources in the library, Web-sites or from the other sources)

2. Problem definition (i.e. understanding problem, goals or requirements) defined

3. Idea generation (i.e. brain storming, creative ideas to improve the design products)

4. Evaluation and decision making (i.e. analyzing

ideas, selecting, utilizing, judging and making design decisions based on established requirements)

5. Implementation (i.e. product delivered on time, product satisfy requirements)

6. Process development (i.e. reviewed for improvement)

5 Points—Student’s response shows breadth and depth of knowledge, and indicate interrelationships among most (5 or 6 of 6) of the listed elements in the engineering design process necessary to produce design products.

4 Points—Student’s response shows more than moderate knowledge, if it includes 4 of the 6 elements (listed here) of the engineering design process.

3 Points—Student’s response shows moderate knowledge, if it include 3 elements of the engineering design process necessary to produce design products.

2 Points—Student’s response shows little knowledge, if it includes 2 elements of the engineering design process.

1 Point---Student’s response indicates an effort made to describe the engineering design process and one element is indicated but the response lacks coherent organization.

0 Point---Student did not make any effort to define the engineering design process and shows no knowledge of any of the elements listed.

Part B: Knowledge of Effective Teamwork

Six specific elements articulated by students.

1. Purpose and Goals (i.e. goals, members know goals)

2. Roles and Responsibilities (i.e. members understand roles and fulfill to support teamwork)

3. Team attitude (i.e. polite acceptance and respect for each other, show pride in team and motivated toward team success, commitment)

4. Planning (i.e. time management)

5. Resource management (team uses talents, knowledge, skills of all members, other resources accessed)

6. Operating procedures (i.e. support effective interaction, discipline, listening, decision making etc.)

5 Points---Student’s response (if include 5-6 of 6) shows detailed knowledge of the listed elements of effective teamwork

4 Points---Student’s response shows above moderate knowledge, if include 4 elements (of listed) of the effective teamwork.

3 Points---Student’s response if include 3 elements of the effective teamwork, shows moderate knowledge of the subject.

2 Points---Student’s response including 2 elements of the effective teamwork shows little knowledge of the subject.

1 Points---Student’s response shows little effort in giving the elements of effective teamwork (one element indicated).

0 Points---Student’s response shows no knowledge about the effective teamwork.


Part C: Knowledge of Effective Communication

Five specific elements articulated by students.

1. Structure (i.e. organization, highly understandable, flow of thoughts)

2. Content (i.e. details, key points, clarity of ideas, complete and accurate information)

3. Relevance to audience (i.e. communicated well and understandable to audience)

4. Team attitude (i.e. co-operation, listening)

5. Involvement (i.e. planning meetings, effective interaction between members)

5 Points--- Student’s response (if include 5of 5) shows detailed knowledge of the listed elements of effective communication.

4Points --- Student’s response shows above moderate knowledge, if include 4 elements (of listed) of the effective communication.

3 Points---Student’s response if include 3 elements of the effective communication, shows moderate knowledge of the subject.

2 Points---Student’s response including 2 elements of the effective communication shows little knowledge of the subject.

1 Points---Student’s response show little effort in giving the elements of effective communication (one element indicated).

0 Points---Student’s response shows no knowledge about the effective communication.


Knowledge of Engineering Design ProcessSix specific elements articulated by students.

• Information gathering

• Problem definition (i.e. understanding problem goals or requirements defined)

• Idea generation (i.e. brain storming, Creative ideas to improve the design products)

• Evaluation and decision making (i.e. analyzing ideas)

• Implementation (i.e. product delivered on time, product satisfy requirements)

• Process development (i.e. reviewed for improvement)

•5 points total: 1 point given for each topic named.

•5 points total: 1 point given for each definition of the main topics

•5 points total: 2 points given for listing the topics in the correct order

2 points given for showing an iteration in the process

1 point given for showing more that one iteration in the process

Grand Total of 15 Points for Section

V.2 TAMU-TIDEE Rubric: 2nd Adaptation (Spring 2002) Piloted in FC Assessment Workshop with Faculty, at

Western Michigan University, Oct 2002

5 points total for each main category:

• 1 point given for each underlying characteristic within the three elements. (Up to 3 points awarded for this part)

• 2 points given for examples or more explanation of why it is important


Knowledge of Effective TeamworkThree specific elements articulated by

students.

• Purpose and planning (goals, members know goals, time management, meetings)

• Rules, roles and responsibilities (members understand and fulfill roles to support teamwork; team uses talents, knowledge, skills of all members, other resources accessed, and communication)

• Team attitude (i.e. polite acceptance and respect for each other, show pride in team and motivated toward team success, commitment, constructive criticism, and communication)



5 points total: 1 point given for each characteristic that relates to the five elements for communication within the team.

5 points total: 1 point given for each characteristic that relates to the five elements for communication outside of the team.

5 points total• 2.5 points for more explanations of

communication within the team• 2.5 points for more explanations of

communication outside the team


Knowledge of Effective CommunicationFive specific elements articulated by

students.

• Structure (i.e. organization, highly understandable, flow of thoughts)

• Content (i.e. details, key points, clarity of ideas, complete and accurate information)

• Relevance to audience (i.e. communicated well and understandable to audience)

• Team attitude (i.e. co-operation, listening)

• Involvement (i.e. planning meetings, interaction)



Knowledge of Engineering Design ProcessSix specific elements articulated by students.

• Information gathering

• Problem definition (i.e. understanding problem goals or requirements defined)

• Idea generation (i.e. brain storming, Creative ideas to improve the design products)

• Evaluation and decision making (i.e. analyzing ideas)

• Implementation (i.e. product delivered on time, product satisfy requirements)

• Process development (i.e. reviewed for improvement)

•5 points total: 1 point given for each of these or similar process actions named.

•5 points total: 1 point given for each addtl “point of information” about a process action. No more than 2 for any one action•5 points total: 2 points given for listing process actions in logical, developmental order similar to that given here. 2 points given for showing an iteration in the process 1 point given for showing more that one iteration in the process Grand Total of 15 Points for Section

V.3 TAMU-TIDEE Rubric: 3rd Adaptation (Fall 2002) Result of WMU Faculty Feedback – For Use on Fall 2002

Electrical Engineering Design Class Pre & Post tests

5 points total for each main category:

• 1 point given for mention of each underlying contibutor to any of the3 main categories given. (Up to 3 points awarded for this part)

• 2 points given for examples or more explanation about how or why a category or any of its contributing elements is important


Knowledge of effective TeamworkThree specific elements articulated by

students.

• Purpose and planning (goals, members know goals, time management, meetings)

• Rules, roles and responsibilities (members understand and fulfill roles to support teamwork; team uses talents, knowledge, skills of all members, other resources accessed, and communication)

• Team attitude (i.e. polite acceptance and respect for each other, show pride in team and motivated toward team success, commitment, constructive criticism, and communication)



5 points total: 1 point given for each characteristic or property that relates to the five elements given with regard to communication within the team.

5 points total: 1 point given for each characteristic that relates to the five elements given with regard to communication with persons outside of the team.

5 points total• 2.5 points for more information or

explanations about communication within the team

• 2.5 points for more information or explanations about communication outside the team


Knowledge of Effective CommunicationFive specific elements articulated by students.

• Structure (i.e. organization, highly understandable, flow of thoughts)

• Content (i.e. details, key points, clarity of ideas, complete and accurate information)

• Relevance to audience (i.e. communicated well and understandable to audience)

• Team attitude (i.e. co-operation, listening)

• Involvement (i.e. planning meetings, interaction)



Application of the TAMU-TIDEE Test for baseline and benchmark measurement in Preparation for AERO CRCD “Smart Materials Project” Evaluation.

Part II

Application

The TAMU-TIDEE Design Knowledge Test was used in assessments which prepared for the evaluation of The CRCD “Smart Materials” Project, and also provided data for use in instrument revisions

Baseline and Benchmark Measurements

• 2001-2002 AERO Senior Design Class

• 2001-2002 Freshman Engineering Class

Comparative Results: 2001-2002 AERO Senior Design Students

Median Mode Median ModeDesign Process 3.50 2.0 4.00 4.0Teamwork 2.00 2.0 2.00 2.0Communication* 2.00 2.0 1.50 1.0

Pre Post

* Validly of question being tested

(N=23)

Senior Design Knowledge Test

Mean Std. Dev. Mean Std. Dev.

Design Process 3.20 1.03 3.65 0.86

Teamwork 2.32 0.77 2.65 1.10

Communication* 2.03 0.84 1.56 0.78

Senior Design Knowledge TestPre Post

(N=23)


Pre Post Prof.

82.6% 39.1% X78.3% 69.6%52.2% 47.8% X52.2% 4.3%43.5% 21.7% X30.4% 13.0% X21.7% 60.9%52.2%30.4%26.1%

47.8%47.8% X30.4%26.1% X

Gather InformationFind a method

Goal

Identify Problem

Response Categories

BrainstormComparison, Evaluate, Analyze, Decide

Conclusions

Implementations

Resulting Detailed Design Choosing One

Finalize

Begin with specifications and requirements

Preliminary Design

Resulting, Detailed Design

Iterations

Percentages calculated on all respondents, N=23 (1student = 4.3%)

'X' represents Prof. response

Highlighted categories match Test scoring rubric

Content Analysis & Comparison with Rubric: Senior AERO Design Students -- Knowledge of Design Process

Pre Post Prof

60.9% 60.9% X52.2% 17.4%26.1% 43.5% X26.1% 34.8% X21.7% 4.3%17.4% 8.7% X17.4% 21.7%13.0% 4.3%13.0% 13.0%13.0% 8.7%13.0% 8.7% 8.7%

17.4%4.3%4.3%

17.4%13.0% X8.7% X

Pre Post Valasik

Set Layout Time for Project

Choose a Member WellTake Responsibility

Response Categories

Communication Between Team Members

Make good eye contact, hands out of pocketKnow your audience

Member understand the Problem

Finish Project On-timeSet Guidelines for group

Split Work

Form (Outline) goalsAbout Attitude

Help each other

Topic

Know Each Other's Strength and WeaknessKnow What to do (recognize their role)

Percentages calculated on all respondents, N=23 (1student =

4.3%)


* 'X' represents Prof's response

Members Know Status of WorkPlanGet Everyone's Contact Information

Content Analysis & Comparison with Rubric: Senior AERO Design Students -- Knowledge of Teamwork

Pre Post Prof

52.2% 34.8% X30.4% 13.0% X26.1% 26.1% X17.4% 17.4% 60.9% X13.0%13.0%13.0%13.0%13.0%8.7% X8.7%

Everyone has the same Technical Data 4.3%4.3%4.3%0.0%

34.8% X13.0%13.0%8.7%0.0%0.0%

Update Changed Info (Documentation)

Be OrganizedConvey a Thought to Outside Sources (write report)

Team Attitude

Spell Check

Contact Often

Meeting

Communicate clear, concise, understood by all

On Time

Get to Know Each OtherConfidence about Work DoneKnow Audience

Open Minded to New IdeasHonestyMotivate Team Members

Be Confident

Check for UnderstandingListenCooperate with Each OtherDiscuss

Percentages calculated on all respondents, N=23 (1student =

4.3%)

X' represents Prof's response

Response Categories


Constructive Critism

Content Analysis & Comparison with Rubric: Senior AERO Design Students -- Knowledge of Teamwork

Median Mode Median ModeDesign Process 3.00 2.0 4.00 4.5Teamwork 3.00 3.0 2.75 3.0Communication* 1.75 1.0 1.50 1.5

Pre Post*


(N=78)

Freshman Knowledge Test

Mean Std. Dev. Mean Std. Dev.

Design Process 2.76 1.14 3.77 1.01

Teamwork 2.79 0.96 2.60 0.78

Communication* 1.77 0.95 1.46 0.89

Freshman Knowledge TestPre Post*

(N=78)


Freshman Benchmark Comparisons: Beginning vs. End of 2001-2002 Yr.

*Different students in Pre and Post test groups

Post Prof.

80.8% X70.5% X88.5% X34.6%12.8% X5.1%

23.1% XX

30.8%

34.6% X64.1% X

X28.2% X

FinalizePreliminary DesignResulting Detailed Design Choosing OneBegin with specifications and requirements

Percentages calculated on all respondents, N=78(1student = 1.28%)

'X' represents Prof. response


Implementations

BrainstormComparison, Evaluate, Analyze, DecideIdentify ProblemConclusions

Resulting, Detailed DesignGoal

Response Categories

Gather InformationFind a methodIterations

Content Analysis & Comparison with Rubric: Profs. & Freshman (Post-test )

Knowledge of Design Process

Post Prof

50.0% X20.5%34.6% X24.4% X33.3% X

X20.5%14.1% X10.3%1.3%

17.9% X2.6% X

3.8% X28.2% X2.6%

* 'X' represents Prof's response

Set Layout Time for Project

Know Each Other's Strength and WeaknessKnow What to do (recognize their role)Form (Outline) goals

Percentages calculated on all respondents, N=78(1student =

1.28%)


Split Work

Communication Between Team Members

Response Categories

About AttitudeMember understand the ProblemHelp each otherFinish Project On-timeSet Guidelines for group

PlanMembers Know Status of Work

Get Everyone's Contact Information

Choose a Member WellTake ResponsibilityMake good eye contact, hands out of pocketKnow your audience


Knowledge of Teamwork

Post Prof

17.9% X10.3% X25.6% X24.4% X9.0% X

3.8%25.6% X37.2% X14.1%19.2% X17.9% X

Percentages calculated on all respondents, N=78(1student =

1.28%)

X' represents Prof's response


Know AudienceOn Time

Motivate Team Members

Be OrganizedBe Confident

Check for UnderstandingListenCooperate with Each OtherDiscussMeetingGet to Know Each OtherConfidence about Work Done

Everyone has the same Technical Data

Open Minded to New IdeasHonesty

Response Categories

Contact OftenConstructive Critism

Team Attitude

Update Changed Info (Documentation)

Communicate clear, concise, understood by allSpell CheckConvey a Thought to Outside Sources (write report)


Knowledge of Teamwork

Concluding ObservationsAbout Student Performance

Both student groups (freshmen and seniors)

• performed well in the pre-test, and improved the most, from pre to post testing, in question 1 (knowledge about the design process).

• performed as well or better, in the pre-test, on question 2 (knowledge about teamwork) than on question 1, their performance declined considerably from pre to post testing.

• Performed worst in the pre-test and declined the most, from pre to post testing, in question 3- knowledge about communication in team design.

Concluding Observations

•Overall, students tended to answer questions as their professors did more than by the Scoring Rubric criteria

•The better the students performed from pre to post testing on a question, the more likely they were to respond to questions as their professors did

•Many of the professors’ answers were not in line with the Scoring Rubric s criteria

•Professors and students both diverged most from the Scoring Rubric s criteria on question 3 (knowledge about communication) and then on question 2 (knowledge about teamwork)

Concluding Observations

•Regardless of revisions aimed at improving alignment of student answers with Scoring Rubric s criteria, performance on question 3 ( communication in team design) did not improve

•Given the discrepancies, on question 3, between the Scoring Rubric criteria and the answers of professors, we believe that conceptions about what is important and what should be taught about communication in team design, may be different at Texas A&M Univeristy than among the engineering professors who developed the TIDEE design assessment instrument used as a prototype for the TAMU-TIDEE adaptations.

REFERENCES & END NOTES1 CRCD: Development of an Integrated Multidisciplinary Curriculum for Intelligent Systems. NSF Proposal Number 0088118; P I: D Lagoudas.

2 Davis, D. C. (2001). Transferable Integrated Design Engineering Education (TIDEE), Mid Program Assessment: A Three Part Assessment of Team Based Design for Entering Juniors. http://www/cea/wsu.edu/TIDEE

3 The adaptations and data collections in this presentation involved only Part I of the TIDEE Three Part Assessment of Team Based Design -- Design Knowledge

Documents

The AERO CRCD “Smart Materials” Project 1 Preparation for Evaluation Adaptation, Evolution and Application of The TAMU-TIDEE Design Knowledge Test Rita