Upload
charlie-gowin
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Unités de recherche éducationnelle/Educational Research Units
L’adulte et l’apprentissage en milieu de travail/Adult and Workplace Learning
Mesure et évaluation des apprentissages/Measurement, Evaluation and Assessment
Séminaire d’après-midi/Afternoon Seminar30 janvier 2013/January 30th , 2013
De 16h à 17h15/4:00 to 5:15 p.m. LMX 388
La présentation sera en anglais/Presentation will be in English
Des rafraichissements seront servis/Refreshments will be served
RSVP avant le 28 janvier à :
RSVP before January 28th at:
Tips on how to develop measurement tools and collect data
Ivana Pavic, Maurice Taylor, Sait Atas and David Trumpower
Tips on how to develop measurement tools
• Involve your learners and instructors in the development (Ivana)
• Ensure the tool has a strong conceptual development (Maurice)
• Pilot test the tool to obtain evidence of reliability, validity, utility (Sait)
• Revise, as necessary, following psychometric analysis (David)
• Talking about the social networks in our lives is very much related to a term called
SOCIAL CAPITAL
• Social capital relates to the connections we have with people
• …..connections to family
• …..connections to friends
• …..connections to the workplace
• …..connections to the neighbourhood
• …..connections to the local community
Why are these connections important?
1) they help us advance our interests in the society we live in
2) they help us co-operate within groups
3) they promote trust, goodwill and a desire to act in a supportive manner
Some examples
• The Ottawa East Community Association
• Immaculata Parent Council• The Art of Running Club• The Ottawa Newcomers Club • Sandy Hill Community Health
Centre Volunteer• One Parent Families Association • Jamaican Ottawa Community Association
What are the benefits of social capital?
• Glue- holds society together by facilitating co-operation within groups of people
• Oil- keeps the economic and social wheels of society rolling smoothly
Social capital is like:
• a resource that people can accumulate and use to their benefit and advantage
• having a bank account of all of the networks in our lives
• The lack of social capital also can hold back people preventing them from achieving their goals which can affect the quality of life
Social capital is an important part of the learning process for
1) individuals
2) the training outcomes of a program
3) the community that we live in
Conceptual development of the tool
• How robust is the concept of social capital?
• OECD: 2000-2012 – framework for social outcomes of learning
• Australian Bureau of Statistics (2004) – indicators for a social capital framework
• A concerted effort to empirically understand adult literacy and social capital outcomes in Scotland, the UK, Australia and New Zealand
Conceptual development
• Interest on Canadian soil
• Canadian Institute for Policy Research Initiatives; National Centre for Literacy
• Is the timing right for developing a measure
• Do you have access to a target population?
Conceptual development
• 1st Subscale: Network Qualities (NQ): including sub-concepts of trust levels, efficacy, diversity and inclusiveness
• 2nd Subscale: Network Structure (NS): including sub-concepts of size, communication mode and power relationships
• 3rd Subscale: Network Transactions (NTS): including sub-concepts of sharing support and sharing knowledge
• 4th Subscale: Network Types (NTY): including sub-concepts of bonding, bridging, and linking
• What are the size of your networks?• How do you
communicate in your networks?
• How do you share support?• How do you share
knowledge?
• What are your levels of trust?• What is your level
of self-confidence?
• How do you bond, bridge, and link with your networks?
4. Network Types
1. Network Qualities
2. Network Structures
3. Network Transactions
Sample Items from SCISub-
scalesItem Strongly
Disagree
DisagreeNeither agree nor
disagree
Agree Strongly Agree M SD
1.NQ I have the self-confidence to influence things in my work and community life. 23.64 5.13
2. NS When I am in a problem situation I have a network consisting of family, friends, and neighbours.
22.85 4.84
3. NTr I believe I am a valued member of my group because of the support that I give in class.
23.10 5.19
4. NTy I believe the program has provided opportunities for me to interact with new outside groups and services.
22.97 5.03
Reliability
…refers to the stability or consistency of scores (over time, across raters, across items).
When a test is reliable, it provides dependable and consistent results.
Methods for Estimating Reliability
• Involves administering the same test to the same group of participants on two different occasions and then correlating the two sets of scores.
Test-Retest Reliability
• Involves two raters independently scoring the same set of tests and then correlating the two rater’s sets of scores.
Inter-rater Reliability
• Involves administering the test once to a single group of participants and determining if items on the test are correlated with one another (as indicated by Cronbach's alpha).
Internal Consistency
Reliability
Let’s have a look at the Social Capital Inventory’s reliability
To test SCI ’ s reliability internal consistency reliability was determined.
*The SCI was found to have adequate reliability.
Subscales Number of items Cronbach's alpha (α)
1. Network Qualities 6 .86
2. Network Structures 6 .81
3. Network Transactions 6 .88
4. Network Types 6 .88
Total Scale 24 .96
Is reliability good enough for a test to be used
A reliability coefficient only indicates whether the attribute measured by the test— whatever it is—is being assessed in a consistent way.
Whether the test is actually assessing what it was designed to measure is addressed by an analysis of the test's validity.
Validity
The term validity refers to whether or not the test measures what it claims to measure.
On a test with high validity the items will be closely linked to the test’s intended focus.
SCI Validity
Three major types:
Content validity
Construct validity
Criterion-related validity
Involves systematic examination of the test content to determine whether it covers a representative sample of the domain to be measured
Typically conducted by panel of subject matter experts
SCI Validity
Three major types:
Criterion-related validity
Content validity
Construct validity
Involves demonstrating that a measurement tool varies systematically with another measure of the same construct
But, requires the existence of another valid measure…
SCI Validity
Three major types:
Construct validity
Criterion-related validity
Content validity
Involves empirical testing of hypotheses about the construct, using the test as a measure of the construct
e.g., if we know that one group should have higher levels of construct X than another group, then comparison of the mean level of construct X (as measured by the test) for the two groups should reveal a significant difference
Ongoing research
- N=115 adult learners seeking literacy and essential skills in formal, non-formal, and informal learning settings- 1 Adult high school (Ontario)- 6 Workplace programs (4 in Manitoba, 2 in
Nova Scotia
- Mixed methods study (SCI, SDLRS, interviews)
Ongoing research - results
- Relatively high SCI means are consistent with qualitative data obtained from instructors and learners
- Those with average to above average scores on the SCI have significantly higher scores on the SDLRS
- Disproportionately higher percentage of females than males had average to above average scores on the SCI, if they do not belong to any clubs
Take-home Tips
2. Seek feedback from Subject Matter Experts
-Teachers
-Colleagues
-Committee members?!
Take-home Tips
3. Pilot test
-Small, representative sample
-Use multiple raters (if applicable)
-Evaluate descriptive statistics (frequency distributions, means, SDs)
-Evaluate reliability (Cronbach alpha, item- total correlations, inter-rater agreement)
Take-home Tips
4. Revise, if necessary
-Delete unreliable items
-Clarify wording
-Modify scale
-Consider difficulty/discriminability
Take-home Tips
5. Include validity checks
-Collect demographic data
-Include other measures related to construct
-Compare with prior studies/theory