Leading to ConclusionsAnalyzing and Interpreting Data

Ben Silliman, Youth Development SpecialistNorth Carolina 4-H

Qualitative Analysis

Leaders also need a way to process those “open ended” observations & comments for decision-making

• To understand the experiences, perceptions, motivations, intentions, and behaviors that – may not yet be coded in an accurate and efficient survey– may offer more in-depth, in-breadth data than a survey

• To engage an audience in dialogue and partnership– valuing them as people and partners– better understanding their needs for education, empowerment

Leading Figures

Tania Leon

Classical music performer,

composer, and conductor

From the organization of her own creative genius to the instantaneous processing of sounds and movements in an orchestra, this multi-talented artist goes beyond the quantitative data (tempo, pitch, volume) to identify and interpret patterns and themes in music and dance. Effective work with a UNC students or other collaborators often depends more on sifting insights than reviewing numbers-thus the value of qualitative analysis.

Purpose of Qualitative Analysis

• Analyze and Interpret Narrative Data from

– Interviews: focus groups, individual interviews– Open-ended questions: self-generated descriptions

or graphics– Observations: on-site observations of groups or

individuals, content analysis of products

Note: qualitative data most insightful when triangulated with other observations and quantitative responses

Purpose of Qualitative Analysis

• Gain insights on a specific theme or audience(vs. generalizing to broader groups)

– Leadership needs of this Council– Strengths of these regional Councils relative to stages

of leadership maturity– Views of youth leaders attending this year’s national

conference about an organizational conflict (or opportunity)

Organizing Qualitative Data

• Pre-Organization (based on theory, experience)

– Established categories used to group comments• Fit ideas to a framework• Focus on predictable categories

• Emergent Organization (grounded theory)

– Frequently mentioned themes (content, process, relationship) named, comments clustered

• Allow critical points to arise from respondents

Organizing Qualitative Data Examples

• Pre-Organization (based on theory, experience)– Leadership Challenge roles: vision, modeling, creative

processing, engaging & empowering, caring– Project Management strengths: recruiting, planning, mobilizing,

implementing, evaluating

• Emergent Organization (grounded theory)

– County Council Leadership Questionnaire:• “Nobody has experience with this”…Content expertise• “People won’t talk with each other”…Relationship issues• “Events are chaotic, thrown together”…Process, organization

Emergent consensus: More training & guidance needed

Qualitative Analysis Steps*

Handling Narrative Data

• Absorbing: Read and re-read to thoroughly understand input before categorizing

• Focusing: Clarifying the purpose for evaluation

• Categorizing: Coding and organizing comments

• Identifying Patterns & Connections: Grouping and inter-relating comments

*adapted from Taylor-Powell, E., & Renner, M. (2003). Analyzing qualitative data. Madison, WI: UW Extension Service bulletin 3658-12.

Qualitative Analysis Step 1: Absorbing What was Said

Practical Step #1: Retreat and Reflect

• Debriefing: Be sure to transcribe or translate comments “as is” in order to process accurately

• Valuing: Examine the quality (focus, breadth, depth) of data to determine worth for examining

• Text-Check: Consider diverse interpretations, limitations, context in any set of comments

• Self-Check: Consider personal biases that affect how you see (and what you don’t see) in the text

Qualitative Analysis Step 1: Absorbing What was Said

Practical Step #2: Don’t lose it

• Marking: Be sure the source and context are clearly indicated and transferred with each item

• Sorting: Physically or electronically group the –marked- comments together

• Duplicating: Copy originals and keep them in a separate location

• Storing securely: Maintain confidentiality by holding material under lock and key

Qualitative Analysis Step 2: Focusing What You Notice

• Remember the Purpose of the Evaluation > What was the research or evaluation question, anyway?

• Identify key questions or emphases– By topic: vision, management skills, caring for others– By time period: public-private settings, early-mid-late

project, routine and crisis events– By case, individual, or group: event, personal,

interpersonal dynamics and systemic issues in or across events

Qualitative Analysis Categorize What You Find

Sample: Open-ended Questions on how Community Service projects teach leadership

As a youth professional, you believe that “hands-on” experiences in community service fosters a variety of leadership skills. As a follow-up with youth leaders who planned and implemented a community service project, you ask an open-ended question:

What did your experience in this community service project teach you about leadership? (Use ”I” statements to describe your learning)

Qualitative Analysis Step 3: Categorize What You Find

• Label and Group individual items > Mark text using pre-set or emergent categories

Feedback from youth after a Community Service Project:What did your experience in this community service project teach

you about leadership? (note broad categories, need to interpret)– Values: responsibility/flexibility, integrity/compromise, acceptance/

accountability, diversity/unity– Concepts: vision/action, task-oriented/people-oriented,

compassion/accountability, product/process– Behaviors: talking/listening, working/playing, thinking/doing,

working alone/working together– Relationships: many/few, close/distant, homogeneous/diverse,

Leading Figures

Jane McKimmon

North Carolina

Extension Educator & Administrator

As an innovative leader, Dr. McKimmon often asked “open-ended” questions, then reflected and prepared a synthesis to create a program. Later, this experience enabled her to present structured evaluations, more efficiently processing client feedback. Although these interactions were often intuitive, data was gathered and interpreted systematically. These are the key elements of qualitative evaluation.

Qualitative Analysis Step 3: Categorize What You Find

• Label and Group individual items > Use existing question or program structure to group comments

Feedback from youth after a Community Service Project:What did your experience in this community service project teach

you about leadership? (note details used to focus descriptions)– Planning the project– Recruitment of helpers– Gaining support– Implementation (doing the project)– Problem solving during the project– Evaluating results

Qualitative Analysis Step 3: Categorize What You Find

• Re-read and distinguish sub-categories > Refine analysis based on research and richness of the data

Feedback from youth after a Community Service Project:(sample comments on communicating as a leader)

– One-way: Recognize the importance of step-by-step instructions

I tried to explain, but forgot and left things out…– Talk/Feedback: Understand the importance of feedback

I learned to check with each person to be sure he understood his job…and was excited about it…

– Two-way: Regular exchange of ideas

Even though I knew the plan, I found that other people gave me good ideas as the project went along …

Qualitative Analysis Step 3: Categorize What You Find

• Relate data to appropriate (multiple) categories> Apply comments to all relevant categories (without over-using)

Feedback from youth after a Community Service Project:(sample comments reflecting innovation)

I tried to explain, but forgot and left things out…

– Communication: ability to give clear directions– Planning: implies development of a project plan

Qualitative Analysis Step 4: Identify Patterns & Connections

• Group within category> Distinguish by similarities and differences

Feedback from youth after a Community Service Project:(sample categories for communication)

Level X: We just argued all the time and the project did not get done…

Level 0: Nothing…I wasn’t the leader…

Level 1: I tried to explain, but forgot and left things out…

Level 2: I learned to check with each person to be sure

he understood his job…and was excited about it…

Level 3: Even though I knew the plan, I found that other people gave me good ideas as the project went along …

Qualitative Analysis Step 4: Identify Patterns & Connections

• Group across categories> Distinguish patterns for key variables

Feedback from youth after a Community Service Project:> Cross-compare for individual experience or observer rating

Level X: Not unusual for new leaders (20% of comments)

We just argued all the time and the project did not get done...

Level 0: Pretty typical for first-time leaders (33%)

Nothing…or…I wasn’t the leader…

Level 1: Typical for new leaders; not for experienced leaders (38%)

I tried to explain, but forgot and left things out…

Qualitative Analysis Step 4: Identify Patterns & Connections

• Group across categories (continued)> Distinguish patterns and frequency for key variables

Feedback from youth after a Community Service Project:> Cross-compare for individual experience or observer rating

Level 2: Rare for new leaders; frequent for experienced leaders (15%)

I learned to check with each person to be sure he understood

his job…and was excited about it…

Level 3: Exceptional, even for experienced leaders (4%)

Even though I knew the plan, I found that other people

gave me good ideas as the project went along …

Qualitative Analysis Step 4: Identify Patterns & Connections

• Try to explain patterns in the data> Explain correlations or cause-effect relationships

Feedback from youth after a Community Service Project:> Growth in leadership (communication) through community service

-Roles in leading community service experiences foster interpersonal perspective-taking and skills of leaders (cause-effect)

-As youth mature, their experience in community service as well as their communication skills will probably both increase (correlation)

-Those who seek leadership skills via service experiences are likely empowered to be more effective leaders later (reciprocal relationship)

Leading Figures

Billy Graham

Preacher, Organization Leader

Raised in Charlotte, NC, Dr. Graham became a practical leader to co-workers, spiritual leader to millions, but also served as leader to an organization holding large meetings through volunteer labor. Careful listening to comments and observation of events—qualitative evaluation--enabled staff to sensitively and creatively respond to opportunities and needs of participants in diverse locations and venues.

           

            

Qualitative Analysis Step 4: Identify Patterns & Connections

• Generate & answer questions from patterns> What context or process factors were typical of leaders who gained most (communication or other skills) from community service experience? (or was it just a matter of experience/maturity?)

Feedback from youth after a Community Service Project:> Context/Process factors that foster leadership communication

-Re-read Top 19% (Level 2 & 3) descriptions/quotes of youth

-Review registration/application materials of Top 19% youth

-Research or review descriptions of community service projects, including mentoring style of older youth or adults working with youth leaders

Qualitative Analysis Step 4: Identify Patterns & Connections

• Explore ‘no gains’ and unanswered questions> Why did most youth experience no gains in communication skills?

> How much ‘learning’ involved direct experience vs. observation of others? (and how accurate are reports of their own experiences)

Reflecting on feedback from leaders:> Accounting for ‘no gains’

-’Hands-on’ experience is not enough (more mentoring needed)

-The ‘open-ended’ question was too open (more focus needed)

> Unanswered questions

-May be implied in some responses

-Would have required

Qualitative Analysis Step 5:Reaching from Insight to Action

• Summarize major themes & connections

Feedback from youth after a Community Service Project:

> Main threads and lessons (related to communication)-About 1 in 5 youth show evidence of growth in communication skills

-Prior experience and adult mentoring are more typical for youth who demonstrated higher levels of communication skills

-Quotation that captures how project leadership improves listening:

Even though I knew the plan, I found that other people

gave me good ideas as the project went along …

Qualitative Analysis Step 5:Reaching from Insight to Action

• Summarize major themes & connections

Feedback from youth after a Community Service Project:

> New learning-For this group (at least) there is a greater-than-expected need for coaching (experiential training and individual mentoring) on communicating effectively as a leader (perspectives and skills)

-Picture Graph: Percent of leadership participants who gained each level of communication skills

Qualitative Analysis Step 5:Reaching from Insight to Action

• Summarize major themes & connections

Feedback from youth after a Community Service Project:

> Application to other settings and audiences-Coaching on how to communicate as a leader should begin early and be repeated often, esp. in non-formal (vs. classroom) situations

-Role modeling is critical for adults and older youth as youth leaders as both the attitude and skills of good communicators

-Picture: Youth leader directing end-of-project debriefing

Qualitative Analysis Step 5:Reaching from Insight to Action

• Summarize major themes & connections

Feedback from youth after a Community Service Project:

> Interpretation to diverse stakeholders-Parents/Volunteer Leaders:

Project leadership requires a major time commitment and willingness to work outside one’s comfort zone…As “hands-on” project leaders, many are learning to be clear and caring communicators. Using evaluation feedback from our most effective youth leaders, all youth leaders will be trained and mentored in key skills such as giving directions, active listening, and involving all team members in planning and problem solving during projects and other club activities.

Qualitative Analysis Step 5:Reaching from Insight to Action

• Summarize major themes & connections

Feedback from youth after a Community Service Project:

> Interpretation to diverse stakeholders-Grant Funders/Commissioners:

Community service teaches important values such as compassion for others and giving back to one’s community. Projects also develop leadership and teamwork, critical skills in today’s workplace and civic arenas. Our ongoing program evaluation indicates that many youth leaders are gaining skills to communicate effectively but most need more training and mentoring…Thus we are requesting funding for a program assistant to train volunteer leaders as leadership coaches. This training will support a five-fold increase in community service among youth, providing in an estimated $50,000 benefit to the county.

Qualitative Analysis Step 5:Reaching from Insight to Action

• Summarize major themes & connections

Feedback from youth after a Community Service Project:

> Interpretation to diverse stakeholders-Business Partners:

Today’s community service project leaders are tomorrow’s business and civic leaders…Many youth are learning practical skills such as giving directions, listening effectively, and engaging team members in problem solving. Please consider volunteering as a Mentoring New Generation Leaders volunteer and help us train 100 more youth leaders in the county to become effective project planners, organizers, and communicators while engaging in needed community service.

Qualitative Analysis Step 5:Reaching from Insight to Action

Capture ideas & action in a published report• Format

– Program Fact Sheet, Brief– Brochure– DVD/Video

• Purpose—connect with stakeholders to– Inform: describe benefits of the program and efforts to

improve it– Invite: seek investments of time, money, and other

resources to support and expand the program