Agenda
2 Main QA approaches Coding exercise 1 Coding exercise 2 Slides on Qualitative Analysis Brainstorming Exercise (if time) Affinity Diagramming Exercise (if
time)
2
Qualitative Research: Common Features of Analytic Methods (Miles & Huberman,1994)1 Affixing codes to a set of field notes
drawn from data collection2 Noting reflections or other remarks
in margin3 Sorting or shifting through the
materials to identify similar phrases, relationships between themes, distinct differences between subgroups and common sequences
Qualitative Research: Common Features of Analytic Methods (Miles & Huberman,1994)4 Isolating patterns and processes,
commonalties and differences, and taking them out to the field in the next wave of data collection
5 Gradually elaborating a small set of generalisations that cover the consistencies discerned in the data base
6 Confronting those generalisations with a formalised body of knowledge in the form of constructs or theories
2 general research approaches
5
deductive approach vs inductive approach
deductive research approach
6
THEORY
HYPOTHESIS
OBSERVATION
CONFIRMATION
Top-down approach
Theory testing
A priori codes
inductive research approach
7
THEORY
TENTATIVE HYPOTHESIS
PATTERN
OBSERVATION
bottom-up approach
Theory building
Emergent codes
8
deductive or inductive
Often use a hybrid approach A set of a priori codes reflecting your
understanding of the topic and your research questions
Emergent codes added as you code the data and find other factors/topics/codes that you had not considered
9
Exercise 1
Open coding Inductive analysis Exploratory research Theory building research
http://b.socrative.com/login/student/ Room: 7f156b7b
10
Exercise 2
Coding with pre-defined categories Deductive analysis Theory Testing
http://b.socrative.com/login/student/ Room: 7f156b7b
11
Qualitative Inquiry - Purpose
The purpose of qualitative inquiry is to produce findings. The Data Collection process is not an end in itself. The culminating activities of qualitative inquiry are analysis, interpretation, and presentation of findings.
Qualitative Inquiry - Challenge
To make sense of massive amounts of data, reduce the volume of information, identify significant patterns and construct a framework for communicating the essence of what the data reveal
Qualitative Inquiry - Problem
‘…have few agreed-on canons for qualitative data analysis, in the sense of shared ground rules for drawing conclusions and verifying sturdiness’ (Miles and Huberman, 1984)
The Creativity of Qualitative Inquiry
‘..the human element of qualitative inquiry is both its strength and weakness - its strength is fully using human insight and experience, its weakness is being so heavily dependent on the researcher’s skill, training, intellect, discipline, and
creativity. The researcher is the instrument of qualitative inquiry, so the quality of the research depends heavily on the qualities of that human being’
(Patton, 1988)
The Science and Art of Qualitative Inquiry (Patton, 1988) The Science
The scientific part is systematic, analytical, rigorous, disciplined, and
critical in perspective The Art
The artistic part is exploring, playful, metaphorical, insightful, and creative
1. Analysis Considerations
1 Words2 Context (tone and inflection)3 Internal consistency (opinion shifts during
groups)4 Frequency and intensity of comments
(counting, content analysis)5 Specificity6 Trends/themes7 Iteration (data collection and analysis is an
iterative process moving back and forth)
2. The Procedures
1 Coding/indexing2 Categorisation3 Abstraction4 Comparison5 Dimensionalisation (relationships)6 Integration7 Iteration8 Refutation (subjecting inferences to scrutiny)9 Interpretation (grasp of meaning - difficult to
describe procedurally)
Critical Thinking ‘Critical Thinking calls for a persistent effort to
examine any belief or supposed form of knowledge in the light of the evidence that supports it and the further conclusions to
which it tends’ (Glaser, 1941) or more simply!
Critical Thinking means weighting up the arguments and evidence for and against.
Critical Thinking• Key points (Glaser, 1941):
– Persistence: Considering an issue carefully and more than once
– Evidence: Evaluating the evidence put forward in support of the belief or viewpoint
– Implications: Considering where the belief or viewpoint leads; what conclusions would follow; are these suitable and rational; and if not, should the belief or viewpoint be reconsidered
Guidance for Creative Thinking
1 Be open2 Generate options3 Divergence before convergence4 Use multiple stimuli - triangulate5 Side track, zig-zag, and circumnavigate6 Change patterns of thinking7 Make linkages8 Trust yourself9 Work and play at it
The Credibility of Qualitative Analysis
1 Rigorous techniques and methods for gathering high-quality data that is carefully analysed, with attention to issues of validity, reliability, and triangulation
2 The credibility of the researcher, which is dependent on training, experience, track record, status, and presentation of self
3 Philosophical belief in the phenomenological paradigm, that is, a fundamental appreciation of naturalistic inquiry, qualitative methods, inductive analysis and holistic thinking
A Credible Qualitative StudyA credible qualitative study needs to address the following issues:
1 What techniques and methods were used to ensure the integrity, validity, and accuracy of the findings
2 What does the researcher bring to study in terms of qualifications, experience, and perspective
3 What paradigm orientation and assumptions ground the study
Principles of Analysing Qualitative Data1 Proceed systematically and rigorously
(minimise human error)2 Record process, memos, journals, etc.3 Focus on responding to research questions4 Appropriate level of interpretation appropriate
for situation5 Time (process of inquiry and analysis are
often simultaneous)6 Seek to explain or enlighten7 Evolutionary/emerging
Inter-rater reliability
What if you have more than one person coding? How much agreement do they have? At what point should you test their
agreement? Other than comparing counts, how can
you validate the coding/analysis? https://www.academia.edu/458025/The_place_of_i
nter-rater_reliability_in_qualitative_research_an_empirical_study
26
27
Articulate:•who users are•their key tasks
User and task descriptions
Goals:
Methods:
Products:
Brainstorm designs
Task centered system design
Participatory design
User-centered design
Evaluate
Psychology of everyday things
User involvement
Representation & metaphors
low fidelity prototyping methods
Throw-away paper prototypes
Participatory interaction
Task / Cognitive scenario walk-through
Refined designs
Graphical screen design
Interface guidelines
Style guides
high fidelity prototyping methods
Testable prototypes
Usability testing
Heuristic evaluation
Completed designs
Alpha/beta systems or complete specification
Field testing
Interface Design and Usability Engineering
brainstorming
the point is: to generate MANY, WIDE-RANGING ideasnutty and absurd are GOOD. go for the
extremes (to get out of the rut)
riff off other’s ideas.
the point is NOT: to generate excellent, complete, feasible
ideas … pressure stifles
to develop or critique ideas … go wide. deep is for later.
process
1. prepare a list of topics / questionsahead of time; or in a preliminary brainstorm
2. facilitator takes team through list of topics switch topic when energy ramps down
3. Note taker takes notes (very important)
4. switch roles so everyone can play
5. ground rules
6. Follow up
brainstorming is like popcorn
ground rules Postpone and withhold your judgment of
ideas: never criticize
Encourage wild and exaggerated ideas
Quantity counts at this stage, not quality
Switch topics when the popcorn slows down
Build on the ideas put forward by others
Every person and every idea has equal worth
Elect a facilitator (calls switches) and a note-taker (one thought per post it!)
Post brain-storm collect the notes
go through carefully, with judgment turned on
look for interesting, surprising ideas that might work ideas that will combine well promising directions on which you should
brainstorm more
keep your notes. at a later design stage, come back to them and see if anything else has become useful in the meantime.
Sometimes you have a lot of ideas to make sense of!
33
work consolidation:abstracting specific insights
one tool: the affinity diagram can use to “consolidate” insights from collected
or generated data. for example: brainstorming about design problems
categories of problems
brainstorming about design ideas categories of ideas
comments from users categories of desirable / successful features
how do you make an affinity diagram?
1. team writes down all data & insights on post-it notes; be sure you can link the post-it detail back to its source!
2. stick one post-it on the wall a whiteboard or big sheet of butcher paper is best
3. arrange the other post-its around it, grouping by affinity to each other. iteration will be required.
4. look at each group and see what it has in common; name and describe each group.
5. “snapshot” the result for documentation1. digital photo your design website or notebook2. transfer post-its onto paper, 1 sheet / notes-cluster
scan website
why does an affinity diagram work?
• use physical arrangement/proximity to understand connections
• openness to serendipity
• low cost to rearrange ideas
• many variants:
arrange along axes rather than by affinity
tie causes to effects
group evidence under assertions
Pooya Jaferian, David Botta, Fahimeh Raja, Kirstie Hawkey, and Konstantin Beznosov. 2008. Guidelines for designing IT security management tools. In Proceedings of the 2nd ACM Symposium on Computer Human Interaction for Management of Information Technology (CHiMiT '08). ACM, New York, NY, USA, , Article 7 , 10 pages. DOI=10.1145/1477973.1477983 http://doi.acm.org/10.1145/1477973.1477983
38
Methodology (Phase I)
• Field studies• Interviews• Questionnaires• Prototyping• Cognitive walkthroughs• Surveying other literature
• Field studies• Interviews• Questionnaires• Prototyping• Cognitive walkthroughs• Surveying other literature
• Field study:• Interviews • Participatory observation
• Field study:• Interviews • Participatory observation
39
Methodology (Phase I)
40
Categorized List of
Guidelines
Categorized List of
Guidelines
41
High level Category
Low level Category
Guideline
Guideline ID number
Methodology (Phase I)
42
Guidelines Framework
Methodology (Phase II)
43
Framework for classification of guidelines
44
Task SpecificTask Specific
Organizational ComplexityOrganizational Complexity
Technological ComplexityTechnological Complexity
Configuration and Deployment
Configuration and Deployment
Diverse Stakeholders
Diverse Stakeholders
General Usability GuidelinesGeneral Usability Guidelines
Specificity
Intensive AnalysisIntensive Analysis
Distributed ITSMDistributed ITSM
CommunicationCommunication
45
Framework for classification of guidelinesTask Specific GuidelinesTask Specific Guidelines
General Usability GuidelinesGeneral Usability Guidelines
Technological Complexity GuidelinesMake tools combinable [8,9,20,26] Use multiple levels of information abstraction [1,4,5,10,12,25,41,42,45]
Help task prioritization [15,44] Use different presentation / interaction methods [1,4,5,29,41,48,49]
Provide customizability [9,33] Support knowledge sharing [9,12,14,27,32,37,47]
Technological Complexity GuidelinesMake tools combinable [8,9,20,26] Use multiple levels of information abstraction [1,4,5,10,12,25,41,42,45]
Help task prioritization [15,44] Use different presentation / interaction methods [1,4,5,29,41,48,49]
Provide customizability [9,33] Support knowledge sharing [9,12,14,27,32,37,47]
Organizational Complexity GuidelinesOrganizational Complexity Guidelines
Diverse Stakeholders Guidelines
Provide flexible reporting [9,18,33,35]Provide an appropriate UI for stakeholders [9,35]
Diverse Stakeholders Guidelines
Provide flexible reporting [9,18,33,35]Provide an appropriate UI for stakeholders [9,35]
Distributed ITSM Guidelines
Support collaboration [6,7,20]Work in a large workflow [8,9,20]
Distributed ITSM Guidelines
Support collaboration [6,7,20]Work in a large workflow [8,9,20]
Communication Guidelines
Provide communication integration [6,7,28,45]Facilitate archiving [17,21]
Communication Guidelines
Provide communication integration [6,7,28,45]Facilitate archiving [17,21]
Intensive Analysis GuidelinesProvide customizable alerting [20]Provide automatic detection [26,41]Provide data correlation and filtering [1,26]
Intensive Analysis GuidelinesProvide customizable alerting [20]Provide automatic detection [26,41]Provide data correlation and filtering [1,26]
Configuration and Deployment GuidelinesMake configuration manageable [3,20]Support rehearsal and planning [3,6,7,20,44]Make configuration easy to change [20,46]Provide meaningful errors [20, 34,46]
Configuration and Deployment GuidelinesMake configuration manageable [3,20]Support rehearsal and planning [3,6,7,20,44]Make configuration easy to change [20,46]Provide meaningful errors [20, 34,46]
Mor
e Sp
ecifi
cM
ore
Spec
ific
Class will be 1 big group3 volunteer note takers
Problem: How to design the user interface for
a car proximity detection system
Brainstorm 3 aspects of the problem: (e.g., physical form factor, safety
issues, input techniques, etc.) go: 5 minutes
affinity diagram exercise
Now take your notes from the earlier brainstorming and create an affinity
diagram
go: 8 minutes
debrief