Data Analysis, Interpretation, and Presentation

Devin SpiveyAsmae Mesbahi El Aouame

Rahul Potghan

Objectives• Difference between qualitative and

quantitative data and analysis.• Analyze data gathered from questionnaires.• Analyze data gathered from interviews.• Analyze data gathered from observation

studies.• Software packages for data analysis.• Pitfalls in data analysis, interpretation, and

presentation.• Presenting your findings.

Qualitative and quantitative

• Quantitative: numbers, translated into numbers.

• Qualitative: difficult to express in numerical terms in a sensible fashion.

Be careful in translating qualitative data into quantitative data.

Types of Data

Simple quantitative analysisSimple quantitative analysis techniques:• Percentages: for standardizing data

(compare large sets of data).• Averages:

Mean: commonly understood average.Median: middle value of the data.Mode: most commonly occurring number.

Initial Analysis: finding averages, outliers, depict any patterns from the graphical representation of data on a spreadsheet.

Initial analysis example: data gathering

• Evaluation study of an e-commerce website: identify transactions’ difficulties faced by users.

• Data gathering methods:

Questionnaires.Observation of a

controlled task.data logging.

Initial analysis example: Visualization of data

Number of errors made

00.5

11.5

22.5

33.5

44.5

1 3 5 7 9 11 13 15 17

User

Num

ber o

f error

s mad

e

Internet use

< once a day

once a day

once a week

2 or 3 times a week

once a month

Initial analysis example: Visualization of data

0123456789

10

0 5 10 15 20

Num

ber o

f error

s mad

e

User

Number of errors made • Outlier: Removed from the

larger data set since it distorts the general patterns.

Interesting case for further analysis.

Activity(1)

• The data in the table represents the time taken by a group of users to select and buy an item from an online shopping website.

Activity(2)Further investigation:

• The values for users N(24) and S(26) are higher than the others.

• Trends: - Users at the beginning of the testing time performed faster than those towards the end of the testing. - O is at the end of the testing but performed well.

How question design affects data analysis

Comparing two products: 2 Phone designs

Initial analysis from data logs (1)

• Research question: investigate how effective Massively Multiplayer Online Role-Playing Games (MMORPGs) are at encouraging interactivity between users.

• Data: data logs and video recordings of players interactions in SWG.

• Ethnography study to identify the locations heavily used by players.

Initial analysis from data logs (2)

• Qualitative analysis –expresses the nature of elements and is represented as themes, patterns, stories

• Qualitative data: difficult to measure sensibly as numbers.e.g. counting number of words to measure dissatisfaction

.

Simple Qualitative Analysis

• -The first step is to gain an overall impression of the data and start looking for patterns.

- Next comes more detailed work using structured frameworks or theories to support the investigation.

- Patterns may relate to a variety of aspects. e.g. behavior, user groups, places or situations where certain events happen.

• In terms of structure:- Unstructured: Not directed by a script. Rich but not replicable. e.g. video material. - Structured: Tightly scripted. Replicable but may lack richness. e.g. questionnaire. - Semi-structured: Guided by a script but interesting issues can be explored in more depth. Provides a good balance between richness and replicability.e.g. interviews.

• When Should I Use Qualitative Vs. Quantitative Research?

http://www.youtube.com/watch?v=638W_s5tRq8&feature=related

Three types of qualitative analysis

1. Identifying recurring patterns and themes

2. Categorizing data

3. Analyzing critical incidents

1. Identifying recurring patterns and themes

1. Identifying recurring patterns and themes

• Emergent from data, dependent on observation framework if used

• Patterns in Quantitative can be find identified by graphical representation but for Qualitative, requires researcher to be immersed in data

- Studying the data- Focusing on the study goals- Keeping clear records of the analysis as it

progresses and close description of themes or patterns that are emerging.

- e.g. Box 8.4, Themes in European culture.

2. Categorizing data• Can be at a high level of detail such as identifying

stories or themes OR at a fine level of detail in which each word, phrase or gesture is analyzed.

• Most challenging aspects :1. Determining meaningful categories that are orthogonal (do not overlap each other in any way).2. Deciding on the appropriate granularity for the categories (word, phrase, sentence, or paragraph level).

• The categorization scheme used must be reliable so that the analysis can be replicated!

- Categorization scheme may be emergent or pre-specified

3. Looking for critical incidents• It is a flexible set of principles that emerged from

work carried out in the United States Army Air Forces where the goal was to identify the critical requirements of “good” and “bad” performance by pilots.

• Two basic principles:1. Reporting facts regarding behavior is preferable to the collection of interpretations, ratings and opinions based on general impressions.2. Reporting should be limited to those behaviors which make a significant contribution to the activity

- Helps to focus in on key event- e.g. Box 8.5, analyzing video material

Tools to support data analysis

• Qualitative data analysis tools– Categorization and theme-based analysis

• N6 can be used to search a body of text to identify categories or words for content analysis

• N6 is used to handle large set of data.– Quantitative analysis of text-based data- Focuses on the number of occurrences of words or words with Similar meanings.

CAQDAS

CAQDAS Networking Project, based at the University of Surrey (http://caqdas.soc.surrey.ac.uk/) support theory-building through the visualization of

relationships between variables that have been coded in the data.

CAQDAS is useful– Grounded Theory– Qualitative Content Analysis – Ethnography

SPSSSPSS is Statistical Package for the Social Science

– It is used by market researchers, health researchers, survey companies, government, education researchers, marketing organizations and others

Statistical packages, e.g. SPSS– You can access, manage, and analyze enormous

amounts of data with SPSS.– SPSS offers statistical test for frequency

distributions, rank correlations, regression analysis, and cluster analysis

Observer Video-Pro

• The Observer Video-Pro is a system for collecting, managing, analyzing, and presenting observational data. – It integrates The Observer software with time code

and multimedia hardware components.

Observer Video-Pro The software allows the user to summarize research

findings in numerical, graphical, or multimedia format

http://www.noldus.com/human-behavior-research/products/the-observer-xt

Theoretical Frameworks for Qualitative Analysis

• Structuring the analysis of qualitative data around a theoretical framework can lead to additional insights that go beyond the results found from the simple techniques introduced earlier.

• Three frameworks are discussed in this section: Grounded theory, Distributed Cognition, and Activity theory.

Grounded Theory• Grounded theory aims to develop theories from

systematic analysis and interpretation of empirical data, i.e. the theory is grounded in the data.

• A grounded theory is developed through alternating data collection and data analysis:– Data is first collected and analyzed to identify

categories, then that analysis leads to the need for further data collection, which is analyzed, and more data is then collected.

– Data collection is driven by the emerging theory.– Approach continues until no further insights emerge

and the theory is well developed.

Grounded Theory• Analysis is mainly to setup Categories.• Category identification and definition is achieved by

“coding” the data, i.e. marking the data up according to the emerging categories.

• Coding has three aspects:– Open coding – the process through which categories are

discovered in the data.– Axial coding – the process of systematically fleshing out

categories and relating them to their subcategories.– Selective coding – the process of refining and integrating

categories to form a larger theoretical scheme. Categories are organized around one central category that forms the backbone of the theory.

Grounded Theory

• Researchers are encouraged to draw on their own theoretical backgrounds to help inform the study, as long as they are alert to the possibility of unintentional bias.

• Emphasizes the important role of empirical data in the derivation of theory.

Distributed Cognition

• The people, environment and artifacts are regarded as one cognitive system

• Focuses on information propagation and transformation

• A distributed cognition analysis results in an event-driven description which emphasizes information and its propagation through the cognitive system under study.

Distributed Cognition• It is recommended to have a deep

understanding of the domain under study. Even taking steps to learn “the trade” under study. This could take more time than a research team has available.

• Alternatively, it is possible to spend a few weeks immersed in the culture and setting of a specific domain to become familiar with it.

Distributed Cognition

• The framework can reveal where information is being ‘distorted’ resulting in poor communication or inefficiency.

• The framework can show when different technologies and the representations displayed via them are effective at mediating certain work activities and how well they are coordinated.

Activity Theory

• Activity theory (AT) is a product of Soviet psychology that explains human behavior in terms of our practical activity with the world.

• AT provides a framework that focuses analysis around the concept of an ‘activity’ and helps to identify tensions between the different elements of the system.

Activity Theory• AT outlines two key models:

– A model that outlines what constitutes an ‘activity’

– A model that outlines the mediating role of artifacts

• AT models activities in a hierarchical way.– ‘Activity’ – Provides a minimum

meaningful context for understanding the individual actions.

– ‘Actions’ – behavior that is characterized by conscious planning.

– ‘Operations’ – routinized behaviors that require little conscious attention.

Activity Theory

• Activity is motivated.• Actions are to accomplish a goal.• Actions involve operations.

Activity Theory• AT models artifacts in two ways:

– Physical– Abstract

• Physical artifacts have physical properties that cause humans to respond to them as direct objects to be acted upon. The object usually embody a set of social practices.– A spoon

• Abstract artifacts follow the idea of ‘mediation’. A fundamental characteristic of human development is the change from a direct mode of acting to one that is mediated by something else.– A set of rules or symbols

Presenting Your Findings• Only make claims that your data supports.• The best method for presenting your finding

depends on the audience, the purpose of the study, and the data gathering and analysis techniques used.

• Graphical representations (numbers, tables, graphs, etc.) may be appropriate for presenting your findings.

• Other techniques are:– Rigorous notations– Using stories– Summarizing your findings

Rigorous Notations

• UML is an example of rigorous notation because it uses notations that have clear syntax and semantics.

Using Stories

• Used as a basis for constructing scenarios.

• May be employed in three ways:– Stories told by the participants– Stories about the participants– Stories constructed from smaller

anecdotes or repeated patterns that are found in the data.

Summarizing the findings

• Summarizing the data by presenting the headline findings, overviews, and detailed content list.

• Numbers and statistical values can be very valuable in a summary.– If you found 800 out of 1000 users

preferred design A over design B. This statement would be a quick indication of your findings.

Summarizing the findings

• Activity• Consider each of the findings below

and the associated summary statement about it.

• What is correct or incorrect about each finding’s statement?

Conclusion• The kind of data analysis that can be done

depends on the data gathering techniques used.• Qualitative and quantitative data may be

collected from any of the main data gathering techniques: interviews, questionnaires, and observation.

• Quantitative data analysis for interaction design usually involves calculating percentages and averages.

• There are three different kinds of averages: mean, mode and median.

Conclusion

• Graphical representations of quantitative data help in identifying patterns, outliers, and the overall view of the data.

• Qualitative data analysis may be framed by theories. Three such theories are grounded theory, activity theory, and distributed cognition.