Maryam Bolouri

Mixed method Research

Toward New Designs in

Applied LinguisticsHashemi &Babaii

MLJ 97,4, 2013

Movement in inquiry in AL

• Emergence of Mixed method• Driving forces behind this shift:1.Advances in natural science2.Development and challenges of social and

educational research3.Criticism against the dichotomy of

qualitative and quantitive research methods

Statement of problem:

• How these 2 methods can be integrated usefully in a way to enhance the research tools?

• How these 2 methods can be integrated usefully in a way to enhance the quality of obtained findings?

Purpose of the study:

• This study attempted to be a first move to study

1.The current status of mixed methods research in AL

2.Explore the nature of mixed research designs, sampling designs and the quality of inferences that can be obtained

Review of literature• Mixed method definition:

1.Simple definition by Bergman (2008): the combination of at least one qualitative and at least one quantitive component in a single research project or program

2.Technical definition by Hanson et. al (2005): the collection, analysis and integration of quantitive and qualitative data in a single or multiphase study

3.Extended definition by Tashakkori & Creswell (2007a): research in which the investigator collects, and analyzes data integrate the findings and draws inferences using both qualitative and quantitive approaches in a single study or program of inquiry

Mixed method origins• Mixed method research has evolved from mono method

research.• The first instances of mixing designs can be traced back to

1800s.• It has been influenced by the emergence of the

“Triangulation”• Navigation: taking bearings on 2 landmarks in order to

locate one’s position• Land surveying: determine the position of point C if we

know the positions of points A and B• Original definition: the use of multiple forms of qualitative

research methods in favor of richly developed multidimensional descriptions of the world

Theoretical issues in mixed method research

• Incompatibility thesis • commensurability validity• Pragmatist position (dialectical, balanced,

transformative or critical interpretive view):• From practice oriented perspective. It is not a major

hindrance, the value is evident and they can coexist in a dialectical relationship.

• The findings can enhance the understanding.• The absent one would remain partially answered or

wholly unanswered questions.

Methodological issues:in mixed method research

• Bazeley (2004): corruption of methods“It more that using a variety of methods….it is an attempt

to serve the dual purposes of generalization and in-depth understanding…”

• High quality mixed method research:Integrations happens to RQ, units of analysis, sample,

instrumentation, data collection methods and analytic strategies

Mixed method validity issues: (Onwuegbuzie &Johnson, 2006): legitimation

• Complementary strength and non-overlapping weakness of both are combined, so the validity

assessment is important and complex.• (Tashakkori & Teddlie, 2008): inference quality• It gathers info that result in meta inferences that

neither of method can do alone. They are an overall conclusion through integration.

Methodological issues:in mixed method research

Methodological issues:in mixed method research

• (Tashakkori & Teddlie, 2003): Inference• A researcher’s construction of the

relationship among people, events, and variables as well as the his/her

construction of respondent’s perceptions, behaviors, and feelings, and how there

relate to each other in a coherent and systematic manner

Research questions:

1. What kinds of research designs are used when method mixing is utilized in AL research?

2. What kinds of sampling designs are used in mixed methods research in the field of AL?

3. How are general conclusions or meta inferences developed in this type of research?

Methodology

1. Data gathering• How to select a representative sample

of all journals?2. Data analysis• What frameworks and typologies

inform data analysis?

Data gathering• 7 comprehensive international peer reviewed AL

journals• Over a period of 14 years 1995---2008 • Journal selection: based on 5 sources:1.The list of professional journals published by MLJ- Weber &

Campbell, 20042.Evaluation of AL journals -Egbert (2007)3.Examination of the frequency of the appearance of ELT

journals selected for presentation in lg teaching btw 1996-2002-Jung, (2004)

4.A study on the use of qualitative and quantitative methods in 332 articles over a 7 year period- Lazarton (2000)

5.Expert judgment (2 PhD holders with research experience of more than 10 years and currently active in the field of AL)

Data gathering (cont.)• The first four sources yielded a comprehensive list of

64 journals• Expert judgment• This exclusion was based on the following five

reasons:1.An indirect relationship to the field of AL such as:

computers and compositions, British journal of educational psychology.

2.Publishing in lg other than English3.Insufficient info in the methodology section of

articles4.Lesser prominence on the international scene5.Strongly delimited research focus such as:

assessing writing, lg and cognitive process

The final list of Journals1. Applied linguistics-AL2. English for specific purpose-ESP3. Language learning-LL4. Language teaching research-LTR5. Language testing-LT6. The modern language journal-MLJ7. TESOL quarterly-TQ

The process of gathering the data sources

• The electronic versions• Screening phase (abstracts and

methodology)• Search function of adobe reader

A total of 273 articles were identified

Review of Data gathering

Data Analysis1. Rigorous qualitative content analysis(rich description of mixing in the process of data

collection and data analysis)Content analysis was based on both:• Manifest content: the concrete terms used

for coding• Latent content: the coder’s interpretation of

the content2. Make use of code-sheets and respective

categories based on three FWs

Creswell et al. (2008) analysis of research designs

• Concurrent designs and Sequential designs• Concurrent triangulation design (parallel)• Concurrent embedded design( btw pre and

post tests)

• Sequential explanatory design : quan->qual

• Sequential exploratory design: qual->quan

• Sequential embedded design: before and after the intervention--qual

Collins et al. (2007) 2 dimensional model of mixed methods

sampling• Time orientation of components (concurrent or sequential)• Relationship of the qualitative and quantitative

samples 1.Identical 2.Multilevel3.Nested4.Parallel

Data Analysis (cont.)• Content analysis based on FWs• Consider the arguments of validity and

inference quality

Tashakkori and Teddlie’s (2008) integrative model of inference quality

in mixed method • It consists of 2 broad categories of 1.Design quality2.Integrative rigor

Data Analysis (cont.)Design quality

1. Design suitability2. Design adequacy

3. Within design consistency

4. Analytic adequacy

Integrative rigor

1. Interpretive consistency

2. Theoretical consistency

3. Interpretive agreement 4. Interpretive

distinctiveness5. Integrative efficacy

What is Integrative efficacy?

• The degree to which inferences made in each strand of mixed methods study are

effectively integrated into a theoretically consistent meta-inference.

• Simply speaking:

effective integration of both qual and quan findings

extended definition of Tashkkori & Creswell (2007a)

1. Mixing should happen at different stages of a particular study

2. Meta inference must be developed based on data integration both qual and quan

It led to the exclusion of 68 articles and the final sample included 205 articles

The reliability of the coding • 1) An independent coder with qualitative

background coded a sample of articles ( 10% of the sample 20 articles)

• 2) Cohen’s Kappa was calculated . It was • 0.82 for coding of research designs • 0.71 for coding of sampling designs.

• 3) These indices were checked against Altman (1991) criteria and proved to be acceptable.

Review of Data Analysis

Research question1: What kinds of research designs are used when method mixing is

utilized in AL research?

Research question1: What kinds of research designs are used when method mixing is

utilized in AL research?• The direct use of mixed designs was directly

acknowledged in only a few studies• . Triangulation at the level of analysis is

referred to as triangulation by analyses(68 articles out of 136 articles that used triangulation)• The term triangulation was used With different wording

such as: Multiple data source, multiple measures, qualitative and quantitive data

• the emergence of mixed method designs that were not in Creswell et al. typology (7 articles) next table

Research question2: What kinds of sampling designs are used in mixed methods research in

the field of AL?

Research question3: How are general conclusions or meta-inferences developed in

this type of research?

• The analyses of discussion and conclusion sections revealed that data integration and consolidation were not presented

• These meta-inferences were offered by researchers in order to:

1.enrich their discussions and conclusions

2.present informative summaries of the findings

Discussions: 1- Use of mixed research design

Concurrent triangulation design (66%)1. Providing the supplementary data in multilevel D.A2. Needs analysis3. Program evaluation (collect data from various sources or by

multiple methods)Lg Testing and ESP journals (Need assessment /program

evaluation)Adopting a particular design could be a practical

considerationsConcurrent embedded designs(MLJ and lg learning)Sequential explanatory design

Discussions: 1- Use of mixed research design

• Sequential exploratory design:• development of questionnaire tests inventories • validating the measurement instruments• Sequential embedded designs • examine cross sections of the test development

and validation process.• ethnographical studies in order to develop

patterns in a more rigorous way.

Discussions 2-Use of mixed sampling design

This study (2013)• Concurrent Identical designs• multilevel concurrent>>>

Concurrent nested• Why: • different samples of articles in

terms of disciplines, size, purpose, rationale for mixing

• Why concurrent identical:• Practicality, feasibility, time

span

Collins et al (2007)• Concurrent Identical

designs• Concurrent

nested>>>multilevel concurrent

Discussions :3-Inference quality

• Why there was scant attention to the concept of integrative efficacy among the articles?

• The concept hasn’t been introduced appropriately to AL research

• It hasn’t been widely used in AL research• It can be explained by Tashakkori and Creswell

distinction btw • Methodology: integration of the 2 approaches

throughout the study• Method: collecting and analyzing different types of

qual and quan data

Implications of the study

• 1) To explore complicated process and outcomes• 2) Concurrent triangulation designs (DA, GA, NA,

…)• 3) Concurrent embedded design (Lg development

or a process within an experiment)• 4) Sequential explanatory designs:• To explore a subsample from large samples used in qual

phase• Analysis of outliers-extreme cases-inconsistent

components or incompatible subsystems• Further analysis of performance on quan measurement

instruments and evaluate DIF items after statistical analyses

Implications of the study(Cont.)

• Sequential exploratory designs: validating evaluation criteria such as tests, surveys, questionnaire, scales or band descriptors

• Sequential embedded design: longitudinal studies of development process in SLA analyze cross sections of test development and

validation processto investigate social aspects of lg learning such as T ID,

L ID, T D, L D, interaction patterns,…

Summary• Innovative mixed method can be used in

design and sampling in AL based on • Research purpose• Requirements of the research contextHigh quality integration in mixed method

depends crucially on design quality and quality of meta-inferences

• This can happen as result of integration of qual and quan components in all of the stages of the study

Penny for thoughts

• What are the evaluative criteria for investigating the integration quality in AL research?

• What is the tendency of some journals in use of particular designs?

(As it was not the main focus of this study, more accurate accounts of it would require further research.)

Thank You for your

patience