A worked example of META-AGGREGATION

Its historyIts philosophy

A worked example

Meta-aggregation• A structured, and process driven approach to systematic

review drawing on the classical understandings and methods associated with systematic review of quantitative literature as practiced by the Cochrane and Campbell Collaboration

• Based on an a-priori protocol– Established, answerable question– Explicit criteria for inclusion– Documented review methods for searching, appraisal,

extraction and synthesis of data

Meta-aggregation: History• Participative consensus project involving

Australian academics

• The outcome = an aggregative approach to the synthesis of qualitative evidence that:– emphasised the complexity of interpretive and critical

understandings of phenomena; recognised the need to ensure that the process is practical and usable;

– balanced utility of the outcomes with the complexity of the material;

– was grounded in pragmatism.

Meta-aggregation: A pragmatic approach

• The philosophy of pragmatism - Charles Peirce, William James, John Dewey – the value of any thought lies in its practical use and consequences. It is the

focus on practical consequences that characterizes meta-aggregation as a synthesis approach

– Central to pragmatism were communities of inquiry or groups of people with a shared interest, problem or issue trying to resolve doubt through critical reasoning.

– Actions are assessed in the light of their practical consequences. Aggregation concentrates on the original researchers’ findings (processed data) and summarizes common and competing findings to produce cross-study generalizations that lead to recommendations for action The utility of an outcome is important.

Research articles

Meta-aggregation: a worked example

• Review Topic: Obstacles to the implementation of Evidence-Based Practice in Belgium: a context-specific study

• Question formulation:– Setting: Belgium (Belgian health care system)– Population: Health Care Practitioners– (Phenomena of) Interest: Obstacles toward the implementation of

Evidence-Based Practice– Comparison: (additional literature search on strategies developed by

countries with other systems)– Evaluation: Experiences and perceptions

Meta-aggregation: a worked example

• Search strategy:– Major databases: Medline, CINAHL, Psychinfo, Embase, Social Sciences abstracts

and ERIC (1990-May 2008). – Where possible we used a methodological filter for qualitative research. – Other information sources:

• ‘Federal research actions’-database from the Belgian governmental department of science

• Consultation of Belgian experts in qualitative research methods and/or Evidence-Based Practice to check on any other published material that could be of use for the synthesis.

• Screening of references– Keywords:

• ‘Evidence-Based Practice (Medical Subject Heading (MeSH) term)’, including Evidence-Based Dentistry, Nursing and Medicine.

• Additional keywords: evidence-based combined with the geographical notion ‘Belgian’, ‘Flemish/Flanders’ or ‘Walloon’.

Meta-aggregation: a worked example

• Inclusion criteria:

– Study type: Qualitative, empirical research papers. Opinion pieces and descriptive articles were excluded.

– Study participants: All types of health care practitioners e.g. physicians, dentists, nurses, physiotherapists, psychologists

– Topic of interest: Obstacles to the implementation of EBP.– Context: the Belgian health care system.– Outcome of interest: Experiences and/or perceptions from

participants.

8 studies met the inclusion criteria

Meta-aggregation: a worked example• Critical appraisal of selected studies:• No criteria based quality appraisal, although the software QARI

provides a standard critical appraisal checklist.– limited amount of studies found– the majority of the selected studies were written by the lead

reviewer, who would also be one of the appraisers. • An overall judgment approach was used instead. This approach

has been proven to deliver the same outcome (Dixon-Woods, 2007). However, it tends to be less explicit about potential reasons for exclusion.

• An evaluation based on the JBI-critical appraisal instrument is recommended for other review teams opting for a meta-aggregative approach to synthesis.

• 85 different findings were found • The findings were classified in 9 major categories:

– (1) evidence is hard to implement– (2) decision making processes are influenced by practitioner variables– (3) decision-making processes are influenced by practitioner variables– (4) commercial/financial interests affect EBP– (5) governmental regulations influence the process of implementation, – (6) EBP is more feasible for practitioners working in a scientific setting compared

to those working in other settings– (7) physicians’ display of power hampers other professionals to put evidence-

based practice into effect– (8) a lack of knowledge and skills hinders the implementation of evidence-based

practice– (9) attitudes of health care practitioners hinder the implementation of EBP.

Meta-aggregation: a worked example

• These categories were further analysed to produce 4 synthesized statements

– Synthesis 1: Evidence might have a limited role in decision-making processes in daily practice, if the importance of the scientific component is not stressed (categories 1, 2 and 3).

– Synthesis 2: Aspects other than quality of care will steer the EBP agenda, if governmental regulations and economic interests are contra-productive for delivering the best possible care (categories 4 and 5).

– Synthesis 3: Although EBP is intended to serve all practitioners, some health care providers will benefit less from EBP than others, if inequity issues between practitioners are not solved and support for field workers is not established (categories 6 and 7).

– Synthesis 4: A lack of competences will hinder the implementation of EBP, if gaps in knowledge and skills are not being filled and efforts to change contra-productive attitudes are not undertaken (categories 8 and 9).

Meta-aggregation: a worked example

Meta-aggregation: a worked example

• Implications for practice and policy

– Consider educating patients and exploring potential information channels that influence patients’ opinions to create space for well-informed decisions (synthesis 1).

– Provide easy and free access to well-structured, compact and relevant information targeted to a particular discipline and consider helpdesks. Screen information, control its quality and translate it to the field (synthesis 1).

– Consider updating the Belgian nomenclature and reimbursement system to bring it in line with the latest evidence (synthesis 2).

– Consider incentives for those practitioners who are engaging themselves for the implementation of EBP in daily practice, to keep them motivated and to prevent from creating a negative spiral in which practitioners tend to believe that efforts to improve practice automatically lead to a loss of income (synthesis 2).

Meta-aggregation: a worked example

• Implications for practice and policy

– Enhance communication and cooperation between physicians and other partners in health care, via their professional groups and journals (synthesis 3)

– Consider programs tailored to the needs of specific disciplines and stimulate multidisciplinary education should be stimulated to create mutual understanding across disciplines (synthesis 3, 4).

– Consider direct access to allied health services to increase autonomy (synthesis 3)

– Integrate EBP in the basic curricula (synthesis 4).

Meta-aggregation: a worked example

The Qualitative Assessment and Review Instrument (QARI)

Designed to provide a systematic process mirroring that taken for systematic reviews of quantitative research, whilst being sensitive to the nature of qualitative data.

Data Extraction

Reduce•The findings of many studies into a single document

Summarise•Methods•Phenomena•Findings

The units of extraction in this process are specific findings and

illustrations from the text that demonstrate the origins of the

findings.

In QARI a finding is defined as: A conclusion reached by the

researcher(s) and often presented as themes or metaphors.

Levels of credibility

Unequivocal - relates to evidence beyond reasonable doubt which may include findings that are matter of fact, directly reported/observed and not open to challengeCredible - those that are, albeit interpretations, plausible in light of data and theoretical framework. They can be logically inferred from the data. Because the findings are interpretive they can be challenged.Not Supported - when 1 nor 2 apply and when most notably findings are not supported by the data

Meta-Synthesis

The aim of meta-synthesis is to assemble findings; categorise these findings into groups on the basis of similarity in meaning;

and to aggregate these to generate a set of statements that adequately represent that aggregation. These statements are

referred to as synthesised findings - and they can be used as a basis for evidence based practice.

In QARI, a synthesised finding is defined as an overarching description of a group of

categorised findings that allow for the generation of recommendations for practice.

Can be stated propositionally as “if-then” statements - for example: “If students are advised reconsider their choice of study , their relatives will sometimes feel as if they are not involved”. (a somewhat awkward and eccentric form)

the declamatory form that emphasizes the probability of the claim: “Relatives of students that are advised to reconsider their choice of study may feel as if they are not involved if strategies to include them are not pursued”.

Should look familiar!