Education Program Experience for the Appraisers of Guidelines using AGREE II Scoring Guide in Korea Sung-Goo Chang 1) 6), Ein-Soon Shin 2), Ji-Eun Jang

Education Program Experience for the Appraisers of Guidelines using AGREE II Scoring Guide in Korea

Sung-Goo Chang1) 6), Ein-Soon Shin2), Ji-Eun Jang2), Min-Ji Kim2), Ji-Yun Yeon2),You-Kyoung Lee3), Heui-Sug Jo4) 6), Dong-Ik Kim5)6)

1) Department of Urology, Kyung Hee University School of Medicine, Seoul, South Korea

2) Research Agency for Clinical Practice Guidelines, KAMS Research Center, The Korean Academy of Medical Sciences, Seoul, South Korea

3) Department of Laboratory Medicine and Genetics, Soonchunhyang University College of Medicine, Bucheon, South Korea

4) Department of Health Management and Policy, Kangwon National University School of Medicine, Chuncheon, South Korea

5) Department of Diagnostic Radiology, Yonsei University College of Medicine, Seoul, South Korea

6) The Korean Academy of Medical Sciences, Seoul, South Korea

http://www.guideline.or.kr

Can we reduce differences among 4 appraisers?

Guideline ratings require a level of judg-ment

1. All 23 AGREE II items are rated on a 7-point scale.

2. Score 1 (strongly Disagree) should be given when there is no informa-

tion or if the concept is very poorly reported.

3. Score 7 (Strongly Agree) should be given when the full criteria in ‘how

to Rate’ and/or ‘Further Considerations’ are articulated in the User’s

Manual.

4. Scores between 2 and 6: a score is assigned depending on the com-

pleteness and quality of reporting. Scores increase as more criteria

are met and considerations addressed. To reduce differences among appraisers1. Plan an effective education program to train qualified appraisers 2. Need to define the concept and details on assessment criteria → develop a scoring guide for anchor points 1, 3, 5, 7 in 23 items 3. Have to identify explicit elements that reflect the operational definition → develop a practical working sheet to identify elements for making correct decision and finding elements that show differences among appraisers


Objectives

To develop an effective education program to train qualified appraisers (multidisciplinary physicians) of guidelines

To apply and examine a Korean AGREE II* scoring guide developed by KAMS that is time-saving and increasing reliability for the assessment of quality of CPGs among appraisers

•Purpose of the AGREE II (Appraisal of Guidelines for Research & Evaluation II)1) Assess the quality of guidelines2) Provide a methodological strategy for the development of guidelines3) Inform what information and how information ought to be reported in guidelines

To maximize the applicability of a Korean AGREE II scoring guide and reduce the disagreement rate among appraisers providing a practical implementation tool such as working sheet to identify elements for correct judgment and finding elements of differences among appraisers


2

Methods: Education program for evaluators

1

3

4

The planned education program was provided to 29 participants who are physicians to assess quality of CPGs on November 29, 2013.

Providing a series of training courses including one hour lecture, demonstration and maximum two hours practice.

To make correct judgment, 29 participants learned how to apply a developed Korean AGREE II scoring guide and practical working sheet to identify elements in detail.

‘Guideline for prevention and treatment of metabolic syndrome in primary care’ was assessed under 23 items of AGREE II instrument. Finally, evaluators made a judgment score after considering both identified elements and scoring guides for anchor points 1, 3, 5, and 7.


Korean version of AGREE II User’s Manual

http://www.agreetrust.org/resource-centre/agree-ii-translations/


Korean AGREE II Scoring Guide - Development

92 draft scoring guides for anchor points 1, 3, 5, and 7 (full score) in 23 items was developed by Executive Committee for CPGs, the Korean Academy of Medical Sciences (KAMS) in Korea.

In consideration of the importance of using a quantitative measure of satisfaction with the elements described in the AGREE II manual (analyzed the user’s manual, how to rate, and additional consideration) , a final draft was developed through a Delphi consensus process.

Consensus to finalize the scoring guide was defined as agreement among at least 70% of the raters.


Korean AGREE II Scoring Guide - Development

Agreement on 88 of 92 draft scoring guidelines was reached in the first Delphi round (95.7%), and agreement for the remaining 4 was achieved in the second round (4.3%).


Korean AGREE II Scoring Guide - an example

평가영역 1. 범위와 목적 (Domain 1. Scope and Purpose)

2. 진료지침에서 다루고자 하는 건강 관련 질문들이 구체적으로 서술되어 있다 .(2. The health question(s) covered by the guideline is (are) specifically described.)

7

진료지침에서 다루는 질문이 PICO 요소를 포함하고 있고 , 별도의 리스트로 되어 있어 찾기 쉽고 내용이 명확히 서술된 경우 (Reported the clinical/health questions with PICO(H) components, easy to find be-cause of provided separate list, and descriptions are clear and concise)

5

질문이 일목요연한 문장이지만 , PICO 의 요소가 없거나 부족하면서 , 별도의 리스트로 되어 있지 않은 경우 (Reported as a structured statements with lack of PICO(H) com-ponents and/or no separate list)

3질문이 단어로만 이루어진 소제목과 같이 최소의 정보로 제시된 경우 (poorly reported or providing a minimum information such as a form of heading with couple of words combination without PICO(H) components)

1진료지침에서 다루는 질문이 제시되어 있지 않은 경우 (not provided the clinical/health questions or no information related PICO(H))


Korean AGREE II Scoring Guide


Define disagreement among appraisers

Disagreement was defined by more than 4 judgment score differences among appraiserson the same item using the 7-point rating scale.

Judgment score of the appraiser #1 : 1Judgment score of the appraiser #2 : 6Judgment score of the appraiser #3 : 5Judgment score of the appraiser #4 : 6

Domain 5. Applicability. 18. The guideline describes facilitators and barriers to its application.

1 Strongly

Disagree2 3 4 5 6

7 Strongly

Agree

Disagreement


Practical implementation tool to identify explicit elements

Develop and apply a practical working sheet to identify elements for correct judgment and easily find elements that show differences among ap-

praisers

영역 3. 개발의 엄격성 (Domain 3 Rigour of Development)14. 진료지침의 갱신 절차가 제시되어 있다 .

(A procedure for updating the guideline is provided)5

(How to rate) Comment ( 평가근거 서술 , page 표기 ) (Yes/No)

지침이 개정될 것이라는 언급 (a statement that the guideline will be updated)

명시되어 있음 (p.10), 이에 대한 자세한 일정은 추후 대한소화기기능성질환 , 운동학회 웹사이트를 통해 공지할 예정임을 언급함 .

Y

개정기간을 명시하거나 개정을 결정하는 판단 기준의 명시 (explicit time interval or explicit criteria to guide decisions about when an update will occur)

개정기간 (4-5 년 ) 명시되어 있음 (p.10), 개정결정 판단기준( 향후 새로운 검사법이나 약제 , 치료법 등이 개발되고 새로운 연구결과가 축적되어 우리나라 국민 건강증진에 필요하다고 판단될 경우 ) 가 명시되어 있음 (p.10)

Y

개정과정의 방법론을 제시 (methodology for the updating procedure is reported)

향후에 신규방법으로 개정을 할지 혹은 수용개발 방법으로 개정 할지에 대한 구체적인 언급이 없음(not described)

N


Results

AGREE II

DomainsItem

AgreementDisagreem

ent

No

response

Total

number of

appraisers

N % N % N % N %

1. Scope and

Purpose

1 29 100.0 0 0.0 0 0.0 29 100.0

2 28 96.6 1 3.4 0 0.0 29 100.0

3 24 82.8 5 17.2 0 0.0 29 100.0

2. Stakeholder

Involvement

4 29 100.0 0 0.0 0 0.0 29 100.0

5 19 65.5 8 27.6 2 6.9 29 100.0

6 28 96.6 0 0.0 1 3.4 29 100.0Stakeholder Involvement. Item 5. showed the highest dis-agreement. The views and preferences of the target popula-

tion (patients, public, etc.) have been sought.


Reasons for the highest disagreement (27.6%) - Stakeholder Involvement. Item 5

Possible reasons:1. Do not have any qualified data on the view and preferences of patients and/or public

in Korea2. No guidelines to consider the patient’s value and preferences in Korea3. Appraisers do not understand the definition for the views and preferences of the

target population (patients, public, etc.)4. Showed the highest disagreement because appraisers could not identify the proper

elements

Terminology: values and preferences (Jeff Andrews et al. GRADE Guidelines 14. 2012)Values and preferences is an overarching term that includes patients’ perspectives, beliefs, expectations, and goals for health and life. More precisely, they refer to the processes that individuals use in considering the potential benefits, harms, costs, limitations, and inconvenience of the management options in relation to one another. For some, the term ‘‘values’’ has the closest connotation to these processes. For others, the connotation of ‘‘preferences’’ best captures the notion of choice. Thus, we use both words together to convey the concept.

Domain 2. Stakeholder Involvement. Item 5. showed the highest disagreement.The views and preferences of the target population (patients, public, etc.) have been sought.


Results

Rigour of Development. Item 11. The health benefits, side effects, and risks have been consid-

ered in formulation the recommendations.

AGREE II

DomainsItem

Agreemen

t

Disagreem

ent

No

response

Total

number of

appraisers

N % N % N % N %

3. Rigour of

Development

7 29 100.0 0 0.0 0 0.0 29 100.0

8 26 89.7 1 3.4 2 6.9 29 100.0

9 25 86.3 1 3.4 3 10.3 29 100.0

10 24 82.8 3 10.3 2 6.9 29 100.0

11 20 69.0 7 24.1 2 6.9 29 100.0

12 23 79.2 2 6.8 4 14.0 29 100.0

13 23 79.2 2 6.8 4 14.0 29 100.0

14 22 76.0 4 13.7 3 10.3 29 100.0


Reasons for the high disagreement (24.1%) - Rigour of Development. Item 11

Domain 3. Rigour of Development. Item 11. The health benefits, side effects, and risks have been considered in formulation the recommendations.

Possible reasons:1. Appraisers did not identify the proper elements because of poor

understanding - need a plan to provide a repeated education session to understand the exact definition of AGREE II items and educate how to identify the elements

2. Appraisers did not consider the description scoring guide by anchor point 1, 3, 5, 7. - need to make/revise a clear and concise description and educate how to apply scoring guide

3. Showed pretty high disagreement because of limited time to read and evaluate guideline - to make correct judgment after reading and identifying the elements


Results

6 to 9 of 29 appraisers (20.7~31.0%) did not rate the 7-point scale on the 3 domains, 9 items among 23 items (39.1%).

AGREE II

Domains Item

Agreemen

t

Disagreem

ent

No

response

Total

number of

appraisers

N % N % N % N %

4. Clarity of

Presentation

15 23 79.3 0 0.0 6 20.7 29 100.0

16 19 65.5 2 6.9 8 27.6 29 100.0

17 22 75.9 1 3.4 6 20.7 29 100.0

5. Applicability

18 18 62.1 2 6.9 9 31.0 29 100.0

19 20 69.0 1 3.4 8 27.6 29 100.0

20 18 62.1 2 6.9 9 31.0 29 100.0

21 19 65.6 1 3.4 9 31.0 29 100.0

6. Editorial

Independence

22 22 75.9 1 3.4 6 20.7 29 100.0

23 17 58.6 4 13.8 8 27.6 29 100.0


No response: did not rating the 7-point scale

- the 3 domains, 9 items among 23 items

Main Reason:. Appraisers did not evaluate domain 4, 5 and 6 (9 items of 23) and showed high rate of no response with ranged 20.7% to 31.0% because of limited time to read and evaluate guideline during a practice session- Need to provide enough time to make correct judgment after reading

guideline carefully and identifying the proper elements

Domain 4. Clarity of

Presentation 20.7%-27.6% (3 of 3 items)

Domain 5. Applicability 27.6%-31.0% (4 of 4 items)

Domain 6. Editorial Independence

20.7%-27.6%

(2 of 2 items)


Limitation and proposed way forward

Limitations

Proposed way forward

Difficult to select a good quality of guideline for education purpose - using an evidence-based ADAPT guideline, could not meet the criteriaLimited time (maximum 2 hours for practice) was one of the difficulties to educate and evaluate a guidelineHard to understand and rate the patient values & preferences (Item 5)because have no experience to identify and collect this element in Korea Could not consider the weight among anchor points (1, 3, 5, 7) in 23 items

Need constant monitoring and updating of the scoring guide to reflect user’s feedbackProviding an easy, clear operational definition for scoring guideDevelopment of the scoring guide in English version will be beneficial not only to domestic appraisers but also to international appraisers


2

Lessons learned and Conclusions

1

Minimizing differences among appraisers is important when measuring quality of CPGs.

We demonstrated that Korean AGREE II scoring guide and practical working sheet were time-saving and useful tool to educate appraisers about CPGs with low disagreement rate.

The detailed scoring guide for 23 items of AGREE II instrument can successfully clarify the guideline assessment process.

3

Applying a practical implementation tool such as working sheet to identify elements for correct judgment was very easy and useful, and helped finding the elements that caused differences among appraisers.

THANK YOU

Authors are free of any real or perceived bias introduced by receipt of any benefit in cash or kind, any

hospitality, or any subsidy derived from any source that may have or be perceived to have an interest in

the outcome of the study. Especially there are no conflicts of interest that relate to industry.

For copies of slides:[email protected] Shin, PhD, MPH, Lic.Ac.

mailto:[email protected]

Documents

Education Program Experience for the Appraisers of Guidelines using AGREE II Scoring Guide in Korea Sung-Goo Chang 1) 6), Ein-Soon Shin 2), Ji-Eun Jang