Quality Indicators for Non-Small Cell Lung CancerOperations With Use of a Modified DelphiConsensus ProcessGail Darling, MD, Richard Malthaner, MD, John Dickie, MD, Leigh McKnight, BMSc,Cindy Nhan, BSc, Amber Hunter, MBA, and Robin S. McLeod, MD, on behalf of theLung Cancer Surgery Expert PanelDivision of Thoracic Surgery, Department of Surgery, University Health Network, Toronto; Division of Thoracic Surgery, Departmentof Surgery, London Health Sciences Centre, Western University, London; Department of Surgery, Lakeridge Health Corporation,Oshawa; Cancer Care Ontario; Department of Surgery and Samuel Lunenfeld Research Institute, Mount Sinai Hospital; andDepartment of Surgery and Department of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada

Background. The aim of this project was to develop a setof quality indicators to assess surgical decision making inthe care of patientswith non-small cell lung cancer (NSCLC).

Methods. A multidisciplinary Expert Panel of 16 phy-sicians used a modified Delphi process to identify qualityindicators that evaluated the processes of care in patientswith NSCLC. A systematic review identified potentialindicators, which were rated on actionability, validity,usefulness, discriminability, and feasibility in tworounds of questionnaires. The first questionnaire wascompleted by the Expert Panel and by the larger thoracicsurgical community of practice; the second questionnairewas sent to only the Expert Panel. Expert Panel membersattended an in-person meeting to review the results of thetwo questionnaires and to compile the final list of in-dicators by consensus.

Accepted for publication March 4, 2014.

Address correspondence to Dr Darling, Toronto General Hospital, Divi-sion of Thoracic Surgery, 200 Elizabeth St, 9N955, Toronto, Ontario M5G2C4, Canada; e-mail: gail.darling@uhn.ca.

� 2014 by The Society of Thoracic SurgeonsPublished by Elsevier Inc

Results. From the literature review, 41 potential in-dicators were identified. An additional 16 indicatorswere suggested by the Expert Panel: 13 indicators inthe two rounds of questionnaires and three after thediscussion at the in-person meeting. One further indi-cator was identified after the in-person meeting. In theend, 17 indicators were chosen from seven domains:preoperative assessment, staging, surgical procedures,pathology, adjuvant therapy, surgical outcomes, andmiscellaneous.Conclusions. By use of a modified Delphi process, 17

indicators to assess the quality of processes of surgicalcare for patients with NSCLC were developed.

(Ann Thorac Surg 2014;-:-–-)� 2014 by The Society of Thoracic Surgeons

nsuring access to high-quality cancer surgical pro-

Ecedures is one of the mandates of Cancer CareOntario (CCO), which is the principal adviser to theOntario government on the delivery of cancer services.An Expert Panel was convened to develop thoracic sur-gical oncology standards because data from the provinceof Ontario and from other jurisdictions reportedimproved outcomes for pneumonectomy and esoph-agectomy for high-volume centers (unpublished data).The standards document stipulated volume levels, sur-geon criteria, and hospital resources for level I and level IIcenters.

Before 2004, approximately 46 hospitals in Ontariowere performing thoracic surgical procedures. After theimplementation of the standards, as of December 31,2010, there were 13 level I and two level II designatedthoracic surgery centers, and approximately 90% of all

procedures were performed in these centers. In an effortto assess the quality of surgical care, an Expert Panel wasconvened to develop a set of quality indicators that wouldassess the surgeon decision making component of careof patients with non-small cell lung cancer (NSCLC).This article outlines the process and outcome of thosedeliberations.

Material and Methods

A modified Delphi method was used to develop the LungCancer Surgery Quality Indicators (LCSQI) (Fig 1). Thismethod combines the nominal group process, which usesstructured meetings to gather information from experts,and the Delphi process, which uses questionnaires toelicit responses in a systematic manner over severalrounds [1]. In the modified Delphi process, participantsattend at least one in-person meeting. During develop-ment of the LCSQI, a systematic review of the literaturewas performed to extract potential indicators, which werereviewed and rated by clinical experts in two rounds ofquestionnaires and at an in-person meeting.

0003-4975/$36.00http://dx.doi.org/10.1016/j.athoracsur.2014.03.001

Fig 1. Modified Delphi process used to select and prioritize qualityindicators for surgical procedures for patients with non-small celllung cancer. (CoP ¼ community of practice).

2 DARLING ET AL Ann Thorac SurgLUNG CANCER OPERATION QUALITY INDICATORS 2014;-:-–-

Panel SelectionThree thoracic surgeons representing both academic andcommunity centers led the panel (GD, JD, RM). Othermembers were nominated by these physician leaders,ensuring that the majority of regions of the Province ofOntario were represented. In addition to thoracic sur-geons, representatives from radiation oncology, medicaloncology, respirology, diagnostic radiology, nuclearmedicine, and CCO administrators were included on thepanel. In total, the Expert Panel was composed of 16members, including the three physician leaders.

Literature SearchA literature search was performed by two of the authors(CN and LM) to identify potential evidence-based qualityindicators that were relevant to surgical decision making(Fig 2). The target population was adult patients withstage I, II, or III NSCLC who underwent surgical resectionwith curative intent. The goal was to identify synthesized

research evidence including clinical practice guidelines,systematic reviews, metaanalyses, population-basedstudies, and consensus statements that supported in-terventions and practices leading to improved patientoutcomes.The MEDLINE database was searched by use of the

following text/MeSH subject headings: “Surgical pro-cedures,” “Operative/Surgery/Surgery Department,”“Hospital/Thoracic Surgery,” “Thoracic Surgical Pro-cedures,” and “Thoracic Neoplasms.” The search waslimited to human studies in the English language pub-lished between 1995 and the second week of January 2009.Two reviewers (CN and LM) independently reviewedthe abstracts, and disagreements were resolved byconsensus. Articles were selected for inclusion if theywere fully published English-language reports.The grey literature was searched with the Google

search engine. Also searched for were nonpublishedorganizational reports, guidelines, and statements. Re-viewers hand-searched the reference lists of the includedarticles and all review articles to identify any additionalarticles missed in the literature search.Each retrieved article was examined by two indepen-

dent reviewers (CN and LM) to extract possible indicatorsfor consideration. For each potential quality indicator, thephase of care, type of article, and citation were collected,and a list of unique indicators with the level of supportingevidence was created. The indicators were organized intoseven categories that covered the continuum of care(preoperative assessment, staging, surgical procedures,pathology, adjuvant therapy, surgical outcomes, andmiscellaneous).

Round 1The list of unique quality indicators was formatted bycategory in a questionnaire and distributed by e-mail toall members of the Expert Panel and to all other thoracicsurgeons practicing at designated centers in Ontario, withthe exception of the three physician leaders. Respondentswere asked to rate the indicators by three criteria: (1)actionability, which refers to whether the care that theindicator assesses is under the control of the stakeholder(ie, the surgeon); (2) validity, which assesses the indicatoras a measure of quality; and (3) usefulness, which de-termines whether the stakeholders would find data on theindicator informative. Each criterion was ranked on aseven-point Likert scale (1 ¼ disagree, 7 ¼ agree).Respondents were also given the opportunity to pro-

vide written comments and were asked to suggest addi-tional indicators. E-mail reminders were sent 2 weeksafter the initial distribution, with follow-up phone calls toExpert Panel members a week later.Questionnaire responses were analyzed and results

organized according to whether there was consensus forinclusion (>51% of respondents ranked all three criteria5, 6, or 7 on the Likert scale), consensus for exclusion(>51% of respondents ranked all three criteria 1, 2, 3, or 4on the Likert scale), or unclear consensus (indicators thatdid not meet the criteria for inclusion or exclusion).

Fig 2. Literature search to identify qualityindicators for surgical procedures for patientswith lung cancer.

3Ann Thorac Surg DARLING ET AL2014;-:-–- LUNG CANCER OPERATION QUALITY INDICATORS

Round 2In round 2, the indicators were organized according tothose for which there was consensus for inclusion,consensus for exclusion, or unclear consensus based onthe results of round 1. Additional indicators suggested inthe first round were included for rating. Information onthe frequency of responses and comments from round 1were also included. Panel members were asked to rate theoriginal indicators on two additional criteria: (1) dis-criminability, which refers to whether the indicator showsvariability in practice and (2) feasibility, which refers towhether the indicator is feasible to measure. Respondentscould answer either “yes”, “no,” or “unsure” to thesecriteria. The additional indicators suggested in round 1were rated by all five criteria (actionability, validity, use-fulness, discriminability, feasibility).

The questionnaire was distributed by e-mail exclu-sively to the 13 Expert Panel members (excluding thethree physician leaders), followed by a reminder e-mail 2weeks later.

In a manner similar to round 1, responses wereanalyzed, and frequencies were calculated and catego-rized as clear consensus for inclusion, clear consensus forexclusion, or unclear consensus. On the basis of theseresults, the indicators were separated into the threegroups for discussion at the Expert Panel meeting.

Round 3Panel members were invited to an in-person Expert Panelmeeting, facilitated by two of the physician leaders (GD,JD). It was anticipated that after two rounds, there wouldbe fewer potential indicators to discuss. However, therewas actually an increase from 41 original indicators to 54for review because of additional indicator suggestionsand only one indicator having clear consensus for exclu-sion after rounds 1 and 2.

The indicators were divided into seven categories ofcare for discussion at the in-person meeting. The goal was

to have approximately 10 to 15 indicators, with one tothree indicators from each of the seven categories. TheExpert Panel members discussed each potential indicatorand by consensus eliminated some indicators, leavingbetween one and five potential indicators in each cate-gory. Modifications, such as wording changes andcombining of indicators, were also made. Discussionresulted in the addition of three more indicators to thelist.At the end of the discussion period, the selected in-

dicators were included in the final (third) questionnaire,which was completed by all Expert Panel members whoattended the in-person meeting. They were asked to rankthe top three indicators in each category that werethought to be the best measures of quality of lung cancersurgical procedures. The top indicator in each categorywas assigned three points, the second was assigned twopoints, and the third was assigned one point. If there werefewer than three indicators in a category, all indicatorswere ranked.The indicator in each category with the most points was

automatically selected for inclusion. The rest of the in-dicators were selected by consensus of the physicianleaders. After the in-person meeting, the physicianleaders made several modifications to the chosen in-dicators, which included the addition of one indicator, theomission of one indicator, and wording modifications toseveral indicators to more accurately reflect the evidence.The evidence for each indicator was also reviewed again.The final list was e-mailed to all Expert Panel membersfor their review and approval.

Results

The development process began in October 2008 andconcluded in December 2011. The Expert Panel includednine surgeons, two radiation oncologists, a medicaloncologist, a respirologist, a radiologist, a nuclear

4 DARLING ET AL Ann Thorac SurgLUNG CANCER OPERATION QUALITY INDICATORS 2014;-:-–-

medicine physician, and a regional vice president ofCCO, representing 11 of 13 regions in Ontario with adesignated thoracic surgery center. As shown in Figure 2,592 articles were identified in the literature search, ofwhich 263 full-text articles were reviewed and 25 articleswere used to identify 41 unique potential LCSQI.

Round 1The response rate for the first-round questionnaire was100% for the Expert Panel and 33% (11/33) for thoracicsurgeons who were not on the panel. One questionnairewas excluded because of missing fields. There was clearconsensus among respondents that all indicators inquestionnaire No. 1 were important, and an additional 13indicators were suggested.

Round 2The second questionnaire contained 54 quality indicators(41 original plus 13 additional indicators suggested fromround 1). The response rate for the second questionnairewas 85%. After round 2, one indicator was excluded, 42indicators had clear consensus for inclusion, and 11 in-dicators had unclear consensus.

Round 3Eleven Expert Panel members (including two of the threeleaders) and the surgical oncology program staff attendedthe meeting. All 54 potential indicators and the accom-panying results of rounds 1 and 2 were presented. Afterthe discussions and by consensus from the Expert Panelmembers, 34 indicators were eliminated. The remaining20 LCSQI across the continuum of care were deemedgood measures of quality and were included in ques-tionnaire No. 3 (Preoperative assessment, 3; Staging, 4;Surgical Procedures, 5; Pathology, 2; Adjuvant Therapy, 1;Surgical Outcomes, 4; Miscellaneous, 1). Seventeen ofthese indicators had been ranked in the rounds 1 and 2questionnaires, and the other three were suggested dur-ing discussion at the in-person meeting.

The results of questionnaire No. 3 were tallied ac-cording to the previously described ranking system, and apreliminary list of quality indicators was chosen, whichwas then modified by the physician leaders. In addition tominor wording changes to several indicators, the physi-cian leaders eliminated one indicator from the pre-liminary list (Proportion of resected stage I and stage IINSCLC patients older than 75 years who receive ananatomic resection). They also added an indicator thathad not been previously discussed (Proportion of patientsscheduled to have an elective lung cancer resection that iscanceled). After these modifications were made, 17LCSQI remained. This final list was distributed to themembers of the Expert Panel for their review, and therewas unanimous agreement, resulting in the final list of 17LCSQI (Table 1).

Comment

Performance measurement is important for health careorganizations interested in assessing and reporting on the

quality of services. Valid, clinically relevant, and sensitiveindicators are essential to this process. Quality indicatorsare measurable elements of practice performance forwhich there is evidence or consensus that they can beused to assess the quality of care [23]. Structural, process,or outcome quality indicators may be chosen, dependingon the intended stakeholders. Thus, in the developmentof indicators, the purpose of the indicators should beexplicitly stated, and the indicators should have facevalidity to the stakeholders. Indicators should be sup-ported by evidence. Depending on the strength of theevidence, expert consensus may be required. Measure-ment of the indicators must be feasible and actionableunder the control of the stakeholders.We chose to use a modified Delphi process rather than

a consensus conference because it avoids domination ofthe group by thought leaders. It ensures anonymityamong respondents, allowing them more freedom intheir responses and opinions [24]. Last, experts can besurveyed several times regardless of their geographicallocation or time constraints [24]. CCO has used thismethod previously to develop quality indicators for sur-gical procedures for colorectal cancer [25] and ovariancancer [26].Although the intention was to follow a modified Delphi

process, our indicator development process variedslightly because we had an increase in the number ofpotential indicators after two rounds of questionnaires(only one potential indicator was excluded, and 13 morewere added), whereas usually indicators would be elim-inated in the rounds 1 and 2 questionnaires. Thus, in anattempt to decrease the total number of indicators, it wasarbitrarily decided that only one to five indicators wouldbe included in each category in the final list. At the in-person meeting, the potential indicators in each cate-gory were discussed, and by informal agreement, 34indicators were omitted, so 20 indicators remained. Theprocess also differed in that 16 additional potential in-dicators were added to the list of indicators identifiedthrough a review of the literature: 13 were suggested inround 1 and three at the in-person meeting. A formalsystematic review to identify supporting evidence forthese indicators was not conducted. In addition, aftercompletion of the third questionnaire, the Expert Panelleaders made several modifications to the indicator list,which included omitting one indicator deemed to beredundant, adding one indicator that had not initiallybeen identified, and making wording modifications tomore accurately reflect current evidence. The list ofLCSQI with the modifications was distributed to theExpert Panel, and the final list was approved.The final list of 17 LCSQI assesses aspects of care that

are at the discretion of surgeons. Fourteen are processmeasures, and three are outcome measures. Processmeasures tend to be more useful because outcome in-dicators may not be immediately available (eg, 5-yearsurvival) and may be affected by factors other than thehealth care system (eg, patient status, preferences). Asshown in Table 1, 12 of the indicators were identified inthe systematic review; four were suggested by the Expert

Table 1. Final List of NSCLC Operation Quality Indicators and Supporting Evidence

Indicator No. Indicator Definition

Evidence

AdditionalComments

Clinical PracticeGuideline Metaanalysis

SystematicReview

Population-BasedStudy

ExpertConsensus

Preoperative assessment1 Proportion of NSCLC patients who underwent

operations who received a CT scan of thechest and a pulmonary function assessmentpreoperatively

Six [2, 3, 4, 5, 6a, 7] One [8] One [9]

2 Time from date of consultation with the surgeon todate of decision to treat

Not identified inevidence; consensusby Expert Panel

Staging3 Proportion of resectable clinical stage I-IIIA NSCLC

patients who receive preoperative PET-CT scanningFive [4, 16, 5, 7, 6a] One [11]

4 Proportion of NSCLC patients with no extrathoracicmetastases who receive invasive mediastinal staging(excluding those who have a normal CT results andnegative PET-CT results in the mediastinum and aperipheral clinical stage 1A tumor)

Five [7, 5, 12, 6a, 2] One [13] Two [9, 11]

5 Proportion of NSCLC patients who receive biopsyof at least one ipsalateral, one contralateral, andone subcarinal node to include suggestive nodes onpreoperative imaging during invasive mediastinalstaging

One [6a] Expert Panel added“suspicious nodes”to indicator

Operation6 Proportion of patients with resected stage I and

stage II NSCLC who receive an anatomicresection

Five [14, 2, 19, 7, 5] One [15] One [16] Expert Panel separatedout indicator foranatomic resectionfrom all resections

7 Proportion of all patients with stage I and stage IINSCLC who receive a resection

Five [14, 2, 10, 7, 5] One [15] One [16]

8 Proportion of all patients with stage I and stage IINSCLC older than 75 years who receive a resection

Two [2, 3] One [8] Expert Panel changedage from 71 yearsto 75 years

9 Proportion of patients found to be unresectableat the time of operation

One [2]

Pathology10 Proportion of patients with resected NSCLC who

receive adequate lymph node dissection orsampling (>10 lymph nodes removed and atleast 3 mediastinal lymph node stations sampled)at the time of resection

Five [14, 19, 10, 5, 7, 4] Two [17, 18] One [11]

(Continued)

5AnnThorac

Surg

DARLIN

GETAL

2014;-:-

–-

LUNG

CANCER

OPERATIO

NQUALITY

INDIC

ATORS

Table 1. Continued

Indicator No. Indicator Definition

Evidence

AdditionalComments

Clinical PracticeGuideline Metaanalysis

SystematicReview

Population-BasedStudy

ExpertConsensus

11 Proportion of patients with resected NSCLCwith an R0 resection

One [10]

Adjuvant therapy12 Proportion of patients with completely resected

stage II NSCLC who received a consultation witha medical oncologist for consideration of adjuvantchemotherapy

One [19] Three [20,15, 21]

One [22] One [11] Expert Panel changedfrom referral to amedical oncologist toconsultation with amedical oncologist

Surgical outcomes13 Proportion of patients undergoing resection who stay

in an acute care hospital >14 days after anatomicresection

Not identified in evidence;consensus by ExpertPanel

14 5-year overall survival rate for resected patients,by stage

Not identified in evidence;consensus by ExpertPanel

15 30-day reoperation rate after surgical resection Not identified in evidence;consensus by ExpertPanel

Miscellaneous16 Proportion of patients with clinical or pathologic

stage IIIA and stage IIIB NSCLC who receivemultidisciplinary evaluation either by directconsultation or discussion at a multidisciplinarycancer conference

Three [10, 19, 3]

17 Proportion of patients scheduled to have an electivelung cancer resection that is canceled

Not identified in evidence;added by leaders ofExpert Panel

a Guideline based on expert consensus.

CT ¼ computed tomography; NSCLC ¼ non-small cell cancer; PET ¼ positron emission tomography.

6DARLIN

GETAL

AnnThorac

Surg

LUNG

CANCER

OPERATIO

NQUALITY

INDIC

ATORS

2014;-:-

–-

7Ann Thorac Surg DARLING ET AL2014;-:-–- LUNG CANCER OPERATION QUALITY INDICATORS

Panel, either in one of the questionnaires or at the in-person meeting; and one indicator was suggested by theleaders of the Expert Panel after the in-person meeting.All 12 indicators identified in the systematic review weresupported by clinical practice guideline recommenda-tions, metaanalysis, or systematic review, with three ofthose indicators also supported by population-basedstudies. The remaining five indicators identified werebased on Expert Panel consensus, and supporting evi-dence was not assessed. The indicators chosen also haveface validity because nine members of the panel werethoracic surgeons. Although morbidity and mortality aretwo commonly measured outcome indicators, CCOalready collects mortality data, so those two terms werenot included in our list. Morbidity was not included inour indicator list primarily because many factors otherthan the surgeon’s judgment or skill influence morbidity.

These indicators were developed to assess the quality oflung cancer surgical procedures in the province ofOntario,specifically to determine the effect of implementation ofthe Thoracic Surgical Oncology Standards, which region-alized thoracic surgical procedures to 15 centers in theprovince. Currently, data for eight indicators can be ob-tained from the Canadian Insititute for Health Informationdatabase (includes all hospital discharges in Canada withthe exception of Quebec), the CCO ePath database (whichcollects pathology and stage data by means of electronicsynoptic reporting), and throughOntarioHealth InsurancePlan billing claims. These initial data will be analyzed andpresented to the thoracic surgeons in Ontario in aggregateand anonymized form. Individual surgeon data is notcollected. Our immediate goal is to determine currentpractice and acceptable variations and then to determinepotential quality gaps. Previous indicators have beendeveloped to assess the spectrum of care from diagnosis totreatment rather than just treatment related to the opera-tions. Hermens and colleagues [27] developed a set of 15indicators to assess compliancewith a guideline previouslydeveloped by their group. Four indicators are similar tothose developed by our group. Reifel [28] reported on a setof indicators developed by Research and Devlopment(RAND) Corporation in 2000. Indicators were againdeveloped to assess the quality of care across the spectrumand for both NSCLC and small cell lung cancer. Some in-dicators are similar to ours, including the proportion ofpatients who have pulmonary function assessment and anelectrocardiogram, and the proportion of stage I and IINSCLC patients who undergo surgical procedures.

Quality indicators are important for evaluating thequality of health care delivery. In thoracic operations,historically the prime indicators used are in-hospitalmortality, length of stay, and 5-year survival. Althoughthese are important, those responsible for health caredelivery require more detailed and timely measures tounderstand the quality of care provided. By use of amodified Delphi process, 17 quality indicators related toprocesses of care and outcome measures were identifiedfor evaluating the quality of thoracic surgical care inOntario. This process ensured that the LCSQI have facevalidity, and it also promotes the quality agenda in a

collegial collaboration among surgeons, other health careproviders, and administrators.

The authors thank the members of the Lung Cancer SurgeryExpert Panel, Drs Paul Chiasson, Bill Evans, Marisa Finlay, KenGehman, Karen Gulenchyn, Matthew Kilmurry, Donna Maziak,Ken Reid, Shafeequr Salahudeen, Michael Sanatani, Julius Toth,Yee Ung, and John Vlasschaert, for their help in developing anddetermining the final list of Lung Cancer Surgery QualityIndicators.

References

1. Fink A, Kosecoff J, Chassin M, Brook RH. Consensusmethods: characteristics and guidelines for use. Am J PublicHealth 1984;74:979–83.

2. British Thoracic Society and Society of Cardiothoracic Sur-geons of Great Britain and Ireland Working Party. Guide-lines on the selection of patients with lung cancer forsurgery. Thorax 2001;56:89–108.

3. Collice GL, Shafazand S, Griffin JP, et al. Physiologic eval-uation of the patient with lung cancer being considered forresectional surgery. Chest 2007;132:161S–77S.

4. Silvestri G, Gould MK, Margolis ML, et al. Noninvasivestaging of non-small cell lung cancer. Chest 2007;132:178S–201S.

5. National Comprehensive Cancer Network. NCCN clinicalpractice guidelines in oncology: non-small cell lung cancer.2009.

6. De Leyn P, Lardinois D, Van Schil P, et al. European trendsin preoperative and intraoperative nodal staging: ESTSguidelines. J Thorac Oncol 2007;2:357–61.

7. Scottish Intercollegiate Guidelines Network. Managementof patients with lung cancer: A national clinical guideline.2005.

8. Jaklitsch M, Mery C, Audisio R. The use of surgery to treatlung cancer in elderly patients. Lancet Oncol 2003;4:463–71.

9. American Thoracic Society/European Respiratory Society.Pretreatment evaluation of non-small cell lung cancer. Am JRespir Crit Care Med 1997;156:320–32.

10. National Institute for Clinical Excellence. Lung cancer: thediagnosis and treatment of lung cancer. Clinical Guideline24. 2005.

11. D’Addario G, Felip E. Non-small-cell lung cancer: ESMOclinical recommendations for diagnosis, treatment andfollow-up. Ann Oncol 2008;19(Suppl 2):ii39–40.

12. Detterbeck FC, Jantz MA, Wallace M, Vansteenkiste J,Silvestri GA. Invasive mediastinal staging of lung cancer.Chest 2007;132:202S–20S.

13. Kramer H, Groen H. Current concepts in the mediastinallymph node staging of non-small cell lung cancer. Ann Surg2003;238:180–8.

14. Scott WJ, Howington J, Feigenberg S, Movsas B, Pisters KI.Treatment of non-small cell lung cancer stage I and stage II.Chest 2007;132:234S–42S.

15. Meert AP, Sculier JP. What has the meta-analysis contributedto today’s standard of care in the treatment of thoracic ma-lignancies? Lung Cancer 2008;61:141–51.

16. Mery C, Pappas A, Bueno R, et al. Similar long-term survivalof elderly patients with non-small cell lung cancer treatedwith lobectomy or wedge resection within the surveillance,epidemiology, and end-results database. Chest 2005;128:237–45.

17. Wright G, Manser RL, Brynes G, Hart D, Campbell DA.Surgery for non-small cell lung cancer: systematic reviewand meta-analysis of randomized controlled trials. Thorax2006;61:597–603.

18. Alam N, Flores RM. Video-assisted thoracic surgery (VATS)lobectomy: the evidence base. JSLS 2007;11:368–74.

8 DARLING ET AL Ann Thorac SurgLUNG CANCER OPERATION QUALITY INDICATORS 2014;-:-–-

19. Robinson L, Ruckdeschel J, Wagner H Jr, Stevens C. Treat-ment of non-small cell lung cancer – stage IIIA. Chest2007;132:243S–65S.

20. Berghmans T, Paesmans M, Meert AP, et al. Survivalimprovement in resectable non-small cell lung cancer with(neo)adjuvant chemotherapy: results of a meta-analysis ofthe literature. Lung Cancer 2005;49:13–23.

21. Nakamura H, Kawasaki N, Taguchi M, Kabasawa K.Survival following lobectomy vs limited resection forstage I lung cancer: a meta analysis. Br J Cancer 2005;92:1033–7.

22. Sedrakyan A, van Der Meulen J, O’Byrne K, Prendiville J,Hill J, Treasure T. Postoperative chemotherapy for non-smallcell lung cancer: a systematic review and meta-analysis.J Thorac Cardiovasc Surg 2004;128:4149.

23. Donabedian A. The quality of care. How can it be assessed?JAMA 1988;260:1743–8.

24. Campbell SM, Braspenning J, Hutchinson A, Marshall M.Research methods used in developing and applying qualityindicators in primary care. BMJ 2003;326:816–9.

25. Gagliardi A, Simunovic M, Langer B, Stern H, Brown A.Development of quality indicators for colorectal cancer sur-gery, using a 3-step modified Delphi approach. Can J Surg2005;48:441–52.

26. Gagliardi A, Fung MF, Langer B, Stern H, Brown A. Devel-opment of ovarian cancer surgery quality indicators using amodified Delphi approach. Gynecol Oncol 2005;97:446–56.

27. Hermens RP, Ouwens MM, Vonk-Okhuijsen SY, et al.Development of quality indicators for diagnosis and treat-ment of patients with non-small cell lung cancer: a first steptoward implementing a multidisciplinary, evidence-basedguideline. Lung Cancer 2006;54:117–24.

28. Reifel J. Quality of care for oncologic conditions and HIV:lung cancer. RAND Monograph Report. Chapter 7:133–71.