A systematic review of Health Technology Assessment tools in sub

Kriza et al. Health Research Policy and Systems 2014, 12:66http://www.health-policy-systems.com/content/12/1/66

REVIEW Open Access

A systematic review of Health TechnologyAssessment tools in sub-Saharan Africa:methodological issues and implicationsChristine Kriza1*, Jill Hanass-Hancock2, Emmanuel Ankrah Odame3, Nicola Deghaye2, Rashid Aman4,Philip Wahlster1, Mayra Marin1, Nicodemus Gebe5, Willis Akhwale6, Isabelle Wachsmuth7

and Peter L Kolominsky-Rabas1

Abstract

Background: Health technology assessment (HTA) is mostly used in the context of high- and middle-incomecountries. Many “resource-poor” settings, which have the greatest need for critical assessment of health technology,have a limited basis for making evidence-based choices. This can lead to inappropriate use of technologies, a problemthat could be addressed by HTA that enables the efficient use of resources, which is especially crucial in such settings.There is a lack of clarity about which HTA tools should be used in these settings. This research aims to provide anoverview of proposed HTA tools for “resource-poor” settings with a specific focus on sub-Saharan Africa (SSA).

Methodology: A systematic review was conducted using basic steps from the PRISMA guidelines. Studies thatdescribed HTA tools applicable for “resource-limited” settings were identified and critically appraised. Only paperspublished between 2003 and 2013 were included. The identified tools were assessed according to a checklist withmethodological criteria.

Results: Six appropriate tools that are applicable in the SSA setting and cover methodological robustness and easeof use were included in the review. Several tools fulfil these criteria, such as the KNOW ESSENTIALS tool, Mini-HTA tool,and Multi-Criteria Decision Analysis but their application in the SSA context remains limited. The WHO CHOICE methodis a standardized decision making tool for choosing interventions but is limited to their cost-effectiveness. Mostevaluation of health technology in SSA focuses on priority setting. There is a lack of HTA tools that can beused for the systematic assessment of technology in the SSA context.

Conclusions: An appropriate HTA tool for “resource-constrained” settings, and especially SSA, should addressall important criteria of decision making. By combining the two most promising tools, KNOW ESSENTIALS andMulti-Criteria Decision Analysis, appropriate analysis of evidence with a robust and flexible methodology couldbe applied for the SSA setting.

Keywords: Health services, Health systems, Health systems research

* Correspondence: [email protected] Centre for Health Technology Assessment (HTA) and PublicHealth (IZPH), University of Erlangen-Nürnberg, Schwabachanlage 6, Erlangen91054, GermanyFull list of author information is available at the end of the article

© 2014 Kriza et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly credited. The Creative Commons Public DomainDedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article,unless otherwise stated.

mailto:[email protected]

http://creativecommons.org/licenses/by/4.0

http://creativecommons.org/publicdomain/zero/1.0/

Kriza et al. Health Research Policy and Systems 2014, 12:66 Page 2 of 13http://www.health-policy-systems.com/content/12/1/66

BackgroundHealth technology assessment (HTA) is a multidisciplin-ary field of policy analysis that examines the medical,economic, social, and ethical implications of the incre-mental value, diffusion, and use of a medical technologyin health care [1]. Currently, HTA is mostly used in thecontext of developed countries. Several methodologiesexist for “resource-poor” settings but implementation ofHTA and transparent use in most African countries isstill limited [2]. In addition, some methodological as-pects of HTA do not fit into the setting of developingcountries and need to be adapted appropriately accordingto specific needs [2]. Especially in “resource-constrained”settings, the need for the systematic evaluation of healthtechnology and of the available alternative technologieshas never been greater [3,4]. HTA is performed in orderto improve the quality of health care and ensure goodvalue for money investments in any setting. It is becauseof this that HTA should form the basis for health technol-ogy policies especially in “resource-poor” settings withlimited health sector budgets.The current lack of HTA in sub-Sahara Africa (SSA)

can be attributed to the lack of capacity to undertakeHTA. Even though countries like South Africa currentlyemploy HTA to a small degree and have begun theprocess of forming a HTA mechanism, the lack ofcapacity coupled with a weak health system capacityto implement interventions are contributing to wide-spread implementation of HTA. Another problematicissue is the limitation of high-quality data availabilityand lack of research evidence, especially in the con-text of “resource-limited” health systems like the assess-ment of health states [5,6]. This reduces the ability for“resource-poor” settings to implement rigorous HTApractices. In addition, “resource-limited” countries havefew resources to support HTA, which then underminesthe ability of HTA to utilize appropriate incentives (thatwould promote more efficient management of resources.Incentives for HTA in resource-limited settings:

What are the incentives for resource-limited coun-tries to develop HTA agencies?1 – Providing international guidelines and model

essential medicines lists as well guidance on goodand ethical governance practices specifically in li-aison with ethic review committee and bioethicscommittee

Low-income countries with very limited capacity andcountries that face major challenges of lack of transparencyin decision-making are likely to face difficulty implement-ing HTA for medicines effectively. Use of internationalguidelines and model essential medicines lists can helpsuch countries to incorporate HTA in their policies.

2 – Complement HTA with evidence-based guidelinesPolicies to encourage doctors to prescribe formulary

medicines and follow evidence-based guidelines are neededto complement HTA.

3 – Strengthening capacity in HTA before to set-upHTA agencies through national capacity buildingworkshops specifically for decision makers andhealth professionals

Capacity for HTA in resource limited settings shouldbe established early and supported; prerequisites andbarriers are extensive but not insurmountable and mustbe considered as HTA processes are developed.Decisions in many “resource-poor” settings can easily

be influenced by past experience without an evidencebase and by preferences of donor agencies and lobbyingpressure for new technologies, for example from commer-cial organisations or global funding and donor organiza-tions [2]. This can lead to the use of technologies whichdo not address health needs and in effect contribute to theinefficient use of resources [7]. Decisions made in thiscontext often do not reflect local values and evidencebased local information on clinical and cost effectiveness[2]. As a result of this, many resources are disproportion-ately allocated or wasted. The need for HTA becomesmore prominent when the need for collective decisionmaking for the good of the whole is necessary. It alsobecomes apparent that HTA in “resource-poor” settingscannot be addressed the same as it is in high-incomecountries [8]. Equity and equality considerations are differ-ent from these in developed countries. These includemore important social issues like poverty reduction [9].There is a clear need for HTA use in “resource-limited”

settings, and SSA countries in particular, as it is thesecountries especially that cannot afford to waste resources.Not only would HTA highlight health technologies thatwould be too costly relative to their benefits, but wouldalso identify potentially harmful and ineffective technolo-gies. In light of a higher percentage of insured patientsstemming from advanced economic development and thepotential to purchase more and more complex, and ex-pensive, health technologies, the introduction of HTA intopublic health policies becomes more evident. The WHOresolution on “Health Intervention and TechnologyAssessment in Support of Universal Health Coverage”passed on 24 May 2014 by the World Health Assemblyprovides an important mandate for SSA countries to ac-celerate their HTA efforts.

ObjectivesThe research objectives of this systematic review are toprovide an overview of HTA tools used in “resource-constrained” settings, with a specific focus on the


assessment on SSA. It was chosen as the focus settingbecause it is the region with the least covered area byHTA practices and also the most “resource-constrained”setting. Another objective is to determine how many ofthe appropriate HTA tools identified in this review wereapplied for the assessment of medical devices.

MethodsSearch methodThe PRISMA guidelines for conducting systematic re-views were followed [10]. Within the study the followingdatabases were searched: PubMed, Science Direct, Scopus,Ebscoh, and EconLit alongside the journals Health Policyand Planning, Cost-Effectiveness and Resource Allocation,and the WHO Bulletin (see Appendix for detailed searchstring). To ensure optimal coverage, additional articleswere found within the reference section of retrieved arti-cles and through citation snowballing by undertakingwider searches by author name for those appearing as keypublishers in the area. Additionally, the web pages of theWHO and the World Bank were searched manually. Thesearch was limited to articles published within the 2003and 2013 timeframe and excluded non-empirical studiesor those that did not focus on a HTA tool.

Selection of manuscripts and data extractionArticles that met the inclusion criteria of an evidence-based HTA tool appropriate for the use in the SSAcontext were retrieved and examined more closely. Thequality of research papers was evaluated according toadequate description of the theoretical framework, back-ground, and methodology [11].For those papers that fulfilled the criteria for quality,

data was extracted according to the following content:date published, study funding source, possible conflictsof interest, study objectives, target population, applica-tion of tool, site/setting, study focus, HTA tool proposedor approach used in the paper, description of tool or ap-proach, stand alone or support tool, aspects of clinical ef-fectiveness, costs and contextual issues addressed by toolor approach, all stakeholders involved, literature search in-corporated, results of implementation, and focus on med-ical technology/intervention.Each study was described by addressing the criteria in

the data extraction form (Table 1).The HTA tool or approach found in each study were

critically appraised using a second data extraction form(Table 2) based on proposed criteria for the assessmentof HTA activities [12]. These criteria are formed by a seriesof 15 principles, as described by Drummond et al. [12],which cover the structure of HTA programs, methods ofHTA, processes for conduct of HTA, and use of HTAs indecision making. This set of principles was utilized to en-sure robustness of the included HTA approaches and

tools. The extraction form prompted i) the context andapplicability of the described tool (geographical focus, tar-get setting, reasons for use, degree of needed training, andexperts); ii) measures for sensitivity and effectiveness (clin-ical effectiveness, cost effectiveness, context sensitivity);iii) the approach to HTA assessment (type of tool, inclu-sion of full social perspective, use of available evidence,transparency, generalizability, result focus, ling to decisionmaking), and iv) if the described tool was piloted for med-ical devices.Criteria addressing areas particular to SSA were in-

cluded in the evaluation to assess how adaptable the toolor approach would be for that setting. The criteria in-cluded the following: ease of training, flexibility of evidencerequirement, economical standing, and local context con-sideration (Table 3). These criteria have been elaborated inconsensus with our Research Consortium Members in-cluding representatives from SSA countries.

ResultsThe search retrieved 1,073 papers in total, of which six ful-filled the inclusion criteria (PRISMA flowchart, Figure 1).Data was extracted from the six papers published between2003 and 2013.HTA tools overview:

� WHO CHOICE cost-effectiveness analysis tool [13]� KNOW ESSENTIALS tool using thirteen criteria

related to context-specific HTA and prioritization ofthese criteria [8]

� Weighting of Multi-Criteria Decision Analysis criteriaon the basis of corresponding HTA report [14]

� Adaptation of Mini-HTA or hospital-based HTAtool for the decision-making related to medical de-vices purchased in a hospital setting [15]

� Equity-oriented toolkit for HTA, containing fourelements: burden of illness, community effectiveness,economic evaluation and knowledge translation [16]

� Mapping system using Analytic Hierarch Process(AHP) methods between different diseases and theirmatching technologies to minimize technologyunderutilization [17]

None of the papers were funded or stated funding byother sources. Four studies proposed novel tools forHTA [8,15-17] and the other two used current HTA ap-proaches originally developed for high-income settings[13,14]. One study specified their target population to bein SSA countries [15], the others focused on “resource-poor” settings [13,14,16,17] and settings without formalHTA [8]. However, all studies included in the review canbe applied in the SSA setting. From the proposed tools,two were stand-alone [8,16] and the other two were sup-port tools [15,17]. The support tools served as decision

Table 1 Extraction form for study characteristics

Author(s) Mathew Abaza and Tawfik Hutubessy et al. Miot et al. Govender et al. Ueffing et al.

Study name KNOW ESSENTIALS:A tool for informeddecisions in the absenceof formal HTA systems

Appropriate medicaltechnologies fordeveloping countries:application tocardiovascular disorders

Generalizedcost-effectivenessanalysis fornational-levelpriority-setting inthe health sector

Field testing of amulticriteria decisionanalysis (MCDA)framework forcoverage of ascreening test forcervical cancer inSouth Africa

Purchasing ofmedical equipmentin public hospitals:the mini-HTA tool

Equity-orientedtoolkit for healthtechnologyassessment andknowledgetranslation:application toscaling up oftraining andeducation forhealth workers

Published date 2011 2008 2003 2012 2011 2009

Study funding source(s) None Not stated Not stated Not stated Not stated Not stated

Possible conflictsof interest

None Not stated None None None Three authorsexpressedcompetinginterests becauseof their affiliationwith the WHO

Objective Describes a tool,“KNOW ESSENTIALS”that includes currentbest evidence onhealth technologies,incorporates relevantcontextual issues, and isobjective, reproducible,transparent, andaffordable

Provide an acquisitionmethodology by whichhealthcare providers canminimize the underutilizationof medical devices to bepurchased when certaindiseases are to be dealtwith and allow non-technicalpersonnel to make correctand appropriate acquisitions

Outline the processby which country leveldecision makers andprogramme managerscan carry out their owncontext-specific analysisof the relative cost-effectivenessof interventions for reducingleading causes of nationaldisease burden usingcost-effective analysis(CEA) information fromthe WHO-CHOICE project

Field testing of theEVIDEM frameworkfor decision-makingon a screening testby a private healthplan in South Africa

Adapt and use theDanish Centre forEvaluation andHealth TechnologyAssessment (DACEHTA)mini-HTA tool toassess pastdecisions made bySouth African hospitalmanagers, as appliedto selected medicaldevices

Propose a toolkitfor decision-makersto scale up trainingand education ofhealth workers

Target population Settings lacking formalHTA systems

Developing countries Low-income settings Low-resource settings South African hospitals Economicallydisadvantagedareas

Was tool/approachapplied?

Yes, pilot tested Yes, pilot tested No Yes Yes No

Site/setting Africa (hypothetical) Not stated n/a South Africa South Africa n/a

Study focus Favourability ofartemisinin-basedtreatment for severeor complicated malariain children usingKNOW ESSENTIALS

Cardiovascular disorderequipment purchases withsupport database software

n/a Cervical cancerscreeningdecision-making

Decision supportchecklist for hospitalmanagers to informdecisions about theacquisition of healthtechnologies (drugs,devices, and otherhealth interventions)

n/a

Krizaet

al.Health

ResearchPolicy

andSystem

s2014,12:66

Page4of

13http://w

ww.health-policy-system

s.com/content/12/1/66

Table 1 Extraction form for study characteristics (Continued)

Expressed need fortool or approach

Lack of formal HTA orlow level of applicationin developing countries;decisions are highlysubjective and expertbased rather thanresearch based

Lack of HTA and therecognition for the needof HTA in developingcountries

Shortage of technical expertiseand health service capacity toutilize CEA information

Need for transparencyand greater access toevidence througha systematic andexplicit process

Existence ofmanagementinformation gaps inSouth African publichospitals and needfor a customized toolto support decisionmakers in medicaldevice management

Need to address theshortage of healthworkers whichare consideredpart of health careresources

HTA tool proposed anddesigned to be appliedfor specific medical technologies

KNOW ESSENTIALS Decision supportdatabase software

n/a n/a Mini-HTA tool/hospital-based HTAtool

Equity-OrientedToolkit (EOT)

HTA approach used in the paper(if new tool not proposed)

n/a n/a Use of CEA informationfrom the WHO-CHOICEproject, generalized CEA

Use of the EVIDEM(Evidence and Value:Impact on DEcisionMaking) framework,brings together HTAand MCDA

n/a n/a

Applied in the context ofmedical devices?

No Yes No Yes Yes No

Description of tool or approach Elements addressingdifferent aspects ofHTA divided intobackground issues(KN, O, W), essentialcriteria (E, S, S, E),and other criteria(N, T, I, A, L, S). Criticalappraisal through asystematic reviewprocess withmeta-analysis, orusing other clearlydefined searchstrategies withjustification. Colourcoding of elementswhen availableinformation favours(green)/does not favour(red) the medicaltechnology, or (yellow)if available informationis insufficient to classifygreen or red. For lastsix elements, notapplicable (white)code available. If anyBackground Issuescoded red, health

Database constructedby compiling data aboutcardiovascular diseaseequipment specifications.Database comprised ofthree main forms. Firstform enables user toselect criteria on thedisease, the brand,equipment type, andnon-diagnostic features.Second form lists relevantequipment according toselected criteria. Equipmentis ordered by a priorityscheme depending ontotal number of diagnosticfeatures, number of unuseddiagnostic features, andnumber of diagnosticfeatures for arrhythmiaslabelled and highlightsless recommendedequipment based onthese features. The thirdform allows user toexamine selectedequipment moreclosely

Generalized CEA identifiescurrent allocative inefficienciesas well as opportunitiespresented by new interventionsand presents it in a way thatcan be translated acrosssettings by i) evaluating thecosts and health benefits of aset of related interventions,singly and in combination, withthe “null scenario”; ii) using CEAresults to classify interventionsinto those that are verycost-effective, cost-ineffective,and somewhere in betweenrather than using the traditionalleague table approach

After a literaturereview and inputform a clinicalcommittee of thehealth plan, a HTAreport on liquidbased cytology (LBC)for cervical cancerscreening wastailored to investigate14 MCDA inclusioncriteria and fourcontextual criteria(appraisedqualitatively)proposed byhealthcare funder.The contents ofreport were tailoredto local context.Committee engagedin workshops wheremembers assignedweights to eachcriterion of the MCDAmatrix and scores forLBC for each criterionof the MCDA matrixbased on the data ofthe HTA report.

Adaptation of theDACEHTA tool, whichis separated into thefollowing sections:introduction,technology, patient,organization, economy.The tool was adaptedinto the followingcluster: patients,technology, economy,and organizationalinfluence. The tool wasused as a prospectivecross-sectional surveyconcerning thedecision-makingprocess of purchasingmedical devices overthe past year, adminis-tered to 21 hospitalmanagers

Adaptation of theWHO’s Needs-BasedToolkit for HealthTechnology Assessmentthat was created toaid health policymakers and plannersto allocate resourcesefficiently, fairly, andeffectively. Aperspective of “equity”was added in the EOT,based on clinical andpopulation healthstatus. Four majorsteps: burden ofillness, communityeffectiveness,economic evaluation,and knowledgetranslation andimplementation.Recommendationswere given for scalingup education andtraining of healthworkers.

Krizaet

al.Health

ResearchPolicy

andSystem

s2014,12:66

Page5of

13http://w



Table 1 Extraction form for study characteristics (Continued)

technology is rejected,if not it proceeds toEssential Criteria. Ifany coded red, healthtechnology is rejected,if not proceeds toOther Criteria. Ifmajority red, healthtechnology is rejected,considered favourablyif mostly green,deferred if mostlyyellow

Members thenassigned qualitativeimpact of system-related criteria on theappraisal. Adoptabilityand utility of frame-work were exploredthrough a post-testing survey

Stand-alone tool or a supporttool for existing decisionmaking process?

Stand alone Support Support Stand alone Support Stand alone

Results of implementation Atemisinin-based treatmentfor severe or complicatedmalaria in children shouldbe incorporated as thefirst-line treatment in theNational guideline.(hypothetical)

Concluded that this toolwould save effort fromtechnical personnel and isfriendly enough to be usedby non-technical personnel.It would also be a helpfultool for the determination ofbudget and other non-diagnostic criteria.

n/a Resulted in aconsideration by thehealth plan to onlyfund for LBC up tothe value ofconventional papsmears. A negotiationprocess was startedwith the pathologylaboratories and thefee for LBC wasreduced to anamount which wasconsideredappropriate for fullfunding; 50% ofmembers felt thatEVIDEM improvedunderstanding of theintervention, accessto quality assessmentof the evidence onthe intervention, andconsideration of allkey elements of thedecision; 56% felt itimprovedtransparency ofdecision making. Nomember thought itworse than existingprocess

Study results showeddeficiencies formedical technology:no sufficientconsideration of risksrelated to a medicaltechnology or on theimpact on staff orcosts

n/a

Krizaet

al.Health

ResearchPolicy

andSystem

s2014,12:66

Page6of

13http://w



Table 2 Extraction form for principles of HTA activities according to Drummond et al. [12]


Study name KNOW ESSENTIALS: A tool forinformed decisions in theabsence of formal HTA systems

Appropriate medicaltechnologies for developingcountries: application tocardiovascular disorders

Generalized cost-effectivenessanalysis for national-levelpriority-setting in the healthsector

Field testing of a multicriteriadecision analysis (MCDA)framework for coverage of ascreening test for cervical cancerin South Africa

Purchasing of medicalequipment in publichospitals: the mini-HTA tool

Equity-oriented toolkit for healthtechnology assessment andknowledge translation: applicationto scaling up of training andeducation for health workers

Structure of HTA Programs

Are the goal andscope of the HTAexplicit andrelevant to its use?

Yes, outcomes of interest areclearly defined at thebeginning of process (nodetailed scoping documentmentioned)

Yes, deals with specificdisease and all relevanthealth technologies(medical devices)

Yes, cost effectiveness analysisof specific health technology

Yes, HTA report clearly outlinesthe purpose

Yes, the form clearly asks todefine medical technologyand scope of proposal

Yes, criteria and requirementsare clearly defined

Is it unbiased andtransparent?

Yes, evidence based and stepshave clear criteria (noindependent party conductingHTA is mentioned)

Yes, systematic andevidence-basedsoftware

Yes, evidence based andsystematic approach

Yes, evidence based andpriorities of stakeholders areclearly defined and addressed

No, the stakeholdersworking on the form canmake subjectiveassessments

Yes, evidence based anddifferent stakeholders areinvolved

Does it include allrelevanttechnologies?

Yes, takes into accountalternatives

Yes, all medical devicesdealing with diseaseand diagnosticprocedure of interestare included indatabase

Yes, the WHO-CHOICE projectused includes an extensivedatabase of evidence

Yes, considers alternatives Yes, considersalternatives

Not stated

Does a clear systemfor setting prioritiesexist?

Not stated, does not mentionpriority setting prior to theimplementation of tool

Yes, systematic search bysoftware of medicaldevices most relevant tostakeholder’s preferences

Not stated, does not mentionpriority setting prior to theimplementation of approach

Yes, weights were assigned tocriteria of the framework bystakeholders

Not stated, does notmention priority settingprior to theimplementation ofapproach

Yes, includes concepts ofneeds assessment and prioritysetting

Methods of HTA

Does it incorporateappropriatemethods forassessing costs andbenefits?

Yes, assesses available evidenceto determine costs of technologyand providing the technologyand its cost-effectiveness. Alsoassesses effectiveness and safetyof technology

No, does not assesscost-effectiveness ofhealth technologies atthis moment

Yes, tool determines cost-effectiveness and assesses thebenefits and drawbacks ofimplementing or not implementingthe health technology along withcombinations of health technologies

Yes, HTA report assesses theeconomics and various benefitsof the intervention

Yes, addresses costs fordifferent stakeholdersand assesses the risksand benefits of thehealth technology

Yes, in its economic evaluation itassesses the benefits and costsas well as the trade-offs be-tween equity and efficiency

Does it consider awide range ofevidence andoutcomes?

Yes, considers availableevidence and long-term out-comes of using or rejecting thehealth technology

Yes, it incorporates allavailable informationabout the medicaldevices

Yes, with the use of theWHO-CHOICE projectdatabases for evidence andone of its tools, PopMod, foranalysing outcomes of usingand rejecting healthtechnology (among otherscenarios)

Yes, a thorough search forevidence in different databasesand other sources is undertakenfor each criterion

Yes, a search and qualityassessment is undertakenof the available literature

Yes, uses a strong evidencebase

Is a full societalperspectiveconsidered?

Yes, takes into account socialissues and interests of differentstakeholders

Not stated Not stated Not stated Yes, takes intoconsideration effects ofproposal on otherdepartments in the hospitaland the cooperation withother hospitals

Yes, includes both societal andindividual determinants

Krizaet

al.Health

ResearchPolicy

andSystem

s2014,12:66

Page7of

13http://w



Table 2 Extraction form for principles of HTA activities according to Drummond et al. [12] (Continued)

Does it explicitlycharacterizeuncertaintysurroundingestimates?

Not stated Not stated Yes, with the use Monte CarloLeague software, an analyticaltool, to find the uncertaintyaround point estimates

Yes, quality of evidence isassessed in the HTA reportportion of approach

Yes, the person filling outthe form notes theuncertainties that applyto the calculations

Not stated

Does it considerand address issuesof generalizabilityand transferability?

Yes, generalizability andtransferability of evidence fromsimilar cohorts needs to bejustified

Not stated Yes, the WHO-CHOICE projectuses international dollars to beable to make meaningful com-parisons and adjustments ac-cording to practice settings aremade to resulting estimates ofgeneralized CEA

Yes, local costs were used whenassessing cost-effectiveness to im-prove transferability

Not stated Yes, considers communityeffectiveness or the “real world”efficacy of an intervention

Processes for Conducting HTA

Are all keystakeholder groupsactively engaged?

Yes, tool includes or considersstakeholders throughout itsprocess

No, only purchaser ofmedical devices isactively engaged andconsideration of keystakeholders is notstated

No, not all stakeholdersaddressed

Yes, key stakeholders areincluded throughout the process

Yes, key stakeholders areincluded or consideredthroughout the process

Yes, forms a national planningauthority that brings togetherdifferent stakeholders

Is all available dataactively beingsought?

Yes, all available data is soughtand used in decision makingtable

Yes, all available dataabout medical devicesincluded is activelysought and regularupdates are mentioned

Yes, all available data is soughtincluding contextual data

Yes, all available data is soughtduring the HTA report process

Yes, all available data issought and consulting alibrarian to ensure qualityis advised

Yes, strong evidence base isneeded for the implementationof this tool

Are findingsmonitored?

Not stated Not stated Not stated Not stated Not stated Not stated

Use of HTA in Decision-Making

Is it timely? Not stated Yes, it is a system ofthree forms that can bedone very quickly

Not stated Not stated Yes, the form takeswithin 5 to 15 hours toanswer (excludingevidence retrieval andassessment)

Not stated

Are findingscommunicatedappropriately todifferent decisionmakers?

Yes, the Decision-Making Tableallows decision makers to seethe evidence related to thecriteria and become informed ofthe health technologiesbeing assessed

Not stated Not stated Yes, the HTA report allowsdecision makers to see theevidence related to the criteriaand become informed of thehealth technologies beingassessed

Yes, the form gives aclear overview fordecision makers

Yes, it has included newadvances in knowledgetranslation

Is the link betweenHTA findings anddecision-makingprocesses transpar-ent and clearlydefined?

Yes, HTA findings and decisionmaking process are clearlyseparate

Not stated Not stated Yes, HTA findings and decisionmaking process are clearlyseparate

Yes, the purpose of themini-HTA is stated to beonly part of the basis ofa proposal for decisionmakers

Not stated

Krizaet

al.Health

ResearchPolicy

andSystem

s2014,12:66

Page8of

13http://w



Table 3 Extraction form for principles for HTA activities in SSA


Study name KNOW ESSENTIALS: A tool forinformed decisions in theabsence of formal HTAsystems

Appropriate medicaltechnologies fordeveloping countries:application tocardiovasculardisorders

Generalized cost-effectiveness analysis fornational-level priority-setting in the health sector

Field testing of amulticriteria decisionanalysis (MCDA)framework for coverageof a screening test forcervical cancer in SouthAfrica

Purchasing of medicalequipment in public hospitals:the mini-HTA tool

Equity-oriented toolkit forhealth technologyassessment and knowledgetranslation: application toscaling up of training andeducation for health workers

Ease of training Yes, it is an easy to use toolwith clear guidelines andcriteria

Yes, it is an easy touse tool with clearinstructions andwould require littletraining

No, special trainingrequired

Yes, not a difficultapproach

Yes, the mini-HTA tool is easy touse and the questions on theform are clear

Yes, not a difficult approach

Is the evidencecollectionrequirementflexible?

Yes, it allows the use of lesssystematic evidencegathering with justification

Yes, all evidence inthe form of dataabout medicaldevices is provided

Yes, data to assess theeffectiveness of a medicaltechnology not only comefrom reviews of evidence,but population surveys andexpert opinion

Yes, available data inlocal context is used tocustomize framework

Yes, available data in the localcontext is used

No, only mentions sourcessuch as The Cochrane Libraryand it is unclear if all othersources are considered

Is approach ortool economical?

Yes, does not requireexpensive equipment orservices

Yes, user would onlyneed a computer andthe software



Yes, tool only comprises of oneform and not expensiveequipment or services arerequired


Does approach ortool address localcontext, includingpoliticalframework?

Yes, it considers the localcontext throughout itsprocess

No, the tool itselfdoes not consider thelocal context

Yes, modificationsaccording to local contextis recommended byapproach

Yes, HTA report istailored to reflect localcontext

Yes, it is flexible and adapted tofit the local context

Yes, involves stakeholders inthe process which bring in acontextual perspective

Considerations Does not include an explicitpriority setting system.However, this tool is used toassess specific healthtechnologies so it is furtherdown in the HTA process.Evidence has the potential ofnot being very robust

Economical standingof this tool isdependent on theprice of the decisionmaking assistancesoftware

Economical standing of thisapproach is dependent ofamount of special trainingneeded

The mini-HTA tool is a fast tool;however, it is not verycomprehensive and only vaguelyaddresses the principles that itproposes. In addition, one of itslimitations is that it is only appliedin the context of hospital settings

Krizaet

al.Health

ResearchPolicy

andSystem

s2014,12:66

Page9of

13http://w



PRISMA flow diagram

Records identified through database searching

(n =1005)Pubmed: n = 810

Scopus: n = 4Science Direct: n = 78

EbscoH: n = 87EconLite: n = 26

Scr

een

ing

Incl

ud

edE

ligib

ility

Iden

tifi

cati

on

Additional records identified through other sources

(n = 68 )WHO Bulletin: n = 63

Health Policy and Plan ning: n = 2World Bank: n = 1

WHO: n = 2

Records after duplicates removed(n = 679)

Records screened(n = 394)

Records excluded(n =337)

high-income country: n = 54not HTA-tool-related: n = 128background on HTA: n = 155

Full-text articles assessed for eligibility

(n = 57 )

Full-text articles excluded, with reasons(n = 51)

only economic analysis: n = 7priority setting focus: n = 29

tool not described sufficiently: n = 3no sufficient HTA information: n = 12

Studies included in qualitative synthesis

(n =6 )

Figure 1 PRISMA flow diagram.


support in the purchase and acquisition of cardiovascu-lar disorder equipment [17] and the scaling up of train-ing and education of health workers [16]. The currentHTA approaches used in the studies were: multi-criteriadecision analysis (MCDA) [14], the WHO-CHOICE pro-ject [13], and generalized cost-effectiveness analysis [13].Four papers applied their tool or approach [8,14,15,17],two of those were pilot tests [8,17]. The other two ap-proaches and tools were not applied and only describedproposed HTA approaches or tools [13,16].Two of the tools were focused on pharmaceuticals [8,13],

whereas three HTA tools were focused on medical devices[14,15,17]. For the medical devices, varying contexts werechosen: one study focused on the introduction of a screen-ing test [14], another on the underutilisation of medical de-vices [17] and a third study focused on the decision-makingprocess for the purchasing of medical devices [15]. Onetool was centred on a health education intervention [16].

Structure of HTA programsAll of the papers were found to explicitly address relevantgoals and scopes of their HTA tool or approach. Five studieswere unbiased and had transparent processes [8,13,14,16,17].One tool required subjective assessment during its process[15]. Five methods included all relevant technologies byconsidering all alternatives [8,14,15,17] or by using an

extensive database to do so [13], while the sixth study didnot state if this was addressed [16].

Methods of HTAA wide range of evidence and outcomes was consideredby all studies [8,13-17] and appropriate methods for asses-sing costs and benefits were taken up by five approaches[8,13-16]. One study did not currently assess the costs[17]. Three considered a full societal perspective [8,15,16],whereas the other three failed to state this [13,14,17]. Onlythree of the tools or approaches stated explicitly character-izing uncertainty surrounding their estimates [13-15]. Is-sues of generalizability and transferability are addressed byfour of the papers [8,13,14,16].

Processes for conducting HTAAll of the approaches required for all available data to besought [8,13-17]. Two of the papers failed to actively in-clude all key stakeholders [13,17], whereas all others in-cluded key stakeholders by including them in the processor considering them throughout [8,14-16]. How findingswould be monitored was not stated in any of the papers.

Use of HTA in decision-makingOnly two studies mentioned a timeframe for completingthe HTA, which also was timely [15,17]. Findings were


communicated appropriately to different decision makersby four tools or approaches [8,14-16]. Three of the pa-pers clearly defined the link between HTA findings anddecision-making processes [8,14,15], while the others didnot state this [13,15,17].

Principles for HTA activities specific for SSAIn the context of SSA, five papers presented tools or ap-proaches that were easy to use [8,14-17], whereas onerequired special training to carry out the generalizedcost-effectiveness analysis [13]. The evidence require-ment was flexible and included available data whetherfrom a literature search or other less rigorous, yet justified,sources when it was not available for five of the studies[8,13-15,17]. One was not as flexible by requiring the datato be collected in databases such as the Cochrane Libraryand no additional alternate sources were mentioned [16].Only one of the papers failed to address the local contextexplicitly in their tool [17].

DiscussionHTA advantages and areas of useThe advantages of using HTA are first and foremost thesystematic evaluation of cost and effectiveness of med-ical technologies and allowing health systems to achievethe greatest good for the greatest number of patients.Chalkidou et al. highlight the importance of HTA for uni-versal health coverage systems by efficiently and equitablyallocating resources [18]. This focused resource use has aneffect on better budgeting and long-term financial sustain-ability of the health systems in SSA countries. It is crucialthat the increased use of HTA in these countries is com-plemented by capacity building and increased expertise inthe HTA area in order to ensure a sustainable infrastruc-ture [19]. International collaboration among HTA bodiescan facilitate the development of methods and more effi-cient assessment processes, and facilitate knowledge trans-fer and capacity-building in less established HTA systemsand programmes.Another area of need is emerging for relevant applications

of HTA, namely its use in global funding organisations as ameans for increased value for money. Teerawattananonet al. have argued for the use of HTA approaches forthe Global Fund to Fight HIV/AIDS, Tuberculosis andMalaria in order to provide strategic directions for theprioritization of health care interventions currently fundedby the organization [20]. Other global health initiativesand national donor countries may follow suit in conduct-ing HTAs before, during, and after grant implementa-tion in order to improve efficiency and identify areas ofunmet need (in June 2014 the Gates Reference Casehas been launched which involves more principled cost-effectiveness analysis in health programme funding bythe Bill and Melinda Gates Foundation). Therefore, the

necessity for the thorough evaluation and fine-tuningof relevant HTA tools and approaches becomes moreevident.

Lack of applicationThe review shows widespread methodological heterogen-eity among the different studies included in the review.The HTA tools or approaches used varied a lot in theircontext and scope. There was a lack of application of sometools for a specific medical technology or intervention inthe SSA setting. The tools analysed in this review wouldbenefit from a wider application and pilot-testing as well asuser friendliness. A direct comparison between applyingthe tools or approaches would also highlight the advan-tages and disadvantages of each tool or approach and guidedecision makers in regards to which tools should be used,in which context, and for which tasks.

Robustness of HTAWhile the need for HTA in SSA is evident, the robustnessof HTA cannot be neglected when resources are limited.The majority of the approaches involved a well-roundedstructure for their HTA that was accomplished with explicitgoals and scopes, unbiased and transparent processes, andthe inclusion of alternative technologies and some also hadpriority setting processes incorporated [14,16,17]. In thiscontext, “priority-setting” focuses on identifying differenthealth technologies for which an evaluation regarding theirinclusion in the health system is warranted, while HTA re-lates to the actual evaluation of a specific technology.The methods utilized in the approaches followed the

principles for HTA activities in most of the studies. How-ever, one approach that seemed to be limited by its pur-pose as a computerized decision-making aide [17] seemedto lack robustness in its methods by not assessing costs.The processes for conducting HTA were limited for someapproaches by the exclusion of key stakeholders [13,17], anintegral dimension of good practice in HTA methodology.In addition, the monitoring of HTA practices lacked in allof the studies. When assessing the use of HTA in decision-making, two approaches showed limitations [13,17]. Oneapproach completely lacked the inclusion of the principlesin this section, but this may be connected with the prioritysetting orientation of the approach [13]. All of the toolsaddressed the issue of economic resource deficiency; how-ever, they lacked specifications explicitly considering thelocal context [17], requiring little training for their use[13] and being flexible with the data necessary for under-taking the HTA when there is limited data or access avail-able [16].

Potential opportunitiesIn general, the review revealed that the majority of ap-proaches were not applied in a stand-alone manner and


were rather used as a support tool to existing decision-making processes. The analysis revealed that twoapproaches show particular promise for further investiga-tion: the KNOW ESSENTIAL tool for its compact, yetcomprehensive coverage [8], and the MCDA approach forthe active involvement of stakeholders in its process [14].These incorporated all aspects important for the SSA con-text into their evaluation such as contextual issues, flexibledata collection practices, and economical and easy use.However, it is important to point out that the MCDA ap-proach is, per se, focused on assessing available evidenceand not generating new evidence as in the case with otherHTA tools. Although several MCDA tools have been ap-plied in the SSA setting, we only included one study inour review as it was explicitly connected with the asses-sing of evidence in the context of a HTA report.Even though the mini-HTA tool [15] also incorporated

the majority of the principles for HTA activities and allof the contextual SSA principles, it had a number of lim-itations related to comprehensive and detailed coveragedue to its restriction to the hospital setting [15]. Thecritical appraisal also revealed that other tools did not meetall of the contextual criteria for SSA [13,16,17] or werelimited to a position further down-stream in the decisionmaking process. Although it is possible to weight the cri-teria applied in the KNOW ESSENTIALS tool, it would bebeneficial and interesting to combine the tool with aspectsof MCDA in order to allow for a more detailed stakeholderevaluation and prioritization of the evidence.A thorough evaluation of the available tools is highly

desirable, involving a wide range of academic and minis-try of health partners in a SSA setting in order to ensurecontext-specific application of HTA tools. Specific em-phasis should be made on the need for HTA evaluationsto allow for differences in the evaluations of pharmaceu-ticals and medical devices. The majority of HTAs cur-rently focus on the assessment of pharmaceuticals andtend to neglect medical devices. Due to the varying focusof the HTA tools that are centred on the evaluation ofmedical devices in our review, it is not possible to drawclear conclusions on the appropriate emphasis of the de-velopment of medical device HTA tools.

Study limitationsThe methodology for the evaluation on each quality cri-teria point highlighted in Table 3 “Extraction Form forPrinciples for HTA Activities in SSA” is based on broadexpert opinion elicitation in our research consortium,which represents different areas and institutions such asacademia, WHO, HTA expertise, and political decisionmakers. Ultimately, the evaluation on the basis of expertopinion is a subjective assessment of the research con-sortium which may be subject to potential bias and assuch has to be highlighted as a study limitation.

ConclusionsOur review has emphasised that there is a lack of HTAtools that can be used for systematically assessing technol-ogy in the SSA context. A clear gap in HTA methodologyfocused on “resource-limited” settings, and particularlythe SSA context, calls for more research into furtherevaluating and developing relevant HTA methods and ap-proaches, especially in the context of the WHO resolutionon “Health Intervention and Technology Assessment inSupport of Universal Health Coverage”.An appropriate HTA tool for “resource-constrained”

settings, and especially SSA, should address all import-ant criteria of decision making. By combining the twomost promising tools, KNOW ESSENTIALS and MCDA,appropriate analysis of evidence with a robust and flex-ible methodology could be applied for the SSA set-ting. Although there are a range of arguments favouringthe need for HTA in the SSA context, advocacy for theimportance of HTA in these settings needs to emergemore clearly [18].

AppendixSearch stringKeywords included synonyms for the following topic(“Health Technology Assessment” OR “HTA” OR “healthtechnology evaluation” OR “priority setting”) AND(“Developing countr*” OR “low income countr*” OR “re-source-limited” OR “resource-constrain” OR “Africa”). Inaddition, the names of all sub-Saharan African countrieswere separately listed as search terms:“Developing countr*” OR “low income countr*” OR

“resource-limited” OR “resource-constrain*” OR Africa*OR Angola OR Benin OR Botswana OR “Burkina Faso”OR Burundi OR Cameroon OR “Cape Verde” OR “CentralAfrican Republic” OR Chad OR Comoros OR CongoOR “Democratic Republic of Congo” OR Djibouti OR“Equatorial Guinea” OR Eritrea OR Ethiopia OR GabonOR Gambia OR Ghana OR Guinea OR “Guinea Bissau”OR “Ivory Coast” OR “Cote d’Ivoire” OR Jamahiriya ORJamahiryia OR Kenya OR Lesotho OR Liberia ORMadagascar OR Malawi OR Mali OR MauritaniaOR Mauritius OR Mayote OR Mozambique OR MocambiqueOR Namibia OR Niger OR Nigeria OR Principe ORReunion OR Rwanda OR “Sao Tome” OR Senegal ORSeychelles OR “Sierra Leone” OR Somalia OR “SouthAfrica” OR “St Helena” OR Sudan OR Swaziland ORTanzania OR Togo OR Tunisia OR Uganda OR “WesternSahara” OR Zambia OR Zimbabwe AND “HealthTechnology Assessment” OR HTA OR “health technol-ogy evaluation” OR “priority setting”.

AbbreviationsHTA: Health technology assessment; MCDA: Multi-criteria decision analysis;SSA: Sub-Saharan Africa.


Competing interestsThe authors declare that they have no competing interests.

Authors’ contributionsCK, BA (Hons), MSc: conception and design, acquisition of data, analysis, andinterpretation of data. JH-H, PhD: conception and design, analysis, andinterpretation of data. EAO, MBCHB, MPH: conception and design, analysis,and interpretation of data. ND, MSc: conception and design, analysis, andinterpretation of data. RA, BPharm, PhD: analysis and interpretation ofdata. PW, PharmD: conception and design, acquisition of data, analysis,and interpretation of data. MM, BSc: acquisition of data, analysis, andinterpretation of data. NG: critical revision of manuscript. WA, MD: criticalrevision of manuscript. IW: critical revision of manuscript. PK-R, MD, PhD,MBA: conception and design, analysis, interpretation of data, and criticalrevision of manuscript. All authors read and approved the finalmanuscript.

AcknowledgmentsThis research is supported by the German Federal Ministry of Education andResearch (BMBF), project grant no. 01EX1013B, as part of the NationalLeading-Edge Cluster Medical Technologies ‘Medical Valley EMN’.

Author details1Interdisciplinary Centre for Health Technology Assessment (HTA) and PublicHealth (IZPH), University of Erlangen-Nürnberg, Schwabachanlage 6, Erlangen91054, Germany. 2Health Economics and HIV/AIDS Research Division(HEARD), University of KwaZulu-Natal, Westville Campus, J block, Level 4,University Road, Private Bag X54001, Durban 4041, South Africa. 3GhanaMinistry of Health, MB 44 Accra, Ghana. 4Centre for Research in TherapeuticSciences (CREATES), Strathmore University, Ole Sangale Road, 59857-00200Nairobi, Kenya. 5Ghana Ministry of Health, M44 Accra, Ghana. 6Kenya Ministryof Public Health and Sanitation, Kenyatta Hospital Grounds, PO Box19982-00202, Nairobi, Kenya. 7World Health Organization, 20 Avenue Appia,Geneva 1211, Switzerland.

Received: 26 June 2014 Accepted: 21 November 2014Published: 2 December 2014

References1. Asua BJ: [The International Network of Agencies for Health Technology

Assessment (INAHTA) or the need for collaboration in the evaluation ofhealth technologies]. Med Clin (Barc) 1999, 112:86–89.

2. Chalkidou K, Levine R, Dillon A: Helping poorer countries make locallyinformed health decisions. BMJ 2010, 12(04):23. 341.

3. Pichon-Riviere A, Augustovski F, García Martí S, Sullivan SD, Drummond M:Transferability of health technology assessment reports in Latin America:An exploratory survey of researchers and decision makers. Int J TechnolAssess Health Care 2012, 28(2):180–186.

4. Panerai RB, Almeida RT, Freire SM, Chaim DM, Miranda MZ, Madureira LC,Aguiar Neto MA: Perspectives on health technology assessment in LatinAmerica. The case of perinatal care in the literature. Int J Technol AssessHealth Care 1993, 9(1):76–84.

5. Kularatna S, Whitty JA, Johnson NW, Scuffham PA: Health state valuation inlow- and middle-income countries: a systematic review of the literature.Value Health 2013, 16(6):1091–1099.

6. Mbondji PE, Kebede D, Soumbey-Alley EW, Zielinski C, Kouvividila W,Lusamba-Dikassa PS: Health information systems in Africa: analysis ofresources, indicators, data management, dissemination and use. J R SocMed 2014, 107(1):Suppl 28–33.

7. Banta D: The uses of modern technologies: problems and perspectivesfor industrialized and developing countries. Bull Pan Am Health Organ1984, 18(2):139–150.

8. Mathew JL: KNOW ESSENTIALS: a tool for informed decisions in theabsence of formal HTA systems. Int J Technol Assess Health Care 2011,27(2):139–150.

9. Baltussen R, Norheim OF, Johri M: Fairness in service choice: an importantyet underdeveloped path to universal coverage. Trop Med Int Health 2011,16(7):838–839.

10. Moher D, Liberati A, Tetzlaff J, Altman DG: Preferred reporting items forsystematic reviews and meta-analyses: the PRISMA statement. BMJ 2009,11(46):49. 339.

11. Mays N, Pope C: Qualitative research in health care. Assessing quality inqualitative research. BMJ 2000, 320(7226):50–52.

12. Drummond MF, Schwartz JS, Jonsson B, Luce BR, Neumann PJ, Siebert U,Sullivan SD: Key principles for the improved conduct of healthtechnology assessments for resource allocation decisions. Int JTechnol Assess Health Care 2008, 24(3):244–258.

13. Hutubessy R, Chisholm D, Edejer T: WHO-CHOICE. Generalizedcost-effectiveness analysis for national-level priority-setting in thehealth sector. Cost Effectiveness Resour Allocation 2003, 1(1):8.

14. Miot J, Wagner M, Khoury H, Rindress D, Goetghebeur M: Field testing of amulticriteria decision analysis (MCDA) framework for coverage of ascreening test for cervical cancer in South Africa. Cost Effectiveness ResourAllocation 2012, 10(1):2.

15. Govender M, Mueller DB, Basu D: Purchasing of medical equipment inpublic hospitals: the mini-HTA tool. S Afr Med J 2011, 101(11):807–808.

16. Ueffing E, Tugwell P, Hatcher Roberts J, Walker P, Hamel N, Welch V:Equity-oriented toolkit for health technology assessment and knowledgetranslation: application to scaling up of training and education forhealth workers. Hum Resour Health 2009, 7:67.

17. Abaza HH, Tawfik BS: Appropriate Medical Technologies for DevelopingCountries. Cairo: Biomedical Engineering Conference; 2008.

18. Chalkidou K, Marten R, Cutler D, Culyer T, Smith R, Teerawattananon Y,Cluzeau F, Li R, Sullivan R, Huang Y, Fan V, Glassman A, Dezhi Y,Gyansa-Lutterodt M, McPherson S, Gadelha C, Sundararaman T, Squires N,Daulaire N, Sadanandan R, Shiro K, Lemgruber A: Health technologyassessment in universal health coverage. Lancet 2013, 382(9910):e48–e49.

19. Odame EA: Systematic review of economic evaluation literature inGhana: is health technology assessment the future? Value in HealthRegional Issues 2013, 2(2):279–283.

20. Teerawattananon Y, McQueston K, Glassman A, Yothasamut J, Myint C:Health technology assessments as a mechanism for increased value formoney: recommendations to the Global Fund. Glob Health 2013, 9(1):35.

doi:10.1186/1478-4505-12-66Cite this article as: Kriza et al.: A systematic review of Health TechnologyAssessment tools in sub-Saharan Africa: methodological issues andimplications. Health Research Policy and Systems 2014 12:66.

Submit your next manuscript to BioMed Centraland take full advantage of:

• Convenient online submission

• Thorough peer review

• No space constraints or color figure charges

• Immediate publication on acceptance

• Inclusion in PubMed, CAS, Scopus and Google Scholar

• Research which is freely available for redistribution

Submit your manuscript at www.biomedcentral.com/submit

Documents

A systematic review of Health Technology Assessment tools in sub