Guidelines for AI procurement (Mobile version) · Fourth Industrial Revolution, Government Digital Service (GDS), Government Commercial Function and Crown Commercial Service. A wide

Version 1

3

Who should use the GuidelinesThese guidelines are aimed at central government departments who areconsidering the suitability of AI technology to improve existing services or as partof future service transformation Other public sector bodies may also follow theseguidelines

These guidelines will be useful to

bull Procurement and commercial practitioners who are responsible for theprocurement of AI technologies and management of contracts

bull Chief Data Information Technology and Innovation Officers considering thesuitability of AI systems to address technology challenges

bull Digital Delivery and Transformation teams who want to use AI technologies indigital transformation projects and programmes

bull Analysts Data Scientists and other Digital Data and Technology (DDaT)professionals who are developing project-specific requirements evaluatingusing or maintaining AI systems

bull Suppliers who want to better understand best practice technical and ethicalstandards for AI procurement in government

Public procurement is led by a framework of procurement rules and regulationsdaggerthis guidance assumes the reader has a sound working knowledge of those rulesand of the end-to-end procurement process Apply commercial judgement whenusing this guide and seek legal advice where appropriate Some research contractsmay be out of scope of procurement legislation

dagger The Public Contracts Regulations 2015 (likely to be the most relevant regulation for AI systems)the Utilities Contracts Regulations 2016 and the Concession Contracts Regulations 2016

4 Guidelines for AI procurement

5

Introduction 6

What is AI and how can it be used in government 7

What is the aim of the guidelines 9

How should the guidelines be used 10

Ten Guidelines for AI Procurement 12

Include your procurement within a strategy for AI adoption 13

Make decisions in a diverse multidisciplinary team 13

Conduct a data assessment before starting your procurement process 14

Assess the benefits and risks of AI deployment 15

Engage effectively with the market from the outset 16

Establish the right route to market and focus on the challenge rather than a specificsolution 16

Develop a plan for governance and information assurance 19

Avoid Black Box algorithms and vendor lock in 19

Focus on the need to address technical and ethical limitations of AI deploymentduring your evaluation 20


AI specific considerations within the procurement process 22

Preparation and planning 24

Publication 28

Selection Evaluation and Award 31

Contract Implementation and Ongoing management 32

Further reading 34

Contents


Introduction

7

What is AI and how can it be used ingovernmentArtificial Intelligence (AI) comprises aset of technologies that have thepotential to greatly improve publicservices by reducing costs enhancingquality and freeing up valuable timefor frontline staff

We are in the early days of deploying AIsystems in government and arecontinuously discovering new benefits forusing AI systems to drive decisionmaking as well as challenges and risksthat need to be addressed

This guidance mostly refers to the use ofmachine learning Machine learning is asubset of AI and refers to thedevelopment of digital systems thatimprove their performance on a giventask over time through experienceMachine learning is the most widely-usedform of AI and has contributed toinnovations like self-driving cars speechrecognition and machine translation

Keen to learn more

There are many new concepts used in the field ofAI and you may find it useful to refer to a glossaryof AI terms in the Annex For more examples onhow AI has been used in the UK public sector youcan explore the case studies in A guide to usingartificial intelligence in the public sector

To learn more about AI technologies the BiscuitBook was developed by data science experts atthe Defence Science and Technology LaboratoryAI Lab to help Ministry of Defence customersunderstand AI data science and machinelearning

A guide to using artificial intelligence inthe public sector provides the followingdefinition of AI

lsquoAI can be defined as the use ofdigital technology to create systemscapable of performing taskscommonly thought to requireintelligencersquo

The development of AI is constantlyevolving but generally it involvesmachines using statistics to find patternsin large amounts of data and using thatdata to perform repetitive tasks withoutthe need for constant human guidance

A guide to using artificial intelligence in

the public sector (Office for AI and GDS)


9

What is the aim of the guidelines

Public procurement can be an enablerfor the adoption of AI and could be usedto improve public service deliveryGovernmentrsquos purchasing power candrive this innovation and spur growth inAI technologies development in the UK

As AI is an emerging technology it can bemore difficult to establish the best routeto market for your requirements toengage effectively with innovativesuppliers or to develop the right AI-specific criteria and terms and conditionsthat allow effective and ethicaldeployment of AI technologies

These guidelines provide a set of guidingprinciples on how to buy AI technologyas well as insights on tackling challengesthat may arise during procurement It isthe first of such guidance and is notexhaustive

These guidelines have been developed bythe Office for AI in collaboration with theWorld Economic Forum Centre for theFourth Industrial Revolution GovernmentDigital Service (GDS) GovernmentCommercial Function and CrownCommercial Service A wide range ofstakeholders from industry academiaand government departments havehelped to shape this update Theguidelines were initiated through theWorld Economic Forumrsquos lsquoUnlockingpublic sector AIrsquo project

Also as part of the project the Office forAI co-created the AI Procurement in aBox a toolkit to help public sectorprocurement professionals across theglobe rethink their approaches to AIprocurement

The AI Procurement in a Box toolkit

provides detailed guidance on AI

procurement and greater detail on the

main issues to consider during the process


How should the guidelines be used

These guidelines provide an overview ofwhat themes to consider when assessingthe viability of an AI system and whatprocurement teams should considerwhen implementing AI technologyprojects As a guiding principle betransparent about your AI project and thetools data and algorithms you will beusing working in the open wherepossible

Not all forms of AI systems will be thesame and increasingly forms of AItechnologies will be built-in to manytypes of technology products AI systemscan be developed from scratch boughtoff the shelf or added to systems that arealready in use

Where suppliers are proposing to utilise AItechnologies as part of the delivery of a non-AIspecific requirement or system - there may be aneed for some additional considerations to betaken into account In such instances discusswith your Chief Digital Information OfficerTechnical Architect to assess the impact AImodels may have on the solution and use yourcommercial judgement to establish appropriateprovisions within the contract to accommodatethe use of AI technologies as part of the delivery

This guidance should be consideredalongside existing policy and guidance inrelation to the use of technology anddigital services

bull The Digital Service Standard

bull The Technology Code of Practice

bull Data Ethics Framework

bull A guide to using artificial intelligencein the public sector

bull Open Data Standards

bull Other Technology standards andguidance

Refer to The Outsourcing Playbook anddefine your purchasing strategy in thesame way as you would for any othertechnology requirement

Also consult sector specific guidance forinstance NHSX A Buyerrsquos Checklist for AIin Health and Care

A Buyerrsquos Checklist for AI in Health and Care (NHSX)

11

What is data ethics

Data ethics is an emerging branch of applied ethics which describes thevalue judgements and approaches we make when generating analysingand disseminating data This includes a sound knowledge of dataprotection law and other relevant legislation and the appropriate use ofnew technologies It requires a holistic approach incorporating goodpractice in computing techniques ethics and information assurance

What is data science

A core aspect of data ethics is using data science appropriately Datascience describes analysis using automated methods to extract knowledgefrom data It covers a range of techniques from finding patterns in datausing traditional analytics to making predictions with machine learning

Data science presents new opportunities for identifying factors foranswering important policy questions mdash factors which might be difficult forpeople to spot on their own It offers huge public benefits in creatingbetter evidence-based policy and in making government operations moretargeted and efficient However we must carefully consider the socialimplications of the data and algorithms used our practices and the qualityassurance processes we follow to ensure this is done well

Why do we need a framework

Advances in computing power and techniques mean newer morepowerful computational models or data science tools are seeing uptakeacross the public sector Coupling this with an increase in skills means wenow have the ability to analyse larger volumes of data more rapidly andmore regularly

Increasingly public servants from across disciplines will need to understandinsights from data and emerging technologies It is crucial that publicservants are equipped to use data-informed insight responsibly andprocesses must be in place to support this

THE DATA ETHICS FRAMEWORK


Ten Guidelines forAI Procurement

13Ten Guidelines for AI Procurement

1 Include your procurementwithin a strategy for AI adoption

2 Make decisions in a diversemultidisciplinary team

Ensure your Technology and Data strategies are updated to incorporate AI technologyadoption Use procurement strategically to support AI adoption across government takeadvantage of economies of scale of AI technology deployment through collaboration andshare your knowledge with interested teams across government

Consider aligning your work with other teams across central government departmentsand organisations leading on relevant AI initiatives

Establish networks within your organisation and across the civil service to share insightsand learn from best practice

Developing evaluating and delivering AI projects will be more effective with diverse teamsthat understand the interdependent disciplines that AI technologies incorporate Thiscould include

bull Domain expertise (for example healthcare transportation)

bull Commercial expertise

bull Systems and data engineering

bull Model development (for example deep learning)

bull Data ethics

bull Visualisationinformation design

Require the successful supplier(s) to assemble a team with the right skill sets and toaddress the need for diversity to mitigate bias in the AI system


Conduct a data assessment beforestarting your procurement process3

Data is currently the basis of the majority of AI-powered solutions Availability of relevantdata is often a prerequisite for any AI system so time should not be spent discussing AIprocurement if no data will be available

bull Ensure data governance mechanisms are in place from the start of the procurementprocess

bull Assess whether relevant data will be available for the project

bull Try to address flaws and potential bias within your data before you go to marketandor have a plan for dealing with data issues if you cannot rectify them yourself

bull Define if and how you will share data with the vendor(s) for the procurement initiativeand the subsequent project


Assess the benefits and risksof AI deployment4

Defining the public benefit goal provides an anchor for the overall project andprocurement process that the AI system is intended to achieve AI technology also bringsspecific risks which must be identified and managed early in the procurement phase

bull Explain in your procurement documentation that the public benefit is a main driver ofyour decision-making process when assessing proposals Consider the human andsocio-economic impact and benefits of your AI system in line with Social Value guidePublic benefit goals must be relevant to what you are procuring (and not generic innature) and must comply with the principles of non-discrimination equal treatmentand proportionality

bull Set out clearly in your procurement documentation why you consider AI to be relevantto the problem and be open to alternative solutions

bull Conduct initial AI impact assessments at the start of the procurement process andensure that your interim findings inform the procurement Be sure to revisit theassessments at key decision points


Engage effectively with themarket from the outset

Establish the right route tomarket and focus on thechallenge rather than aspecific solution

5

6

Government spending can be used to create a fair competitive market which leads tobetter AI systems Early engagement with AI vendors can result in more relevantresponses increasing the likelihood of a successful procurement and better projectdelivery Focus on proportionality in your approach and do not impose unnecessaryburdens that would deter suppliers including start-ups small and medium sizedenterprises (SMEs) voluntary community and social enterprises (VCSEs) suppliers andthose owned by under-represented groups from competing

bull Engage with AI suppliers early and within your planning phase

bull Reach out in various ways to a wide variety of AI suppliers

bull Encourage an open environment that supports competition in the AI ecosystem

The AI systems being procured must address the challenges you want to solve andpromote a responsible innovative response from the market Carefully writtenrequirements can help a supplier understand what you need and propose their bestsolution Tell suppliers about the situation or challenge and let them propose a solutionthat meets your needs

bull Refer to the guidance in The Outsourcing Playbook for commercial best practice

bull Explore different routes to market to acquire AI systems for instance InnovationPartnerships the GovTech catalyst and the Crown Commercial Servicersquos AI DynamicPurchasing System

bull Provide a clear problem statement rather than detailed specifications for a solution

bull Prioritise an iterative approach to product development and reflect this accordingly inthe invitation to tender




Develop a plan for governanceand information assurance

Avoid Black Box algorithmsand vendor lock in

7

8

You must establish appropriate oversight mechanisms to allow scrutiny of AI systemsthroughout their lifecycle You will need to apply different considerations depending onthe AI use case and the risk profile of the project and ensure that your approach canwithstand scrutiny Highlight the need to comply with existing laws and regulations andsupport the standardisation of norms through your procurement documentation

Be sure to refer to existing codes of practice guidance and regulations when draftingyour requirement and ensure these are carried over to the terms of the contract wheresuitable

bull Adhere to the Tech Code of Practice and Government Design Principles the DataEthics Framework and other relevant standards

bull Maximise transparency in AI decision-making to give users confidence that an AIsystem functions well

Encourage explainability and interpretability of algorithms and make this one of yourdesign criteria This means using methods and techniques that allow the results to beunderstood by your team Highly lsquoexplainablersquo outputs from your AI system will be able tobe interpreted by your team and by other suppliers This will also make it more likely foryou to be able to engage with other suppliers to continue or build upon your AI system inthe future limiting the risk of vendor lock-in


Focus on the need to addresstechnical and ethicallimitations of AI deploymentduring your evaluation9

Make use of the experience within your multidisciplinary team to support the evaluationprocess and ensure there is broad expertise conducting the tender evaluation

bull Have suppliers highlighted andor addressed any issues of bias within the data Dothey clearly explain why their strategies are appropriate and proportionate Do theyhave a plan for addressing any issues that you may have missed and underline theimportance of an agile project delivery

bull Has consideration been given to any required integration with existing services ortechnology

bull Does their governance approach meet your requirements

bull Have the appropriate technical standards been adhered to


Assess the benefits andrisks of AI deployment10

For AI-powered solutions in the public sector implementation plans sustainable andongoing evaluation methods and mechanisms to feed back into the data model arecrucial to ensure ethical use Further the functionality and consequences of AI systemsmay not be apparent in the procurement process and often only become evident duringdeployment requiring extended communication and information sharing between thebuyer and supplier

bull Consider during AI procurement that lifespan testing not a one-time decision isrequired

bull Ensure that knowledge transfer and training is part of your requirement

bull Ensure that you make training and explanations for non-specialists that might need tounderstand the AI system part of your requirement

bull Ensure you have the appropriate ongoing support and hosting arrangements in place


AI specificconsiderationswithin theprocurementprocess

23AI specific considerations within the procurement process

This section raises specific considerations to be addressed throughout theprocurement process

1 Preparation and Planning

2 Publication

3 Selection Evaluation and Award

4 Contract Implementation and Ongoing Management

As a general principle any AI procurement should be investigated with the mindset ofldquohow could AI technologies potentially benefit usrdquo rather than ldquohow can we make ourproblem fit an AI system solutionrdquo

Treat AI technology like you would any other technological solution and use it inappropriate situations Every service is different and many of the decisions you makeabout technology will be unique to your service


Preparation is the key to achievingflexible and efficient procurementprocesses that encourage broadparticipation which are open andaccessible to all Work in the openwhere possible and comply with pre-procurement legal requirements Thisincludes making due considerationsrequired by the Public Services (SocialValue) Act 2012 (as amended) asapplicable and assessing applicationof the Public Sector Equality Dutyunder the Equality Act 2010

Before you commence any procurementproject likely to include deployment of AIsystems ensure you have considered thefollowing

Multidisciplinary teams

Bring people into your team who havethe knowledge and experience toconsider whether AI is a viable andappropriate solution Seek to establish amultidisciplinary team with a diversecombination of roles and skills to supportthe procurement and implementation ofyour AI system A team with a diverse skillset will help you to conduct data andimpact assessments and ensure thatyour business case and procurementprocess reflects their key findings

Some specialist roles to consider for your AIproject team

bull Data Architect

bull Data Scientist

bull Data Engineer

bull Technical Architect

bull Delivery Manager

bull Security Architect

bull Commercial Manager

You may not need all of these roles from the verybeginning but consider your needs before youstart It is useful to consult experts to ensure thatyou have the right foundations in place to go tomarket and have verified that you can integratean AI system with your existing processestechnologies and services

It is important to ensure robust practicesare in place and that work is carried outwithin the teamrsquos skillset If your teamlacks expertise you could reach out toprofessional networks within yourorganisation or across government togather important insight into yourdesired use case

You may also consider completing adiscovery exercise as part of the decisionmaking process to explore yourrequirements

Preparation and planning


Keen to learn more

Examples for teams and organisations that mightbe useful to consult are the Office for AI the GDSthe Centre for Data Ethics and Innovation orteams and organisations with specific domainknowledge Also learn about best practices andshare knowledge and feedback via expertcommunities Knowledge Hub the Digital BuyingCommunity the Data Science Community ofInterest or other similar networks

Data assessment and governance

Ensure that a discovery into your data isconducted before you go to market Youmay need to seek out specific expertiseto support this data architects and datascientists should lead this process Thiswill help you and your team tounderstand the complexitiescompleteness and limitations of the datayou have available

If a thorough assessment of the dataproves difficult or has not been mademake it a requirement in the invitation-to-tender to conduct a comprehensivecheck of the data the AI system will useto base its decisions upon

Your data governance needs to cover alldata activities related to your project

bull Granting data access to projectmembers

bull Storing data in other locations foranalysis

bull Reviewing data consent

Depending on the sensitivity of yourproject and data it is worth consideringreleasing data to suppliers during theprocurement process This can allowsuppliers to gain insight into the availabledata and improve responses to theinvitation to tender Ensure you providethe same data to all suppliers at thesame time and that you are acting inaccordance with Data Protectionlegislation and GDPR and consider theuse of Non-Disclosure Agreements(NDAs) or supplier engagement events tosupport this The Data Ethics Framework- Principle 3 provides further informationon data governance and proportionality

Consider the use of anonymisation techniques tohelp safeguard data privacy including dataaggregation masking and synthetic dataInvitations-to-tender should encourage innovativetechnological approaches that make less intrusiveuse of data or that achieve the same or similaroutcomes with less sensitive datasets


AI Impact assessment

Your AI impact assessment should beinitiated at the project design stageEnsure that the solution design andprocurement process seeks to mitigateany risks that you identify in theassessment Your AI impact assessmentshould be an iterative process as withoutknowing the specification of the AIsystem you will acquire it is not possibleto conduct a complete assessment

Your AI impact assessment shouldoutline

bull Your user needs and the publicbenefit of your AI system

bull Human and socio-economic impactsof your AI system - this will help toensure it delivers social value benefits

bull Consequences for your existingtechnical and procedural landscape

bull Data quality and any potentialinaccuracy or bias

bull Any potential unintendedconsequences

bull Whole-of-life cost considerationsincluding ongoing support andmaintenance requirements

Associated risks and their respectivemitigation strategies must be providedand agreed upon within the impactassessment and should include lsquogonogorsquo key decision points where applicableReview your impact assessment at these

decision points or every time asubstantial change to the design of an AIsystem is made

Keen to learn more

Data protection impact assessments and equalityimpact assessments can provide a useful startingpoint for assessing potential unintendedconsequences For examples of risk assessmentquestionnaires for automated decision makingrefer to the Government of Canadarsquos Directive onAutomated Decision Making and the frameworkon Algorithmic Impact Assessments from AI Now

Preliminary Market Engagement

Preliminary market engagement will helpto understand if whether and how AI canbe part of the solution Learning takenfrom your preliminary marketengagement will help you to better defineyour problem statement and can alsohelp you to determine the scope andfeasibility of your requirements

Your preliminary market engagement shouldactively seek out suppliers that can help toimprove service delivery including SMEs and anyVCSEs who are experts in AI design and deliveryacross the country

All preliminary market engagement mustobserve the principles of publicprocurement and be handled in such away that no supplier gains a preferentialadvantage In practice this means notsetting the technical specification to suita particular solution or supplier andmaking sure any information shared isalso available during the procurementprocess


Keen to learn more

Completing preliminary market engagement or adiscovery phase will help you to understand yourproblem before you commit to buying or buildinga service Find more information on how toorganise your discovery phase in A guide to usingartificial intelligence in the public sector sectionon planning and preparing for artificialintelligence implementation

Procurement approach and vehicle

There are currently a number of routes tomarket to purchase AI systemsDepending upon which kind ofchallenges you need to addressFramework agreements including G-Cloud Digital Outcomes and Specialistsand the Spark Dynamic PurchasingSystem (DPS) are useful starting points toconsider

Innovation-oriented procurementprocedures provide opportunities toaccelerate the adoption of newtechnologies within government andpromote innovation and ethicaldevelopment of AI These can include

bull Agile procurement processes thatallow you to go to market at differentstages and can include proof-of-concepts to test the technologiesbefore the solution goes liveDiscovery phases or proofs-of-concepts can demonstrate if the AIsystem is likely to meet your widerrequirements

bull Technology contests demonstratorsand challenge-based procurement

processes have vendors competeagainst each other based on their AIskills and include an evaluation of thetechnologies applied to thechallenges they mean to addressThese processes focus on innovationand allow you to explore differentapproaches Examples include theGovTech Catalyst or the ScottishGovernment-run CivTechreg acceleratorprogramme

bull Innovation Partnerships enable theprocurement of technologies thatcannot be delivered by the currentoptions available to the market Theupdate of the Public ContractsRegulations highlights theopportunities that this route tomarkets can unlock This researchreport provides analysis on how thesepartnerships can work on a local level

bull Specialist AI Procurement Frameworksor Dynamic Purchasing Systems thatprescribe the terms and conditionsapplying to any subsequent contractand allow you to assess suppliersagainst a set of predefined criteriathat can include ethical requirementsThe Dynamic Purchasing System for AIfrom Crown Commercial Service is thefirst example of this kind of novelapproach


Publication

Whichever route to market you chosebe mindful that AI technologies arerapidly developing with newtechnologies and products constantlycoming to market Use output-basedrequirements in your invitation-to-tender that focus on describing thechallenges and opportunities you arefacing This will allow suppliers todetermine which technologies aremost appropriate for yourrequirements

Drafting your requirement

The drafting of your requirement candrive innovation and set the foundationsfor the effective responsible and ethicaldeployment of AI technologies

Use output-based requirements whichallow the supplier to propose how theywill respond to your requirement You willhave to draft sufficiently detailed problemstatements backed by user needs andrequired performance

Key considerations when drafting your AIrequirement

bull Start with your problem statement

Set out clearly what challenge you areaiming to address including anylimitations and additional functionalrequirements Describe why you considerAI to be relevant to the challenge andremain open to alternative solutions

bull Highlight your data strategy andrequirements

Describe how the AI system will fit withinyour current data strategy and practicesbased on your data discovery Referenceyour data protection impact assessmentwhere possible and add the findings ofyour data assessment your datarequirements and details on your datagovernance approach

bull Focus on data quality bias andlimitations

Use the insights from your data discoveryto highlight known limitations of the datain your invitation to tender and asksuppliers to describe their strategies onhow to address these shortcomingsHave a plan for addressing relevantlimitations you may have missed and asksuppliers for their strategies to mitigatethem

bull Underline the need for you tounderstand the supplierrsquos AIapproach

Draft evaluation questions that give youinformation about the algorithms andmodels including how the supplierselects variables and what AI techniquesthe model is based on (for examplesupervised unsupervised orreinforcement learning) and try toestablish any limitations of the modelSeek clarity on the origin and nature ofany data the supplier trained thealgorithms with andor plans on bringingto the project You can also consider therequirement for independent audits of


the algorithms Ensure your evaluationcriteria appropriately assess these points

bull Consider strategies to avoid vendorlock-in and lsquoBlack Boxrsquo AI systems

Avoid relying on lsquoblack-boxrsquo algorithmsUnderline the need for an lsquoexplainableapproachrsquo to AI development (the extentto which an AI systemrsquos decision makingprocess can be understood) in yourinvitation-to-tender Highly lsquoexplainablersquooutputs from your AI system will be ableto be interpreted by your team and byother suppliers This will increase thelikelihood for you to be able to engagewith other suppliers in the future tocontinue or build upon the initial AIsystem limiting the risk of vendor lock-inConsider addressing these issues in yourprocurement documentation Goodpractice could involve adopting openstandards royalty-free licensingagreements and public domainpublication terms

Apply the Data Ethics Framework -Principle 6 Make your work transparentand be accountable and consider the useof other tools for example independentaudits For more information on this topicread the policy briefing of the RoyalSociety on Explainable AI

bull Indicate the importance ofintellectual property ownership

Suppliers may not wish to disclose detailsof the inner workings of their solution inorder to protect their intellectualproperty (IP) and commercial advantageDuring the design and deployment of the

AI system it is likely that either a newalgorithm will be developed or anexisting one will be tailored (for examplere-trained through your data) Considerwhether you or the supplier should ownany new intellectual property and whocan best exploit the IP generated Ensurethat arrangements are mutuallybeneficial and fair

Consider the 10th Government DesignPrinciple and ensure your work is openand available to others for reuse

bull Mention any integration withassociated technologies or services

Work with your Data and TechnicalArchitects to define any associatedtechnologies or services that the AIsystem will have to integrate with Theremay be specific design criteria requiredthat potential suppliers need to be awareof when responding to your invitation-to-tender

bull Consider your ongoing support andmaintenance requirements

Operational or service staff must haveenough knowledge of or training on theAI system to understand how to use itand act on its outputs You may wish tohighlight the importance of training andknowledge transfer to ensure your teamis up-skilled during the life of the contractand has a deep understanding of thesolution the supplier has put in placeand consider how non-specialist userscan be supported


Consider ongoing support requirementshosting or additional development thatmay be required beyond the term of yourinitial contract If you have an in-houseteam taking over support andmaintenance of the service ensure theyhave been consulted

bull Add considerations on liability andrisk

Allocate risk to the parties best able tomanage it Correct risk allocation iscrucial to the long term viability of theservice but also to achieving best valueLiability for certain areas will reside withthe department particularly around theuse and application of the AI-poweredsolution and in relation to data accessand transfer Liability may also need to sitwith the supplier including areas focusedaround technical security and qualityassurance Underline theseconsiderations in your invitation-to-tender Refer to chapter 8 of TheOutsourcing Playbook for moreinformation on Risk Allocation

The application must also have an easyway to report any suspectedunauthorised behaviour to relevantauthorities within or outside theorganisation

Use the experience of the team tosupport the drafting process andengage with users and stakeholders toestablish the core areas of assessmentwhich should be included Allrequirements should be transparent andshould not discriminate against particulartypes of suppliers for instance SMEs and

VCSEs or those from countries withwhich the UK has trade agreements withprocurement obligations

Keen to learn more

Follow the commercial best practice outlinedwithin The Outsourcing Playbook and adhere tothe Tech Code of Practice and GovernmentDesign Principles

The Outsourcing Playbook

(Government Commercial Function)


During the selection and evaluationstages consider the suppliersrsquo responsesto your requirement Make use of theexperience within your multidisciplinaryteam to support the evaluation processand to ensure there is a broad base ofexpertise conducting the evaluation

There are key robust practices that youcan ask for suppliers to demonstratewhen providing AI-powered solutions andthen look out for when evaluatingtenders Robust practices may includebut are not limited to

bull Having an internal AI ethics approachwith examples of how it has beenapplied to design develop and deployAI-powered solutions

bull Processes to ensure accountabilityover outputs of algorithms

bull Avoiding outputs that could beunfairly discriminatory

bull Designing for reproducibility

bull Testing the model under a range ofconditions

bull Defining acceptable modelperformance

bull Robust and proportionate securityprovision

As part of the evaluation process alsoreview the specialist skills qualificationsand diversity of the team that will developand deploy the AI system This can alsohelp to anticipate or detect unfair bias in

the system

Be aware that suppliersrsquo responses willgive you an indication of a supplierrsquosgeneral approach Do not expect a fullydetailed and definite plan as AIdevelopment is an iterative process andthe system will invariably change andevolve as the project progresses

Keen to learn more

The resources developed by the World EconomicForum in partnership with the Office for AI alsoprovide examples for evaluating responses in AItenders and will provide you with some ideas onfurther considerations during the evaluationstage The Magenta Book can provide generalguidance on monitoring and evaluation strategiesin government

Selection Evaluation and Award


As with any contract time and careshould be taken to onboard the newsupplier in accordance with TheOutsourcing Playbook and best practicefor Contract Management

An AI system may need continuedsupport throughout its lifecycleAccepting the potential impact of anysupport gaps or employing outsideexpertise both come at a cost Thisshould be factored in when purchasingan AI-powered solution

Process-based governance andauditability

Consider implementing process-basedgovernance frameworks as suggested inthe Guidance for Understanding AI ethicsand safety This provides a basis tointegrate norms values and principlesinforming procedures and protocols thatdefine the project workflow The AlanTuring Institute calls it a lsquoPBG (Process-Based Governance) Frameworkrsquo that willprovide your team with an overview of

bull The relevant team members and rolesinvolved in each governance action

bull The relevant stages of the workflow inwhich intervention and targetedconsideration are necessary to meetgovernance goals

bull Explicit time frames for anyevaluations follow-up actions re-assessments and continuousmonitoring

bull Clear and well-defined protocols forlogging activity and for implementingmechanisms to support end-to-endauditability

Enable end-to-end auditability byimplementing process logs that gatherthe data across the modelling trainingtesting verifying and implementationphases of the project lifecycle Such a logshould allow for the variable accessibilityand presentation of information withdifferent users in mind to achieveinterpretable and justifiable AI

Model testing

Testing the model on an ongoing basis isnecessary to maintain its accuracy Aninaccurate model can result in erroneousdecisions that negatively impact citizensTherefore establish with the supplierhow the efficacy of the model will bemonitored once deployed The NationalCyber Security Centre Guidance forassessing intelligent tools for cybersecurity also highlights the importance ofthese considerations

Contract Implementationand Ongoing management


Knowledge transfer and training

Evaluate the thoroughness and logic ofthe knowledge transfer plan to ensurethat teams internally will be able to usethe tool appropriately on their own oncethe project is finalised If this cannot beguaranteed ensure in-house capacity isestablished through hiring and retainingor further maintenance contracts

Operational or service staff must haveenough knowledge of or training on theAI system to understand how to use itand act on its outputs Address the needfor staff training and support to avoid themisuse of AI applications with the AIsupplier The application must have aneasy way to report any suspectedunauthorised behaviour to relevantauthorities within or outside theorganisation

End-of-life

Consider what the end-of-life processesfor your AI system and the data shouldlook like Auditable methods of datacleaning and collection are key for you toeffectively apply the guidelines Definingend-of-contract roles and processes forboth the contracting authority and thesupplier is important Ensure the contractincludes such considerations and test ifyour contract management processesare sufficiently robust to adequatelysupport the end-of-life for the AI systemas part of the wider process it isembedded in

For information on fairness during data

selection refer to Understanding

artificial intelligence ethics and safety A

guide for the responsible design and

implementation of AI systems in the

public sector (The Alan Turing Institute)


Further reading

― AI Procurement in a Boxwwwweforumorgprojectsunlocking-public-sector-artificial-intelligence

― Assessing if artificial intelligence is the right solutionwwwgovukguidanceassessing-if-artificial-intelligence-is-the-right-solution

― Examples of real-world artificial intelligence usewwwgovukgovernmentcollectionsa-guide-to-using-artificial-intelligence-in-the-public-sectorexamples-of-artificial-intelligence-use

― Explaining decisions made with AIwwwicoorgukfor-organisationsguide-to-data-protectionkey-data-protection-themesexplaining-decisions-made-with-ai

―Managing your artificial intelligence projectwwwgovukguidancemanaging-your-artificial-intelligence-project

―National Cyber Security Centre guidance for assessing intelligenttools for cyber securitywwwncscgovukcollectionintelligent-security-tools

― Planning and preparing for artificial intelligence implementationwwwgovukguidanceplanning-and-preparing-for-artificial-intelligence-implementation

― The Technology Code of Practicewwwgovukgovernmentpublicationstechnology-code-of-practicetechnology-code-of-practice

―Understanding artificial intelligence ethics and safetywwwgovukguidanceunderstanding-artificial-intelligence-ethics-and-safety

35

GUIDELINES FOR AI PROCUREMENTDELIVERY TEAM

OFFICE FOR ARTIFICIAL INTELLIGENCESabine Gerdon

CABINET OFFICEDavid Brambley-Crawshaw

The guidelines will be updated as government use of AItechnologies evolves

If you are in the process of or considering procuring an AIsystem or interested in providing feedback on theguidelines please contact ai-procurement-guidelinesofficeforaigovuk


37


About Office for Artificial IntelligenceThe Office for Artificial Intelligence is a joint BEIS-DCMS unit responsible foroverseeing implementation of the AI and Data Grand Challenge

Its mission is to drive responsible and innovative uptake of AI technologies for thebenefit of everyone in the UK The Office for AI does this by engagingorganisations fostering growth and delivering recommendations around dataskills and public and private sector adoption

wwwgovukofficeforai Twitter officeforai

BEIS 1 Victoria Street London SW1H 0ETDCMS 100 Parliament Street London SW1A 2BQ

copy Crown copyright 2020

You may re-use this information (excluding logos) free of charge inany format or medium under the terms of the Open GovernmentLicence v30 To view this licence visit OGL or emailpsinationalarchivesgsigovuk Where we have identified any thirdparty copyright information you will need to obtain permission fromthe copyright holders concerned

Published June 2020

About World Economic ForumCentre for the Fourth Industrial RevolutionThe Centre for the Fourth Industrial Revolution is a hub for globalmultistakeholder cooperation to develop policy frameworks and advancecollaborations that accelerate the benefits of science and technology

wwwweforumorgcentre-for-the-fourth-industrial-revolution

World Economic Forum Centre for the Fourth Industrial Revolution1201 Ralston Avenue San Francisco CA 94129

3

Who should use the GuidelinesThese guidelines are aimed at central government departments who areconsidering the suitability of AI technology to improve existing services or as partof future service transformation Other public sector bodies may also follow theseguidelines

These guidelines will be useful to

bull Procurement and commercial practitioners who are responsible for theprocurement of AI technologies and management of contracts

bull Chief Data Information Technology and Innovation Officers considering thesuitability of AI systems to address technology challenges

bull Digital Delivery and Transformation teams who want to use AI technologies indigital transformation projects and programmes

bull Analysts Data Scientists and other Digital Data and Technology (DDaT)professionals who are developing project-specific requirements evaluatingusing or maintaining AI systems

bull Suppliers who want to better understand best practice technical and ethicalstandards for AI procurement in government

Public procurement is led by a framework of procurement rules and regulationsdaggerthis guidance assumes the reader has a sound working knowledge of those rulesand of the end-to-end procurement process Apply commercial judgement whenusing this guide and seek legal advice where appropriate Some research contractsmay be out of scope of procurement legislation

dagger The Public Contracts Regulations 2015 (likely to be the most relevant regulation for AI systems)the Utilities Contracts Regulations 2016 and the Concession Contracts Regulations 2016


5

Introduction 6

















Publication 28



Further reading 34

Contents


Introduction

7




Keen to learn more









9


























11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020





5

Introduction 6

















Publication 28



Further reading 34

Contents


Introduction

7




Keen to learn more









9


























11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020




5

Introduction 6

















Publication 28



Further reading 34

Contents


Introduction

7




Keen to learn more









9


























11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020





Introduction

7




Keen to learn more









9


























11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020




7




Keen to learn more









9


























11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020





9


























11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020




9


























11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020



















11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020




11

What is data ethics






















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020

















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020















bull Data ethics



















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020




















5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020













5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020







5

6















7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020









7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020








7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020







7

8

























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020























2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020
















2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020









2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020







2 Publication











bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020










bull Data Architect

bull Data Scientist

bull Data Engineer










Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020





Keen to learn more























Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020
















Keen to learn more







Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020





Keen to learn more











Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020





Publication


































Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020
























Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020











Keen to learn more















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020















the system


Keen to learn more













Model testing







End-of-life









Further reading










35







37








Published June 2020














Model testing







End-of-life









Further reading










35







37








Published June 2020








End-of-life









Further reading










35







37








Published June 2020





Further reading










35







37








Published June 2020




35







37








Published June 2020





37








Published June 2020




37








Published June 2020











Published June 2020




Documents

Guidelines for AI procurement (Mobile version) · Fourth Industrial Revolution, Government Digital Service (GDS), Government Commercial Function and Crown Commercial Service. A wide