The reality of buying AI
Gemma Nash and Matthew Mo consider the implications of the government's new Guidelines for Artificial Intelligence Procurement.
On June 8, 2020, the UK Government published Guidelines for Artificial Intelligence (AI) procurement. The purpose of the Guidelines is to provide central government departments and other public sector bodies with a set of guiding principles for purchasing AI technology.
It also covers guidance on tackling challenges that may occur during the procurement process. The Guidelines have been developed by the UK Government’s Office for AI in collaboration with the World Economic Forum Centre for the Fourth Industrial Revolution, Digital Service, Government Commercial Function and Crown Commercial Service. In connection with this project, the Office for AI has also co-created the AI Procurement in a box toolkit that provides a guide for the public sector globally to rethink the procurement of AI.
This article looks at the purpose of the Guidelines and Toolkit, provides a brief summary of both, as well as a summary of additional guidance available to those working in the public sector seeking to adopt AI solutions. While much of the UK Government’s guidance to date has been targeted at the adoption of AI in the public sector, it will also serve as a useful guide for private sector organisations, particularly where they are seeking to offer AI solutions to public sector bodies.
Purpose of the Guidelines
The Guidelines are another step towards the UK Government’s strategy to lead in the AI and data revolution, as further set out in Government’s AI Sector Deal (published May 2019) and Industrial Strategy (published November 2017). The UK Government has identified the public sector as an area that could greatly benefit from the use of AI, such as through increased productivity and efficiencies in order to benefit citizens and the wider economy.
The overall aim of the Guidelines is to further the UK Government’s plans in this respect to encourage AI adoption within the public sector to help realise these benefits. In addition, the Guidelines also promote the UK Government’s aim of setting the standard for ensuring the ethical use of AI.
The Guidelines
The Guidelines itself provide for a number of considerations that need to be factored into the procurement process. It is separated into two key sections: (i) top 10 considerations; and (ii) AI-specific considerations within the procurement process.
Top 10 Considerations
The Guidelines set out the following top 10 considerations:
1. Include your procurement within a strategy for AI adoption.
2. Make decisions in a diverse multidisciplinary team in order to mitigate against AI bias.
3. Conduct a data assessment before starting your procurement process.
4. Assess the benefits and risks of AI deployment, including defining the public benefit goal.
5. Engaging effectively with the market from the outset.
6. Establish the right route to market and focus on the challenge rather than a specific solution.
7. Develop a plan for governance and information assurance.
8. Avoid Black Box algorithms and vendor lock in.
9. Focus on the need to address technical and ethical limitations of AI deployment during your evaluation.
10. Consider the lifecycle management of the AI system. AI-specific considerations within the procurement process
In addition, the Guidelines provide for a number of factors to consider at different stages of the procurement process:
• Preparation and Planning. Considerations at this stage of the process include use of multidisciplinary teams, assessment of data and governance, conducting an AI impact assessment (for which a Data Protection Impact Assessment may be a useful starting point), carrying out preliminary market engagement, and determining the procurement approach vehicle (i.e. framework agreements).
• Publication. At this stage of the process you need to consider how best to draft your AI requirements. Among other considerations, the Guidelines recommend drafting output-based requirements which allow suppliers to best respond to those specific needs.
• Selection, Evaluation and Award. When selecting the supplier they should be able to demonstrate robust practices, such as having an internal AI ethics approach, processes for ensuring accountability over algorithmic outputs, avoiding discrimination, model testing and performance, designing for reproducibility, and security.
• Contract Implementation and Ongoing Management. The Guidelines note that AI systems “may need continued support throughout its lifecycle”. Such on-going support includes implementing process-based governance frameworks, on-going model testing, knowledge transfer and training, and end-of-life processes.
AI Procurement in a box Toolkit
In addition to the Guidelines, the Office for AI has also co-created, with the World Economic Forum, the AI Procurement in a box Toolkit which provides more detailed guidance for public sector officials looking to purchase AI solutions. The Toolkit identifies that there are challenges with public sector adoption, such as public sector officials lacking “the appropriate knowledge and expertise to make strategic buying decision for AI-powered tool”, as well as “[u]ncertainty about ethical considerations”. The Toolkit aims to tackle these challenges by providing more detailed guidance for those public sector officials.
The Toolkit is split into the following parts:
• Guidelines. A set of guidelines and principles that should be taken into account when procuring AI products.
• Workbook. A more detailed workbook to guide public sector officials through the guidelines. The workbook also includes example decision criteria that can be used when carrying out an AI impact assessment.
• Challenges and opportunities during implementation. This provides more practical guidance on the types of challenges that may arise during the procurement process. These have been developed following a series of workshops in which participants tested the guidelines against potential use cases. The following were key themes that came out of the workshops: (i) data governance; (ii) people (skills, culture and leadership); (iii) procurement processes; and (iv) ethics, i.e. accountability, liability and transparency.
Other AI Guidance for the Public Sector
This more recent development from the UK Government further adds to the growing set of guidance on the use of AI for the public sector. Other examples of such guidance include:
• Guide to using AI for the Public Sector. Published by the UK Government in June 2019, the guidance covers: (i) how to assess if AI will meet the needs of users; (ii) how the public sector can best use AI; and (iii) how to implement AI ethically, fairly and safely.
• Data Ethics Framework. The UK Government’s Data Ethics Framework provides for a set of principles on how data should be used in the public sector, setting a high standard for transparency and accountability.
• Code of conduct for data-driven health and care technology. Developed by NHSX (the arm of the NHS driving forward digital transformation within the healthcare system), the code provides a set of principles for adopting data-driven technologies (such as AI) into the UK public health sector.
• A Buyer’s Checklist for AI in Health and Care. Also developed by NHSX, the checklist provides considerations to help support those responsible for making decisions on procuring AI solutions within the healthcare sector.
• ICO’s AI Auditing Framework. The Information Commissioner’s Office (the UK data protection authority, also known as the “ICO”) began its AI Auditing project in March last year, focused on developing an AI auditing framework for ensuring compliance with data protection legislation. Following the introduction of the project, the ICO has published a number of guidance documents and blogs (see here), including a draft of its AI Auditing Framework (which was subject to public consultation earlier this year), as well as guidance on Explaining decisions made with AI. The ICO’s work has much broader application than just the public sector, as it is aimed at any organisation seeking to adopt AI solutions.
Conclusion
Public sector bodies, and private organisations seeking to supply AI solutions to the public sector, should use these guidelines to assist with the development and deployment of AI solutions. The guidelines should assist organisations with ensuring that any use of AI solutions meet high ethical and moral standards.
A key recommendation coming out of the guidance thus far is to carry out an AI impact assessment prior to adopting the technology. The assessment should weigh up the impact of the technologies use, and ensure that any risks are mitigated in advance of adoption. Organisations should also be aware that where the use of the technology involves the processing of personal data, there may be a mandatory requirement to carry out a data protection impact assessment under applicable data protection law.
Gemma Nash is an information law specialist and Matthew Mo a commercial lawyer at Bevan Brittan