by Gergely Rideg
The timeliness of regulating artificial intelligence (AI) is beyond debate. In addition to the opportunities offered by new technologies, there is now a detailed mapping of the risks surrounding AI systems.
Highly advanced AI is predicted by prominent figures from different disciplines as a technology with gigantic risks.[1] Risk factors are linked to a broad spectrum of fundamental human rights, highlighting the potential dangers of using AI.[2]
On 11/27/2023, artificial intelligence regulation reached another milestone. On this day, the National Security Agency (NSA), UK National Cyber Security Centre (NCSC-UK), U.S Cybersecurity and Infrastructure Security Agency (CISA), and other partners have released its global guidelines, Guidelines for secure AI system development for ensure the secure artificial intelligence development.
23 partners from 18 countries have joined the document. From Chile to France and Japan, agencies from all corners of the globe have contributed to the document. This reflects the cross-national challenges of AI systems from a cybersecurity perspective.
Of course, this document is not without precedent, as the National Cyber Security Centre previously published another document[3] in August 2022, entitled "Principles for the security of machine learning", which also addressed fundamental principles to address and prevent the additional risks inherent in machine learning systems.
As digitalisation becomes more and more prevalent in our lives and we spend more and more time online, the security of these technologies becomes crucial. As well as expecting IT hardware to be secure, it is equally important that software provides the right level of security. Public administrations, among others, are also trying to get on board this digitalisation trend and many services are now available either optionally or exclusively in digital form. There is no doubt that serious problems can arise when some public services become dysfunctional, for example when they stop working, as a result of cyber-attacks. Failures in software and hardware products increase the attack surface on which cybercriminals can cause damage.
The importance of these guidelines is also underlined by the fact that a number of non-governmental organisations have contributed to their development. It is encouraging to note that the list includes several major global IT companies such as Google, Microsoft, Amazon, IBM, etc., which are playing a key role in the digital development process.
Tech giants such as Microsoft, which are pioneers in the development of various AI systems at many points - think of their partnership with OpenAI - naturally have their own AI security protocols.[4] In the document "AI security risk assessment framework", published on 9 December 2021, they explicitly address machine learning security assassment, within which they specifically address the secure storage, access and integrity of the data used, the types of sensitive data, and they elaborate on the security criteria for models. It includes professional guidelines such as that the source of the data collected should be verified before use, the source should be stored with the data and documented. In addition to these, it is also more specific in its cultivation and includes criteria specifically for model teaching and development.[5]
Looking more closely at the NCSC published guidelines document we discusse, we see that, as in the Microsoft document, artificial intelligence is specifically defined as machine learning applications, and within that, all types of machine learning AI are included. In line with the technological developments and trends of the time, AI systems are typically based on machine learning models.
In order to be clear from an application perspective, the document also provides a definition of machine learning applications. „ML applications as applications that:
-
involve software components (models) that allow computers to recognise and bring context to patterns in data without the rules having to be explicitly programmed by a human
-
generate predictions, recommendations, or decisions based on statistical reasoning”
As to the reasons for the creation of this document, the document declares the following. Just as in other areas of our lives, new tools and new techniques in programme development provide opportunities for new abuses. Before the advent of car use, it was not natural for an accident to be caused by deliberately damaging the parts of a car. Artificial intelligence systems contain new vulnerabilities that can be exploited by prepared malicious actors, both on the hardware and software side. The paper draws attention to this by stressing that attackers can induce unintended behaviour in machine learning systems by using so-called adversarial machine learning, leading to the problem repeatedly mentioned by lawyers that the output of the system cannot be predicted. In the case of machine learning AI systems, this so-called black box effect is present anyway, in the case of such cyber-attacks this unintended behaviour is deliberately induced.
There can also be cases where users are allowed to perform unauthorised operations, or data poisoning, where the training data as a data domain is corrupted.
The structure of the document follows the 4 phases in the lifecycle of AI systems development, namely secure design, secure development, secure deployment, and secure operation and maintenance.
This so-called life cycle includes the requirement that once the software containing the AI systems is created, it is monitored during its use, with each update meeting cybersecurity criteria.
In fact, in each chapter, the document suggests considerations and measures that will help reduce the overall risk of the organisational AI system development process.
One such suggestion is to consider the security benefits and trade-offs of each model when selecting an AI model at the design stage of development.
By integrating well-established principles like "security-by-design" and "security-by-default," the publication outlines the existing vulnerabilities specific to AI and suggests ways to consider them during the development process. Typically, end users lack the understanding to grasp the risks associated with AI. Additionally, cybersecurity authorities emphasize the importance for AI system operators to educate users about potential risks and provide guidance on the secure utilization of these systems.[6]
While the document certainly describes the guidelines in sufficient detail for its intended purpose, the agencies emphasize that the measures and adherence to such guidelines are not a substitute for the development of a proper cybersecurity practice and risk management program or protocol. Such research itself should be used in conjunction with established cybersecurity risk management and incident response best practices. The guidelines set out in this document are closely aligned with the good practices for software development lifecycle practices that have already been identified in subsequent documents:
-
the NCSC's Secure development and deployment guidance
-
the National Institute of Standards and Technology (NIST) Secure Software Development Framework (SSDF)
The document is not binding legislation that would impose a strict obligation on companies developing AI systems, and thus cannot be used to enforce its provisions. However, it is noted that the use of new technologies and the success of AI systems are based on trust and confidence in them, which cannot be achieved by legislation alone. This document can be successful and can be considered a milestone because of the significant international partnership and contributors.
Increasing user awareness creates the need for the system to be used to comply with cybersecurity recommendations. This kind of user confidence can be achieved by applying a standard, by obtaining a certificate, with which an operator can not only be successful, but also help build a more secure digital future for its users.
[1] Around April 2023, it was reported in various media that decisive people such as Elon Musk, Yuval Noah Harari and Steve Wozniak, among others, are calling for an immediate halt to the development of certain types of artificial intelligence systems.– Die Rückkehr des Wunderglaubens – https://www.spiegel.de/wissenschaft/kuenstliche-intelligenz-die-rueckkehr-des-wunderglaubens-kolumne-a-d53eb350-b5b5-4888-9bf8-8fc510d018b8
[2] Udvary Sándor: Az önvezető gépjárművek egyes felelősségi kérdései, Pro Publico Bono – Magyar Közigazgatás, 7. évf., 2019/2, 146–155.
[3] https://www.ncsc.gov.uk/files/Principles-for-the-security-of-machine-learning.pdf
[4] Best practices for AI security risk management - https://www.microsoft.com/en-us/security/blog/2021/12/09/best-practices-for-ai-security-risk-management/
[5] Microsoft Security AI Security Risk Assessment, Best practices and guidance to secure AI systems - https://github.com/Azure/AI-Security-Risk-Assessment/blob/main/AI_Risk_Assessment_v4.1.4.pdf
[6] Internationale Cybersicherheitsbehörden veröffentlichen Leitfaden zur Entwicklung sicherer KI-Systeme - https://www.bsi.bund.de/DE/Service-Navi/Presse/Pressemitteilungen/Presse2023/231127_Leitfaden-sicher-KI-Systeme.html
