Sittings · Document

OPINION (COM(2021)0206 – C90146/2021 – 2021/0106(COD)) 2022-04-22

On the proposal for a regulation of the European Parliament and of the Council laying down harmonised rules on Artificial Intelligence (Artificial Intelligence Act) and amending certain Union Legislative Acts

Committee on the Environment, Public Health and Food Safety · Rapporteur: Susana Solís Pérez

PA_Legam

SHORT JUSTIFICATION

The proposal for Regulation of the European Parliament and the Council laying down harmonized rules on Artificial Intelligence (hereinafter “AI Act”) is part of the broader agenda to boost Europe in the digital age and achieve its environmental and climate objectives. This stems from the fact that AI currently plays a role in all aspects of European daily life activities.

AI systems will become more and more embedded into products and services therefore requiring a horizontal legislative approach as set out in the AI Act. The Rapporteur is fully aligned with this as she considers that we must establish the common rules to provide a cross-cutting approach to all sectors, including the healthcare sector. By doing so, the European Union has a chance to lead and set the standards of AI worldwide, as it has already done with data protection through GDPR. The EU could also become a global leader in niche sectors that require a very forward-looking perspective such as the regulation of neurological rights.

Overall the AI act should preserve European values, facilitating the distribution of AI’s benefits across society, protecting individuals, companies and the environment from risks while boosting innovation and employment and making Europe a leader in the field.

In this regard, the Rapporteur wants to emphasize the importance of sandboxes in certain areas (e.g. Health) and how it could be extended to other areas such as Hospitals, Health Authorities and research centers in order to reinforce and expand the leading position of the health system in all the Member States and at EU level. Health is wealth. By applying AI in health using interoperable health data we could further increase this wealth from health systems to society at large. The Rapporteur also highlights the potential implications of AI systems in mental health.

The Rapporteur for the opinion deems that the proposal insufficiently anticipates the risks of not having a common and consistent regulatory approach.

As a horizontal legislative initiative, the proposed AI Act is expected to intersect with several regulations currently in place (e.g GDPR or MDR) and several legislative initiatives that might intersect in the future such as the European Health Data Space. All these initiatives should be aligned with the AI Act to ensure a common and consistent regulatory approach therefore avoiding duplication of functions or discoordination among bodies and authorities at both the EU and Member State level.

The Rapporteur for the opinion is concerned that the AI Act does not provide sufficient protection to the environment.

The Special Eurobarometer 513 Climate change published in 2021 shows that tackling climate and environmental-related challenges is one of the main concerns for European citizens. Therefore, the Rapporteur proposes that the AI Act shall include the environment among the areas that require a high level of protection. In order to do so, the environment has been included in all the recitals and articles together with health, safety and the protection of fundamental rights. This will entail the classification as “high risk AI” of all those systems that can have major negative implications on the environment. At the same time, the Rapporteur has reinforced the right to proper redress mechanisms in case of negative environmental impacts as set out in the Aarhus Convention, and has set the principle of “Do no significant harm” as established in the Taxonomy Regulation as a limit to ensure that AI systems abide with the EU´s high level of environmental standards and rights.

The Rapporteur for the opinion considers that the AI Act shall not just cover users but must expand its scope to end recipients too.

Many of the applications mentioned in the proposed AI Act will involve not just users but end recipients. In the case of healthcare applications this distinction is crucial as there is a clear differentiation between the intended use and capabilities of patients and doctors. Therefore, the draft report now includes a new definition of end recipients and grants them the appropriate degree of transparency and provision of specific information.

AMENDMENTS

The Committee on the Environment, Public Health and Food Safety calls on the Committee on the Internal Market and Consumer Protection, as the committee responsible, to take into account the following amendments:

Amendment 1

Proposal for a regulation

Recital 1

Text proposed by the Commission

Amendment

(1) The purpose of this Regulation is to improve the functioning of the internal market by laying down a uniform legal framework in particular for the development, marketing and use of artificial intelligence in conformity with Union values. This Regulation pursues a number of overriding reasons of public interest, such as a high level of protection of health, safety and fundamental rights, and it ensures the free movement of AI-based goods and services cross-border, thus preventing Member States from imposing restrictions on the development, marketing and use of AI systems, unless explicitly authorised by this Regulation.

(1) The purpose of this Regulation is to improve the functioning of the internal market by laying down a uniform legal framework for the design, development, marketing and use of artificial intelligence and of sustainable and green artificial intelligence in conformity with Union priorities and values while minimising any risk of adverse and discriminatory impacts on people and adverse impacts on the environment. This Regulation pursues a number of overriding reasons of public interest, such as a high level of protection of biodiversity, the climate and the environment, health, safety, and fundamental rights, and it ensures the free movement of AI-based goods and services cross-border, thus preventing Member States from imposing restrictions on the design, development, marketing and use of AI systems, unless explicitly authorised by this Regulation.

Amendment 2

Proposal for a regulation

Recital 1 a (new)

Text proposed by the Commission

Amendment

(1a) This Regulation should serve as a basis to promote health, wellbeing, prevent diseases, and foster a supportive environment for healthier lifestyles in a sustainable and climate neutral way and, in particular, facilitate the achievement of the UN Sustainable Development Goals, the Paris Agreement and net-zero transition by 2050 across different sectors. Member States can establish additional requirements other than those established under this Regulation provided they are justified for reasons of public interest, the protection of legal rights, the protection of the climate, the environment and biodiversity.

Amendment 3

Proposal for a regulation

Recital 1 b (new)

Text proposed by the Commission

Amendment

(1b) This Regulation should preserve the values of the Union facilitating the distribution of artificial intelligence benefits across society, protecting individuals, companies and the environment from risks while boosting innovation and employment and making Europe a leader in the field.

Amendment 4

Proposal for a regulation

Recital 2

Text proposed by the Commission

Amendment

(2) Artificial intelligence systems (AI systems) can be easily deployed in multiple sectors of the economy and society, including cross border, and circulate throughout the Union. Certain Member States have already explored the adoption of national rules to ensure that artificial intelligence is safe and is developed and used in compliance with fundamental rights obligations. Differing national rules may lead to fragmentation of the internal market and decrease legal certainty for operators that develop or use AI systems. A consistent and high level of protection throughout the Union should therefore be ensured, while divergences hampering the free circulation of AI systems and related products and services within the internal market should be prevented, by laying down uniform obligations for operators and guaranteeing the uniform protection of overriding reasons of public interest and of rights of persons throughout the internal market based on Article 114 of the Treaty on the Functioning of the European Union (TFEU). To the extent that this Regulation contains specific rules on the protection of individuals with regard to the processing of personal data concerning restrictions of the use of AI systems for ‘real-time’ remote biometric identification in publicly accessible spaces for the purpose of law enforcement, it is appropriate to base this Regulation, in as far as those specific rules are concerned, on Article 16 of the TFEU. In light of those specific rules and the recourse to Article 16 TFEU, it is appropriate to consult the European Data Protection Board.

(2) Artificial intelligence systems (AI systems) can be easily deployed in multiple sectors of the economy and society, including cross border, and circulate throughout the Union. Certain Member States have already explored the adoption of national rules to ensure that artificial intelligence is safe and is developed and used in compliance with fundamental rights obligations. Differing national rules may lead to fragmentation of the internal market and decrease legal certainty for operators that develop or use AI systems. A consistent and high level of protection throughout the Union should therefore be ensured, while divergences hampering the free circulation of AI systems and related products and services within the internal market should be prevented, by laying down uniform obligations for operators and guaranteeing the uniform protection of overriding reasons of public interest and of rights of persons, end users and end recipients throughout the internal market based on Article 114 of the Treaty on the Functioning of the European Union (TFEU). To the extent that this Regulation contains specific rules on the protection of individuals with regard to the processing of personal data concerning restrictions of the use of AI systems for ‘real-time’ remote biometric identification in publicly accessible spaces for the purpose of law enforcement, it is appropriate to base this Regulation, in as far as those specific rules are concerned, on Article 16 of the TFEU. In light of those specific rules and the recourse to Article 16 TFEU, it is appropriate to consult the European Data Protection Board.

Amendment 5

Proposal for a regulation

Recital 3

Text proposed by the Commission

Amendment

(3) Artificial intelligence is a fast evolving family of technologies that can contribute to a wide array of economic and societal benefits across the entire spectrum of industries and social activities. By improving prediction, optimising operations and resource allocation, and personalising digital solutions available for individuals and organisations, the use of artificial intelligence can provide key competitive advantages to companies and support socially and environmentally beneficial outcomes, for example in healthcare, farming, education and training, infrastructure management, energy, transport and logistics, public services, security, justice, resource and energy efficiency, and climate change mitigation and adaptation.

(3) Artificial intelligence is a fast evolving family of technologies that can contribute to a wide array of economic, environmental and societal benefits across the entire spectrum of industries and social activities. By improving prediction, optimising operations and resource allocation, and personalising digital solutions available for individuals and organisations, the use of artificial intelligence can provide key competitive advantages to companies and support socially and environmentally beneficial outcomes, for example in food safety, by reducing the use of pesticides, the protection of nature, the conservation and restoration of biodiversity and ecosystems, environmental monitoring, access to and provision of medicines and healthcare, including mental health, carbon farming, education and training, infrastructure management, crisis management, management of natural disasters, energy, sustainable transport and logistics, public services, security, justice, resource and energy efficiency, and climate change mitigation and adaptation.

Amendment 6

Proposal for a regulation

Recital 4

Text proposed by the Commission

Amendment

(4) At the same time, depending on the circumstances regarding its specific application and use, artificial intelligence may generate risks and cause harm to public interests and rights that are protected by Union law. Such harm might be material or immaterial.

Amendment 7

Proposal for a regulation

Recital 4 a (new)

Text proposed by the Commission

Amendment

(4a) In its White Paper on "Artificial Intelligence - A European approach to excellence and trust" of 19 February 2020, the Commission recalled that artificial intelligence can contribute to finding solutions to some of the most pressing societal challenges, including the fight against climate change, biodiversity loss and environmental degradation and highlighted the potential benefits and risks of artificial intelligence in relation to safety, health and wellbeing of individuals.

Amendment 8

Proposal for a regulation

Recital 4 b (new)

Text proposed by the Commission

Amendment

(4b) Tackling climate change and environmental-related challenges and reaching the objectives of the Paris Agreement are at the core of the Communication of the Commission on the “European Green Deal”, adopted on 11 December 2019, where the Commission recalled the role of digital technologies such as artificial intelligence, 5G, cloud and edge computing and the internet of things in achieving a sustainable future and to accelerate and maximise the impact of policies to deal with climate change mitigation and adaptation, protect the environment and address biodiversity loss.

Amendment 9

Proposal for a regulation

Recital 4 c (new)

Text proposed by the Commission

Amendment

(4c) AI applications can bring environmental and economic benefits and strengthen predictive capabilities that contribute to the fight against climate change, to meeting the Sustainable Development Goals (SDGs) and to achieving our target of becoming the first climate-neutral continent. In this sense, the use of AI has the potential to reduce global greenhouse gas emissions by up to 4 % by 2030. It has also been estimated that ICT technologies are capable of reducing ten times more greenhouse gas emissions than their own footprint 1a.. In terms of environment, artificial intelligence has a strong potential to solve environmental issues such as reducing resource consumption, promoting decarbonisation, boosting the circular economy, balancing supply and demand in electricity grids or optimising logistic routes. Artificial intelligence also has the potential to contribute to strengthening environmental administration and governance by facilitating administrative decisions related to environmental heritage management, monitoring violations and environmental fraud, and encouraging citizen participation in biodiversity conservation initiatives. Moreover, the analysis of large volumes of data can lead to a better understanding of environmental challenges and a better monitoring of trends and impacts. The intelligent management of large volumes of information related to the environment also provides solutions for better environmental planning, decision-making and monitoring of environmental threats and can inform and encourage environmentally sustainable business, providing better information to reorient sustainable decision-making in different business models, and thereby improving the efficiency of resource, energy and material use through smart-Industry initiatives and machine to machine (M2M) and internet of things (IoT) technologies.

_____________

1a. https://www.europarl.europa.eu/cmsdata/231979/Working%20Paper%20-%20AIDA%20Hearing%20on%20AI%20and%20Green%20Deal.pdf

Amendment 10

Proposal for a regulation

Recital 4 d (new)

Text proposed by the Commission

Amendment

(4d) The predictive analytics capabilities provided by models based on artificial intelligence can support a better maintenance of energy systems and infrastructure, as well as anticipate the patterns of society's interaction with natural resources, thus facilitating better resource management. Artificial intelligence can serve in climate change mitigation for example through the European Union's Earth observation programme Copernicus that has the potential to be the programme needed to acquire accurate scientific information that secures science-based decision-making and implementation of the Union’s climate, biodiversity and other environmental policies. The United Kingdom's withdrawal from the European Union has caused a significant funding gap to the aforementioned Copernicus programme, which endangers the whole future of Copernicus and which needs to be acutely solved by guaranteeing sufficient funds as well as data processing support so that advanced and automatized technology and artificial intelligence based monitoring and analysing of all central environmental indicators will be guaranteed in the future. In addition, traditional identification of species has been time consuming and costly, which hinders real time biodiversity assessments. The integration of AI systems has the potential to move away from manual sorting and identification of species, which can play a role in animal conservation by allowing authorities to quickly identify, observe and monitor endangered species populations and help inform additional measures if needed for conservation purposes.

Amendment 11

Proposal for a regulation

Recital 4 e (new)

Text proposed by the Commission

Amendment

(4e) In order to ensure the dual green and digital transition and secure the technological resilience of the EU, to reduce the carbon footprint of artificial intelligence and achieve the objectives of the new European Green Deal, this Regulation contributes to the promotion of a green and sustainable artificial intelligence and to the consideration of the environmental impact of AI systems throughout their lifecycle. Sustainability should be at the core at the European artificial intelligence framework to guarantee that the development of artificial intelligence is compatible with sustainable development of environmental resources for current and future generations, at all stages of the lifecycle of artificial intelligence products; sustainability of artificial intelligence should encompass sustainable data sources, data centres, resource use, power supplies and infrastructures.

Amendment 12

Proposal for a regulation

Recital 4 f (new)

Text proposed by the Commission

Amendment

(4f) Despite the high potential solutions to the environmental and climate crisis offered by artificial intelligence, the design, training and execution of algorithms imply a high energy consumption and, consequently, high levels of carbon emissions. Artificial intelligence technologies and data centres have a high carbon footprint due to increased computational energy consumption, and high energy costs due to the volume of data stored and the amount of heat, electric and electronic waste generated, thus resulting in increased pollution. These environmental and carbon footprints are expected to increase overtime as the volume of data transferred and stored and the increasing development of artificial intelligence applications will continue to grow exponentially in the years to come. It is therefore important to minimise the climate and environmental footprint of artificial intelligence and related technologies and that AI systems and associated machinery are designed sustainably to reduce resource usage and energy consumption, thereby limiting the risks to the environment.

Amendment 13

Proposal for a regulation

Recital 4 g (new)

Text proposed by the Commission

Amendment

(4g) To promote the sustainable development of AI systems and in particular to prioritise the need for sustainable, energy efficient data centres, requirements for efficient heating and cooling of data centres should be consistent with the long-term climate and environmental standards and priorities of the Union and comply with the principle of 'do no significant harm' within the meaning of Article 17 of Regulation (EU) 2020/852 on the establishment of a framework to facilitate sustainable investment, and should be fully decarbonised by January 2050. In this regard, Member States and telecommunications providers should collect and publish information relating to the energy performance and environmental footprint for artificial intelligence technologies and date centres including information on the energy efficiency of algorithms to establish a sustainability indicator for artificial intelligence technologies. A European code of conduct for data centre energy efficiency can establish key sustainability indicators to measure four basic dimensions of a sustainable data centre, namely, how efficiently it uses energy, the proportion of energy generated from renewable energy sources, the reuse of any waste and heat and the usage of freshwater.

Amendment 14

Proposal for a regulation

Recital 5

Text proposed by the Commission

Amendment

(5) A Union legal framework laying down harmonised rules on artificial intelligence is therefore needed to foster the development, use and uptake of artificial intelligence in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, as recognised and protected by Union law. To achieve that objective, rules regulating the placing on the market and putting into service of certain AI systems should be laid down, thus ensuring the smooth functioning of the internal market and allowing those systems to benefit from the principle of free movement of goods and services. By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development of secure, trustworthy and ethical artificial intelligence, as stated by the European Council33 , and it ensures the protection of ethical principles, as specifically requested by the European Parliament34 .

(5) A Union legal framework laying down harmonised rules on artificial intelligence is therefore needed to foster the design, development, use and uptake of sustainable and green artificial intelligence in the internal market aligned with the European Green Deal objectives, that at the same time meets a high level of protection of public interests, such as health and safety, environment and climate change, food security and the protection of fundamental rights, as recognised and protected by Union law. To achieve that objective, rules regulating the placing on the market and putting into service of certain AI systems should be laid down, thus ensuring the smooth functioning of the internal market and allowing those systems to benefit from the principle of free movement of goods and services. By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development of secure, non-biased, trustworthy and ethical artificial intelligence, as stated by the European Council[33], and it ensures the protection of ethical principles, as specifically requested by the European Parliament[34].

_________________

33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6.

34 European Parliament resolution of 20 October 2020 with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies, 2020/2012(INL).

Amendment 15

Proposal for a regulation

Recital 6

Text proposed by the Commission

Amendment

(6) The notion of AI system should be clearly defined to ensure legal certainty, while providing the flexibility to accommodate future technological developments. The definition should be based on the key functional characteristics of the software, in particular the ability, for a given set of human-defined objectives, to generate outputs such as content, predictions, recommendations, or decisions which influence the environment with which the system interacts, be it in a physical or digital dimension. AI systems can be designed to operate with varying levels of autonomy and be used on a stand-alone basis or as a component of a product, irrespective of whether the system is physically integrated into the product (embedded) or serve the functionality of the product without being integrated therein (non-embedded). The definition of AI system should be complemented by a list of specific techniques and approaches used for its development, which should be kept up-to–date in the light of market and technological developments through the adoption of delegated acts by the Commission to amend that list.

(6) The notion of AI system should be clearly defined to ensure legal certainty, while providing the flexibility to accommodate future technological developments, such as neurotechnology, which may put mental privacy at risk and require legislative proposals to protect neurodata and other sensitive health data. The definition should be based on the key functional characteristics of the software, in particular the ability, for a given set of human-defined objectives, to generate outputs such as content, predictions, recommendations, or decisions which influence the environment with which the system interacts, be it in a physical or digital dimension. AI systems can be designed to operate with varying levels of autonomy and be used on a stand-alone basis or as a component of a product, irrespective of whether the system is physically integrated into the product (embedded) or serve the functionality of the product without being integrated therein (non-embedded). The definition of AI system should be complemented by a list of specific techniques and approaches used for its development, which should be kept up-to–date in the light of market and technological developments through the adoption of delegated acts by the Commission to amend that list.

Amendment 16

Proposal for a regulation

Recital 13

Text proposed by the Commission

Amendment

(13) In order to ensure a consistent and high level of protection of public interests as regards health, safety and fundamental rights, common normative standards for all high-risk AI systems should be established. Those standards should be consistent with the Charter of fundamental rights of the European Union (the Charter) and should be non-discriminatory and in line with the Union’s international trade commitments.

(13) In order to ensure a consistent and high level of protection of public interests as regards health, safety, fundamental rights or the environment, common normative standards for all high-risk AI systems should be established. Those standards should be consistent with the Charter of fundamental rights of the European Union (the Charter) and should be non-discriminatory and in line with the Union’s international trade commitments.

Amendment 17

Proposal for a regulation

Recital 13 c (new)

Text proposed by the Commission

Amendment

(13c) Artificial intelligence can unlock solutions in the health sector that could save millions of lives, respond to unmet needs, improve our standard of living and improve patient care and health outcomes, especially in diagnosis, prognosis and treatment, patient engagement, adherence, management and follow-up, clinical decision-making, including predictive analytics, screening and optimization of clinical pathways, and pathology. Artificial intelligence can also improve prevention strategies, health system management and in the organization and provision of health services and medical care, including health promotion and disease prevention interventions. It also has the potential to foster the competitiveness of stakeholders and to improve the cost-effectiveness and sustainability of health services and medical care. The Union has the potential to become a leader in the application of artificial intelligence in the healthcare sector.

Amendment 18

Proposal for a regulation

Recital 16

Text proposed by the Commission

Amendment

(16) The placing on the market, putting into service or use of certain AI systems intended to distort human behaviour, whereby physical or psychological harms are likely to occur, should be forbidden. Such AI systems deploy subliminal components individuals cannot perceive or exploit vulnerabilities of children and people due to their age, physical or mental incapacities. They do so with the intention to materially distort the behaviour of a person and in a manner that causes or is likely to cause harm to that or another person. The intention may not be presumed if the distortion of human behaviour results from factors external to the AI system which are outside of the control of the provider or the user. Research for legitimate purposes in relation to such AI systems should not be stifled by the prohibition, if such research does not amount to use of the AI system in human-machine relations that exposes natural persons to harm and such research is carried out in accordance with recognised ethical standards for scientific research.

(16) The placing on the market, putting into service or use of certain AI systems intended to distort human behaviour, whereby physical, psychological harms or disruption of the sense of oneself are likely to occur, should be forbidden. Such AI systems deploy subliminal components individuals cannot perceive or exploit vulnerabilities of children and people due to their age, physical or mental incapacities. They do so with the intention to materially distort the behaviour of a person and in a manner that causes or is likely to cause harm to that or another person or remove ultimate control over personal decision-making, with unknown manipulation from external neurotechnologies. The intention may not be presumed if the distortion of human behaviour results from factors external to the AI system which are outside of the control of the provider or the user. Research for legitimate purposes in relation to such AI systems should not be stifled by the prohibition, if such research does not amount to use of the AI system in human-machine relations that exposes natural persons to harm and such research is carried out in accordance with recognised ethical standards for scientific research.

Amendment 19

Proposal for a regulation

Recital 27

Text proposed by the Commission

Amendment

(27) High-risk AI systems should only be placed on the Union market or put into service if they comply with certain mandatory requirements. Those requirements should ensure that high-risk AI systems available in the Union or whose output is otherwise used in the Union do not pose unacceptable risks to important Union public interests as recognised and protected by Union law. AI systems identified as high-risk should be limited to those that have a significant harmful impact on the health, safety and fundamental rights of persons in the Union and such limitation minimises any potential restriction to international trade, if any.

(27) High-risk AI systems should only be placed on the Union market or put into service if they comply with certain mandatory requirements. Those requirements should ensure that high-risk AI systems available in the Union or whose output is otherwise used in the Union do not pose unacceptable risks to important Union climate priorities, environmental imperatives and public interests as recognised and protected by Union law. AI systems identified as high-risk should be limited to those that have a significant harmful impact on the health, safety, greenhouse gas emissions, crucial environmental parameters like biodiversity or soil pollution and fundamental rights of persons in the Union and such limitation minimises any potential restriction to international trade, if any.

Amendment 20

Proposal for a regulation

Recital 27 a (new)

Text proposed by the Commission

Amendment

(27a) According to the definition of the World Health Organisation (WHO), "health is a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity." In order to improve the health of the population in the Union and reduce health inequalities, it is essential not to focus only on physical health. Digital technologies and especially artificial intelligence can have a direct negative impact on mental health. At the same time, the full potential of artificial intelligence should be unleashed in the development of prediction, detection, and treatment solutions for mental health. The right to physical and mental health is a fundamental human right and universal health coverage is a SDG that all signatories have committed to achieve by 2030.

Amendment 21

Proposal for a regulation

Recital 28

Text proposed by the Commission

Amendment

(28) AI systems could produce adverse outcomes to health and safety of persons, in particular when such systems operate as components of products. Consistently with the objectives of Union harmonisation legislation to facilitate the free movement of products in the internal market and to ensure that only safe and otherwise compliant products find their way into the market, it is important that the safety risks that may be generated by a product as a whole due to its digital components, including AI systems, are duly prevented and mitigated. For instance, increasingly autonomous robots, whether in the context of manufacturing or personal assistance and care should be able to safely operate and performs their functions in complex environments. Similarly, in the health sector where the stakes for life and health are particularly high, increasingly sophisticated diagnostics systems and systems supporting human decisions should be reliable and accurate. The extent of the adverse impact caused by the AI system on the fundamental rights protected by the Charter is of particular relevance when classifying an AI system as high-risk. Those rights include the right to human dignity, respect for private and family life, protection of personal data, freedom of expression and information, freedom of assembly and of association, and non-discrimination, consumer protection, workers’ rights, rights of persons with disabilities, right to an effective remedy and to a fair trial, right of defence and the presumption of innocence, right to good administration. In addition to those rights, it is important to highlight that children have specific rights as enshrined in Article 24 of the EU Charter and in the United Nations Convention on the Rights of the Child (further elaborated in the UNCRC General Comment No. 25 as regards the digital environment), both of which require consideration of the children’s vulnerabilities and provision of such protection and care as necessary for their well-being. The fundamental right to a high level of environmental protection enshrined in the Charter and implemented in Union policies should also be considered when assessing the severity of the harm that an AI system can cause, including in relation to the health and safety of persons.

(28) AI systems could produce adverse outcomes to health and safety of persons or to the environment, in particular when such systems operate as components of products. Consistently with the objectives of Union harmonisation legislation to facilitate the free movement of products in the internal market and to ensure that only safe and otherwise compliant products find their way into the market, it is important that the safety risks that may be generated by a product as a whole due to its digital components, including AI systems, are duly prevented and mitigated. For instance, increasingly autonomous robots, whether in the context of manufacturing or personal assistance and care should be able to safely operate and performs their functions in complex environments. Similarly, in the health sector where the stakes for life and health are particularly high, increasingly sophisticated diagnostics systems and systems supporting human decisions should be reliable and accurate. Digital health should not dehumanise care nor diminish the doctor-patient relationship, but should assist doctors in diagnosing or treating patients more effectively, while keeping in mind the necessary human oversight and abiding by relevant data protection rules. The extent of the adverse impact caused by the AI system on the fundamental rights protected by the Charter is of particular relevance when classifying an AI system as high-risk. Those rights include the right to human dignity, respect for private and family life, protection of personal and health data, freedom of expression and information, freedom of assembly and of association, and non-discrimination, consumer protection, workers’ rights, rights of persons with disabilities, right to an effective remedy and to a fair trial, right of defence and the presumption of innocence, right to good administration. In addition to those rights, it is important to highlight that children have specific rights as enshrined in Article 24 of the EU Charter and in the United Nations Convention on the Rights of the Child (further elaborated in the UNCRC General Comment No. 25 as regards the digital environment), both of which require consideration of the children’s vulnerabilities and provision of such protection and care as necessary for their well-being. The fundamental right to a high level of environmental protection enshrined in the Charter and implemented in Union policies should also be considered when assessing the severity of the harm that an AI system can cause, including in relation to the health and safety of persons or to the environment.

Amendment 22

Proposal for a regulation

Recital 28 a (new)

Text proposed by the Commission

Amendment

(28a) In terms of health and patients’ rights, AI systems can play a major role in improving the health of individual patients and the performance of public health systems. However, when artificial intelligence is deployed in the context of health, patients may be exposed to potential specific risks that could lead to physical or psychological harm, for example, when different biases related to age, ethnicity, sex or disabilities in algorithms lead to incorrect diagnoses. The lack of transparency around the functioning of algorithms also makes it difficult to provide patients with the relevant information they need to exercise their rights, such as informed consent. In addition, artificial intelligence’s reliance on large amounts of data, many of them being personal data, may affect the protection of medical data, due to patients’ limited control over the use of their personal data and the cybersecurity vulnerabilities of AI systems. All of this means that special caution must to be taken when artificial intelligence is applied in clinical or healthcare settings. In order to improve the health outcomes of the population in Member States, it is essential to have a clear liability framework in place for artificial intelligence medical applications and medicine development.

Amendment 23

Proposal for a regulation

Recital 28 b (new)

Text proposed by the Commission

Amendment

(28b) AI systems not covered by Regulation (EU) 2017/745 with an impact on health or healthcare should be classified as high-risk and be covered by this Regulation. Healthcare is one of the sectors where many artificial intelligence applications are being deployed in the Union and is a market posing potential high risk to human health. Regulation (EU) 2017/745 only covers medical devices and software with an intended medical purpose, but excludes many artificial intelligence applications used in health, like artificial intelligence administrative and management systems used by healthcare professionals in hospitals or other healthcare setting and by health insurance companies and many fitness and health apps which provides artificial intelligence powered recommendations. These applications may present new challenges and risks to people, because of their health effects or the processing of sensitive health data. In order to control this, potential specific risks that could lead to any physical or psychological harm or the misuse of sensitive health data, these AI systems should be classified as high-risk.

Amendment 24

Proposal for a regulation

Recital 31

Text proposed by the Commission

Amendment

(31) The classification of an AI system as high-risk pursuant to this Regulation should not necessarily mean that the product whose safety component is the AI system, or the AI system itself as a product, is considered ‘high-risk’ under the criteria established in the relevant Union harmonisation legislation that applies to the product. This is notably the case for Regulation (EU) 2017/745 of the European Parliament and of the Council47 and Regulation (EU) 2017/746 of the European Parliament and of the Council48 , where a third-party conformity assessment is provided for medium-risk and high-risk products.

(31) The classification of an AI system as high-risk pursuant to this Regulation should not, unless duly justified, mean that the product whose safety component is the AI system, or the AI system itself as a product, is considered ‘high-risk’ under the criteria established in the relevant Union harmonisation legislation that applies to the product. This is notably the case for Regulation (EU) 2017/745 of the European Parliament and of the Council47 and Regulation (EU) 2017/746 of the European Parliament and of the Council48, where a third-party conformity assessment is provided for medium-risk and high-risk products. To ensure consistency and legal clarity, where the provided risk-based system already takes into account potential associated risks, artificial intelligence components should continue to be assessed as part of the overall device.

_________________

47 Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (OJ L 117, 5.5.2017, p. 1).

48 Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on in vitro diagnostic medical devices and repealing Directive 98/79/EC and Commission Decision 2010/227/EU (OJ L 117, 5.5.2017, p. 176).

Amendment 25

Proposal for a regulation

Recital 32

Text proposed by the Commission

Amendment

(32) As regards stand-alone AI systems, meaning high-risk AI systems other than those that are safety components of products, or which are themselves products, it is appropriate to classify them as high-risk if, in the light of their intended purpose, they pose a high risk of harm to the health and safety or the fundamental rights of persons, taking into account both the severity of the possible harm and its probability of occurrence and they are used in a number of specifically pre-defined areas specified in the Regulation. The identification of those systems is based on the same methodology and criteria envisaged also for any future amendments of the list of high-risk AI systems.

(32) As regards stand-alone AI systems, meaning high-risk AI systems other than those that are safety components of products, or which are themselves products, it is appropriate to classify them as high-risk if, in the light of their intended purpose, they pose a high risk of harm to the health, safety or the fundamental rights of persons or the environment, taking into account both the severity of the possible harm and its probability of occurrence and they are used in a number of specifically pre-defined areas specified in the Regulation. The identification of those systems is based on the same methodology and criteria envisaged also for any future amendments of the list of high-risk AI systems.

Amendment 26

Proposal for a regulation

Recital 34

Text proposed by the Commission

Amendment

(34) As regards the management and operation of critical infrastructure, it is appropriate to classify as high-risk the AI systems intended to be used as safety components in the management and operation of road traffic and the supply of water, gas, heating and electricity, since their failure or malfunctioning may put at risk the life and health of persons at large scale and lead to appreciable disruptions in the ordinary conduct of social and economic activities.

(34) As regards the management and operation of critical infrastructure, it is appropriate to classify as high-risk the AI systems intended to be used as safety components in the management and operation of road traffic, the supply of water and gas, healthcare systems, natural or anthropogenic disaster control mechanisms, heating and electricity, since their failure or malfunctioning may put at risk the life and health of persons and environment at large scale and lead to appreciable disruptions in the ordinary conduct of social and economic activities.

Amendment 27

Proposal for a regulation

Recital 37

Text proposed by the Commission

Amendment

(37) Another area in which the use of AI systems deserves special consideration is the access to and enjoyment of certain essential private and public services and benefits necessary for people to fully participate in society or to improve one’s standard of living. In particular, AI systems used to evaluate the credit score or creditworthiness of natural persons should be classified as high-risk AI systems, since they determine those persons’ access to financial resources or essential services such as housing, electricity, and telecommunication services. AI systems used for this purpose may lead to discrimination of persons or groups and perpetuate historical patterns of discrimination, for example based on racial or ethnic origins, disabilities, age, sexual orientation, or create new forms of discriminatory impacts. Considering the very limited scale of the impact and the available alternatives on the market, it is appropriate to exempt AI systems for the purpose of creditworthiness assessment and credit scoring when put into service by small-scale providers for their own use. Natural persons applying for or receiving public assistance benefits and services from public authorities are typically dependent on those benefits and services and in a vulnerable position in relation to the responsible authorities. If AI systems are used for determining whether such benefits and services should be denied, reduced, revoked or reclaimed by authorities, they may have a significant impact on persons’ livelihood and may infringe their fundamental rights, such as the right to social protection, non-discrimination, human dignity or an effective remedy. Those systems should therefore be classified as high-risk. Nonetheless, this Regulation should not hamper the development and use of innovative approaches in the public administration, which would stand to benefit from a wider use of compliant and safe AI systems, provided that those systems do not entail a high risk to legal and natural persons. Finally, AI systems used to dispatch or establish priority in the dispatching of emergency first response services should also be classified as high-risk since they make decisions in very critical situations for the life and health of persons and their property.

(37) Another area in which the use of AI systems deserves special consideration is the access to and enjoyment of certain essential private and public services, including healthcare, and benefits necessary for people to fully participate in society or to improve one’s standard of living. In particular, AI systems used to evaluate the credit score or creditworthiness of natural persons should be classified as high-risk AI systems, since they determine those persons’ access to financial resources or essential services such as housing, electricity, healthcare and telecommunication services. AI systems used for this purpose may lead to discrimination of persons or groups and perpetuate historical patterns of discrimination, for example based on racial or ethnic origins, disabilities, age, sexual orientation, or create new forms of discriminatory impacts. Considering the very limited scale of the impact and the available alternatives on the market, it is appropriate to exempt AI systems for the purpose of creditworthiness assessment and credit scoring when put into service by small-scale providers for their own use. Natural persons applying for or receiving public assistance benefits and services from public authorities are typically dependent on those benefits and services and in a vulnerable position in relation to the responsible authorities. If AI systems are used for determining whether such benefits and services should be denied, reduced, revoked or reclaimed by authorities, they may have a significant impact on persons’ livelihood, health and wellbeing, and may infringe their fundamental rights, such as the right to social protection, non-discrimination, human dignity or an effective remedy. Those systems should therefore be classified as high-risk. Nonetheless, this Regulation should not hamper the development and use of innovative approaches in the public administration, which would stand to benefit from a wider use of compliant and safe AI systems, provided that those systems do not entail a high risk to legal and natural persons. Finally, AI systems used to dispatch or establish priority in the dispatching of emergency first response services, disease prevention, diagnosis, control and treatment should also be classified as high-risk since they make decisions in very critical situations for the life and health of persons and their property or the environment.

Amendment 28

Proposal for a regulation

Recital 38

Text proposed by the Commission

Amendment

(38) Actions by law enforcement authorities involving certain uses of AI systems are characterised by a significant degree of power imbalance and may lead to surveillance, arrest or deprivation of a natural person’s liberty as well as other adverse impacts on fundamental rights guaranteed in the Charter. In particular, if the AI system is not trained with high quality data, does not meet adequate requirements in terms of its accuracy or robustness, or is not properly designed and tested before being put on the market or otherwise put into service, it may single out people in a discriminatory or otherwise incorrect or unjust manner. Furthermore, the exercise of important procedural fundamental rights, such as the right to an effective remedy and to a fair trial as well as the right of defence and the presumption of innocence, could be hampered, in particular, where such AI systems are not sufficiently transparent, explainable and documented. It is therefore appropriate to classify as high-risk a number of AI systems intended to be used in the law enforcement context where accuracy, reliability and transparency is particularly important to avoid adverse impacts, retain public trust and ensure accountability and effective redress. In view of the nature of the activities in question and the risks relating thereto, those high-risk AI systems should include in particular AI systems intended to be used by law enforcement authorities for individual risk assessments, polygraphs and similar tools or to detect the emotional state of natural person, to detect ‘deep fakes’, for the evaluation of the reliability of evidence in criminal proceedings, for predicting the occurrence or reoccurrence of an actual or potential criminal offence based on profiling of natural persons, or assessing personality traits and characteristics or past criminal behaviour of natural persons or groups, for profiling in the course of detection, investigation or prosecution of criminal offences, as well as for crime analytics regarding natural persons. AI systems specifically intended to be used for administrative proceedings by tax and customs authorities should not be considered high-risk AI systems used by law enforcement authorities for the purposes of prevention, detection, investigation and prosecution of criminal offences.

(38) Actions by law enforcement authorities involving certain uses of AI systems are characterised by a significant degree of power imbalance and may lead to surveillance, arrest or deprivation of a natural person’s liberty as well as other adverse impacts on fundamental rights guaranteed in the Charter. In particular, if the AI system is not trained with high quality data, does not meet adequate requirements in terms of its accuracy or robustness, or is not properly designed and tested before being put on the market or otherwise put into service, it may single out people in a discriminatory or otherwise incorrect or unjust manner. Furthermore, the exercise of important procedural fundamental rights, such as the right to an effective remedy, including the right to access to justice for environmental matters as established in the UNECE Convention on Access to Information, Public Participation in Decision-making and Access to Justice in Environmental Matters (“tAarhus Convention”) applicable to the Union institutions and bodies through Regulation 1367/2006 of the European Parliament and of the Council* and to a fair trial as well as the right of defence and the presumption of innocence, could be hampered, in particular, where such AI systems are not sufficiently transparent, explainable and documented. It is therefore appropriate to classify as high-risk a number of AI systems intended to be used in the law enforcement context where accuracy, reliability and transparency is particularly important to avoid adverse impacts, retain public trust and ensure accountability and effective redress. In view of the nature of the activities in question and the risks relating thereto, those high-risk AI systems should include in particular AI systems intended to be used by law enforcement authorities for individual risk assessments, polygraphs and similar tools or to detect the emotional state of natural person, to detect ‘deep fakes’, for the evaluation of the reliability of evidence in criminal proceedings, for predicting the occurrence or reoccurrence of an actual or potential criminal offence based on profiling of natural persons, or assessing personality traits and characteristics or past criminal behaviour of natural persons or groups, for profiling in the course of detection, investigation or prosecution of criminal offences, as well as for crime analytics regarding natural persons. AI systems specifically intended to be used for administrative proceedings by tax and customs authorities should not be considered high-risk AI systems used by law enforcement authorities for the purposes of prevention, detection, investigation and prosecution of criminal offences.

_________________

* Regulation (EC) No 1367/2006 of the European Parliament and of the Council of 6 September 2006 on the application of the provisions of the Aarhus Convention on Access to Information, Public Participation in Decision-making and Access to Justice in Environmental Matters to Community institutions and bodies (OJ L 264, 25.09.2006, p. 13).

Amendment 29

Proposal for a regulation

Recital 40

Text proposed by the Commission

Amendment

(40) Certain AI systems intended for the administration of justice and democratic processes should be classified as high-risk, considering their potentially significant impact on democracy, rule of law, individual freedoms as well as the right to an effective remedy and to a fair trial. In particular, to address the risks of potential biases, errors and opacity, it is appropriate to qualify as high-risk AI systems intended to assist judicial authorities in researching and interpreting facts and the law and in applying the law to a concrete set of facts. Such qualification should not extend, however, to AI systems intended for purely ancillary administrative activities that do not affect the actual administration of justice in individual cases, such as anonymisation or pseudonymisation of judicial decisions, documents or data, communication between personnel, administrative tasks or allocation of resources.

(40) Certain AI systems intended for the administration of justice and democratic processes should be classified as high-risk, considering their potentially significant impact on democracy, rule of law, individual freedoms as well as the right to an effective remedy and to a fair trial. In particular, to address the risks of potential biases, errors and opacity, as well as related serious ethical concerns regarding machine autonomy and decision-making, it is appropriate to qualify as high-risk AI systems intended to assist judicial authorities in researching and interpreting facts and the law and in applying the law to a concrete set of facts. Such qualification should not extend, however, to AI systems intended for purely ancillary administrative activities that do not affect the actual administration of justice in individual cases, such as anonymisation or pseudonymisation of judicial decisions, documents or data, communication between personnel, administrative tasks or allocation of resources.

Amendment 30

Proposal for a regulation

Recital 43

Text proposed by the Commission

Amendment

(43) Requirements should apply to high-risk AI systems as regards the quality of data sets used, technical documentation and record-keeping, transparency and the provision of information to users and end recipients, human oversight, and robustness, accuracy and cybersecurity. Those requirements are necessary to effectively mitigate the risks for health, safety and fundamental rights and more widely for the climate and the environment, as applicable in the light of the intended purpose of the system, and no other less trade restrictive measures are reasonably available, thus avoiding unjustified restrictions to trade. To avoid any potential misalignment or duplication, the Commission should clearly determine where any relevant sectoral legislation may take precedence concerning data governance and any associated management practices or quality criteria.

Amendment 31

Proposal for a regulation

Recital 43 a (new)

Text proposed by the Commission

Amendment

(43a) These requirements should also take into account the international environmental and human rights principles and instruments including the Aarhus Convention, Resolution 48/13 adopted by the Human Rights Council on 8 October 2021 on the human right to a clean, healthy and sustainable environment, as well as international climate commitments outlined in the 2018 IPCC Special Report to limit global average temperatures to 1,5 degrees.

Amendment 32

Proposal for a regulation

Recital 43 b (new)

Text proposed by the Commission

Amendment

(43b) The Union commits to progressing towards the recognition of the right to a clean, healthy and sustainable environment, as laid out in Resolution 48/13 of the UN Human Rights Council.

Amendment 33

Proposal for a regulation

Recital 44

Text proposed by the Commission

Amendment

(44) High data quality is essential for the performance of many AI systems, especially when techniques involving the training of models are used, with a view to ensure that the high-risk AI system performs as intended and safely and it does not become the source of discrimination prohibited by Union law. High quality training, validation and testing data sets require the implementation of appropriate data governance and management practices. Training, validation and testing data sets should be sufficiently relevant, representative and free of errors and complete in view of the intended purpose of the system. They should also have the appropriate statistical properties, including as regards the persons or groups of persons on which the high-risk AI system is intended to be used. In particular, training, validation and testing data sets should take into account, to the extent required in the light of their intended purpose, the features, characteristics or elements that are particular to the specific geographical, behavioural or functional setting or context within which the AI system is intended to be used. In order to protect the right of others from the discrimination that might result from the bias in AI systems, the providers shouldbe able to process also special categories of personal data, as a matter of substantial public interest, in order to ensure the bias monitoring, detection and correction in relation to high-risk AI systems.

(44) High data quality is essential for the performance of many AI systems, especially when techniques involving the training of models are used, with a view to ensure that the high-risk AI system performs as intended and safely and it does not become the source of discrimination prohibited by Union law. High quality training, validation and testing data sets require the implementation of appropriate data governance and management practices. Training, validation and testing data sets should be sufficiently relevant, representative and free of errors and complete in view of the intended purpose of the system. They should also have the appropriate statistical properties, including as regards the persons or groups of persons on which the high-risk AI system is intended to be used. In particular, training, validation and testing data sets should take into account, to the extent required in the light of their intended purpose, the features, characteristics or elements that are particular to the specific geographical, behavioural or functional setting or context within which the AI system is intended to be used. In order to protect the right of others from the discrimination that might result from the bias in AI systems, that is, to ensure algorithmic non-discrimination, the providers should be able to process also special categories of personal data, as a matter of substantial public interest, in order to ensure the bias monitoring, detection and correction in relation to high-risk AI systems.

Amendment 34

Proposal for a regulation

Recital 45

Text proposed by the Commission

Amendment

(45) For the development of high-risk AI systems, certain actors, such as providers, notified bodies and other relevant entities, such as digital innovation hubs, testing experimentation facilities and researchers, should be able to access and use high quality datasets within their respective fields of activities which are related to this Regulation. European common data spaces established by the Commission and the facilitation of data sharing between businesses and with government in the public interest will be instrumental to provide trustful, accountable and non-discriminatory access to high quality data for the training, validation and testing of AI systems. For example, in health, the European health data space will facilitate non-discriminatory access to health data and the training of artificial intelligence algorithms on those datasets, in a privacy-preserving, secure, timely, transparent and trustworthy manner, and with an appropriate institutional governance. Relevant competent authorities, including sectoral ones, providing or supporting the access to data may also support the provision of high-quality data for the training, validation and testing of AI systems.

(45) For the development of high-risk AI systems, certain actors, such as providers, notified bodies and other relevant entities, such as digital innovation hubs, research and scientific institutes, health authorities, hospitals, testing experimentation facilities and researchers, should be able to have increased access and use of high quality datasets within their respective fields of activities which are related to this Regulation. European common data spaces established by the Commission and the facilitation of data sharing between businesses and with government in the public interest will be instrumental to provide trustful, accountable and non-discriminatory access to high quality data for the training, validation and testing of AI systems. For example, in health, the European health data space will facilitate non-discriminatory access to health data and the training of artificial intelligence algorithms on those datasets, in a privacy-preserving, secure, timely, transparent and trustworthy manner, and with an appropriate institutional governance. Artificial intelligence applications for medicines and healthcare should support the interoperability of health data and epidemiological information to better provide doctors with the necessary support to diagnose and treat patients more effectively to improve patient outcomes. Member States should put in place incentives to ensure that the data is completely interoperable to unlock the full potential of Europe´s high quality healthcare services, while complying with Regulation (EU) 2016/679. Relevant competent authorities, including sectoral ones, providing or supporting the access to data may also support the provision of high-quality data for the training, validation and testing of AI systems.

Amendment 35

Proposal for a regulation

Recital 46

Text proposed by the Commission

Amendment

(46) Having information on how high-risk AI systems have been developed and how they perform throughout their lifecycle is essential to verify compliance with the requirements under this Regulation. This requires keeping records and the availability of a technical documentation, containing information which is necessary to assess the compliance of the AI system with the relevant requirements. Such information should include the general characteristics, capabilities and limitations of the system, algorithms, data, training, testing and validation processes used as well as documentation on the relevant risk management system. The technical documentation should be kept up to date.

(46) Having information on how high-risk AI systems have been designed and developed and how they perform throughout their lifecycle is essential to verify compliance with the requirements under this Regulation. This requires keeping records and the availability of a technical documentation, containing information which is necessary to assess the compliance of the AI system with the relevant requirements. Such information should include the general characteristics, capabilities and limitations of the system, algorithms, data, training, testing and validation processes used as well as documentation on the relevant risk management system. The technical documentation should be kept up to date.

Amendment 36

Proposal for a regulation

Recital 46a

Text proposed by the Commission

Amendment

(46a) Artificial intelligence should contribute to the European Green Deal and the green transition and be used by governments and businesses to benefit people and the planet. In this regard, the Commission and Member States should encourage the design, development, deployment and use of energy efficient and low carbon AI systems through the development of best practice procedures and the publication of guidelines and methodologies. In addition, the Commission should develop a procedure, methodology, minimum standards and scale, to be applied to all AI systems on a voluntary basis, to facilitate a multicriteria disclosure of information on the energy used in the training, re-training, fine tuning and execution of AI systems and a quantitative assessment of how the AI system affects climate change mitigation and adaption, including their carbon intensity.

Amendment 37

Proposal for a regulation

Recital 47

Text proposed by the Commission

Amendment

(47) To address the opacity that may make certain AI systems incomprehensible to or too complex for natural persons, a certain degree of transparency should be required for high-risk AI systems. Users should be able to interpret the system output and use it appropriately. High-risk AI systems should therefore be accompanied by relevant documentation and instructions of use and include concise and clear information, including in relation to possible risks to fundamental rights and discrimination, where appropriate.

(47) To address the opacity that may make certain AI systems incomprehensible to or too complex for natural persons, a certain degree of transparency should be required for high-risk AI systems. Users and end recipients should be able to interpret the system output and use it appropriately. High-risk AI systems should therefore be accompanied by relevant documentation and instructions of use and include concise and clear information, including in relation to possible risks to fundamental rights and discrimination, where appropriate.

Amendment 38

Proposal for a regulation

Recital 48

Text proposed by the Commission

Amendment

(48) High-risk AI systems should be designed and developed in such a way that natural persons can oversee their functioning. For this purpose, appropriate human oversight measures should be identified by the provider of the system before its placing on the market or putting into service. In particular, where appropriate, such measures should guarantee that the system is subject to in-built operational constraints that cannot be overridden by the system itself and is responsive to the human operator, and that the natural persons to whom human oversight has been assigned have the necessary competence, training and authority to carry out that role.

Amendment 39

Proposal for a regulation

Recital 48 a (new)

Text proposed by the Commission

Amendment

(48a) The recommendations regarding human oversight from the opinion of the Committee on the Environment, Public Health and Food Safety for the Committee on Legal Affairs with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies (2020/2012(INL)) are to complement this Regulation.

Amendment 40

Proposal for a regulation

Recital 49

Text proposed by the Commission

Amendment

(49) High-risk AI systems should perform consistently throughout their lifecycle and meet an appropriate level of accuracy, robustness and cybersecurity in accordance with the generally acknowledged state of the art. The level of accuracy and accuracy metrics should be communicated to the users.

Amendment 41

Proposal for a regulation

Recital 50

Text proposed by the Commission

Amendment

(50) The technical robustness is a key requirement for high-risk AI systems. They should be resilient against risks connected to the limitations of the system (e.g. errors, faults, inconsistencies, unexpected situations) as well as against malicious actions that may compromise the security of the AI system and result in harmful or otherwise undesirable behaviour. Failure to protect against these risks could lead to safety impacts or negatively affect the fundamental rights, for example due to erroneous decisions or wrong or biased outputs generated by the AI system.

(50) The technical robustness is a key requirement for high-risk AI systems. They should be resilient against risks connected to the limitations of the system (e.g. errors, faults, inconsistencies, unexpected situations) as well as against malicious actions that may compromise the security of the AI system and result in harmful or otherwise undesirable behaviour. Failure to protect against these risks could lead to safety impacts, negative environmental implications, or negatively affect the fundamental rights, for example due to erroneous decisions or wrong or biased outputs generated by the AI system.

Amendment 42

Proposal for a regulation

Recital 54

Text proposed by the Commission

Amendment

(54) The provider should establish a sound quality management system, ensure the accomplishment of the required conformity assessment procedure, draw up the relevant documentation and establish a robust post-market monitoring system. Public authorities which put into service high-risk AI systems for their own use may adopt and implement the rules for the quality management system as part of the quality management system adopted at a national or regional level, as appropriate, taking into account the specificities of the sector and the competences and organisation of the public authority in question.

(54) The provider should establish a sound quality management system, ensure the accomplishment of the required conformity assessment procedure, draw up the relevant documentation, including the energy consumption and carbon intensity of the system and establish a robust post-market monitoring system. Public authorities which put into service high-risk AI systems for their own use may adopt and implement the rules for the quality management system as part of the quality management system adopted at a national or regional level, as appropriate, taking into account the specificities of the sector and the competences and organisation of the public authority in question. Where this overlaps with any relevant and applicable sectoral legislation, the relevant terminology should be appropriately harmonised to avoid any unnecessary fragmentation.

Amendment 43

Proposal for a regulation

Recital 59 a (new)

Text proposed by the Commission

Amendment

(59a) Considering the specific nature and potential uses of AI systems which can be addressed to natural persons who are not users or operators, it is important to ensure the protection of certain rights, notably regarding transparency and the provision of information, to end recipients such as patients of healthcare services, students, consumers, etc. The current legislation should aim at providing the appropriate type and degree of transparency as well as the provision of specific information to end recipients and establish a clear difference with users as it can increase the protection and usability of AI systems and components.

Amendment 44

Proposal for a regulation

Recital 68

Text proposed by the Commission

Amendment

(68) Under certain conditions, rapid availability of innovative technologies may be crucial for health and safety of persons and for society as a whole. It is thus appropriate that under exceptional reasons of public security or protection of life and health of natural persons and the protection of industrial and commercial property, Member States could authorise the placing on the market or putting into service of AI systems which have not undergone a conformity assessment.

(68) Under certain conditions, rapid availability of innovative technologies may be crucial for health and safety of persons, the environment and climate change and for society as a whole. It is thus appropriate that under exceptional reasons of public security or protection of life and health of natural persons, the protection of the environment and the protection of industrial and commercial property, Member States could authorise the placing on the market or putting into service of AI systems which have not undergone a conformity assessment.

Amendment 45

Proposal for a regulation

Recital 70

Text proposed by the Commission

Amendment

(70) Certain AI systems intended to interact with natural persons or to generate content may pose specific risks of impersonation or deception irrespective of whether they qualify as high-risk or not. In certain circumstances, the use of these systems should therefore be subject to specific transparency obligations without prejudice to the requirements and obligations for high-risk AI systems. In particular, natural persons should be notified that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use. Moreover, natural persons should be notified when they are exposed to an emotion recognition system or a biometric categorisation system. Such information and notifications should be provided in accessible formats for persons with disabilities. Further, users, who use an AI system to generate or manipulate image, audio or video content that appreciably resembles existing persons, places or events and would falsely appear to a person to be authentic, should disclose that the content has been artificially created or manipulated by labelling the artificial intelligence output accordingly and disclosing its artificial origin.

(70) Certain AI systems intended to interact with natural persons or to generate content may pose specific risks of impersonation or deception irrespective of whether they qualify as high-risk or not. In these circumstances, the use of these systems should therefore be subject to specific transparency obligations without prejudice to the requirements and obligations for high-risk AI systems. In particular, natural persons should be notified that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use. Moreover, natural persons should be notified when they are exposed to an emotion recognition system or a biometric categorisation system. Such information and notifications should be provided in a timely and accessible format paying particular attention to persons with disabilities. Further, users, who use an AI system to generate or manipulate image, audio or video content that appreciably resembles existing persons, places or events and would falsely appear to a person to be authentic, should disclose that the content has been artificially created or manipulated by labelling the artificial intelligence output accordingly and disclosing its artificial origin.

Amendment 46

Proposal for a regulation

Recital 71

Text proposed by the Commission

Amendment

(71) Artificial intelligence is a rapidly developing family of technologies that requires novel forms of regulatory oversight and a safe space for experimentation, while ensuring responsible innovation and integration of appropriate safeguards and risk mitigation measures. To ensure a legal framework that is innovation-friendly, future-proof and resilient to disruption, national competent authorities from one or more Member States should be encouraged to establish artificial intelligence regulatory sandboxes to facilitate the development and testing of innovative AI systems under strict regulatory oversight before these systems are placed on the market or otherwise put into service.

(71) Artificial intelligence is a rapidly developing family of technologies that requires novel and effective forms of regulatory oversight and a safe space for experimentation, while ensuring responsible innovation and integration of appropriate safeguards and risk mitigation measures. To ensure a legal framework that is innovation-friendly, future-proof, sustainable and resilient to disruption, national competent authorities from one or more Member States should be encouraged to establish artificial intelligence regulatory sandboxes to facilitate the development and testing of innovative AI systems, with particular emphasis on the promotion of sustainable and green AI systems, under strict regulatory oversight before these systems are placed on the market or otherwise put into service.

Amendment 47

Proposal for a regulation

Recital 72

Text proposed by the Commission

Amendment

(72) The objectives of the regulatory sandboxes should be to foster AI innovation by establishing a controlled experimentation and testing environment in the development and pre-marketing phase with a view to ensuring compliance of the innovative AI systems with this Regulation and other relevant Union and Member States legislation; to enhance legal certainty for innovators and the competent authorities’ oversight and understanding of the opportunities, emerging risks and the impacts of AI use, and to accelerate access to markets, including by removing barriers for small and medium enterprises (SMEs) and start-ups. To ensure uniform implementation across the Union and economies of scale, it is appropriate to establish common rules for the regulatory sandboxes’ implementation and a framework for cooperation between the relevant authorities involved in the supervision of the sandboxes. This Regulation should provide the legal basis for the use of personal data collected for other purposes for developing certain AI systems in the public interest within the AI regulatory sandbox, in line with Article 6(4) of Regulation (EU) 2016/679, and Article 6 of Regulation (EU) 2018/1725, and without prejudice to Article 4(2) of Directive (EU) 2016/680. Participants in the sandbox should ensure appropriate safeguards and cooperate with the competent authorities, including by following their guidance and acting expeditiously and in good faith to mitigate any high-risks to safety and fundamental rights that may arise during the development and experimentation in the sandbox. The conduct of the participants in the sandbox should be taken into account when competent authorities decide whether to impose an administrative fine under Article 83(2) of Regulation 2016/679 and Article 57 of Directive 2016/680.

(72) The objectives of the regulatory sandboxes should be to foster AI innovation by establishing a controlled experimentation and testing environment in the development and pre-marketing phase with a view to ensuring compliance of the innovative AI systems with this Regulation and other relevant Union and Member States legislation; to enhance legal certainty for innovators and the competent authorities’ oversight and understanding of the opportunities, emerging risks and the impacts of AI use, and to accelerate access to markets, including by removing barriers for small and medium enterprises (SMEs) and start-ups. To ensure uniform implementation across the Union and economies of scale, it is appropriate to establish common rules for the regulatory sandboxes’ implementation and a framework for cooperation between the relevant authorities involved in the supervision of the sandboxes. This Regulation should provide the legal basis for the use of personal data collected for other purposes for developing certain AI systems in the public interest within the AI regulatory sandbox, in line with Article 6(4) of Regulation (EU) 2016/679, and Article 6 of Regulation (EU) 2018/1725, and without prejudice to Article 4(2) of Directive (EU) 2016/680. Participants in the sandbox should ensure appropriate safeguards and cooperate with the competent authorities, including by following their guidance and acting expeditiously and in good faith to mitigate any high-risks to safety, health, the environment and fundamental rights that may arise during the development and experimentation in the sandbox. The conduct of the participants in the sandbox should be taken into account when competent authorities decide whether to impose an administrative fine under Article 83(2) of Regulation 2016/679 and Article 57 of Directive 2016/680.

Amendment 48

Proposal for a regulation

Recital 73 a (new)

Text proposed by the Commission

Amendment

(73a) In order to promote a more sustainable and greener innovation, the Commission and Member States should publish guidelines and methodologies for efficient algorithms that provide data and pre-trained models in view of a rationalisation of training activity. The development of best practice procedures would also support the identification and subsequent development of solutions to the most pressing environmental challenges of AI systems, including on the development of the green AI label.

Amendment 49

Proposal for a regulation

Recital 74

Text proposed by the Commission

Amendment

(74) In order to minimise the risks to implementation resulting from lack of knowledge and expertise in the market as well as to facilitate compliance of providers and notified bodies with their obligations under this Regulation, the AI-on demand platform, the European Digital Innovation Hubs and the Testing and Experimentation Facilities established by the Commission and the Member States at national or EU level should possibly contribute to the implementation of this Regulation. Within their respective mission and fields of competence, they may provide in particular technical and scientific support to providers and notified bodies.

(74) In order to minimise the risks to implementation resulting from lack of knowledge and expertise in the market as well as to facilitate compliance of providers and notified bodies with their obligations under this Regulation, the AI-on demand platform, the European Digital Innovation Hubs, the European Institute of Innovation and Technology, and the Testing and Experimentation Facilities established by the Commission and the Member States at national or EU level should possibly contribute to the implementation of this Regulation. Within their respective mission and fields of competence, they may provide in particular technical and scientific support to providers and notified bodies.

Amendment 50

Proposal for a regulation

Recital 76

Text proposed by the Commission

Amendment

(76) In order to facilitate a smooth, effective and harmonised implementation of this Regulation a European Artificial Intelligence Board should be established. The Board should be responsible for a number of advisory tasks, including issuing opinions, recommendations, advice or guidance on matters related to the implementation of this Regulation, the establishment of an artificial intelligence sustainability taskforce for the sustainable development of artificial intelligence and the development towards a harmonised criteria for sustainable technical specifications, existing standards and best practice regarding the requirements established in this Regulation and to provide expert advice to and assisting the Commission on specific questions related to artificial intelligence to better address emerging cross-border challenges arising from rapid technological development.

Amendment 51

Proposal for a regulation

Recital 76 a (new)

Text proposed by the Commission

Amendment

(76a) To ensure a common and consistent approach regarding the deployment and implementation of AI systems in the various areas and sectors concerned and to exploit potential synergies and complementarities, the Board should cooperate closely with other relevant sectoral advisory groups established at Union level, such as boards, committees and expert groups, including organisations from the civil society such as NGOs, consumer associations, and industry representatives with competence in areas related to digital technologies or artificial intelligence, such as governance, exchange, access or use and re-use of data, including health data or environmental information, while avoiding duplication of work.

Amendment 52

Proposal for a regulation

Recital 78

Text proposed by the Commission

Amendment

(78) In order to ensure that providers of high-risk AI systems can take into account the experience on the use of high-risk AI systems for improving their systems and the design and development process or can take any possible corrective action in a timely manner, all providers should have a post-market monitoring system in place. This system is also key to ensure that the possible risks emerging from AI systems which continue to ‘learn’ after being placed on the market or put into service can be more efficiently and timely addressed. In this context, providers should also be required to have a system in place to report to the relevant authorities any serious incidents or any breaches to national and Union law protecting fundamental rights resulting from the use of their AI systems.

Amendment 53

Proposal for a regulation

Recital 81

Text proposed by the Commission

Amendment

(81) The development of AI systems other than high-risk AI systems in accordance with the requirements of this Regulation may lead to a larger uptake of trustworthy artificial intelligence in the Union. Providers of non-high-risk AI systems should be encouraged to create codes of conduct intended to foster the voluntary application of the mandatory requirements applicable to high-risk AI systems. Providers should also be encouraged to apply on a voluntary basis additional requirements related, for example, to environmental sustainability, accessibility to persons with disability, stakeholders’ participation in the design and development of AI systems, and diversity of the development teams. The Commission may develop initiatives, including of a sectorial nature, to facilitate the lowering of technical barriers hindering cross-border exchange of data for AI development, including on data access infrastructure, semantic and technical interoperability of different types of data.

(81) The development of AI systems other than high-risk AI systems in accordance with the requirements of this Regulation should lead to a larger uptake of trustworthy artificial intelligence in the Union. Providers of non-high-risk AI systems should be encouraged to create codes of conduct intended to foster the voluntary application of the mandatory requirements applicable to high-risk AI systems. Providers should also be encouraged to apply on a voluntary basis additional requirements related, for example, to take a risk-based approach to focus on the direct and indirect effects on environmental sustainability, energy efficiency and carbon intensity, accessibility to persons with disability, stakeholders’ participation in the design and development of AI systems, and diversity of the development teams. The Commission may develop initiatives, including of a sectorial nature, to facilitate the lowering of technical barriers hindering cross-border exchange of data for AI development, including on data access infrastructure, semantic and technical interoperability of different types of data.

Amendment 54

Proposal for a regulation

Article 1 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) harmonised rules for the placing on the market, the putting into service and the use of artificial intelligence systems (‘AI systems’) in the Union;

(a) harmonised rules to ensure the protection for the public interest, the health and safety of consumers and the protection of the environment for the placing on the market, the putting into service and the use of artificial intelligence systems (‘AI systems’) in the Union;

Amendment 55

Proposal for a regulation

Article 3 – paragraph 1 – point 4 a (new)

Text proposed by the Commission

Amendment

(4a) ‘end recipient’ means any natural or legal person, other than an operator, to whom the output of an AI system is intended or to whom that output is provided;

Amendment 56

Proposal for a regulation

Article 3 – paragraph 1 – point 14

Text proposed by the Commission

Amendment

(14) ‘safety component of a product or system’ means a component of a product or of a system which fulfils a safety function for that product or system or the failure or malfunctioning of which endangers the health and safety of persons or property;

Amendment 57

Proposal for a regulation

Article 3 – paragraph 1 – point 15

Text proposed by the Commission

Amendment

(15) ‘instructions for use’ means the information provided by the provider to inform the user of in particular an AI system’s intended purpose and proper use, inclusive of the specific geographical, behavioural or functional setting within which the high-risk AI system is intended to be used;

(15) ‘instructions for use’ means the information provided by the provider to inform the user and end recipient of in particular an AI system’s intended purpose and proper use, inclusive of the specific geographical, behavioural or functional setting within which the high-risk AI system is intended to be used;

Amendment 58

Proposal for a regulation

Article 3 – paragraph 1 – point 24 b (new)

Text proposed by the Commission

Amendment

(24b) “green AI label” means a label by which the less carbon intensive and most energy efficient AI systems are recognised and that promotes the techniques and procedures used for a better efficiency;

Amendment 59

Proposal for a regulation

Article 3 – paragraph 1 – point 34

Text proposed by the Commission

Amendment

(34) ‘emotion recognition system’ means an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data;

(34) ‘emotion recognition system’ means an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric or neurological data;

Amendment 60

Proposal for a regulation

Article 5 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) the placing on the market, putting into service or use of an AI system that deploys subliminal techniques beyond a person’s consciousness in order to materially distort a person’s behaviour in a manner that causes or is likely to cause that person or another person physical or psychological harm;

(a) the placing on the market, putting into service or use of an AI system that deploys subliminal, psychological techniques beyond a person’s consciousness in order to materially distort a person’s behaviour in a manner that causes or is likely to cause that person or another person economic, physical or psychological harm;

Justification

Discriminatory AI driven price optimisation strategies should not be permitted. For example: insurance firms targeting price increases at consumers who are perceived by AI systems as less likely to switch providers

Amendment 61

Proposal for a regulation

Article 5 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) the placing on the market, putting into service or use of an AI system that exploits any of the vulnerabilities of a specific group of persons due to their age, physical or mental disability, in order to materially distort the behaviour of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;

(b) the placing on the market, putting into service or use of an AI system that exploits any of the vulnerabilities of a specific group of persons due to their age, physical or mental disability including addiction, bereavement or distress, in order to materially distort the behaviour of a person pertaining to that group in a manner that causes or is likely to cause that person or another person economic, physical or psychological harm;

Justification

The protection of vulnerable citizens should include those suffering from temporary vulnerabilities including addiction or bereavement to ensure protection from the use of AI driven persuasion profiling used in dating and gambling websites for example.

Amendment 62

Proposal for a regulation

Article 7 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) the AI systems are intended to be used in any of the areas listed in points 1 to 8 of Annex III;

(a) the AI systems are intended to be used in any of the areas listed in points 1 to 9 of Annex III;

Amendment 63

Proposal for a regulation

Article 7 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) the AI systems pose a risk of harm to the health and safety, or a risk of adverse impact on fundamental rights, that is, in respect of its severity and probability of occurrence, equivalent to or greater than the risk of harm or of adverse impact posed by the high-risk AI systems already referred to in Annex III.

(b) the AI systems pose a risk of harm to the health and safety, or a risk of adverse impact on climate change mitigation and adaptation, the environment and fundamental rights, that is, in respect of its severity and probability of occurrence, equivalent to or greater than the risk of harm or of adverse impact posed by the high-risk AI systems already referred to in Annex III.

Amendment 64

Proposal for a regulation

Article 7 – paragraph 2 – introductory part

Text proposed by the Commission

Amendment

2. When assessing for the purposes of paragraph 1 whether an AI system poses a risk of harm to the health and safety or a risk of adverse impact on fundamental rights that is equivalent to or greater than the risk of harm posed by the high-risk AI systems already referred to in Annex III, the Commission shall take into account the following criteria:

2. When assessing for the purposes of paragraph 1 whether an AI system poses a risk of harm to the health and safety or a risk of adverse impact on the climate, the environment or fundamental rights that is equivalent to or greater than the risk of harm posed by the high-risk AI systems already referred to in Annex III, the Commission shall take into account the following criteria:

Amendment 65

Proposal for a regulation

Article 7 – paragraph 2 – point c

Text proposed by the Commission

Amendment

(c) the extent to which the use of an AI system has already caused harm to the health and safety or adverse impact on the fundamental rights or has given rise to significant concerns in relation to the materialisation of such harm or adverse impact, as demonstrated by reports or documented allegations submitted to national competent authorities;

(c) the extent to which the use of an AI system has already caused harm to the health and safety or adverse impact on the climate, the environment and fundamental rights or has given rise to significant concerns in relation to the materialisation of such harm or adverse impact, as demonstrated by reports or documented allegations submitted to national competent authorities;

Amendment 66

Proposal for a regulation

Article 7 – paragraph 2 – point d

Text proposed by the Commission

Amendment

(d) the potential extent of such harm or such adverse impact, in particular in terms of its intensity and its ability to affect a plurality of persons;

(d) the potential extent of such harm or such adverse impact, in particular in terms of its intensity and its ability to affect a plurality of persons, the environment and biodiversity;

Amendment 67

Proposal for a regulation

Article 7 – paragraph 2 – point e

Text proposed by the Commission

Amendment

(e) the extent to which potentially harmed or adversely impacted persons are dependent on the outcome produced with an AI system, in particular because for practical or legal reasons it is not reasonably possible to opt-out from that outcome;

(e) the extent to which potentially harmed or adversely impacted persons, including end recipients, are dependent on the outcome produced with an AI system, in particular because for practical or legal reasons it is not reasonably possible to opt-out from that outcome;

Amendment 68

Proposal for a regulation

Article 7 – paragraph 2 – point f

Text proposed by the Commission

Amendment

(f) the extent to which potentially harmed or adversely impacted persons are in a vulnerable position in relation to the user of an AI system, in particular due to an imbalance of power, knowledge, economic or social circumstances, or age;

Amendment 69

Proposal for a regulation

Article 7 – paragraph 2 – point g

Text proposed by the Commission

Amendment

(g) the extent to which the outcome produced with an AI system is easily reversible, whereby outcomes having an impact on the health or safety of persons shall not be considered as easily reversible;

(g) the extent to which the outcome produced with an AI system is easily reversible, whereby outcomes having an adverse impact on the climate, the environment, on biodiversity or negatively affecting the ability to achieve greenhouse gas reduction targets or the health or safety of persons shall not be considered as easily reversible;

Amendment 70

Proposal for a regulation

Article 9 – paragraph 4 – subparagraph 1

Text proposed by the Commission

Amendment

The risk management measures referred to in paragraph 2, point (d) shall be such that any residual risk associated with each hazard as well as the overall residual risk of the high-risk AI systems is judged acceptable, provided that the high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse. Those residual risks shall be communicated to the user.

Amendment 71

Proposal for a regulation

Article 10 – paragraph 3

Text proposed by the Commission

Amendment

3. Training, validation and testing data sets shall be relevant, representative, free of errors and complete. They shall have the appropriate statistical properties, including, where applicable, as regards the persons or groups of persons on which the high-risk AI system is intended to be used. These characteristics of the data sets may be met at the level of individual data sets or a combination thereof.

3. Training, validation and testing data sets shall be relevant, representative, free of errors and complete. They shall have the appropriate statistical properties, including, where applicable, as regards the persons or groups of persons on which the high-risk AI system is intended to be used, including end recipients. These characteristics of the data sets may be met at the level of individual data sets or a combination thereof.

Amendment 72

Proposal for a regulation

Article 10 – paragraph 4

Text proposed by the Commission

Amendment

4. Training, validation and testing data sets shall take into account, to the extent required by the intended purpose, the characteristics or elements that are particular to the specific geographical, behavioural or functional setting within which the high-risk AI system is intended to be used.

4. Training, validation and testing data sets shall take into account, to the extent required by the intended purpose, the characteristics or elements that are particular to the specific geographical, environmental, behavioural or functional setting within which the high-risk AI system is intended to be used.

Amendment 73

Proposal for a regulation

Article 10 – paragraph 5

Text proposed by the Commission

Amendment

5. To the extent that it is strictly necessary for the purposes of ensuring bias monitoring, detection and correction in relation to the high-risk AI systems, the providers of such systems may process special categories of personal data referred to in Article 9(1) of Regulation (EU) 2016/679, Article 10 of Directive (EU) 2016/680 and Article 10(1) of Regulation (EU) 2018/1725, subject to appropriate safeguards for the fundamental rights and freedoms of natural persons, including technical limitations on the re-use and use of state-of-the-art security and privacy-preserving measures, such as pseudonymisation, or encryption where anonymisation may significantly affect the purpose pursued.

5. To the extent that it is strictly necessary for the purposes of ensuring bias monitoring, detection and correction in relation to the high-risk AI systems and to ensure algorithmic non-discrimination, the providers of such systems may process special categories of personal data referred to in Article 9(1) of Regulation (EU) 2016/679, Article 10 of Directive (EU) 2016/680 and Article 10(1) of Regulation (EU) 2018/1725, subject to appropriate safeguards for the fundamental rights and freedoms of natural persons, including technical limitations on the re-use and use of state-of-the-art security and privacy-preserving measures, such as pseudonymisation, or encryption where anonymisation may significantly affect the purpose pursued.

Amendment 74

Proposal for a regulation

Article 13 – title

Text proposed by the Commission

Amendment

Transparency and provision of information to users

Transparency and provision of information to users and end recipients

Amendment 75

Proposal for a regulation

Article 13 – paragraph 1

Text proposed by the Commission

Amendment

1. High-risk AI systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system’s output and use it appropriately. An appropriate type and degree of transparency shall be ensured, with a view to achieving compliance with the relevant obligations of the user and of the provider set out in Chapter 3 of this Title.

1. High-risk AI systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users and end recipients to interpret the system’s output and use it appropriately. An appropriate type and degree of transparency shall be ensured, with a view to achieving compliance with the relevant obligations of the user, end recipient and of the provider set out in Chapter 3 of this Title.

Amendment 76

Proposal for a regulation

Article 13 – paragraph 2

Text proposed by the Commission

Amendment

2. High-risk AI systems shall be accompanied by instructions for use in an appropriate digital format or otherwise that include concise, complete, correct and clear information that is relevant, accessible and comprehensible to users.

Amendment 77

Proposal for a regulation

Article 13 – paragraph 3 – point b – point iii

Text proposed by the Commission

Amendment

(iii) any known or foreseeable circumstance, related to the use of the high-risk AI system in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, which may lead to risks to the health and safety or fundamental rights;

Amendment 78

Proposal for a regulation

Article 13 – paragraph 3 a (new)

Text proposed by the Commission

Amendment

3a. High-risk AI systems shall be designed, developed and used in such a way to ensure that the outputs are sufficiently transparent, relevant, accessible and comprehensible to the end recipients, in accordance with the intended purpose.

Amendment 79

Proposal for a regulation

Article 14 – paragraph 2

Text proposed by the Commission

Amendment

2. Human oversight shall aim at preventing or minimising the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular when such risks persist notwithstanding the application of other requirements set out in this Chapter.

2. Human oversight shall aim at preventing or minimising disinformation as well as the risks to health, safety, the climate and environment or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular when such risks persist notwithstanding the application of other requirements set out in this Chapter.

Amendment 80

Proposal for a regulation

Article 14 – paragraph 4 – point e

Text proposed by the Commission

Amendment

(e) be able to intervene on the operation of the high-risk AI system or interrupt the system through a “stop” button or a similar procedure.

(e) be able to intervene on the operation of the high-risk AI system or interrupt the system through a “stop” button or a similar procedure provided that the intended function of the AI system is not affected in a way that risks health, safety or fundamental rights.

Amendment 81

Proposal for a regulation

Article 26 – paragraph 1 – point c

Text proposed by the Commission

Amendment

(c) the system bears the required conformity marking and is accompanied by the required documentation and instructions of use.

(c) the system bears the required conformity marking and is accompanied by the required concise and clear documentation and instructions of use, including in relation to possible risks to fundamental rights and discrimination.

Amendment 82

Proposal for a regulation

Article 43 – paragraph 2

Text proposed by the Commission

Amendment

2. For high-risk AI systems referred to in points 2 to 8 of Annex III, providers shall follow the conformity assessment procedure based on internal control as referred to in Annex VI, which does not provide for the involvement of a notified body. For high-risk AI systems referred to in point 5(b) of Annex III, placed on the market or put into service by credit institutions regulated by Directive 2013/36/EU, the conformity assessment shall be carried out as part of the procedure referred to in Articles 97 to101 of that Directive.

2. For high-risk AI systems referred to in points 2 to 9 of Annex III, providers shall follow the conformity assessment procedure based on internal control as referred to in Annex VI, which does not provide for the involvement of a notified body. For high-risk AI systems referred to in point 5(b) of Annex III, placed on the market or put into service by credit institutions regulated by Directive 2013/36/EU, the conformity assessment shall be carried out as part of the procedure referred to in Articles 97 to101 of that Directive.

Amendment 83

Proposal for a regulation

Article 43 – paragraph 6

Text proposed by the Commission

Amendment

6. The Commission is empowered to adopt delegated acts to amend paragraphs 1 and 2 in order to subject high-risk AI systems referred to in points 2 to 8 of Annex III to the conformity assessment procedure referred to in Annex VII or parts thereof. The Commission shall adopt such delegated acts taking into account the effectiveness of the conformity assessment procedure based on internal control referred to in Annex VI in preventing or minimizing the risks to health and safety and protection of fundamental rights posed by such systems as well as the availability of adequate capacities and resources among notified bodies.

6. The Commission is empowered to adopt delegated acts to amend paragraphs 1 and 2 in order to subject high-risk AI systems referred to in points 2 to 8 of Annex III to the conformity assessment procedure referred to in Annex VII or parts thereof. The Commission shall adopt such delegated acts taking into account the effectiveness of the conformity assessment procedure based on internal control referred to in Annex VI in preventing or minimizing the risks to health and safety, to the environment and protection of fundamental rights posed by such systems as well as the availability of adequate capacities and resources among notified bodies.

Amendment 84

Proposal for a regulation

Article 52 – paragraph 1

Text proposed by the Commission

Amendment

1. Providers shall ensure that AI systems intended to interact with natural persons are designed and developed in such a way that natural persons are informed that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use. This obligation shall not apply to AI systems authorised by law to detect, prevent, investigate and prosecute criminal offences, unless those systems are available for the public to report a criminal offence.

1. Providers shall ensure that AI systems intended to interact with natural persons are designed and developed in such a way that natural persons are informed that they are interacting with an AI system, especially in the healthcare sector, unless this is obvious from the circumstances and the context of use. This obligation shall not apply to AI systems authorised by law to detect, prevent, investigate and prosecute criminal offences, unless those systems are available for the public to report a criminal offence.

Amendment 85

Proposal for a regulation

Article 52 – paragraph 3 a (new)

Text proposed by the Commission

Amendment

3 a. Recipients of an AI system in the domain of healthcare shall be informed of their interaction with an AI system.

Amendment 86

Proposal for a regulation

Article 52 – paragraph 3 b (new)

Text proposed by the Commission

Amendment

3b. Public and administrative authorities which adopt decisions with the assistance of AI systems shall provide a clear and intelligible explanation. The explanation shall be accessible for persons with disabilities and other vulnerable groups.

Amendment 87

Proposal for a regulation

Article 53 – paragraph 3

Text proposed by the Commission

Amendment

3. The AI regulatory sandboxes shall not affect the supervisory and corrective powers of the competent authorities. Any significant risks to health and safety and fundamental rights identified during the development and testing of such systems shall result in immediate mitigation and, failing that, in the suspension of the development and testing process until such mitigation takes place.

3. The AI regulatory sandboxes shall not affect the supervisory and corrective powers of the competent authorities. Any significant risks to climate mitigation, the environment, health and safety and fundamental rights identified during the development and testing of such systems shall result in immediate mitigation and, failing that, in the suspension of the development and testing process until such mitigation takes place.

Amendment 88

Proposal for a regulation

Article 53 – paragraph 4

Text proposed by the Commission

Amendment

4. Participants in the AI regulatory sandbox shall remain liable under applicable Union and Member States liability legislation for any harm inflicted on third parties as a result from the experimentation taking place in the sandbox.

4. Participants in the AI regulatory sandbox shall remain liable under applicable Union and Member States liability legislation for any harm inflicted on third parties or the environment as a result from the experimentation taking place in the sandbox.

Amendment 89

Proposal for a regulation

Article 54 – paragraph 1 – point a – point ii

Text proposed by the Commission

Amendment

(ii) public safety and public health, including disease prevention, control and treatment;

(ii) public safety and public health, including disease detection, diagnosis, prevention, control and treatment, and the health challenges in relation to the inter-linkage between human and animal health, in particular zoonotic diseases;

Amendment 90

Proposal for a regulation

Article 54 – paragraph 1 – point a – point iii

Text proposed by the Commission

Amendment

(iii) a high level of protection and improvement of the quality of the environment;

(iii) a high level of protection and improvement of the quality of the environment, protection of biodiversity, pollution as well as climate change mitigation and adaptation;

Amendment 91

Proposal for a regulation

Article 54 – paragraph 1 – point a – point iii a (new)

Text proposed by the Commission

Amendment

(iiia) the principle of data minimisation shall be upheld, meaning that the data acquisition and processing shall be kept to what is strictly necessary for the purpose of the artificial intelligence application;

Amendment 92

Proposal for a regulation

Article 56 – paragraph 2 – point b

Text proposed by the Commission

Amendment

(b) coordinate and contribute to guidance and analysis by the Commission and the national supervisory authorities and other competent authorities on emerging issues across the internal market with regard to matters covered by this Regulation;

(b) coordinate and contribute to guidance and analysis by the Commission and the national supervisory authorities as well as advisory and expert groups, including organisations from the civil society such as NGOs, consumer associations, and industry representatives and other competent authorities on emerging issues across the internal market with regard to matters covered by this Regulation;

Amendment 93

Proposal for a regulation

Article 57 – paragraph 1

Text proposed by the Commission

Amendment

1. The Board shall be composed of the national supervisory authorities, who shall be represented by the head or equivalent high-level official of that authority, and the European Data Protection Supervisor. Other national authorities may be invited to the meetings, where the issues discussed are of relevance for them.

1. The Board shall be composed of the national supervisory authorities, who shall be represented by the head or equivalent high-level official of that authority, and the European Data Protection Supervisor. Other national authorities including those which are members of relevant advisory and expert groups at Union level, may be invited to the meetings, where the issues discussed are of relevance for them.

Amendment 94

Proposal for a regulation

Article 57 – paragraph 4

Text proposed by the Commission

Amendment

4. The Board may invite external experts and observers to attend its meetings and may hold exchanges with interested third parties to inform its activities to an appropriate extent. To that end the Commission may facilitate exchanges between the Board and other Union bodies, offices, agencies and advisory groups.

4. The Board may invite external experts, ethicists and observers to attend its meetings and may hold exchanges with interested third parties including organisations from the civil society such as NGOs, consumer associations, human rights groups and intergovernmental organisations and industry representatives to inform its activities to an appropriate extent. To that end the Commission may facilitate exchanges between the Board and other relevant Union bodies, offices, agencies and expert advisory groups.

Amendment 95

Proposal for a regulation

Article 58 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) collect and share expertise and best practices among Member States;

(a) collect and share technical and regulatory expertise and best practices among Member States;

Justification

It should be specified that the technical as well as from the regulatory views are considered.

Amendment 96

Proposal for a regulation

Article 58 – paragraph 1 – point c a (new)

Text proposed by the Commission

Amendment

(ca) ensure that there is a common and consistent approach among the different advisory and expert groups established at Union level on matters covered by this Regulation or related to AI systems.

Amendment 97

Proposal for a regulation

Article 59 – paragraph 1

Text proposed by the Commission

Amendment

1. National competent authorities shall be established or designated by each Member State for the purpose of ensuring the application and implementation of this Regulation. National competent authorities shall be organised so as to safeguard the objectivity and impartiality of their activities and tasks.

1. National competent authorities shall be established or designated by each Member State for the purpose of ensuring the application and implementation of this Regulation and horizontal Union legislation. National competent authorities shall be organised so as to safeguard the objectivity, consistency and impartiality of their activities and tasks to avoid any conflicts of interest.

Amendment 98

Proposal for a regulation

Article 59 – paragraph 4

Text proposed by the Commission

Amendment

4. Member States shall ensure that national competent authorities are provided with adequate financial and human resources to fulfil their tasks under this Regulation. In particular, national competent authorities shall have a sufficient number of personnel permanently available whose competences and expertise shall include an in-depth understanding of artificial intelligence technologies, data and data computing, fundamental rights, health and safety risks and knowledge of existing standards and legal requirements.

4. Member States shall ensure that national competent authorities are provided with adequate financial and human resources to fulfil their tasks under this Regulation. In particular, national competent authorities shall have a sufficient number of personnel permanently available whose competences and expertise shall include an in-depth understanding of artificial intelligence technologies, data protection and data computing, fundamental rights, health and safety risks, environmental risks and knowledge of existing standards and legal requirements.

Amendment 99

Proposal for a regulation

Article 62 – paragraph 1 – subparagraph 1

Text proposed by the Commission

Amendment

Providers of high-risk AI systems placed on the Union market shall report any serious incident or any malfunctioning of those systems which constitutes a breach of obligations under Union law intended to protect fundamental rights to the market surveillance authorities of the Member States where that incident or breach occurred.

Providers of high-risk AI systems placed on the Union market shall report any serious incident or any malfunctioning of those systems which constitutes a breach of obligations under Union law intended to protect health, safety, fundamental rights and the environment to the market surveillance authorities of the Member States where that incident or breach occurred.

Amendment 100

Proposal for a regulation

Article 65 – paragraph 1

Text proposed by the Commission

Amendment

1. AI systems presenting a risk shall be understood as a product presenting a risk defined in Article 3, point 19 of Regulation (EU) 2019/1020 insofar as risks to the health or safety or to the protection of fundamental rights of persons are concerned.

1. AI systems presenting a risk shall be understood as a product presenting a risk defined in Article 3, point 19 of Regulation (EU) 2019/1020 insofar as risks to the health or safety, the protection of consumers and the environment or where the protection of fundamental rights of persons are concerned.

Amendment 101

Proposal for a regulation

Article 67 – paragraph 1

Text proposed by the Commission

Amendment

1. Where, having performed an evaluation under Article 65, the market surveillance authority of a Member State finds that although an AI system is in compliance with this Regulation, it presents a risk to the health or safety of persons, to the compliance with obligations under Union or national law intended to protect fundamental rights or to other aspects of public interest protection, it shall require the relevant operator to take all appropriate measures to ensure that the AI system concerned, when placed on the market or put into service, no longer presents that risk, to withdraw the AI system from the market or to recall it within a reasonable period, commensurate with the nature of the risk, as it may prescribe.

1. Where, having performed an evaluation under Article 65, the market surveillance authority of a Member State finds that although an AI system is in compliance with this Regulation, it presents a risk to the health or safety of persons, to the environment, to the compliance with obligations under Union or national law intended to protect fundamental rights or to other aspects of public interest protection, it shall require the relevant operator to take all appropriate measures to ensure that the AI system concerned, when placed on the market or put into service, no longer presents that risk, to withdraw the AI system from the market or to recall it within a reasonable period, commensurate with the nature of the risk, as it may prescribe.

Amendment 102

Proposal for a regulation

Article 69 – paragraph 2

Text proposed by the Commission

Amendment

2. The Commission and the Board shall encourage and facilitate the drawing up of codes of conduct intended to foster the voluntary application to AI systems of requirements related for example to environmental sustainability, accessibility for persons with a disability, stakeholders participation in the design and development of the AI systems and diversity of development teams on the basis of clear objectives and key performance indicators to measure the achievement of those objectives.

2. The Commission and the Board shall encourage and facilitate the drawing up of codes of conduct intended to foster the voluntary application to AI systems of requirements related to a European code for data centre energy efficiency that shall contain key indicators related to environmental sustainability, resource usage, energy efficiency and carbon intensity, the proportion of energy generated from renewable energy sources and reuse of any heat or waste. This could be extended to encourage the accessibility for persons with a disability, stakeholders’ participation in the design and development of the AI systems as well as the diversity of development teams on the basis of clear objectives and key performance indicators to measure the achievement of those objectives. In order to facilitate the voluntary application of environmental assessments, the Commission shall develop, by means of an implementing act, a procedure, methodology, minimum standards and scale to facilitate the disclosure of information on the energy used in the training and execution of AI systems and their carbon intensity to promote the development of energy efficient and low carbon AI systems, which shall be applicable to all AI systems on a voluntary basis. Those AI systems voluntarily participating shall include this information in the technical documentation referred to in article 11.

Amendment 103

Proposal for a regulation

Article 84 – paragraph 3 a (new)

Text proposed by the Commission

Amendment

3a. Within ...[two years after the date of application of this Regulation referred to in Article 85(2)] and every two years thereafter, the Commission shall evaluate the environmental impact and effectiveness of this Regulation with regards to energy use or other environmental impact of AI systems. By January 2050, the Commission shall present a proposal to regulate the energy efficiency to ensure the full decarbonisation of AI technologies.

Amendment 104

Proposal for a regulation

Annex I – point a

Text proposed by the Commission

Amendment

(a) Machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning;

(a) Machine learning approaches, including supervised, unsupervised, reinforcement learning and computational scientific discovery, using a wide variety of methods including deep learning;

Amendment 105

Proposal for a regulation

Annex III – paragraph 1 – point 5 – point a

Text proposed by the Commission

Amendment

(a) AI systems intended to be used by public authorities or on behalf of public authorities to evaluate the eligibility of natural persons for public assistance benefits and services, as well as to grant, reduce, revoke, or reclaim such benefits and services;

(a) AI systems intended to be used by public authorities or on behalf of public authorities to evaluate the eligibility of natural persons for public assistance benefits and services, including healthcare service and health literacy, as well as to grant, reduce, revoke, or reclaim such benefits and services;

Amendment 106

Proposal for a regulation

Annex III – paragraph 1 – point 8 a (new)

Text proposed by the Commission

Amendment

8a. Health, health care, long-term care and health insurance:

(a) AI systems not covered by Regulation (EU) 2017/745 intended to be used in the health, health care and long-term care sectors that have indirect and direct effects on health or that use sensitive health data.

(b) Artificial intelligence administrative and management systems used by healthcare professionals in hospitals and other healthcare settings and by health insurance companies that process sensitive data of people’s health.

Justification

The proposal assumes that all AI applications used in the context of health are covered by Regulation (EU) 2017/745. However, this Regulation only covers medical devices and software with an intended medical purpose, such as treatment of patients. This excludes health related AI applications (for example, apps to track medication) and administrative AI systems used by doctors in a hospital or other healthcare setting that still present new challenges and possible risks to people, because of their effects on health or the use of sensitive health data and life choices.

Amendment 107

Proposal for a regulation

Annex IV – paragraph 1 – point 1 – point g

Text proposed by the Commission

Amendment

(g) instructions of use for the user and, where applicable installation instructions;

(g) clear and concise instructions of use for the user and the end recipient, including in relation to possible risks to fundamental rights and discrimination and, where applicable installation instructions;

Amendment 108

Proposal for a regulation

Annex IV – paragraph 1 – point 2 – point g a (new)

Text proposed by the Commission

Amendment

(ga) the computational complexity of the system and its software components, its data use, including the validation and testing of systems.

Amendment 109

Proposal for a regulation

Annex IV – paragraph 1 – point 3

Text proposed by the Commission

Amendment

3. Detailed information about the monitoring, functioning and control of the AI system, in particular with regard to: its capabilities and limitations in performance, including the degrees of accuracy for specific persons or groups of persons on which the system is intended to be used and the overall expected level of accuracy in relation to its intended purpose; the foreseeable unintended outcomes and sources of risks to health and safety, fundamental rights and discrimination in view of the intended purpose of the AI system; the human oversight measures needed in accordance with Article 14, including the technical measures put in place to facilitate the interpretation of the outputs of AI systems by the users; specifications on input data, as appropriate;

3. Detailed information and fully accessible about the monitoring, functioning and control of the AI system, in particular with regard to: its capabilities and limitations in performance, including the degrees of accuracy for specific persons or groups of persons on which the system is intended to be used and the overall expected level of accuracy in relation to its intended purpose; the foreseeable unintended outcomes and sources of risks to the environment, health and safety, fundamental rights and discrimination in view of the intended purpose of the AI system; the human oversight measures needed in accordance with Article 14, including the technical measures put in place to facilitate the interpretation of the outputs of AI systems by the users; specifications on input data, as appropriate;

PROCEDURE – COMMITTEE ASKED FOR OPINION

Title

Harmonised rules on Artificial Intelligence (Artificial Intelligence Act) and amending certain Union Legislative Acts

References

COM(2021)0206 – C9-0146/2021 – 2021/0106(COD)

Committees responsible

Date announced in plenary

IMCO

7.6.2021

LIBE

7.6.2021

Opinion by

Date announced in plenary

ENVI

7.6.2021

Rapporteur for the opinion

Date appointed

Susana Solís Pérez

15.9.2021

Rule 58 – Joint committee procedure

Date announced in plenary

16.12.2021

Discussed in committee

13.1.2022

Date adopted

15.3.2022

Result of final vote

–:

Members present for the final vote

Mathilde Androuët, Nikos Androulakis, Bartosz Arłukowicz, Margrete Auken, Simona Baldassarre, Marek Paweł Balt, Traian Băsescu, Aurélia Beigneux, Monika Beňová, Hildegard Bentele, Sergio Berlato, Alexander Bernhuber, Malin Björk, Simona Bonafè, Delara Burkhardt, Pascal Canfin, Sara Cerdas, Mohammed Chahim, Tudor Ciuhodaru, Nathalie Colin-Oesterlé, Esther de Lange, Christian Doleschal, Marco Dreosto, Bas Eickhout, Cyrus Engerer, Cornelia Ernst, Eleonora Evi, Agnès Evren, Pietro Fiocchi, Raffaele Fitto, Malte Gallée, Andreas Glück, Catherine Griset, Jytte Guteland, Teuvo Hakkarainen, Anja Hazekamp, Martin Hojsík, Pär Holmgren, Jan Huitema, Yannick Jadot, Adam Jarubas, Petros Kokkalis, Athanasios Konstantinou, Ewa Kopacz, Joanna Kopcińska, Peter Liese, Sylvia Limmer, Javi López, César Luena, Fulvio Martusciello, Liudas Mažylis, Joëlle Mélin, Tilly Metz, Silvia Modig, Dolors Montserrat, Alessandra Moretti, Dan-Ştefan Motreanu, Ville Niinistö, Ljudmila Novak, Grace O’Sullivan, Jutta Paulus, Stanislav Polčák, Jessica Polfjärd, Nicola Procaccini, Luisa Regimenti, Frédérique Ries, María Soraya Rodríguez Ramos, Sándor Rónai, Rob Rooken, Silvia Sardone, Christine Schneider, Günther Sidl, Ivan Vilibor Sinčić, Linea Søgaard-Lidell, Maria Spyraki, Nils Torvalds, Edina Tóth, Véronique Trillet-Lenoir, Petar Vitanov, Alexandr Vondra, Mick Wallace, Emma Wiesner, Michal Wiezik, Tiemo Wölken, Anna Zalewska

Substitutes present for the final vote

Maria Arena, Marlene Mortler, Susana Solís Pérez

FINAL VOTE BY ROLL CALL IN COMMITTEE ASKED FOR OPINION

Key to symbols: