Sittings · Compare

What changed

From · opinion parliamentary committee · 2022-06-16 CULT-AD-719637 on the proposal for a regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union Legislative Acts

→

To · opinion parliamentary committee · 2022-06-14 ITRE-AD-719801 on the proposal for a regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts

+274 added · −204 removed · 51 modified paragraphs hide unchanged text

SHORT JUSTIFICATION

On 24 April 2021, the European Commission published its legislative proposal laying down harmonised rules on artificial intelligence (AI Act), which introduces a regulatory framework with the objective of ensuring that AI systems placed on the European Union market are safe to use and respect fundamental rights and European Union values.

Introduction

Furthermore, the proposal facilitates development of a single market for lawful, safe and trustworthy AI applications, enhances governance and effective enforcement of existing law on fundamental rights and safety requirements applicable to AI systems and ensures legal certainty to facilitate investment and innovation in AI.

The Rapporteur welcomes the Commission’s proposal on an Artificial Intelligence Act and especially the horizontal risk-based approach that it puts forward. This approach will allow for the development of AI systems in line with European values and for the fostering of social trust in these new technologies, so that the EU can fulfil the full economic and social benefits of AI.

Overall, the Rapporteur welcomes the European Commission’s proposal; however, would like to suggest a few amendments mainly to extend the list of high-risk AI applications in areas of education, media and culture under Annex III and to modify certain provisions related to banned practices under Article 5.

The Rapporteur is of the opinion that through the AI Act, we need to create an environment with the right balance between freedom and supervision. The Rapporteur proposes that further provisions are made in order for companies, especially start-ups and SMEs, to remain competitive and creative in the face of new obligations required of them. The Rapporteur believes this will increase both the legitimacy and relevance of the AI Act. We need to provide companies with clearer guidelines, simpler tools and more efficient resources to cope with regulation. This would allow us to support AI innovation, development and market uptake.

More specifically, the Rapporteur reflects on the increased deployment of AI technologies in education and training facilities. Therefore, he proposes listing, among high-risk technologies, also those AI technologies used for monitoring of students during tests and technologies used to determine an area or a programme a student should study. Regarding media and culture, the Rapporteur suggests listing high-risk AI technologies used to create or disseminate machine-generated news articles used by news media outlets and AI technologies used to recommend or rank audiovisual content.

Therefore, the Rapporteur’s draft pursues four main objectives in this direction:

In addition, the Rapporteur proposes to extend the ban on deployment of social scoring systems to usage by public and private entities given the inherent threat of discrimination and exclusion of certain groups or individuals.

1. Enhancing measures to support innovation, such as the ones foreseen for regulatory sandboxes, with a particular focus on start-ups and SMEs

Finally, in light of the danger that deployment of remote biometric identification systems in publicly accessible places poses to citizens’ fundamental rights, freedom of assembly, work of investigative journalists, activists and political representatives, the Rapporteur proposes to ban deployment of such technologies in publicly accessible places following up on the Parliament resolution of 6 October 2021 on “Artificial intelligence in criminal law and its use by the police and judicial authorities in criminal matters”.

2. Providing a concise and internationally recognised definition of Artificial Intelligence System and setting high but realistic standards for accuracy, robustness, cybersecurity and data

3. Encouraging the uptake of AI systems by industry by placing an emphasis on social trust and value chain responsibility

4. Future-proofing the Act through better linkages to the green transition and possible changes in the industry, technology and power of AI

This draft opinion focuses mainly on issues related to ITRE’s competences but also broader issues related to innovation, competitiveness, research, sustainability and future changes in industry.

Supporting innovation, focus on start-ups and SMEs, enhancing regulatory sandboxes

The Rapporteur welcomes the introduction of Article 55 on measures for small-scale providers, but believes SMEs and start-ups should be more involved throughout the AI Act in a holistic approach. More specifically, in the development of Codes of Conduct, standardisation, and representation in the European Artificial Intelligence Board. By far, one of the biggest focuses for the Rapporteur is the provision of opportunities to SMEs and start-ups to participate in the AI regulatory sandboxes. This is why the Rapporteur proposes to strengthen the existing provisions by giving the regulatory sandboxes a more European dimension, preserving the unity of the Single Market and calling for the development of an EU AI Regulatory Sandboxing Programme whose modalities are set out in a new Annex.

Clear definition and realistic standards

The Rapporteur calls for the use of an internationally recognised definition of Artificial Intelligence System, which would be in line with the EU’s broader goals of setting global standards, working closely with transatlantic partners and likeminded allies and providing legal certainty for businesses, citizens and civil society. The Rapporteur believes that high standards for accuracy, robustness and cybersecurity as well as data and data governance are key to developing safe AI systems that protect fundamental rights. The key here is to balance this aim with the practical and pragmatic approach needed for achieving it. The Rapporteur calls for a common European authority on benchmarking that brings together national metrology and benchmarking authorities to set a unified approach to measurement of accuracy, robustness, and other relevant criteria.

Encouraging uptake of AI systems, fostering social trust, value chain responsibility

To encourage uptake and deployment of AI systems, the Rapporteur believes we need to foster social trust of both businesses and citizens. The Rapporteur seeks to address the challenge of social trust by encouraging a collaborative relationship between developers and users of AI that is better aligned to their responsibilities along the value chain, strengthening the Codes of Conduct and enhancing the measures on regulatory sandboxes to enable compliance-by-design. This in turn creates a healthy and integrated ecosystem, which will help reduce legal uncertainty and implementation gaps, all of which in turn will increase social trust.

Future-proofing, sustainability and changes in the industry and power of AI

AI is a mature and ready-to-use technology that can be used to process the ever growing amount of data created along industrial processes. To facilitate investments to AI-based analysis and optimisation solutions, this regulation should provide a predictable environment for low-risk industrial solutions. Furthermore, this Regulation should take into account future changes in the industry and power of AI. This is why the Rapporteur proposes great involvement of the High Level Expert Group on AI with both the Commission and the European Artificial Intelligence Board as well as the monitoring of market trends and foresight by the European AI Board.

AMENDMENTS

The Committee on ~~Culture~~Industry, Research and ~~Education~~Energy calls on the Committee on the Internal Market and Consumer Protection and the Committee on Civil Liberties, Justice and Home Affairs, as the committees responsible, to take into account the following amendments:

Amendment 1

Proposal for a regulation

~~Recital 1~~

Recital 3 a (new)

Text proposed by the Commission

Amendment

(1) The purpose of this Regulation is to improve the functioning of the internal market by laying down a uniform legal framework in particular for the development, marketing and use of artificial intelligence in conformity with Union values. This Regulation pursues a number of overriding reasons of public interest, such as a high level of protection of health, safety and fundamental rights, and it ensures the free movement of AI-based goods and services cross-border, thus preventing Member States from imposing restrictions on the development, marketing and use of AI systems, unless explicitly authorised by this Regulation.

(3a) Furthermore, in order for the Member States to reach their climate targets and to meet the United Nation’s Sustainable Development Goals (SDGs), Union companies should be encouraged to utilise available technological advancements in realising this goal. AI is a well-developed and ready-to-use technology that can be used to process ever-growing amount of data created along industrial processes. To facilitate investments in AI-based analysis and optimisation solutions that can help to achieve the climate goals, this Regulation should provide a predictable and proportionate environment for low- risk industrial solutions. To ensure coherence, this requires that AI systems themselves need to be designed sustainably to reduce resource usage and energy consumption, thereby limiting the damage to the environment.

(1) The purpose of this Regulation is to improve the functioning of the internal market by laying down a uniform legal framework, based on ethical principles in particular for the development, marketing and use of artificial intelligence in conformity with Union values, minimising any risk of adverse and discriminatory impact on people and without hindering innovation. This Regulation pursues a number of overriding reasons of public interest, such as a high level of protection of health, safety and fundamental rights, democracy, the rule of law and the environment, and it ensures the free movement of AI-based goods and services cross-border, thus preventing Member States from imposing restrictions on the development, marketing and use of AI systems, unless explicitly authorised by this Regulation.

Amendment 2

Proposal for a regulation

~~Recital 2~~

Recital 3 b (new)

Text proposed by the Commission

Amendment

(2) Artificial intelligence systems (AI systems) can be easily deployed in multiple sectors of the economy and society, including cross border, and circulate throughout the Union. Certain Member States have already explored the adoption of national rules to ensure that artificial intelligence is safe and is developed and used in compliance with fundamental rights obligations. Differing national rules may lead to fragmentation of the internal market and decrease legal certainty for operators that develop or use AI systems. A consistent and high level of protection throughout the Union should therefore be ensured, while divergences hampering the free circulation of AI systems and related products and services within the internal market should be prevented, by laying down uniform obligations for operators and guaranteeing the uniform protection of overriding reasons of public interest and of rights of persons throughout the internal market based on Article 114 of the Treaty on the Functioning of the European Union (TFEU). To the extent that this Regulation contains specific rules on the protection of individuals with regard to the processing of personal data concerning restrictions of the use of AI systems for ‘real-time’ remote biometric identification in publicly accessible spaces for the purpose of law enforcement, it is appropriate to base this Regulation, in as far as those specific rules are concerned, on Article 16 of the TFEU. In light of those specific rules and the recourse to Article 16 TFEU, it is appropriate to consult the European Data Protection Board.

(3b) Furthermore, in order to foster the development of artificial intelligence in line with Union values, the Union needs to address the main gaps and barriers blocking the potential of the digital transformation including the shortage of digitally skilled workers, cybersecurity concerns, lack of investment and access to investment, and existing and potential gaps between large companies and SMEs. Special attention should be paid to ensuring that the benefits of artificial intelligence and innovation in new technologies are felt across all regions of the Union and that sufficient investment and resources are provided especially to those regions that may be lagging behind in some digital indicators.

(2) Artificial intelligence systems (AI systems) can be easily deployed in multiple sectors of the economy and society, including cross border, and circulate throughout the Union. Certain Member States have already explored the adoption of national rules to ensure that artificial intelligence is trustworthy and safe and is developed and used in compliance with fundamental rights obligations. Differing national rules may lead to fragmentation of the internal market and decrease legal certainty for operators that develop or use AI systems. A consistent and high level of protection throughout the Union should therefore be ensured in order to achieve trustworthy AI, while divergences hampering the free circulation, innovation, deployment and uptake of AI systems and related products and services within the internal market should be prevented, by laying down uniform obligations for operators and guaranteeing the uniform protection of overriding reasons of public interest and of rights of persons throughout the internal market based on Article 114 of the Treaty on the Functioning of the European Union (TFEU). To the extent that this Regulation contains specific rules on the protection of individuals with regard to the processing of personal data concerning restrictions of the use of AI systems for ‘real-time’ remote biometric identification in publicly accessible spaces for the purpose of law enforcement, it is appropriate to base this Regulation, in as far as those specific rules are concerned, on Article 16 of the TFEU. In light of those specific rules and the recourse to Article 16 TFEU, it is appropriate to consult the European Data Protection Board.

Amendment 3

Proposal for a regulation

Recital 36

Text proposed by the Commission

Amendment

(3) Artificial intelligence is a fast evolving family of technologies that can contribute to a wide array of economic and societal benefits across the entire spectrum of industries and social activities. By improving prediction, optimising operations and resource allocation, and personalising digital solutions available for individuals and organisations, the use of artificial intelligence can provide key competitive advantages to companies and support socially and environmentally beneficial outcomes, for example in healthcare, farming, education and training, infrastructure management, energy, transport and logistics, public services, security, justice, resource and energy efficiency, and climate change mitigation and adaptation.

(6) The notion of AI system should be clearly defined to ensure legal certainty, while providing the flexibility to accommodate future technological developments. The definition should be based on the key functional characteristics of the software, in particular the ability, for a given set of human-defined objectives, to generate outputs such as content, predictions, recommendations, or decisions which influence the environment with which the system interacts, be it in a physical or digital dimension. AI systems can be designed to operate with varying levels of autonomy and be used on a stand-alone basis or as a component of a product, irrespective of whether the system is physically integrated into the product (embedded) or serve the functionality of the product without being integrated therein (non-embedded). The definition of AI system should be complemented by a list of specific techniques and approaches used for its development, which should be kept up-to–date in the light of market and technological developments through the adoption of delegated acts by the Commission to amend that list.

(3) Artificial intelligence is a fast evolving family of technologies that can contribute and is already contributing to a wide array of economic and societal benefits across the entire spectrum of industries and social activities, if developed in accordance with ethical principles. By improving prediction, optimising operations and resource allocation, and personalising digital solutions available for individuals and organisations, the use of artificial intelligence can provide key competitive advantages to companies and support socially and environmentally beneficial outcomes, for example in healthcare, farming, education and training, media, sports, culture infrastructure management, energy, transport and logistics, public services, security, justice, resource and energy efficiency, and climate change mitigation and adaptation.

(6) The notion of AI system should be clearly defined to ensure legal certainty, while providing the flexibility to accommodate future technological developments. This definition should be in line with definitions that have been accepted internationally. The definition should be based on the key functional characteristics of the AI system, in particular the ability, for a given set of human-defined objectives, to make predictions, recommendations, or decisions influencing real or virtual environments. More specifically, the definition of AI system should take into account key features such as the ability to perceive real and/or virtual environments, to abstract such perceptions into models through analysis in an automated manner and to use model inference to formulate options for information or action. AI systems are designed to operate with varying levels of autonomy and be used on a stand-alone basis or as a component of a product, irrespective of whether the system is physically integrated into the product (embedded) or serve the functionality of the product without being integrated therein (non-embedded). The definition of AI system should be complemented by a list of specific techniques and approaches used for its development, which should be kept up-to–date in the light of market and technological developments through the adoption of delegated acts by the Commission to amend that list. While drafting these delegated acts, the Commission should insure the input of all relevant stakeholders including the technical experts and developers of AI systems. This consultation could take place through existing bodies such as the High Level Expert Group on AI or a newly established similar advisory body that is closely included in the work of the European Artificial Intelligence Board. Furthermore, the Commission should engage in dialogue with key international organisations such as the Organisation for Economic Cooperation and Development and other key organisations working on the definition of AI systems to ensure alignment between definitions of AI, while keeping the prerogative of the Union to set its own definition and standards through enacting legislation.

Amendment 4

Proposal for a regulation

~~Recital 4~~

Recital 12 a (new)

Text proposed by the Commission

Amendment

(12a) This Regulation should not undermine research and development activity and should respect freedom of science. It is therefore necessary to ensure that this Regulation does not otherwise affect scientific research and development activity on AI systems. As regards product oriented research activity by providers, this Regulation should apply insofar as such research leads to or entails placing an AI system on the market or putting it into service. Under all circumstances, any research and development activity should be carried out in accordance with recognised ethical standards for scientific research.

(4) At the same time, depending on the circumstances regarding its specific application and use, artificial intelligence may generate risks and cause harm to public interests and rights, that are protected by Union law including fundamental rights of workers, people in learning processes and socially engaged people, privacy, data protection and informational self-determination, societal or environmental rights. Such harm might be material or immaterial.

Amendment 5

Proposal for a regulation

Recital 529

Text proposed by the Commission

Amendment

(5) A Union legal framework laying down harmonised rules on artificial intelligence is therefore needed to foster the development, use and uptake of artificial intelligence in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, as recognised and protected by Union law. To achieve that objective, rules regulating the placing on the market and putting into service of certain AI systems should be laid down, thus ensuring the smooth functioning of the internal market and allowing those systems to benefit from the principle of free movement of goods and services. By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development of secure, trustworthy and ethical artificial intelligence, as stated by the European Council33 , and it ensures the protection of ethical principles, as specifically requested by the European Parliament34 .

(29) As regards high-risk AI systems that are safety components of products or systems, or which are themselves products or systems falling within the scope of Regulation (EC) No 300/2008 of the European Parliament and of the Council39 , Regulation (EU) No 167/2013 of the European Parliament and of the Council40 , Regulation (EU) No 168/2013 of the European Parliament and of the Council41 , Directive 2014/90/EU of the European Parliament and of the Council42 , Directive (EU) 2016/797 of the European Parliament and of the Council43 , Regulation (EU) 2018/858 of the European Parliament and of the Council44 , Regulation (EU) 2018/1139 of the European Parliament and of the Council45 , and Regulation (EU) 2019/2144 of the European Parliament and of the Council46 , it is appropriate to amend those acts to ensure that the Commission takes into account, on the basis of the technical and regulatory specificities of each sector, and without interfering with existing governance, conformity assessment and enforcement mechanisms and authorities established therein, the mandatory requirements for high-risk AI systems laid down in this Regulation when adopting any relevant future delegated or implementing acts on the basis of those acts.

(5) A Union legal framework laying down harmonised rules on artificial intelligence is therefore needed to foster the development, use and uptake of artificial intelligence in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, as recognised and protected by Union law, democracy, the rule of law and of the environment. To achieve that objective, rules regulating the placing on the market and putting into service of certain AI systems should be laid down, thus ensuring the smooth functioning of the internal market and allowing those systems to benefit from the principle of free movement of goods and services. By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development of secure, trustworthy and ethical artificial intelligence based on fundamental rights, as stated by the European Council33, and it ensures the protection of ethical principles, as specifically requested by the European Parliament34, with a human-centric approach and in compliance with freedom of expression, freedom of speech, media freedom, pluralism and diversity.

__________________

~~33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6.~~

39 Regulation (EC) No 300/2008 of the European Parliament and of the Council of 11 March 2008 on common rules in the field of civil aviation security and repealing Regulation (EC) No 2320/2002 (OJ L 97, 9.4.2008, p. 72).

~~33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6.~~

34 European Parliament resolution of 20 October 2020 with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies, 2020/2012(INL).

40 Regulation (EU) No 167/2013 of the European Parliament and of the Council of 5 February 2013 on the approval and market surveillance of agricultural and forestry vehicles (OJ L 60, 2.3.2013, p. 1).

41 Regulation (EU) No 168/2013 of the European Parliament and of the Council of 15 January 2013 on the approval and market surveillance of two- or three-wheel vehicles and quadricycles (OJ L 60, 2.3.2013, p. 52).

42 Directive 2014/90/EU of the European Parliament and of the Council of 23 July 2014 on marine equipment and repealing Council Directive 96/98/EC (OJ L 257, 28.8.2014, p. 146).

43 Directive (EU) 2016/797 of the European Parliament and of the Council of 11 May 2016 on the interoperability of the rail system within the European Union (OJ L 138, 26.5.2016, p. 44).

44 Regulation (EU) 2018/858 of the European Parliament and of the Council of 30 May 2018 on the approval and market surveillance of motor vehicles and their trailers, and of systems, components and separate technical units intended for such vehicles, amending Regulations (EC) No 715/2007 and (EC) No 595/2009 and repealing Directive 2007/46/EC (OJ L 151, 14.6.2018, p. 1).

45 Regulation (EU) 2018/1139 of the European Parliament and of the Council of 4 July 2018 on common rules in the field of civil aviation and establishing a European Union Aviation Safety Agency, and amending Regulations (EC) No 2111/2005, (EC) No 1008/2008, (EU) No 996/2010, (EU) No 376/2014 and Directives 2014/30/EU and 2014/53/EU of the European Parliament and of the Council, and repealing Regulations (EC) No 552/2004 and (EC) No 216/2008 of the European Parliament and of the Council and Council Regulation (EEC) No 3922/91 (OJ L 212, 22.8.2018, p. 1).

46 Regulation (EU) 2019/2144 of the European Parliament and of the Council of 27 November 2019 on type-approval requirements for motor vehicles and their trailers, and systems, components and separate technical units intended for such vehicles, as regards their general safety and the protection of vehicle occupants and vulnerable road users, amending Regulation (EU) 2018/858 of the European Parliament and of the Council and repealing Regulations (EC) No 78/2009, (EC) No 79/2009 and (EC) No 661/2009 of the European Parliament and of the Council and Commission Regulations (EC) No 631/2009, (EU) No 406/2010, (EU) No 672/2010, (EU) No 1003/2010, (EU) No 1005/2010, (EU) No 1008/2010, (EU) No 1009/2010, (EU) No 19/2011, (EU) No 109/2011, (EU) No 458/2011, (EU) No 65/2012, (EU) No 130/2012, (EU) No 347/2012, (EU) No 351/2012, (EU) No 1230/2012 and (EU) 2015/166 (OJ L 325, 16.12.2019, p. 1).

Amendment 6

Proposal for a regulation

~~Recital 5 a (new)~~

Recital 44

Text proposed by the Commission

Amendment

(5a) In order to help promote the development, uptake and understanding of AI, the Union needs to put further effort into education and training, thus, inter alia, addressing the shortage of ICT professionals and AI undergraduate courses, digitally skilled workers as well as lack of even basic digital skills amongst a significant share of the population of the Union.

(44) High data quality is essential for the performance of many AI systems, especially when techniques involving the training of models are used, with a view to ensure that the high-risk AI system performs as intended and safely and it does not become the source of discrimination prohibited by Union law. High quality training, validation and testing data sets require the implementation of appropriate data governance and management practices. Training, validation and testing data sets should be sufficiently relevant, representative and free of errors and complete in view of the intended purpose of the system. They should also have the appropriate statistical properties, including as regards the persons or groups of persons on which the high-risk AI system is intended to be used. In particular, training, validation and testing data sets should take into account, to the extent required in the light of their intended purpose, the features, characteristics or elements that are particular to the specific geographical, behavioural or functional setting or context within which the AI system is intended to be used. In order to protect the right of others from the discrimination that might result from the bias in AI systems, the providers shouldbe able to process also special categories of personal data, as a matter of substantial public interest, in order to ensure the bias monitoring, detection and correction in relation to high-risk AI systems.

(44) High data quality is essential for the performance of many AI systems, especially when techniques involving the training of models are used, with a view to ensure that the high-risk AI system performs as intended and safely and it does not become the source of discrimination prohibited by Union law. High quality training, validation and testing data sets require the implementation of appropriate data governance and management practices. Training, validation and testing data sets are designed with the best possible efforts to ensure that they are relevant, representative, free of errors and appropriately vetted for errors in view of the intended purpose of the system. They should also have the appropriate statistical properties, including as regards the persons or groups of persons on which the high-risk AI system is intended to be used, with specific attention to the mitigation of possible biases in the datasets, that might lead to risks to fundamental rights or discriminatory outcomes for the persons affected by the high-risk AI system. In particular, training, validation and testing data sets should take into account, to the extent required in the light of their intended purpose, the features, characteristics or elements that are particular to the specific geographical, behavioural, contextual or functional setting or context within which the AI system is intended to be used, with specific attention to women, vulnerable groups and children. In order to protect the right of others from the discrimination that might result from the bias in AI systems, the providers should be able to process also special categories of personal data, as a matter of substantial public interest, in order to ensure the bias monitoring, detection and correction in relation to high-risk AI systems.

Amendment 7

Proposal for a regulation

~~Recital 5 b (new)~~

Recital 46

Text proposed by the Commission

Amendment

(5b) Lack of both public and private investment is currently undermining development and use of AI systems across the Union, especially when compared to other major industrial economies. Special attention, incentives and support should be devised to promote AI uptake amongst SMEs, including those in education and cultural and creative sectors and industries.

(46) Having information on how high-risk AI systems have been developed and how they perform throughout their lifecycle is essential to verify compliance with the requirements under this Regulation. This requires keeping records and the availability of a technical documentation, containing information which is necessary to assess the compliance of the AI system with the relevant requirements. Such information should include the general characteristics, capabilities and limitations of the system, algorithms, data, training, testing and validation processes used as well as documentation on the relevant risk management system. The technical documentation should be kept up to date.

(46) Having comprehensible information on how high-risk AI systems have been developed and how they perform throughout their lifecycle is essential to verify compliance with the requirements under this Regulation and to allow users to make informed and autonomous decisions about their use. This requires keeping records and the availability of a technical documentation, containing information which is necessary to assess the compliance of the AI system with the relevant requirements. Such information should include the general characteristics, capabilities and limitations of the system, algorithms, data, training, testing and validation processes used as well as documentation on the relevant risk management system. The technical documentation should be kept up to date.

Amendment 8

Proposal for a regulation

Recital 949

Text proposed by the Commission

Amendment

(9) For the purposes of this Regulation the notion of publicly accessible space should be understood as referring to any physical place that is accessible to the public, irrespective of whether the place in question is privately or publicly owned. Therefore, the notion does not cover places that are private in nature and normally not freely accessible for third parties, including law enforcement authorities, unless those parties have been specifically invited or authorised, such as homes, private clubs, offices, warehouses and factories. Online spaces are not covered either, as they are not physical spaces. However, the mere fact that certain conditions for accessing a particular space may apply, such as admission tickets or age restrictions, does not mean that the space is not publicly accessible within the meaning of this Regulation. Consequently, in addition to public spaces such as streets, relevant parts of government buildings and most transport infrastructure, spaces such as cinemas, theatres, shops and shopping centres are normally also publicly accessible. Whether a given space is accessible to the public should however be determined on a case-by-case basis, having regard to the specificities of the individual situation at hand.

(9) For the purposes of this Regulation the notion of publicly accessible space should be understood as referring to any physical or virtual place that is accessible to the public, irrespective of whether the place in question is privately or publicly owned. Therefore, the notion does not cover places that are private in nature and normally not freely accessible for third parties, including law enforcement authorities, unless those parties have been specifically invited or authorised, such as homes, private clubs, offices, warehouses and factories, and other private spaces. The same principle should apply to protected virtual publicly accessible spaces. However, the mere fact that certain conditions for accessing a particular space may apply, such as admission tickets or age restrictions, does not mean that the space is not publicly accessible within the meaning of this Regulation. Consequently, in addition to public spaces such as streets, parks, sport complexes relevant parts of government buildings and most transport infrastructure, spaces such as cinemas, theatres, shops, museums, libraries monuments, cultural sites, cultural institutions and shopping centres are normally also publicly accessible. Whether a given space is accessible to the public should however be determined on a case-by-case basis, having regard to the specificities of the individual situation at hand.

(49) High-risk AI systems should perform consistently throughout their lifecycle and meet an appropriate level of accuracy, robustness and cybersecurity in accordance with the generally acknowledged state of the art. Accuracy metrics and their expected level should be defined with the primary objective to mitigate risks and negative impact of the AI system to individuals and the society as a whole. The expected level of accuracy and accuracy metrics should be communicated in a clear, transparent, easily understandable and intelligible way to the users. The declaration of accuracy metrics cannot however be considered proof of future levels but relevant methods need to be applied to ensure sustainable levels during use. While standardisation organisations exist to establish standards, coordination on benchmarking is needed to establish how these standards should be met and measured. The European Artificial Intelligence Board should bring together national metrology and benchmarking authorities and provide non-binding guidance to address the technical aspects of to how to measure the appropriate levels of accuracy and robustness.

Amendment 9

Proposal for a regulation

Recital 1350

Text proposed by the Commission

Amendment

(13) In order to ensure a consistent and high level of protection of public interests as regards health, safety and fundamental rights, common normative standards for all high-risk AI systems should be established. Those standards should be consistent with the Charter of fundamental rights of the European Union (the Charter) and should be non-discriminatory and in line with the Union’s international trade commitments.

(50) The technical robustness is a key requirement for high-risk AI systems. They should be resilient against risks connected to the limitations of the system (e.g. errors, faults, inconsistencies, unexpected situations) as well as against malicious actions that may compromise the security of the AI system and result in harmful or otherwise undesirable behaviour. Failure to protect against these risks could lead to safety impacts or negatively affect the fundamental rights, for example due to erroneous decisions or wrong or biased outputs generated by the AI system.

(13) In order to ensure a consistent and high level of protection of public interests as regards health, safety, fundamental rights, democracy, the rule of law, as well as the environment, a set of ethical principles and common normative standards for all high-risk AI systems should be established. Those standards should be consistent with the Charter of fundamental rights of the European Union (the Charter), the communication of the Commission of 11 December 2019 entitled ‘The European Green Deal’ and the European Declaration of 26 January 2022 on Digital Rights and Principles for the Digital Decade and should be non-discriminatory and in line with the Union’s international trade commitments.

Amendment 10

Proposal for a regulation

~~Recital 14 b (new)~~

Recital 51

Text proposed by the Commission

Amendment

(14b) AI literacy refers to skills, knowledge and understanding that allows both citizens and operators in the context of the obligations set out in this Regulation to make an informed deployment and use of AI systems, as well as to gain awareness about the opportunities and risks of AI and thereby promote its democratic control. AI literacy should not be limited to learning about tools and technologies, but should also aim to equip citizens and operators in the context of the obligations set out in this Regulation with the critical thinking skills required to identify harmful or manipulative uses, as well as to improve their agency and their ability to fully comply with and benefit from trustworthy AI. It is therefore necessary that the Commission, the Member States and operators of AI systems, in cooperation with all relevant stakeholders, promote the development of AI literacy, in all sectors of society, for citizens of all ages, including women and girls, and that progress in that regard is closely followed.

(51) Cybersecurity plays a crucial role in ensuring that AI systems are resilient against attempts to alter their use, behaviour, performance or compromise their security properties by malicious third parties exploiting the system’s vulnerabilities. Cyberattacks against AI systems can leverage AI specific assets, such as training data sets (e.g. data poisoning) or trained models (e.g. adversarial attacks or confidentiality attacks), or exploit vulnerabilities in the AI system’s digital assets or the underlying ICT infrastructure. To ensure a level of cybersecurity appropriate to the risks, suitable measures should therefore be taken by the providers of high-risk AI systems, as well as the notified bodies, competent national authorities and market surveillance authorities, also taking into account as appropriate the underlying ICT infrastructure. High-risk AI should be accompanied by security solutions and patches for the lifetime of the product, or in case of the absence of dependence on a specific product, for a time that needs to be stated by the manufacturer.

Amendment 11

Proposal for a regulation

Recital 1561

Text proposed by the Commission

Amendment

(15) Aside from the many beneficial uses of artificial intelligence, that technology can also be misused and provide novel and powerful tools for manipulative, exploitative and social control practices. Such practices are particularly harmful and should be prohibited because they contradict Union values of respect for human dignity, freedom, equality, democracy and the rule of law and Union fundamental rights, including the right to non-discrimination, data protection and privacy and the rights of the child.

(15) Aside from the many beneficial uses of artificial intelligence, that technology can also be misused and provide novel and powerful tools for manipulative, exploitative and social control practices. Such practices are particularly harmful and should be prohibited because they contradict Union values of respect for human dignity, freedom, equality, democracy and the rule of law and Union fundamental rights, including the right to non-discrimination, the protection of employees and workers, data protection and privacy and gender equality and the rights of the child.

(61) Standardisation should play a key role to provide technical solutions to providers to ensure compliance with this Regulation. Compliance with harmonised standards as defined in Regulation (EU) No 1025/2012 of the European Parliament and of the Council54 should be a means for providers to demonstrate conformity with the requirements of this Regulation. In addition to technical details, the standardisation process should also take into account risks to fundamental rights, the environment, and society as a whole and other democratic and sociotechnical aspects of the AI system, and should ensure that the relevant subject-matter experts are included and consulted in the standardisation process. The standardisation process should be transparent in terms of legal and natural persons participating in the standardisation activities. However, the Commission could adopt common technical specifications in areas where no harmonised standards exist or where they are insufficient. In developing these common specifications Commission should involve views of relevant stakeholders, in particular when the common specifications address specific fundamental rights concerns. In particular, the Commission should adopt common specifications setting out how risk management systems give specific consideration to impact on children.

~~Amendment 12Proposal for a regulation~~

__________________

~~Recital 27~~

__________________

54 Regulation (EU) No 1025/2012 of the European Parliament and of the Council of 25 October 2012 on European standardisation, amending Council Directives 89/686/EEC and 93/15/EEC and Directives 94/9/EC, 94/25/EC, 95/16/EC, 97/23/EC, 98/34/EC, 2004/22/EC, 2007/23/EC, 2009/23/EC and 2009/105/EC of the European Parliament and of the Council and repealing Council Decision 87/95/EEC and Decision No 1673/2006/EC of the European Parliament and of the Council (OJ L 316, 14.11.2012, p. 12).

Amendment 12

Proposal for a regulation

Recital 71

Text proposed by the Commission

Amendment

(27) High-risk AI systems should only be placed on the Union market or put into service if they comply with certain mandatory requirements. Those requirements should ensure that high-risk AI systems available in the Union or whose output is otherwise used in the Union do not pose unacceptable risks to important Union public interests as recognised and protected by Union law. AI systems identified as high-risk should be limited to those that have a significant harmful impact on the health, safety and fundamental rights of persons in the Union and such limitation minimises any potential restriction to international trade, if any.

(27) High-risk AI systems should only be placed on the Union market or put into service if they comply with certain mandatory requirements. Those requirements should ensure that high-risk AI systems available in the Union or whose output is otherwise used in the Union do not pose unacceptable risks to important Union public interests as recognised and protected by Union law. AI systems identified as high-risk should be limited to those that have a harmful impact on the health, safety and fundamental rights of persons in the Union, as well as on society and on the environment, and such limitation minimises any potential restriction to international trade, if any.

(71) Artificial intelligence is a rapidly developing family of technologies that requires novel forms of regulatory oversight and a safe space for experimentation, while ensuring responsible innovation and integration of appropriate and ethically justified safeguards and risk mitigation measures. To ensure a legal framework that is innovation-friendly, future-proof and resilient to disruption, national competent authorities from one or more Member States should be encouraged to establish artificial intelligence regulatory sandboxes and make such regulatory sandboxes widely available throughout the Union, in order to facilitate the development and testing of innovative AI systems under strict regulatory oversight before these systems are placed on the market or otherwise put into service. Any significant risks identified during the development and testing of AI systems shall result in immediate mitigation and, failing that, in the suspension of the development and testing process until such mitigation takes place.

Amendment 13

Proposal for a regulation

Recital 2872

Text proposed by the Commission

Amendment

(28) AI systems could produce adverse outcomes to health and safety of persons, in particular when such systems operate as components of products. Consistently with the objectives of Union harmonisation legislation to facilitate the free movement of products in the internal market and to ensure that only safe and otherwise compliant products find their way into the market, it is important that the safety risks that may be generated by a product as a whole due to its digital components, including AI systems, are duly prevented and mitigated. For instance, increasingly autonomous robots, whether in the context of manufacturing or personal assistance and care should be able to safely operate and performs their functions in complex environments. Similarly, in the health sector where the stakes for life and health are particularly high, increasingly sophisticated diagnostics systems and systems supporting human decisions should be reliable and accurate. The extent of the adverse impact caused by the AI system on the fundamental rights protected by the Charter is of particular relevance when classifying an AI system as high-risk. Those rights include the right to human dignity, respect for private and family life, protection of personal data, freedom of expression and information, freedom of assembly and of association, and non-discrimination, consumer protection, workers’ rights, rights of persons with disabilities, right to an effective remedy and to a fair trial, right of defence and the presumption of innocence, right to good administration. In addition to those rights, it is important to highlight that children have specific rights as enshrined in Article 24 of the EU Charter and in the United Nations Convention on the Rights of the Child (further elaborated in the UNCRC General Comment No. 25 as regards the digital environment), both of which require consideration of the children’s vulnerabilities and provision of such protection and care as necessary for their well-being. The fundamental right to a high level of environmental protection enshrined in the Charter and implemented in Union policies should also be considered when assessing the severity of the harm that an AI system can cause, including in relation to the health and safety of persons.

(72) The objectives of the regulatory sandboxes should be to foster AI innovation by establishing a controlled experimentation and testing environment in the development and pre-marketing phase with a view to ensuring compliance of the innovative AI systems with this Regulation and other relevant Union and Member States legislation; to enhance legal certainty for innovators and the competent authorities’ oversight and understanding of the opportunities, emerging risks and the impacts of AI use, and to accelerate access to markets, including by removing barriers for small and medium enterprises (SMEs) and start-ups. To ensure uniform implementation across the Union and economies of scale, it is appropriate to establish common rules for the regulatory sandboxes’ implementation and a framework for cooperation between the relevant authorities involved in the supervision of the sandboxes. This Regulation should provide the legal basis for the use of personal data collected for other purposes for developing certain AI systems in the public interest within the AI regulatory sandbox, in line with Article 6(4) of Regulation (EU) 2016/679, and Article 6 of Regulation (EU) 2018/1725, and without prejudice to Article 4(2) of Directive (EU) 2016/680. Participants in the sandbox should ensure appropriate safeguards and cooperate with the competent authorities, including by following their guidance and acting expeditiously and in good faith to mitigate any high-risks to safety and fundamental rights that may arise during the development and experimentation in the sandbox. The conduct of the participants in the sandbox should be taken into account when competent authorities decide whether to impose an administrative fine under Article 83(2) of Regulation 2016/679 and Article 57 of Directive 2016/680.

(28) AI systems could produce adverse outcomes to health and safety of persons, in particular when such systems operate as components of products. Consistently with the objectives of Union harmonisation legislation to facilitate the free movement of products in the internal market and to ensure that only safe and otherwise compliant products find their way into the market, it is important that the safety risks that may be generated by a product as a whole due to its digital components, including AI systems, are duly prevented and mitigated. For instance, increasingly autonomous robots, whether in the context of manufacturing or personal assistance and care should be able to safely operate and performs their functions in complex environments. Similarly, in the health sector where the stakes for life and health are particularly high, increasingly sophisticated diagnostics systems and systems supporting human decisions should be reliable and accurate. The extent of the adverse impact caused by the AI system on the fundamental rights protected by the Charter is of particular relevance when classifying an AI system as high-risk. Those rights include the right to human dignity, respect for private and family life, protection of personal data, freedom of expression and information, freedom of assembly and of association, and non-discrimination, right to education, consumer protection, workers’ rights. Special attention should be paid to gender equality, rights of persons with disabilities, right to an effective remedy and to a fair trial, right of defence and the presumption of innocence, right to good administration, protection of intellectual property rights and ensuring cultural diversity. In addition to those rights, it is important to highlight that children have specific rights as enshrined in Article 24 of the EU Charter and in the United Nations Convention on the Rights of the Child (further elaborated in the UNCRC General Comment No. 25 as regards the digital environment), both of which require consideration of the children’s vulnerabilities and provision of such protection and care as necessary for their well-being. The fundamental right to a high level of environmental protection enshrined in the Charter and implemented in Union policies should also be considered when assessing the harm that an AI system can cause, including in relation to the health and safety of persons or to the environment, taking into account the extraction and consumption of natural resources, waste and the carbon footprint of those AI systems.

(72) The objectives of the regulatory sandboxes should be to foster AI innovation by establishing a controlled experimentation and testing environment in the development and pre-marketing phase with a view to ensuring compliance of the innovative AI systems with this Regulation and other relevant Union and Member States legislation, as well as with the Charter of fundamental rights of the European Union and the General Data Protection Regulation; to enhance legal certainty for innovators and the competent authorities’ oversight and understanding of the opportunities, emerging risks and the impacts of AI use, to provide safeguards needed to build trust and reliance on AI systems and to accelerate access to markets, including by removing barriers for small and medium enterprises (SMEs) and start-ups; to contribute to achieving the targets on AI as set in the Policy Programme “Path to the Digital Decade"; to contribute to the development of ethical, socially responsible and environmentally sustainable AI systems; to permit effective participation of SMEs and start-ups in regulatory sandboxes, compliance costs should be kept to a reasonable level to ensure the development of trustworthy European artificial intelligence solutions; it is appropriate to establish common rules for the regulatory sandboxes’ implementation and a framework for cooperation between the relevant authorities involved in the supervision of the sandboxes, while encouraging innovation. This Regulation should provide the legal basis for the use of personal data collected for other purposes for developing certain AI systems in the public interest within the AI regulatory sandbox, in line with Article 6(4) of Regulation (EU) 2016/679, and Article 6 of Regulation (EU) 2018/1725, and without prejudice to Article 4(2) of Directive (EU) 2016/680. Participants in the sandbox should ensure appropriate safeguards and cooperate with the competent authorities, including by following their guidance and acting expeditiously and in good faith to mitigate any high-risks to safety and fundamental rights that may arise during the development and experimentation in the sandbox. The conduct of the participants in the sandbox should be taken into account when competent authorities decide whether to impose an administrative fine under Article 83(2) of Regulation 2016/679 and Article 57 of Directive 2016/680.

Amendment 14

Proposal for a regulation

~~Recital 33~~

Recital 72 a (new)

Text proposed by the Commission

Amendment

(33) Technical inaccuracies of AI systems intended for the remote biometric identification of natural persons can lead to biased results and entail discriminatory effects. This is particularly relevant when it comes to age, ethnicity, sex or disabilities. Therefore, ‘real-time’ and ‘post’ remote biometric identification systems should be classified as high-risk. In view of the risks that they pose, both types of remote biometric identification systems should be subject to specific requirements on logging capabilities and human oversight.

(72a) It is desirable for the establishment of regulatory sandboxes, which is at present left to the discretion of Member States, as a next step to be made obligatory, with properly established criteria, to ensure both the effectiveness of the AI system and easier access for enterprises, in particular SMEs. Research enterprises and institutions should be involved in developing the conditions for the creation of regulatory sandboxes.

(33) Technical inaccuracies of AI systems intended for the biometric identification of natural persons can lead to biased results and entail discriminatory effects. This is particularly relevant when it comes to age, ethnicity, sex or disabilities. Therefore, ‘real-time’ and ‘post’ remote biometric identification systems should be classified as high-risk. In view of the risks that they pose, both types of biometric identification systems should be subject to specific requirements on logging capabilities and human oversight. Non-remote biometric identification systems intended to be used in publicly accessible spaces, workplaces and education and training institutions can also present a high risk. The high risk of non-remote biometric identification systems intended to be used in publicly accessible spaces, workplaces and education and training institutions should be determined on a case-by-case basis.

Amendment 15

Proposal for a regulation

~~Recital 34 a (new)~~

Recital 73

Text proposed by the Commission

Amendment

(34a) Deployment of AI systems in education is crucial in order to help modernise entire education systems, to increase educational quality, both offline and online, and to accelerate digital education, thus also making it available to a broader audience. AI-aided digital education, whilst not a replacement for an in-person learning, is increasingly necessary to promote societal and economic growth, promote inclusiveness and increase educational attainment and accessibility to individuals.

(73) In order to promote and protect innovation, it is important that the interests of small-scale providers and users of AI systems are taken into particular account. To this objective, Member States should develop initiatives, which are targeted at those operators, including on AI literacy, awareness raising and information communication. Member States should utilise existing channels and where appropriate, establish new dedicated channels for communication with SMEs, start-ups, users and other innovators to provide guidance and respond to queries about the implementation of this Regulation. Such existing channels could include, inter alia, ENISA’s Computer Security Incident Response Teams, National Data Protection Agencies, the AI-on demand platform, the European Digital Innovation Hubs and other relevant instruments funded by EU programmes as well as the Testing and Experimentation Facilities established by the Commission and the Member States at national or Union level. Where appropriate, these channels should work together to create synergies and ensure homogeneity in their guidance to start-ups, SMEs and users. Moreover, the specific interests and needs of small-scale providers shall be taken into account when Notified Bodies set conformity assessment fees. The Commission should regularly assess the certification and compliance costs for SMEs and start-ups, including through transparent consultations with SMEs, start-ups and users and work with Member States to lower such costs. For example, translation costs related to mandatory documentation and communication with authorities may constitute a significant cost for providers and other operators, notably those of a smaller scale. Member States should possibly ensure that one of the languages determined and accepted by them for relevant providers’ documentation and for communication with operators is one which is broadly understood by the largest possible number of cross-border users. Medium-sized enterprises which recently changed from the small to medium-size category within the meaning of Recommendation 2003/361/EC should have access to these initiatives and guidance for a period of time deemed appropriate by the Member States, as these new medium-sized enterprises may sometimes lack the legal resources and training necessary to ensure proper understanding and compliance with provisions.

Amendment 16

Proposal for a regulation

~~Recital 35~~

Recital 76 a (new)

Text proposed by the Commission

Amendment

(35) AI systems used in education or vocational training, notably for determining access or assigning persons to educational and vocational training institutions or to evaluate persons on tests as part of or as a precondition for their education should be considered high-risk, since they may determine the educational and professional course of a person’s life and therefore affect their ability to secure their livelihood. When improperly designed and used, such systems may violate the right to education and training as well as the right not to be discriminated against and perpetuate historical patterns of discrimination.

(76a) An AI advisory council (‘the Advisory Council’) should be established as a sub-group of the Board consisting of relevant representatives from industry, research, academia, civil society, standardisation organisations, social partners, SMEs, fundamental rights experts and other relevant stakeholders representing all Member States to maintain geographical balance. The Advisory Council should support the work of the Board by providing advice relating to the tasks of the Board. The Advisory Council should nominate a representative to attend meetings of the Board and to participate in its work.

(35) AI systems used in education or training, notably for determining access or assigning persons to educational and training institutions to evaluate persons on tests as part of or as a precondition for their education or for determining the areas of study a student should follow should be considered high-risk, since they may determine the educational and professional course of a person’s life and therefore affect their ability to secure their livelihood. When improperly designed, developed and used, such systems may violate the right to education and training as well as the right not to be discriminated against and perpetuate historical patterns of discrimination, for example against women, certain age groups, persons with disabilities, or persons of certain racial or ethnic origins or sexual orientation. AI systems used to monitor students’ behaviour and emotion during tests at education and training institutions should be considered high-risk, since they are also interfering with students’ rights to privacy and data protection. The use of AI to check assessments, such as exam papers for plagiarism, should not be considered high-risk.

Amendment 17

Proposal for a regulation

Recital 3681

Text proposed by the Commission

Amendment

(36) AI systems used in employment, workers management and access to self-employment, notably for the recruitment and selection of persons, for making decisions on promotion and termination and for task allocation, monitoring or evaluation of persons in work-related contractual relationships, should also be classified as high-risk, since those systems may appreciably impact future career prospects and livelihoods of these persons. Relevant work-related contractual relationships should involve employees and persons providing services through platforms as referred to in the Commission Work Programme 2021. Such persons should in principle not be considered users within the meaning of this Regulation. Throughout the recruitment process and in the evaluation, promotion, or retention of persons in work-related contractual relationships, such systems may perpetuate historical patterns of discrimination, for example against women, certain age groups, persons with disabilities, or persons of certain racial or ethnic origins or sexual orientation. AI systems used to monitor the performance and behaviour of these persons may also impact their rights to data protection and privacy.

(81) The development of AI systems other than high-risk AI systems in accordance with the requirements of this Regulation may lead to a larger uptake of trustworthy artificial intelligence in the Union. Providers of non-high-risk AI systems should be encouraged to create codes of conduct intended to foster the voluntary application of the mandatory requirements applicable to high-risk AI systems. Providers should also be encouraged to apply on a voluntary basis additional requirements related, for example, to environmental sustainability, accessibility to persons with disability, stakeholders’ participation in the design and development of AI systems, and diversity of the development teams. The Commission may develop initiatives, including of a sectorial nature, to facilitate the lowering of technical barriers hindering cross-border exchange of data for AI development, including on data access infrastructure, semantic and technical interoperability of different types of data.

(36) AI systems used in employment, employment support workers management and access to self-employment, notably for the recruitment and selection of persons, for making decisions on promotion and termination and for task allocation, for monitoring compliance with workplace rules and for monitoring or evaluation of persons in work-related relationships, should also be classified as high-risk, since those systems may appreciably impact the health, safety and security rules applicable in their work and at their workplaces and future career prospects and livelihoods of these persons. Relevant work-related relationships should involve employees and persons providing services through platforms as referred to in the Commission Work Programme 2021. Such persons should in principle not be considered users within the meaning of this Regulation. Throughout the recruitment process and in the evaluation, promotion, or retention of persons in work-related relationships, such systems may perpetuate historical patterns of discrimination, for example against women, certain age groups, persons with disabilities, or persons of certain racial or ethnic origins or sexual orientation. AI systems used to monitor the performance and behaviour of these persons may also impact their rights to data protection and privacy. In this regard, specific requirements on transparency, information and human oversight should apply. Trade unions and workers’ representatives should be informed and they should have access to any relevant documentation created under this Regulation for high-risk AI systems deployed or used in their work or at their workplace.

(81) The development of AI systems other than high-risk AI systems in accordance with the requirements of this Regulation may lead to a larger uptake of trustworthy, socially responsible and environmentally sustainable artificial intelligence in the Union. Providers of non-high-risk AI systems should be encouraged to create codes of conduct intended to foster the voluntary application of the mandatory requirements applicable to high-risk AI systems. Providers should also be encouraged to apply on a voluntary basis additional requirements related, for example, to environmental sustainability, accessibility to persons with disability, stakeholders’ participation in the design and development of AI systems, and diversity of the development teams. The Commission may develop initiatives, including of a sectorial nature, to facilitate the lowering of technical barriers hindering cross-border exchange of data for AI development, including on data access infrastructure, semantic and technical interoperability of different types of data.

Amendment 18

Proposal for a regulation

~~Recital 70~~

Article 2 – paragraph 5 a (new)

Text proposed by the Commission

Amendment

(70) Certain AI systems intended to interact with natural persons or to generate content may pose specific risks of impersonation or deception irrespective of whether they qualify as high-risk or not. In certain circumstances, the use of these systems should therefore be subject to specific transparency obligations without prejudice to the requirements and obligations for high-risk AI systems. In particular, natural persons should be notified that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use. Moreover, natural persons should be notified when they are exposed to an emotion recognition system or a biometric categorisation system. Such information and notifications should be provided in accessible formats for persons with disabilities. Further, users, who use an AI system to generate or manipulate image, audio or video content that appreciably resembles existing persons, places or events and would falsely appear to a person to be authentic, should disclose that the content has been artificially created or manipulated by labelling the artificial intelligence output accordingly and disclosing its artificial origin.

5a. This Regulation shall not affect research activities regarding AI systems insofar as such activities do not lead to or entail placing an AI system on the market or putting it into service. These research activities shall not violate the fundamental rights of the affected persons.

(70) Certain AI systems used to interact with natural persons or to generate content may pose specific risks of impersonation or deception irrespective of whether they qualify as high-risk or not. In certain circumstances, the use of these systems should therefore be subject to specific transparency obligations without prejudice to the requirements and obligations for high-risk AI systems. In particular, natural persons should be notified that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use or where the content is evidently used to form part of a creative, artistic or fictional cinematographic work. Moreover, natural persons should be notified when they are exposed to an emotion recognition system or a biometric categorisation system. Such information and notifications should include a disclaimer and should be provided inaccessible formats for children, the elderly, migrants and persons with disabilities or other vulnerabilities. Further, users, who use an AI system to generate or manipulate image, audio or video content, texts or scripts that appreciably resembles existing persons, places or events and would falsely appear to a person to be authentic, should disclose in a clear manner that the content has been artificially created or manipulated by labelling the artificial intelligence output accordingly and disclosing its artificial origin.

Amendment 19

Proposal for a regulation

~~Recital 73~~

Article 2 – paragraph 5 b (new)

Text proposed by the Commission

Amendment

5b. This Regulation shall not apply to AI systems, including their output, specifically developed and put into service for the sole purpose of scientific research in the general interest of the Union.

Amendment 20

Proposal for a regulation

~~Recital 74~~

Article 3 – paragraph 1 – point 1

Text proposed by the Commission

Amendment

(1) ‘artificial intelligence system’ (AI system) means software that is developed with one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with;

(74) In order to minimise the risks to implementation resulting from lack of knowledge and expertise in the market as well as to facilitate compliance of providers and notified bodies with their obligations under this Regulation, the AI-on demand platform, the European Digital Innovation Hubs and the Testing and Experimentation Facilities established by the Commission and the Member States at national or EU level should possibly contribute to the implementation of this Regulation. Within their respective mission and fields of competence, they may provide in particular technical and scientific support to providers and notified bodies. It is necessary for the Commission to also create a pan-European network of universities and researchers focused on AI for enhanced studying and research on the impact of AI and to update the Digital Education Action Plan established in the communication of the Commission of 30 September 2020 entitled ‘Digital Education Action Plan 2021-2027 – Resetting education and training for the digital age’, in order to integrate AI and robotics innovation in education.

(1) ‘artificial intelligence system’ (AI system) means a machine-based system that can, with varying levels of autonomy, for a given set of human-defined objectives, make predictions, content, recommendations, or decisions influencing real or virtual environments they interact with;

Amendment 21

Proposal for a regulation

~~Recital 76~~

Article 3 – paragraph 1 – point 1 a (new)

Text proposed by the Commission

Amendment

(76) In order to facilitate a smooth, effective and harmonised implementation of this Regulation a European Artificial Intelligence Board should be established. The Board should be responsible for a number of advisory tasks, including issuing opinions, recommendations, advice or guidance on matters related to the implementation of this Regulation, including on technical specifications or existing standards regarding the requirements established in this Regulation and providing advice to and assisting the Commission on specific questions related to artificial intelligence.

(1a) ‘autonomy’ means that an AI system operates by interpreting certain input and by using a set of pre-determined objectives, without being limited to such instructions, despite the system’s behaviour being constrained by, and targeted at, fulfilling the goal it was given and other relevant design choices made by its developer;

(76) In order to facilitate a smooth, effective and harmonised implementation of this Regulation a European Artificial Intelligence Board should be established. The Board should be responsible for a number of advisory tasks, including issuing opinions, recommendations, advice or guidance on matters related to the implementation of this Regulation, including on technical specifications or existing standards regarding the requirements established in this Regulation and providing expert advice to and assisting the Commission on specific questions related to artificial intelligence and to addressing the challenges rising from the fast evolving development of AI technologies.

Amendment 22

Proposal for a regulation

~~Recital 83~~

Article 3 – paragraph 1 – point 2

Text proposed by the Commission

Amendment

(1) ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or that has an AI system developed with a view to placing it on the market or putting it into service under its own name or trademark, whether for payment or free of charge;

(83) In order to ensure trustful and constructive cooperation of competent authorities on Union and national level, all parties involved in the application of this Regulation should respect the confidentiality of information and data obtained in carrying out their tasks. It is appropriate for a new set of common European guidelines and standards to be set up in order to protect privacy while making an effective use of the data available.

(2) ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or that has an AI system developed and places that system on the market or puts it into service under its own name or trademark, whether for payment or free of charge;

Amendment 23

Proposal for a regulation

~~Recital 85~~

Article 3 – paragraph 1 – point 14

Text proposed by the Commission

Amendment

(85) In order to ensure that the regulatory framework can be adapted where necessary, the power to adopt acts in accordance with Article 290 TFEU should be delegated to the Commission to amend the techniques and approaches referred to in Annex I to define AI systems, the Union harmonisation legislation listed in Annex II, the high-risk AI systems listed in Annex III, the provisions regarding technical documentation listed in Annex IV, the content of the EU declaration of conformity in Annex V, the provisions regarding the conformity assessment procedures in Annex VI and VII and the provisions establishing the high-risk AI systems to which the conformity assessment procedure based on assessment of the quality management system and assessment of the technical documentation should apply. It is of particular importance that the Commission carry out appropriate consultations during its preparatory work, including at expert level, and that those consultations be conducted in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making58 . In particular, to ensure equal participation in the preparation of delegated acts, the European Parliament and the Council receive all documents at the same time as Member States’ experts, and their experts systematically have access to meetings of Commission expert groups dealing with the preparation of delegated acts.

(14) ‘safety component of a product or system’ means a component of a product or of a system which fulfils a safety function for that product or system or the failure or malfunctioning of which endangers the health and safety of persons or property;

(85) In order to ensure that the regulatory framework can be adapted where necessary, the power to adopt acts in accordance with Article 290 TFEU should be delegated to the Commission to amend the techniques and approaches referred to in Annex I to define AI systems, the Union harmonisation legislation listed in Annex II, the high-risk AI systems listed in Annex III, the provisions regarding technical documentation listed in Annex IV, the content of the EU declaration of conformity in Annex V, the provisions regarding the conformity assessment procedures in Annex VI and VII and the provisions establishing the high-risk AI systems to which the conformity assessment procedure based on assessment of the quality management system and assessment of the technical documentation should apply. It is of particular importance that the Commission carry out appropriate consultations during its preparatory work, including at expert level, and that those consultations be conducted in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making58. Such consultations should involve qualified specialists from different areas of society including from the private sector, researchers and scientists, education, media and culture representatives, trade unions, consumer, parental and data protection organizations with skills and knowledge relevant to the task. In particular, to ensure equal participation in the preparation of delegated acts, the European Parliament and the Council receive all documents at the same time as Member States’ experts, and their experts systematically have access to meetings of Commission expert groups dealing with the preparation of delegated acts.

(14) ‘safety component of a product or system’ means a component of a product or of a system which fulfils a safety function for that product or system and the failure or malfunctioning of which endangers the health and safety of persons or property;

~~__________________~~

~~58 OJ L 123, 12.5.2016, p. 1.~~

Amendment 24

Proposal for a regulation

~~Recital 86 a (new)~~

Article 3 – paragraph 1 – point 44 – introductory part

Text proposed by the Commission

Amendment

(86a) Given the rapid technological developments and the required technical expertise in conducting the assessment of high-risk AI systems, the delegation of powers and the implementing powers of the Commission should be exercised with as much flexibility as possible. The Commission should regularly review Annex III without undue delay, at least every six months, while consulting with the relevant stakeholders, including ethics experts, anthropologists, sociologists, mental health specialists and any other relevant scientists and researchers, as well as with parent associations.

(44) ‘serious incident’ means any incident that directly or indirectly leads, might have led or might lead to any of the following:

(44) ‘serious incident’ means any incident or malfunctioning of an AI system that directly or indirectly leads, might have led or might lead to any of the following:

Amendment 25

Proposal for a regulation

Article 13 – paragraph 1 – point c44 – point a

Text proposed by the Commission

Amendment

(c) harmonised transparency rules for AI systems intended to interact with natural persons, emotion recognition systems and biometric categorisation systems, and AI systems used to generate or manipulate image, audio or video content;

(a) the death of a person or serious damage to a person’s health, to property or the environment,

~~(c) harmonised transparency rules for AI systems;~~

(a) the death of a person or serious damage to a person’s fundamental rights, health, safety, property or the environment,

Amendment 26

Proposal for a regulation

Article 23 – paragraph 41 – point 44 – point b a (new)

Text proposed by the Commission

Amendment

~~4a. This Regulation shall not affect or undermine academic research or development of AI systems and their outputs for the purpose of academic research.~~

(ba) breach of obligations under Union law intended to protect fundamental rights.

Amendment 27

Proposal for a regulation

Article 3 – paragraph 1 – point 444 a (new)

Text proposed by the Commission

Amendment

(4) ‘user’ means any natural or legal person, public authority, agency or other body using an AI system under its authority, except where the AI system is used in the course of a personal non-professional activity;

(44a) ‘regulatory sandbox’ means a facility established by one or more Member States’ competent authorities in collaboration with the Commission or by the European Data Protection Supervisor, that provides an appropriate controlled and flexible environment to facilitate the safe development, testing and validation of innovative AI systems for a limited time before their placement on the market or putting into service pursuant to a specific plan;

(4) ‘user’ means any natural or legal person, public authority, educational and training institution, agency or other body using an AI system under its authority, except where the AI system is used in the course of a personal non-professional activity;

Amendment 28

Proposal for a regulation

Article 3 – paragraph 1 – point 3544 b (new)

Text proposed by the Commission

Amendment

(35) ‘biometric categorisation system’ means an AI system for the purpose of assigning natural persons to specific categories, such as sex, age, hair colour, eye colour, tattoos, ethnic origin or sexual or political orientation, on the basis of their biometric data;

(44b) ‘AI literacy’ means the skills, knowledge and understanding regarding AI systems that are necessary for compliance with and enforcement of this Regulation;

(35) ‘biometric categorisation system’ means an AI system that uses biometric data, or other physical, physiological or behavioural data, for the purpose of assigning natural persons to specific categories, such as sex, age, hair colour, eye colour, tattoos, ethnic origin or sexual or political orientation, on the basis of their biometric data;

Amendment 29

Proposal for a regulation

Article 3 – paragraph 1 – point 3944 c (new)

Text proposed by the Commission

Amendment

~~(39) ‘publicly accessible space’ means any physical place accessible to the public, regardless of whether certain conditions for access may apply;~~

(44c) ‘deep fake’ means manipulated or synthetic audio and/or visual material that gives an authentic impression, in which events appear to be taking place, which never happened, and which has been produced using techniques in the field of artificial intelligence, including machine learning and deep learning, without the user, or end-user being aware that the audio and/or visual material has been produced using artificial intelligence;

~~(39) ‘publicly accessible space’ means any place accessible to the public, regardless of whether certain conditions for access may apply;~~

Amendment 30

Proposal for a regulation

Article 3 – paragraph 1 – point 44 ~~– point~~d a(new)

Text proposed by the Commission

Amendment

~~(a) the death of a person or serious damage to a person’s health, to property or the environment,~~

(44d) ‘critical infrastructure’ means an asset, system or part thereof which is necessary for the delivery of a service that is essential for the maintenance of vital societal functions or economic activities within the meaning of Article 2(4) and (5) of Directive ____ on the resilience of critical entities (2020/0365(COD));

~~(a) the death of a person or serious damage to a person’s fundamental rights, health, to property or the environment, to democracy or the democratic rule of law,~~

Amendment 31

Proposal for a regulation

Article 3 – paragraph 1 – point 44 ae (new)

Text proposed by the Commission

Amendment

(44a) ‘education and training institutions’ means providers of education and training, irrespective of the age of the persons receiving the education and training, including preschools, childcare, primary schools, secondary schools, tertiary education providers, vocational education and training and any type of lifelong learning providers;

(44e) ‘personal data’ means personal data as defined in Article 4, point (1), of Regulation (EU) 2016/679;

Amendment 32

Proposal for a regulation

Article 3 – paragraph 1 – point 44 bf (new)

Text proposed by the Commission

Amendment

(44b) ‘cultural institutions’ means institutions such as libraries, museums, theatres, concert halls, exhibition centres, architectural ensembles and multi-purpose arts venues, as well as their virtual sections, which organise cultural education, democratic exchanges and research and provide ways and means of engaging with cultural heritage;

(44f) ‘non personal data’ means data other than personal data as defined in point (1) of Article 4 of Regulation (EU) 2016/679.

Amendment 33

Proposal for a regulation

Article 34 – paragraph 1 ~~– point 44 c (new)~~

Text proposed by the Commission

Amendment

~~(44c) 'deep fake' means manipulated or synthetic audio, visual or audiovisual content, text or scripts which feature persons purported to be authentic and truthful;~~

The Commission is empowered to adopt delegated acts in accordance with Article 73 to amend the list of techniques and approaches listed in Annex I, in order to update that list to market and technological developments on the basis of characteristics that are similar to the techniques and approaches listed therein.

The Commission is empowered to adopt delegated acts in accordance with Article 73 to amend the list of techniques and approaches listed in Annex I within the scope of the AI system as defined in Article 3, point (1), in order to update that list to market and technological developments on the basis of characteristics that are similar to the techniques and approaches listed therein.

Amendment 34

Proposal for a regulation

Article 34 – paragraph 1 ~~– point 44 d~~a (new)

Text proposed by the Commission

Amendment

~~(44d) 'AI literacy' means the skills, knowledge and understanding regarding AI systems.~~

When drafting these delegated acts, the Commission shall ensure the input of all relevant stakeholders such as technical experts and developers of AI systems.

Amendment 35

Proposal for a regulation

~~Article 4 a (new)~~

Article 10 – paragraph 1

Text proposed by the Commission

Amendment

~~Article 4a~~

1. High-risk AI systems which make use of techniques involving the training of models with data shall be developed on the basis of training, validation and testing data sets that meet the quality criteria referred to in paragraphs 2 to 5.

~~Trustworthy AI~~

1. High-risk AI systems which make use of techniques involving the training of models with data shall be developed on the basis of training, assessment, validation and testing data sets that meet the quality criteria referred to in paragraphs 2 to 5 as far as this is feasible from a technical point of view while taking into account the latest state-of-the-art measures, according to the specific market segment or scope of application.

~~1. All AI systems in the Union shall be developed, deployed and used in full respect of the Charter of Fundamental Rights of the European Union (the Charter).~~

~~2. In view of promoting trustworthy AI in the Union, and without prejudice to the requirements set out in Title III for high-risk AI systems, all AI systems shall be developed, deployed and used:~~

~~(a) in a lawful, fair and transparent manner (‘the principle of lawfulness, fairness and transparency’);~~

(b) in a manner that ensures that natural persons are always able to make informed decisions regarding such systems and that such systems do not undermine or override human autonomy (‘the principle of human agency and oversight’);

(c) in a manner that ensures their safe, accurate and reliable performance, with embedded safeguards to prevent any kind of individual or collective harm (‘the principle of safety, accuracy, reliability and robustness’);

~~(d) in a manner that guarantees privacy and data protection (‘the principle of privacy’);~~

~~(e) in a manner that privileges the integrity and quality of data, including with regard to access (‘the principle of data governance’);~~

(f) in a traceable, auditable and explainable manner that ensures responsibility and accountability for their outcomes and supports redress (‘the principle of traceability, auditability, explainability and accountability’);

(g) in a manner that does not discriminate against persons or groups of persons on the basis of unfair bias and that includes, to that end, the participation and input of relevant stakeholders(‘the principle of non-discrimination and diversity’);

(h) in an environmentally sustainable manner that minimises their environmental footprint, including with regard to the extraction and consumption of natural resources (‘the principle of environmental sustainability’);

(i) in a socially responsible manner that minimises their negative societal impact, especially with regard to social and gender inequalities and democratic processes (‘the principle of social responsibility’).

Amendment 36

Proposal for a regulation

~~Article 4 b (new)~~

Article 10 – paragraph 1 a (new)

Text proposed by the Commission

Amendment

~~Article 4b~~

1a. Techniques such as unsupervised learning and reinforcement learning, that do not use validation and testing data sets, shall be developed on the basis of training data sets that meet the quality criteria referred to in paragraphs 2 to 5.

~~AI literacy~~

1. When implementing this Regulation, the Union and the Member States shall promote measures and tools for the development of a sufficient level of AI literacy, across sectors and groups of operators concerned, including through education and training, skilling and reskilling programmes and while ensuring a proper gender and age balance, in view of allowing a democratic control of AI systems.

2. Providers and users of AI systems shall promote tools and shall take measures to ensure a sufficient level of AI literacy of their staff and any other persons dealing with the operation and use of AI systems on their behalf, taking into account their technical knowledge, experience, education and training and the environment in which the AI systems are to be used, and considering the persons or groups of persons on which the AI systems are to be used.

3. Such literacy tools and measures shall consist, in particular, of the teaching and learning of basic notions and skills about AI systems and their functioning, including the different types of products and uses, their risks and benefits and the severity of the harm they can cause and its probability of occurrence.

4. The level of AI literacy shall be considered to be sufficient where it contributes to the ability of operators to fully comply with and benefit from trustworthy AI, and in particular with the requirements laid down in this Regulation.

Amendment 37

Proposal for a regulation

Article 610 – paragraph 21 b (new)

Text proposed by the Commission

Amendment

~~2. In addition to the high-risk AI systems referred to in paragraph 1, AI systems referred to in Annex III shall also be considered high-risk.~~

1b. Providers of high-risk AI systems that utilise data collected and/or managed by third parties may rely on representations from those third parties with regard to quality criteria referred to in paragraph 2, points (a), (b) and (c)

2. In addition to the high-risk AI systems referred to in paragraph 1, AI systems referred to in Annex III shall also be considered high-risk due to their risk to cause harm to health, safety, the environment, fundamental rights or to democracy and the rule of law.

Amendment 38

Proposal for a regulation

Article 710 – paragraph 12 – ~~point~~introductory apart

Text proposed by the Commission

Amendment

~~(a) the AI systems are intended to be used in any of the areas listed in points 1 to 8 of Annex III;~~

2. Training, validation and testing data sets shall be subject to appropriate data governance and management practices. Those practices shall concern in particular,

~~(a) the AI systems are intended to be used in any of the areas listed in Annex III;~~

2. Training, assessment, validation and testing data sets shall be subject to appropriate data governance and management practices for the entire lifecycle of data processing. Those practices shall concern in particular, the following elements:

Amendment 39

Proposal for a regulation

Article 710 – paragraph 12 – point ba a (new)

Text proposed by the Commission

Amendment

(b) the AI systems pose a risk of harm to the health and safety, or a risk of adverse impact on fundamental rights, that is, in respect of its severity and probability of occurrence, equivalent to or greater than the risk of harm or of adverse impact posed by the high-risk AI systems already referred to in Annex III.

(aa) transparency as regards the original purpose of data collection;

(b) the AI systems pose a risk of harm to the health and safety, or a risk of adverse impact on fundamental rights, democracy and the rule of law, or the environment that is, in respect of its severity and probability of occurrence, equivalent to or greater than the risk of harm or of adverse impact posed by the high-risk AI systems already referred to in Annex III.

Amendment 40

Proposal for a regulation

Article 710 – paragraph 2 – ~~introductory~~point ~~part~~b

Text proposed by the Commission

Amendment

2. When assessing for the purposes of paragraph 1 whether an AI system poses a risk of harm to the health and safety or a risk of adverse impact on fundamental rights that is equivalent to or greater than the risk of harm posed by the high-risk AI systems already referred to in Annex III, the Commission shall take into account the following criteria:

(b) data collection;

2. When assessing for the purposes of paragraph 1 whether an AI system poses a risk of harm to the health and safety or a risk of adverse impact on fundamental rights, democracy and the rule of law, or the environment that is equivalent to or greater than the risk of harm posed by the high-risk AI systems already referred to in Annex III, the Commission shall take into account the following criteria:

(b) data collection processes;

Amendment 41

Proposal for a regulation

Article 710 – paragraph 2 a– ~~(new)~~point f

Text proposed by the Commission

Amendment

~~2a. The Commission shall conduct the assessment referred to in paragraph 2 annually under the consultation conditions laid down in Article 73.~~

(f) examination in view of possible biases;

(f) examination in view of possible biases that are likely to affect health and safety of persons, negatively impact fundamental rights or lead to discrimination prohibited by Union law; including the cases where data outputs are used as an input for future operations (‘feedback loops’);

Amendment 42

Proposal for a regulation

Article 910 – paragraph ~~4 – subparagraph~~ 2 – point ~~d a (new)~~g

Text proposed by the Commission

Amendment

~~(da) provision of a sufficient level of AI literacy;~~

(g) the identification of any possible data gaps or shortcomings, and how those gaps and shortcomings can be addressed.

(g) the identification of possible data gaps or shortcomings, and how those gaps and shortcomings can be addressed;

Amendment 43

Proposal for a regulation

Article 910 – paragraph 82 – point g a (new)

Text proposed by the Commission

Amendment

8. When implementing the risk management system described in paragraphs 1 to 7, specific consideration shall be given to whether the high-risk AI system is likely to be accessed by or have an impact on children.

(ga) the purpose and the environment in which the system is to be used.

Amendment 44

Proposal for a regulation

Article 10 – paragraph ~~2 – point f~~3

Text proposed by the Commission

Amendment

~~(f) examination in view of possible biases;~~

3. Training, validation and testing data sets shall be relevant, representative, free of errors and complete. They shall have the appropriate statistical properties, including, where applicable, as regards the persons or groups of persons on which the high-risk AI system is intended to be used. These characteristics of the data sets may be met at the level of individual data sets or a combination thereof.

~~(f) examination in view of possible biases, in particular deviations that could affect the health and safety of people or could lead to discrimination;~~

3. Training, validation and testing datasets are designed with the best possible efforts to ensure that they are relevant, representative and appropriately vetted for errors in view of the intended purpose of the AI system. In particular, they shall have the appropriate statistical properties, including, where applicable, as regards the persons or groups of persons on which the high-risk AI system is intended to be used. These characteristics of the data sets may be met at the level of individual data sets or a combination thereof.

Amendment 45

Proposal for a regulation

Article 10 – paragraph ~~2 – point g a (new)~~4

Text proposed by the Commission

Amendment

~~(ga) the purpose and the environment in which the system is to be used;~~

4. Training, validation and testing data sets shall take into account, to the extent required by the intended purpose, the characteristics or elements that are particular to the specific geographical, behavioural or functional setting within which the high-risk AI system is intended to be used.

4. Training, validation and testing data sets shall take into account, to the extent required by the intended purpose, the characteristics or elements that are particular to the specific geographical, behavioural, contextual or functional setting within which the high-risk AI system is intended to be used.

Amendment 46

Proposal for a regulation

Article 1315 – paragraph 1

Text proposed by the Commission

Amendment

1. High-risk AI systems shall be designed and developed in such a way ~~to ensure~~ that ~~their operation is sufficiently transparent to enable users~~they toachieve, ~~interpret~~in the ~~system’s output and use it appropriately. An appropriate type and degree~~light of ~~transparency shall be ensured, with a view to achieving~~their ~~compliance~~intended ~~with~~purpose, ~~the~~an ~~relevant~~appropriate ~~obligations~~level of ~~the~~accuracy, ~~user~~robustness and ~~of the~~cybersecurity, ~~provider~~and ~~set~~perform ~~out~~consistently in ~~Chapter~~those 3respects ofthroughout ~~this~~their ~~Title.~~lifecycle.

1. High-risk AI systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent to enable providers and users and other relevant stakeholders to easily interpret the system’s functioning and output and use it appropriately on the basis of informed decisions, with a view to achieving compliance with the relevant obligations set out in Chapter 3 of this Title.

1. High-risk AI systems shall be designed and developed following the principle of security by design and by default. In the light of their intended purpose, they should achieve an appropriate level of accuracy, robustness, safety, and cybersecurity, and perform consistently in those respects throughout their lifecycle. Compliance with these requirements shall include implementation of state-of-the-art measures, according to the specific market segment or scope of application.

Amendment 47

Proposal for a regulation

Article 1315 – paragraph 31 a (new)

Text proposed by the Commission

Amendment

~~3a. In order to comply with the obligations laid down in this Article, providers and users shall ensure a sufficient level of AI literacy in accordance with Article 4b.~~

1a. To address the technical aspects of to how to measure the appropriate levels of accuracy and robustness set out in paragraph 1 of this Article, the European Artificial Intelligence Board shall bring together national metrology and benchmarking authorities and provide non-binding guidance on the matter as set out in Article 56, paragraph 2, point (a).

Amendment 48

Proposal for a regulation

Article 1415 – paragraph 51 ab (new)

Text proposed by the Commission

Amendment

~~5a. In order to comply with the obligations laid down in this Article, providers and users shall ensure a sufficient level of AI literacy in accordance with Article 4b.~~

1b. To address any emerging issues across the internal market with regard to cybersecurity, the European Union Agency for Cybersecurity (ENISA) shall be involved alongside the European Artificial Intelligence Board as set out Article 56, paragraph 2, point (b).

Amendment 49

Proposal for a regulation

Article 2915 – paragraph ~~1 a (new)~~2

Text proposed by the Commission

Amendment

1a. In order to comply with the obligations laid down in this Article, as well as to be able to justify their possible non-compliance, users of high-risk AI systems shall ensure a sufficient level of AI literacy in accordance with Article 4b.

2. The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.

2. The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use. The language used shall be clear, free of misunderstandings or misleading statements.

Amendment 50

Proposal for a regulation

Article 4115 – paragraph 23 – subparagraph 1

Text proposed by the Commission

Amendment

~~2. The Commission, when preparing the common specifications referred to in paragraph 1, shall gather the views of relevant bodies or expert groups established under relevant sectorial Union law.~~

High-risk AI systems shall be resilient as regards errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems.

2. The Commission, when preparing the common specifications referred to in paragraph 1, shall gather the views of relevant stakeholders, including industry representatives, SMEs and other relevant bodies or expert groups established under relevant sectorial Union law.

Technical and organisational measures shall be taken to ensure that high-risk AI systems shall be as resilient as possible regarding errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems.

Amendment 51

Proposal for a regulation

Article 5215 – paragraph 13 – subparagraph 2

Text proposed by the Commission

Amendment

1. Providers shall ensure that AI systems intended to interact with natural persons are designed and developed in such a way that natural persons are informed that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use. This obligation shall not apply to AI systems authorised by law to detect, prevent, investigate and prosecute criminal offences, unless those systems are available for the public to report a criminal offence.

The robustness of high-risk AI systems may be achieved through technical redundancy solutions, which may include backup or fail-safe plans.

1. Providers shall ensure that AI systems used to interact with natural persons are designed and developed in such a way that natural persons are informed that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use. This obligation shall not apply to AI systems authorised by law to detect, prevent, investigate and prosecute criminal offences, unless those systems are available for the public to report a criminal offence.

The robustness of high-risk AI systems may be achieved by the appropriate provider with input from the user, where necessary, through technical redundancy solutions, which may include backup or fail-safe plans.

Amendment 52

Proposal for a regulation

Article 5215 – paragraph 23 – subparagraph 3

Text proposed by the Commission

Amendment

2. Users of an emotion recognition system or a biometric categorisation system shall inform of the operation of the system the natural persons exposed thereto. This obligation shall not apply to AI systems used for biometric categorisation, which are permitted by law to detect, prevent and investigate criminal offences.

High-risk AI systems that continue to learn after being placed on the market or put into service shall be developed in such a way to ensure that possibly biased outputs due to outputs used as an input for future operations (‘feedback loops’) are duly addressed with appropriate mitigation measures.

2. Users of an emotion recognition system or a biometric categorisation system shall inform, in a timely, clear and intelligible manner, of the operation of the system the natural persons exposed thereto. That information shall also include, as appropriate, the rights and processes to allow natural persons to appeal against the application of such AI systems to them. This obligation shall not apply to AI systems used for biometric categorisation, which are permitted by law to detect, prevent and investigate criminal offences.

High-risk AI systems that continue to learn after being placed on the market or put into service shall be developed in such a way to ensure that possibly biased outputs influencing input for future operations (‘feedback loops’) and malicious manipulation of inputs used in learning during operation are duly addressed with appropriate mitigation measures.

Amendment 53

Proposal for a regulation

Article 5215 – paragraph 34 – subparagraph 1

Text proposed by the Commission

Amendment

Users of an AI system that generates or manipulates image, audio or video content that appreciably resembles existing persons, objects, places or other entities or events and would falsely appear to a person to be authentic or truthful (‘deep fake’), shall disclose that the content has been artificially generated or manipulated.

High-risk AI systems shall be resilient as regards attempts by unauthorised third parties to alter their use or performance by exploiting the system vulnerabilities.

Users of an AI system that generates or manipulates image, audio, text, scripts or video content that appreciably resembles existing persons, objects, places, text, scripts or other entities or events and would falsely appear to a person to be authentic or truthful (‘deep fake’), shall disclose, in an appropriate clear, repetitive, timely and visible manner, that the content has been artificially generated or manipulated.

High-risk AI systems shall be resilient as regards to attempts by unauthorised third parties to alter their use, behaviour, outputs or performance by exploiting the system vulnerabilities.

Amendment 54

Proposal for a regulation

Article 5215 – paragraph 34 – subparagraph 23

Text proposed by the Commission

Amendment

However, the first subparagraph shall not apply where the use is authorised by law to detect, prevent, investigate and prosecute criminal offences or it is necessary for the exercise of the right to freedom of expression and the right to freedom of the arts and sciences guaranteed in the Charter of Fundamental Rights of the EU, and subject to appropriate safeguards for the rights and freedoms of third parties.

The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent and control for attacks trying to manipulate the training dataset (‘data poisoning’), inputs designed to cause the model to make a mistake (‘adversarial examples’), or model flaws.

However, the first subparagraph shall not apply where the use forms part of an evidently artistic, creative or fictional cinematographic or analogous work or where it is necessary for the exercise of the right to freedom of expression and the right to freedom of the arts and sciences guaranteed in the Charter of Fundamental Rights of the EU, and subject to appropriate safeguards for the rights and freedoms of third parties.

The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent, detect, respond to, resolve and control for attacks trying to manipulate the training dataset (‘data poisoning’), or pre-trained components used in training (‘model poisoning’) , inputs designed to cause the model to make a mistake (‘adversarial examples’ or ‘model evasion’), confidentiality attacks or model flaws, which could lead to harmful decision-making.

Amendment 55

Proposal for a regulation

Article 5240 – paragraph 31 a (new)

Text proposed by the Commission

Amendment

3a. Providers and users of AI systems that recommend, disseminate and order news or creative and cultural content shall disclose, in an appropriate, easily accessible, clear and visible manner, the main parameters used for the moderation of content and personalized suggestions. That information shall include a disclaimer.

The Commission shall ensure that the process of developing harmonised standards takes into account risks to fundamental rights, environment and society as a whole.

Amendment 56

Proposal for a regulation

Article 5240 – paragraph 31 b (new)

Text proposed by the Commission

Amendment

3b. The information referred to in this Article shall be provided to the natural persons in a timely, clear and visible manner, at the latest at the time of the first interaction or exposure. Such information shall be made accessible when the exposed natural person is a person with disabilities, a child or where he or she belongs to a vulnerable group. It shall be complete, where possible, with intervention or flagging procedures for the exposed natural person, taking into account the generally acknowledged state of the art and relevant harmonised standards and common specifications.

The Commission shall ensure that the process of developing harmonised standards on AI systems is open to stakeholders, including SMEs in accordance with Articles 5 and 6 of Regulation (EU) No 1025/2012.

Amendment 57

Proposal for a regulation

Article 5640 – paragraph ~~2 – point a~~1 ac (new)

Text proposed by the Commission

Amendment

~~(aa) work towards promoting uptake of AI within the Union, especially amongst SMEs;~~

To this end the Commission shall direct funds in accordance with Article 17 of Regulation (EU) No 1025/2012 to facilitate their effective participation.

Amendment 58

Proposal for a regulation

Article 5740 – paragraph 1 d (new)

Text proposed by the Commission

Amendment

1. The Board shall be composed of the national supervisory authorities, who shall be represented by the head or equivalent high-level official of that authority, and the European Data Protection Supervisor. Other national authorities may be invited to the meetings, where the issues discussed are of relevance for them.

The Commission shall review the harmonised standards before their publication in the Official Journal and prepare a report outlining their adequacy with paragraphs 1a and 1b of this Article.

1. The Board shall be composed of the national supervisory authorities, who shall be represented by the head or equivalent high-level official of that authority, and the European Data Protection Supervisor. Other national or international authorities and relevant stakeholders, including from the private sector, shall be invited to the meetings, where the issues discussed are of relevance for them.

Amendment 59

Proposal for a regulation

Article 5741 – paragraph 41

Text proposed by the Commission

Amendment

4. The Board may invite external experts and observers to attend its meetings and may hold exchanges with interested third parties to inform its activities to an appropriate extent. To that end the Commission may facilitate exchanges between the Board and other Union bodies, offices, agencies and advisory groups.

1. Where harmonised standards referred to in Article 40 do not exist or where the Commission considers that the relevant harmonised standards are insufficient or that there is a need to address specific safety or fundamental right concerns, the Commission may, by means of implementing acts, adopt common specifications in respect of the requirements set out in Chapter 2 of this Title. Those implementing acts shall be adopted in accordance with the examination procedure referred to in Article 74(2).

4. The Board shall, where relevant, invite external experts and observers to attend its meetings and may hold exchanges with interested third parties to inform its activities to an appropriate extent. To that end the Commission may facilitate exchanges between the Board and other Union bodies, offices, agencies and advisory groups, including the High-Level Expert Group on AI.

1. Where harmonised standards referred to in Article 40 do not exist or where the Commission considers that the relevant harmonised standards are insufficient or that there is a need to address specific and pressing safety or fundamental right concern that cannot be sufficiently settled by development of harmonised standards, the Commission may, by means of implementing acts, adopt common specifications in respect of the requirements set out in Chapter 2 of this Title. Those implementing acts shall be adopted in accordance with the examination procedure referred to in Article 74(2).

Amendment 60

Proposal for a regulation

Article 6941 – paragraph 32

Text proposed by the Commission

Amendment

3. Codes of conduct may be drawn up by individual providers of AI systems or by organisations representing them or by both, including with the involvement of users and any interested stakeholders and their representative organisations. Codes of conduct may cover one or more AI systems taking into account the similarity of the intended purpose of the relevant systems.

2. The Commission, when preparing the common specifications referred to in paragraph 1, shall gather the views of relevant bodies or expert groups established under relevant sectorial Union law.

3. Codes of conduct may be drawn up by individual providers of AI systems or by organisations representing them or by both, including with the involvement of users and any interested stakeholders and their representative organisations, including in particular trade unions and consumers organisations. Codes of conduct may cover one or more AI systems taking into account the similarity of the intended purpose of the relevant systems.

2. The Commission, when preparing the common specifications referred to in paragraph 1, shall gather the views of developers and providers of High-risk AI systems as well as relevant stakeholders, such as SME's and start-ups, civil society and social partners or expert groups established under relevant sectorial Union law.

Amendment 61

Proposal for a regulation

Article 6942 – paragraph ~~3 b (new)~~1

Text proposed by the Commission

Amendment

~~3b. In order to comply with the obligations laid down in this Article, providers and users shall ensure a sufficient level of AI literacy in accordance with Article 4b.~~

1. Taking into account their intended purpose, high-risk AI systems that have been trained and tested on data concerning the specific geographical, behavioural and functional setting within which they are intended to be used shall be presumed to be in compliance with the requirement set out in Article 10(4).

1. Taking into account their intended purpose, high-risk AI systems that have been trained and tested on data concerning the specific geographical, behavioural, contextual and functional setting within which they are intended to be used shall be presumed to be in compliance with the requirement set out in Article 10(4).

Amendment 62

Proposal for a regulation

Article 7142 – paragraph 12

Text proposed by the Commission

Amendment

1. In compliance with the terms and conditions laid down in this Regulation, Member States shall lay down the rules on penalties, including administrative fines, applicable to infringements of this Regulation and shall take all measures necessary to ensure that they are properly and effectively implemented. The penalties provided for shall be effective, proportionate, and dissuasive. They shall take into particular account the interests of small-scale providers and start-up and their economic viability.

2. High-risk AI systems that have been certified or for which a statement of conformity has been issued under a cybersecurity scheme pursuant to Regulation (EU) 2019/881 of the European Parliament and of the Council63 and the references of which have been published in the Official Journal of the European Union shall be presumed to be in compliance with the cybersecurity requirements set out in Article 15 of this Regulation in so far as the cybersecurity certificate or statement of conformity or parts thereof cover those requirements.

1. In compliance with the terms and conditions laid down in this Regulation, Member States shall lay down the rules on penalties, including administrative fines, applicable to infringements of this Regulation and shall take all measures necessary to ensure that they are properly and effectively implemented. The penalties provided for shall be effective, proportionate, and dissuasive. They shall take into particular account the interests and market position of small-scale providers and start-up and their economic viability.

2. High-risk AI systems that have been certified or for which a statement of conformity has been issued under a cybersecurity scheme pursuant to Regulation (EU) 2019/881 of the European Parliament and of the Council63 and the references of which have been published in the Official Journal of the European Union shall be presumed to be in compliance with the cybersecurity requirements set out in Article 15 of this Regulation, where applicable, in so far as the cybersecurity certificate or statement of conformity or parts thereof cover those requirements.

__________________

63 Regulation (EU) 2019/881 of the European Parliament and of the Council of 17 April 2019 on ENISA (the European Union Agency for Cybersecurity) and on information and communications technology cybersecurity certification and repealing Regulation (EU) No 526/2013 (Cybersecurity Act) (OJ L 151, 7.6.2019, p. 1).

Amendment 63

Proposal for a regulation

Article 7343 – paragraph 31 – subparagraph 1 a (new)

Text proposed by the Commission

Amendment

3a. Before adopting a delegated act, the Commission shall consult with the relevant institutions and stakeholders in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making.

Should the provider already have established internal organisation and structures for existing conformity assessments or requirements under other existing rules, the provider may utilise those, or parts of those, existing compliance structures, so long as they also have the capacity and competence needed to fulfil the requirements for the product set out in this Regulation.

Amendment 64

Proposal for a regulation

~~Annex III – paragraph 1 – point 2 – point a~~

Article 43 – paragraph 5

Text proposed by the Commission

Amendment

~~(a) AI systems intended to be used as safety components in the management and operation of road traffic and the supply of water, gas, heating and electricity.~~

~~(a) AI systems intended to be used as safety components in the management and operation of road traffic and the supply of water, gas, heating, telecommunications, and electricity.~~

5. The Commission is empowered to adopt delegated acts in accordance with Article 73 for the purpose of updating Annexes VI and Annex VII in order to introduce elements of the conformity assessment procedures that become necessary in light of technical progress. The Commission shall consult the European Artificial Intelligence Board established in Article 56 as well as all relevant stakeholders.

Amendment 65

Proposal for a regulation

~~Annex III – paragraph 1 – point 3 – introductory part~~

Article 43 – paragraph 6

Text proposed by the Commission

Amendment

~~3. Education and vocational training:~~

6. The Commission is empowered to adopt delegated acts to amend paragraphs 1 and 2 in order to subject high-risk AI systems referred to in points 2 to 8 of Annex III to the conformity assessment procedure referred to in Annex VII or parts thereof. The Commission shall adopt such delegated acts taking into account the effectiveness of the conformity assessment procedure based on internal control referred to in Annex VI in preventing or minimizing the risks to health and safety and protection of fundamental rights posed by such systems as well as the availability of adequate capacities and resources among notified bodies.

~~3. Education and training:~~

Amendment 66

Proposal for a regulation

~~Annex~~Article ~~III~~44 – paragraph 1 ~~– point 3 – point a~~

Text proposed by the Commission

Amendment

~~(a) AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and vocational training institutions;~~

1. Certificates issued by notified bodies in accordance with Annex VII shall be drawn-up in an official Union language determined by the Member State in which the notified body is established or in an official Union language otherwise acceptable to the notified body.

~~(a) AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and training institutions;~~

1. Certificates issued by notified bodies in accordance with Annex VII shall be drawn-up in one or several official languages determined by the Member State in which the notified body is established or in one or several official languages otherwise acceptable to the notified body.

Amendment 67

Proposal for a regulation

~~Annex~~Article ~~III~~48 – paragraph 1 ~~– point 3 – point b~~

Text proposed by the Commission

Amendment

(b) AI systems intended to be used for the purpose of assessing students in educational and vocational training institutions and for assessing participants in tests commonly required for admission to educational institutions.

1. The provider shall draw up a written EU declaration of conformity for each AI system and keep it at the disposal of the national competent authorities for 10 years after the AI system has been placed on the market or put into service. The EU declaration of conformity shall identify the AI system for which it has been drawn up. A copy of the EU declaration of conformity shall be given to the relevant national competent authorities upon request.

(b) AI systems intended to be used for the purpose of assessing students in educational and training institutions and for assessing participants in tests commonly required for admission to educational institutions.

Amendment 68

Proposal for a regulation

~~Annex~~Article ~~III~~51 – paragraph 1 ~~– point 3 – point b~~ a (new)

Text proposed by the Commission

Amendment

~~(ba) AI systems intended to be used for the purpose of determining the study programme or areas of study to be followed by students in education and training institutions;~~

A high-risk AI system designed, developed, trained, validate, tested or approved to be placed on the market or put into service, outside the Union, can be registered in the EU database referred to in Article 60 and placed on the market or put into service in the Union only if it is proven that at all the stages of its design, development, training, validation, testing or approval, all the obligations required from such AI systems in the Union have been met.

Amendment 69

Proposal for a regulation

~~Annex~~Article ~~III~~51 – paragraph 1 ~~– point 3 – point~~ b b (new)

Text proposed by the Commission

Amendment

~~(bb) AI systems intended to be used for monitoring and detecting prohibited behaviour of students during tests at education and training institutions;~~

Before using a high-risk AI system referred to in Article 6(2) the user or where applicable the authorised representative shall register the uses of that system in the EU database referred to in the Article 60. A new registration entry shall be complemented by the user for each high risk use of the AI system.

Amendment 70

Proposal for a regulation

~~Annex~~Article ~~III~~53 – paragraph 1 ~~– point 4 – introductory part~~

Text proposed by the Commission

Amendment

~~4. Employment, workers management and access to self-employment:~~

1. AI regulatory sandboxes established by one or more Member States competent authorities or the European Data Protection Supervisor shall provide a controlled environment that facilitates the development, testing and validation of innovative AI systems for a limited time before their placement on the market or putting into service pursuant to a specific plan. This shall take place under the direct supervision and guidance by the competent authorities with a view to ensuring compliance with the requirements of this Regulation and, where relevant, other Union and Member States legislation supervised within the sandbox.

~~4. Employment and employment support, workers management and access to self-employment:~~

1. AI regulatory sandboxes established by one or more Member States competent authorities in collaboration with the Commission, or the European Data Protection Supervisor shall provide a controlled environment that facilitates the safe development, testing and validation of innovative AI systems for a limited time before their placement on the market or putting into service pursuant to a specific plan, SMEs, start-ups, enterprises, innovators or other relevant actors could be included as partners in the regulatory sandboxes. This shall take place under the direct supervision and guidance of the Commission in collaboration with the competent authorities with a view to identifying risks, in particular to health, safety, and fundamental rights, and ensuring compliance with the requirements of this Regulation and, where relevant, other Union and Member States legislation supervised within the sandbox. The Commission shall play a complementary role, allowing those Member States with demonstrated experience with sandboxing to build on their expertise and, on the other hand, assisting and providing technical understanding and resources to those Member States that seek guidance on the set-up and running of these regulatory sandboxes.

Amendment 71

Proposal for a regulation

~~Annex III – paragraph 1 – point 4 – point b~~

Article 53 – paragraph 2

Text proposed by the Commission

Amendment

(b) AI intended to be used for making decisions on promotion and termination of work-related contractual relationships, for task allocation and for monitoring and evaluating performance and behavior of persons in such relationships.

2. Member States shall ensure that to the extent the innovative AI systems involve the processing of personal data or otherwise fall under the supervisory remit of other national authorities or competent authorities providing or supporting access to data, the national data protection authorities and those other national authorities are associated to the operation of the AI regulatory sandbox.

(b) AI intended to be used for making decisions on establishment, promotion and termination of work-related relationships, for task allocation, for monitoring compliance with workplace rules and for monitoring and evaluating performance and behavior of persons in such relationships.

2. Member States, in collaboration with the Commission, shall ensure that to the extent the innovative AI systems involve the processing of personal data or otherwise fall under the supervisory remit of other national authorities or competent authorities providing or supporting access to data, the national data protection authorities and those other national authorities are associated to the operation of the AI regulatory sandbox.

~~ANNEX: LIST OF ENTITIES OR PERSONS FROM WHOM THE RAPPORTEUR HAS RECEIVED INPUT~~

Amendment 72

The following list is drawn up on a purely voluntary basis under the exclusive responsibility of the rapporteur. The rapporteur has received input from the following entities or persons in the preparation of the opinion, until the adoption thereof in committee:

Proposal for a regulation

~~Entity and/or person~~

Article 53 – paragraph 3

~~AccessNow~~

Text proposed by the Commission

~~Baptiste Caramiaux, CNRS researcher, Sorbonne Université in Paris~~

Amendment

~~BEUC~~

3. The AI regulatory sandboxes shall not affect the supervisory and corrective powers of the competent authorities. Any significant risks to health and safety and fundamental rights identified during the development and testing of such systems shall result in immediate mitigation and, failing that, in the suspension of the development and testing process until such mitigation takes place.

~~Defenddigitalme~~

3. The AI regulatory sandboxes shall not affect the supervisory and corrective powers of the competent authorities, including at regional or local level. Any significant risks to health and safety, fundamental rights, democracy or the environment, identified during the development and testing of AI systems, shall result in immediate mitigation and, failing that, in the suspension of the development and testing process until such mitigation takes place.

~~EDRi~~

Amendment 73

~~European Disability Forum~~

Proposal for a regulation

~~Huawei~~

Article 53 – paragraph 5

~~International Federation of Library Associations and Institutions~~

Text proposed by the Commission

~~Joanna Bryson, Professor of Ethics and Technology, Hertie School~~

Amendment

~~Liber Europe~~

5. Member States’ competent authorities that have established AI regulatory sandboxes shall coordinate their activities and cooperate within the framework of the European Artificial Intelligence Board. They shall submit annual reports to the Board and the Commission on the results from the implementation of those scheme, including good practices, lessons learnt and recommendations on their setup and, where relevant, on the application of this Regulation and other Union legislation supervised within the sandbox.

~~Maldita.es~~

5. Member States’ competent authorities and the Commission shall coordinate their activities with regard to AI regulatory sandboxes and cooperate within the framework of the European Artificial Intelligence Board. The Commission shall submit annual reports to the European Artificial Intelligence Board on the results from the implementation of those schemes, including best practices, computational energy use and efficiency, lessons learnt and recommendations on their setup and, where relevant, on the application of this Regulation and other Union legislation supervised within the sandbox. SMEs, start-ups, enterprises and other innovators shall be invited to share their good practices, lessons learnt and recommendations on their AI sandboxes with Member States’ competent authorities.

~~Martin Senftlebe, Professor of Intellectual Property Law, University of Amsterdam~~

Amendment 74

~~Meaning Processing Ltd.~~

Proposal for a regulation

~~Saidot Ltd.~~

Article 53 – paragraph 6 a (new)

~~Scio.cz~~

Text proposed by the Commission

Amendment

6a. The Commission shall establish an EU AI Regulatory Sandboxing Work Programme whose modalities referred to in Article 53(6) shall cover the elements set out in Annex IXa. The Commission shall proactively coordinate with national and local authorities, where relevant.

Amendment 75

Proposal for a regulation

Article 55 – title

Text proposed by the Commission

Amendment

Measures for small-scale providers and users

Measures for SMEs, start-ups and users

Amendment 76

Proposal for a regulation

Article 55 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) provide small-scale providers and start-ups with priority access to the AI regulatory sandboxes to the extent that they fulfil the eligibility conditions;

(a) provide SMEs and start-ups, established in the Union, with priority access to the AI regulatory sandboxes, to the extent that they fulfil the eligibility conditions;

Amendment 77

Proposal for a regulation

Article 55 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) organise specific awareness raising activities about the application of this Regulation tailored to the needs of the small-scale providers and users;

(b) organise specific awareness raising and enhanced digital skills development activities on the application of this Regulation tailored to the needs of SMEs, start-ups and users;

Amendment 78

Proposal for a regulation

Article 55 – paragraph 1 – point c

Text proposed by the Commission

Amendment

(c) where appropriate, establish a dedicated channel for communication with small-scale providers and user and other innovators to provide guidance and respond to queries about the implementation of this Regulation.

(c) utilise existing dedicated channels and where appropriate, establish new dedicated channels for communication with SMEs, start-ups, users and other innovators to provide guidance and respond to queries about the implementation of this Regulation;

Amendment 79

Proposal for a regulation

Article 55 – paragraph 1 – point c a (new)

Text proposed by the Commission

Amendment

(ca) foster the participation of SMEs and other relevant stakeholders in the standardisation development process.

Amendment 80

Proposal for a regulation

Article 55 – paragraph 2

Text proposed by the Commission

Amendment

2. The specific interests and needs of the small-scale providers shall be taken into account when setting the fees for conformity assessment under Article 43, reducing those fees proportionately to their size and market size.

2. The specific interests and needs of the SMEs, start-ups and users shall be taken into account when setting the fees for conformity assessment under Article 43, reducing those fees proportionately to development stage, their size, market size and market demand. The Commission shall regularly assess the certification and compliance costs for SMEs and start-ups, including through transparent consultations with SMEs, start-ups and users and shall work with Member States to lower such costs where possible. The Commission shall report on these findings to the European Parliament and to the Council as part of the report on the evaluation and review of this Regulation provided for in Article 84(2).

Amendment 81

Proposal for a regulation

Article 57 – paragraph 3 a (new)

Text proposed by the Commission

Amendment

3a. The Board shall establish an AI Advisory Council (Advisory Council). The Advisory Council shall be composed of relevant representatives from industry, research, academia, civil society, standardisation organisations, and other relevant stakeholders or third parties appointed by the Board, representing all Member States to maintain geographical balance. The Advisory Council shall support the work of the Board by providing advice relating to the tasks of the Board. The Advisory Council shall nominate a relevant representative, depending on the configuration in which the Board meets, to attend meetings of the Board and to participate in its work. The composition of the Advisory Council and its recommendations to the Board shall be made public.

Amendment 82

Proposal for a regulation

Annex I – point c

Text proposed by the Commission

Amendment

Amendment 83

Proposal for a regulation

Annex IX a (new)

Text proposed by the Commission

Amendment

ANNEX IXa

Modalities for an EU AI regulatory sandboxing work programme

1. The AI Regulatory Sandboxes shall be part of the EU AI Regulatory Sandboxing Work Programme (‘sandboxing programme’) to be established by the Commission in collaboration with Member States.

2. The Commission shall play a complementary role, allowing those Member States with demonstrated experience with sandboxing to build on their expertise and the expertise of relevant stakeholders from industry, SMEs, academia and civil society and, on the other hand, assisting and providing technical understanding and resources to those Member States that seek guidance on the set-up of these regulatory sandboxes.

3. The criteria for the access to the regulatory sandbox shall be transparent and competitive.

4. Participants in the sandboxing programme, in particular small-scale providers, are granted access to pre-deployment services, such as preliminary registration of their AI system, compliance R&D support services, and to all the other relevant elements of the Union’s AI ecosystem and other Digital Single Market initiatives such as Testing & Experimentation Facilities, Digital Hubs, Centres of Excellence, and EU benchmarking capabilities; and to other value-adding services such as standardisation documents and certification, consultation and support to conduct impact assessments of the AI systems to fundamental rights, environment or the society at large, an online social platform for the community, contact databases, existing portal for tenders and grant making and lists of EU investors.

5. The sandboxing programme shall, in a later development phase, aim at assisting Member States in developing and managing two types of regulatory sandboxes: Physical Regulatory Sandboxes for AI systems embedded in physical products or services and Cyber Regulatory Sandboxes for AI systems operated and used on a stand-alone basis, not embedded in physical products or services.

6. The sandboxing programme shall work with the already established Digital Innovation Hubs in Member States to provide a dedicated point of contact for entrepreneurs to raise enquiries with competent authorities and to seek non-binding guidance on the conformity of innovative products, services or business models embedding AI technologies.

7. One of the objectives of the sandboxing programme is to enable firms’ compliance with this Regulation at the design stage of the AI system (‘compliance-by-design’). To do so, the programme shall facilitate the development of software tools and infrastructure for testing, benchmarking, assessing and explaining dimensions of AI systems relevant to sandboxes, such as accuracy, robustness and cybersecurity as well as minimisation of risks to fundamental rights, environment and the society at large.

8. The sandboxing programme shall be rolled out in a phased fashion, with the various phases launched by the Commission upon success of the previous phase.

9. The sandboxing programme will have a built-in impact assessment procedure to facilitate the review of cost-effectiveness against the agreed-upon objectives. This assessment shall be drafted with input from Member States based on their experiences and shall be included as part of the Annual Report submitted by the Commission to the European Artificial Intelligence Board.

PROCEDURE – COMMITTEE ASKED FOR OPINION

Date announced in plenary

~~CULT~~

ITRE

7.6.2021

Date appointed

~~Marcel Kolaja~~

Eva Maydell

~~13.7.2021~~

11.1.2022

Rule 58 – Joint committee procedure

Discussed in committee

~~15.3.2022~~

21.3.2022

~~Date adopted~~

~~15.6.2022~~

~~Result of final vote~~

–:

Members present for the final vote

Asim Ademov, Ilana Cicurel, Tomasz Frankowski, Romeo Franz, Catherine Griset, Sylvie Guillaume, Hannes Heide, Petra Kammerevert, Niyazi Kizilyürek, Predrag Fred Matić, Dace Melbārde, Peter Pollák, Diana Riba i Giner, Andrey Slabakov, Massimiliano Smeriglio, Michaela Šojdrová, Sabine Verheyen, Theodoros Zagorakis, Milan Zver

François-Xavier Bellamy, Hildegard Bentele, Tom Berendsen, Vasile Blaga, Michael Bloss, Marc Botenga, Cristian-Silviu Buşoi, Jerzy Buzek, Maria da Graça Carvalho, Ignazio Corrao, Ciarán Cuffe, Nicola Danti, Pilar del Castillo Vera, Martina Dlabajová, Christian Ehler, Niels Fuglsang, Lina Gálvez Muñoz, Jens Geier, Nicolás González Casares, Christophe Grudler, Henrike Hahn, Robert Hajšel, Ivars Ijabs, Romana Jerković, Seán Kelly, Łukasz Kohut, Zdzisław Krasnodębski, Andrius Kubilius, Thierry Mariani, Marisa Matias, Eva Maydell, Iskra Mihaylova, Dan Nica, Angelika Niebler, Niklas Nienaß, Mauri Pekkarinen, Mikuláš Peksa, Tsvetelina Penkova, Morten Petersen, Pina Picierno, Clara Ponsatí Obiols, Manuela Ripa, Robert Roos, Sara Skyttedal, Maria Spyraki, Patrizia Toia, Pernille Weiss, Carlos Zorrinho

Substitutes present for the final vote

~~Loucas Fourlas, Ibán García Del Blanco, Marcel Kolaja, Domènec Ruiz Devesa~~

Franc Bogovič, Andrea Caroppo, Jakop G. Dalunde, Jens Gieseke, Klemen Grošelj, Elena Kountoura, Marian-Jean Marinescu, Dace Melbārde, Dominique Riquet, Rob Rooken, Susana Solís Pérez

Substitutes under Rule 209(7) present for the final vote

~~Karen Melchior~~

Alessandra Basso, Bas Eickhout, Carlo Fidanza, Rob Rooken

FINAL VOTE BY ROLL CALL IN COMMITTEE ASKED FOR OPINION