What changed

PA_Legam

SHORT JUSTIFICATION

On 24 April 2021, the European Commission published its legislative proposal laying down harmonised rules on artificial intelligence (AI Act), which introduces a regulatory framework with the objective of ensuring that AI systems placed on the European Union market are safe to use and respect fundamental rights and European Union values.

Furthermore, the proposal facilitates development of a single market for lawful, safe and trustworthy AI applications, enhances governance and effective enforcement of existing law on fundamental rights and safety requirements applicable to AI systems and ensures legal certainty to facilitate investment and innovation in AI.

Overall, the Rapporteur welcomes the European Commission’s proposal; however, would like to suggest a few amendments mainly to extend the list of high-risk AI applications in areas of education, media and culture under Annex III and to modify certain provisions related to banned practices under Article 5.

More specifically, the Rapporteur reflects on the increased deployment of AI technologies in education and training facilities. Therefore, he proposes listing, among high-risk technologies, also those AI technologies used for monitoring of students during tests and technologies used to determine an area or a programme a student should study. Regarding media and culture, the Rapporteur suggests listing high-risk AI technologies used to create or disseminate machine-generated news articles used by news media outlets and AI technologies used to recommend or rank audiovisual content.

In addition, the Rapporteur proposes to extend the ban on deployment of social scoring systems to usage by public and private entities given the inherent threat of discrimination and exclusion of certain groups or individuals.

Finally, in light of the danger that deployment of remote biometric identification systems in publicly accessible places poses to citizens’ fundamental rights, freedom of assembly, work of investigative journalists, activists and political representatives, the Rapporteur proposes to ban deployment of such technologies in publicly accessible places following up on the Parliament resolution of 6 October 2021 on “Artificial intelligence in criminal law and its use by the police and judicial authorities in criminal matters”.

AMENDMENTS

The Committee on LegalCulture Affairsand Education calls on the Committee on the Internal Market and Consumer Protection and the Committee on Civil Liberties, Justice and Home Affairs, as the committees responsible, to take into account the following amendments:

Amendment 1

Proposal for a regulation

Recital 51

Text proposed by the Commission

Amendment

(5) A Union legal framework laying down(1) harmonisedThe rulespurpose onof artificialthis intelligenceRegulation is therefore needed to fosterimprove the development, use and uptakefunctioning of artificial intelligence in the internal market that at the sameby timelaying meetsdown a high level of protectionuniform oflegal publicframework interests,in suchparticular asfor healththe anddevelopment, safetymarketing and the protectionuse of fundamental rights, asartificial recognisedintelligence andin protectedconformity bywith Union law. To achieve that objective, rules regulating the placing on thevalues. marketThis andRegulation puttingpursues intoa servicenumber of certainoverriding AIreasons systemsof shouldpublic beinterest, laidsuch down,as thusa ensuringhigh thelevel smoothof functioningprotection of the internalhealth, marketsafety and allowing thosefundamental systemsrights, toand benefitit fromensures the principle of free movement of AI-based goods and services. By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development ofservices secure,cross-border, trustworthythus andpreventing ethicalMember artificialStates intelligence,from asimposing statedrestrictions byon the European Council33development, ,marketing and it ensures the protectionuse of ethicalAI principles,systems, asunless specificallyexplicitly requestedauthorised by the European Parliament34this .Regulation.

(5) A Union legal framework laying down(1) harmonisedThe rulespurpose onof artificialthis intelligenceRegulation is therefore needed to fosterimprove the development, use and uptakefunctioning of artificial intelligence in the internal market that at the sameby timelaying meetsdown a high level of protectionuniform oflegal publicframework interests,in suchparticular asfor healththe anddevelopment, safetymarketing and the protectionuse of fundamental rights, asartificial recognisedintelligence andin protectedconformity bywith Union law. To achieve that objective, rules regulating the placing on thevalues. marketThis andRegulation puttingpursues intoa servicenumber of certain AI systems should be laid down, thus ensuring the smoothoverriding functioningreasons of the internal market and allowing those systems to benefitpublic frominterest, thesuch principleas ofa freehigh movementlevel of goods and services. Furthermore, clear rules supporting the developmentprotection of AI systems should be laid down, thus enabling a European ecosystem ofhealth, publicsafety and private actors creating AI Systems in line with thefundamental Europeanrights, values.as Whilewell providersas of AI systems should not be hindered, the placing on the marketsociety and putting into service of certain AI Systems requires strict rules, including the ban of certain practices of AI Systems. By laying down those rules,environment, thisand Regulationit supportsensures the objective of thefree Unionmovement of being a global leader inAI-based thegoods developmentand ofservices secure,cross-border, trustworthythus andpreventing ethicalMember artificialStates intelligence,from asimposing statedrestrictions byon the Europeandevelopment, Council33,marketing and it ensures the protectionuse of ethicalAI principles,systems, asunless specificallyexplicitly requestedauthorised by the Europeanthis Parliament34.Regulation.

__________________

__________________

33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6.

33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6.

34 European Parliament resolution of 20 October 2020 with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies, 2020/2012(INL).

34 European Parliament resolution of 20 October 2020 with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies, 2020/2012(INL).

Or. en

Proposal for a regulation

Recital 5 a (new)

Recital 4

Text proposed by the Commission

Amendment

(5a) To ensure the development of secure, trustworthy and ethical AI, the European Commission established the High-Level Expert Groupon Artificial Intelligence. In formulating both Ethics guidelines for Trustworthy AI and a corresponding Assessment List for Trustworthy Artificial Intelligence, this independent group solidified the foundational ambition for ‘Trustworthy AI’. As noted by the group, Trustworthiness is a prerequisite for people and societies to develop, deploy and use AI systems. Without AI Systems – and the human beings behind them – being demonstrably worthy of trust, serious and unwanted consequences may ensue and their uptake might be hindered, preventing the realisation of the potentially vast social and economic Benefits that trustworthy AI systems can bring. This approach should be seen as the basis of a European approach to ensure and scale AI that is both innovative and ethical.

(4) At the same time, depending on the circumstances regarding its specific application and use, artificial intelligence may generate risks and cause harm to public interests and rights that are protected by Union law. Such harm might be material or immaterial.

(4) At the same time, depending on the circumstances regarding its specific application and use, artificial intelligence may generate risks and cause harm to public interests and rights that are protected by Union law, whether individual, societal or environmental. Such harm might be material or immaterial.

Or. en

Proposal for a regulation

Recital 65

Text proposed by the Commission

Amendment

(6) The notion of AI system should be clearly defined to ensure legal certainty, while providing the flexibility to accommodate future technological developments. The definition should be based on the key functional characteristics of the software, in particular the ability, for a given set of human-defined objectives, to generate outputs such as content, predictions, recommendations, or decisions which influence the environment with which the system interacts, be it in a physical or digital dimension. AI systems can be designed to operate with varying levels of autonomy and be used on a stand-alone basis or as a component of a product, irrespective of whether the system is physically integrated into the product (embedded) or serve the functionality of the product without being integrated therein (non-embedded). The definition of AI system should be complemented by a list of specific techniques and approaches used for its development, which should be kept up-to–date in the light of market and technological developments through the adoption of delegated acts by the Commission to amend that list.

(5) A Union legal framework laying down harmonised rules on artificial intelligence is therefore needed to foster the development, use and uptake of artificial intelligence in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, as recognised and protected by Union law. To achieve that objective, rules regulating the placing on the market and putting into service of certain AI systems should be laid down, thus ensuring the smooth functioning of the internal market and allowing those systems to benefit from the principle of free movement of goods and services. By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development of secure, trustworthy and ethical artificial intelligence, as stated by the European Council33 , and it ensures the protection of ethical principles, as specifically requested by the European Parliament34 .

(6) The notion of AI system should be clearly defined to ensure legal certainty, while providing the flexibility to accommodate future technological developments. The definition should be in line with already existing generaly accepted AI definitions that have found international acceptance. Moreover, it should be based on the key functional characteristics of artificial intelligence distinguishing it from more classic software systems and modelling approaches such as logistic regression and other techniques that are similarly transparent, explainable and interpretable. For the purposes of this Regulation, the definition should be based on the key functional characteristics of the AI system, in particular its ability, for a given set of human-defined objectives, to make predictions, recommendations, or decisions that influence real or virtual environments, whereby it uses machine and/or human-based data and inputs to (i) perceive real and/or virtual environments; (ii) abstract these perceptions into models through analysis inan automated manner (e.g., with machine learning), or manually; and (iii) use model inference to formulate options for outcomes. The definition should include AI systems which are designed to operate with varying levels of autonomy and can be and to be designed to operate with varying levels of autonomy and be used on a stand-alone basis or as a component of a product, irrespective of whether the system is physically integrated into the product (embedded) or serve the functionality of the product without being integrated therein (non-embedded). Systems, which provide for automatic tasks solution, should not be considered to be AI system, unless the systems acts significant autonomously. The definition of AI system should be complemented by a list of specific techniques and approaches used for its development, which should be kept up-to–date in the light of market and technological developments through the adoption of delegated acts by the Commission to amend that list. Should the definition of ‘artificial intelligence system’ from international organisations such as the OECD be adjusted in the coming years, the European Commission should engage in dialogue with the OECD to ensure alignment between the two definitions. Should the AI Act still be undergoing legislative procedure, the co-legislators should consider these latest developments during the legislative process, so as to ensure alignment, legal clarity and broad international acceptance of the AI Act Definition of ‘AI systems’.

(5) A Union legal framework laying down harmonised rules on artificial intelligence is therefore needed to foster the development, use and uptake of artificial intelligence in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, as recognised and protected by Union law, as well as of society and the environment. To achieve that objective, rules regulating the placing on the market and putting into service of certain AI systems should be laid down, thus ensuring the smooth functioning of the internal market and allowing those systems to benefit from the principle of free movement of goods and services. By laying down those rules, this Regulation supports the objective of the Union of being a global leader in the development of secure, trustworthy and ethical artificial intelligence, as stated by the European Council33 , and it ensures the protection of ethical principles, as specifically requested by the European Parliament34 .

__________________

__________________

33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6.

33 European Council, Special meeting of the European Council (1 and 2 October 2020) – Conclusions, EUCO 13/20, 2020, p. 6.

34 European Parliament resolution of 20 October 2020 with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies, 2020/2012(INL).

34 European Parliament resolution of 20 October 2020 with recommendations to the Commission on a framework of ethical aspects of artificial intelligence, robotics and related technologies, 2020/2012(INL).

Or. en

Proposal for a regulation

Recital 6 a (new)

Recital 8

Text proposed by the Commission

Amendment

(6a) An AI systems of a mixed nature due to the combination or the convergence of listed high-risk purposes and non-listed low-risk purposes should fall completely under Title III of this Regulation, if the purposes are inextricably linked. The same rule should apply to minor AI components in a large software system, meaning that the AI component should be addressed separately unless it is inextricably linked with the software system.

(8) The notion of remote biometric identification system as used in this Regulation should be defined functionally, as an AI system intended for the identification of natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, and without prior knowledge whether the targeted person will be present and can be identified, irrespectively of the particular technology, processes or types of biometric data used. Considering their different characteristics and manners in which they are used, as well as the different risks involved, a distinction should be made between ‘real-time’ and ‘post’ remote biometric identification systems. In the case of ‘real-time’ systems, the capturing of the biometric data, the comparison and the identification occur all instantaneously, near-instantaneously or in any event without a significant delay. In this regard, there should be no scope for circumventing the rules of this Regulation on the ‘real-time’ use of the AI systems in question by providing for minor delays. ‘Real-time’ systems involve the use of ‘live’ or ‘near-‘live’ material, such as video footage, generated by a camera or other device with similar functionality. In the case of ‘post’ systems, in contrast, the biometric data have already been captured and the comparison and identification occur only after a significant delay. This involves material, such as pictures or video footage generated by closed circuit television cameras or private devices, which has been generated before the use of the system in respect of the natural persons concerned.

(8) The notion of remote biometric identification system as used in this Regulation should be defined functionally, as an AI system intended for the identification of natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, irrespectively of the particular technology, processes or types of biometric data used. Considering their different characteristics and manners in which they are used, as well as the different risks involved, a distinction should be made between ‘real-time’ and ‘post’ remote biometric identification systems. In the case of ‘real-time’ systems, the capturing of the biometric data, the comparison and the identification occur all instantaneously, near-instantaneously or in any event without a significant delay. In this regard, there should be no scope for circumventing the rules of this Regulation on the ‘real-time’ use of the AI systems in question by providing for minor delays. ‘Real-time’ systems involve the use of ‘live’ or ‘near-‘live’ material, such as video footage, generated by a camera or other device with similar functionality. In the case of ‘post’ systems, in contrast, the biometric data have already been captured and the comparison and identification occur only after a significant delay. This involves material, such as pictures or video footage generated by closed circuit television cameras or private devices, which has been generated before the use of the system in respect of the natural persons concerned.

Or. en

Proposal for a regulation

Recital 89

Text proposed by the Commission

Amendment

(8) The notion of remote biometric identification system as used in this Regulation should be defined functionally, as an AI system intended for the identification of natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, and without prior knowledge whether the targeted person will be present and can be identified, irrespectively of the particular technology, processes or types of biometric data used. Considering their different characteristics and manners in which they are used, as well as the different risks involved, a distinction should be made between ‘real-time’ and ‘post’ remote biometric identification systems. In the case of ‘real-time’ systems, the capturing of the biometric data, the comparison and the identification occur all instantaneously, near-instantaneously or in any event without a significant delay. In this regard, there should be no scope for circumventing the rules of this Regulation on the ‘real-time’ use of the AI systems in question by providing for minor delays. ‘Real-time’ systems involve the use of ‘live’ or ‘near-‘live’ material, such as video footage, generated by a camera or other device with similar functionality. In the case of ‘post’ systems, in contrast, the biometric data have already been captured and the comparison and identification occur only after a significant delay. This involves material, such as pictures or video footage generated by closed circuit television cameras or private devices, which has been generated before the use of the system in respect of the natural persons concerned.

(9) For the purposes of this Regulation the notion of publicly accessible space should be understood as referring to any physical place that is accessible to the public, irrespective of whether the place in question is privately or publicly owned. Therefore, the notion does not cover places that are private in nature and normally not freely accessible for third parties, including law enforcement authorities, unless those parties have been specifically invited or authorised, such as homes, private clubs, offices, warehouses and factories. Online spaces are not covered either, as they are not physical spaces. However, the mere fact that certain conditions for accessing a particular space may apply, such as admission tickets or age restrictions, does not mean that the space is not publicly accessible within the meaning of this Regulation. Consequently, in addition to public spaces such as streets, relevant parts of government buildings and most transport infrastructure, spaces such as cinemas, theatres, shops and shopping centres are normally also publicly accessible. Whether a given space is accessible to the public should however be determined on a case-by-case basis, having regard to the specificities of the individual situation at hand.

(8) The notion of remote biometric identification system as used in this Regulation should be defined functionally, as an AI system intended for the identification of natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, and without prior knowledge whether the targeted person will be present and can be identified, irrespectively of the particular technology, processes or types of biometric data used. Considering their different characteristics and manners in which they are used, as well as the different risks involved, a distinction should be made between ‘real-time’ and ‘post’ remote biometric identification systems. In the case of ‘real-time’ systems, the capturing of the biometric data, the comparison and the identification occur all instantaneously, near-instantaneously or in any event without a significant delay. In this regard, there should be no scope for circumventing the rules of this Regulation on the ‘real-time’ use of the AI systems in question by providing for minor delays. ‘Real-time’ systems involve the use of ‘live’ or ‘near-‘live’ material, such as video footage, generated by a camera or other device with similar functionality. In the case of ‘post’ systems, in contrast, the biometric data have already been captured and the comparison and identification occur only after a significant delay. This involves material, such as pictures or video footage generated by closed circuit television cameras or private devices, which has been generated before the use of the system in respect of the natural persons concerned. The notion of remote biometric identification system should not cover AI systems, which allow for an identification of a natural person, if theses AI Systems are under the control of the natural person, such as door locks, security systems etc.

(9) For the purposes of this Regulation the notion of publicly accessible space should be understood as referring to any physical or virtual place that is accessible to the public, irrespective of whether the place in question is privately or publicly owned. Therefore, the notion does not cover places that are private in nature and normally not freely accessible for third parties, including law enforcement authorities, unless those parties have been specifically invited or authorised, such as homes, private clubs, offices, warehouses and factories. The same principle should apply to virtual publicly accessible spaces. However, the mere fact that certain conditions for accessing a particular space may apply, such as admission tickets or age restrictions, does not mean that the space is not publicly accessible within the meaning of this Regulation. Consequently, in addition to public spaces such as streets, relevant parts of government buildings and most transport infrastructure, spaces such as cinemas, theatres, shops and shopping centres are normally also publicly accessible. Whether a given space is accessible to the public should however be determined on a case-by-case basis, having regard to the specificities of the individual situation at hand.

Or. en

Proposal for a regulation

Recital 1113

Text proposed by the Commission

Amendment

(11) In light of their digital nature, certain AI systems should fall within the scope of this Regulation even when they are neither placed on the market, nor put into service, nor used in the Union. This is the case for example of an operator established in the Union that contracts certain services to an operator established outside the Union in relation to an activity to be performed by an AI system that would qualify as high-risk and whose effects impact natural persons located in the Union. In those circumstances, the AI system used by the operator outside the Union could process data lawfully collected in and transferred from the Union, and provide to the contracting operator in the Union the output of that AI system resulting from that processing, without that AI system being placed on the market, put into service or used in the Union. To prevent the circumvention of this Regulation and to ensure an effective protection of natural persons located in the Union, this Regulation should also apply to providers and users of AI systems that are established in a third country, to the extent the output produced by those systems is used in the Union. Nonetheless, to take into account existing arrangements and special needs for cooperation with foreign partners with whom information and evidence is exchanged, this Regulation should not apply to public authorities of a third country and international organisations when acting in the framework of international agreements concluded at national or European level for law enforcement and judicial cooperation with the Union or with its Member States. Such agreements have been concluded bilaterally between Member States and third countries or between the European Union, Europol and other EU agencies and third countries and international organisations.

(13) In order to ensure a consistent and high level of protection of public interests as regards health, safety and fundamental rights, common normative standards for all high-risk AI systems should be established. Those standards should be consistent with the Charter of fundamental rights of the European Union (the Charter) and should be non-discriminatory and in line with the Union’s international trade commitments.

deleted

(13) In order to ensure a consistent and high level of protection of public interests as regards health, safety and fundamental rights, as well as of society and the environment, common normative standards for all high-risk AI systems should be established. Those standards should be consistent with the Charter of fundamental rights of the European Union (the Charter) and should be non-discriminatory and in line with the Union’s international trade commitments.

Or. en

Proposal for a regulation

Recital 1416

Text proposed by the Commission

Amendment

(14) In order to introduce a proportionate and effective set of binding rules for AI systems, a clearly defined risk-based approach should be followed. That approach should tailor the type and content of such rules to the intensity and scope of the risks that AI systems can generate. It is therefore necessary to prohibit certain artificial intelligence practices, to lay down requirements for high-risk AI systems and obligations for the relevant operators, and to lay down transparency obligations for certain AI systems.

(16) The placing on the market, putting into service or use of certain AI systems intended to distort human behaviour, whereby physical or psychological harms are likely to occur, should be forbidden. Such AI systems deploy subliminal components individuals cannot perceive or exploit vulnerabilities of children and people due to their age, physical or mental incapacities. They do so with the intention to materially distort the behaviour of a person and in a manner that causes or is likely to cause harm to that or another person. The intention may not be presumed if the distortion of human behaviour results from factors external to the AI system which are outside of the control of the provider or the user. Research for legitimate purposes in relation to such AI systems should not be stifled by the prohibition, if such research does not amount to use of the AI system in human-machine relations that exposes natural persons to harm and such research is carried out in accordance with recognised ethical standards for scientific research.

(14) In order to introduce a proportionate and effective set of binding rules for AI systems, a clearly defined risk-based approach should be followed. That approach should tailor the type and content of such rules to the intensity and scope of the risks that AI systems can generate. It is therefore necessary to prohibit certain artificial intelligence practices, to lay down requirements for high-risk AI systems and obligations for the relevant operators, and to lay down transparency obligations for certain AI systems. This approach should also take into consideration that the development of AI systems, compared to the placing on the market and putting into service of an AI system, only leads to lower risks as long as the AI System is only used in the development environment. Therefore, the risk based approach should allow for the development of AI systems, disregarding whether the developed AI systems will eventually be put on the market.

(16) The placing on the market, putting into service or use of certain AI systems intended to distort human behaviour, whereby material or non-material harm, including physical, psychological or economic harm is likely to occur, should be forbidden. Such AI systems deploy subliminal components individuals cannot perceive or exploit vulnerabilities of children and people due to their age, physical or mental capacity. They do so with the intention to materially distort the behaviour of a person and in a manner that causes or is likely to cause harm to that or another person. The intention may not be presumed if the distortion of human behaviour results from factors external to the AI system which are outside of the control of the provider or the user. Research for legitimate purposes in relation to such AI systems should not be stifled by the prohibition, if such research does not amount to use of the AI system in human-machine relations that exposes natural persons to harm and such research is carried out in accordance with recognised ethical standards for scientific research.

Or. en

Proposal for a regulation

Recital 3317

Text proposed by the Commission

Amendment

(33) Technical inaccuracies of AI systems intended for the remote biometric identification of natural persons can lead to biased results and entail discriminatory effects. This is particularly relevant when it comes to age, ethnicity, sex or disabilities. Therefore, ‘real-time’ and ‘post’ remote biometric identification systems should be classified as high-risk. In view of the risks that they pose, both types of remote biometric identification systems should be subject to specific requirements on logging capabilities and human oversight.

(17) AI systems providing social scoring of natural persons for general purpose by public authorities or on their behalf may lead to discriminatory outcomes and the exclusion of certain groups. They may violate the right to dignity and non-discrimination and the values of equality and justice. Such AI systems evaluate or classify the trustworthiness of natural persons based on their social behaviour in multiple contexts or known or predicted personal or personality characteristics. The social score obtained from such AI systems may lead to the detrimental or unfavourable treatment of natural persons or whole groups thereof in social contexts, which are unrelated to the context in which the data was originally generated or collected or to a detrimental treatment that is disproportionate or unjustified to the gravity of their social behaviour. Such AI systems should be therefore prohibited.

(33) Technical inaccuracies of AI systems intended for the remote biometric identification of natural persons can lead to biased results and entail discriminatory effects. This is particularly relevant when it comes to age, ethnicity, sex or disabilities. Therefore, ‘real-time’ and ‘post’ remote biometric identification systems should be classified as high-risk, except for the purpose of remote client on-boarding or authentication of a user through a device. In view of the risks that they pose, both types of remote biometric identification systems should be subject to specific requirements on logging capabilities and human oversight.

(17) AI systems providing social scoring of natural persons for general purpose may lead to discriminatory outcomes and the exclusion of certain groups. They may violate the right to dignity and non-discrimination and the values of equality and justice. Such AI systems evaluate or classify the trustworthiness of natural persons based on their social behaviour in multiple contexts or known or predicted personal or personality characteristics. Such AI systems should be therefore prohibited.

Or. en

Proposal for a regulation

Recital 3718

Text proposed by the Commission

Amendment

(37) Another area in which the use of AI systems deserves special consideration is the access to and enjoyment of certain essential private and public services and benefits necessary for people to fully participate in society or to improve one’s standard of living. In particular, AI systems used to evaluate the credit score or creditworthiness of natural persons should be classified as high-risk AI systems, since they determine those persons’ access to financial resources or essential services such as housing, electricity, and telecommunication services. AI systems used for this purpose may lead to discrimination of persons or groups and perpetuate historical patterns of discrimination, for example based on racial or ethnic origins, disabilities, age, sexual orientation, or create new forms of discriminatory impacts. Considering the very limited scale of the impact and the available alternatives on the market, it is appropriate to exempt AI systems for the purpose of creditworthiness assessment and credit scoring when put into service by small-scale providers for their own use. Natural persons applying for or receiving public assistance benefits and services from public authorities are typically dependent on those benefits and services and in a vulnerable position in relation to the responsible authorities. If AI systems are used for determining whether such benefits and services should be denied, reduced, revoked or reclaimed by authorities, they may have a significant impact on persons’ livelihood and may infringe their fundamental rights, such as the right to social protection, non-discrimination, human dignity or an effective remedy. Those systems should therefore be classified as high-risk. Nonetheless, this Regulation should not hamper the development and use of innovative approaches in the public administration, which would stand to benefit from a wider use of compliant and safe AI systems, provided that those systems do not entail a high risk to legal and natural persons. Finally, AI systems used to dispatch or establish priority in the dispatching of emergency first response services should also be classified as high-risk since they make decisions in very critical situations for the life and health of persons and their property.

(18) The use of AI systems for ‘real-time’ remote biometric identification of natural persons in publicly accessible spaces for the purpose of law enforcement is considered particularly intrusive in the rights and freedoms of the concerned persons, to the extent that it may affect the private life of a large part of the population, evoke a feeling of constant surveillance and indirectly dissuade the exercise of the freedom of assembly and other fundamental rights. In addition, the immediacy of the impact and the limited opportunities for further checks or corrections in relation to the use of such systems operating in ‘real-time’ carry heightened risks for the rights and freedoms of the persons that are concerned by law enforcement activities.

deleted

(18) The use of AI systems for ‘real-time’ and ‘post’ remote biometric identification of natural persons in publicly accessible spaces is considered particularly intrusive in the rights and freedoms of the concerned persons, to the extent that it may affect the private life of a large part of the population, evoke a feeling of constant surveillance and indirectly dissuade the exercise of the freedom of assembly and other fundamental rights. In addition, the immediacy of the impact and the limited opportunities for further checks or corrections in relation to the use of such systems operating in ‘real-time’ carry heightened risks for the rights and freedoms of the persons.

Or. en

Proposal for a regulation

Recital 4119

Text proposed by the Commission

Amendment

(41) The fact that an AI system is classified as high risk under this Regulation should not be interpreted as indicating that the use of the system is necessarily lawful under other acts of Union law or under national law compatible with Union law, such as on the protection of personal data, on the use of polygraphs and similar tools or other systems to detect the emotional state of natural persons. Any such use should continue to occur solely in accordance with the applicable requirements resulting from the Charter and from the applicable acts of secondary Union law and national law. This Regulation should not be understood as providing for the legal ground for processing of personal data, including special categories of personal data, where relevant.

(19) The use of those systems for the purpose of law enforcement should therefore be prohibited, except in three exhaustively listed and narrowly defined situations, where the use is strictly necessary to achieve a substantial public interest, the importance of which outweighs the risks. Those situations involve the search for potential victims of crime, including missing children; certain threats to the life or physical safety of natural persons or of a terrorist attack; and the detection, localisation, identification or prosecution of perpetrators or suspects of the criminal offences referred to in Council Framework Decision 2002/584/JHA38 if those criminal offences are punishable in the Member State concerned by a custodial sentence or a detention order for a maximum period of at least three years and as they are defined in the law of that Member State. Such threshold for the custodial sentence or detention order in accordance with national law contributes to ensure that the offence should be serious enough to potentially justify the use of ‘real-time’ remote biometric identification systems. Moreover, of the 32 criminal offences listed in the Council Framework Decision 2002/584/JHA, some are in practice likely to be more relevant than others, in that the recourse to ‘real-time’ remote biometric identification will foreseeably be necessary and proportionate to highly varying degrees for the practical pursuit of the detection, localisation, identification or prosecution of a perpetrator or suspect of the different criminal offences listed and having regard to the likely differences in the seriousness, probability and scale of the harm or possible negative consequences.

(41) The fact that an AI system is classified as high risk under this Regulation should not be interpreted as indicating that the use of the system is necessarily lawful under other acts of Union law or under national law compatible with Union law, such as on the protection of personal data, on the use of polygraphs and similar tools or other systems to detect the emotional state of natural persons. Any such use should continue to occur solely in accordance with the applicable requirements resulting from the Charter and from the applicable acts of secondary Union law and national law. As far as applicable, this Regulation may be understood as providing for the legal ground for processing of personal data, including special categories of personal data, where relevant.

(19) Given the detrimental effect of those systems to freedoms a liberties of impacted individuals, the use of those systems in publicly accessible spaces should be prohibited.

__________________

38 Council Framework Decision 2002/584/JHA of 13 June 2002 on the European arrest warrant and the surrender procedures between Member States (OJ L 190, 18.7.2002, p. 1).

Or. en

Proposal for a regulation

Recital 4420

Text proposed by the Commission

Amendment

(44) High data quality is essential for the performance of many AI systems, especially when techniques involving the training of models are used, with a view to ensure that the high-risk AI system performs as intended and safely and it does not become the source of discrimination prohibited by Union law. High quality training, validation and testing data sets require the implementation of appropriate data governance and management practices. Training, validation and testing data sets should be sufficiently relevant, representative and free of errors and complete in view of the intended purpose of the system. They should also have the appropriate statistical properties, including as regards the persons or groups of persons on which the high-risk AI system is intended to be used. In particular, training, validation and testing data sets should take into account, to the extent required in the light of their intended purpose, the features, characteristics or elements that are particular to the specific geographical, behavioural or functional setting or context within which the AI system is intended to be used. In order to protect the right of others from the discrimination that might result from the bias in AI systems, the providers should be able to process also special categories of personal data, as a matter of substantial public interest, in order to ensure the bias monitoring, detection and correction in relation to high-risk AI systems.

(20) In order to ensure that those systems are used in a responsible and proportionate manner, it is also important to establish that, in each of those three exhaustively listed and narrowly defined situations, certain elements should be taken into account, in particular as regards the nature of the situation giving rise to the request and the consequences of the use for the rights and freedoms of all persons concerned and the safeguards and conditions provided for with the use. In addition, the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement should be subject to appropriate limits in time and space, having regard in particular to the evidence or indications regarding the threats, the victims or perpetrator. The reference database of persons should be appropriate for each use case in each of the three situations mentioned above.

(44) High data quality is essential for the performance of many AI systems, especially when techniques involving the training of models are used, with a view to ensure that the high-risk AI system performs as intended and safely and it does not become the source of discrimination prohibited by Union law. The processing of personal data in order to achieve the aforementioned shall be a legitimate interest. High quality training, validation and testing data sets require the implementation of appropriate data governance and management practices. Training, validation and testing data sets should be sufficiently relevant, and representative in view of the intended purpose of the system. They should also have the appropriate statistical properties, including as regards the persons or groups of persons on which the high-risk AI system is intended to be used. If it is necessary for the aforementioned purpose to use existing sets of personal data originally collected and stored for a different purpose their use for the aforementioned purpose shall be deemed compatible with the original purpose as long as the personal data is not transferred to any third party. In particular, training, validation and testing data sets should take into account, to the extent required in the light of their intended purpose, the features, characteristics or elements that are particular to the specific geographical, behavioural or functional setting or context within which the AI system is intended to be used. In order to protect the right of others from the discrimination that might result from the bias in AI systems, the providers should be able to process also special categories of personal data, as a matter of substantial public interest, in order to ensure the bias monitoring, detection and correction in relation to high-risk AI systems.

deleted

Or. en

Proposal for a regulation

Recital 4621

Text proposed by the Commission

Amendment

(46) Having information on how high-risk AI systems have been developed and how they perform throughout their lifecycle is essential to verify compliance with the requirements under this Regulation. This requires keeping records and the availability of a technical documentation, containing information which is necessary to assess the compliance of the AI system with the relevant requirements. Such information should include the general characteristics, capabilities and limitations of the system, algorithms, data, training, testing and validation processes used as well as documentation on the relevant risk management system. The technical documentation should be kept up to date.

(21) Each use of a ‘real-time’ remote biometric identification system in publicly accessible spaces for the purpose of law enforcement should be subject to an express and specific authorisation by a judicial authority or by an independent administrative authority of a Member State. Such authorisation should in principle be obtained prior to the use, except in duly justified situations of urgency, that is, situations where the need to use the systems in question is such as to make it effectively and objectively impossible to obtain an authorisation before commencing the use. In such situations of urgency, the use should be restricted to the absolute minimum necessary and be subject to appropriate safeguards and conditions, as determined in national law and specified in the context of each individual urgent use case by the law enforcement authority itself. In addition, the law enforcement authority should in such situations seek to obtain an authorisation as soon as possible, whilst providing the reasons for not having been able to request it earlier.

(46) Having information on how high-risk AI systems have been developed and how they perform throughout their lifecycle is essential to verify compliance with the requirements under this Regulation. This requires keeping records and the availability of a technical documentation, containing information which is necessary to assess the compliance of the AI system with the relevant requirements. While preserving trade secrets, such information should include the general characteristics, capabilities and limitations of the system, algorithms, data, training, testing and validation processes used as well as documentation on the relevant risk management system. The technical documentation should be kept up to date.

deleted

Or. en

Proposal for a regulation

Recital 5122

Text proposed by the Commission

Amendment

(51) Cybersecurity plays a crucial role in ensuring that AI systems are resilient against attempts to alter their use, behaviour, performance or compromise their security properties by malicious third parties exploiting the system’s vulnerabilities. Cyberattacks against AI systems can leverage AI specific assets, such as training data sets (e.g. data poisoning) or trained models (e.g. adversarial attacks), or exploit vulnerabilities in the AI system’s digital assets or the underlying ICT infrastructure. To ensure a level of cybersecurity appropriate to the risks, suitable measures should therefore be taken by the providers of high-risk AI systems, also taking into account as appropriate the underlying ICT infrastructure.

(22) Furthermore, it is appropriate to provide, within the exhaustive framework set by this Regulation that such use in the territory of a Member State in accordance with this Regulation should only be possible where and in as far as the Member State in question has decided to expressly provide for the possibility to authorise such use in its detailed rules of national law. Consequently, Member States remain free under this Regulation not to provide for such a possibility at all or to only provide for such a possibility in respect of some of the objectives capable of justifying authorised use identified in this Regulation.

(51) Cybersecurity plays a crucial role in ensuring that AI systems are resilient against attempts to alter their use, behaviour, performance or compromise their security properties by malicious third parties exploiting the system’s vulnerabilities. Cyberattacks against AI systems can leverage AI specific assets, such as training data sets (e.g. data poisoning) or trained models (e.g. adversarial attacks), or exploit vulnerabilities in the AI system’s digital assets or the underlying ICT infrastructure. To ensure a level of cybersecurity appropriate to the risks, suitable measures should therefore be taken by the providers of high-risk AI systems but also by national competent authorities and notified bodies, also taking into account as appropriate the underlying ICT infrastructure.

deleted

Or. en

Proposal for a regulation

Recital 6223

Text proposed by the Commission

Amendment

(62) In order to ensure a high level of trustworthiness of high-risk AI systems, those systems should be subject to a conformity assessment prior to their placing on the market or putting into service.

(23) The use of AI systems for ‘real-time’ remote biometric identification of natural persons in publicly accessible spaces for the purpose of law enforcement necessarily involves the processing of biometric data. The rules of this Regulation that prohibit, subject to certain exceptions, such use, which are based on Article 16 TFEU, should apply as lex specialis in respect of the rules on the processing of biometric data contained in Article 10 of Directive (EU) 2016/680, thus regulating such use and the processing of biometric data involved in an exhaustive manner. Therefore, such use and processing should only be possible in as far as it is compatible with the framework set by this Regulation, without there being scope, outside that framework, for the competent authorities, where they act for purpose of law enforcement, to use such systems and process such data in connection thereto on the grounds listed in Article 10 of Directive (EU) 2016/680. In this context, this Regulation is not intended to provide the legal basis for the processing of personal data under Article 8 of Directive 2016/680. However, the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for purposes other than law enforcement, including by competent authorities, should not be covered by the specific framework regarding such use for the purpose of law enforcement set by this Regulation. Such use for purposes other than law enforcement should therefore not be subject to the requirement of an authorisation under this Regulation and the applicable detailed rules of national law that may give effect to it.

(62) In order to ensure a high level of trustworthiness of high-risk AI systems, those systems should be subject to a conformity assessment prior to their placing on the market or putting into service. AI systems, including general purpose AI systems, are frequently used as components of other AI or non-AI software systems. In order to increase trust in the value chain and to give certainty to businesses about the performance of their systems, providers may voluntarily apply for a third-party conformity assessments.

(23) The use of AI systems for ‘real-time’ and ‘post’ remote biometric identification of natural persons in publicly accessible spaces necessarily involves the processing of biometric data. The rules of this Regulation that prohibit such use, which are based on Article 16 TFEU, should apply as lex specialis in respect of the rules on the processing of biometric data contained in Article 10 of Directive (EU) 2016/680, thus regulating such use and the processing of biometric data involved in an exhaustive manner.

Or. en

Proposal for a regulation

Recital 6524

Text proposed by the Commission

Amendment

(65)(24) InAny orderprocessing toof carrybiometric outdata third-partyand conformityother assessmentpersonal fordata involved in the use of AI systems intendedfor tobiometric beidentification, usedother forthan in connection to the use of ‘real-time’ remote biometric identification ofsystems persons,in notifiedpublicly bodiesaccessible shouldspaces befor designatedthe underpurpose thisof Regulationlaw enforcement as regulated by thethis nationalRegulation, competentincluding authorities,where providedthose theysystems are compliantused by competent authorities in publicly accessible spaces for other purposes than law enforcement, should continue to comply with aall setrequirements resulting from Article 9(1) of requirements,Regulation notably(EU) on2016/679, independence,Article competence10(1) of Regulation (EU) 2018/1725 and absenceArticle 10 of conflictsDirective of(EU) interests.2016/680, as applicable.

(65) In order to carry out third-party conformity assessment for AI systems intended to be used for the remote biometric identification of persons, notified bodies should be designated under this Regulation by the national competent authorities, provided they are compliant with a set of requirements, notably on independence, competence, absence of conflicts of interests, and minimum cybersecurity requirements.

deleted

Or. en

Proposal for a regulation

Recital 6627

Text proposed by the Commission

Amendment

(66) In line with the commonly established notion of substantial modification for products regulated by Union harmonisation legislation, it is appropriate that an AI system undergoes a new conformity assessment whenever a change occurs which may affect the compliance of the system with this Regulation or when the intended purpose of the system changes. In addition, as regards AI systems which continue to ‘learn’ after being placed on the market or put into service (i.e. they automatically adapt how functions are carried out), it is necessary to provide rules establishing that changes to the algorithm and its performance that have been pre-determined by the provider and assessed at the moment of the conformity assessment should not constitute a substantial modification.

(27) High-risk AI systems should only be placed on the Union market or put into service if they comply with certain mandatory requirements. Those requirements should ensure that high-risk AI systems available in the Union or whose output is otherwise used in the Union do not pose unacceptable risks to important Union public interests as recognised and protected by Union law. AI systems identified as high-risk should be limited to those that have a significant harmful impact on the health, safety and fundamental rights of persons in the Union and such limitation minimises any potential restriction to international trade, if any.

(66) In line with the commonly established notion of substantial modification for products regulated by Union harmonisation legislation, it is appropriate that an AI system undergoes a new conformity assessment whenever a change occurs which may affect the compliance of the system with this Regulation or when the intended purpose of the system changes. If such a case materialises, the provider should follow a clear procedure with fixed deadlines, transparency requirements and reporting duties. In addition, as regards AI systems which continue to ‘learn’ after being placed on the market or put into service (i.e. they automatically adapt how functions are carried out), it is necessary to provide rules establishing that changes to the algorithm and its performance that have been planned or considered at the moment of the conformity assessment should not constitute a substantial modification. In addition, it should not be considered as substantial modification if the user trains an AI system. In this situation, the user should clearly delimit the effects that the learning can have for the AI system.

(27) High-risk AI systems should only be placed on the Union market or put into service if they comply with certain mandatory requirements. Those requirements should ensure that high-risk AI systems available in the Union or whose output is otherwise used in the Union do not pose unacceptable risks to important Union public interests as recognised and protected by Union law. AI systems identified as high-risk should be limited to those that have a significant harmful impact on the health, safety and fundamental rights of persons in the Union, as well as on society and on the environment, and such limitation minimises any potential restriction to international trade, if any.

Or. en

Proposal for a regulation

Recital 8433

Text proposed by the Commission

Amendment

(84) Member States should take all necessary measures to ensure that the provisions of this Regulation are implemented, including by laying down effective, proportionate and dissuasive penalties for their infringement. For certain specific infringements, Member States should take into account the margins and criteria set out in this Regulation. The European Data Protection Supervisor should have the power to impose fines on Union institutions, agencies and bodies falling within the scope of this Regulation.

(33) Technical inaccuracies of AI systems intended for the remote biometric identification of natural persons can lead to biased results and entail discriminatory effects. This is particularly relevant when it comes to age, ethnicity, sex or disabilities. Therefore, ‘real-time’ and ‘post’ remote biometric identification systems should be classified as high-risk. In view of the risks that they pose, both types of remote biometric identification systems should be subject to specific requirements on logging capabilities and human oversight.

(84) Member States should take all necessary measures to ensure that the provisions of this Regulation are implemented, including by laying down effective, proportionate and dissuasive penalties for their infringement. For certain specific infringements, Member States should take into account the margins and criteria set out in this Regulation. The European Data Protection Supervisor should have the power to impose fines on Union institutions, agencies and bodies falling within the scope of this Regulation. As these fines are proportional to turnover, there is a risk that the provider of an AI system requests a user of that AI system, with smaller turnover, to take on the role of provider in exchange for the initial provider to reimburse the penalties the AI system could face, reducing the penalty the initial provider would otherwise face. To prevent such abuse, the penalties and litigation costs under this Regulation should not be subject to contractual clauses or other arrangements.

(33) Technical inaccuracies of AI systems intended for the biometric identification of natural persons can lead to biased results and entail discriminatory effects. This is particularly relevant when it comes to age, ethnicity, sex or disabilities. Therefore, ‘real-time’ and ‘post’ remote biometric identification systems, as well as non-remote biometric identification systems intended to be used in publicly accessible spaces, workplaces and education and training institutions should be classified as high-risk. In view of the risks that they pose, both types of biometric identification systems should be subject to specific requirements on logging capabilities and human oversight.

Or. en

Proposal for a regulation

Article 1 – paragraph 1 – point a

Recital 35

Text proposed by the Commission

Amendment

(a) harmonised rules for the placing on the market, the putting into service and the use of artificial intelligence systems (‘AI systems’) in the Union;

(35) AI systems used in education or vocational training, notably for determining access or assigning persons to educational and vocational training institutions or to evaluate persons on tests as part of or as a precondition for their education should be considered high-risk, since they may determine the educational and professional course of a person’s life and therefore affect their ability to secure their livelihood. When improperly designed and used, such systems may violate the right to education and training as well as the right not to be discriminated against and perpetuate historical patterns of discrimination.

(a) harmonised rules for the placing on the market, the putting into service and the use of trustworthy artificial intelligence systems (‘AI systems’) in the Union;

(35) AI systems used in education or training, notably for determining access or assigning persons to educational and training institutions to evaluate persons on tests as part of or as a precondition for their education or for determining the areas of study a student should follow should be considered high-risk, since they may determine the educational and professional course of a person’s life and therefore affect their ability to secure their livelihood. When improperly designed and used, such systems may violate the right to education and training as well as the right not to be discriminated against and perpetuate historical patterns of discrimination. AI systems used to monitor students during tests at education and training institutions should be considered high-risk, since they are also interfering with students’ rights to privacy and data protection.

Or. en

Proposal for a regulation

Article 1 – paragraph 1 – point e

Recital 35 a (new)

Text proposed by the Commission

Amendment

(e) rules on market monitoring and surveillance.

(35a) AI systems used in media and culture, in particular those that create and disseminate machine-generated news articles and those that suggest or prioritize audiovisual content should be considered high-risk, since those systems may influence society, spread disinformation and misinformation, have a negative impact on elections and other democratic processes and impact cultural and linguistic diversity.

(e) rules on market monitoring, market surveillance and governance;

Or. en

Proposal for a regulation

Article 1 – paragraph 1 – point e a (new)

Recital 36

Text proposed by the Commission

Amendment

(ea) measures in support of innovation with a particular focus on SMEs and start-ups.

(36) AI systems used in employment, workers management and access to self-employment, notably for the recruitment and selection of persons, for making decisions on promotion and termination and for task allocation, monitoring or evaluation of persons in work-related contractual relationships, should also be classified as high-risk, since those systems may appreciably impact future career prospects and livelihoods of these persons. Relevant work-related contractual relationships should involve employees and persons providing services through platforms as referred to in the Commission Work Programme 2021. Such persons should in principle not be considered users within the meaning of this Regulation. Throughout the recruitment process and in the evaluation, promotion, or retention of persons in work-related contractual relationships, such systems may perpetuate historical patterns of discrimination, for example against women, certain age groups, persons with disabilities, or persons of certain racial or ethnic origins or sexual orientation. AI systems used to monitor the performance and behaviour of these persons may also impact their rights to data protection and privacy.

(36) AI systems used in employment, workers management and access to self-employment, notably for the recruitment and selection of persons, for making decisions on promotion and termination and for task allocation, for monitoring compliance with workplace rules and for monitoring or evaluation of persons in work-related relationships, should also be classified as high-risk, since those systems may appreciably impact future career prospects and livelihoods of these persons. Relevant work-related relationships should involve employees and persons providing services through platforms as referred to in the Commission Work Programme 2021. Such persons should in principle not be considered users within the meaning of this Regulation. Throughout the recruitment process and in the evaluation, promotion, or retention of persons in work-related relationships, such systems may perpetuate historical patterns of discrimination, for example against women, certain age groups, persons with disabilities, or persons of certain racial or ethnic origins or sexual orientation. AI systems used to monitor the performance and behaviour of these persons may also impact their rights to data protection and privacy.

Or. en

Proposal for a regulation

Article 2 – paragraph 1 – point b

Recital 85

Text proposed by the Commission

Amendment

(b) users of AI systems located within the Union;

(85) In order to ensure that the regulatory framework can be adapted where necessary, the power to adopt acts in accordance with Article 290 TFEU should be delegated to the Commission to amend the techniques and approaches referred to in Annex I to define AI systems, the Union harmonisation legislation listed in Annex II, the high-risk AI systems listed in Annex III, the provisions regarding technical documentation listed in Annex IV, the content of the EU declaration of conformity in Annex V, the provisions regarding the conformity assessment procedures in Annex VI and VII and the provisions establishing the high-risk AI systems to which the conformity assessment procedure based on assessment of the quality management system and assessment of the technical documentation should apply. It is of particular importance that the Commission carry out appropriate consultations during its preparatory work, including at expert level, and that those consultations be conducted in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making58 . In particular, to ensure equal participation in the preparation of delegated acts, the European Parliament and the Council receive all documents at the same time as Member States’ experts, and their experts systematically have access to meetings of Commission expert groups dealing with the preparation of delegated acts.

(b) users of AI systems who are physically present or established within the Union;

(85) In order to ensure that the regulatory framework can be adapted where necessary, the power to adopt acts in accordance with Article 290 TFEU should be delegated to the Commission to amend the techniques and approaches referred to in Annex I to define AI systems, the Union harmonisation legislation listed in Annex II, the high-risk AI systems listed in Annex III, the provisions regarding technical documentation listed in Annex IV, the content of the EU declaration of conformity in Annex V, the provisions regarding the conformity assessment procedures in Annex VI and VII and the provisions establishing the high-risk AI systems to which the conformity assessment procedure based on assessment of the quality management system and assessment of the technical documentation should apply. It is of particular importance that the Commission carry out appropriate consultations during its preparatory work, including at expert level, and that those consultations be conducted in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making58. Those consultations should involve the participation of a balanced selection of stakeholders, including consumer organisations, associations representing the persons concerned, representatives of businesses from different sectors and of different sizes, as well as researchers and scientists. In particular, to ensure equal participation in the preparation of delegated acts, the European Parliament and the Council receive all documents at the same time as Member States’ experts, and their experts systematically have access to meetings of Commission expert groups dealing with the preparation of delegated acts.

__________________

__________________

58 OJ L 123, 12.5.2016, p. 1.

58 OJ L 123, 12.5.2016, p. 1.

Or. en

Proposal for a regulation

Article 2 – paragraph 1 – point c

Recital 86 a (new)

Text proposed by the Commission

Amendment

(c) providers and users of AI systems that are located in a third country, where the output produced by the system is used in the Union;

(86a) Given the rapid technological developments and the required technical expertise in conducting the assessment of high-risk AI systems, the delegation of powers and the implementing powers of the Commission should be exercised with as much flexibility as possible. The Commission should regularly review Annex III without undue delay, at least every six months, while consulting with the relevant stakeholders, including ethics experts, anthropologists, sociologists, mental health specialists and any other relevant scientists and researchers, as well as with parent associations.

(c) providers of AI systems that are located in a third country, where the output, meaning predictions, recommendations or decisions produced by the system and influencing the environment it interacts with, is used in the Union and puts at risk the health, safety or fundamental rights of natural persons physically present in the Union, insofar as the provider has permitted or is involved in such use;

Or. en

Proposal for a regulation

Article 23 – paragraph 1 – point c a (new)34

Text proposed by the Commission

Amendment

(ca) importers, distributors, and authorised representatives of providers of AI systems;

(34) ‘emotion recognition system’ means an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data;

(34) ‘emotion recognition system’ means an AI system for the purpose of identifying or inferring emotions, states of mind or intentions of natural persons on the basis of their biometric data or data relating to their physiological or behavioral characteristics;

Or. en

Proposal for a regulation

Article 23 – paragraph 21 – introductorypoint part35

Text proposed by the Commission

Amendment

2. For high-risk AI systems that are safety components of products or systems, or which are themselves products or systems, falling within the scope of the following acts, only Article 84 of this Regulation shall apply:

(35) ‘biometric categorisation system’ means an AI system for the purpose of assigning natural persons to specific categories, such as sex, age, hair colour, eye colour, tattoos, ethnic origin or sexual or political orientation, on the basis of their biometric data;

2. For AI systems classified as high-risk AI in accordance with Article 6 related to products covered by Union harmonisation legislation listed in Annex II, section B systems that are safety components of products or systems, or which are themselves products, falling within the scope of the following acts only Article 84 of this Regulation shall apply.

(35) ‘biometric categorisation system’ means an AI system that uses biometric data, or other physical, physiological or behavioral data, for the purpose of assigning natural persons to specific categories, such as sex, age, hair colour, eye colour, tattoos, ethnic origin or sexual or political orientation, on the basis of their biometric data;

Or. en

Proposal for a regulation

Article 23 – paragraph 21 – point a36

Text proposed by the Commission

Amendment

(a) Regulation (EC) No 300/2008;

(36) ‘remote biometric identification system’ means an AI system for the purpose of identifying natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, and without prior knowledge of the user of the AI system whether the person will be present and can be identified ;

deleted

(36) ‘remote biometric identification system’ means an AI system for the purpose of identifying natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database;

Or. en

Proposal for a regulation

Article 23 – paragraph 21 – point b39

Text proposed by the Commission

Amendment

(b) Regulation (EU) No 167/2013;

(39) ‘publicly accessible space’ means any physical place accessible to the public, regardless of whether certain conditions for access may apply;

deleted

(39) ‘publicly accessible space’ means any physical or virtual place accessible to the public, regardless of whether certain conditions for access may apply;

Or. en

Proposal for a regulation

Article 23 – paragraph 21 – point c44 a (new)

Text proposed by the Commission

Amendment

(c) Regulation (EU) No 168/2013;

(44a) ‘education and training institutions’ means providers where people of different ages gain education and training, including preschools, childcare, primary schools, secondary schools, tertiary education providers, vocational education and training and any type of lifelong learning providers.

deleted

Or. en

Proposal for a regulation

Article 25 – paragraph 21 – point da

Text proposed by the Commission

Amendment

(d) Directive 2014/90/EU;

(a) the placing on the market, putting into service or use of an AI system that deploys subliminal techniques beyond a person’s consciousness in order to materially distort a person’s behaviour in a manner that causes or is likely to cause that person or another person physical or psychological harm;

deleted

(a) the placing on the market, putting into service or use of an AI system that deploys techniques with the effect or likely effect of materially distorting a person’s behaviour in a manner that causes or is likely to cause that person or another person material or non-material harm including physical, psychological or economic harm;

Or. en

Proposal for a regulation

Article 25 – paragraph 21 – point eb

Text proposed by the Commission

Amendment

(e) Directive (EU) 2016/797;

(b) the placing on the market, putting into service or use of an AI system that exploits any of the vulnerabilities of a specific group of persons due to their age, physical or mental disability, in order to materially distort the behaviour of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;

deleted

(b) the placing on the market, putting into service or use of an AI system that exploits any of the vulnerabilities of a person due to their known or predicted personality or social or economic situation or due to their age, physical or mental capacity, in order to materially distort the behaviour of a person in a manner that causes or is likely to cause that person or another person material or non-material harm, including physical, psychological or economic harm;

Or. en

Proposal for a regulation

Article 25 – paragraph 21 – point fc – introductory part

Text proposed by the Commission

Amendment

(f) Regulation (EU) 2018/858;

(c) the placing on the market, putting into service or use of AI systems by public authorities or on their behalf for the evaluation or classification of the trustworthiness of natural persons over a certain period of time based on their social behaviour or known or predicted personal or personality characteristics, with the social score leading to either or both of the following:

deleted

(c) the placing on the market, putting into service or use of AI systems for the evaluation or classification of the trustworthiness of natural persons based on their social behaviour or known or predicted personal or personality characteristics;

Or. en

Proposal for a regulation

Article 25 – paragraph 21 – point gc – point i

Text proposed by the Commission

Amendment

(g) Regulation (EU) 2018/1139;

(i) detrimental or unfavourable treatment of certain natural persons or whole groups thereof in social contexts which are unrelated to the contexts in which the data was originally generated or collected;

deleted

Proposal for a regulation

Article 25 – paragraph 21 – point hc – point ii

Text proposed by the Commission

Amendment

(h) Regulation (EU) 2019/2144.

(ii) detrimental or unfavourable treatment of certain natural persons or whole groups thereof that is unjustified or disproportionate to their social behaviour or its gravity;

deleted

Proposal for a regulation

Article 25 – paragraph 31 – point d – introductory part

Text proposed by the Commission

Amendment

3. This Regulation shall not apply to AI systems developed or used exclusively for military purposes.

(d) the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement, unless and in as far as such use is strictly necessary for one of the following objectives:

3. This Regulation shall not apply to AI systems developed or used exclusively for military or national security purposes.

(d) the use of ‘real-time’ and ‘post’ remote biometric identification systems of natural persons in publicly accessible spaces.

Or. en

Proposal for a regulation

Article 25 – paragraph 31 a– (new)point d – point i

Text proposed by the Commission

Amendment

3a. This Regulation shall not affect any research, testing and development activity regarding AI systems, in particular but without limitation if it is conducted in an AI regulatory sandbox based on Article 53 of this Regulation, prior to the system being placed on the market or putting it into service. It shall neither apply to AI systems, including their output, specifically developed and put into service for the sole Purpose of scientific research, testing and development.

(i) the targeted search for specific potential victims of crime, including missing children;

deleted

Or. en

Proposal for a regulation

Article 25 – paragraph 31 b– (new)point d – point ii

Text proposed by the Commission

Amendment

3b. With the exception of Article 23a(2), this Regulation shall not apply to providers of general purpose AI systems, unless those systems have been adapted to a specific intended purpose that falls within the scope of this Regulation. This provision shall apply irrespective of whether the general purpose AI system is open source software or not.

(ii) the prevention of a specific, substantial and imminent threat to the life or physical safety of natural persons or of a terrorist attack;

deleted

Or. en

Proposal for a regulation

Article 25 – paragraph 31 c– (new)point d – point iii

Text proposed by the Commission

Amendment

3c. Title III of this Regulation shall not apply to AI systems that are used in a sole business-to-business (B2B) environment without any foreseeable impact or effect on end-users or other natural persons.

(iii) the detection, localisation, identification or prosecution of a perpetrator or suspect of a criminal offence referred to in Article 2(2) of Council Framework Decision 2002/584/JHA62 and punishable in the Member State concerned by a custodial sentence or a detention order for a maximum period of at least three years, as determined by the law of that Member State.

deleted

__________________

62 Council Framework Decision 2002/584/JHA of 13 June 2002 on the European arrest warrant and the surrender procedures between Member States (OJ L 190, 18.7.2002, p. 1).

Or. en

Proposal for a regulation

Article 35 – paragraph 1 – point 12

Text proposed by the Commission

Amendment

(1) ‘artificial intelligence system’ (AI system) means software that is developed with one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with;

2. The use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement for any of the objectives referred to in paragraph 1 point d) shall take into account the following elements:

(1) ‘artificial intelligence system’ (AI system) means a machine based system that is developed with the techniques and approaches listed in Annex I and is capable of influencing the environment by producing an output (predictions, recommendations or decisions) for a given set of objectives. It uses machine and/or human-based data and inputs to (i) perceive real and/or virtual environments; (ii) abstract these perceptions into models through analysis in an automated manner (e.g., with machine learning), or manually; and (iii) use model inference to formulate options for outcomes. AI systems are designed to operate with varying levels of autonomy.

deleted

(a) the nature of the situation giving rise to the possible use, in particular the seriousness, probability and scale of the harm caused in the absence of the use of the system;

(b) the consequences of the use of the system for the rights and freedoms of all persons concerned, in particular the seriousness, probability and scale of those consequences.

In addition, the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement for any of the objectives referred to in paragraph 1 point d) shall comply with necessary and proportionate safeguards and conditions in relation to the use, in particular as regards the temporal, geographic and personal limitations.

Or. en

Justification

Although the AI Act is an EU Regulation, it should use the wording developed by the OECD. Using this widely accepted definition will help the EU to better cooperate with non-EU democracies such as the USA, Canada or UK. Together, it will be easier to promote international standards based on our democratic values. The new definition for AI systems moreover creates legal certainty while providing enough flexibility by accommodating future technological developments.

Amendment 38

Proposal for a regulation

Article 35 – paragraph 1 – point 1 a (new)3

Text proposed by the Commission

Amendment

(1a) 'general purpose AI system' means AI system that is able to perform generally applicable functions such as image or speech recognition, audio or video generation, pattern detection, question answering, and translation, and is able to have multiple intended purposes;

3. As regards paragraphs 1, point (d) and 2, each individual use for the purpose of law enforcement of a ‘real-time’ remote biometric identification system in publicly accessible spaces shall be subject to a prior authorisation granted by a judicial authority or by an independent administrative authority of the Member State in which the use is to take place, issued upon a reasoned request and in accordance with the detailed rules of national law referred to in paragraph 4. However, in a duly justified situation of urgency, the use of the system may be commenced without an authorisation and the authorisation may be requested only during or after the use.

deleted

The competent judicial or administrative authority shall only grant the authorisation where it is satisfied, based on objective evidence or clear indications presented to it, that the use of the ‘real-time’ remote biometric identification system at issue is necessary for and proportionate to achieving one of the objectives specified in paragraph 1, point (d), as identified in the request. In deciding on the request, the competent judicial or administrative authority shall take into account the elements referred to in paragraph 2.

Or. en

Proposal for a regulation

Article 35 – paragraph 1 – point 1 b (new)4

Text proposed by the Commission

Amendment

(1b) 'autonomous’ means an AI-system that operates by interpreting certain input and by using a set of pre-determined objectives, without being limited to specific instructions, despite the system’s behaviour being constrained by, and targeted at, fulfilling the goal it was given and other relevant design choices made by its provider;

4. A Member State may decide to provide for the possibility to fully or partially authorise the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement within the limits and under the conditions listed in paragraphs 1, point (d), 2 and 3. That Member State shall lay down in its national law the necessary detailed rules for the request, issuance and exercise of, as well as supervision relating to, the authorisations referred to in paragraph 3. Those rules shall also specify in respect of which of the objectives listed in paragraph 1, point (d), including which of the criminal offences referred to in point (iii) thereof, the competent authorities may be authorised to use those systems for the purpose of law enforcement.

deleted

Or. en

Proposal for a regulation

Article 37 – paragraph 1 – point 1 cintroductory (new)part

Text proposed by the Commission

Amendment

(1c) ‘risk’ means the combination of the probabilityof occurrence of a harm and the severity of that harm;

1. The Commission is empowered to adopt delegated acts in accordance with Article 73 to update the list in Annex III by adding high-risk AI systems where both of the following conditions are fulfilled:

1. The Commission is empowered to adopt delegated acts in accordance with Article 73 to update the list in Annex III by adding high-risk AI systems where the AI systems pose a risk of harm to the health and safety, or a risk of adverse impact on fundamental rights, that is, in respect of its severity and probability of occurrence.

Or. en

Proposal for a regulation

Article 37 – paragraph 1 – point 2a

Text proposed by the Commission

Amendment

(2) ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or that has an AI system developed with a view to placing it on the market or putting it into service under its own name or trademark, whether for payment or free of charge;

(a) the AI systems are intended to be used in any of the areas listed in points 1 to 8 of Annex III;

(2) ‘developer’ means a natural or legal person, public authority, agency or other body, other than the deployer, that places an AI system on the market or putsit into service under its own name or trademark, whether for payment or free of charge or that adapts general purpose AI systems to a specific intended purpose;

deleted

Or. en

Justification

The modification of 'provider' to developer should be made consequentially throughout the text. 'Developer' is the term that is used in the AI community.

Amendment 42

Proposal for a regulation

Article 37 – paragraph 12 – pointintroductory 3part

Text proposed by the Commission

Amendment

(3) ‘small-scale provider’ means a provider that is a micro or small enterprise within the meaning of Commission Recommendation 2003/361/EC61 ;

2. When assessing for the purposes of paragraph 1 whether an AI system poses a risk of harm to the health and safety or a risk of adverse impact on fundamental rights that is equivalent to or greater than the risk of harm posed by the high-risk AI systems already referred to in Annex III, the Commission shall take into account the following criteria:

deleted

2. When assessing for the purposes of paragraph 1 whether an AI system poses a risk of harm to the health and safety or a risk of adverse impact on fundamental rights, society or the environment that is equivalent to or greater than the risk of harm posed by the high-risk AI systems already referred to in Annex III, the Commission shall take into account the following criteria:

__________________

61 Commission Recommendation of 6 May 2003 concerning the definition of micro, small and medium-sized enterprises (OJ L 124, 20.5.2003, p. 36).

Or. en

Justification

Deletion due to proposed changes in Art 55.

Amendment 43

Proposal for a regulation

Article 37 – paragraph 1 –2 pointa 4(new)

Text proposed by the Commission

Amendment

(4) ‘user’ means any natural or legal person, public authority, agency or other body using an AI system under its authority, except where the AI system is used in the course of a personal non-professional activity;

2a. The Commission shall conduct the assessment referred to in paragraph 2 without undue delay and at least every six months, under the consultation conditions laid down in this Regulation, in particular in Article 73.

(4) ‘deployer’ means any natural or legal person, public authority, agency or other body using an AI system under its authority, except where the AI system is used in the course of a personal non-professional activity;

Or. en

Justification

The modification of 'user' to deployer should be made consequentially throughout the text. 'Deployer' is the term that is used in the AI community. 'User' would at the same time lead to legal overlaps and contradictions with other laws such as the GDPR.

Amendment 44

Proposal for a regulation

Article 373 – paragraph 1 – point 43 a (new)

Text proposed by the Commission

Amendment

(4a) ‘end-user’ means any natural person who, in the context of employment or contractual agreement with the deployer, uses the AI system under the authority of the deployer;

3a. Before adopting a delegated act, the Commission shall consult with the relevant institutions and stakeholders in accordance with the principles laid down in the Interinstitutional Agreement of 13 April 2016 on Better Law-Making.

Or. en

Proposal for a regulation

ArticleAnnex 3III – paragraph 1 – point 111 – point a a (new)

Text proposed by the Commission

Amendment

(11) ‘putting into service’ means the supply of an AI system for first use directly to the user or for own use on the Union market for its intended purpose;

(aa) AI systems which use data relating to physical, physiological or behavioural characteristics, including biometric data;

(11) ‘putting into service’ means the first use of an AI system within the Union by the user or the end-user for the purpose for which it was intended;

Or. en

Proposal for a regulation

ArticleAnnex 3III – paragraph 1 – point 141 – point a b (new)

Text proposed by the Commission

Amendment

(14) ‘safety component of a product or system’ means a component of a product or of a system which fulfils a safety function for that product or system or the failure or malfunctioning of which endangers the health and safety of persons or property;

(ab) AI systems that are intended be used for the non-remote biometric identification of natural persons in publicly accessible spaces, as well as in workplaces and in education and training institutions.

(14) ‘safety component of a product or system’ means, in line with the relevant Union harmonisation legislation listed in Annex II, a component of a productor of a system which fulfils a safety function for that product or system so that the malfunction of this component significantly and almost certainly directly or immediately endangers the health and safety of persons;

Or. en

Proposal for a regulation

ArticleAnnex 3III – paragraph 1 – point 153 – introductory part

Text proposed by the Commission

Amendment

(15) ‘instructions for use’ means the information provided by the provider to inform the user of in particular an AI system’s intended purpose and proper use, inclusive of the specific geographical, behavioural or functional setting within which the high-risk AI system is intended to be used;

3. Education and vocational training:

(15) ‘instructions for use’ means the information provided by the provider to inform the user of in particular an AI system’s intended purpose and proper use;

3. Education and training:

Or. en

Proposal for a regulation

ArticleAnnex 3III – paragraph 1 – point 233 – point a

Text proposed by the Commission

Amendment

(23) ‘substantial modification’ means a change to the AI system following its placing on the market or putting into service which affects the compliance of the AI system with the requirements set out in Title III, Chapter 2 of this Regulation or results in a modification to the intended purpose for which the AI system has been assessed;

(a) AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and vocational training institutions;

(23) ‘substantial modification’ means a change to the AI system following its placing on the market or putting into service, which creates a new or increased risk and significantly affects the compliance of the AI system with the requirements set out in Title III, Chapter 2 of this Regulation or results in a modification to the intended purpose for which the AI system has been assessed;

(a) AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and training institutions;

Or. en

Proposal for a regulation

ArticleAnnex 3III – paragraph 1 – point 243 – point b

Text proposed by the Commission

Amendment

(24) ‘CE marking of conformity’ (CE marking) means a marking by which a provider indicates that an AI system is in conformity with the requirements set out in Title III, Chapter 2 of this Regulation and other applicable Union legislation harmonising the conditions for the marketing of products (‘Union harmonisation legislation’) providing for its affixing;

(b) AI systems intended to be used for the purpose of assessing students in educational and vocational training institutions and for assessing participants in tests commonly required for admission to educational institutions.

(24) ‘CE marking of conformity’ (CE marking) means a marking by which a provider indicates that a product with an embedded AI system is in conformity with the requirements set out in Title III, Chapter 2 of this Regulation and other applicable Union legislation harmonising the conditions for the marketing of products (‘Union harmonisation legislation’) providing for its affixing;

(b) AI systems intended to be used for the purpose of assessing students in educational and training institutions and for assessing participants in tests commonly required for admission to educational institutions;

Or. en

Proposal for a regulation

ArticleAnnex 3III – paragraph 1 – point 333 – point b a (new)

Text proposed by the Commission

Amendment

(33) ‘biometric data’ means personal data resulting from specific technical processing relating to the physical, physiological or behavioural characteristics of a natural person, which allow or confirm the unique identification of that natural person, such as facial images or dactyloscopic data;

(ba) AI systems intended to be used for monitoring and detecting prohibited behaviour of students during tests at education and training institutions;

(33) ‘biometric data’ means personal data resulting from specific technical processing relating to the physical, or physiological or behavioural characteristics of a natural person, which confirm the unique identification of that natural person, such as dactyloscopic data;

Or. en

Proposal for a regulation

ArticleAnnex 3III – paragraph 1 – point 343 – point b b (new)

Text proposed by the Commission

Amendment

(34) ‘emotion recognition system’ means an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data;

(bb) AI systems intended to be used for the purpose of determining the study program or areas of study to be followed by students in education and training institutions;

(34) ‘emotion recognition system’ means an AI system for the purpose of identifying or inferring emotions of natural persons on the basis of their biometric or other data obtained, read or interpreted from an individual;

Or. en

Proposal for a regulation

ArticleAnnex 3III – paragraph 1 – point 373 a (new)

Text proposed by the Commission

Amendment

(37) ‘‘real-time’ remote biometric identification system’ means a remote biometric identification system whereby the capturing of biometric data, the comparison and the identification all occur without a significant delay. This comprises not only instant identification, but also limited short delays in order to avoid circumvention.

3a. Media and culture

(37) ‘‘real-time’ remote biometric identification system’ means a remote biometric identification system whereby the capturing of biometric data, the comparison and the identification all occur without a significant delay.

(a) AI systems used by news media outlets intended to be used to create or disseminate machine-generated news articles;

(b) AI systems used by audiovisual media services intended to make recommendations to natural persons or to prioritize or rank specific audiovisual content in the online interface of the audiovisual media service.

Or. en

Proposal for a regulation

Article 3 – paragraph 1 – point 44 – introductory part

Text proposed by the Commission

Amendment

(44) ‘serious incident’ means any incident that directly or indirectly leads, might have led or might lead to any of the following:

(44) ‘serious incident’ means any incident that directly or indirectly leads to any of the following:

Or. en

Amendment 54

Proposal for a regulation

Article 3 – paragraph 1 – point 44 – point a

Text proposed by the Commission

Amendment

(a) the death of a person or serious damage to a person’s health, to property or the environment,

(a) the death of a person or serious damage to a person’s health,

Or. en

Amendment 55

Proposal for a regulation

Article 3 – paragraph 1 – point 44 a (new)

Text proposed by the Commission

Amendment

(44a) ‘incident’means a faulty operation of an AI system;

Or. en

Amendment 56

Proposal for a regulation

Article 3 – paragraph 1 – point 44 b (new)

Text proposed by the Commission

Amendment

(44b) 'personal data' means data as defined in point (1) of Article 4 of Regulation (EU) 2016/679;

Or. en

Amendment 57

Proposal for a regulation

Article 3 – paragraph 1 – point 44 c (new)

Text proposed by the Commission

Amendment

(44c) ‘non-personal data’ means data other than personal data as defined in point (1) of Article 4 of Regulation (EU) 2016/679;

Or. en

Amendment 58

Proposal for a regulation

Article 3 – paragraph 1 – point 44 d (new)

Text proposed by the Commission

Amendment

(44d) ‘deep fake’ means manipulated or synthetic audio, image or video content that would falsely appear to be authentic or truthful, and which features depictions of persons appearing to say or do things they did not say or do, without their consent, produced using AI techniques, including machine learning and deep learning;

Or. en

Amendment 59

Proposal for a regulation

Article 3 – paragraph 1 – point 44 e (new)

Text proposed by the Commission

Amendment

(44e) 'regulatory sandbox’ means a framework which, by providing a structured context for experimentation, enable where appropriate in a real-world or digital environment the testing of innovative technologies, products, services or approaches for a limited time and in a limited part of a sector or area under regulatory supervision ensuring that appropriate safeguards are in place.

Or. en

Amendment 60

Proposal for a regulation

Article 4 – paragraph 1

Text proposed by the Commission

Amendment

The Commission is empowered to adopt delegated acts in accordance with Article 73 to amend the list of techniques and approaches listed in Annex I, in order to update that list to market and technological developments on the basis of characteristics that are similar to the techniques and approaches listed therein.

The Commission is empowered to adopt delegated acts in accordance with Article 73, after ensuring adequate consultation with relevant stakeholders, to amend the list of techniques and approaches listed in Annex I within the scope of the definition of an AI system as provided for in Article 3(1), in order to update that list to market and technological developments on the basis of transparent characteristics.

Or. en

Amendment 61

Proposal for a regulation

Article 4 a (new)

Text proposed by the Commission

Amendment

Article 4 a

Trustworthy AI

1. Providers of an AI system shall, throughout all stages of development of the AI system, acknowledge the EU Charta of Fundamental Rights and ensure that the AI system is lawful, ethical and robust.

(a) ‘lawful’ means that the AIsystem is developed to operate in accordance with European, national and international legally binding rules;

(b) ‘ethical’ means that the AI system is developed taking into account the specific benefits of the AI system while respecting the freedom and autonomy of human beings, human dignity as well as mental and physical integrity, and to be fair and explicable;

(c) ‘robust’ means that the AI system performs in a safe, secure and reliable manner, with embedded safeguards to as much as possible prevent any unintended adverse impacts.

2. The concept set out inparagraph 1 shall be taken into account by the European Standardisation Organisations as outcome-based objectives when they develop appropriate harmonised standards for high risk AI systems as referred to in Article 40(2b).

For all other AI systems, the voluntary application on the basis of technical specifications and solutions through codes of conducts as referred to in Article 69(1a) is highly encouraged.

3. Providers and users of high-risk AI systems shall complete a trustworthy technology assessment, incompliance with paragraph 1 and as part of the requirements under Article 16(a) and 29(4).

4. The Commission shall provide an assessment list for developing Trustworthy AI and create an AI toolkit for AI and data protection risk mitigation and management.

Or. en

Justification

The EU Institutions regularly speak about the European approach to AI and underline that our AI is “trustworthy”. However, until now a concrete concept behind this term is missing. The AI Act did not fill this conceptual gap either. In particular, it did not take the INL on AI ethics, written by the JURI committee into account, which could have solved the problem. Therefore, our new Article aims to address the gap and outline, what makes European AI unique. It thereby draws heavily on the work of the High Level Expert Group on AI. This article is at the same time a political signal to the European Standardisation Organisations (ESOs), which told us frequently that standardisation experts are waiting for directions. The political level should create a framework key requirements on meta level, which the ESOs could then translate into specified technical standards. Without clear political indications, ESOs will have problems to quickly adopt harmonised Standards.

Amendment 62

Proposal for a regulation

Article 5 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) the placing on the market, putting into service or use of an AI system that deploys subliminal techniques beyond a person’s consciousness in order to materially distort a person’s behaviour in a manner that causes or is likely to cause that person or another person physical or psychological harm;

(a) the placing on the market, putting into service or use of an AI system with the objective to significantly and materially distorting a person’s behaviour or directly causing that person or another person significant harm;

Or. en

Amendment 63

Proposal for a regulation

Article 5 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) the placing on the market, putting into service or use of an AI system that exploits any of the vulnerabilities of a specific group of persons due to their age, physical or mental disability, in order to materially distort the behaviour of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;

(b) the placing on the market, putting into service or use of an AI system that exploits any of the vulnerabilities of a specific group of persons with the objective to significantly and materially distort the behaviour of a person pertaining to that group to cause that person or another person significant harm;

Or. en

Amendment 64

Proposal for a regulation

Article 5 – paragraph 1 – point c – point i

Text proposed by the Commission

Amendment

(i) detrimental or unfavourable treatment of certain natural persons or whole groups thereof in social contexts which are unrelated to the contexts in which the data was originally generated or collected;

(i) detrimental or unfavourable treatment of certain natural persons or groups thereof in social contexts which are unrelated to the contexts in which the data was originally generated or collected;

Or. en

Amendment 65

Proposal for a regulation

Article 5 – paragraph 1 – point c – point ii

Text proposed by the Commission

Amendment

(ii) detrimental or unfavourable treatment of certain natural persons or whole groups thereof that is unjustified or disproportionate to their social behaviour or its gravity;

(ii) detrimental or unfavourable treatment of certain natural persons or groups thereof that is unjustified or disproportionate to their social behaviour or its gravity;

Or. en

Amendment 66

Proposal for a regulation

Article 5 – paragraph 1 – point d – introductory part

Text proposed by the Commission

Amendment

(d) the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement, unless and in as far as such use is strictly necessary for one of the following objectives:

(d) the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces by law enforcement authorities or on their behalf for the purpose of law enforcement, unless used for one of the following objectives:

Or. en

Amendment 67

Proposal for a regulation

Article 5 – paragraph 3 – introductory part

Text proposed by the Commission

Amendment

3. As regards paragraphs 1, point (d) and 2, each individual use for the purpose of law enforcement of a ‘real-time’ remote biometric identification system in publicly accessible spaces shall be subject to a prior authorisation granted by a judicial authority or by an independent administrative authority of the Member State in which the use is to take place, issued upon a reasoned request and in accordance with the detailed rules of national law referred to in paragraph 4. However, in a duly justified situation of urgency, the use of the system may be commenced without an authorisation and the authorisation may be requested only during or after the use.

3. As regards paragraphs 1, point (d) and 2, each use for the purpose of law enforcement of a ‘real-time’ remote biometric identification system in publicly accessible spaces shall be subject to a prior authorisation granted by a judicial authority or by an independent administrative authority of the Member State in which the use is to take place, issued upon a reasoned request and in accordance with the detailed rules of national law referred to in paragraph. However, in a duly justified situation of urgency, the use of the system may be commenced without an authorisation if such authorisation is requested without undue delay, and, if such authorisation is rejected, the system’s use is stopped with immediate effect.

Or. en

Amendment 68

Proposal for a regulation

Article 5 – paragraph 4

Text proposed by the Commission

Amendment

4. A Member State may decide to provide for the possibility to fully or partially authorise the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement within the limits and under the conditions listed in paragraphs 1, point (d), 2 and 3. That Member State shall lay down in its national law the necessary detailed rules for the request, issuance and exercise of, as well as supervision relating to, the authorisations referred to in paragraph 3. Those rules shall also specify in respect of which of the objectives listed in paragraph 1, point (d), including which of the criminal offences referred to in point (iii) thereof, the competent authorities may be authorised to use those systems for the purpose of law enforcement.

4. A Member State may decide to provide for the possibility to fully or partially authorise the use of ‘real-time’ remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement within the limits and under the conditions listed in paragraphs 1, point (d), 2 and 3. That Member State shall lay down in its national law the necessary detailed rules for the request, issuance and exercise of, as well as supervision and reporting relating to, the authorisations referred to in paragraph 3. Those rules shall also specify in respect of which of the objectives listed in paragraph 1, point (d), including which of the criminal offences referred to in point (iii) thereof, the competent authorities may be authorised to use those systems for the purpose of law enforcement.

Or. en

Amendment 69

Proposal for a regulation

Article 6 – paragraph 1 – introductory part

Text proposed by the Commission

Amendment

1. Irrespective of whether an AI system is placed on the market or put into service independently from the products referred to in points (a) and (b), that AI system shall be considered high-risk where both of the following conditions are fulfilled:

1. An AI system that is itself a product shall be considered as high risk if, under the applicable Union harmonisation legislation listed in Annex II, it is classified as high-risk or an equivalent thereof and has to undergo a third-party conformity assessment for meeting essential safety requirements prior to placing it on the market or putting it into service.

An AI system intended to be used as a relevant safety component of a product under the applicable Union harmonisation legislation listed in Annex II, shall be considered as high risk if such Union harmonisation legislation classifies it as high-risk or an equivalent thereof and requires it to undergo a third-party conformity assessment for meeting essential safety requirements with a view to placing it on the market or putting it into service.

Or. en

Amendment 70

Proposal for a regulation

Article 6 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) the AI system is intended to be used as a safety component of a product, or is itself a product, covered by the Union harmonisation legislation listed in Annex II;

deleted

Or. en

Amendment 71

Proposal for a regulation

Article 6 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) the product whose safety component is the AI system, or the AI system itself as a product, is required to undergo a third-party conformity assessment with a view to the placing on the market or putting into service of that product pursuant to the Union harmonisation legislation listed in Annex II.

deleted

Or. en

Amendment 72

Proposal for a regulation

Article 6 – paragraph 2

Text proposed by the Commission

Amendment

2. In addition to the high-risk AI systems referred to in paragraph 1, AI systems referred to in Annex III shall also be considered high-risk.

2. In addition to the high-risk AI systems referred to in paragraph 1, an AI systems shall also be considered high-risk if its intended purpose means that it will be deployed in a critical area referred to in Annex III and in such a manner that significant harm is likely to arise. The provider of an AI system, where the intended purpose falls under a critical area referred to in Annex III, shall be required to carry out an assessment of the risks that could be posed to the health and safety or fundamental rights of potentially impacted persons.

Or. en

Justification

We support the approach in the 2020 AI White Paper by the European Commission, which introduced a specific threshold to determine the risk of an AI system. Only when the sector as well as the intended use of the AI system involves significant risks, the AI system shall be considered as high risk. Unfortunately, the AI Act not only gave up on this balanced approach, the current Annex III is also way too broad and vague. As a result, it would categorize whole sectors as high risk, whereby numerous AI systems with hardly any risk would face the burdensome obligations of Chapter 2. In other words, this is not a risk-based approach! In contrast, our system reintroduces the White Paper threshold and makes sure that only those AI systems are categorized as “high risk” that fulfil clear and transparent criteria. An EU that strives for global leadership in AI will only be able to reach this goal with a truly risk-based approach that strikes the right balance between promoting innovation and protecting fundamental rights.

Amendment 73

Proposal for a regulation

Article 6 – paragraph 2 a (new)

Text proposed by the Commission

Amendment

2a. This high risk assessment shall contain the following elements:

(a a description of the AI system, including the intended purpose, the potential persons impacted, the extent to which any outcomes produced are subject to human review or intervention, and where applicable, the legitimate interest pursued by the user;

(b) an assessment of the potential benefits provided by the use of the AI system, as well as reticence risk and/or opportunity costs of not using the AI for individuals, groups of individuals, or society at large. This includes weighing the benefits of deploying the AI system against keeping the status quo;

(c) an assessment of any potentialmaterial risks of harm, including likelihood and severity of that harm, to the health and safety or fundamental rights of potentially impacted persons, including:

(i) the extent to which the AI system has already been evaluated and proven to havecaused material harm as demonstrated by studies or reports published by thenational competent authorities;

(ii) the extent to which potentially impacted persons are dependent on the outcome produced from the AI system, in particular because for practical or legal reasons it is not reasonably possible to opt-out from that outcome;

(iii) the extent to which the outcome produced by the AI system is easily reversible;

(iv) the extent to which potentially impacted persons are in a vulnerable position in relation to the user of the AI system, in particular due to an imbalance of power, knowledge, economic or social circumstances, or age.

(d) measures taken to address ormitigate the identified risks, including to the extent existing Union legislation provides for:

(i) effective measures of redress in relation to the risks posed by an AI system, with the exclusion of claims for damages;

(ii) effective measures to prevent or substantially minimise those risks.

Or. en

Amendment 74

Proposal for a regulation

Article 6 – paragraph 2 b (new)

Text proposed by the Commission

Amendment

2b. The Commission, in close cooperation with the Board and the High level Expert Group on AI, shall provide guidelines for high risk assessments according to paragraph 3 as well as best practices, to serve as additional support for providers that aim to comply with this Article. The national competent authorities shall also provide direct consultation for providers in this regard.

Or. en

Amendment 75

Proposal for a regulation

Article 7 – paragraph 1 – introductory part

Text proposed by the Commission

Amendment

1. The Commission is empowered to adopt delegated acts in accordance with Article 73 to update the list in Annex III by adding high-risk AI systems where both of the following conditions are fulfilled:

1. The Commission is empowered to adopt delegated acts in accordance with Article 73, after ensuring adequate consultation with relevant stakeholders, to update the list in Annex III by adding high-risk AI systems where both of the following conditions are fulfilled:

Or. en

Amendment 76

Proposal for a regulation

Article 7 – paragraph 2 – point a a (new)

Text proposed by the Commission

Amendment

(aa) the general capabilities and functionalities of the AI system independent of its intended purpose;

Or. en

Amendment 77

Proposal for a regulation

Article 7 – paragraph 2 – point b a (new)

Text proposed by the Commission

Amendment

(ba) the extent to which the AI system acts autonomously;

Or. en

Amendment 78

Proposal for a regulation

Article 7 – paragraph 2 – point e a (new)

Text proposed by the Commission

Amendment

(ea) the potential misuse and malicious use of the AI system and of the technology underpinning it;

Or. en

Amendment 79

Proposal for a regulation

Article 7 – paragraph 2 – point g a (new)

Text proposed by the Commission

Amendment

(ga) the extent of the availability and use of demonstrated technical solutions and mechanisms for the control, reliability and corrigibility of the AI system;

Or. en

Amendment 80

Proposal for a regulation

Article 7 – paragraph 2 – point g b (new)

Text proposed by the Commission

Amendment

(gb) the extent of human oversight and the possibility for a human to intercede in order to override a decision;

Or. en

Amendment 81

Proposal for a regulation

Article 7 – paragraph 2 – point g c (new)

Text proposed by the Commission

Amendment

(gc) magnitude and likelihood of benefit of the deployment of the AI system for individuals, groups, or society at large;

Or. en

Amendment 82

Proposal for a regulation

Article 7 – paragraph 2 – point g d (new)

Text proposed by the Commission

Amendment

(gd) reticence risk and/or opportunity costs of not using the AI for individuals, groups of individuals, or society at large.

Or. en

Amendment 83

Proposal for a regulation

Article 7 – paragraph 2 – point h – introductory part

Text proposed by the Commission

Amendment

(h) the extent to which existing Union legislation provides for:

(h) the extent to which existing Union legislation, in particular the GDPR, provides for:

Or. en

Amendment 84

Proposal for a regulation

Article 8 – paragraph 1

Text proposed by the Commission

Amendment

1. High-risk AI systems shall comply with the requirements established in this Chapter.

1. High-risk AI systems shall comply with the essential requirements established in this Chapter, taking into account the generally acknowledged state of the art, including as reflected in relevant industry standards.

Or. en

Amendment 85

Proposal for a regulation

Article 9 – paragraph 1

Text proposed by the Commission

Amendment

1. A risk management system shall be established, implemented, documented and maintained in relation to high-risk AI systems.

1. A risk management system shall be established, implemented, documented and maintained in relation to high-risk AI systems if the high-risk AI system poses a risk of harm to the health and safety or a risk of adverse impact on fundamental rights.

Or. en

Amendment 86

Proposal for a regulation

Article 9 – paragraph 2 – introductory part

Text proposed by the Commission

Amendment

2. The risk management system shall consist of a continuous iterative process run throughout the entire lifecycle of a high-risk AI system, requiring regular systematic updating. It shall comprise the following steps:

2. The risk management system shall consist of a continuous iterative process run throughout the entire lifecycle of a high-risk AI system, requiring regular review of the suitability of the risk management process. It shall comprise the following steps:

Or. en

Amendment 87

Proposal for a regulation

Article 9 – paragraph 2 – point a

Text proposed by the Commission

Amendment

(a) identification and analysis of the known and foreseeable risks associated with each high-risk AI system;

(a) identification and analysis of the known and foreseeable risks of harms most likely to occur to the health, safety or to the fundamental rights in view of the intended purpose of the high-risk AI system;

Or. en

Amendment 88

Proposal for a regulation

Article 9 – paragraph 2 – point b

Text proposed by the Commission

Amendment

(b) estimation and evaluation of the risks that may emerge when the high-risk AI system is used in accordance with its intended purpose and under conditions of reasonably foreseeable misuse;

deleted

Or. en

Amendment 89

Proposal for a regulation

Article 9 – paragraph 2 – point c

Text proposed by the Commission

Amendment

(c) evaluation of other possibly arising risks based on the analysis of data gathered from the post-market monitoring system referred to in Article 61;

(c) evaluation of new risks consistent with those described in paragraph (2a) of this Article and identified based on the analysis of data gathered from the post-market monitoring system referred to in Article 61;

Or. en

Amendment 90

Proposal for a regulation

Article 9 – paragraph 2 – point d

Text proposed by the Commission

Amendment

(d) adoption of suitable risk management measures in accordance with the provisions of the following paragraphs.

(d) adoption of appropriate and targeted risk management measures, designed to address identified known and foreseeable risks to health and safety or fundamental human rights, in accordance with the provisions of the following paragraphs;

Or. en

Amendment 91

Proposal for a regulation

Article 9 – paragraph 2 – point d a (new)

Text proposed by the Commission

Amendment

(da) completion of a use-case specific and evidence-based trustworthy technology assessment taking into account the principles laid down in Article 4a.

The risks referred to in this paragraph shall concern only those which may be foreseen or reasonably mitigated or even eliminated through the development or design of the high-risk AI system, or the provision of adequate technical information.

Or. en

Amendment 92

Proposal for a regulation

Article 9 – paragraph 3

Text proposed by the Commission

Amendment

3. The risk management measures referred to in paragraph 2, point (d) shall give due consideration to the effects and possible interactions resulting from the combined application of the requirements set out in this Chapter 2. They shall take into account the generally acknowledged state of the art, including as reflected in relevant harmonised standards or common specifications.

3. The risk management measures referred to in paragraph 2, point (d) shall give due consideration to the effects and possible interactions resulting from the combined application of the requirements set out in this Chapter 2, with a view to minimising risks more effectively while achieving an appropriate balance in implementing the measures to fulfil those requirements.

Or. en

Amendment 93

Proposal for a regulation

Article 9 – paragraph 4 – introductory part

Text proposed by the Commission

Amendment

4. The risk management measures referred to in paragraph 2, point (d) shall be such that any residual risk associated with each hazard as well as the overall residual risk of the high-risk AI systems is judged acceptable, provided that the high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse. Those residual risks shall be communicated to the user.

4. The risk management measures referred to in paragraph 2, point (d) shall be such that any significant residual risk associated with each hazard as well as the overall residual risk of the high-risk AI systems is reasonably judged to be acceptable, having regards to the benefits that the high-risk AI system is reasonably expected to deliver and provided that the high-risk AI system is used in accordance with its intended purpose. Those significant residual risks shall be communicated to the user.

Or. en

Amendment 94

Proposal for a regulation

Article 9 – paragraph 4 – subparagraph 1 – introductory part

Text proposed by the Commission

Amendment

In identifying the most appropriate risk management measures, the following shall be ensured:

In identifying the most appropriate risk management measures, the following shall be pursued:

Or. en

Amendment 95

Proposal for a regulation

Article 9 – paragraph 4 – subparagraph 1 – point a

Text proposed by the Commission

Amendment

(a) elimination or reduction of risks as far as possible through adequate design and development;

(a) reduction of identified and evaluated risks as far as commercially reasonable and technologically feasible through adequate design and development of the high risk AI system;

Or. en

Amendment 96

Proposal for a regulation

Article 9 – paragraph 4 – subparagraph 1 – point b

Text proposed by the Commission

Amendment

(b) where appropriate, implementation of adequate mitigation and control measures in relation to risks that cannot be eliminated;

deleted

Or. en

Amendment 97

Proposal for a regulation

Article 9 – paragraph 4 – subparagraph 1 – point c

Text proposed by the Commission

Amendment

(c) provision of adequate information pursuant to Article 13, in particular as regards the risks referred to in paragraph 2, point (b) of this Article, and, where appropriate, training to users.

(c) provisions of adequate information pursuant to Article 13, and, where appropriate, training to users.

Or. en

Amendment 98

Proposal for a regulation

Article 9 – paragraph 4 – subparagraph 2

Text proposed by the Commission

Amendment

In eliminating or reducing risks related to the use of the high-risk AI system, due consideration shall be given to the technical knowledge, experience, education, training to be expected by the user and the environment in which the system is intended to be used.

In reducing risks related to the use of the high-risk AI system, due consideration shall be given to the technical knowledge, experience, education, training to be expected by the user and the environment in which the system is intended to be used.

Or. en

Amendment 99

Proposal for a regulation

Article 9 – paragraph 5

Text proposed by the Commission

Amendment

5. High-risk AI systems shall be tested for the purposes of identifying the most appropriate risk management measures. Testing shall ensure that high-risk AI systems perform consistently for their intended purpose and they are in compliance with the requirements set out in this Chapter.

5. High-risk AI systems shall be evaluated for the purposes of identifying the most appropriate and targeted risk management measures and weighing any such measures against the potential benefits and intended goals of the system. Evaluations shall ensure that high-risk AI systems are in compliance with the relevant requirements set out in this Chapter.

Or. en

Amendment 100

Proposal for a regulation

Article 9 – paragraph 9

Text proposed by the Commission

Amendment

9. For credit institutions regulated by Directive 2013/36/EU, the aspects described in paragraphs 1 to 8 shall be part of the risk management procedures established by those institutions pursuant to Article 74 of that Directive.

9. For providers and AI systems already covered by Union law that require them to carry out specific risk assessments, the aspects described in paragraphs 1 to 8 may be combined with the risk assessment procedures established by that Union law or deemed to be covered as part of it.

Or. en

Amendment 101

Proposal for a regulation

Article 10 – paragraph 1

Text proposed by the Commission

Amendment

1. High-risk AI systems which make use of techniques involving the training of models with data shall be developed on the basis of training, validation and testing data sets that meet the quality criteria referred to in paragraphs 2 to 5.

1. High-risk AI systems which make use of techniques involving the training of models with data shall be, as far as this can be reasonably expected and is feasible from a technical point of view, developed on the basis of training, validation and testing data sets that meet the quality criteria referred to in paragraphs 2 to 5.

Or. en

Amendment 102

Proposal for a regulation

Article 10 – paragraph 2 – introductory part

Text proposed by the Commission

Amendment

2. Training, validation and testing data sets shall be subject to appropriate data governance and management practices. Those practices shall concern in particular,

2. Training, validation and testing data sets shall be subject to appropriate data governance and management practices for the entire lifecycle of data processing. Those practices shall concern, where relevant,

Or. en

Amendment 103

Proposal for a regulation

Article 10 – paragraph 2 – point a

Text proposed by the Commission

Amendment

(a) the relevant design choices;

(a) the design choices;

Or. en

Amendment 104

Proposal for a regulation

Article 10 – paragraph 2 – point c

Text proposed by the Commission

Amendment

(c) relevant data preparation processing operations, such as annotation, labelling, cleaning, enrichment and aggregation;

(c) data preparation processing operations, such as annotation, labelling, cleaning, enrichment and aggregation;

Or. en

Amendment 105

Proposal for a regulation

Article 10 – paragraph 2 – point e

Text proposed by the Commission

Amendment

(e) a prior assessment of the availability, quantity and suitability of the data sets that are needed;

deleted

Or. en

Amendment 106

Proposal for a regulation

Article 10 – paragraph 2 – point f

Text proposed by the Commission

Amendment

(f) examination in view of possible biases;

(f) examination in view of possible biases that are likely to affect health and safety of persons or lead to discrimination prohibited by Union law;

Or. en

Amendment 107

Proposal for a regulation

Article 10 – paragraph 2 – point g

Text proposed by the Commission

Amendment

(g) the identification of any possible data gaps or shortcomings, and how those gaps and shortcomings can be addressed.

(g) the identification of significant data gaps or shortcomings, and how those gaps and shortcomings can be addressed.

Or. en

Amendment 108

Proposal for a regulation

Article 10 – paragraph 2 – point g a (new)

Text proposed by the Commission

Amendment

(ga) the context of the use as well as the intended purpose of the AI system.

Or. en

Amendment 109

Proposal for a regulation

Article 10 – paragraph 3

Text proposed by the Commission

Amendment

3. Training, validation and testing data sets shall be relevant, representative, free of errors and complete. They shall have the appropriate statistical properties, including, where applicable, as regards the persons or groups of persons on which the high-risk AI system is intended to be used. These characteristics of the data sets may be met at the level of individual data sets or a combination thereof.

3. High risk AI systems should be designed and developed with the best efforts to ensure that, where appropriate, training datasets, validation and testing data sets are sufficiently relevant and representative in view of the intended purpose of the AI system. These characteristics of the data sets may be met at the level of individual datasets or a combination thereof.

Or. en

Amendment 110

Proposal for a regulation

Article 10 – paragraph 3 a (new)

Text proposed by the Commission

Amendment

3a. In assessing the quality of a data set, account shall be taken to the extent to which the data set is constructed with a view to fulfilling in particular the following aspects:

a) provides a similar output for relevant demographic Groups impacted by the system;

b) minimizes disparities in outcomes for relevant demographic groups impacted by the system, in case where the system allocates resources or opportunities to natural persons;

c) minimizes the potential for stereotyping, demeaning, or erasing relevant demographic groups impacted by the system where the system describes, depicts, or otherwise represents people, cultures, or society.

Or. en

Amendment 111

Proposal for a regulation

Article 10 – paragraph 4

Text proposed by the Commission

Amendment

4. Training, validation and testing data sets shall take into account, to the extent required by the intended purpose, the characteristics or elements that are particular to the specific geographical, behavioural or functional setting within which the high-risk AI system is intended to be used.

4. The processing of lawfully collected personal data to train, validate and test data sets of an AI system with the purpose of meeting the requirements of this regulation shall be considered a legitimate interest for the provider or user in accordance with Article 6(1f) GDPR as well as a compatible purpose in accordance with Article 6(4) GDPR as long as appropriate safeguards in line with Article 89 GDPR are in place to prevent abusive processing of personal data.

Or. en

Amendment 112

Proposal for a regulation

Article 10 – paragraph 5

Text proposed by the Commission

Amendment

5. To the extent that it is strictly necessary for the purposes of ensuring bias monitoring, detection and correction in relation to the high-risk AI systems, the providers of such systems may process special categories of personal data referred to in Article 9(1) of Regulation (EU) 2016/679, Article 10 of Directive (EU) 2016/680 and Article 10(1) of Regulation (EU) 2018/1725, subject to appropriate safeguards for the fundamental rights and freedoms of natural persons, including technical limitations on the re-use and use of state-of-the-art security and privacy-preserving measures, such as pseudonymisation, or encryption where anonymisation may significantly affect the purpose pursued.

5. To the extent that it is strictly necessary for the purposes of ensuring bias monitoring, detection and correction in relation to the high-risk AI systems the providers of such systems may also process special categories of personal data referred to in Article 9(1) of Regulation (EU) 2016/679, Article 10 of Directive (EU) 2016/680 and Article 10(1) of Regulation (EU) 2018/1725,subject to appropriate safeguards for the fundamental rights and freedoms of natural persons, including:

(i) technical limitations on the re-use and use of state-of-the-art security and privacy-preserving measures, such as pseudonymisation, or encryption where anonymisation may significantly affect the purpose pursued;

(ii) measures ensuring confidentiality, integrity, availability and resilience of processing systems and services, and the ability to restore the availability and access to special category personaldata in a timely manner in the event of a physical or technical incident;

(iii) processes for regularly testing, assessing and evaluating the effectiveness of technical and organisational measures in order to ensure the security of the processing;

(iv) measures for user identification, authorisation, protection of data during transmission, protection of data during storage, ensuring physical security of locations at which personal data are processed, internal IT and IT security governance and management, certification/assurance of processes and products;

(v) measures for ensuring data minimisation, data quality, limited data retention, and data portability and ensuring erasure.

Or. en

Amendment 113

Proposal for a regulation

Article 10 – paragraph 6 a (new)

Text proposed by the Commission

Amendment

6a. Providers and user may comply with the obligations set out in this Article through the use of third-parties that offer certified compliance services including verification of data governance, data set integrity, and datatraining, validation and testing practices.

Or. en

Amendment 114

Proposal for a regulation

Article 11 – paragraph 1 – introductory part

Text proposed by the Commission

Amendment

1. The technical documentation of a high-risk AI system shall be drawn up before that system is placed on the market or put into service and shall be kept up-to date.

1. The technical documentation of a high-risk AI system shall be drawn up, where possible, relevant and without compromising intellectual property rights or trade secrets, before that system is placed on the market or put into service and shall be kept up-to date.

Or. en

Amendment 115

Proposal for a regulation

Article 11 – paragraph 1 – subparagraph 1

Text proposed by the Commission

Amendment

The technical documentation shall be drawn up in such a way to demonstrate that the high-risk AI system complies with the requirements set out in this Chapter and provide national competent authorities and notified bodies with all the necessary information to assess the compliance of the AI system with those requirements. It shall contain, at a minimum, the elements set out in Annex IV.

The technical documentation shall be drawn up, where possible, relevant and without compromising intellectual property rights or trade secrets, in such a way to demonstrate that the high-risk AI system complies with the requirements set out in this Chapter and provide national competent authorities and notified bodies with all the necessary information to assess the compliance of the AI system with those requirements. It shall contain, at a minimum, the elements set out in Annex IV.

Or. en

Amendment 116

Proposal for a regulation

Article 11 – paragraph 2

Text proposed by the Commission

Amendment

2. Where a high-risk AI system related to a product, to which the legal acts listed in Annex II, section A apply, is placed on the market or put into service one single technical documentation shall be drawn up containing all the information set out in Annex IV as well as the information required under those legal acts.

2. Where a high-risk AI system related to a product, to which the legal acts listed in Annex II, section A apply, is placed on the market or put into service appropriate technical documentation shall be drawn up containing all the information set out in Annex IV as well as the information required under those legal acts.

Or. en

Amendment 117

Proposal for a regulation

Article 12 – paragraph 1

Text proposed by the Commission

Amendment

1. High-risk AI systems shall be designed and developed with capabilities enabling the automatic recording of events (‘logs’) while the high-risk AI systems is operating. Those logging capabilities shall conform to recognised standards or common specifications.

1. High-risk AI systems shall technically allow the automatic recording of events (‘logs’) over the duration of the lifecycle of the system as long as these logs are in compliance with the GDPR.

Or. en

Amendment 118

Proposal for a regulation

Article 12 – paragraph 2

Text proposed by the Commission

Amendment

2. The logging capabilities shall ensure a level of traceability of the AI system’s functioning throughout its lifecycle that is appropriate to the intended purpose of the system.

2. In order to ensure a level of traceability of the AI system’s functioning, which is appropriate to the intended purpose of the system, the logging capabilities shall enable the recording of events relevant for the identification of situations that may:

(i) result in the AI system presenting a risk within the meaning of Article 65(1); or

(ii) lead to a substantial modification that facilitate the post-market monitoring referred to in Article 61.

Or. en

Amendment 119

Proposal for a regulation

Article 12 – paragraph 3

Text proposed by the Commission

Amendment

3. In particular, logging capabilities shall enable the monitoring of the operation of the high-risk AI system with respect to the occurrence of situations that may result in the AI system presenting a risk within the meaning of Article 65(1) or lead to a substantial modification, and facilitate the post-market monitoring referred to in Article 61.

deleted

Or. en

Amendment 120

Proposal for a regulation

Article 12 – paragraph 4

Text proposed by the Commission

Amendment

4. For high-risk AI systems referred to in paragraph 1, point (a) of Annex III, the logging capabilities shall provide, at a minimum:

deleted

(a) recording of the period of each use of the system (start date and time and end date and time of each use);

(b) the reference database against which input data has been checked by the system;

(c) the input data for which the search has led to a match;

(d) the identification of the natural persons involved in the verification of the results, as referred to in Article 14 (5).

Or. en

Amendment 121

Proposal for a regulation

Article 13 – paragraph 1

Text proposed by the Commission

Amendment

1. High-risk AI systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system’s output and use it appropriately. An appropriate type and degree of transparency shall be ensured, with a view to achieving compliance with the relevant obligations of the user and of the provider set out in Chapter 3 of this Title.

1. High-risk AI systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent, with a view to achieving compliance with the relevant obligations of the user and of the provider set out in Chapter 3of this Title. Transparency shall thereby mean that, to the extent that can be reasonably expected and is feasible in technical terms at the time when the AI system is placed on the market, the AI system is interpretable to the provider, in that the provider can understand the rationale of decisions taken by the high risk AI system, while enabling the user to understand and use the AI system appropriately, by generally knowing how the AI system works and what data it ingests.

Or. en

Amendment 122

Proposal for a regulation

Article 13 – paragraph 2

Text proposed by the Commission

Amendment

2. High-risk AI systems shall be accompanied by instructions for use in an appropriate digital format or otherwise that include concise, complete, correct and clear information that is relevant, accessible and comprehensible to users.

2. High-risk AI systems shall be accompanied by instructions for use in an appropriate digital format or otherwise that include concise, correct and clear information that helps supporting informed decision-making by users and is relevant, accessible and comprehensible to users

Or. en

Amendment 123

Proposal for a regulation

Article 13 – paragraph 3 – introductory part

Text proposed by the Commission

Amendment

3. The information referred to in paragraph 2 shall specify:

3. To the extent necessary to achieve the outcomes referred to in paragraph 1, information referred to in paragraph 2 shall specify:

Or. en

Amendment 124

Proposal for a regulation

Article 13 – paragraph 3 – point a

Text proposed by the Commission

Amendment

(a) the identity and the contact details of the provider and, where applicable, of its authorised representative;

(a) the identity and the contact details of the entity or entities responsible for the AI system’s compliance with this Act, regardless of their qualification as provider or user and, where applicable, of their authorised representatives;

Or. en

Amendment 125

Proposal for a regulation

Article 13 – paragraph 3 – point b – point ii

Text proposed by the Commission

Amendment

(ii) the level of accuracy, robustness and cybersecurity referred to in Article 15 against which the high-risk AI system has been tested and validated and which can be expected, and any known and foreseeable circumstances that may have an impact on that expected level of accuracy, robustness and cybersecurity;

(ii) the level of accuracy, robustness and cybersecurity referred to in Article 15 against which the high-risk AI system has been tested and validated and which can be expected;

Or. en

Amendment 126

Proposal for a regulation

Article 13 – paragraph 3 – point b – point iii

Text proposed by the Commission

Amendment

(iii) any known or foreseeable circumstance, related to the use of the high-risk AI system in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, which may lead to risks to the health and safety or fundamental rights;

deleted

Or. en

Amendment 127

Proposal for a regulation

Article 13 – paragraph 3 – point c

Text proposed by the Commission

Amendment

(c) the changes to the high-risk AI system and its performance which have been pre-determined by the provider at the moment of the initial conformity assessment, if any;

(c) the changes to the high-risk AI system;

Or. en

Amendment 128

Proposal for a regulation

Article 13 – paragraph 3 – point e

Text proposed by the Commission

Amendment

(e) the expected lifetime of the high-risk AI system and any necessary maintenance and care measures to ensure the proper functioning of that AI system, including as regards software updates.

(e) and any necessary maintenance and care measures to ensure the proper functioning of that AI system, including as regards software updates.

Or. en

Amendment 129

Proposal for a regulation

Article 13 – paragraph 3 – point e a (new)

Text proposed by the Commission

Amendment

(ea) a description of the mechanisms included within the AI system that allow users to properly collect, store and interpret the logs in accordance with Article 12(1), where relevant.

Or. en

Amendment 130

Proposal for a regulation

Article 14 – paragraph 1

Text proposed by the Commission

Amendment

1. High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which the AI system is in use.

1. Where proportionate to the risks associated with the high-risk system and where technical safeguards are not sufficient, high-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they be effectively overseen by natural persons during the period in which the AI system is in use.

Or. en

Amendment 131

Proposal for a regulation

Article 14 – paragraph 2

Text proposed by the Commission

Amendment

2. Human oversight shall aim at preventing or minimising the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular when such risks persist notwithstanding the application of other requirements set out in this Chapter.

2. Human oversight shall aim at preventing or minimising the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose, in particular when such risks persist notwithstanding the application of other requirements set out in this Chapter.

Or. en

Amendment 132

Proposal for a regulation

Article 14 – paragraph 3 – introductory part

Text proposed by the Commission

Amendment

3. Human oversight shall be ensured through either one or all of the following measures:

3. The degree of human oversight shall be adapted to the specific risks, the level of automation, and context of the AI system and shall be ensured through either one or all of the following types of measures:

Or. en

Amendment 133

Proposal for a regulation

Article 14 – paragraph 4 – introductory part

Text proposed by the Commission

Amendment

4. The measures referred to in paragraph 3 shall enable the individuals to whom human oversight is assigned to do the following, as appropriate to the circumstances:

4. For the purpose of implementing paragraphs 1 to 3, the high-risk AI system shall be provided to the user in such a way that natural persons to whom human oversight is assigned are enabled, as appropriate and proportionate to the circumstances and in accordance with industry standards:

Or. en

Amendment 134

Proposal for a regulation

Article 14 – paragraph 4 – point a

Text proposed by the Commission

Amendment

(a) fully understand the capacities and limitations of the high-risk AI system and be able to duly monitor its operation, so that signs of anomalies, dysfunctions and unexpected performance can be detected and addressed as soon as possible;

(a) to be aware of and sufficiently understand the relevant capacities and limitations of the high-risk AI system and be able to duly monitor its operation;

Or. en

Amendment 135

Proposal for a regulation

Article 14 – paragraph 4 – point b

Text proposed by the Commission

Amendment

(b) remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (‘automation bias’), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;

(b) remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (‘automation bias’);

Or. en

Amendment 136

Proposal for a regulation

Article 14 – paragraph 4 – point c

Text proposed by the Commission

Amendment

(c) be able to correctly interpret the high-risk AI system’s output, taking into account in particular the characteristics of the system and the interpretation tools and methods available;

(c) to correctly interpret the high-risk AI system’s output, taking into account for example the interpretation tools and methods available;

Or. en

Amendment 137

Proposal for a regulation

Article 14 – paragraph 4 – point e

Text proposed by the Commission

Amendment

(e) be able to intervene on the operation of the high-risk AI system or interrupt the system through a “stop” button or a similar procedure.

(e) be able to intervene on the operation of the high-risk AI system or interrupt, where reasonable and technically feasible, the system through a “stop” button or a similar procedure, except if the human interference increases the risks or would negatively impact the performance in consideration of generally acknowledged state-of-the-art.

Or. en

Amendment 138

Proposal for a regulation

Article 15 – paragraph 1

Text proposed by the Commission

Amendment

1. High-risk AI systems shall be designed and developed in such a way that they achieve, in the light of their intended purpose, an appropriate level of accuracy, robustness and cybersecurity, and perform consistently in those respects throughout their lifecycle.

1. High-risk AI systems shall be designed and developed in such a way that they achieve, in the light of their intended purpose and to the extent that can be reasonably expected and is in accordance with relevant industrial standards, an appropriate level of accuracy, reliability, robustness and cybersecurity, and the basic pillars of information security and protection, such as confidentiality, integrity and availability as well as to perform consistently in those respects throughout their lifecycle while taking their evolving nature into account.

Or. en

Amendment 139

Proposal for a regulation

Article 15 – paragraph 2

Text proposed by the Commission

Amendment

2. The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.

2. The range of expected performance, and the operational factors that affect that performance, shall be declared, where possible, in the accompanying instructions of use.

Or. en

Amendment 140

Proposal for a regulation

Article 15 – paragraph 3 – introductory part

Text proposed by the Commission

Amendment

3. High-risk AI systems shall be resilient as regards errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems.

3. High-risk systems shall be designed and developed with safety and security-by-design mechanism so that they achieve, in the light of their intended purpose, an appropriate level of cyber resilience as regards errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems.

Or. en

Amendment 141

Proposal for a regulation

Article 15 – paragraph 3 – subparagraph 1

Text proposed by the Commission

Amendment

The robustness of high-risk AI systems may be achieved through technical redundancy solutions, which may include backup or fail-safe plans.

The robustness of high-risk AI systems may be achieved through diverse technical redundancy solutions, which may include reasonably designed backup or fail-safe plans by the appropriate provider or user or as mutually agreed by the provider and the user.

Or. en

Amendment 142

Proposal for a regulation

Article 15 – paragraph 3 – subparagraph 2

Text proposed by the Commission

Amendment

High-risk AI systems that continue to learn after being placed on the market or put into service shall be developed in such a way to ensure that possibly biased outputs due to outputs used as an input for future operations (‘feedback loops’) are duly addressed with appropriate mitigation measures.

High-risk AI systems that continue to learn after being placed on the market or put into service shall be developed in such a way to ensure that possibly biased outputs that influence the input for future operations (‘feedback loops’) are duly addressed with appropriate mitigation measures.

Or. en

Amendment 143

Proposal for a regulation

Article 15 – paragraph 3 a (new)

Text proposed by the Commission

Amendment

3a. In accordance with Article 42(2), the compliance with Article 15 for high-risk AI system that have already been certified or for which a statement of conformity has been issued under a cybersecurity scheme pursuant to Regulation (EU) 2019/881 shall be assumed.

Or. en

Amendment 144

Proposal for a regulation

Article 15 – paragraph 4 – subparagraph 1

Text proposed by the Commission

Amendment

The technical solutions aimed at ensuring the cybersecurity of high-risk AI systems shall be appropriate to the relevant circumstances and the risks.

The technical solutions and organisational measures designed to uphold the cybersecurity of high-risk AI systems shall be appropriate to the relevant circumstances and the risks.

Or. en

Amendment 145

Proposal for a regulation

Article 15 – paragraph 4 – subparagraph 2

Text proposed by the Commission

Amendment

The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent and control for attacks trying to manipulate the training dataset (‘data poisoning’), inputs designed to cause the model to make a mistake (‘adversarial examples’), or model flaws.

Technical solutions may include, where appropriate, measures to prevent and control for attacks trying to manipulate the training dataset (‘data poisoning’), inputs designed to cause the model to make a mistake (‘adversarial examples’), model flaws, or exploratory attacks that may aim to extract knowledge, algorithms, trade secrets or training information from the AI.

Or. en

Amendment 146

Proposal for a regulation

Article 16 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) ensure that their high-risk AI systems are compliant with the requirements set out in Chapter 2 of this Title;

(a) ensure that their high-risk AI systems are compliant with the requirements set out in Chapter 2 of this Title, including the requirement to conduct an ethical technology assessment as part of Article 9(2)(da);

Or. en

Amendment 147

Proposal for a regulation

Article 16 – paragraph 1 – point a a (new)

Text proposed by the Commission

Amendment

(aa) indicate their name, registered trade name or registered trade mark, the address at which they can be contacted on the high-risk AI system or, where that is not possible, on its packaging or its accompanying documentation, as applicable;

Or. en

Amendment 148

Proposal for a regulation

Article 16 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) have a quality management system in place which complies with Article 17;

deleted

Or. en

Amendment 149

Proposal for a regulation

Article 16 – paragraph 1 – point c

Text proposed by the Commission

Amendment

(c) draw-up the technical documentation of the high-risk AI system;

(c) keep the documentation referred to in Article 18;

Or. en

Amendment 150

Proposal for a regulation

Article 16 – paragraph 1 – point e

Text proposed by the Commission

Amendment

(e) ensure that the high-risk AI system undergoes the relevant conformity assessment procedure, prior to its placing on the market or putting into service;

(e) carry out the relevant conformity assessment procedure, prior to its placing on the market or putting into service;

Or. en

Amendment 151

Proposal for a regulation

Article 16 – paragraph 1 – point j

Text proposed by the Commission

Amendment

(j) upon request of a national competent authority, demonstrate the conformity of the high-risk AI system with the requirements set out in Chapter 2 of this Title.

(j) upon reasoned request of a national competent authority, provide the relevant information and documentation to demonstrate the conformity of the high-risk AI system.

Or. en

Amendment 152

Proposal for a regulation

Article 17

Text proposed by the Commission

Amendment

[...]

deleted

Or. en

Justification

Requiring every provider to establish a quality management system for a high-risk system is disproportionate and unjustified. It would create huge costs and administrative burdens. At the same time, it does not have added value as most of its element are already or can at least be included in the risk assessment of Art 9. Moreover, many AI systems already fall under existing quality management system, such as those defined by ISO 9001. If the EU wants to promote innovation in AI, in particular by European SMEs and start-ups, it must not overburden them with unnecessary obligations.

Amendment 153

Proposal for a regulation

Article 18 – paragraph 1

Text proposed by the Commission

Amendment

1. Providers of high-risk AI systems shall draw up the technical documen­tation referred to in Article 11 in accordance with Annex IV.

1. The provider shall, for a period of 3 years after the AI system has been placed on the market or put into service, keep at the disposal of the national competent authorities:

(a) the technical documentation referred to in Article 11 and Annex IV;

(b) the documentation concerning the changes approved by notified bodies where applicable;

(c) the decisions and other documents issued by the notified bodies where applicable;

(d) the EU declaration of conformity referred to in Article 48.

Or. en

Amendment 154

Proposal for a regulation

Article 20 – paragraph 1

Text proposed by the Commission

Amendment

1. Providers of high-risk AI systems shall keep the logs automatically generated by their high-risk AI systems, to the extent such logs are under their control by virtue of a contractual arrangement with the user or otherwise by law. The logs shall be kept for a period that is appropriate in the light of the intended purpose of high-risk AI system and applicable legal obligations under Union or national law.

1. Providers of high-risk AI systems shall keep the logs automatically generated by their high-risk AI systems, to the extent such logs are under their control by law as well as under their factual control and to the extent that it is technically feasible. They shall keep them for a period of at least six months, unless provided otherwise in applicable Union or national law.

Or. en

Amendment 155

Proposal for a regulation

Article 21 – paragraph 1

Text proposed by the Commission

Amendment

Providers of high-risk AI systems which consider or have reason to consider that a high-risk AI system which they have placed on the market or put into service is not in conformity with this Regulation shall immediately take the necessary corrective actions to bring that system into conformity, to withdraw it or to recall it, as appropriate. They shall inform the distributors of the high-risk AI system in question and, where applicable, the authorised representative and importers accordingly.

Providers of high-risk AI systems which consider or have reason to consider that a high-risk AI system which they have placed on the market or put into service is not in conformity with this Regulation shall immediately, where applicable, investigate the causes in collaboration with the user and take the necessary corrective actions to bring that system into conformity, to withdraw it or to recall it, as appropriate. They shall inform the distributors of the high-risk AI system in question and, where applicable, the authorised representative and importers accordingly.

Or. en

Amendment 156

Proposal for a regulation

Article 22 – paragraph 1

Text proposed by the Commission

Amendment

Where the high-risk AI system presents a risk within the meaning of Article 65(1) and that risk is known to the provider of the system, that provider shall immediately inform the national competent authorities of the Member States in which it made the system available and, where applicable, the notified body that issued a certificate for the high-risk AI system, in particular of the non-compliance and of any corrective actions taken.

Where the high-risk AI system presents a risk within the meaning of Article 65(1) and that risk is known to the provider of the system, that provider shall immediately inform the market surveillance authorities of the Member States in which it made the system available and, where applicable, the notified body that issued a certificate for the high-risk AI system, in particular of the non-compliance and of any relevant corrective actions taken by the provider.

Or. en

Amendment 157

Proposal for a regulation

Article 23 – paragraph 1

Text proposed by the Commission

Amendment

Providers of high-risk AI systems shall, upon request by a national competent authority, provide that authority with all the information and documentation necessary to demonstrate the conformity of the high-risk AI system with the requirements set out in Chapter 2 of this Title, in an official Union language determined by the Member State concerned. Upon a reasoned request from a national competent authority, providers shall also give that authority access to the logs automatically generated by the high-risk AI system, to the extent such logs are under their control by virtue of a contractual arrangement with the user or otherwise by law.

Providers of high-risk AI systems shall, upon a reasoned request by a national competent authority, provide that authority with all the information and documentation necessary to demonstrate the conformity of the high-risk AI system with the requirements set out in Chapter 2 of this Title, in an official Union language determined by the Member State concerned. Upon a reasoned request from a national competent authority, providers shall also give that authority access to the logs automatically generated by the high-risk AI system, to the extent such logs are under their control by virtue of a contractual arrangement with the user or otherwise by law. Any information submitted in accordance with the provision of this article shall be considered by the national competent authority a trade secret of the company that is submitting such information and kept strictly confidential.

Or. en

Amendment 158

Proposal for a regulation

Article 23 a (new)

Text proposed by the Commission

Amendment

Article 23 a

Conditions for other persons to be subject to the obligations of a provider

1. Concerning high risk AI systems any natural or legal personshall be considered a provider for the purposes of this Regulation and shall be subject to the obligations of the provider under Article 16, in any of the following circumstances:

(a) they put their name or trademark on a high-risk AI system already placed on the market or put into service, without prejudice to contractual arrangements stipulating that the obligations are allocated otherwise;

(b) they make a substantial modification to or modify the intended purpose of a high-risk AI system already placed on the market or put into service;

(c) they modify the intended purpose of a non-high-risk AI system already placed on the market or put it to service, in a way which makes the modified system a high-risk AI system;

(d) they adapt a general purpose AI system to a specific intended purpose and placed it on the market or put it into service.

2. Where the circumstances referred to in paragraph 1 occur, the provider that initially placed the high-risk AI system on the market or put it into service shall no longer be considered a provider for the purposes of this Regulation.

The initial provider subject to the previous sentence, shall upon request and without compromising its own intellectual property rights or trade secrets, provide the new provider referred to in paragraph (1a), (1b) or (1c) with all essential, relevant and reasonably expected information that is necessary to comply with the obligations set out in this Regulation.

The provider of a general purpose AI system shall, after placing it on the market or putting it to service and without compromising its own intellectual property rights or trade secrets, provide the new provider referred to in paragraph (1d) with all essential, relevant and reasonably expected information that is necessary to comply with the obligations set out in this Regulation.

3. For high-risk AI systems that are safety components of products to which the legal acts listed in Annex II, section A apply, the manufacturer of those products shall be considered the provider of the high-risk AI system and shall be subject to the obligations referred to in Article 16 under either of the following scenarios:

(i) the high-risk AI system is placed on the market together with the product under the name or trademark of the product manufacturer; or

(ii) the high-risk AI system is put into service under the name or trademark of the product manufacturer after the product has been placed on the market.

4. Third parties involved in the sale and the supply of software including general purpose application programming interfaces (API), software tools and components, providers who develop and train AI systems on behalf of a deploying company in accordance with their instruction, or providers of network services shall not be considered providers for the purposes of this Regulation.

Or. en

Justification

AI technologies are highly complex and are often based on components and inputs from various actors worldwide. Those various actors as well as their roles within the supply chain were so far however not adequately addressed by the Commission’s proposal. Based on the Council’s latest proposal, this article in combination with Art 16, 25, 26, 27, 29 tries to clarify which actor is responsible in which situation and tries to strike the right balance, in particular when it comes to the responsibilities of producers and users.

Amendment 159

Proposal for a regulation

Article 24 – paragraph 1

Text proposed by the Commission

Amendment

Where a high-risk AI system related to products to which the legal acts listed in Annex II, section A, apply, is placed on the market or put into service together with the product manufactured in accordance with those legal acts and under the name of the product manufacturer, the manufacturer of the product shall take the responsibility of the compliance of the AI system with this Regulation and, as far as the AI system is concerned, have the same obligations imposed by the present Regulation on the provider.

deleted

Or. en

Amendment 160

Proposal for a regulation

Article 25 – paragraph 1

Text proposed by the Commission

Amendment

1. Prior to making their systems available on the Union market, where an importer cannot be identified, providers established outside the Union shall, by written mandate, appoint an authorised representative which is established in the Union.

1. Prior to making their systems available on the Union market providers established outside the Union shall, by written mandate, appoint an authorised representative which is established in the Union.

Or. en

Amendment 161

Proposal for a regulation

Article 25 – paragraph 2 – introductory part

Text proposed by the Commission

Amendment

2. The authorised representative shall perform the tasks specified in the mandate received from the provider. The mandate shall empower the authorised representative to carry out the following tasks:

2. The authorised representative shall perform the tasks specified in the mandate received from the provider. For the purpose of this Regulation, the mandate shall empower the authorised representative to carry out only the following tasks:

Or. en

Amendment 162

Proposal for a regulation

Article 25 – paragraph 2 – point a

Text proposed by the Commission

Amendment

(a) keep a copy of the EU declaration of conformity and the technical documentation at the disposal of the national competent authorities and national authorities referred to in Article 63(7);

(a) verify that the EU declaration of conformity and the technical documentation have been drawn up and that an appropriate conformity assessment procedure has been carried out by the provider;

Or. en

Amendment 163

Proposal for a regulation

Article 25 – paragraph 2 – point b a (new)

Text proposed by the Commission

Amendment

(ba) keep at the disposal of the national competent authorities and national authorities referred to in Article63(7), for a period ending 3 years after the high-risk AI system has been placed on the market or put into service, a copy of the EU declaration of conformity, the technical documentation and, if applicable, the certificate issued by the notified body;

Or. en

Amendment 164

Proposal for a regulation

Article 25 – paragraph 2 – point c

Text proposed by the Commission

Amendment

(c) cooperate with competent national authorities, upon a reasoned request, on any action the latter takes in relation to the high-risk AI system.

(c) cooperate with national competent authorities, upon a reasoned request, on any action the latter takes in relation to the high-risk AI system.

Or. en

Amendment 165

Proposal for a regulation

Article 25 – paragraph 2 – point c a (new)

Text proposed by the Commission

Amendment

(ca) comply with the registration obligations referred to in Article 51 or, if the registration is carried out by the provider itself, verify that the information referred to in point 3 of Annex VIII is correct.

The authorised representative shall terminate the mandate if it considers or has reason to consider that the provider acts contrary to its obligations under this Regulation. In such a case, it shall also immediately inform the market surveillance authority of the Member State in which it is established, as well as, where applicable, the relevant notified body, about the termination of the mandate and the reasons thereof.

Or. en

Amendment 166

Proposal for a regulation

Article 26 – paragraph 1 – introductory part

Text proposed by the Commission

Amendment

1. Before placing a high-risk AI system on the market, importers of such system shall ensure that:

1. Before placing a high-risk AI system on the market, importers of such system shall ensure that such a system is in conformity with this Regulation by verifying that:

Or. en

Amendment 167

Proposal for a regulation

Article 26 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) the appropriate conformity assessment procedure has been carried out by the provider of that AI system

(a) the relevant conformity assessment procedure referred to in Article 43 has been carried out by the provider of that AI system;

Or. en

Amendment 168

Proposal for a regulation

Article 26 – paragraph 1 – point c a (new)

Text proposed by the Commission

Amendment

(ca) the authorised representative referred to in Article 25 has been established by the provider.

Or. en

Amendment 169

Proposal for a regulation

Article 26 – paragraph 2

Text proposed by the Commission

Amendment

2. Where an importer considers or has reason to consider that a high-risk AI system is not in conformity with this Regulation, it shall not place that system on the market until that AI system has been brought into conformity. Where the high-risk AI system presents a risk within the meaning of Article 65(1), the importer shall inform the provider of the AI system and the market surveillance authorities to that effect.

2. Where an importer considers or has reason to consider that a high-risk AI system is not in conformity with this Regulation, or is falsified, or accompanied by falsified documentation, it shall not place that system on the market until that AI system has been brought into conformity. Where the high-risk AI system presents a risk within the meaning of Article 65(1), the importer shall inform the provider of the AI system and the market surveillance authorities to that effect.

Or. en

Amendment 170

Proposal for a regulation

Article 26 – paragraph 4 a (new)

Text proposed by the Commission

Amendment

4a. Importers shall keep, for a period ending 3 years after the AI system has been placed on the market or put into service, a copy of the certificate issued by the notified body, where applicable, of the instructions for use and of the EU declaration of conformity.

Or. en

Amendment 171

Proposal for a regulation

Article 26 – paragraph 5

Text proposed by the Commission

Amendment

5. Importers shall provide national competent authorities, upon a reasoned request, with all necessary information and documentation to demonstrate the conformity of a high-risk AI system with the requirements set out in Chapter 2 of this Title in a language which can be easily understood by that national competent authority, including access to the logs automatically generated by the high-risk AI system to the extent such logs are under the control of the provider by virtue of a contractual arrangement with the user or otherwise by law. They shall also cooperate with those authorities on any action national competent authority takes in relation to that system.

5. Where no authorized representative has been established, importers shall provide national competent authorities, upon a reasoned request, with all necessary information and documentation to demonstrate the conformity of a high-risk AI system with the requirements set out in Chapter 2 of this Title in a language which can be easily understood by that national competent authority. To this purpose they shall also ensure that the technical documentation can be made available to those authorities.

Or. en

Amendment 172

Proposal for a regulation

Article 26 – paragraph 5 a (new)

Text proposed by the Commission

Amendment

5a. Importers shall cooperate with national competent authorities on any action those authorities take in relation to an AI system.

Or. en

Amendment 173

Proposal for a regulation

Article 27 – paragraph 1

Text proposed by the Commission

Amendment

1. Before making a high-risk AI system available on the market, distributors shall verify that the high-risk AI system bears the required CE conformity marking, that it is accompanied by the required documentation and instruction of use, and that the provider and the importer of the system, as applicable, have complied with the obligations set out in this Regulation.

1. Before making a high-risk AI system available on the market, distributors shall verify that the high-risk AI system bears the required CE conformity marking, that it is accompanied by the required documentation and instruction of use, and that the provider and the importer of the system, as applicable, have complied with their obligations set out in Article 16 and in Article 26(3) respectively.

Or. en

Amendment 174

Proposal for a regulation

Article 27 – paragraph 2

Text proposed by the Commission

Amendment

2. Where a distributor considers or has reason to consider that a high-risk AI system is not in conformity with the requirements set out in Chapter 2 of this Title, it shall not make the high-risk AI system available on the market until that system has been brought into conformity with those requirements. Furthermore, where the system presents a risk within the meaning of Article 65(1), the distributor shall inform the provider or the importer of the system, as applicable, to that effect.

2. Where a distributor considers or has reason to consider, on the basis of the information in its possession, that a high-risk AI system is not in conformity with the requirements set out in Chapter 2 of this Title, it shall not make the high-risk AI system available on the market until that system has been brought into conformity with those requirements. Furthermore, where the system presents a risk within the meaning of Article 65(1), the distributor shall inform the provider or the importer of the system, as applicable, to that effect, and the market surveillance authorities.

Or. en

Amendment 175

Proposal for a regulation

Article 27 – paragraph 4

Text proposed by the Commission

Amendment

4. A distributor that considers or has reason to consider that a high-risk AI system which it has made available on the market is not in conformity with the requirements set out in Chapter 2 of this Title shall take the corrective actions necessary to bring that system into conformity with those requirements, to withdraw it or recall it or shall ensure that the provider, the importer or any relevant operator, as appropriate, takes those corrective actions. Where the high-risk AI system presents a risk within the meaning of Article 65(1), the distributor shall immediately inform the national competent authorities of the Member States in which it has made the product available to that effect, giving details, in particular, of the non-compliance and of any corrective actions taken.

4. A distributor that considers or has reason to consider, on the basis of the information in its possession, that a high-risk AI system which it has made available on the market is not in conformity with the requirements set out in Chapter 2 of this Title shall take the corrective actions necessary to bring that system into conformity with those requirements, to withdraw it or recall it or shall ensure that the provider, the importer or any relevant operator, as appropriate, takes those corrective actions. Where the high-risk AI system presents a risk within the meaning of Article 65(1),the distributor shall immediately inform the provider or the importer of the system and inform the national competent authorities of the Member States in which it has made the product available to that effect, giving details, in particular, of the non-compliance and of any corrective actions taken.

Or. en

Amendment 176

Proposal for a regulation

Article 27 – paragraph 5

Text proposed by the Commission

Amendment

5. Upon a reasoned request from a national competent authority, distributors of high-risk AI systems shall provide that authority with all the information and documentation necessary to demonstrate the conformity of a high-risk system with the requirements set out in Chapter 2 of this Title. Distributors shall also cooperate with that national competent authority on any action taken by that authority.

5. Upon a reasoned request from a national competent authority, distributors of high-risk AI systems shall provide that authority with all the information and documentation regarding its activities as described in paragraphs 1 to 4.

Or. en

Amendment 177

Proposal for a regulation

Article 27 – paragraph 5 a (new)

Text proposed by the Commission

Amendment

5a. Importers shall cooperate with national competent authorities on any action those authorities take in relation to an AI system.

Or. en

Amendment 178

Proposal for a regulation

Article 28

Text proposed by the Commission

Amendment

Article 28

deleted

Obligations of distributors, importers, users or any other third-party

1. Any distributor, importer, user or other third-party shall be considered a provider for the purposes of this Regulation and shall be subject to the obligations of the provider under Article 16, in any of the following circumstances:

(a) they place on the market or put into service a high-risk AI system under their name or trademark;

(b) they modify the intended purpose of a high-risk AI system already placed on the market or put into service;

(c) they make a substantial modification to the high-risk AI system.

2. Where the circumstances referred to in paragraph 1, point (b) or (c), occur, the provider that initially placed the high-risk AI system on the market or put it into service shall no longer be considered a provider for the purposes of this Regulation.

Or. en

Amendment 179

Proposal for a regulation

Article 29 – paragraph 1

Text proposed by the Commission

Amendment

1. Users of high-risk AI systems shall use such systems in accordance with the instructions of use accompanying the systems, pursuant to paragraphs 2 and 5.

1. Users of high-risk AI systems shall use such systems and implement human oversight in accordance with the instructions of use accompanying the systems, pursuant to paragraphs 2 and 5 of this Article.

Or. en

Amendment 180

Proposal for a regulation

Article 29 – paragraph 1 a (new)

Text proposed by the Commission

Amendment

1a. To the extent the user exercise control over the high-risk AI system, that user shall only assign human oversight to natural persons who have the necessary competence, training and authority.

Or. en

Amendment 181

Proposal for a regulation

Article 29 – paragraph 2

Text proposed by the Commission

Amendment

2. The obligations in paragraph 1 are without prejudice to other user obligations under Union or national law and to the user’s discretion in organising its own resources and activities for the purpose of implementing the human oversight measures indicated by the provider.

2. The obligations in paragraph 1 and 1a are without prejudice to other user obligations under Union or national law and to the user’s discretion in organising its own resources and activities for the purpose of implementing the human oversight measures indicated by the provider.

Or. en

Amendment 182

Proposal for a regulation

Article 29 – paragraph 3

Text proposed by the Commission

Amendment

3. Without prejudice to paragraph 1, to the extent the user exercises control over the input data, that user shall ensure that input data is relevant in view of the intended purpose of the high-risk AI system.

3. Without prejudice to paragraph 1, to the extent the user exercises control over the input data, that user shall ensure that input data is relevant and representative in view of the intended purpose of the high-risk AI system. To the extent the user exercise control over the high-risk AI system, that user shall also ensure that relevant and appropriate robustness and cybersecurity measures are in place and are regularly adjusted or updated.

Or. en

Amendment 183

Proposal for a regulation

Article 29 – paragraph 4 – introductory part

Text proposed by the Commission

Amendment

4. Users shall monitor the operation of the high-risk AI system on the basis of the instructions of use. When they have reasons to consider that the use in accordance with the instructions of use may result in the AI system presenting a risk within the meaning of Article 65(1) they shall inform the provider or distributor and suspend the use of the system. They shall also inform the provider or distributor when they have identified any serious incident or any malfunctioning within the meaning of Article 62 and interrupt the use of the AI system. In case the user is not able to reach the provider, Article 62 shall apply mutatis mutandis.

4. Users shall monitor the operation of the high-risk AI system on the basis of the instructions of use and, when relevant, inform providers in accordance with Article 61. To the extent the user exercise control over the high-risk AI system, users shall also perform risk assessments in line with Article 9 but limited to the potential adverse effects of using the high-risk AI system, the respective mitigation measures, and the trustworthy technology assessment. When they have reasons to consider that the use in accordance with the instructions of use may result in the AI system presenting a risk within the meaning of Article 65(1) or a risk to the trustworthy technology assessment set out Article 4(a) they shall inform the provider or distributor and suspend the use of the system. They shall also inform the provider or distributor when they have identified any serious incident and interrupt the use of the AI system. In case the user is not able to reach the provider, Article 62 shall apply mutatis mutandis.

Or. en

Amendment 184

Proposal for a regulation

Article 29 – paragraph 5 – introductory part

Text proposed by the Commission

Amendment

5. Users of high-risk AI systems shall keep the logs automatically generated by that high-risk AI system, to the extent such logs are under their control. The logs shall be kept for a period that is appropriate in the light of the intended purpose of the high-risk AI system and applicable legal obligations under Union or national law.

5. Users of high-risk AI systems shall keep the logs automatically generated by that high-risk AI system, to the extent such logs are under their control. They shall keep them for a period of at least six months, unless provided otherwise in applicable Union or national law.

Or. en

Amendment 185

Proposal for a regulation

Article 29 – paragraph 6

Text proposed by the Commission

Amendment

6. Users of high-risk AI systems shall use the information provided under Article 13 to comply with their obligation to carry out a data protection impact assessment under Article 35 of Regulation (EU) 2016/679 or Article 27 of Directive (EU) 2016/680, where applicable.

6. Users of high-risk AI systems shall use the information provided under Article 13 to comply with their obligation to carry out a data protection impact assessment under Article 35 of Regulation (EU) 2016/679 or Article 27 of Directive (EU) 2016/680 and may revert in part to those data protection impact assessments for fulfilling the obligations set out in this Article.

Or. en

Amendment 186

Proposal for a regulation

Article 29 – paragraph 6 a (new)

Text proposed by the Commission

Amendment

6a. Where a user of a high risk AI system is obliged pursuant to Regulation (EU) 2016/679 to provide information regarding the use of automated decision making procedures, the user shall not be obliged to provide information on how the AI system reached a specific result. When fulfilling the information obligations under Regulation (EU) 2016/679, the user shall not be obliged to provide information beyond the information he or she received from the provider under Article 13 of this Regulation.

Or. en

Amendment 187

Proposal for a regulation

Article 29 – paragraph 6 b (new)

Text proposed by the Commission

Amendment

6b. The provider shall be obliged to cooperate closely with the user and in particular provide the user with the necessary information to allow the fulfilment of the obligations set out in this Article.

Or. en

Amendment 188

Proposal for a regulation

Article 29 – paragraph 6 c (new)

Text proposed by the Commission

Amendment

6c. Users shall cooperate with national competent authorities on any action those authorities take in relation to an AI system.

Or. en

Amendment 189

Proposal for a regulation

Article 39 – paragraph 1

Text proposed by the Commission

Amendment

Conformity assessment bodies established under the law of a third country with which the Union has concluded an agreement may be authorised to carry out the activities of notified Bodies under this Regulation.

1. In line with EU commitments under the World Trade Organization (WTO) Agreement on Technical Barriers to Trade (TBT), the Commission shall endeavour to maximise the acceptance of test results produced by competent conformity assessment bodies, independent of the territory in which they may be established, where necessary to demonstrate conformance with applicable requirements of the Regulation.

2. Conformity assessment bodies established under the law of a third country may carry out the activities of Notified Bodies under this regulation where they have been accredited as competent by an accreditation body, whether established in the territory of the EU or a third country, that is a signatory of an international accreditation or conformity assessment scheme based on rigorous peer-review processes, such as the International Laboratory Accreditation Collaboration (ILAC) Mutual Recognition Arrangement (MRA) and International Accreditation Forum (IAF) Multilateral Recognition Arrangement (MLA).

3. In addition, where conformity assessment bodies established under the law of a third country have not been accredited by signatory bodies of such international accreditation or conformity assessment schemes, third-country conformity assessment bodies may carry out the activities of Notified Bodies where international mutual recognition arrangements, conformity assessment protocols, or other agreements exist between the EU and the country in which the conformity assessment body is established.

Or. en

Amendment 190

Proposal for a regulation

Article 40 – paragraph 1

Text proposed by the Commission

Amendment

High-risk AI systems which are in conformity with harmonised standards or parts thereof the references of which have been published in the Official Journal of the European Union shall be presumed to be in conformity with the requirements set out in Chapter 2 of this Title, to the extent those standards cover those requirements.

1. High-risk AI systems which are in conformity with harmonised standards or parts thereof the references of which have been published in the Official Journal of the European Union shall be presumed to be in conformity with the requirements set out in Chapter 2 of this Title, to the extent those standards cover those requirements.

2. When issuing a standardisation request to European standardisation organisations in accordance with Article 10 of Regulation (EU) 1025/2012, the Commission shall specify that standards are coherent, easy to implement and drafted in such a way that they aim to fulfil in particular the following objectives:

a) ensure that AI systems placed on the market or put into service in the Union are safe and respect Union values and strengthen the Union's digital sovereignty;

b) take into account the concept of trustworthy AI set out in Article 4(a);

c) promote investment and innovation in AI, as well as competitiveness and growth of the Union market; d) enhance multistakeholder governance, representative of allrelevant European stakeholders (e.g. industry, SMEs, civil society, researchers);

d) contribute to strengthening global cooperation on standardisation in the field of AI that is consistent with Union values and interests.

The Commission shall request the European standardisation organisations to provide evidence of their best efforts to fulfil the above objectives.

3. The Commission shall issue standardisation requests covering all essential requirements of the Regulation in accordance with Article 10 of Regulation (EU) No 1025/2012 no later than 6 months after the date of entry into force of the Regulation.

Or. en

Amendment 191

Proposal for a regulation

Article 41 – paragraph 1

Text proposed by the Commission

Amendment

1. Where harmonised standards referred to in Article 40 do not exist or where the Commission considers that the relevant harmonised standards are insufficient or that there is a need to address specific safety or fundamental right concerns, the Commission may, by means of implementing acts, adopt common specifications in respect of the requirements set out in Chapter 2 of this Title. Those implementing acts shall be adopted in accordance with the examination procedure referred to in Article 74(2).

1. Where the Commission considers that there is a need to address serious fundamental right concerns and provided that relevant harmonised standards referred to in Article 40 are insufficient, do not exist or are not foreseeable to be developed soon, the Commission may, by means of implementing acts, adopt common specifications in respect of the requirements set out in Chapter 2 of this Title. Those implementing acts shall be adopted in accordance with the examination procedure referred to in Article 74(2).

Or. en

Amendment 192

Proposal for a regulation

Article 41 – paragraph 1 a (new)

Text proposed by the Commission

Amendment

1a. When deciding to draft and adopt common specifications, the Commission shall consult the Board, European standardisation organisations as well as the relevant stakeholders, and duly justify why it decided not to use harmonised standards. The abovementioned organisations shall be regularly consulted as the Commission drafts the common specifications.

Or. en

Amendment 193

Proposal for a regulation

Article 42 – paragraph 1

Text proposed by the Commission

Amendment

1. Taking into account their intended purpose, high-risk AI systems that have been trained and tested on data concerning the specific geographical, behavioural and functional setting within which they are intended to be used shall be presumed to be in compliance with the requirement set out in Article 10(4).

1. High-risk AI systems that have been trained and tested on data reflecting the specific geographical, behavioural or functional setting within which they are intended to be used shall be presumed to be in compliance with the respective requirements set out in Article 10(4).

Or. en

Amendment 194

Proposal for a regulation

Article 42 – paragraph 2

Text proposed by the Commission

Amendment

2. High-risk AI systems that have been certified or for which a statement of conformity has been issued under a cybersecurity scheme pursuant to Regulation (EU) 2019/881 of the European Parliament and of the Council63 and the references of which have been published in the Official Journal of the European Union shall be presumed to be in compliance with the cybersecurity requirements set out in Article 15 of this Regulation in so far as the cybersecurity certificate or statement of conformity or parts thereof cover those requirements.

2. High-risk AI systems that have been certified or for which a statement of conformity has been issued under a cybersecurity scheme Regulation (EU) 2019/881 of the European Parliament and of the Council or pursuant to other harmonization legislation in the field of security of network and information systems and electronic communications networks and services and the references of which have been published in the Official Journal of the European Union shall be presumed to be in compliance with the cybersecurity requirements set out in Article 15 of this Regulation in so far as the cybersecurity certificate or statement of conformity or parts thereof cover those requirements.

__________________

__________________

63 Regulation (EU) 2019/881 of the European Parliament and of the Council of 17 April 2019 on ENISA (the European Union Agency for Cybersecurity) and on information and communications technology cybersecurity certification and repealing Regulation (EU) No 526/2013 (Cybersecurity Act) (OJ L 151, 7.6.2019, p. 1).

63 Regulation (EU) 2019/881 of the European Parliament and of the Council of 17 April 2019 on ENISA (the European Union Agency for Cybersecurity) and on information and communications technology cybersecurity certification and repealing Regulation (EU) No 526/2013 (Cybersecurity Act) (OJ L 151, 7.6.2019, p. 1).

Or. en

Amendment 195

Proposal for a regulation

Article 43 – paragraph 1 – introductory part

Text proposed by the Commission

Amendment

1. For high-risk AI systems listed in point 1 of Annex III, where, in demonstrating the compliance of a high-risk AI system with the requirements set out in Chapter 2 of this Title, the provider has applied harmonised standards referred to in Article 40, or, where applicable, common specifications referred to in Article 41, the provider shall follow one of the following procedures:

1. For high-risk AI systems listed in point 1 of Annex III, where, in demonstrating the compliance of a high-risk AI system with the requirements set out in Chapter 2 of this Title, the provider has applied harmonised standards referred to in Article 40, or, where applicable, common specifications referred to in Article 41, the provider shall opt for one of the following procedures:

Or. en

Amendment 196

Proposal for a regulation

Article 43 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) the conformity assessment procedure based on internal control referred to in Annex VI;

(a) the conformity assessment procedure based on internal control referred to in Annex VI; or

Or. en

Amendment 197

Proposal for a regulation

Article 43 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) the conformity assessment procedure based on assessment of the quality management system and assessment of the technical documentation, with the involvement of a notified body, referred to in Annex VII.

(b) the conformity assessment procedure based on assessment of the technical documentation, with the involvement of a notified body, referred to in Annex VII.

Or. en

Amendment 198

Proposal for a regulation

Article 43 – paragraph 4 – introductory part

Text proposed by the Commission

Amendment

4. High-risk AI systems shall undergo a new conformity assessment procedure whenever they are substantially modified, regardless of whether the modified system is intended to be further distributed or continues to be used by the current user.

4. High-risk AI systems that have already been subject to a conformity assessment procedure shall undergo a new conformity assessment procedure whenever they are substantially modified, regardless of whether the modified system is intended to be further distributed or continues to be used by the current user.

Or. en

Amendment 199

Proposal for a regulation

Article 43 – paragraph 4 – subparagraph 1

Text proposed by the Commission

Amendment

For high-risk AI systems that continue to learn after being placed on the market or put into service, changes to the high-risk AI system and its performance that have been pre-determined by the provider at the moment of the initial conformity assessment and are part of the information contained in the technical documentation referred to in point 2(f) of Annex IV, shall not constitute a substantial modification.

For high-risk AI systems that continue to learn after being placed on the market or put into service, changes to the high-risk AI system and its performance that have been pre-determined by the provider at the moment of the initial conformity assessment and are part of the information contained in the technical documentation referred to in point 2(f) of Annex IV, shall not constitute a substantial modification. The same should apply to updates of the AI system for security reasons in general and to protect against evolving threats of manipulation of the system.

This paragraph only applies if the Member State has established a legal framework, which allows the provider of a high risk AI system, which autonomously make substantial modifications to itself, to regularly perform an automated real-time conformity assessment procedure.

Or. en

Amendment 200

Proposal for a regulation

Article 43 – paragraph 4 a (new)

Text proposed by the Commission

Amendment

4a. Any provider may voluntarily apply for a third-party conformity assessment regardless of the risk level of their AI system.

Or. en

Amendment 201

Proposal for a regulation

Article 43 – paragraph 5

Text proposed by the Commission

Amendment

5. The Commission is empowered to adopt delegated acts in accordance with Article 73 for the purpose of updating Annexes VI and Annex VII in order to introduce elements of the conformity assessment procedures that become necessary in light of technical progress.

5. After consulting the AI Board referred to in Article 56 and after providing substantial evidence, followed by thorough consultation and the involvement of the affected stakeholders, the Commission is empowered to adopt delegated acts in accordance with Article 73 for the purpose of updating Annexes VI and Annex VII in order to introduce elements of the conformity assessment procedures that become necessary in light of technical progress.

Or. en

Amendment 202

Proposal for a regulation

Article 43 – paragraph 6

Text proposed by the Commission

Amendment

6. The Commission is empowered to adopt delegated acts to amend paragraphs 1 and 2 in order to subject high-risk AI systems referred to in points 2 to 8 of Annex III to the conformity assessment procedure referred to in Annex VII or parts thereof. The Commission shall adopt such delegated acts taking into account the effectiveness of the conformity assessment procedure based on internal control referred to in Annex VI in preventing or minimizing the risks to health and safety and protection of fundamental rights posed by such systems as well as the availability of adequate capacities and resources among notified bodies.

6. After consulting the AI Board referred to in Article 56 and after providing substantial evidence, followed by thorough consultation and the involvement of the affected stakeholders, the Commission is empowered to adopt delegated acts to amend paragraphs 1 and 2 in order to subject high-risk AI systems referred to in points 2 to 8 of Annex III to the conformity assessment procedure referred to in Annex VII or parts thereof. The Commission shall adopt such delegated acts taking into account the effectiveness of the conformity assessment procedure based on internal control referred to in Annex VI in preventing or minimizing the risks to health and safety and protection of fundamental rights posed by such systems as well as the availability of adequate capacities and resources among notified bodies.

Or. en

Amendment 203

Proposal for a regulation

Article 50

Text proposed by the Commission

Amendment

Article 50

deleted

Document retention

The provider shall, for a period ending 10 years after the AI system has been placed on the market or put into service, keep at the disposal of the national competent authorities:

(a) the technical documentation referred to in Article 11;

(b) the documentation concerning the quality management system referred to Article 17;

(c) the documentation concerning the changes approved by notified bodies where applicable;

(d) the decisions and other documents issued by the notified bodies where applicable;

(e) the EU declaration of conformity referred to in Article 48.

Or. en

Amendment 204

Proposal for a regulation

Article 51 – paragraph 1

Text proposed by the Commission

Amendment

Before placing on the market or putting into service a high-risk AI system referred to in Article 6(2), the provider or, where applicable, the authorised representative shall register that system in the EU database referred to in Article 60.

Before placing on the market or putting into service a high-risk AI system referred to in Article 6(2) and (2a), the provider or, where applicable, the authorised representative shall register that system in the EU database referred to in Article 60.

Or. en

Amendment 205

Proposal for a regulation

Article 52 – paragraph 1

Text proposed by the Commission

Amendment

1. Providers shall ensure that AI systems intended to interact with natural persons are designed and developed in such a way that natural persons are informed that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use. This obligation shall not apply to AI systems authorised by law to detect, prevent, investigate and prosecute criminal offences, unless those systems are available for the public to report a criminal offence.

1. Providers shall ensure that AI systems intended to interact with natural persons are designed and developed in such away that the provider itself or the user can inform the natural person exposed to an AI system that they are interacting with an AI system, unless this is obvious from the circumstances and the context of use. Where relevant, this information shall also include which functions are AI enabled, if there is human oversight and who is responsible for the decision-making process. This obligation shall not apply to AI systems authorised by law to detect, prevent, investigate and prosecute criminal offences, unless those systems are available for the public to report a criminal offence.

Or. en

Amendment 206

Proposal for a regulation

Article 52 – paragraph 3 – introductory part

Text proposed by the Commission

Amendment

3. Users of an AI system that generates or manipulates image, audio or video content that appreciably resembles existing persons, objects, places or other entities or events and would falsely appear to a person to be authentic or truthful (‘deep fake’), shall disclose that the content has been artificially generated or manipulated.

3. Users of an AI system that generates or manipulates audio or visual content that would falsely appear to be authentic or truthful and which features depictions of people appearing to say or do things they did not say or do, without their consent (‘deep fake’), shall disclose that the content has been artificially generated or manipulated. Disclosure shall mean labelling the content in a way that informs that the content is inauthentic and that is clearly visible for the recipient of that content. To label the content, users shall take into account the generally acknowledged state of the art and relevant harmonised standards and specifications.

Or. en

Amendment 207

Proposal for a regulation

Article 52 – paragraph 3 – subparagraph 1

Text proposed by the Commission

Amendment

However, the first subparagraph shall not apply where the use is authorised by law to detect, prevent, investigate and prosecute criminal offences or it is necessary for the exercise of the right to freedom of expression and the right to freedom of the arts and sciences guaranteed in the Charter of Fundamental Rights of the EU, and subject to appropriate safeguards for the rights and freedoms of third parties.

However, the first subparagraph shall not apply where the use of an AI system that generates or manipulates audio or visual content is authorized by law to detect, prevent, investigate and prosecute criminal offences or it is necessary for the exercise of the right to freedom of expression and the right to freedom of the arts and sciences guaranteed in the Charter of Fundamental Rights of the EU, and subject to appropriate safeguards for the rights and freedoms of third parties.

Or. en

Amendment 208

Proposal for a regulation

Article 52 – paragraph 3 a (new)

Text proposed by the Commission

Amendment

3a. The information referred to in paragraphs 1 to 3 shall be provided to natural persons in a clear and visible manner at the latest at the time of the first interaction or exposure. The mechanism shall, where relevant and appropriate, be completed with intervention or flagging procedures for the exposed natural person.

Or. en

Amendment 209

Proposal for a regulation

Article 53 – paragraph 1

Text proposed by the Commission

Amendment

1. AI regulatory sandboxes established by one or more Member States competent authorities or the European Data Protection Supervisor shall provide a controlled environment that facilitates the development, testing and validation of innovative AI systems for a limited time before their placement on the market or putting into service pursuant to a specific plan. This shall take place under the direct supervision and guidance by the competent authorities with a view to ensuring compliance with the requirements of this Regulation and, where relevant, other Union and Member States legislation supervised within the sandbox.

1. The competent authorities of the Member States shall establish physical and digital AI regulatory sandboxes six months prior to the entry into application of this Regulation based on well-established criteria that provide a controlled environment that facilitates the development, testing and validation of innovative AI systems for a limited time before their placement on the market or putting into service pursuant to a specific plan. This shall take place under the direct supervision and guidance by the respective national competent authorities with a view to identify risks to health and safety and fundamental rights, test mitigation measures for identified risks, demonstrate prevention of these risks and otherwise ensuring compliance with the requirements of this Regulation and, where relevant, other Union and Member States legislation supervised within the sandbox.

Or. en

Amendment 210

Proposal for a regulation

Article 53 – paragraph 1 a (new)

Text proposed by the Commission

Amendment

1a. This article shall also apply to AI systems for which full compliance with the requirements of Title III Chapter 2 requires an initial phase of placing the systems on the market or putting them into service and using the experiences gained in such initial phase to further develop the AI system so as to fully fulfil the requirements of Title III Chapter 2.

Or. en

Amendment 211

Proposal for a regulation

Article 53 – paragraph 5

Text proposed by the Commission

Amendment

5. Member States’ competent authorities that have established AI regulatory sandboxes shall coordinate their activities and cooperate within the framework of the European Artificial Intelligence Board. They shall submit annual reports to the Board and the Commission on the results from the implementation of those scheme, including good practices, lessons learnt and recommendations on their setup and, where relevant, on the application of this Regulation and other Union legislation supervised within the sandbox.

5. Member States’ competent authorities shall establish AI regulatory sandboxes as much as possible through national and regional initiatives, in particular through European digital innovation hubs, and closely coordinate their activities as well as cooperate within the framework of the European Artificial Intelligence Board. They shall submit annual reports to the Board and the Commission on the results from the implementation of those schemes, including good practices, lessons learnt and recommendations on their setup and, where relevant, on the application of this Regulation and other Unionlegislation supervised within the sandbox. These annual reports or abstracts shall be made available to the public, online, in order to further enable innovation within the Union.

Or. en

Amendment 212

Proposal for a regulation

Article 53 – paragraph 6

Text proposed by the Commission

Amendment

6. The modalities and the conditions of the operation of the AI regulatory sandboxes, including the eligibility criteria and the procedure for the application, selection, participation and exiting from the sandbox, and the rights and obligations of the participants shall be set out in implementing acts. Those implementing acts shall be adopted in accordance with the examination procedure referred to in Article 74(2).

6. The modalities and the conditions of the operation of the AI regulatory sandboxes, including the eligibility criteria and the procedure for the application, selection, participation and exiting from the sandbox, and the rights and obligations of the participants shall be set out in implementing acts in accordance with the Council’s communication(11/2020) and in strong cooperation with relevant stakeholders. Those implementing acts shall be adopted in accordance with the examination procedure referred to in Article 74(2). Member States shall anticipate demand for using the sandboxes and scale the sandboxes’ capacity accordingly, to ensure that as many economic operators as possible can access them.

Or. en

Amendment 213

Proposal for a regulation

Article 54 – paragraph 1 a (new)

Text proposed by the Commission

Amendment

1a. Provided that the conditions of paragraph 1 are met, personal data processed for developing and testing innovative AI systems in the sandbox shall be considered compatible for the purposes of Article 6(4) GDPR.

Or. en

Amendment 214

Proposal for a regulation

Article 55 – title

Text proposed by the Commission

Amendment

Measures for small-scale providers and users

Measures for providers and users that are SMEs or start ups

Or. en

Amendment 215

Proposal for a regulation

Article 55 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) provide small-scale providers and start-ups with priority access to the AI regulatory sandboxes to the extent that they fulfil the eligibility conditions;

(a) provide SMEs and start-ups with priority access to and make AI regulatory sandboxes reusable as well as affordable to the extent that SMEs and start-ups fulfil the eligibility conditions;

Or. en

Amendment 216

Proposal for a regulation

Article 55 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) organise specific awareness raising activities about the application of this Regulation tailored to the needs of the small-scale providers and users;

(b) organise specific awareness raising activities about the application of this Regulation tailored to the needs of the SMEs and start ups;

Or. en

Amendment 217

Proposal for a regulation

Article 55 – paragraph 1 – point c

Text proposed by the Commission

Amendment

(c) where appropriate, establish a dedicated channel for communication with small-scale providers and user and other innovators to provide guidance and respond to queries about the implementation of this Regulation.

(c) where appropriate, establish a dedicated channel for communication with SMEs and startups and other innovators to provide guidance and respond to queries about the implementation of this Regulation;

Or. en

Amendment 218

Proposal for a regulation

Article 55 – paragraph 1 – point c a (new)

Text proposed by the Commission

Amendment

(ca) consult representative organisations of SMEs and start ups and involve them in the development of relevant standards;

Or. en

Amendment 219

Proposal for a regulation

Article 55 – paragraph 1 – point c b (new)

Text proposed by the Commission

Amendment

(cb) create development paths and services for SMEs and start ups, ensuring that government support is provided at all stages of their development, in particular by promoting digital tools and developing AI transition plans;

Or. en

Amendment 220

Proposal for a regulation

Article 55 – paragraph 1 – point c c (new)

Text proposed by the Commission

Amendment

(cc) promote industry best practices and responsible approaches to AI development and use self-regulatory commitments as a criterion for public procurement projects or as a factor that allows more opportunities to use and share data responsibly;

Or. en

Amendment 221

Proposal for a regulation

Article 55 – paragraph 1 – point c d (new)

Text proposed by the Commission

Amendment

(cd) offer tax breaks for doing research, better access to computer capacities and datasets, an EU-Visa schema for tech-talents,temporary support in technology scouting or in paying salaries of AI specialists, and state aid exemptions in the area of AI education, training and reskilling of employees;

Or. en

Amendment 222

Proposal for a regulation

Article 55 – paragraph 1 – point c e (new)

Text proposed by the Commission

Amendment

(ce) reduce extensive reporting, information or documentation obligations, establish a single EU online portal in different languages concerning all necessary procedures and formalities to operate in another EU country, a single point of contact in the home country that can certify the company’s eligibility to provide services in another EU country as well as a standardized EU-wide VAT declaration in the respective native language;

Or. en

Amendment 223

Proposal for a regulation

Article 55 – paragraph 2

Text proposed by the Commission

Amendment

2. The specific interests and needs of the small-scale providers shall be taken into account when setting the fees for conformity assessment under Article 43, reducing those fees proportionately to their size and market size.

2. The specific interests and needs of SMEs and start ups shall be taken into account when setting the fees for conformity assessment under Article 43, reducing those fees proportionately to their size and market size, by granting subsidies or even exempting SMEs and start ups from paying.

Or. en

Amendment 224

Proposal for a regulation

Article 56 – title

Text proposed by the Commission

Amendment

Establishment of the European Artificial Intelligence Board

Establishment of the European Artificial Intelligence Board and re-establishment of the High Level Expert Group on AI

Or. en

Justification

The amendments to Art 56, 57 and 58 shall make sure that the Board is capable to fulfil its crucial role in harmonising AI policies across the Union in an impartial manner. While not being a full AI agency, the Board has the competence to streamline via guidance, coordinate among various governance actors, and make binding decisions in case national competent authorities cannot agree on an important issues. The AMs will help to ensure a balanced and effective AI governance.

Amendment 225

Proposal for a regulation

Article 56 – paragraph 1

Text proposed by the Commission

Amendment

1. A ‘European Artificial Intelligence Board’ (the ‘Board’) is established.

1. A ‘European Artificial Intelligence Board’ (the ‘Board’) is established. The Board shall have a secretariat, a strong mandate as well as sufficient resources and skilled personnel at its disposal for assistance in the proper performance of its tasks laid down in Article 58.

Or. en

Amendment 226

Proposal for a regulation

Article 56 – paragraph 2 – point c

Text proposed by the Commission

Amendment

(c) assist the national supervisory authorities and the Commission in ensuring the consistent application of this Regulation.

(c) assist the national supervisory authorities and the Commission in ensuring the consistent application of this Regulation, in particular in line with the consistency mechanism referred to in Article 59 a(3).

Or. en

Amendment 227

Proposal for a regulation

Article 56 – paragraph 2 a (new)

Text proposed by the Commission

Amendment

2a. The High Level Expert group on AI is re-established, shall act as advisory body to the Board and shall be regularly invited to share its practical and technical expertise with the Board.

Or. en

Amendment 228

Proposal for a regulation

Article 57 – paragraph 1

Text proposed by the Commission

Amendment

1. The Board shall be composed of the national supervisory authorities, who shall be represented by the head or equivalent high-level official of that authority, and the European Data Protection Supervisor. Other national authorities may be invited to the meetings, where the issues discussed are of relevance for them.

1. The Board shall be composed of the national supervisory authorities, who shall be represented by the head or equivalent high-level official of that authority, the European Data Protection Supervisor as well as the Chairperson of the EU Agency for Fundamental Rights, the Executive director of the EU Agency for Cybersecurity, the Chair of the High Level Expert Group on AI, the Director-General of the Joint Research Centre, and the presidents of the European Committee for Standardization, the European Committee for Electrotechnical Standardization, and the European Telecommunications Standards Institute. Other national authorities may be invited to the meetings, where the issues discussed are of relevance for them.

Or. en

Amendment 229

Proposal for a regulation

Article 57 – paragraph 2

Text proposed by the Commission

Amendment

2. The Board shall adopt its rules of procedure by a simple majority of its members, following the consent of the Commission. The rules of procedure shall also contain the operational aspects related to the execution of the Board’s tasks as listed in Article 58. The Board may establish sub-groups as appropriate for the purpose of examining specific questions.

2. The Board shall adopt its rules of procedure by a simple majority of its members. The rules of procedure shall also contain the operational aspects related to the execution of the Board’s tasks as listed in Article 58. The Board may establish sub-groups as appropriate for the purpose of examining specific questions.

Or. en

Amendment 230

Proposal for a regulation

Article 57 – paragraph 3

Text proposed by the Commission

Amendment

3. The Board shall be chaired by the Commission. The Commission shall convene the meetings and prepare the agenda in accordance with the tasks of the Board pursuant to this Regulation and with its rules of procedure. The Commission shall provide administrative and analytical support for the activities of the Board pursuant to this Regulation.

3. The Board shall be chaired by the Commission. The Board secretariat shall convene the meetings and prepare the agenda in accordance with the tasks of the Board pursuant to this Regulation and with its rules of procedure. The Board secretariat shall provide administrative and analytical support for the activities of the Board pursuant to this Regulation.

Or. en

Amendment 231

Proposal for a regulation

Article 57 – paragraph 4

Text proposed by the Commission

Amendment

4. The Board may invite external experts and observers to attend its meetings and may hold exchanges with interested third parties to inform its activities to an appropriate extent. To that end the Commission may facilitate exchanges between the Board and other Union bodies, offices, agencies and advisory groups.

4. The Board shall institutionalise a structured dialogue with the High Level Expert Group on AI that takes place at least once a month. The Board shall also regularly invite external experts, in particular AI researchers and developers, and observers to attend its meetings, may hold exchanges with interested third parties to inform its activities to an appropriate extent, and hold appropriate consultations with relevant stakeholders and ensure appropriate participation. The Commission may facilitate exchanges between the Board and other Union bodies, offices, agencies and advisory groups.

Or. en

Amendment 232

Proposal for a regulation

Article 58 – paragraph 1 – point b

Text proposed by the Commission

Amendment

(b) contribute to uniform administrative practices in the Member States, including for the functioning of regulatory sandboxes referred to in Article 53;

(b) contribute to uniform administrative practices in the Member States, including for the assessment, establishing, managing with the meaning of fostering cooperation and guaranteeing consistency among regulatory sandboxes, and functioning of regulatory sandboxes referred to in Article 53;

Or. en

Amendment 233

Proposal for a regulation

Article 58 – paragraph 1 – point c – point iii a (new)

Text proposed by the Commission

Amendment

(iii a) on the need for the amendment of each of the Annexes as referred to in Article 73 as well as all other provisions in this Regulation that the Commission can amend, in light of the available evidence.

Or. en

Amendment 234

Proposal for a regulation

Article 58 – paragraph 1 – point c a (new)

Text proposed by the Commission

Amendment

(ca) carry out annual reviews and analyses of the complaints sent to and findings made by national competent authorities, of the serious incidents reports referred to in Article 62, and of the new registration in the EU Database referred to in Article 60 to identify trends and potential emerging issues threatening the future health and safety and fundamental rights of citizens that are not adequately addressed by this Regulation;

Or. en

Amendment 235

Proposal for a regulation

Article 58 – paragraph 1 – point c b (new)

Text proposed by the Commission

Amendment

(cb) carry out biannual horizon scanning and foresight exercises to extrapolate the impact the trends and emerging issues can have on the Union;

Or. en

Amendment 236

Proposal for a regulation

Article 58 – paragraph 1 – point c c (new)

Text proposed by the Commission

Amendment

(cc) annually publish recommendations to the Commission, in particular on the categorization of prohibited practices, high-risk systems, and codes of conduct for AI systems that are not classified as high-risk;

Or. en

Amendment 237

Proposal for a regulation

Article 58 – paragraph 1 – point c d (new)

Text proposed by the Commission

Amendment

(cd) encourage and facilitate the drawing up of codes of conduct as referred to in Article 69;

Or. en

Amendment 238

Proposal for a regulation

Article 58 – paragraph 1 – point c e (new)

Text proposed by the Commission

Amendment

(ce) coordinate among national competent authorities and make sure that the consistency mechanism in Article 59a (3) is observed, in particular for all major cross-border cases;

Or. en

Amendment 239

Proposal for a regulation

Article 58 – paragraph 1 – point c f (new)

Text proposed by the Commission

Amendment

(cf) adopt binding decisions for national competent authorities in case the consistency mechanism is not able to solve the conflict among national competent authorities as it is clarified in Article 59a (6).

Or. en

Amendment 240

Proposal for a regulation

Article 59 – paragraph 1

Text proposed by the Commission

Amendment

1. National competent authorities shall be established or designated by each Member State for the purpose of ensuring the application and implementation of this Regulation. National competent authorities shall be organised so as to safeguard the objectivity and impartiality of their activities and tasks.

1. Each Member State shall establish or designate one national competent authority as well as one national supervisory authority. The national competent authority shall be organised so as to safeguard the objectivity and impartiality of their activities and tasks, in particular by building up staff comprising of experts from different areas such as cyber security, competition law or data protection.

Or. en

Amendment 241

Proposal for a regulation

Article 59 – paragraph 2

Text proposed by the Commission

Amendment

2. Each Member State shall designate a national supervisory authority among the national competent authorities. The national supervisory authority shall act as notifying authority and market surveillance authority unless a Member State has organisational and administrative reasons to designate more than one authority.

2. The national competent authority shall be in charge to ensure the application and implementation of this Regulation. With regard to high-risk AI systems, related to products to which legal acts listed in Annex II apply, the competent authorities designated under those legal acts shall continue to lead the administrative procedures. However, to the extent a case involves aspects covered by this Regulation, the competent authorities shall be bound by measures issued by the national competent authority designated under this Regulation. The national supervisory authority shall act as notifying authority and market surveillance authority.

Or. en

Amendment 242

Proposal for a regulation

Article 59 – paragraph 3

Text proposed by the Commission

Amendment

3. Member States shall inform the Commission of their designation or designations and, where applicable, the reasons for designating more than one authority.

3. The national competent authority in each Member State shall be the lead authority, ensure adequate coordination and act as single point of contact for this Regulation. Member States shall inform the Commission of their designations.

Or. en

Amendment 243

Proposal for a regulation

Article 59 – paragraph 4

Text proposed by the Commission

Amendment

4. Member States shall ensure that national competent authorities are provided with adequate financial and human resources to fulfil their tasks under this Regulation. In particular, national competent authorities shall have a sufficient number of personnel permanently available whose competences and expertise shall include an in-depth understanding of artificial intelligence technologies, data and data computing, fundamental rights, health and safety risks and knowledge of existing standards and legal requirements.

4. Member States shall ensure that national competent authority is provided with adequate financial and human resources to fulfil their tasks under this Regulation. In particular, national competent authorities shall have a sufficient number of personnel permanently available whose competences and expertise shall include an in-depth understanding of artificial intelligence technologies, data and data computing, fundamental rights, health and safety risks and knowledge of existing standards and legal requirements.

Or. en

Amendment 244

Proposal for a regulation

Article 59 – paragraph 5

Text proposed by the Commission

Amendment

5. Member States shall report to the Commission on an annual basis on the status of the financial and human resources of the national competent authorities with an assessment of their adequacy. The Commission shall transmit that information to the Board for discussion and possible recommendations.

5. Member States shall report to the Commission on an annual basis on the status of the financial and human resources of the national competent authority with an assessment of their adequacy. The Commission shall transmit that information to the Board for discussion and possible recommendations.

Or. en

Amendment 245

Proposal for a regulation

Article 59 – paragraph 6

Text proposed by the Commission

Amendment

6. The Commission shall facilitate the exchange of experience between national competent authorities.

6. The Commission and Board shall facilitate the exchange of experience between national competent authorities.

Or. en

Amendment 246

Proposal for a regulation

Article 59 – paragraph 7

Text proposed by the Commission

Amendment

7. National competent authorities may provide guidance and advice on the implementation of this Regulation, including to small-scale providers. Whenever national competent authorities intend to provide guidance and advice with regard to an AI system in areas covered by other Union legislation, the competent national authorities under that Union legislation shall be consulted, as appropriate. Member States may also establish one central contact point for communication with operators.

7. National competent authorities may provide guidance and advice on the implementation of this Regulation and consult provider and user on issues that were discovered during the trustworthy technology assessment referred to in Article 9(2) (d a),including to SMEs and start-ups. Whenever national competent authorities intend to provide guidance and advice with regard to an AI system in areas covered by other Union legislation, the competent national authorities under that Union legislation shall be consulted, as appropriate. Member States may also establish one central contact point for communication with operators.

Or. en

Amendment 247

Proposal for a regulation

Article 59 – paragraph 8

Text proposed by the Commission

Amendment

8. When Union institutions, agencies and bodies fall within the scope of this Regulation, the European Data Protection Supervisor shall act as the competent authority for their supervision.

8. When Union institutions, agencies and bodies fall within the scope of this Regulation, the European Data Protection Supervisor shall act as the competent authority for their supervision and coordination.

Or. en

Amendment 248

Proposal for a regulation

Article 59 a (new)

Text proposed by the Commission

Amendment

Article 59 a

Consistency mechanism

1. Each national competent authority shall perform the tasks assigned to and the exercise of the powers conferred on it in accordance with this Regulation on the territory of its own Member State.

2. The national competent authority of the Member State where the provider or user is present or established shall be competent to act as lead national competent authority for across-border case that involves an AI-system that falls under this Regulation and that is being placed on the market or put into service in two or more Member States.

3. In order to contribute to the consistent application of this Regulation throughout the Union, national competent authorities shall cooperate with each other and, where relevant, with the Commission and the Board, through the consistency mechanism as set out in the following paragraphs.

4. The lead national competent authority shall cooperate with the other supervisory authorities in an endeavour to reach consensus. The lead national competent authority and the other national competent authorities concerned shall exchange all relevant information with each other, provide mutual assistance and execute joint operations.

5. The lead national competent authority shall, without delay, communicate the relevant information on the matter to the other national competent authorities concerned. It shall without delay submit a draft decision to the other national competent authorities concerned for their opinion and take due account of their views.

6. In case the Board, after being notified by another national competent authority, finds that the lead national competent authority did not use its investigative, corrective or authorisation power despite being notified by another national competent authority or came to a decision that is clearly incompatible with provisions of this Regulation, other national competent authorities may address the case on their own, taking into account the procedure described in paragraph 3.

Or. en

Justification

Most major cases will have a cross-border background, which is why the cooperation among national competent authorities is key if the AIA wants to ensure secure AI systems and a strong protection of fundamental rights. This article follows the GDPR approach but adjusts a few issues that have proven to be problematic in practice (e.g. passive DPAs).

Amendment 249

Proposal for a regulation

Article 60 – paragraph 1

Text proposed by the Commission

Amendment

1. The Commission shall, in collaboration with the Member States, set up and maintain a EU database containing information referred to in paragraph 2 concerning high-risk AI systems referred to in Article 6(2) which are registered in accordance with Article 51.

1. The Commission shall, in collaboration with the Member States and by building on the existing Business Registries in line with Directive 2012/17/EU, set up and maintain a EU database containing information referred to in paragraph 2 concerning high-risk AI systems referred to in Article 6(2) and (2a) which are registered in accordance with Article 51.

Or. en

Amendment 250

Proposal for a regulation

Article 61 – paragraph 2

Text proposed by the Commission

Amendment

2. The post-market monitoring system shall actively and systematically collect, document and analyse relevant data provided by users or collected through other sources on the performance of high-risk AI systems throughout their lifetime, and allow the provider to evaluate the continuous compliance of AI systems with the requirements set out in Title III, Chapter 2.

2. The post-market monitoring system shall actively and systematically collect, document and analyse relevant data provided by users and end-user or collected through other sources, to the extent such data are readily accessible to the provider and taking into account the limits resulting from data protection, copyright and competition law, on the performance of high-risk AI systems throughout their lifetime, and allow the provider to evaluate the continuous compliance of AI systems with the requirements set out in Title III, Chapter 2.

Or. en

Amendment 251

Proposal for a regulation

Article 62 – title

Text proposed by the Commission

Amendment

Reporting of serious incidents and of malfunctioning

Reporting of serious incidents

Or. en

Amendment 252

Proposal for a regulation

Article 62 – paragraph 1 – introductory part

Text proposed by the Commission

Amendment

1. Providers of high-risk AI systems placed on the Union market shall report any serious incident or any malfunctioning of those systems which constitutes a breach of obligations under Union law intended to protect fundamental rights to the market surveillance authorities of the Member States where that incident or breach occurred.

1. Providers of high-risk AI systems placed on the Union market shall report any serious incident to the market surveillance authorities of the Member States where that incident or breach occurred.

Or. en

Amendment 253

Proposal for a regulation

Article 62 – paragraph 1 – subparagraph 1

Text proposed by the Commission

Amendment

Such notification shall be made immediately after the provider has established a causal link between the AI system and the incident or malfunctioning or the reasonable likelihood of such a link, and, in any event, not later than 15 days after the providers becomes aware of the serious incident or of the malfunctioning.

Such notification shall be made without undue delay after the provider or user has established a causal link between the AI system and the serious incident or the reasonable likelihood of such a link, and, in any event, not later than 72 hours after the providers becomes aware of the serious incident.

No report under this Article is required if the serious incident also leads to reporting requirements under other laws. In that case, the authorities competent under those laws shall forward the received report to the national competent authority.

Or. en

Amendment 254

Proposal for a regulation

Article 62 – paragraph 2

Text proposed by the Commission

Amendment

2. Upon receiving a notification related to a breach of obligations under Union law intended to protect fundamental rights, the market surveillance authority shall inform the national public authorities or bodies referred to in Article 64(3). The Commission shall develop dedicated guidance to facilitate compliance with the obligations set out in paragraph 1. That guidance shall be issued 12 months after the entry into force of this Regulation, at the latest.

2. Upon receiving a notification related to a serious incident referred to in Article 3(44), the relevant market surveillance authority shall inform the national public authorities or bodies referred to in Article 64(3). The Commission shall develop dedicated guidance to facilitate compliance with the obligations set out in paragraph 1. That guidance shall be issued 12 months after the entry into force of this Regulation, at the latest.

Or. en

Amendment 255

Proposal for a regulation

Article 62 – paragraph 3

Text proposed by the Commission

Amendment

3. For high-risk AI systems referred to in point 5(b) of Annex III which are placed on the market or put into service by providers that are credit institutions regulated by Directive 2013/36/EU and for high-risk AI systems which are safety components of devices, or are themselves devices, covered by Regulation (EU) 2017/745 and Regulation (EU) 2017/746, the notification of serious incidents or malfunctioning shall be limited to those that that constitute a breach of obligations under Union law intended to protect fundamental rights.

3. For high-risk AI systems referred to in point 5(b) of Annex III which are placed on the market or put into service by providers that are subject to regulations that require solutions equivalent to those set out in this Regulation, the notification of serious incidents shall be limited to those referred to in Article 3(44).

Or. en

Amendment 256

Proposal for a regulation

Article 64 – paragraph 1

Text proposed by the Commission

Amendment

1. Access to data and documentation in the context of their activities, the market surveillance authorities shall be granted full access to the training, validation and testing datasets used by the provider, including through application programming interfaces (‘API’) or other appropriate technical means and tools enabling remote access.

1. When appropriate and proportionate, market surveillance authorities may request access to data and documentation in the context of their activities. The market surveillance authorities shall only be granted, access to those training, validation and testing datasets used by the provider that are strictly necessary for the purpose of its request.

Or. en

Amendment 257

Proposal for a regulation

Article 64 – paragraph 2

Text proposed by the Commission

Amendment

2. Where necessary to assess the conformity of the high-risk AI system with the requirements set out in Title III, Chapter 2 and upon a reasoned request, the market surveillance authorities shall be granted access to the source code of the AI system.

deleted

Or. en

Amendment 258

Proposal for a regulation

Article 67

Text proposed by the Commission

Amendment

Article 67

deleted

Compliant AI systems which present a risk

1. Where, having performed an evaluation under Article 65, the market surveillance authority of a Member State finds that although an AI system is in compliance with this Regulation, it presents a risk to the health or safety of persons, to the compliance with obligations under Union or national law intended to protect fundamental rights or to other aspects of public interest protection, it shall require the relevant operator to take all appropriate measures to ensure that the AI system concerned, when placed on the market or put into service, no longer presents that risk, to withdraw the AI system from the market or to recall it within a reasonable period, commensurate with the nature of the risk, as it may prescribe.

2. The provider or other relevant operators shall ensure that corrective action is taken in respect of all the AI systems concerned that they have made available on the market throughout the Union within the timeline prescribed by the market surveillance authority of the Member State referred to in paragraph 1.

3. The Member State shall immediately inform the Commission and the other Member States. That information shall include all available details, in particular the data necessary for the identification of the AI system concerned, the origin and the supply chain of the AI system, the nature of the risk involved and the nature and duration of the national measures taken.

4. The Commission shall without delay enter into consultation with the Member States and the relevant operator and shall evaluate the national measures taken. On the basis of the results of that evaluation, the Commission shall decide whether the measure is justified or not and, where necessary, propose appropriate measures.

5. The Commission shall address its decision to the Member States.

Or. en

Amendment 259

Proposal for a regulation

Article 68 – paragraph 2

Text proposed by the Commission

Amendment

2. Where the non-compliance referred to in paragraph 1 persists, the Member State concerned shall take all appropriate measures to restrict or prohibit the high-risk AI system being made available on the market or ensure that it is recalled or withdrawn from the market.

2. Where the non-compliance referred to in paragraph 1 persists, the Member State concerned shall take all appropriate and proportionate measures to restrict or prohibit the high-risk AI system being made available on the market.

Or. en

Amendment 260

Proposal for a regulation

Article 68 a (new)

Text proposed by the Commission

Amendment

Article 68 a

Right to lodge a complaint with a supervisory authority

1. Every citizen who considers that his or her right to protection of personal data has been infringed by the use of a prohibited AI system or a high-risk AI system shall have the right to lodge a complaint with the authority in charge to handle complaints under Article 77 of Regulation (EU) 2016/679 in the Member State of his or her habitual residence, place of work or place of the alleged infringement.

2. The supervisory authority with which the complaint has been lodged shall inform the complainant on the progress and the outcome of the complaint.

Or. en

Amendment 261

Proposal for a regulation

Article 69 – paragraph 1 a (new)

Text proposed by the Commission

Amendment

1a. The Commission and the Board shall encourage and facilitate the drawing up of codes of conduct intended to foster the voluntary application of the concept of trustworthy AI set out in Article 4(a) to AI systems other than high-risk AI systems on the basis of technical specifications and solutions that are appropriate means of ensuring compliance with such requirements in light of the intended purpose of the systems.

Or. en

Amendment 262

Proposal for a regulation

Article 69 – paragraph 4

Text proposed by the Commission

Amendment

4. The Commission and the Board shall take into account the specific interests and needs of the small-scale providers and start-ups when encouraging and facilitating the drawing up of codes of conduct.

4. The Commission and the Board shall take into account the specific interests and needs of SMEs and start-ups when encouraging and facilitating the drawing up of codes of conduct.

Or. en

Amendment 263

Proposal for a regulation

Article 70 – paragraph 1 – introductory part

Text proposed by the Commission

Amendment

1. National competent authorities and notified bodies involved in the application of this Regulation shall respect the confidentiality of information and data obtained in carrying out their tasks and activities in such a manner as to protect, in particular:

1. National competent authorities and notified bodies involved in the application of this Regulation shall put effective cybersecurity measures in place and respect the confidentiality of information and data obtained in carrying out their tasks and activities in such a manner as to protect, in particular:

Or. en

Amendment 264

Proposal for a regulation

Article 70 – paragraph 1 – point a

Text proposed by the Commission

Amendment

(a) intellectual property rights, and confidential business information or trade secrets of a natural or legal person, including source code, except the cases referred to in Article 5 of Directive 2016/943 on the protection of undisclosed know-how and business information (trade secrets) against their unlawful acquisition, use and disclosure apply.

(a) intellectual property rights, and confidential business information or trade secrets of a natural or legal person in line with the 2016 EU Trade Secrets Directive (Directive 2016/943) as well as the 2004 Directive on the enforcement of intellectual property rights (Directive 2004/48/EC), including source code, except the cases referred to in Article 5 of Directive 2016/943 on the protection of undisclosed know-how and business information (trade secrets)against their unlawful acquisition, use and disclosure apply.

Or. en

Amendment 265

Proposal for a regulation

Article 70 – paragraph 1 – point c a (new)

Text proposed by the Commission

Amendment

(ca) the principles of purpose limitation and data minimization, meaning that national competent authorities minimize the quantity of data requested for disclosure in line with what is absolutely necessary for the perceived risk and its assessment, and they must not keep the data for any longer than absolutely necessary.

Or. en

Amendment 266

Proposal for a regulation

Article 70 – paragraph 4

Text proposed by the Commission

Amendment

4. The Commission and Member States may exchange, where necessary, confidential information with regulatory authorities of third countries with which they have concluded bilateral or multilateral confidentiality arrangements guaranteeing an adequate level of confidentiality.

4. The Commission and Member States may, if consistent with the provisions contained in EU trade agreements with third countries, exchange, where necessary, confidential information with regulatory authorities of third countries with which they have concluded bilateral or multilateral confidentiality arrangements guaranteeing an adequate level of confidentiality.

Or. en

Amendment 267

Proposal for a regulation

Article 71 – paragraph 3 – introductory part

Text proposed by the Commission

Amendment

3. The following infringements shall be subject to administrative fines of up to 30 000 000 EUR or, if the offender is company, up to 6 % of its total worldwide annual turnover for the preceding financial year, whichever is higher:

3. Non-compliance with the prohibition of the AI practices referred to in Article 5 shall be subject to administrative fines of up to 20000 000 EUR or, if the offender is a company, up to 4% of its total worldwide annual turnover for the preceding financial year, whichever is higher.

Or. en

Amendment 268

Proposal for a regulation

Article 71 – paragraph 3 – point a

Text proposed by the Commission

Amendment

(a) non-compliance with the prohibition of the artificial intelligence practices referred to in Article 5;

deleted

Or. en

Amendment 269

Proposal for a regulation

Article 71 – paragraph 3 – point b

Text proposed by the Commission

Amendment

(b) non-compliance of the AI system with the requirements laid down in Article 10.

deleted

Or. en

Amendment 270

Proposal for a regulation

Article 71 – paragraph 4

Text proposed by the Commission

Amendment

4. The non-compliance of the AI system with any requirements or obligations under this Regulation, other than those laid down in Articles 5 and 10, shall be subject to administrative fines of up to 20 000 000 EUR or, if the offender is a company, up to 4 % of its total worldwide annual turnover for the preceding financial year, whichever is higher.

4. The grossly negligent non-compliance of the provider or user of the AI system with the respective requirements or obligations under this Regulation, other than those laid down in Articles 5, shall be subject to administrative fines of up to 10 000 000 EUR or, if the offender is a company, up to 2 % of its total worldwide annual turnover for the preceding financial year, whichever is higher.

Or. en

Amendment 271

Proposal for a regulation

Article 71 – paragraph 6 – introductory part

Text proposed by the Commission

Amendment

6. When deciding on the amount of the administrative fine in each individual case, all relevant circumstances of the specific situation shall be taken into account and due regard shall be given to the following:

6. Fines may be imposed in addition to or instead of non-monetary measures such as orders or warnings. When deciding on whether to impose a fine or on the amount of the administrative fine in each individual case, all relevant circumstances of the specific situation shall be taken into account and due regard shall be given to the following:

Or. en

Amendment 272

Proposal for a regulation

Article 71 – paragraph 6 – point c a (new)

Text proposed by the Commission

Amendment

(ca) the intentional or negligent character of the infringement;

Or. en

Amendment 273

Proposal for a regulation

Article 71 – paragraph 6 – point c b (new)

Text proposed by the Commission

Amendment

(cb) any action taken by the provider to mitigate the harm or damage suffered by the affected persons;

Or. en

Amendment 274

Proposal for a regulation

Article 71 – paragraph 6 – point c c (new)

Text proposed by the Commission

Amendment

(cc) any relevant previous infringements by the provider;

Or. en

Amendment 275

Proposal for a regulation

Article 71 – paragraph 6 – point c d (new)

Text proposed by the Commission

Amendment

(cd) the degree of cooperation with the national competent authorities, in order to remedy the infringement and mitigate the possible adverse effects of the infringement;

Or. en

Amendment 276

Proposal for a regulation

Article 71 – paragraph 6 – point c e (new)

Text proposed by the Commission

Amendment

(ce) the manner in which the infringement became known to the national competent authority, in particular whether, and if so to what extent, the provider notified the infringement;

Or. en

Amendment 277

Proposal for a regulation

Article 71 – paragraph 6 – point c f (new)

Text proposed by the Commission

Amendment

(cf) any other aggravating or mitigating factor applicable to the circumstances of the case, such as financial benefits gained, or losses avoided, directly or indirectly, from the infringement.

Or. en

Amendment 278

Proposal for a regulation

Article 71 – paragraph 8 a (new)

Text proposed by the Commission

Amendment

8a. Administrative fines shall not be applied to a participant in a regulatory sandbox, who was acting in line with the recommendation issued by the supervisory authority;

Or. en

Amendment 279

Proposal for a regulation

Article 71 – paragraph 8 b (new)

Text proposed by the Commission

Amendment

8b. The penalties referred to in this article as well as the associated litigation costs and indemnification claims may not be the subject of contractual clauses or other form of burden-sharing agreements between the providers and distributors, importers, users, or any other third-parties.

Or. en

Amendment 280

Proposal for a regulation

Article 73 – paragraph 2 a (new)

Text proposed by the Commission

Amendment

2a. The delegation of power referred to in Article4, Article 7(1), Article 11(3), Article 43(5) and (6) and Article 48(5) shall undergo due process, be proportionate and be based on a permanent and institutionalised exchange with the relevant stakeholders as well as the Board and the High Level Expert Group on AI.

Or. en

Amendment 281

Proposal for a regulation

Article 83 – paragraph 2

Text proposed by the Commission

Amendment

2. This Regulation shall apply to the high-risk AI systems, other than the ones referred to in paragraph 1, that have been placed on the market or put into service before [date of application of this Regulation referred to in Article 85(2)], only if, from that date, those systems are subject to significant changes in their design or intended purpose.

2. This Regulation shall apply to the high-risk AI systems, other than the ones referred to in paragraph 1, that have been placed on the market or put into service before [date of application of this Regulation referred to in Article 85(2)], only if, from that date, those systems are subject to substantial modification as defined in Article 3(23) in their design or intended purpose.

Or. en

Amendment 282

Proposal for a regulation

Article 85 – paragraph 2

Text proposed by the Commission

Amendment

2. This Regulation shall apply from [24 months following the entering into force of the Regulation].

2. This Regulation shall apply from [48 months following the entering into force of the Regulation].

Or. en

Amendment 283

Proposal for a regulation

Article 85 – paragraph 3 a (new)

Text proposed by the Commission

Amendment

3a. Member States shall not until ... [24 months after the date of application of this Regulation] impede the making available of AI systems and products which were placed on the market in conformity with Union harmonisation legislation before [the date of application of this Regulation].

Or. en

Amendment 284

Proposal for a regulation

Article 85 – paragraph 3 b (new)

Text proposed by the Commission

Amendment

3b. At the latest by six months after entry into force of this Regulation, the European Commission shall submit a standardization request to the European Standardisation Organisations in order to ensure the timely provision of all relevant harmonised standards that cover the essential requirements of this regulation. Any delay in submitting the standardisation request shall add to the transitional period of 24 months as stipulated in paragraph 4.

Or. en

Amendment 285

Proposal for a regulation

Annex I – point a

Text proposed by the Commission

Amendment

(a) Machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning;

(a) Machine learning and optimization approaches, including but not limited to evolutionary computing as well as supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning;

Or. en

Justification

The justification for a lex specialis on AI by the Commission was based on the specific characteristics, such as autonomy and opacity, of (rather new) machine-learning and data-driven AI applications. It was argued that they are so far not adequately covered by existing laws. Their existence would therefore demand new laws. Symbolic AI (dominant from the 1950s-90s) is however already covered by numerous EU and national laws. Point (b) and (c) fall exactly in this category. It is therefore not justified to address them - again - within the AI Act. Their inclusion would be contradictory to the impact assessment as well as better regulation principles.

Amendment 286

Proposal for a regulation

Annex I – point b

Text proposed by the Commission

Amendment

(b) Logic- and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems;

deleted

Or. en

Amendment 287

Proposal for a regulation

Annex I – point c

Text proposed by the Commission

Amendment

(c) Statistical approaches, Bayesian estimation, search and optimization methods.

deleted

Or. en

Amendment 288

Proposal for a regulation

Annex II – Part A – point 6

Text proposed by the Commission

Amendment

6. Directive 2014/53/EU of the European Parliament and of the Council of 16 April 2014 on the harmonisation of the laws of the Member States relating to the making available on the market of radio equipment and repealing Directive 1999/5/EC (OJ L 153, 22.5.2014, p. 62);

deleted

Or. en

Amendment 289

Proposal for a regulation

Annex II – Part B – point 7 a (new)

Text proposed by the Commission

Amendment

7a. Regulation (EU) 2017/745 and Regulation (EU) 2017/746.

Or. en

Amendment 290

Proposal for a regulation

Annex III – title

Text proposed by the Commission

Amendment

HIGH-RISK AI SYSTEMS REFERRED TO IN ARTICLE 6(2)

CRITICAL AREAS REFERRED TO IN ARTICLE 6(2)

Or. en

Amendment 291

Proposal for a regulation

Annex III – paragraph 1 – introductory part

Text proposed by the Commission

Amendment

High-risk AI systems pursuant to Article 6(2) are the AI systems listed in any of the following areas:

Critical areas pursuant to Article 6(2) are the ones listed in the following areas:

Or. en

Amendment 292

Proposal for a regulation

Annex III – paragraph 1 – point 1 – introductory part

Text proposed by the Commission

Amendment

1. Biometric identification and categorisation of natural persons:

1. Biometric identification systems intended to be used for the ‘real-time’ and ‘post’ remote biometric identification, excluding biometric authentication, of natural persons (i.e., revealing their identity or tracking their behaviour)without their expressed or implied agreement;

Or. en

Amendment 293

Proposal for a regulation

Annex III – paragraph 1 – point 1 – point a

Text proposed by the Commission

Amendment

(a) AI systems intended to be used for the ‘real-time’ and ‘post’ remote biometric identification of natural persons;

deleted

Or. en

Amendment 294

Proposal for a regulation

Annex III – paragraph 1 – point 2 – introductory part

Text proposed by the Commission

Amendment

2. Management and operation of critical infrastructure:

2. Critical infrastructure:

Or. en

Amendment 295

Proposal for a regulation

Annex III – paragraph 1 – point 2 – point a

Text proposed by the Commission

Amendment

(a) AI systems intended to be used as safety components in the management and operation of road traffic and the supply of water, gas, heating and electricity.

(a) AI systems used as safety components in the management and operation of road traffic to the extent that they are not embedded in a motor vehicle,

Or. en

Amendment 296

Proposal for a regulation

Annex III – paragraph 1 – point 2 – point a a (new)

Text proposed by the Commission

Amendment

(aa) AI systems intended to be used as safety components in the management and operation of the supply of water, gas, heating and electricity, provided the failure of the AI system might lead to an imminent threat to such supply;

Or. en

Amendment 297

Proposal for a regulation

Annex III – paragraph 1 – point 3 – point a

Text proposed by the Commission

Amendment

(a) AI systems intended to be used for the purpose of determining access or assigning natural persons to educational and vocational training institutions;

(a) AI systems intended to be used for the purpose of determining and making autonomous decision on the admission of natural persons to educational and vocational training institutions;

Or. en

Amendment 298

Proposal for a regulation

Annex III – paragraph 1 – point 3 – point b

Text proposed by the Commission

Amendment

(b) AI systems intended to be used for the purpose of assessing students in educational and vocational training institutions and for assessing participants in tests commonly required for admission to educational institutions.

(b) AI systems intended to be used for the purpose of assessing the learning outcome of students in educational and vocational training institutions and for assessing participants in tests commonly required for admission to these institutions.

Or. en

Amendment 299

Proposal for a regulation

Annex III – paragraph 1 – point 4 – introductory part

Text proposed by the Commission

Amendment

4. Employment, workers management and access to self-employment:

4. Employment, workers management and access to self-employment:

Or. en

Amendment 300

Proposal for a regulation

Annex III – paragraph 1 – point 4 – point a

Text proposed by the Commission

Amendment

(a) AI systems intended to be used for recruitment or selection of natural persons, notably for advertising vacancies, screening or filtering applications, evaluating candidates in the course of interviews or tests;

(a) AI systems intended to be used for the sole purpose of recruitment or selection of natural persons, and making autonomous decisions on promotion and termination of work-related contractual relationships and evaluating performance and behavior of persons in such relationships.

Or. en

Amendment 301

Proposal for a regulation

Annex III – paragraph 1 – point 4 – point b

(b) AI intended to be used for making decisions on promotion and termination of work-related contractual relationships, for task allocation and for monitoring and evaluating performance and behavior of persons in such relationships.

deleted

(b) AI intended to be used for making decisions on promotion and termination of work-related relationships, for task allocation, for monitoring compliance with workplace rules and for monitoring and evaluating performance and behavior of persons in such relationships.

Or. en

Amendment 302

Proposal for a regulation

Annex III – paragraph 1 – point 5 – point b

Text proposed by the Commission

Amendment

(b) AI systems intended to be used to evaluate the creditworthiness of natural persons or establish their credit score, with the exception of AI systems put into service by small scale providers for their own use;

deleted

Or. en

Amendment 303

Proposal for a regulation

Annex III – paragraph 1 – point 8 – point a

Text proposed by the Commission

Amendment

(a) AI systems intended to assist a judicial authority in researching and interpreting facts and the law and in applying the law to a concrete set of facts.

(a) AI systems intended to be used by a judicial authority, administrative body or on their behalf for interpreting facts or the law and for applying the law to a concrete set of facts.

Or. en

Amendment 304

Proposal for a regulation

Annex IV – paragraph 1 – point 2 – point b

Text proposed by the Commission

Amendment

(b) the design specifications of the system, namely the general logic of the AI system and of the algorithms; the key design choices including the rationale and assumptions made, also with regard to persons or groups of persons on which the system is intended to be used; the main classification choices; what the system is designed to optimise for and the relevance of the different parameters; the decisions about any possible trade-off made regarding the technical solutions adopted to comply with the requirements set out in Title III, Chapter 2;

(b) when relevant, proportionate and with respect of intellectual property rights and trade secrets, the design specifications of the system, namely the general logic of the AI system and of the algorithms; the key design choices including the rationale and assumptions made, also with regard to persons or groups of persons on which the system is intended to be used; the main classification choices; what the system is designed to optimise for and the relevance of the different parameters; the decisions about any possible trade-off made regarding the technical solutions adopted to comply with the requirements set out in Title III, Chapter 2;

Or. en

Amendment 305

Proposal for a regulation

Annex IV – paragraph 1 – point 5

Text proposed by the Commission

Amendment

5. A description of any change made to the system through its lifecycle;

deleted

Or. en

Amendment 306

Proposal for a regulation

Annex VII – title

Text proposed by the Commission

Amendment

CONFORMITY BASED ON ASSESSMENT OF QUALITY MANAGEMENT SYSTEM AND ASSESSMENT OF TECHNICAL DOCUMENTATION

CONFORMITY BASED ON ASSESSMENT OF TECHNICAL DOCUMENTATION

Or. en

Amendment 307

Proposal for a regulation

Annex VII – point 2 – paragraph 1

Text proposed by the Commission

Amendment

The approved quality management system for the design, development and testing of AI systems pursuant to Article 17 shall be examined in accordance with point 3 and shall be subject to surveillance as specified in point 5. The technical documentation of the AI system shall be examined in accordance with point 4.

The technical documentation of the AI system shall be examined in accordance with point 4.

Or. en

Amendment 308

Proposal for a regulation

Annex VII – point 3

Text proposed by the Commission

Amendment

3. Quality management system

deleted

3.1. The application of the provider shall include:

(a) the name and address of the provider and, if the application is lodged by the authorised representative, their name and address as well;

(b) the list of AI systems covered under the same quality management system;

(c) the technical documentation for each AI system covered under the same quality management system;

(d) the documentation concerning the quality management system which shall cover all the aspects listed under Article 17;

(e) a description of the procedures in place to ensure that the quality management system remains adequate and effective;

(f) a written declaration that the same application has not been lodged with any other notified body.

3.2. The quality management system shall be assessed by the notified body, which shall determine whether it satisfies the requirements referred to in Article 17.

The decision shall be notified to the provider or its authorised representative.

The notification shall contain the conclusions of the assessment of the quality management system and the reasoned assessment decision.

3.3. The quality management system as approved shall continue to be implemented and maintained by the provider so that it remains adequate and efficient.

3.4. Any intended change to the approved quality management system or the list of AI systems covered by the latter shall be brought to the attention of the notified body by the provider.

The proposed changes shall be examined by the notified body, which shall decide whether the modified quality management system continues to satisfy the requirements referred to in point 3.2 or whether a reassessment is necessary.

The notified body shall notify the provider of its decision. The notification shall contain the conclusions of the examination of the changes and the reasoned assessment decision.

Or. en

Amendment 309

Proposal for a regulation

Annex VIII – point 5 a (new)

Text proposed by the Commission

Amendment

5a. The outcome of the trustworthy technology assessment;

Or. en

ANNEX: LIST OF ENTITIES OR PERSONS FROM WHOM THE RAPPORTEUR HAS RECEIVED INPUT

The following list is drawn up on a purely voluntary basis under the exclusive responsibility of the rapporteur. The rapporteur has received input from the following entities or persons in the preparation of the draft opinion:

Entity and/or person

AccessNow

Baptiste Caramiaux, CNRS researcher, Sorbonne Université in Paris

BEUC

Defenddigitalme

EDRi

European Disability Forum

Huawei

International Federation of Library Associations and Institutions

Joanna Bryson, Professor of Ethics and Technology, Hertie School

Liber Europe

Maldita.es

Martin Senftlebe, Professor of Intellectual Property Law, University of Amsterdam

Meaning Processing Ltd.

Saidot Ltd.

Scio.cz