The EU AI Act – the reality is better than many feared

The European Union’s Artificial Intelligence Act came into force this month. Much of the text addresses high-risk AI systems used within the EU, regardless of whether they are based in the EU or in a third country, such as the US. 

The EU explains:

AI systems are always considered high-risk if [the AI] profiles individuals, i.e. automated processing of personal data to assess various aspects of a person’s life, such as work performance, economic situation, health, preferences, interests, reliability, behaviour, location or movement.

Under the new Act, developers of all high-risk systems must: establish a risk management system; conduct data governance; draw up technical documentation to demonstrate compliance, design the AI for record-keeping, and to achieve appropriate levels of accuracy, robustness, and cybersecurity; establish a quality management system to ensure compliance; and design their systems to allow deployers to implement human oversight.

Meanwhile, developers and deployers of more “limited-risk AI systems” must ensure that users are aware they are interacting with an AI (for example, chatbots and deep fakes). This is a sensible provision, aimed at preventing the deliberate deception and manipulation of end-users.

As for general-purpose AIs (GPAIs), models present systemic risks when the cumulative amount of compute used for its training is greater than 10²⁵ floating point operations (FLOPs) says the Act.

It explains:

Providers must notify the [European] Commission if their model meets this criterion within two weeks. The provider may present arguments that, despite meeting the criteria, their model does not present systemic risks. The Commission may decide on its own, or via a qualified alert from the scientific panel of independent experts, that a model has high impact capabilities, rendering it systemic.

Significantly, all GPAI model providers must also comply with the EU’s Copyright Directive, and publish a summary about the content used for training.

IP-centric businesses will welcome that provision, with the unlicensed scraping of copyrighted data to train Large Language Models and generative systems the subject of ongoing litigation in the US. 

These practices were also called out by the UK’s House of Lords’ Communications and Digital Committee in its LLM Inquiry last year, and in its subsequent report this year (see diginomica, passim). Alas, the government “sat on its hands” on the issue, noted the Committee, so the EU Act may persuade Britain’s policymakers to look at it again.

Citizen protections

In other areas, the Act makes citizen protections paramount, with the following types of AI system expressly prohibited:

• Those deploying subliminal, manipulative, or deceptive techniques to distort behaviour and impair informed decision-making, causing “significant harm”.

– This is an interesting provision, as it could, conceivably, be applied to AI-infused social platforms, where algorithms might favour divisive content, or force people towards fake content or scams. I predict that test cases will be brought against the likes of Meta Platforms and X Corp to see if these clauses can be used in this way.

– Also included in this broad definition are risks such as “voice-activated toys that encourage dangerous behaviour in children”.

• Those exploiting vulnerabilities related to age, disability, or socio-economic circumstances in order to distort behaviour, causing “significant harm”.
• Biometric categorization systems that infer attributes, such as race, political opinions, trade union membership, faith, or sexual orientation from someone’s physical appearance, perhaps leading to them being treated incorrectly, unfairly, or with prejudice. Such a system might make completely false predictions about an individual, based on biased human assumptions.
• Social scoring, such as evaluating or classifying individuals or groups based on social behaviour or personal traits, causing detrimental or unfavourable treatment of some when compared with others.
• Systems that assess the risk of an individual committing criminal offenses, based solely on profiling or personality traits. However, exceptions apply when AI is used to augment human assessments that are based on objective, verifiable facts linked directly to criminal activity. 

– This is a timely development, with some countries, such as Argentina last week, announcing that they intend to use AI to predict crime, risking some individuals or groups being subjected to flawed assessments based on biased human data. 

– Essentially, therefore, the EU is seeking to prevent the adoption of so-called ‘thought crime’ concepts: the prediction that an offence will be committed by someone who appears similar to a criminal or suspect.

– An early example of automated bias in the criminal justice system was the COMPAS sentencing guidelines algorithm used in some US courtrooms to predict recidivism. A 2016 ProPublica investigation found that COMPAS predicted that black defendants were more likely to reoffend than white. This led to harsher sentencing of black Americans, due to historic systemic bias in parts of the US legal system.

• Systems that compile facial recognition databases by scraping facial images from the internet or CCTV footage.
• Those that infer emotions in workplaces or educational institutions, except for medical or safety reasons. 

– Again, this is a sensible provision: sentiment-and emotion-reading systems are clumsy and lack sufficiently diverse training data. As a result, they may give wildly inaccurate readings for some individuals, for neurodiverse people, and for workers from different races or cultures. 

• Real-time remote biometric identification – such as live facial recognition – in public spaces for law enforcement, except when searching for missing persons, abduction victims, and other vulnerable or trafficked people.

– Live facial recognition has been found, in study after study, to give less accurate readings for women and for black and other ethnic minority citizens, leading to a high risk of misidentification. These problems are rooted in poor diversity in training data systems that are trained on more white males than other groups – and, in some cases, also in imaging systems whose sensors have been calibrated on white skin tones.

• However, the EU says this will be balanced with the need to prevent substantial and imminent threat to life, terrorist attack, or locating the suspects of serious crimes.

Big-picture aims

Overall, the European Parliament says its aim is ensuring that AI systems used within the EU are safe, transparent, traceable, non-discriminatory and environmentally friendly. AI systems should be overseen by people, rather than by automation, it adds.

Addressing some developers’ fears that regulation would impede innovation, the EU notes:

As part of its digital strategy, the EU wants to regulate artificial intelligence (AI) to ensure better conditions for the development and use of this innovative technology. AI can create many benefits, such as better healthcare; safer and cleaner transport; more efficient manufacturing; and cheaper and more sustainable energy.

As with the General Data Protection Regulation (GDPR) – which, though flawed, has become the gold standard for seeking to prevent the abuse of citizen data – the financial penalties are severe. The Act mandates fines of up to seven percent of global annual turnover for violations in the form of banned applications, up to three percent for breaches of the Act’s other provisions, and up to 1.5% for supplying incorrect data to regulators. 

My take

On the face of it, an apparently sincere move to protect citizens from the worst excesses and dangers of a technology that may have unpredictable social effects. 

Indeed, its provisions make it hard for vendors to claim that the EU is seeking to rein in US companies and innovation in general. What the Act does do is impose a duty of care and transparency on the vendors of high-risk systems – and more than a few on the developers of lower-risk and general-purpose AIs too. 

Fine tuning and stress testing will follow, with a phased rollout of compliance measures and standards.