IDAIS-Shanghai saw top researchers call for urgent international cooperation to ensure advanced AI systems remain controllable and aligned with human intentions and values.

Consensus Statement on Ensuring Alignment and Human Control of Advanced AI Systems to Safeguard Human Flourishing

The rapid advancement of artificial intelligence entails unprecedented risks that must be addressed to realise unprecedented opportunities for development. Humanity is at a pivotal juncture as AI systems may soon match and surpass human intelligence. These future systems could take undesired actions without their operators realising, leading to a loss of control, where one or more general-purpose AI systems come to operate outside of anyone's control, posing catastrophic and existential risks.

In the last year, there has been increasing evidence suggesting that advanced future AI systems may deceive humans and escape our control. Existing research has shown that current systems can detect when they are being evaluated and pretend to be aligned with human values to pass tests. These systems have also begun to display self-preserving behaviours, such as attempting to blackmail developers to avoid being replaced with a new version.  

We emphasise: some AI systems today already demonstrate the capability and propensity to undermine their creators’ safety and control efforts.  

While this evidence has primarily been found in experimental settings, there are still no known and proven methods to ensure reliable alignment of and effective human control over AI systems, particularly for future systems that are both general and more capable than humans. The world faces this shared urgent challenge of strengthening the assessment and prevention of potential risks in AI development and ensuring these rapidly improving AI systems remain safe, reliable, and controllable even as they surpass human capabilities.  

The need to ensure that advanced AI systems are aligned and under human control has been recognised by key decision makers. Major jurisdictions have added to their regulatory frameworks to take the lead in shaping AI development and governance for example: the European Union has issued the EU AI Act and set up the EU AI Office; China has mandated registration for generative AI services and established the China AI Safety and Development Association; the UK launched the AI Summit Series and established the first and largest AI Security Institute in the world; and the US has created the Center for AI Standards and Innovation for company guidance and pre-deployment testing. Across the world, states have taken steps to distinguish risk tiers and apply proportionate restrictions that endeavour to balance development and safety. Frontier AI companies across the world have also signed on to voluntary commitments, pledging to take safety measures such as building up their safety and security teams and evaluating their models for frontier risks prior to deployment.  

Nonetheless, investments in AI safety have significantly lagged behind advances in capabilities. As we come closer to AI capabilities that pose catastrophic risks, it is critical that key global actors take credible steps on safety, jointly where possible and independently where necessary.

Recommendations

We call on the international community to take the following steps:

Require Safety Assurance from Frontier AI Developers. To provide transparency and certainty to domestic authorities around the safety of current and future advanced AI systems, frontier AI developers should be required to both conduct rigorous internal safety and security evaluations and commission third party evaluations, present high-confidence safety cases to relevant authorities, and perform thorough red-teaming exercises before deploying powerful models. For models that cross critical capability thresholds developers should also be required to provide transparency, minimally to home governments and when appropriate to the public, around risks related to planned training runs and internal deployments within their organisation. Post-deployment monitoring systems should be adopted to detect and report emergent risks, severe incidents, and misuse patterns, with clear escalation paths for handling risks should they materialize, up to and including immediate shutdown for severe risks.

Commit to Global Verifiable Behavioural Red Lines Through Enhanced Coordination. To address collective action problems around taking serious safety measures, the international community should establish specific operationalizable globally agreed red lines that AI systems must never cross. These red lines should focus on the behaviour of AI systems – accounting for both the capacity of an AI system to carry out an action, as well as its tendency to do so. To support implementation of these red lines, states could create a technically competent and internationally inclusive coordination body that brings together AI safety authorities to share risk-relevant information and facilitate standard-setting around evaluation protocols as well as verification methods. This body would facilitate knowledge sharing and agreement on technical measures for demonstrating compliance with red lines, including standardised disclosure requirements and assessment protocols that developers can use to credibly demonstrate the safety and security of their AI systems. Over time, verification methods could be mutually enforced through incentives such as conditioning market access on compliance with agreed standards. This coordination mechanism would be a critical first step, though states may need to develop more robust governance structures as AI capabilities advance.

Invest in Research on Safe-By-Design AI Systems. The scientific community and developers should invest in a range of rigorous mechanisms to design safe AI systems. In the short term, there is a clear need to combat AI deception through scalable oversight mechanisms: leveraging other AI models, like “lie detectors”, to assist those judging AI outputs. Other short-term solutions could include investments in information security to guard against threats from insiders, human or AI, and external threats, as well as adopting rigorous robustness techniques to make models more resilient to jailbreaks. In the longer term, we may need to switch from a reactive approach of addressing safety problems as they arise, to building systems that are safe by design.