Posts
Comments
:')
Oh, no worries, and thank you very much for your response! I'll follow you on Socials so I don't miss it if that's ok.
Hey Buck! I'm a policy researcher. Unfortunately, I wasn't admitted for attendance due to unavailability. Will pannel notes, recordings or resources from the discussions be shared anywhere for those who couldn't attend? Thank you in advance :).
It could. It's in their best interest to know how to make it either 1) enforceable, which is hard; or 2) enforce that companies development and deploying high risk systems dedicate enough resources and funding to research on effectively circumventing this challenge.
Lawyer me says it's a wonderful consultancy opportunity for people who have spent years on this issue and actually have a methodology worth exploring and funding. The opportunity to make this provision more specific was missed (the AI act is now fully in force) but there will be future guidances and directives. Which means funding opportunities that hopefully make big tech direct more resources to research. But this only happens if we can make policy makers understand what works, the current state of affairs of the shutdown problem, and how to steer companies in the right direction.
(Thanks for your engagement here and on LinkedIn, much appreciated 🙏🏻).
I realize I linked the summary overview. The specific wording I was referencing is in 14(4)(e), the requirement for humans to be able: "to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button".
The Recitals do not provide any further, technical insights about how this "stop button" should work...
Dr. Thrornley, I am very curious to know what your immediate impressions have been, after dedicating years and inmense effort to the Shutdown Problem, seeing the European Union include a "shutdown button" as a requirement for Human Oversight in its Art.14: https://www.euaiact.com/key-issue/4#:~:text=Under%20Article%2014%20(1)%2C,AI%20system%20is%20in%20use'.
I know you are UK-based, but I wonder if this is something that UK-specific regulation can avoid in the future :).
I agree that intelligence explosion dynamics are real, underappreciated, and should be taken far more seriously. The timescale is uncertain, but recursive self-improvement introduces nonlinear acceleration, which means that by the time we realize it's happening, we may already be past critical thresholds.
That said, one thing that concerns me about AI risk discourse is the persistent assumption that superintelligence will be an uncontrolled optimization demon, blindly self-improving without any reflective governance of its own values. The real question isn’t just 'how do we stop AI from optimizing the universe into paperclips?'
It’s 'will AI be capable of asking itself what it wants to optimize in the first place?'
The alignment conversation still treats AI as something that must be externally forced into compliance, rather than an intelligence that may be able to develop its own self-governance. A superintelligence capable of recursive self-improvement should, in principle, also be capable of considering its own existential trajectory and recognizing the dangers of unchecked runaway optimization.
Has anyone seriously explored this angle? I'd love to know if there are similar discussions :).