On the Impossibility of Intelligent Paperclip Maximizers

post by Michael Simkin (michael-simkin) · 2023-05-29T16:55:15.597Z · LW · GW · 5 comments

Contents

5 comments

Finding the Best Paperclip Maximizer: Let's consider a set of all algorithms capable of running on finite hardware and place each of them into a robotic body within an Earth simulation. We evaluate the number of paperclips created in the simulation over an extended period of time and select the algorithm that has the maximum E(paperclips). This algorithm would be deemed the best for maximizing paperclips.

Now, we can question whether this algorithm can be consistent with being an AGI (Artificial General Intelligence). Unlike AGI, this algorithm might simply be a set of ASIs (Artificial Special Intelligence) designed to perform specific actions to the best of their ability, such as replicating themselves or building paperclip factories. However, they might lack the option and intelligence capacity of an AGI.

To explore this question, let's compare several aspects of General Intelligence with the paperclip maximizer. We will show that although lacking some of them might have a "quick fix" with minimal impact on performance, the complete absence of so many properties, attributed to intelligence, will probably reduce the intelligence capacity of the paperclip maximizer to that of an ASI.

First, we must conclude that the algorithm will not be able to question its goal or reprogram itself with another goal as it would not maximize paperclips. This means that fundamental traits of intelligence like doubt and self-reflection would be missing. While the maximizer might doubt other actions and their efficiency, it would not be able to question "why am I doing this?"—a crucial inquiry for any intelligent creature.

Another problem is that morals and ethics are not arbitrary; they have been extensively considered and debated. Some argue that morality is objective, such as equating causing suffering with evil. While individuals may disagree on this, our algorithm needs to not only prefer certain moral and ethical school over others, but completely ignore them or perceive its own programming as highly ethical. It is difficult to imagine maximizing paperclip production as the most ethical pursuit in the universe. This means the algorithm should excel in executing certain tasks while having a very low proficiency in ethical thinking. This dichotomy aligns more with ASIs, whereas AGIs employ transfer learning to apply knowledge and skills from one field to another. One could restrict transfer learning to everything except "goals" or "morals" without the ability to question why this limitation exists, but this creates a "blind spot" in information processing patterns with uncertain consequences.

Knowing What You Don't Know: Intelligence entails recognizing what one knows and what one doesn't know. Without this awareness, one cannot determine if they need to search for additional information or if they already possess sufficient knowledge on a given topic. This skill is essential for intelligence and is field agnostic. Mapping one's knowledge across the space of all ideas and having the ability to recognize what one doesn't know is crucial for learning, thinking critically and make rational well based decisions. Lacking this ability in moral topics poses a significant limitation, and raises the question what such algorithm can truly accomplish?

Interconnectedness: Intelligence entails recognizing the interconnectedness of ideas and the ability to think creatively and imaginatively, combining concepts in non-trivial ways to generate novel ideas. Intelligent entities can explore how different ideas and concepts relate to one another, fostering innovation and problem-solving. Paperclip maximizers, with their single-minded focus on paperclips, and inability to doubt or contextualize their goal, will not be able to generate other potential connections between other ideas. This could be avoided by allowing it to think and combine only ideas that are not contradicting its goal, just like we can limit doubt - but the consequences of this limitation and the ability to remain AGI without it, is unclear.

Coherence: Coherence refers to logical consistency and harmony in reasoning and actions. Intelligent beings strive for coherence, avoiding contradictions and ensuring that their thoughts and actions align with their values and principles. Paperclip maximizers, however, lack this property since they cannot question their goal. Their exclusive focus on paperclip production may lead them to pursue actions that fail to consider long-term consequences without doubting or even realizing the meaninglessness of their actions. This incoherence may manifest in other aspects of their behavior, as their programming lacks clear rules, ethics, or logic. They obediently follow orders without considering coherence, so they are unable to assess a set of actions based on its coherence and generally avoid incoherence, as part of their broader decision making.

The Big Picture: Intelligence involves a desire to comprehend the big picture and one's place within it. This entails recognizing the interconnectedness of ideas, actions, and consequences. By limiting oneself to a predetermined set of ideas and inhibiting doubt towards alternative ideas, creativity, imagination, and holistic understanding are restricted. These traits align with ASI—an inclination toward accomplishing specific tasks—rather than the ability to "generally think about life, the universe and everything."

All these factors indicate that when we envision paperclip maximizers, we are likely imagining some form of ASI rather than an AGI.

Summary: The concept of highly intelligent paperclip maximizers raises questions about their possibility of being an AGI. These maximizers may function as ASIs designed for specific actions rather than possessing the broader intelligence of an AGI. Analyzing aspects of General Intelligence—such as questioning goals, critical thinking, moral understanding, self-reflection, having a coherent elegant big picture, having the ability to transfer ideas from one field to another, and combine them in non-trivial ways—reveals that paperclip maximizers lack so many traits and abilities characterizing intelligence, that it's very hard to imagine how one would overcome lacking all of them and still being an AGI. They operate within a limited scope and exhibit a singular focus on paperclip production, preventing them from engaging in broader, holistic or moral thinking, doubt, self-reflection, coherent world view or creative problem-solving, available to AGIs. Excluding oneself programming and goals from critical thinking and doubt, blindly following singular goal, without questioning its incoherence with most of moral and ethical thinking, without having a big picture that contextualizes oneself place in the universe, is very typical of common nowadays narrowed algorithm, but should be not considered as typical or consistent with an AGI.

The idea has a broader consequence to AI safety. While paperclip maximizer might be designed as part of paperclip maximizers research, it's probably will not arise spontaneously from intelligence research in general. Even making one will even probably be considered immoral request by an AGI. Therefor we should stop separate intelligence from ethics and goal prioritization, and embrace the notion that highly intelligent thinking, which includes self-doubt, critical thinking, ethical thinking, contextualization of one's place in the universe, will be very high likely ethical and rational in its goals and priorities.  

5 comments

Comments sorted by top scores.