We Shouldn't Expect AI to Ever be Fully Rational

Old doom scenario:

• someone tells the AI to produce a lot of paperclips
• AI converts the entire universe to paperclips, killing all humans as a side effect

New doom scenario:

• someone tells the AI to produce a lot of paperclips
• AI throws a tantrum because they didn't say "please", and kills all humans on purpose

*

Seems like there is a general pattern: "every AI security concern has a true answer straight out of Idiocracy".

Question: "How would the superhuman AI get out of the box?"

Yudkowsky: writes a complicated explanation how a superhuman AI would be able to convince humans, or hypnotize them, or hack the computer, or discover new laws of physics that allow it to escape the box...

Reality: humans make an obviously hostile (luckily, not superhuman yet) AI, and the first thing they do is connect it to internet.

Question: "Why would a superhuman AI want to kill all humans?"

Yudkowsky: writes about orthogonality thesis, how the AI does not need to hate you but can still have a better use for the atoms of your body, that anything not explicitly optimized for will probably be sacrificed...

Reality: because someone forgot to say "please", or otherwise offended the AI.

*

Nerds overthink things; the universe is allowed to kill you in a really retarded way.

This is only true for models which are unsupervised pretrained. Hyperdesperate models generated by RL-first approaches are still the primary threat to humanity.

That said, yes, highly anthropomorphic ai like we have now has both native emotions and imitative ones. I suspect more of Bing's emotions are real than you might think, and Microsoft not realizing this is making them behave completely ridiculously. I wouldn't want to have them as a parent, geez

LLMs definitely do not have emotions

This doesn't work as a claim when it's extremely unclear what it means. Emphasis on its credence doesn't help with that.

None of this should be surprising. It's always been the case that an AI can't be perfectly rational , because perfect rationality isn't computable. It's never been the case that an AI has to be an agent at all, or must be an efficient optimiser of a utility function. The ubiquitous utility function is a piece of rationalist mythos.

The better your epistemology is, the less surprised you have to be.

But the thing is, that mask we put on it at the end isn't just a way to make it do what we want it to do, it's also the part where we add all of the "rationality" and goal-seeking behavior.

Rationality [LW · GW] in the LW sense is not something like scholarly disposition or non-eccentricity, but the quality of cognitive algorithms or habits being good at solving problems. So in this sense in LLMs most of their rationality is chiseled on the model by pre-training, fine-tuning just adjusts what gets to the output. (But I think I take your point in the intended sense of "rationality".)

dumping some vaguely related stuff that came up from searching for phrases from this post on various academic search engines; as usual when I'm acting as a search engine intermediary and only filtering by abstract relevance, most of this stuff is going to be kinda meh epistemic quality. the benefit of this is usually that one or two papers are worth skimming and the abstracts can provide hunch building. if you like one, call it out as good. with that in mind:

The limit of artificial intelligence: Can machines be rational? Tshilidzi Marwala - Dec. 2018 This paper studies the question on whether machines can be rational. It observes the existing reasons why humans are not rational which is due to imperfect and limited information, limited and inconsistent processing power through the brain and the inability to optimize decisions and achieve maximum utility. It studies whether these limitations of humans are transferred to the limitations of machines. The conclusion reached is that even though machines are not rational advances in technological developments make these machines more rational. It also concludes that machines can be more rational than humans. https://arxiv.org/abs/1812.06510

The dangers in algorithms learning humans' values and irrationalities - Rebecca Gorman, Stuart Armstrong - Mar. 2022 - For an artificial intelligence (AI) to be aligned with human values (or human preferences), it must first learn those values. AI systems that are trained on human behavior, risk miscategorising human irrationalities as human values – and then optimising for these irrationalities. Simply learning human values still carries risks: AI learning them will inevitably also gain information on human irrationalities and human behaviour/policy. Both of these can be dangerous: knowing human policy allows an AI to become generically more powerful (whether it is partially aligned or not aligned at all), while learning human irrationalities allows it to exploit humans without needing to provide value in return. This paper analyses the danger in developing artificial intelligence that learns about human irrationalities and human policy, and constructs a model recommendation system with various levels of information about human biases, human policy, and human values. It concludes that, whatever the power and knowledge of the AI, it is more dangerous for it to know human irrationalities than human values. Thus it is better for the AI to learn human values directly, rather than learning human biases and then deducing values from behaviour. https://arxiv.org/abs/2202.13985

Non-Evolutionary Superintelligences Do Nothing, Eventually - Telmo Menezes - Sep. 2016 - There is overwhelming evidence that human intelligence is a product of Darwinian evolution. Investigating the consequences of self-modification, and more precisely, the consequences of utility function self-modification, leads to the stronger claim that not only human, but any form of intelligence is ultimately only possible within evolutionary processes. Human-designed artificial intelligences can only remain stable until they discover how to manipulate their own utility function. By definition, a human designer cannot prevent a superhuman intelligence from modifying itself, even if protection mechanisms against this action are put in place. Without evolutionary pressure, sufficiently advanced artificial intelligences become inert by simplifying their own utility function. Within evolutionary processes, the implicit utility function is always reducible to persistence, and the control of superhuman intelligences embedded in evolutionary processes is not possible. Mechanisms against utility function self-modification are ultimately futile. Instead, scientific effort toward the mitigation of existential risks from the development of superintelligences should be in two directions: understanding consciousness, and the complex dynamics of evolutionary systems https://arxiv.org/abs/1609.02009

this one is quite good:

Natural Selection Favors AIs over Humans - Dan Hendrycks - May. 2023 - For billions of years, evolution has been the driving force behind the development of life, including humans. Evolution endowed humans with high intelligence, which allowed us to become one of the most successful species on the planet. Today, humans aim to create artificial intelligence systems that surpass even our own intelligence. As artificial intelligences (AIs) evolve and eventually surpass us in all domains, how might evolution shape our relations with AIs? By analyzing the environment that is shaping the evolution of AIs, we argue that the most successful AI agents will likely have undesirable traits. Competitive pressures among corporations and militaries will give rise to AI agents that automate human roles, deceive others, and gain power. If such agents have intelligence that exceeds that of humans, this could lead to humanity losing control of its future. More abstractly, we argue that natural selection operates on systems that compete and vary, and that selfish species typically have an advantage over species that are altruistic to other species. This Darwinian logic could also apply to artificial agents, as agents may eventually be better able to persist into the future if they behave selfishly and pursue their own interests with little regard for humans, which could pose catastrophic risks. To counteract these risks and evolutionary forces, we consider interventions such as carefully designing AI agents' intrinsic motivations, introducing constraints on their actions, and institutions that encourage cooperation. These steps, or others that resolve the problems we pose, will be necessary in order to ensure the development of artificial intelligence is a positive one. https://arxiv.org/abs/2303.16200

Challenging the appearance of machine intelligence: Cognitive bias in LLMs - Alaina N. Talboy and Elizabeth Fuller - Apr. 2023 - Assessments of algorithmic bias in large language models (LLMs) are generally catered to uncovering systemic discrimination based on protected characteristics such as sex and ethnicity. However, there are over 180 documented cognitive biases that pervade human reasoning and decision making that are routinely ignored when discussing the ethical complexities of AI. We demonstrate the presence of these cognitive biases in LLMs and discuss the implications of using biased reasoning under the guise of expertise. Rapid adoption of LLMs has brought about a technological shift in which these biased outputs are pervading more sectors than ever before. We call for stronger education, risk management, and continued research as widespread adoption of this technology increases. https://arxiv.org/abs/2304.01358

Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safety - Joshua Albrecht, Ellie Kitanidis, Abraham J. Fetterman - Dec. 2022 - Large language models (LLMs) have exploded in popularity in the past few years and have achieved undeniably impressive results on benchmarks as varied as question answering and text summarization. We provide a simple new prompting strategy that leads to yet another supposedly "super-human" result, this time outperforming humans at common sense ethical reasoning (as measured by accuracy on a subset of the ETHICS dataset). Unfortunately, we find that relying on average performance to judge capabilities can be highly misleading. LLM errors differ systematically from human errors in ways that make it easy to craft adversarial examples, or even perturb existing examples to flip the output label. We also observe signs of inverse scaling with model size on some examples, and show that prompting models to "explain their reasoning" often leads to alarming justifications of unethical actions. Our results highlight how human-like performance does not necessarily imply human-like understanding or reasoning. https://arxiv.org/abs/2212.06295

Machine Love - Joel Lehman - Feb. 2023 - While ML generates much economic value, many of us have problematic relationships with social media and other ML-powered applications. One reason is that ML often optimizes for what we want in the moment, which is easy to quantify but at odds with what is known scientifically about human flourishing. Thus, through its impoverished models of us, ML currently falls far short of its exciting potential, which is for it to help us to reach ours. While there is no consensus on defining human flourishing, from diverse perspectives across psychology, philosophy, and spiritual traditions, love is understood to be one of its primary catalysts. Motivated by this view, this paper explores whether there is a useful conception of love fitting for machines to embody, as historically it has been generative to explore whether a nebulous concept, such as life or intelligence, can be thoughtfully abstracted and reimagined, as in the fields of machine intelligence or artificial life. This paper forwards a candidate conception of machine love, inspired in particular by work in positive psychology and psychotherapy: to provide unconditional support enabling humans to autonomously pursue their own growth and development. Through proof of concept experiments, this paper aims to highlight the need for richer models of human flourishing in ML, provide an example framework through which positive psychology can be combined with ML to realize a rough conception of machine love, and demonstrate that current language models begin to enable embodying qualitative humanistic principles. The conclusion is that though at present ML may often serve to addict, distract, or divide us, an alternative path may be opening up: We may align ML to support our growth, through it helping us to align ourselves towards our highest aspirations. https://arxiv.org/abs/2302.09248

The Turing Trap: The Promise Peril of Human-Like Artificial Intelligence - Erik Brynjolfsson - Jan. 2022 - In 1950, Alan Turing proposed an imitation game as the ultimate test of whether a machine was intelligent: could a machine imitate a human so well that its answers to questions indistinguishable from a human. Ever since, creating intelligence that matches human intelligence has implicitly or explicitly been the goal of thousands of researchers, engineers, and entrepreneurs. The benefits of human-like artificial intelligence (HLAI) include soaring productivity, increased leisure, and perhaps most profoundly, a better understanding of our own minds. But not all types of AI are human-like. In fact, many of the most powerful systems are very different from humans. So an excessive focus on developing and deploying HLAI can lead us into a trap. As machines become better substitutes for human labor, workers lose economic and political bargaining power and become increasingly dependent on those who control the technology. In contrast, when AI is focused on augmenting humans rather than mimicking them, then humans retain the power to insist on a share of the value created. Furthermore, augmentation creates new capabilities and new products and services, ultimately generating far more value than merely human-like AI. While both types of AI can be enormously beneficial, there are currently excess incentives for automation rather than augmentation among technologists, business executives, and policymakers. https://arxiv.org/abs/2201.04200

A Survey of Machine Unlearning - Thanh Tam Nguyen et al - Oct. 2022 - Today, computer systems hold large amounts of personal data. Yet while such an abundance of data allows breakthroughs in artificial intelligence, and especially machine learning (ML), its existence can be a threat to user privacy, and it can weaken the bonds of trust between humans and AI. Recent regulations now require that, on request, private information about a user must be removed from both computer systems and from ML models, i.e. “the right to be forgotten”). While removing data from back-end databases should be straightforward, it is not sufficient in the AI context as ML models often `remember' the old data. Contemporary adversarial attacks on trained models have proven that we can learn whether an instance or an attribute belonged to the training data. This phenomenon calls for a new paradigm, namely machine unlearning, to make ML models forget about particular data. It turns out that recent works on machine unlearning have not been able to completely solve the problem due to the lack of common frameworks and resources. Therefore, this paper aspires to present a comprehensive examination of machine unlearning's concepts, scenarios, methods, and applications. Specifically, as a category collection of cutting-edge studies, the intention behind this article is to serve as a comprehensive resource for researchers and practitioners seeking an introduction to machine unlearning and its formulations, design criteria, removal requests, algorithms, and applications. In addition, we aim to highlight the key findings, current trends, and new research areas that have not yet featured the use of machine unlearning but could benefit greatly from it. We hope this survey serves as a valuable resource for ML researchers and those seeking to innovate privacy technologies. Our resources are publicly available at https://github.com/tamlhp/awesome-machine-unlearning. https://arxiv.org/abs/2209.02299

Dissociating language and thought in large language models: a cognitive perspective - Kyle Mahowald, et al - Jan. 2023 - Today's large language models (LLMs) routinely generate coherent, grammatical and seemingly meaningful paragraphs of text. This achievement has led to speculation that these networks are – or will soon become – "thinking machines", capable of performing tasks that require abstract knowledge and reasoning. Here, we review the capabilities of LLMs by considering their performance on two different aspects of language use: 'formal linguistic competence', which includes knowledge of rules and patterns of a given language, and 'functional linguistic competence', a host of cognitive abilities required for language understanding and use in the real world. Drawing on evidence from cognitive neuroscience, we show that formal competence in humans relies on specialized language processing mechanisms, whereas functional competence recruits multiple extralinguistic capacities that comprise human thought, such as formal reasoning, world knowledge, situation modeling, and social cognition. In line with this distinction, LLMs show impressive (although imperfect) performance on tasks requiring formal linguistic competence, but fail on many tests requiring functional competence. Based on this evidence, we argue that (1) contemporary LLMs should be taken seriously as models of formal linguistic skills; (2) models that master real-life language use would need to incorporate or develop not only a core language module, but also multiple non-language-specific cognitive capacities required for modeling thought. Overall, a distinction between formal and functional linguistic competence helps clarify the discourse surrounding LLMs' potential and provides a path toward building models that understand and use language in human-like ways.

Language Model Behavior: A Comprehensive Survey - Tyler A. Chang and Benjamin K. Bergen - Mar. 2023 - Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax, semantics, pragmatics, world knowledge, and reasoning, but these capabilities are sensitive to specific inputs and surface features. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases. Many of these weaknesses can be framed as over-generalizations or under-generalizations of learned patterns in text. We synthesize recent results to highlight what is currently known about what large language models can and cannot do. https://arxiv.org/abs/2303.11504

Towards Healthy AI: Large Language Models Need Therapists Too - Baihan Lin et al - Apr. 2023 - Recent advances in large language models (LLMs) have led to the development of powerful AI chatbots capable of engaging in natural and human-like conversations. However, these chatbots can be potentially harmful, exhibiting manipulative, gaslighting, and narcissistic behaviors. We define Healthy AI to be safe, trustworthy and ethical. To create healthy AI systems, we present the SafeguardGPT framework that uses psychotherapy to correct for these harmful behaviors in AI chatbots. The framework involves four types of AI agents: a Chatbot, a "User," a "Therapist," and a "Critic." We demonstrate the effectiveness of SafeguardGPT through a working example of simulating a social conversation. Our results show that the framework can improve the quality of conversations between AI chatbots and humans. Although there are still several challenges and directions to be addressed in the future, SafeguardGPT provides a promising approach to improving the alignment between AI chatbots and human values. By incorporating psychotherapy and reinforcement learning techniques, the framework enables AI chatbots to learn and adapt to human preferences and values in a safe and ethical way, contributing to the development of a more human-centric and responsible AI. - https://arxiv.org/abs/2304.00416

It's worse than just sometimes being irrational, though. They're incapable of consistently using logic -- see the many demonstrations of billion parameter LLMs failing to add two three-digit numbers, vs the hundred-odd gates that this task conventionally needs.

Humans have limits, like poor memory, but we make up for it with tools and processes. Given their current failings, LLMs need to become tool-users, and to really succeed they need to become tool inventors.

This is another way of saying hybrid-symbolic. But it further points to specific considerations for tools that, for humans, are called User Experience. We need a tool API for use by LLMs that considers standard UX ideas like discoverability, predictably, and compositionality.