LessWrong 2.0 Reader

So you're basically working with a maximally-shattered model of agency where life consists of a bunch of independent activities that can be fully observed post-hoc and which have no connection between them?

So e.g. if you sometimes feel like eating one kind of food and other times feel like eating another kind of food, you just think "ah, my food preference arbitrarily changed", not "my situation changed to make so that the way to objectively improve my food intake is different now than it was in the past"?

Expected utility is what you have before the outcome of an action is known. Actual utility is what you have after the outcome is known. Here, the utility function has remained the same and you have acquired knowledge of the outcome.

Someone no longer finding a thing valuable that they used to, has either re-evaluated the thing in the light of new information about it, or changed the value they (their utility function) put on it.

Yeah, I agree we need improvement. I don't know how many people it's important to reach, but I am willing to believe you that this will hit maybe 10%. I expect the 10% to be people with above-average impact on the future, but I don't know what %age of people is enough.

90% is an extremely ambitious goal. I would be surprised if 90% of the population can be reliably convinced by logical arguments in general.

I don't agree that I am making unwarranted assumptions; I think what you call "assumptions" are merely observations about the meanings of words. I agree that it is hard to program an AI to determine who the "he"s refer to, but I think as a matter of fact the meanings of those words don't allow for any other possible interpretation. It's just hard to explain to an AI what the meanings of words are. Anyway I'm not sure if it is productive to argue this any further as we seem to be repeating ourselves.

When will this be revealed?

Similarly in Baba is You: when people don't have a crisp understanding of the puzzle, they tend to grasp and straws and motivatedly-reason their way into accepting sketchy sounding premises [LW · GW]. But, the true solution to a level often feels very crisp and clear and inevitable. 

A few of the scientists I’ve read about have realized their big ideas in moments of insight (e.g., Darwin for natural selection, Einstein for special relativity). My current guess about what’s going on is something like: as you attempt to understand a concept you don’t already have, you’re picking up clues about what the shape of the answer is going to look like (i.e., constraints). Once you have these constraints in place, your mind is searching for something which satisfies all of them (both explicitly and implicitly), and insight is the thing that happens when you find a solution that does.

At least, this is what it feels like for me when I play Baba is You (i.e., when I have the experience you’re describing here). I always know when a fake solution is fake, because it’s really easy to tell that it violates one of the explicit constraints the game has set out (although sometimes in desperation I try it anyway :p). But it’s immediately clear when I've landed on the right solution (even before I execute it), because all of the constraints I’ve been holding in my head get satisfied at once. I think that’s the “clicking” feeling.

Darwin’s insight about natural selection was also shaped by constraints. His time on the Beagle had led him to believe that “species gradually become modified,” but he was pretty puzzled as to how the changes were being introduced. If you imagine a beige lizard that lives in the sand, for instance, it seems pretty clear that it isn’t the lizard itself (its will) which causes its beigeness, nor is it the sand that directly causes the coloring (as in, physically causes it within the lizards lifetime). But then, how are changes introduced, if not by the organism, and not by the environment directly? He was stuck on this for awhile, when: “I can remember the very spot in the road, whilst in my carriage, when to my joy the solution occurred to me.”

There’s more going on to Darwin’s story than that, but I do think it has elements of the sort of thing you're describing here. Jeff Hawkins also describes insight as a constraint satisfaction problem pretty explicitly (I might’ve gotten this idea from him), and he experienced it when coming up with the idea of a thousand brains.

Anyway, I don’t have a strong sense of how crucial this sort of thing is to novel conceptual inquiry in general, but I do think it’s quite interesting. It seems like one of the ways that someone can go from a pre-paradigmatic grasping around for clues sort of thing to a fully formed solution.

Thinking further, a key part of it is that temperature has a tendency to mix stuff together, due to the associated microscopic kinetic energy.

with a less than fifty percent change of killing more than one billion people

Typo, 'change' should be 'chance'

predictable

damn that hit me

Thank you very much for the criticism. Jeremias: I genuinely appreciate it. Please allow me to provide some context before commenting. Professor W.J. Wukmir developed the orectic theory, which I used in this post to provide a theory of emotions. The man in question lived in Barcelona, where he arrived around 1960 and was stateless until his death in 1981. Unfortunately, his theories have been lost to the course of time. A few months ago, the last of his disciples passed away. I'm afraid his theory will be forgotten in some second-hand bookstore as a rarity. His disciples said that in his youth (in the early 1930s), Wukmir had intense disputes with Freud and Adler in the Vienna circle to which the three belonged. Freud saw him as a revolutionary, while Wukmir believed that Freud was unable to understand him. I apologize for writing this preview, but my purpose was only to offer light on the origins of this theory.

The critique raises some important issues that should be carefully considered. However, I believe there are some fundamental misunderstandings concerning the nature and purpose of emotions as described by Wukmir's orectic theory, which I will address. I apologize for not being more clear in explaining them. 
Emotions, according to this theory, are not just "shortcut heuristics" but also essential systems of vital orientation. Emotions are more than just quick evaluations in the absence of detail; they are complex, multidimensional evaluations that incorporate multiple aspects of an organism's internal state, external environment, and previous experiences. This valuation process takes place at all levels of an organism, from individual cells to complex brain systems. The goal here is to emphasize the relevance of the "valuation" process rather than focusing solely on the emotions involved. 

Your critique implies a distinction between emotion and cognition, which my argument explicitly rejects. According to the orectic theory, emotion and cognition are inextricably related parts of the same process of vital orientation. Every cognitive act entails emotional valuation, and every emotional response contains cognitive components. This integration is not a "ghost in the machine," but rather an essential component of how living systems process information and make decisions. This is what we require from AI: process information and make decisions.

Your claim that AI can make decisions without emotions ignores the importance of emotions in decision-making, according to orectic theory. It's probably my fault for not explaining it more concisely. Emotions are not an add-on to decision-making; they are essential to the process of analyzing possibilities and deciding courses of action. According to this viewpoint, assigning meaning and value to stimuli and prospective actions is an emotional process.

The notion that ethics can be "arbitrarily programmed" without emotional capacity misses the importance of emotions in moral judgment. In my opinion, ethical decisions are fundamentally dependent on emotional valuations of situations and the resulting outcomes. A strictly logical system of ethics without emotional components would lack the critical ability to attach meaning and value to different outcomes.
Your argument appears to presuppose a restricted definition of intelligence that focuses on information processing and decision-making in a mechanical sense. The theory I described, as well as many modern perspectives in cognitive research, advocate for a more embodied and embedded conception of intelligence that must include emotional processes. From this standpoint, a "fully functioning artificial intelligence" would have to have emotion-like mechanisms.

While it is true that more precise prompts may elicit different answers, the purpose of the GPT-4o example was to demonstrate the limitations of simply language-based AI in grasping emotional context. Human communication is highly reliant on emotional understanding that extends beyond the literal meaning of words, and this understanding is rooted in shared embodied experiences.
 

Finally, the notion that language can completely compensate for the lack of nonverbal emotional cues overlooks the embodied character of emotional cognition. In my opinion, our emotional knowledge is largely based on our bodily experiences and cannot be completely reduced to linguistic descriptions.