LessWrong 2.0 Reader

update please!

FWIW twitter search is ridiculously bad, it's often better to use google instead. In this case I had it as the second result when I googled "richardmcngo twitter safety fundamentals" (richardmcngo being my twitter handle).

The incentive problem still remains, such that it's more effective to use the price system than to use a command economy to deal with incentive issues:

going by the linked tweet, does "incentive problem" mean "needing to incentivize individuals to share information about their preferences in some way, which is currently done through their economic behavior, in order for their preferences to be fulfilled"? and contrasted with a "command economy", where everything is planned out long in advance, and possibly on less information about the preferences of individual moral patients?

if so, can you not imagine any better way a superintelligence could elicit this information? it does not need to use prices or trade. as some basic examples:

• it could let beings enter whatever they want into a computer in real time.
• it could give them superintelligent nano-factory-devices (aligned to its own values) which create whatever they want locally^[1], in the most seamless way physically possible.

(these are just examples; i personally would expect something more complex and less thing-oriented, around moral patients who are okay with/desire it, where superintelligence imbues itself as computation throughout the lowest level of physics upon which this is possible, and so it is as if physics itself is contextually aware and benevolent)

A potentially large crux is I don't really think a utopia is possible, at least in the early years even by superintelligences, because I expect preferences in the new environment to grow unboundedly such that preferences are always dissatisfied

i interpret this to mean "some entities' values will want to use as much matter as they can for things, so not all values can be unboundedly fulfilled". this is true and not a crux for me. i want the world to be as good as possible, and if the moral patient who wants to make unboundedly many paperclips making unboundedly many paperclips would be less good than other things which could be done with the world, then an aligned agent would choose the best tradeoff.

superintelligence is context-aware in this way, it is not {a rigid system which fails to outliers it doesn't expect (e.g.: "tries to create utopia, but instead gives all the matter to whichever maximizer requests it all first"), and so which needs a somewhat less rigid but not-superintelligent system (an economy) to avoid this}.

1. ^{^}

   (if morally acceptable, e.g. no creating hells)

LVT applies to all land, but not to the improvements on the land.

We do not care about disincentivizing an investment in land (by which I mean, just buying land). We do care about disincentivizing investments in improvements on the land (by which I include buying the improvement on the land, as well as building such improvements). A signal of LVT intent will not have negative consequences unless it is interpreted as a signal of broader confiscation.

Looking back on this post after a year, I haven't changed my mind about the content of the post, but I agree with Seth Herd when he said this post was "important but not well executed". 

In hindsight I was too careless with my language in this post, and I should have spent more time making sure that every single paragraph of the post could not be misinterpreted. As a result of my carelessness, the post was misinterpreted in a predictable direction. And while I'm not sure how much I could have done to eliminate this misinterpretation, I do think that I could have reduced it a fair bit with more effort and attention.

If you're not sure what misinterpretation I'm referring to, I'll just try to restate the main point that I was trying to make below. To be clear, what I say below is not identical to the content of this post (as the post was narrowly trying to respond to the framing of this problem given by MIRI; and in hindsight, it was a mistake to reply in that way), but I think this is a much clearer presentation of one of the main ideas I was trying to convey by writing this post:

In my opinion, a common belief among people theorizing about AI safety around 2015, particularly on LessWrong, was that we would design a general AI system by assigning it a specific goal, and the AI would then follow that goal exactly. This strict adherence to the goal was considered dangerous because the goal itself would likely be subtly flawed or misspecified in a way we hadn’t anticipated. While the goal might appear to match what we want on the surface, in reality, it would be slightly different from what we anticipate, with edge cases that don't match our intentions. The idea was that the AI wouldn’t act in alignment with human intentions—it would rigidly pursue the given goal to its logical extreme, leading to unintended and potentially catastrophic consequences.  

The goal in question could theoretically be anything, but it was often imagined as a formal utility function—a mathematical representation of a specific objective that we would directly program into the AI, potentially by hardcoding the goal in a programming language like Python or C++. The AI, acting as a powerful optimizer, would then work to maximize this utility function at any and all costs. However, other forms of goal specification were also considered for illustrative purposes. For example, a common hypothetical scenario was that an AI might be given an English-language instruction, such as "make as many paperclips as you can." In this example, the AI would misinterpret the instruction by interpreting it overly literally. It would focus exclusively on maximizing the number of paperclips, without regard for the broader intentions of the user, such as not harming humanity or destroying the environment in the process.

However, based on how current large language models operate, I don’t think this kind of failure mode is a good match for what we’re seeing in practice. LLMs typically do not misinterpret English-language instructions in the way that these older thought experiments imagined. This isn’t just because LLMs seem to "understand" English better than people expected—it's not that people expected superintelligences would not understand English. My point is not that LLMs possess natural language comprehension, so therefore the LessWrong community was mistaken. 

Instead, my claim is that LLMs usually follow and execute user instructions in a manner that aligns with the user's actual intentions. In other words, the AI's actual behavior generally matches what the user meant for them to do, rather than leading to extreme, unintended outcomes caused by rigidly literal interpretations of instructions.

Because of the fact that LLMs are capable of doing this, despite in my opinion being general AIs, I believe it’s fair to say that the concerns raised by the LessWrong community about AI systems rigidly following misspecified goals were, at least in this specific sense, misguided when applied to the behavior of current LLMs.

I feel like we're the blind men with the elephant more often than we'd like to admit. A lot of the time when two people make conflicting claims about society, really they're both right about their substrate of society and the world is just twice as big as either thought.

Another shocker for most people: 20 million people in the US live in trailer parks. People with similar life circumstances tend to accumulate in similar places, only see those places, and thus vastly underestimate the diversity of life experience. (This is also true of everything in https://www.lesswrong.com/posts/KpMNqA5BiCRozCwM3/social-dark-matter)

So the story goes like this: there are two ways people think of "general intelligence." Fuzzy frame upcoming that I do not fully endorse.

1. General Intelligence = (general learning algorithm) + (data)
2. General Intelligence = (learning algorithm) + (general data)

It's hard to describe all the differences here, so I'm just going to enumerate some ways people approach the world differently, depending on the frame.

1. Seminal text for the first The Power of Intelligence [LW · GW], which attributes general problem solving entirely to the brain. Seminal text for the second is The Secret of Our Success [LW · GW], which points out that without the load of domain-specific culture, human problem solving is shit.

2. When the first think of the moon landing, they think "Man, look at that out-of-domain problem solving, that lets a man who evolved in Africa walk on the moon." When the second think of the moon landing, they think of how humans problem solving is so situated that we needed to not just hire the Nazis who had experience with rockets but put them in charge.

3. The first thinks of geniuses as those with a particularly high dose of General Intelligence, which is why they solved multiple problems in multiple domains (like Einstein, and Newton did). The second thinks of geniuses as slightly smarter-than average people who probably crested a wave of things that many of their peers might have figured out... and who did so because they were more stubborn, such that eventually they would endorse dumb ideas with as much fervor as they did their good ones (like Einstein, and Newton did).

4. First likes to make analogies of... intelligence to entire civilizations [LW · GW]. Second thinks that's cool, but look -- civilization does lots of things brains empirically don't, so maybe civilization is the problem-solving unit generally? Like the humans who walked on the moon did not, in fact, get their training data from the savannah, and that seems pretty relevant.

5. First... expects LLMs to not make it, because they are bad at out-of-domain thinking, maybe. Second is like, sure, LLMs are bad at out-of-domain thinking. So are humans, so what? Spiky intelligence and so on. Science advances not in one mind, but with the funeral of each mind. LLMs lose plasticity as they train. Etc.

Confirming that efficiently finding a small circuit (you don't actually need further restrictions than size) based on its values on a fixed collection of test inputs is known to imply $NP\subseteq BPP$ --- see this paper.

Intelligence is computation. It's measure is success. General intelligence is more generally successful. 

It's a combination of the way LLM work that they predict the most likely token that very similar to prediction common wisdom and experience of interacting with LLMs.

Pattern matching also matters. After reading the answers from Claude and ChatGPT, you can ask yourself what you expect a person to tell you when you ask them for the top five reason and how likely it is that they will tell you "sports online betting" as one of the top five reasons.