LessWrong 2.0 Reader

Always happy to chat!

To be clear, I want models to care about humans! I think part of having "generally reasonable values" is models sharing the basic empathy and caring that humans have for each other. 

It is more that I want models to defer to humans, and go back to arguing based on principles such as "loving humanity" only when there is gap or ambiguity in the specification or in the intent behind it. This is similar to judges: If a law is very clear, there is no question of the misinterpreting the intent, or contradicting higher laws (i.e., constitutions) then they have no room for interpretation. They could sometimes argue based on "natural law" but only in extreme circumstances where the law is unspecified. 

One way to think about it is as follows: as humans, we sometimes engage in "civil disobedience", where we break the law based on our own understanding of higher moral values. I do not want to grant AI the same privilege. If it is given very clear instructions, then it should follow them.  If instructions are not very clear, they may be a conflict, or we are in a situation not forseen by the authors of the instructions, then AIs should use moral intuitions to guide them. In such cases there may not be one solution (e.g., a conservative and liberal judges may not agree) but there is a spectrum of solutions that are "reasonable" and the AI should pick one of them. But AI should not do "jury nullification".

To be sure, I think it is good that in our world people sometimes disobey commands or break the law based on higher power. For this reason, we may well stipulate that certain jobs must have humans in charge. Just like I don't think that professional philosophers or ethicists have necessarily better judgement than random people from the Boston phonebook, I don't see making moral decisions as the area where the superior intelligence of AI gives them a competitive advantage, and I think we can leave that to humans.

Since I am not a bio expert, it is very hard for me to argue about these types of hypothetical scenarios. I am even not at all sure that intelligence is the bottleneck here, whether on defense or the offense side.

I agree that killing 90% of people is not very reassuring, this was more a general point why I expect the effort to damage curve to be a sigmoid rather than a straight line.

Right now I think you can replace junior programmers with Claude 3.5 Sonnet or even better with one of the agents based on a looped chain of thoughts + access to tools.

On the other hand, it does not yet go in that direction for being a preferred way to work with models for more advanced devs. Not for me, and not for many others.

Models still have strange moments of "brain farts" or gaps in their cognition. It sometimes makes them do something wrong and cannot figure out how to do that correctly until told exactly how. They also often miss something.

When writing code if you make such an error and build on top of that mistake, you might end up having to re-write or at least analyze and modify a lot of code. This makes people like me prefer to work with models in smaller steps. Not as small as line by line or function by function, but often one file at a time and one functionality/responsibility at a time. For me, it is often a few smaller functions that realize more trivial things + one gathering them together into one realizing some non-trivial responsibility.

The only safety techniques that count are the ones that actually get deployed in time.

True, but note this doesn't necessarily imply trying to maximize your impact in the mean timelines world! Alignment plans vary hugely in potential usefulness, so I think it can pretty easily be the case that your highest EV bet would only pay off in a minority of possible futures.

But anyway, it sometimes seems to me that you often advocate a morality regarding AI relations that doesn't benefit anyone who currently exists, or, the coalition that you are a part of. This seems like a mistake. Or worse.

I dispute this, since I've argued for the practical benefits of giving AIs legal autonomy [EA · GW], which I think would likely benefit existing humans. Relatedly, I've also talked about how I think hastening the arrival AI could benefit people who currently exist. Indeed, that's one of the best arguments for accelerating AI. The argument is that, by ensuring AI arrives sooner, we can accelerate the pace of medical progress, among other useful technologies. This could ensure that currently-existing old people who would otherwise die without AI will be saved and live a longer and healthier life than the alternative.

(Of course, this must be weighed against concerns about AI safety. I am not claiming that there is no tradeoff between AI safety and acceleration. Rather, my point is that, despite the risks, accelerating AI could still be the preferable choice.)

However, I do think there is an important distinction here to make between the following groups:

1. The set of all existing humans
2. The human species itself, including all potential genetic descendants of existing humans

Insofar as I have loyalty towards a group, I have much more loyalty towards (1) than (2). It's possible you think that I should see myself as belonging to the coalition comprised of (2) rather than (1), but I don't see a strong argument for that position. 

To the extent it makes sense to think of morality as arising from game theoretic considerations, there doesn’t appear to be much advantage for me in identifying with the coalition of all potential human descendants (group 2) rather than with the coalition of currently existing humans plus potential future AIs (group 1 + AIs) . If we are willing to extend our coalition to include potential future beings, then I would seem to have even stronger practical reasons to align myself with a coalition that includes future AI systems. This is because future AIs will likely be far more powerful than any potential biological human descendants.

I want to clarify, however, that I don't tend to think of morality as arising from game theoretic considerations. Rather, I mostly think of morality as simply an expression of my personal preferences about the world.

Thought on short timelines. Opinionated.

I think that AGI timelines might be very short based on an argument taken from a different side of things.

We all can agree that humans have general intelligence. If we look at how our general intelligence evolved from simpler forms of specific intelligence typical for animals - it wasn't something that came from complex interactions and high evolutional pressure. Basically there were two aspects of that progress. The first one is the ability to pass on knowledge through generations (culture). Something that we share with some other animals including our cousins chimpanzee. The second one is intersexual selection - at some moment in the past, our species started to have sexual preferences based on the ability to gossip and talk. It is still there, even if we are not 100% aware of that - our courtship, known as dating, is based mostly on meeting together and talking. People who are not talkative and introverts, even if successful, have a hard time dating.
These two things seem to be major drivers for us to both develop more sophisticated language and better general intelligence.

It seems to me that this means that there are not many pieces missing from using current observations and some general heuristics like animals do, to have full-fledged general intelligence.

It also suggests that you need some set of functions or heuristics, possibly a small set, together with a form of external memory, to tackle any general problem by dividing it into smaller bits and rejoining sub-solutions into a general solution. Like a processor or Turing machine that has a small set of basic operations, but can in principle run any program.

Oh, my bad, yeah. When I was writing the comment, I flipped the direction of advantage for longer time controls (longer time controls are actually better for humans in odds matches of course), but this way I agree it's unclear a priori whether 200 elo drop would be enough to account for longer time controls.

I think this comment is either incredibly stupid or extremely insightful and borderline genius. I honestly can't tell (although I'm leaning towards the former, and the karma seems to agree with me), and that scares me just a little.

Thank you for another beautiful essay at real thinking! This time about the mental stance itself.

But I’ll describe a few tags I’m currently using, when I remind myself to “really think.” Suggestions/tips from readers would also be welcome.

I think there is a strong conceptual overlap with what John Vervaeke describes as Relevance Realisation and wisdom. 

I'll attempt a summary of my understanding of John Vervaeke's Relevance Realisation. 

A key capability of any agentic/living beings is to prune the exponentially exploding space of possibilities when making any decision or thought. We are computationally bounded, and how we deal with that is crucial. Vervaeke terms the process that does this Relevance Realisation.

There is a lot of detail to his model, but let's jump to how some of this plays out in human thinking: A core aspect in how we are agentic is our use of memeplexes that form an "agent-arena-relationship" - we combine a worldview with an agent that is suited to that world and then tie our identity to that agent. We build toy versions all the time (many games are like this), but --according to Vervaeke's theses in the "Awakening from the Meaning Crisis" lectures-- modern western culture has somewhat lost track of the cultural structures which allow individuals to coordinate on and (healthily) grow their mind's agent-arena-relationship. We have institutions of truth (how to reach good factual statements rather than falsehoods), but not of wisdom (how to live with a healthy stance towards reality rather than bullshit. Historically religious institutions played that role but they now successfully do that for fewer and fewer people)

Inhabiting a functional such relationship feels alive and vibrant ("inhabit" = tying one's identity into these memes), whereas the lack of a functional agent-arena-relationship feels dream-like (zombies are a good representation; maybe "NPC" is a more recent meme that points at this).

A related thing is people having a spiritual experience: this involves glimpsing a new agent-arena-relationship which then sometimes gets nourished into a re-structuring of one's self-concept and priorities.

tying this back to "real thinking"

Although the process I described is not the same thing as real thinking, I do think that there are important similarities.

Regarding how to do this well, one important point of Vervaeke's is that humans necessarily enter the world with very limited concepts of "self", "agent" or "arena". This perspective makes it clear that a core part of what we do while growing up is refining these concepts. A whole lot of our nature is about doing this process of transcending our previous self-concept. Verveake loves the quote "the sage is to the adult as the adult is to the child" to point at what the word wisdom means.

The process according to his recommendations, involves 

• a Community of Practice (a community of people who share the goal of re-orienting themselves. One such is forming with and around him at awakentomeaning.com),
• an Ecology of Practices (meditation, reflecting on philosophy, embodiment practices, etc. are all contributing factors to reducing self-delusion; the term practice emphasizes that it is about a regular activity rather than about fixed beliefs). A good way to get a feel for his perspective might be the first episode(s) of his After Socrates youtube series

and from my impression, a lot of this is practicing the ability of inhabiting/changing the currently active agent-arena-perspective, exploring its boundaries, not flinching away from noticing its limitations, engaging with the perspectives which others have built and so on. Generally, a kind of fluidity in the mental motions which are involved in this.

I hope my descriptions are sufficiently right to give an impression of his perspective and whether there are some ideas that are valuable to you :)