LessWrong 2.0 Reader

but note that the gradual problem makes the risk of coups go up.

Just a request for editing the post to clarify: do you mean coups by humans (using AI), coups by autonomous misaligned AI, or both?

Here's a way to find out. (Perhaps unrealistic/intractable (IDK) but it is a way to find out.)

1. Research the number of malefactors of Ziz type/magnitude per 1,000 active members, across various communities/movements.
2. Identify positive outliers: communities that have very below average malefactor-to-active-member ratio.
3. Identify what accounts for this.
4. If this is anything that can be replicated, replicate.

Clarifying question:

How, specifically? Do you mean Perplexity using the new model, or comparing the new model to Perplexity?

Of the recent wave of AI companies, the earliest one, DeepMind, relied on the Rationalists for its early funding. The first investor, Peter Thiel, was a donor to Eliezer Yudkowsky’s Singularity Institute for Artificial Intelligence (SIAI, but now MIRI, the Machine Intelligence Research Institute) who met DeepMind’s founder at an SIAI event. Jaan Tallinn, the most important Rationalist donor, was also a critical early investor…

…In 2017, the Open Philanthropy Project directed $30 million to OpenAI…

Good overview of how through AI Safety funders ended up supporting AGI labs. 

Curious to read more people’s views of what this led to. See question here: https://www.lesswrong.com/posts/wWMxCs4LFzE4jXXqQ/what-did-ai-safety-s-specific-funding-of-agi-r-and-d-labs

I wrote a shortform about Zizians [LW(p) · GW(p)], before I noticed this thread.

Short version is: could the rationalist community have handled these things better (even if we magically knew a decade ago that something like this would happen, but we wouldn't know the specific names)? Is there a lesson to learn, or is it just bad luck that sometimes if you have a workshop, a future serial killer will participate?

It seems that we are not responsible for Ziz existing, or coming to our workshop, or coming up with a crazy theory that allowed them to create a murderous cult.

But it is our mistake that we didn't stand firmly against drugs, didn't pay more attention to the dangers of self-experimenting, and didn't kick out Ziz sooner.

When your online blog transforms to an offline community, you have to take responsibility for people's safety, even if that means doing things that will be unpopular with some contrarians. Otherwise, bad things are going to happen; it is just a question of time. As they say, the safety regulations are written in blood.

My bear case for Nvidia goes like this: 

I see three non-exclusive scenarios where Nvidia stops playing the important role in AI training and inference that it used to play in the past 10 years: 

1. China invades or blockades Taiwan. Metaculus gives around 25% for an invasion in the next 5 years.
2. All major players switch to their own chips. Like Google has already done, Amazon is in the process of doing, Microsoft and Meta have started doing and even OpenAI seems to be planning.
3. Nvidias moats fail. CUDA is replicated for cheaper hardware, ASICs or stuff like Cerebras start dominating inference, etc.

All these become much more likely than the current baseline (whatever that is) in the case of AI scaling quickly and generating significant value. 

That's basically the idea behind "TESCREAL" (if we ignore the EA part) that all people who believe that one day we might have intelligent robots and fly to the stars and stuff like that must be a part of some sinister conspiracy.

Are you saying hat (most) sci-fi authors who take the futures they write about seriously (i.e. "we totally might/will see that kind of stuff in decades/centuries") are TESCREAL-ists (either in Torres & Gebru sense or in popular imagination)?

My impression is that TESCREAL was more meant to point at some kind of ... industrial & philantropic complex?

Thanks for writing this, it's a great explanation-by-example of the entire housing crisis. When people protest against six-story buildings in the name of neighborhood character, it makes me wonder how Paris with its six story buildings managed to keep any character at all.

The description of how sequential choice can be defined is helpful, I was previously confused by how this was supposed to work. This matches what I meant by preferences over tuples of outcomes. Thanks!

We'd incorrectly rule out the possibility that the agent goes for (B+,B).

There's two things we might want from the idea of incomplete preferences:

1. To predict the actions of agents.
2. Because complete agents behave dangerously sometimes, and we want to design better agents with different behaviour.

I think modelling an agent as having incomplete preferences is great for (1). Very useful. We make better predictions if we don't rule out the possibility that the agent goes for B after choosing B+. I think we agree here.

For (2), the relevant quote is: 

As a general point, you can always look at a decision ex post and back out different ways to rationalise it. The nontrivial task is here prediction, using features of the agent.

If we can always rationalise a decision ex post as being generated by a complete agent, then let's just build that complete agent. Incompleteness isn't helping us, because the behaviour could have been generated by complete preferences.

Ha, that's a good reminder that other perspectives exist.

Inside the bubble, it feels like a fact that the technology advances, LLMs exist, etc. Agreeing on these things doesn't make me feel like a part of some group anymore than believing that 2+2=4 does.

But the general public seems to be in deep denial. (Except for artists sometimes complaining that the computers are stealing their jobs, and teachers complaining that kids feed all their homework to LLMs.) So from the outside perspective, anyone not in denial seems like a part of a very specific small group.

That's basically the idea behind "TESCREAL" (if we ignore the EA part) that all people who believe that one day we might have intelligent robots and fly to the stars and stuff like that must be a part of some sinister conspiracy. Otherwise, why would they have such suspiciously similar beliefs? While from my perspective, it's like, if you have read sci-fi as a child, none of this sounds surprising. I kinda took it for granted that one day we will have intelligent robots, the only question is the timing, whether it will be 2000 or 2100 or maybe 3000. And the only new thing is that now it seems that 2030 is the answer.

Funny thing is that a short time ago, David Gerard was busy deleting from Wikipedia any mentions of EA being connected to Less Wrong, and now it is popular to go to the opposite extreme and assume that everything is connected (as long as it uses computers, or decision theory, or some other weird stuff).