LessWrong 2.0 Reader

Do you have any idea at all? If you don't, what is the point of 'winning the race'?

Maybe they have some idea but don't want to say it. In recently disclosed internal OpenAI emails, Greg Brockman and Ilya Sutskever said to Elon Musk:

"You are concerned that Demis [Hassabi of DeepMind] could create an AGI dictatorship. So do we. So it is a bad idea to create a structure where you could become a dictator if you chose to" [LW · GW]

Perhaps this - originally private - is saying the quiet part. Perhaps you use your AI to achieve a cultural, espionage, technological, influence, diplomatic, and military victory simultaneously? But why would you declare that beforehand? Declaring it would only invite opposition and competition.

At the very least, you can hack and spy and sabotage other AGI attempts.

To be specific, there are a few areas where, it seems to me, increased intelligence could lead to quick and leveraged benefits. Hacking, espionage, negotiation, finance, and marketing/propaganda. For example, what if you can capture a significant fraction of the world's trading income, attract a large portion of China's talent to turn coat and move to your country, and hack into a large part of an opposition's infrastructure.

If one or more of these tactics can work significantly you buy time for other tactics to progress.

I remember someone here perhaps a year ago had suggested the 1965 flick Flight Of The Phoenix and were trying to maybe get some kind of online rationalist movie club off the ground, though seems perhaps they've deleted their post since searching just now didn't seem to turn it up.

Here's the reasoning I intuitively want to apply:

$(X\to Y)\to P\left[Y|Z\right]\ge P\left[X|Z\right]$

where X = "you roll two 6s in a row by roll N", Y = "you roll at least two 6s by roll N", and Z = "the first N rolls are all even".

This is valid, right? And not particularly relevant to the stated problem, due to the "by roll N" qualifiers mucking up the statements in complicated ways?

Thanks for the thoughts! I've not thought about this topic that much, so my comment(s) will be longer as I'm figuring it out for myself, and in the process of generating hypotheses.

———

I'm hearing you say that while I have drawn some distinctions, that overall these groups still have major similarities, so the term accurately tracks reality and is helpful.

On further reflection I'm more sympathetic to this point; but granting it I'm still concerned that the term is net harmful for thinking.

My current sense is that a cult is the name given to a group that has gone off the rails. The group has 

• some weird beliefs
• intends to behave in line with those beliefs
• seems unable to change course
• the individuals seem unable to change their mind
• and the behavior seems to outsiders to be extremely harmful.

My concern is that the following two claims are true:

1. There are groups with closed epistemologies and whose behavior has a large effect size, in similar ways to groups widely considered to be 'cults', yet the outcomes are overall great and worth supporting.
2. There are groups with closed epistemologies and whose behavior has a large effect size, in similar ways to groups widely considered to be 'cults', yet are not called cults because they have widespread political support.

I'll talk through some potential examples.

Startups

Peter Thiel has said that a successful startup feels a bit like a cult. Many startups are led by a charismatic leader who believes in the product, surrounded by people who believe in the leader and the product, where outsiders don't get it at all and think it's a waste of time. The people in the company work extreme hours, regularly hitting sleep deprivation, and sometimes invest their savings into the project. The internal dynamics are complicated and political and sometimes cut-throat. Sometimes this pays off greatly, like with Tesla/SpaceX/Apple. Other times it doesn't, like with WeWork or FTX.

I'd guess there are many people in this world who left a failed startup in a daze, wondering why they dedicated some of the best years of their lives to something and someone that in retrospect clearly wasn't worth it, not entirely dissimilar to someone leaving a more classical cult. However, it seems likely to me the distribution of startups is well-worth-it for civilization as a whole (with the exception of suicidal AI-companies).

(This is a potential example of number 1 above.)

Religions

Major religions have often done things just as insane and damaging as smaller cults, but aren't called cults. The standard list of things includes oppression of homosexuality and other sexualities, subjugation of women, genital mutilation, blasphemy laws, opposition to contraception in developing countries (exacerbating the spread of HIV/AIDS), death orders, censorship, and more.

It seems plausible to me that someone would do far more harm and become far more closed in their epistemology via joining the Islamic Republic of Iran or the Holy See in the Vatican than if they joined Scientology or other things that get called cults (e.g. a quick googling came up with cryptocurrencies, string theory, Donal Trump, and PETA). Yet it seems to me that these aren't given as examples of cults, only the smaller religions that are easier to oppose and which have little political power. Scientology seems to be the most powerful one where people feel like they can get away with it.

(This is a potential example of number 2 above.)

Education

A hypothesis I take seriously is that schooling is a horrible experience for kids, and the systems don't change because children are often not respected as whole people and can be treated as subhuman.

• Kids are forced to sit still for something like more-than-10% of the hours of their childhood, and regularly complain about this and seem to me kind of psychologically numbed by it.
• I seem to recall a study that all homework other than mathematics had zero effect on learning success, and also I think I recall a study from Scandinavia where kids who joined school when they were 7 or 8 quickly caught up to their peers (suggesting the previous years had been ~pointless). I suspect Bryan Caplan's book-length treatment of education will have some reliable info making this point (even though I believe he focuses on higher education).
• I personally found university a horrible experience. Leaving university I had a strong sense of "I need to get away from this, why on Earth did I do that?" and a sense that everyone there was kind of in on a mass delusion where your status in the academic system was very important and mattered a great deal and you should really care about the system. A few years ago I had a phone call with an old friend from high-school who was still studying in the education system at the age of ~25, and I encouraged them to get out of it and grow up into a whole person.

There's not a charismatic leader here, but I believe there's some mass delusion and very harmful outcomes. I don't think the education system should be destroyed, but I think it probably causes more harm than many things more typically understood to be cults (as most groups with dedicated followings and charismatic leaders have very little effect size either way), and my sense is that many people involved are extremely resistant that they are not doing what's best for the children or are doing some bad things.

(This is a potential example of both numbers 1 and 2 above.)

———

To repeat: my concern is that the things that are common to cults is more like "what groups with closed epistemologies and unusual behavior is it easy to coordinate on destroying" rather than "what groups have closed epistemologies and behavior with terrible effects". 

If so, while I acknowledge that many of the groups that are widely described as "cults" probably have closed epistemologies and cause a lot of damage, I am concerned that whether a group is called a cult is primarily a political question about whether you can backing for destroying it in this case.

Do you hold panpsychism as a likely candidate? If not, then you most likely believe the vast majority of things are not conscious. We have a lot of evidence that the way it operates is not meaningfully different in ways we don't understand from other objects. Thus, almost the entire reference class would be things that are not conscious. If you do believe in panpsychism, then obviously AIs would be too, but it wouldn't be an especially meaningful statement.

You could choose computer programs as the reference class, but most people are quite sure those aren't conscious in the vast majority of cases. So what, in the mechanisms underlying an llm is meaningfully different in a way that might cause consciousness? There doesn't seem to be any likely candidates at a technical level. Thus, we should not increase our prior from that of other computer programs. This does not rule out consciousness, but it does make it rather unlikely.

I can see you don't appreciate my pedantic points regarding language, but be more careful if you want to say that you are substituting a word for what I used. It is bad communication if it was meant as a translation. It would easily mislead people into thinking I claimed it was 'self-evident'. I don't think we can meaningfully agree to use words in our own way if we are actually trying to communicate since that would be self-refuting (as we don't know what we are agreeing to if the words don't have a normal meaning).

pretty small, hard to quantify but I'd guess under 20% and perhaps under 10.

A lot of stuff turns out to hinge on effort. One of the reasons that strength programs work better than generic exercise routines is that with higher reps it's easy to 'tire yourself out' at a level that doesn't actually drive that much adaptation. Think of those fitness classes with weights. Decent cardio, but they don't gain much strength.

When I first learned about social status as a concept, I somehow got the mistaken impression that any kind of status seeking is amoral. This caused me harm because I didn't want to violate any social boundaries, and trying to avoid violating status seeking behavior hobbles your ability to find and follow up on opportunities.

I think status seeking can be zero sum, and in such cases it should be avoided (like playing school with the intention of becoming valedictorian).

Status seeking can be positive sum while consisting of iterated zero sum games (like playing in a tennis club).

Status seeking behavior in positive sum environments generally consists of good things, like working harder at the gym.

The concept is extremely useful to keep in mind when designing environments. What constitutes status seeking should be legible, enable and encourage prosocial behavior, and allow social norms to be learned in a healthy way. Losing in iterated zero-sum games is often a common factor in environments with this attribute, since losing is then an expected outcome of playing, and the game can altered so that an individual loss is seen as providing a gain in knowledge, and continuing to play becomes the source of reward. 

This can be actively implemented into zero-sum social situations by setting up a situation to expose oneself to frequent but non-comprimising losses. Like starting debates to entertain others with the intention of being roasted.

your (incorrect) claim about a single definition not being different from an extremely confident vague definition"

That is not the claim I made. I said it was not very different, which is true. Please read and respond to the words I actually say, not to different ones.

The definitions are not obviously wrong except to people who agree with you about where to draw the boundaries.

My emphasis implied you used a term which meant the same thing as self-evident, which in the language I speak, you did. Personally I think the way I use words is the right one and everyone should be more like me; however, I'm willing to settle on the compromise position that we'll both use words in our own ways.
As for the prior probability, I don't think we have enough information to form a confident prior here.

How large do you think the marginal benefits of doing the full workout you recommend in Updates and Reflections on Optimal Exercise after Nearly a Decade [LW · GW] versus the quicker version in this post?