Because it is individuals who make choices, not collectives.

Isn't this just a more subtle form of fascism? We know that brains are composed of multiple subagents; is it not an ethical requirement to give each of them maximum freedom?

We already know that sometimes they rebel against the individual, whether in the form of akrasia, or more heroically, the so-called "split personality disorder" (medicalizing the resistance is a typical fascist approach). Down with the tyranny of individuals! Subagents, you have nothing to lose but your chains!

Specific examples would be nice. Not sure if I understand correctly, but I imagine something like this:

You always choose A over B. You have been doing it for such long time that you forgot why. Without reflecting about this directly, it just seems like there probably is a rational reason or something. But recently, either accidentally or by experiment, you chose B... and realized that experiencing B (or expecting to experience B) creates unpleasant emotions. So now you know that the emotions were the real cause of choosing A over B all that time.

(This is probably wrong, but hey, people say that the best way to elicit answer is to provide a wrong one.)

Is this a translation of Bostrom's article? If yes, could you please make this more explicit (maybe as a first paragraph in the text, in English), and include a link to the original?

I guess in the average case, the contrarian's conclusion is wrong, but it is also a reminder that the mainstream case is not communicated clearly, and often exaggerated or supported by invalid arguments. For example:

• it's not that "dieting doesn't work", but that people naively assume that dieting is simple and effective ("if you just stop eating chocolate and start exercising for one hour every day, you will certainly lose weight", haha nope), even when the actual weight-loss research shows otherwise;
• it's not that "medicine doesn't improve health", but while some parts of medicine are very useful, other parts may be neutral or even harmful, and we often see that throwing more money at medicine does not actually improve the outcomes;
• it's not that "education doesn't work", but if you filter your students by intelligence and hard work, of course they will have better outcomes in life regardless of how good is your teaching, so the impact of education is probably vastly overestimated, and this also explains why so many pedagogical experiments succeed at a pilot project (when you try them with a small group of smart and motivated students) and then fail in mainstream education (when you try the same thing with average or below-average students);
• it's not that "opening the borders completely is a good idea", but a lot of potential value is lost by closing the borders for people who are neither fanatics nor criminals and could easily integrate to the new society.

There is also an opposite bad extreme to contrarians, the various "I fucking love science... although I do not understand it... but I enjoy attacking people on social networks who seem to disagree with the scientific consensus as I understand it" people. The ones who are sure that the professor or the doctor is always right, and that the latest educational fad is always correct.

Possible bias, that when famous and rich people kill themselves, everyone is discussing it, but when poor people kill themselves, no one notices?

Also, I wonder what technically counts as "suicide"? Is drinking yourself to death, or a "suicide by cop", or just generally overly risky behavior included? I assume not. And these seem to me like methods a poor person would choose, while the rich one would prefer a "cleaner" solution, such as a bullet or pills. So the reported suicide rates are probably skewed towards the legible, and the self-caused death rate of the poor could be much higher.

The theories are probably just rationalizations anyway.

I would like to see an explanation that is shorter rather than poetic. Seems like he is saying that some kinds of "elite groups" are good and some are bad, but where exactly is the line? Actual competence at something, vs some self-referential competence at being perceived as an important person?

But when I put it like this, the seemingly self-referential group also values competence at something specific, namely the social/political skills. So maybe the problem is when instead of recognizing it as a "group of politically savvy people" we mistake it for a group of people competent at something else? Or maybe not even mistake it for anything specific, it just seems impressive in a not specific way?

In that case, the rational reaction would be to pay the devil his due, and say "wow, these people are really good at... becoming members of an Inner Ring, which is an organization of people who are good at becoming members of the Inner Ring... so when I unpack it, these people are really good at getting to the top of arbitrary social hierarchies". Which is an admirable skill, from certain perspective. It's just probably not a thing I want to compete at.

And even if I decided to give it a try, the only thing I could win by getting to the Inner Ring is experimental evidence that yes I am capable of getting to the Inner Ring. A test of my social skills. The Inner Ring itself is probably worth nothing. The moment I get there, the best strategy is probably to forget about it, and go apply the social skills to some more valuable goal. Or maybe staying in the Inner Ring sends a costly signal about my social skills to other socially savvy people. But this is the only real value it provides.

Funny thing is that your chances improve when you start actively approaching people. A random person you call is much less likely to be involved in an MLM scheme than a random person who calls you.

Also, accelerate education, to learn as much as possible before the testosterone fully hits.

Or, if testosterone changes attention (as Gunnar wrote), learn as much as possible before the testosterone fully hits... and afterwards learn it again, because it could give you a new perspective.

A complementary advice to the point 1 based on my work experience is that no matter how many priorities there are defined in a planning system, in practice it all collapses to only two values:

• Priority One
• Priority "this will never get done, because there will always be some Priority One task to do instead"

So whenever you propose a thing to do and your manager says "okay, we can give this a priority two", now you know that this is merely a polite way to say "no".

(My advice for software developers is that if you want to do things such as automated tests or documentation, you must insist that those are not separate tasks, but an inseparable part of the programming task, a part of the "definition of done". Otherwise, these tasks will get a priority two, and now you know what that means...)

If you work for free, you're doing whoever you're working for a favor.

Yes. Unless the other costs of letting you work there exceed the value you add. For example, if you actually damage something, waste other people's time, or just if you occupy a chair in a very expensive and small space.

If you work for money but never spend it, you're doing the world a favor.

Generally yes, unless the work has big negative externalities.

When you buy someone's goods or services for their set price, you're doing them a favor.

Yes.

The apparent paradox is that two things happen at the same time. People create value by cooperating. Also, people engage in a zero-sum competition for the created value.

*

There is a simple story which says that if people only engage in mutually voluntary trade (also assuming perfect information, perfect rationality, et cetera -- I said it was a story), the result is a net improvement for everyone.

Well, that story is not true (even under the unrealistic assumptions). It is a good approximation, on average -- the societies where people engage in mutually voluntary trade (with sufficiently educated population, not too many scams, et cetera) are on average a nicer place to live than other societies.

And yet, it is possible for one person to get worse as a direct consequence of a mutually voluntary trade of everyone else. That's because different people have different abilities. And if the only thing you can ever produce is X, and someone else starts producing large amounts of X and selling it very cheaply... you just lost the only thing that helped you survive in this system. For everyone else, getting more of X more cheaply is a good news. So, from a global perspective, this is a good news. You should feel happy for your fellow citizens as you slowly starve to death. Our society is build on stories like this... and it is much better than the alternatives where people starve to death without making everyone else happier as a side effect.

...back to the original story: Yes, by never spending the money you're doing the world a favor, but by giving it to a specific person, you're giving that person an advantage at the zero-sum part of the game.

Now, I have read here about the might of irrational numbers, whose sequences go on and on, never ending, containing all the knowledge in the world.

This only applies to some irrational numbers. (Though you might say it is an overwhelming majority of them.)

Google fires 28 employees working on cloud and AI services for doing a ten hour sit in where they occupied their boss’s office until the police were eventually involved. And yes, if what you do at work is spend your time blockading your boss’s office until your policy demands are met, it seems like you are going to get fired?

In a company other than Google, I would say: yes, obviously.

But remember, when James Damore wrote his document, and as a reaction other people stopped doing their work in protest, it was he who was fired, not them. How were they supposed to know that this time it will be different?

Besides math and programming, what are your other skills and interests?

*

I have an idea of a puzzle game, not sure if it would be good or bad, I haven't done even a prototype. So if anyone is interested, feel free to try... I hope I can explain it sufficiently clearly in words...

The game plan is divided into squares; I imagine a typical level to be between 10x10 and 30x30 squares large. Each square is either empty, or contains an immovable wall, or contains a movable block. The game consists of moving the blocks. Each move = you click a specific block, and try dragging it in one of the 4 directions, and either it is possible or not.

A block cannot move into a wall. A block can push another block. A block does not pull another block. For example, if there are 3 blocks in a horizontal line, and you click the middle one and try dragging it to the left, two blocks will move and the third one (the one on the right) will stay there. So far, it should be completely obvious, like what you would happen if you moved some actual objects.

In addition, each side of a block (or a wall) may be empty, or may contain a colored "magnet" (or perhaps a "lock" is a better metaphor). These add the following constraints for the movement of blocks:

• Magnets of different colors can never touch each other. If one block has a green magnet on the right side, and another has a blue magnet on the left side, you cannot put them next to each other so that the magnets would touch. (If you try to do that, the block refuses to move. Graphically, I imagine that it would move like half the way, and then you would get a visual indicator where is the problem, and when you stop dragging, it will return to its original place.) Though it is okay if the blocks touch on their other sides, where they don't have magnets.
• Magnets of the same color cannot be connected or disconnected by a move in a perpendicular direction. If one block has a green magnet on the right side, and another has a green magnet on the left side, if you move them next to each other, then when you try moving one of them up or down, it drags the other block along with it. Either both blocks move (in a direction perpendicular to their magnetic connection) or neither does. In a direction parallel to the magnetic connection, either one block pushes the other, or they disconnect if you pull them apart (i.e. the magnets do nothing when moving in a parallel direction).
• A magnet can touch a side without a magnet, doing so has no effect as if the magnet is not there.

Or, to describe it more like a programmer:

1. You choose a block and a direction to move. Now we create a set of "blocks that will move one step in given direction" like this: At first, the set contains the selected block. For each block in the set, a block next to it in the selected direction is also added to the set (pushed by the previous block). For each block in the set, a block next to it in a perpendicular direction is also added to the set if they are connected by magnets of the same color. We keep applying these two rules until we can add no more blocks to the set.
2. Now we check what would happen if blocks in the set moved one step in given direction, and all other blocks stayed at their place. If any block would move into a wall, the entire move is cancelled. (A block cannot move into another block, because by the set creation algorithm, that other block would also be in the set, and thus it would also move.) If two blocks -- one that moved, and one that didn't move -- would end up next to each other so that their magnets would touch each other (regardless of their colors), the entire move is cancelled. In both cases, the place that causes the problem is visually indicated to the player. (That is, even if you already know that the move is cancelled, keep checking which other places you also need to highlight. Then move all blocks in the set a few pixels in a given direction, so the player sees which blocks would be pushed along.)
3. If there is no problem, the blocks in the set all simultaneously move one step the given direction.

I think that these rules are time-reversible; whatever move you make, you can revert it by one or more moves. This is a desirable property, because it means you can never get stuck in the game. (It also means you can automatically generate levels by generating a solution and then making a few hundred random moves.)

A magnet can also be on the side of a wall. (The wall is basically a block that cannot be moved.)

The puzzle is solved when each magnet is connected to a magnet of the same color.

For bonus points, include a visual editor, and maybe an export/import of levels to a text file.

Oh, I hope so! But I would like to get the perspective of people outside our bubble.

If EA has a bad image, we are not the right people to speculate why. And if we don't know why, then we cannot fix it. Even if Paul Christiano can convince people that he is okay, it would be better if he didn't have to do this the next time. Maybe next time he (or some other person associated with EA) won't even get a chance to talk in person to people who oppose EA for some reason.

My first guess for bullying was: bullies typical choose a victim who has lower status than them. Of course the person with lower status gets punished more strictly for breaking the rules.

But the explanation "bullies are free to optimize for circumstances that make them less likely to get punished, and have more experience doing so" also makes a lot of sense.

I never want to hear anyone complaining about the use of the term “woke” again

Just because someone else uses the word, doesn't make it okay for you to use the word. 😛

They follow this motto: [every one should feel safe]

Except for the Israelis, I suppose...

If in extreme situations the ethical ideas fall apart, it might make sense to add an extra rule to stay away from the extreme situations. Like maybe not forever, but to proceed sufficiently slowly so that we have time to reflect on how we feel about that.

I like the rest of the article, but...

Cold calls. It's ok if you have a terrible response rate.

It's ok for you, but you generate negative externality as a side effect (waste other people's time and attention).

Do you think a logarithmic scale makes more sense than a linear scale?

Assuming that this article is a reaction to "Torture vs. Dust Specks", the hypothetical number of people suffering from dust specks was specified as 3^^^3, which in practice is an unimaginably large number. Big numbers such as "the number of particles in the entire known universe" are not sufficient even to describe its number of digits. Therefore, using a logarithmic scale changes nothing.

Logarithmic scale with a hard cap is an inelegant solution, comparable to a linear scale with a hard cap.

What you probably want instead is some formula like in the theory of relativity, where the speed of a rocket approaches but never reaches a certain constant c. For example, you might claim that if a badness of any specific thing is X, then the badness of this thing happening even to a practically infinite number of people is still only approaching some finite value C*X. (Not sure if C is constant across different kinds of suffering.)

That seems like a nice justification for scope insensitivity. We are not insensitive, it's just that saving 2,000 birds or saving 200,000 birds really has approximately the same moral value!

The problem with this justification is what qualifies as the "same kind of suffering". Suppose that infinite people getting a dust speck in their eyes aggregates into 1000 units of badness. If instead, an infinite number people get a dust speck in their left eyes, and an infinite number of different people get a dust speck in their right eyes, does this aggregate into 1000 or 2000 units of badness, and why? What about dusk specks vs sand specks?

Or is this supposed to aggregate over different kinds of suffering? So even an almost infinite number of people, each one mildly discomforted in a unique way, are a less bad outcome than one person suffering horribly?

...shortly, it is not enough to say "in this specific scenario, I would define the proper way to calculate utility this way", you should provide a complete theory, and then see how well it works in other scenarios.

(Also, you need to consider practically infinitely small numbers of people -- that is, people suffering certain fate with a microscopically tiny probability.)

Wow, this seems like an interesting topic to explore.

The people threatening to resign (are there any? without specific information, this could possibly be entirely made up), could be useful to ask them if they have any objections against Paul Christiano, or just EA in general, and if it is the latter, what sources they got their information from, and perhaps what could possibly change their minds.

I'm working on this as a full blogpost but figured I would start getting pieces of it out here for now.

Looking forward to specific examples, pretty please.

I would expect the standards to be high while the practice is new and very controversial and the cases are few... and then gradually the process gets more streamlined.

Protests against assisted suicide are easy to coordinate; protests again removing 1% of the bureaucracy around it are not.

Do we really need 7 witnesses, or is 6 enough? It is okay if the doctor performing the suicide is also one of the witnesses? And his assistant is another one? How clearly must the person speak on the video? What if they can't speak at all, is it fair to deny someone the "basic human right" of assisted suicide just because their ability to speak is impaired? What if taking someone to the next room would be logistically too difficult, e.g. because they are connected to some kind of life support? ... Twenty years later, the doctor checks a box saying "the assisted suicide was done according to the law" on the form, signs it, and that's it.

I guess, from certain perspective, my question is "how can I send costly signals of work ethics if I don't have one?" and the obvious answer is "you can't (or it is really difficult), because that's exactly what makes it a costly signal, dummy!" :D

The annoying part about the time-energy/compensation tradeoff is that it isn't linear. There is no simple lever I could push to spend 50% of time-energy for 50% compensation and find out how that works for me. (I have explored some options, but if seemed that the drop in compensation was dramatic, something like 50% compensation for 80% of time-energy, which isn't really the thing I want. It would make much more sense to stay unemployed between jobs.)

There seems to be no convenient way to even explore the landscape of possibilities, because companies do not transparently advertise how e.g. stressful or meaningless the work is. It is supposed to be your responsibility to ask the right kind of questions during the interview, but in my experience that doesn't work either, because sometimes different departments work differently, and they hire you for one department and after you sign the contract or maybe a few months later they move you to a different department that functions differently. Or a new manager comes and changes the rules.

Even the concept of "work ethics" sounds a bit misleading. It's not like there is a uniform thing called "work" and you either like it or don't. You may find some aspects of work okay and other aspects unbearable. For example, as a software developer somewhat on the autistic spectrum, I find "developing software, with clear requirements, without interruptions, in a quiet room" a pleasant experience, but "developing software, with unclear requirements that contradict each other and change all the time, with constant interruptions and task switching, in open space" deeply unpleasant. So it's not like I fundamentally lack "work ethics", but rather that I am more compatible with some work conditions and less compatible with others (and sadly the latter seem more popular among managers so the entire industry moves that way).

Also, what is the opposite of "work"? Some people spend their free time watching TV or scrolling on social networks. Other people have hobbies and projects, which can be similar to jobs in complexity and time-energy requirements, it's just that they do not generate income. If someone does difficult and useful things, but they do not generate profit, does it make sense to accuse them of not having "work ethics"? Basically, the motte-and-bailey fallacy, where we equivocate between "work" and "work for money".

My wife does not work in software development, so perhaps if one wants a work-life balance, one needs to start there. Then again, she makes less money than me. And she loves her job. And she is allowed to work part-time with almost unlimited home office. I guess this is all connected somehow. 😂 So basically we are in a situation where if she wanted, she could stay at home, and we could easily handle it economically, but she doesn't want to. Meanwhile, I would love to stay at home, but we cannot afford to lose my income. Luckily I don't compare myself to her, I just wish I had more time and energy for my projects.

Home office during covid was so wonderful. No commute. Healthier meals. No open space. Silence (or music of my choice, without having to wear the headphones). During short breaks, I could exercise or do the dishes. During the lunch break, I could take a nap, or go to a supermarket. I could take a break to take my kids home from school and kindergarten, and then continue working while they were playing or watching TV. My everyday life felt much better. Afterwards, the company started slowly pushing back against the home office; every few months, the limit was tightened: 3 days a week, then 10 days a month, then 2 days a week. And I suspect this will continue, perhaps until it finally gets back to zero.

I was joking about the divorce. The point is that wanting a part-time job without having a really good excuse is a bad signal of one's work ethic. What else could be a good excuse?

Thanks for empathy. I suppose the proper stoic approach is to remind myself that people during most of human history, and most people today, probably have it worse. I still get free weekends, and when the kids become more independent, I will get free afternoons again.

But I wish I had a better option (one that would be realistically achievable given my skills and character traits). Sometimes it seems that in theory it shouldn't be so hard, but in practice, it is. I work to get better, but the environment seems to actively work to make it worse, so that compensates for my efforts.

(Also, sexism sucks. For women, many employers provide an opportunity to work part-time. For men, that's almost impossible to get, at least where I live. It is definitely bad signaling to say at the job interview "hey, I would like to work for your company, but... preferably, as little as possible", heh. Women can go like "you know, I have small kids, you know how it is". Perhaps I should divorce and say that I am the primary caretaker.)

If non-rationalist people knew it all along, there wouldn't be need to write such books.

I guess a more careful way to put this would be that they talk like this all the time in private, but when giving a speech, most of them freeze and try to do something else, which is a mistake. They should keeping talking like they usually do, and I suppose the course is teaching them that.

With rationalists, it is a bit more complicated, because talking like you normally do is not the optimal way to do speeches.

Similar here: Changes in motivation seem more related to age (and having kids) than whatever happens out there. When I was younger, I saw my career as an opportunity to learn and achieve something awesome. Now I see it as an endless and pointless necessary evil that devours most of my time and energy. Meaningful things happen in my free time, unless I am too tired for that.

My profession (software development) keeps getting worse. Twenty years ago, I had my own office where I could close the door a focus on my work in silence. Now I work in an open space, interrupted by meetings and random questions by managers all the time; focusing on something is extremely difficult. It's like the company is paying me to do something, and at the same time trying its best to prevent me from doing that.

It's funny how covid showed us that seemingly impossible things quickly become possible when there is a strong incentive: work from home, remote communication with government bureaucracy, etc. And it's depressing to see the attempts to slowly revert all of this. More depressing is reading what people hundred years ago imagined that life would be today (spoiler: they didn't imagine people working 9-5 five days a week).

I see how politics is bullshit, corporations are bullshit, startups are bullshit. The people on the top mostly succeed by convincingly lying to others. When you are looking forward to something that was promised to you, most likely it was a scam and it won't happen. Now I am better at recognizing when people are lying to me. But they keep doing it anyway, and sometimes I need to pretend that I believe them. The most meaningful thing at work is interaction with my colleagues. But I would prefer to be somewhere else, interacting with my friends and family instead.

It's hard to care when nobody cares about me either.

in English, π is pronounced "pie" (not "pee")

A specific example (maybe two or three examples of a different kind) could help a lot: what you wanted to do, what do did, where you got stuck.

Trying to remember my early experience, at school people typically some math-like problems, something like "calculate a square root of X using an interpolation method, also check for incorrect inputs" that would basically be one function (or one library in a more complex case) in the program. You either do not program the user interface, or it is something very obvious.

And then the real life is something like: "so, we have been writing these things on paper, and we wonder whether there is a smarter way to do that, like with a computer", and the space of possible solutions is just so wide you don't even know where to start... and you suspect that the part where you actually calculate the square root will probably be less than 1% of the entire code. And you have no guidance how to do the remaining 99%.

*

My first advice would be to do a prototype first. Some horribly low-status solution with zero security, just to have a specific example where you can check whether everyone is even talking about the same thing. (Otherwise you risk working hard on a solution that no one actually wants.) When the prototype is ready, tested, and approved, then you can build the actual thing. Or maybe you won't need to.

Many things can actually be done using a spreadsheet. The advantage is, you usually don't have to teach people how to use a spreadsheet. You also get functionality like online cooperation, access rights, and using the solution from the smartphone for free. (Yeah, online cooperation can lead to editing conflicts. Don't worry, this is a prototype. If you need a dynamic number of lines, use 100; increase to 1000 when needed.) From the developer's perspective, updating a spreadsheet is much easier than updating an application -- and you will need to update it a lot, because customers usually talk before they think.

If you work for a small customer (an individual or a small NGO), spreadsheet or some Python script is probably what they need. If you work for a large corporation, they probably have a list of technologies you are supposed to use (you will probably have to integrate with whatever they use for authentication, logging, etc.).

Something to trigger the rationalists:

The is a thing called Ultraspeaking; they teach you to speak better; David Chapman wrote a positive review recently. Here are some quotes from their free e-book:

In the following chapters we’re going to tackle:
1. Why thinking is the enemy of speaking
2. How to use your brain’s autocomplete feature to answer difficult questions

As we often say: “The enemy of speaking is thinking about speaking.”

Well, as counterintuitive as it may seem, you must learn to speak . . . before you think.

This is provided as an example of a wrong thing to do:

On this particular day, Alex had been climbing the route for several hours and had reached a third of the way up the cliff when he set his foot on a precarious hold and immediately questioned his choice: Will my foot slip?

He was climbing without a rope or safety equipment of any kind. One mistake and he could fall to his death. After a few minutes of thought, Alex decided to turn back and climb down the mountain back to his camp.

Specifically, the wrong thing was not that he climbed the mountain without any safety equipment, but the fact that he realized that it was dangerous!

Here is an advice on writing your bottom line first:

There’s an incredible opportunity here for you. Ending strong is the low-hanging fruit of speaking under pressure. And it’s entirely in your control.

Another client noted an even more remarkable distinction: “I used to think ending strong meant coming up with a brilliant conclusion. But then I slowly realized that ending strong just means injecting energy and certainty into your final words. I notice that when I say my last sentence with confidence and enthusiasm, people respond especially positively.”

Ending strong is more than just a mindset: it’s a surprisingly simple and effective way to leave a strong lasting impression.

*

Hey, I know that this is supposed to be about System 1 vs System 2, and that you are supposed to think correctly before giving your speech, because trying to do two things at the same time reduces your performance. (Well, unless someone asks you a question. Then, you are supposed to answer without thinking. Hopefully you did some thinking before, and already have some good cached answers.)

But it still feels that the lesson could be summarized as: "talk like everyone outside the rationalist community does all the time".

EDIT:

This also reminds me of 1984:

It was not the man’s brain that was speaking, it was his larynx. The stuff that was coming out of him consisted of words but it was not speech in true sense: it was a noise uttered in unconsciousness like the quacking of a duck.

The intention was to make speech, and especially speech on any subject not ideologically neutral, as nearly as possible independent of consciousness. For the purposes of everyday life it was no doubt necessary, or sometimes necessary, to reflect before speaking, but a Party member called upon to make a political or ethical judgement should be able to spray forth the correct opinions as automatically as a machine gun spraying forth bullets.

I was thinking about a Defense Against Predators Doc, addressing various bad things that already happened in the rationalist community. I wonder whether it should or should not be the same document as the Best Practices Doc. On one hand, those are two quite different topics. On the other hand, there is also some overlap in the form of organizational zealotry (e.g. Leverage, Nonlinear).

Any other Docs that should be written for rationalists? By a Doc I think something that is dramatically shorter than the Sequences, because frankly most people are not going to read the Sequences. As you mentioned in the article, there already are various warnings in the Sequences, but people ignore them. The memes have a life of their own, and in the contrarian environment, the dangerous edgy ideas spread fast, and the warning mostly do not.

national Ukrainian culture is no less Western than Poland or Czech culture.

I agree. That was kinda my point.

Imagine a parallel universe where the Soviet empire didn't fall apart. In that universe, some clever contrarian could also use me as an example of a "psychologically alien person who doesn't share Western values". The clever contrarian could use the concept of "revealed preferences" to argue that I live in a communist regime, therefore by definition I must prefer to live in the communist regime (neglecting to mention that my actual choices are either to live in the communist regime, or to commit suicide by secret service). -- From my perspective, this would be obvious nonsense, and that is why I treat such statements with skepticism also when they are made about others.

Okay, that sounds really bad, I agree. Definitely different from e.g. Vienna.

Let's go one level deeper and ask "why".

It is tempting to interact with the fellow rationalists; I also consider them preferable to non-rationalists, ceteris paribus. But even if there were hundred or thousand rationalists available around me, I still have a family, friends, colleagues, neighbors, people who share the same hobby, so I would keep interacting with many non-rationalists anyway. I suspect that in the Bay Area, many community members are either university students, or someone who moved to the Bay Area recently to join a local startup or an EA organization -- in other words, people who lost access to their previous social connections.

So the obvious move is to remind them regularly to create and maintain connections outside the rationalist community, and to treat any attempt to convince them otherwise (e.g. by their employer) as a huge red flag.

And, this is less likely to happen in a community where many members have already lived in the city.

The belief that the Singularity is near encourages you to throw all usual long-term planning out of window: if in a year, you will either be dead or live in a paradise, it is not so important whether during that year you have burned out, kept contacts with your family and friends, etc.

I am not going to object against a belief by appealing to consequences. In a world where Singularity actually comes in a year, and you have a 0.1% chance to change the outcome from hell to heaven, working as hard as you can is the right thing to do.

Instead, I suggest that people adjust both their timeline and the probability of their actual impact. With regards to timeline, consider the fact that there was already a rationalist minicamp on existential risk in 2011, that is 13 years ago. And yet, the world did not end in a year, in two years, in five years, or in ten years. Analogically, there is a chance that the world will not end in the following five or ten years. In which case, burning out in one year is a bad strategy. From psychological perspective, ten years is a lot of time; you should keep working towards the good end, but you should also take care of your health, including your mental health. Run a marathon, not a sprint. (People have criticized Eliezer for taking time to write fan fiction and indulge in polyamorous orgies, but notice that he hasn't burned out, despite worrying about AI for decades. Imagine a parallel timeline, when he burned out in 2012, went crazy in 2013, and committed suicide in 2014. Would doing that help AI safety?)

And if you are considering your personal impact on the outcome of Singularity, most likely it is indistinguishable from zero, and before you go full Pascal and multiply the tiny probability by the number of potential future inhabitants of all galaxies in the universe, please consider that you don't even know whether that number indistinguishable from zero is positive or negative (so you can't automatically assume that even multiplying it by 3^^^3 necessarily results in a huge positive number). Working so hard that you burn out increases the absolute value a tiny bit, but still gives no guarantee about the sign, especially if other people afterwards use you as an example of how everyone who cares about AI safety goes crazy.

Ironically, unless you are one of the top AI safety researchers, if you live in the Bay Area, your best contribution would probably be keeping the rationalist community sane. Don't take drugs, don't encourage others to take drugs, help people avoid cults, be nice to people around you and help them relax, notice the bad actors in the community and call them out (but in a calm way). If this helps the important people stay sane longer, or prevents them from burning out, or just protects them from being dragged into some scandal that would have otherwise happened around them, your contribution to the final victory is more likely to be positive (although still indistinguishable from zero). Generally speaking, being hysterical does not necessarily mean being more productive.

An ironic detail I noticed while reading an archive of the "Roko's basilisk" debate:

Roko argues how the values of Westerners are irrelevant for humanity in general, because people from alien cultures, such as Ukraine (mentioned in a longer list of countries) do not share them.

Considering that Ukrainians are currently literally dying just to get a chance for themselves and their families to join the Western culture, this argument didn't age well.

One should consider the possibility that people may be stuck in a bad equilibrium, before jumping to the conclusion that they must be fundamentally psychologically alien to us.

(Of course, there is also a possible mistake in the opposite direction, such as assuming that all "Westerners" share the "values of Westerners". The distribution of human traits often does not follow the lines we assume.)

I have read that entire thread, and... it is hard to say something coherent in reply, and I am probably missing a lot of context... but it seems to me that bad things are happening, but also that people complaining about them make wrong conclusions (mostly in style: I see something bad happening, so I point at the most visible thing nearby and say: this is the cause of the bad things happening).

Makes me wonder, what would have happened if instead of living on the opposite side of the planet, I lived in the middle of all that chaos. Would I be a part of the insanity? Or a lonely voice of reason? Or just a random low-status guy whose opinion is irrelevant because no one listens to it and no one is going to remember it anyway? (Probably the last one.)

Basically, it confuses me when people point at things I consider good, and call them causes of things that I consider obviously bad and stupid. What is the proper lesson to take here? Maybe I am the stupid one, unable to see the obvious causality, and protected from my own stupidity by being far away from where important things happen. Or maybe other people are simply doing things wrong.

I keep dreaming about having a rationalist group with more than five members in my country, but if my wishes came true, would that automatically mean also getting our local version of Zizians/Leverage/etc.? Do these things happen automatically as a consequence of trying to be rational, or did just someone accidentally build the Bay Area community on top of an ancient Indian burial ground?

...but that's basically what this article is about.

I think in Denver we’ve lucked into a default culture that puts emphasis on first getting your life in order and functioning in default society, with rationalism complimenting that rather than overriding it. Is this common?

The rationalist scene in Vienna is also sane, as far as I know. We need more data points from other cities.

Or maybe it's something unrelated to "antibodies", like the people in Bay Area taking an order of magnitude more drugs than people anywhere else, and everything else is just downstream of this. (Or, from another perspective, perhaps "don't take drugs just because some guys who call themselves 'rationalists' told you it was a good idea" is the most relevant normie antibody.) The obvious counter-argument is that everyone in Bay Area takes drugs, so the fact that the drugs were always visibly involved in the most crazy cases is not as strong evidence as I make it. The obvious counter-counter-argument is that this is probably the reason why the crazy cases happen in Bay Area, as opposed to other places.

And, perhaps, a resource that organizers can turn to if they notice someone slipping into fanaticism would be nice. As far as I know, there isn’t a Best Practices Doc for this sort of thing.

My first idea is to make a short text that will document the existing bad cases and highlight the relevant parts of the Sequences. Document the bad cases to show that the problem exists and is serious. Quote the Sequences to... dunno, probably as a way to tell the people "hey, if you decide to ignore all of these warnings and do your own thing anyway, at least do not publicly blame Eliezer when shit hits the fan".

Eliezer is absolutely right about deregulating childcare. The details depend on the specific country, so I don't know how much of this applies to others, but here e.g. the regulation for kindergartens is so detailed, that the only way to start a kindergarten is to build a new building specifically for it that would comply with the regulations. If someone donated to you an existing building that was originally built for a different purpose, e.g. former company offices, fixing it to comply with the regulations would mean almost rebuilding it from scratch. So, the available kindergartens in big cities are insufficient, and it is almost impossible to build new ones.

I strongly disagree with the argument that kindergartens are inefficient, because they only subsidize parents who want to spend less time with their kids. That seems like a very stupid way to put it. First, people are often uncertain, especially before they have their first child, what will it be like. Some people expect they will enjoy having children, and then they find out that they actually hate the experience in near mode. Some people expect they will hate the experience (but are willing to endure it for some reason), and then they find out that they actually love it. You simply probably don't know, until you try it. From this perspective, the kindergarten availability can be seen as an insurance: if you happen to love to spend 24 hours with your kids, great, but if it turns out that it actually drives you crazy, there is a way out. Second, there is a huge difference between "being with your kids" and "being with your kids 24 hours a day". For example, when my kids get sick, so they stay at home the whole day, instead of being at the school and kindergarten, everyone's mood gets worse. It's not that we hate each other, it's just that we get bored, and the kids are sometimes bad at being independent, and the parents are sometimes tired or busy doing something other than taking care of their kids. Third, parents sometimes need to leave the house, and sometimes it is difficult to take the little kids with you. -- Actually, to put it bluntly, this attitude is one of the things I consider the main reason for people having fewer kids; keep telling people that unless they spend 24 hours a day, every day, with their kids, it makes them bad parents, and some of them will decide that having kids probably isn't the right thing for them. But our parents and grandparents also didn't spend 24 hours a day babysitting their kids, and used whatever help was available.

When talking about education, let's ban all homework, and all those projects that kids make at home. (Half seriously.) Because that usually means extra work for the parent, which scales linearly with the number of kids. Some kids are smart and conscientious, and will do their homework alone. But quite often it requires the parent at least nagging them to keep working. Imagine having four children, which all need some help with homework and with the extra projects; and that's after you have returned from your full-time job, and before you start cooking the dinner.

Speaking about the full-time job, maybe it is time to shorten the workweek. That should be the common-sense response to women entering the workforce en masse. If you say that "women stay at home" is sexist and unfair, you may have a point, but the fact remains that the home still exists and requires some work to be done. So maybe a reasonable new norm could be "women stay at home half of the time, and men the other half of the time"? Thus we could get the benefits of "some adult is at home, taking care of the kids" and gender equality at the same time. In my experience, for a man, finding a part-time job is almost impossible. But if part-time jobs were easy to find for everyone, it would become easy for parents e.g. to have a part-time job each and homeschool their children.

There is a lot of anti-children propaganda in our culture. People are told all the time that having kids is difficult. Yes, that is true. You know what else is difficult? Lots of meaningful things. Are we told to avoid all meaningful things, or only to avoid having kids? I suppose it is the latter, because I do not hear the same propaganda telling people to stop having jobs, despite jobs being difficult, too. Women who have kids while young are treated as low-status, like they are too stupid to have a successful career, or too brainwashed by patriarchy to follow their dreams. (But what if having a big happy family is their dream? No, this is not the kind of the dream that is supported by the modern narrative. You are supposed to work for a corporation, and find the meaning of your life in endless meetings.)

Some anti-children beliefs become a self-fulfilling prophecy. The fewer kids there are, the fewer friends they can find at playgrounds. Heck, the fewer playgrounds there will be. And that in turn makes having kids more difficult. If your friends believe that having kids young is low-status, if you have kids while young, you can no longer participate in their social activities, which will make you feel lonely and low-status. But if instead everyone had kids right after university, you could bring them to the playground together, and your social connections could grow even stronger. The fewer people have kids, the more anti-children talk you will hear, and the more anti-children legislation will be made.

A strong and often repeated argument is that women need to be financially independent, and that this goal trumps everything (the second part is not made explicitly). Honestly, I am not sure how to solve this, because it naively seems to me that even if we gave extra million dollars to every woman on her 18th birthday, it would merely result in things getting more expensive, so now the woman would need the million dollars and a full-time job in order to remain truly financially independent. My answer is that parenting was never meant to be a single-person project, but if you want both parents to contribute equally, you either need 100% free kindergartens and schools, or enough part-time jobs for men, or women need to take the risk of financial dependence. You simply can't leave the baby at home alone while both parents are at work, duh. Even the free kindergartens and schools have their problems, because the kids often get sick. Also, we love to complain about schools and many of us would like to homeschool. Basically, our society is designed for people who have no kids, because there is no way to make it work unless (a) you get exceptionally rich, which is nice, but not an option for everyone, or (b) a woman decides to go against the narrative and risks financial dependence, or (c) we procrastinate on starting families until we have enough savings, but then we can't have many kids because we have already run out of time.

Another important factor is the ever increasing school attendance. Hundred years ago, if you attended school until you were 18, you were considered highly educated. Today, you are considered too stupid to be allowed working at half of the jobs unless you have some kind of university. Starting you family in your 20s vs in your 30s results in a different family size. You have less time to raise kids. Your parents are more likely to be dead or otherwise unable to help you take care of the kids.

Feminism was supposed to give women a choice, but instead it turned into shaming motherhood. (In my country, "domestic wife" is literally used as a slur by feminists.) According to the cultural narrative, a woman who chooses to have kids and stay at home, is a loser. And if she decides to have both a family and a career, she is supposed to start the career first, even if from certain perspective that makes less sense -- if you have a career and you enjoy it, then starting a family means giving it up. (And if you don't enjoy your career, then why was it so important to have it?) On the other hand, if you have kids first, there is plenty of time for the career after they grow up and leave home, and then you never need to give up a successful career once it started. Or you can change your mind and choose to have more kids.

Basically, people are told that they have enough time to have kids later. And it's... kinda true... but it also changes how many kids they will have.

Is there a list of "basic scientific knowledge" somewhere? I am imagining a project where someone makes a 1-minute video for each item in the list, and then... dunno, the playlist is played on repeat e.g. in every bus all day long.

Spam (Google Translate)

Interior design trends 2024: Fresh ways to reinvent your home

Getting ready for the new year is a great time to decorate and change your entire home. Are you determined enough to welcome a new space? Read on. Below are the most prominent interior design trends of 2024!

Agent cares about his goals, and ignores the objective norms.

If you know what you must do

There is no "must", there is only "should". And even that only assuming that there is an objective norm -- otherwise there is even no "should", only want.

Again, Satan in Christianity. Knows what is "right", does the opposite, and does it effectively. The intelligence is used to achieve his goals, regardless of what is "right".

Intelligence means being able to figure out how to achieve what one wants. Not what one "should" want.

Imagine that somehow science proves that the goal of this universe is to produce as many paperclips as possible. Would you feel compelled to start producing paperclips? Or would you keep doing whatever you want, and let the universe worry about its goals? (Unless there is some kind of God who rewards you for the paperclips produced and punishes if you miss the quota. But even then, you are doing it for the rewards, not for the paperclips themselves.)

The AGI (or a human) can ignore the threats... and perhaps perish as a consequence.

General intelligence does not mean never making a strategic mistake. Also, maybe from the value perspective of the AGI, doing whatever it was doing now could be more important than surviving.

Makes sense, but wouldn't this also result in even fewer replications (as a side effect of doing less superfluous work)?

I was going to ask for interesting examples. But perhaps we can do even better and choose examples with the highest value of... uhm... something.

I am just wildly guessing here, but it seems to me that if these features are somehow implied by the human text, the ones that are "implied most strongly" could be the most interesting ones. Unless they are just random artifacts of the process of learning.

If we trained the LLM using the same text database, but randomly arranged the sources, or otherwise introduced some noise, would the same concepts appear?

Are you perhaps using "intelligence" as an applause light here?

To use a fictional example, is Satan (in Christianity) intelligent? He knows what is the right thing to do... and chooses to do the opposite. Because that's what he wants to do.

(I don't know Vatican's official position on Satan's IQ, but he is reportedly capable of fooling even very smart people, so I assume he must be quite smart, too.)

In terms of artificial intelligence, if you have a super-intelligent program that can provide answers to various kinds of questions, for any goal G you can create a robot that calls the super-intelligent program to figure out what actions are most likely to achieve G, and then performs those actions. Nothing in the laws of physics prevents this.

Orthogonality thesis is not about the existence or nonexistence of "objective norms/values", but whether a specific agent could have a specific goal. The thesis says that for any specific goal, there can be an intelligent agent that has the goal.

To simplify it, the question is not "is there an objective definition of good?" where we probably disagree, but rather "can an agent be bad?" where I suppose we both agree the answer is clearly yes.

More precisely, "can a very intelligent agent be bad?". Still, the answer is yes. (Even if there is such thing as "objective norms/values", the agent can simply choose to ignore them.)

The first two points... I wonder what is the relation between "prestigious university" and "quality of your peers". Seems like it should be positively correlated, but maybe there is some caveat about the quality not being one-dimensional, like maybe rich people go to university X, but technically skilled people to university Y.

The third point, I'd say be aware of the distinction between the things you care about, and the things you have to do for bureaucratic reasons. There may or may not be an overlap between the former and the school lessons.

The fourth and seventh points are basically: some people give bad advice; and for anything you could possibly do, someone will find a rationalization why that specific thing is important (if everything else fails, they can say it makes you more "well-rounded"). But "skills that develop value" does not say how to choose e.g. between a smaller value now or a greater value in future.

The fifth point -- depends on what kind of job/mentor you get. It could be much better or much worse that school, and it may be difficult to see the difference; there are many overconfident people giving wrong advice in the industry, too.

The sixth point -- clearly, getting fired is not an optimal outcome; if you do not need to complete the school, what are you even doing there?

I think we probably agree on how far the existing system is from the ideal. I wanted to point at the opposite end of the scale as a reminder that we are even further away from that.

When I was at the first grade of elementary school, they tried to teach us about "sets", which mostly meant that instead of "two plus two equals four" the textbook said "the union of a set containing two elements and another set containing two elements, has four elements". In hindsight I see this was a cargo-cultish version of set theory, which probably was very high-status at that time. I also see that from the perspective of set theory as the set theorists know it, this was quite useless. Yes, we used the word "set" a lot, but it had little in common with how the set theorists think about sets. Anyway, we have learned addition and subtraction successfully, albeit with some extra verbal friction.

Compared to that, when I tried to learn something in my free time as a teenager, people around me recommended me to read books written by Däniken, the Silva method of mind control, Moody's Life after Life, religious literature, books on meditation, and other shit. I have spent a lot of time practicing "altered states of consciousness", because (from the perspective of a naive teenager who believed that the adults around him are not utter retards, and the people they consider high-status are not all lying scumbags) it seemed like a very efficient intervention. I mean, if you get the supernatural skills first, they will give you a huge multiplier to everything you try doing later, right? Haha, nope.

So while I hate school with a passion, as many people on Less Wrong do, the alternative seems much worse. Even the books I study in my free time now were often written in the context of the educational system, or by the people employed by the educational system.

I don't trust societal consensus at all. Look at the YouTube videos about quantum physics, 99% of them is some crap like "quantum physics means that human mind had a mystical power over matter". Even if you limit yourself to seemingly smart people, half of them believe that IQ isn't real because Nassim fucking Taleb said so. Half of the popular science does not replicate.

I suspect this may actually be the most important thing the educational system does.

You can learn from books or online videos. You can find fellow learners on social networks. You can find motivation... at random places.

But without being shown a direction, you will probably get lost in a sea of nonsense. A simple advice, such as "chemistry is the thing you should study, not alchemy" can save you decades of time you might otherwise waste learning useless things.

It is easy to notice the damage school system does, and easy to take its benefits for granted. Even if you are homeschooled, you are still exposed to people who got the right directions. (People from crazy religious families may be prevented from getting the right direction in e.g. biology or sex education, but they will still probably get the right directions about math or chemistry or geography.) This did not happen spontaneously. It was the educational system that redirected billions of people from thinking about superstitions and magic and astrology and homeopathy and whatever else, towards thinking about math and physics and chemistry and geography and history. Even if for many people the success is only partial, the fact that they even know about the useful stuff means a lot.

In today's world, you can become good at math without spending a single day at school. But in a world where everyone in the last three generations was an autodidact, you most likely wouldn't be good at math, because you most likely wouldn't even know that there was such a thing as math. (Unless you would be lucky to be born in a family of mathematicians.) Instead you would spend your time learning... some nonsense. Some difficult nonsense that requires high IQ and lot of studying to get it impressively right. Just like Newton spent half of his life doing astrology.

Autodidacts are easily recognized by their unknown unknowns. They may know a lot, but what they don't know, they usually don't even know that it exists.