While I don't necessary approve of the conclusion, there are many important points that people seem to underestimate. (Briefly, before discussing them, I suspect that what currently makes death penalty so expensive is all the legal processes around it, so if we replaced all death penalty with life sentence in prison, we would spend more money on the prisoners, but less money on the lawyers, potentially saving money on the net.)

The key part is: While the truly horrible people are few, they cause vastly disproportional damage, plus all kinds of secondary damage (the population living in fear, not trusting each other, spending more money on security, or just not trying many projects because of the perceived risk). Eliminating these people from the streets might change the situation dramatically, possibly in ways that many of us can't even imagine. It could change a low-trust society into a high-trust society, with various positive impacts on mental health and economy.

I have also seen similar dynamics in other situations. For example, in school, there is often one child in a classroom that constantly keeps disrupting lessons, frustrating the teachers and reducing the learning opportunities for all their classmates. Removing that one child can dramatically change the entire experience for everyone involved. I have seen incredible changes during a week or two when the disruptive child got sick; but after their return the situation reverted to the usual.

There is a related problem, that crime is often difficult to prove (beyond the shadow of doubt, without doing something illegal yourself, etc.). For example, rape is often only witnessed by the two people involved; we get a "he said, she said" situation. Another example, in Slovakia the law about illegal drugs was changed so that it is no longer illegal to have the amount of drugs you need for your own consumption; only selling drugs is illegal. Sounds reasonable at first sight (catching the dealers is strategically more important than catching the users)... until you realize that in practice, selling is almost impossible to prove. Because you would have to be there, at the exact moment. A police arriving five minutes later (which is already a super optimistic scenario) can't prove anything: all people involved will claim that no exchange happened, that everyone who has drugs on them has already arrived with them, that it is all for their personal consumption, and that all they wanted was to talk. So I have literally the situation that people are selling drugs on my street, they are not even trying hard to hide (other than making sure that no one is closer than 10 meters to them at the moment the money changes hands), and they do it with complete impunity. So again, we have a situation where worrying about being needlessly harsh to the users made the crime itself... not literally legal, but not punished in practice either.

Which suggest a need of another "tough on crime" approach, like: if X is a crime you want to prevent, but it is extremely easy to pretend that X is Y, and there is no good reason why Y should be legal... then maybe you should also make Y illegal, just to make sure that the people doing X actually get punished. Basically, when making laws about Y, don't think about Y in isolation, but also about all things that can be plausibly made into Y. Even if the punishment for Y is smaller than the punishment for X, it should definitely be nonzero, and if it happens repeatedly, it is very likely X in disguise, so the punishments for repeated Y should be comparable to X.

When we look at experience itself, there is no fixed “I” to be found. Boundaries between self and other aren’t innate to reality but drawn after the fact. We carve up this vast space of experience into “mine” and “yours,” but these divisions are somewhat arbitrary.

The boundaries are somewhat arbitrary, but it seems to me that if we keep going in this direction far enough, at the end of the road is equanimity with the universe being converted to paperclips. (Which would be a wrong thing in my current unenlightened opinion.) After all, there is no sharp boundary between "me" and a paperclip.

So, one problem seems to be that humans are slow, and evaluating all options would require too much time, so you need to prune the option tree a lot. I am not sure what is the optimal strategy here; seems like all the lottery winners have focused on analyzing the happy path, but we don't know how much luck was involved at actually staying on the happy path, and what was the average outcome when they deviated from it.

Another problem is that human prediction and motivation are linked in a bad way, where having a better model of the world sometimes makes you less motivated, so sometimes lying to yourself can be instrumentally useful... the problem is, you cannot figure out how much instrumentally useful exactly, because you are lying to yourself, duh.

This model gives an explanation for why people who are very successful often say they cannot imagine failure.

Another important piece of data would be, how many of the people who cannot imagine failure actually do succeed, and what typically happens to them when they don't. Maybe nothing serious. Maybe they often ruin their lives.

More generally, what is stopping people from making RL forum posters on eg Reddit that will improve themselves?

Could be a problem with not enough learning data -- you get banned for making the bad comments before you get enough feedback to learn how to write the good ones? Also, people don't necessarily upvote based on your comment alone; they may also take your history into account (if you were annoying in the past, they may get angry also about a mediocre comment, while if you were nice in the past, they may be more forgiving). Also, comments happen in a larger context -- a comment that is downvoted in one forum could be upvoted in a different forum, or in the same forum but a different thread, or maybe even the same thread but a different day (for example, if your comment just says what someone else already said before).

Maybe someone is already experimenting with this on Facebook, but the winning strategy seems to be reposting cute animal videos, or posting an AI generated picture of a nice landscape with the comment "wow, I didn't know that nature is so beautiful in <insert random country>". (At least these seem to be over-represented in my feed.)

Task: write LessWrong posts and comments. Reward signal: get LessWrong upvotes.

Sounds like a good way to get banned. But as a thought experiment, you might start at some place where people judge content less strictly, and gradually move towards more difficult environments? Like, before LW, you should probably master the "change my view" subreddit. Before that, probably Hacker News. I am not sure about the exact progression. One problem is that the easier environments might teach the model actively bad habits that would later prevent it from succeeding in the stricter environments.

But, to state the obvious, this is probably not a desirable thing, because the model could get high LW karma by simply exploiting our biases, or just posting a lot (after it succeed to make a positive comment on average).

Ah, I meant something like people have 11 units of resources, but they need ~10 to survive, so 10 of them would have to invest all their savings... which is unlikely to happen, because coordination is hard.

You are right that companies with shares are the way to overcome it. I was thinking deeper in history, where e.g. science could only happen because someone had a rich sponsor. Without the rich sponsors, science probably would not have happened; Newton would be too busy picking the apples. Only a small fraction of rich people becomes sponsors of science, but it's better than nothing.

Perfect equality could mean that if somehow things go wrong, they will go equally wrong for everyone, so no one will have the slack necessary to find the way out... is my intuition about this scenario.

Leave them up, other people may be curious too, but too shy to ask.

Also, sometimes inequality functions a bit like division of labor.

Imagine that everyone has 11 units of resources, and you need 20 to start a project. Compare to a situation where most people have 10 units of resources and one person has 30. There is no guarantee that the rich person will start the project, but the chances are probably higher than in the first scenario.

This article feels like arguing against a statement that was probably never made on Less Wrong.

I even think I remember Yudkowsky saying that individual differences in IQ are unfair, and that in the glorious transhuman future of course everyone should get at least IQ 200, or something like that.

The implicit assumption that anyone could reason as we do if they simply tried harder.

Frustration or dismissal when others fail to grasp concepts we find intuitive.

For me, the frustrating thing is that many of those people who have the sufficiently high intelligence still choose to be irrational. There is a book "What Intelligence Tests Miss: The Psychology of Rational Thought" by Keith Stanovich that used to be popular here, and it is precisely about how intelligence isn't rationality.

Personalized AI tutors could help those with lower cognitive capabilities catch up, but access to these technologies is uneven. If high-quality AI education tools remain expensive or exclusive, the divide will only widen.

Ironically, this part seems like you making the very mistake that you are accusing rationalists of. Suppose that perfect personalized AI tutors are available to everyone, for free. What happens? I think the divide will widen anyway, simply because the more intelligent people will benefit more from the AI tutors.

This is a mistake people frequently make when discussing education, even if we completely ignore the AI or computers in general. Yeah, education sucks for everyone, both for the smart, the average, and the stupid, in a different way for everyone. It could be made much better, and everyone could benefit from that. However, that alone does not imply that optimal education for everyone would make the differences disappear. If I simplify it a lot, if the improved education helped everyone achieve twice as much as they can achieve now, it would be better for everyone, and yet the differences would now be twice as big, not smaller. You would need to argue that there are ways to dramatically improve the education of the stupid, but no comparable ways to dramatically improve the education of the smart, which goes against the usual experience of smart kids being bored at school most of the time.

Recognizing your cognitive privilege is the first step toward engaging with others more fairly and constructively.

I agree, but in the current (or maybe recent) political climate, acknowledging that some people are more intelligent than others would get you in deep trouble.

Intelligence should not be the sole determinant of value or opportunity. If it is largely unearned, then structuring society around cognitive hierarchies is deeply unjust.

Here, the tricky thing is how to decouple "good life" from "making decisions". Capitalism kinda conflates those two things: if you have more money, you can live a more convenient life, and you can also decide the use of more resources. This already assumes some cognitive capabilities -- merely throwing more money at stupid people doesn't necessarily improve their lives; many of them get scammed, spend the money on drugs, etc. Democracy is also built on the assumption that letting stupid people vote creates better outcomes for the stupid people, which again is not necessarily true.

So the problem is how to create some kind of kindergarten environment for the stupid, where they can live as well as possible, but won't be expected to manage the environment. It is a historical experience that not letting people make the decisions often leads to abuse. The problem is that without cognitive abilities, letting people make the decisions also often leads to abuse... so, I guess, it is not obvious how the society should be structured.

What implicit assumptions do you make about others based on your own cognitive experiences?

I don't anymore. I have advanced towards the state of silent despair.

How should the recognition of cognitive privilege shape discussions on AI ethics and policy?

Mostly by realizing that the machines of the future will probably be smarter than all of us? Which again is not a controversial thought in this community. The original plan was to build a Friendly AI, and that still seems to me like a good outcome to strive for, even if I have no idea how to contribute to it.

In my experience, it’s scary how often such vibes prove correct in the end.

Yeah. I have the same experience. (Unless it is a selective memory, of course.)

But that alone does not prove that the experience is universal. Maybe we are just exceptionally well calibrated. If so, there is a potential self-improvement area: figure out how well calibrated you are about people, and maybe try to improve somehow. Not sure how, though. Make predictions about other people, and check them later? A prediction market for whether your neighbors will divorce, and which of your friends will end in prison?

It also seems possible (at least to my inner psychoanalyst) that some people have an unconscious desire to get hurt in some ways. Like, the real bottleneck is not detecting the bad vibes; it is acting on the information. Some people choose to ignore the bad vibes; some people may even be attracted to them. ("This person seems dangerous. I am sure it will be okay." "This person seems safe. Boring!")

TikTok as intermittent reinforcement, a slot machine for children.

An interesting point made in the video, TikTok rewards new accounts by giving them more attention than they would normally get. So the user starts with a more positive experience ("hey, everyone loves me, this is an awesome platform"), and when they lose it later, they probably respond by trying harder.

Which is the same strategy that many games use, where completing the first levels is easy, and then it gets exponentially harder (but don't worry, you can always pay to win).

Also, I think Chinese online shops use this strategy where they give huge discounts to new users, so you remember "oh, buying here is much cheaper than buying anywhere else" and you stay there even when it gradually stops being true. (I tried it with AliExpress, different people get different prices.)

i’ve heard ppl who lost a lot of weight talk about some angry cynicism when people start treating them better, even ppl they’ve known for a long time. I’m having a bit of that now that twitter seems to like me. i’ve been consistently myself this entire time, what’s happening.

How we perceive ourselves is often very different from how other people see us, and the huge jumps in popularity when you change one detail make it painfully obvious that people care about this little detail a lot, and about the rest of you... well, not much. That hurts, if you primarily identify with the rest.

It's like, dunno, winning at a poetry contest, and then finding out that the voters actually didn't listen to your poetry at all, they just voted for you because they e.g. liked your shoes. And you're like: okay, my shoes are nice and I am happy that someone appreciates that, but this was supposed to be a fucking poetry contest, does anyone care about that at all? And the next time you are trying to compose a poem, it feels like you shouldn't bother, because apparently no one cares about that, and even when it seemed so, it was incidental.

There is some overlap, so not all is lost. For example, people around me appreciate that I am smart, or a good listener, which are important parts of my identity. But things like... uhm, all those things that I would put in a "generally being a good person" set... they probably matter way less than the fact that I have nice blue eyes.

When I scroll through my old Facebook posts, it is often sad to see how the stupid stuff gets upvoted a lot, but the things that seem important to me are mostly ignored. I would prefer a system that rewards effort and thought more than it rewards cat videos. But Facebook is what it is, and handle this thing by spending less time on Facebook.

And what do these people constantly yell at us, if we have ears to hear? That they, their preferences and causes get no respect.

Seems like polarization is the cause of the problem. It doesn't feel like all causes can be cared about, only the selected ones. Then people switch to zero-sum mode.

Then again, people probably overestimate (by orders of magnitude) how much support the other causes actually get. They often get tons of attention, but not enough funding, etc.

We could probably make everyone happy by paying more lip service to the things most people care about, and simultaneously increasing the funding for the effectively altruistic things from epsilon to twice the epsilon. But there are probably no incentives to do that, so nope, not going to happen.

A theory from Benjamin Hoffman on various Trump executive order fiascos: That the administrative class feels compelled to do perverse interpretations of the (usually very poorly drafted) EOs.

Is anyone even able to coherently model Trump? I admit I don't pay enough attention to him, but his words seem to me mostly random, one day contradicting what he said on another day, it's all just vibes... like if he is talking to a group of X people, then X is best, the next day he is talking to Y people, so obviously Y is best, etc.

My point is, if the EOs are poorly drafted and there is no way to model what he actually wants... then you shouldn't blame the administrative class for failing to do the impossible. (Although, blaming the administrative class is fully vibes-compatible with Trump fans. "The Czar is good, the boyars/bureaucrats are bad.")

Rationalists have noticed this tendency too, but they usually come to the wrong conclusion: “If there is no clear reason not to do A, then as a rational person, I should be fine with A.”

True. If there is no legible reason. If there is no reason that your opponent is willing to accept. That basically means that a sufficiently forceful opponent who can use the right keywords can push you anywhere. It works not only against objections you can't put in words, but also against objections that are low-status.

(For example, there is no "clear" reason why people shouldn't experiment with drugs. At least, no reason that the people who like to experiment with drugs would accept as "clear", if they can instead accuse you of being too stupid to think independently and do our own online research.)

From inside, almost everything I can do is "easy". Otherwise I wouldn't be able to do it, right? The trick is noticing that many things that are "easy" for me are actually quite difficult for other people. And even there, who do you compare yourself to? If you are constantly surrounded by experts, the things that are considered "easy" by your entire bubble can still be a mystery for 99.9% of population.

people with potential didn't have enough time to read LW to become rationalists? Or rationalists don't have time for LW?

Both of that. There are probably some people out there, who would be a great fit for LW, but instead they are busy doing research or making money or changing the world somehow. Also, some people who have formerly spent a lot of time on LW are now doing something more efficient with their time.

Isn't posting your thoughts in net is usually a very cost benefit action where thousands of people can read post written once?

Yeah, but 99.9% of those people won't remember what you wrote the next day, so the actual impact can be very small. Also, instead on LW you could post on your own blog, or maybe write a book, those are also ways to approach many people. Some of those may be more efficient.

It's good that we have the LW books for busy people; selection of the best articles instead of having read all of that.

And I admit I have much more free time than the most of people.

That is a great starting position (much better than having no free time -- then it is very difficult to think about your life or try to improve it, if there is no time to do that). But if you use that free time to figure out what you want to do and actually start doing it, then... probably in a year, you will have less free time. Not necessarily in a bad way; you can be busy doing things that you totally love. But you won't have so much time to read LW anymore.

This is a paradox of self-improvement groups (and LW kinda belongs to that category). If you actually keep improving, at some moment you will probably leave the group, because some other action will be more useful for you than staying at the group. That's the entire point -- the group is supposed to make you more efficient at the life outside the group. If it fails to achieve that, then the group is not really good at its stated goal.

Death tax without a gift tax would simply be a tax on people who die unexpectedly. Because if you know that you are going to die tomorrow, you can donate all your belongings to your children today.

Even if you don't know the exact day, if you trust your children, you can simply donate them everything now, and then continue living in a house they legally own, etc. (Though then you are screwed if your children die before you. But this just means that the system introduces a lot of randomness.)

Oh, and if you have a 100% gift tax, you also need to make all kinds of helping each other illegal. Not only you shouldn't give you children money, you shouldn't even give them food, probably not even good advice, if you want to be consistent. Otherwise we get all kinds of tax loopholes, such as allowing your adult children live in a house that you own (so they can save the money they would otherwise spend on rent or buying their own house), or babysitting your grandchildren for free (so that your children can save money they would spend on babysitters). Cooking for your children, helping them fix things, etc., all tax loopholes.

Minimizing friction is surprisingly difficult. I keep plain-text notes in a hierarchical editor (cherrytree), but even that feels too complicated sometimes. This is not just about the tool... what you actually need is a combination of the tool and the right way to use it.

(Every tool can be used in different ways. For example, suppose you write a diary in MS Word. There are still options such as "one document per day" or "one very long document for all", and things in between like "one document per month", which all give different kinds of friction. The one megadocument takes too much time to load. It is more difficult to search in many small documents. Or maybe you should keep your current day in a small document, but once in a while merge the previous days into the megadocument? Or maybe switch to some application that starts faster than MS Word?)

Forgetting is an important part. Even if you want to remember forever, you need some form of deprioritizing. Something like "pages you haven't used for months will get smaller, and if you search for keywords, they will be at the bottom of the result list". But if one of them suddenly becomes relevant again, maybe the connected ones become relevant, too? Something like associations in brain. The idea is that remembering the facts is only a part of the problem; making the relevant ones more accessible is another. Because searching in too much data is ultimately just another kind of friction.

It feels like a smaller version of the internet. Years ago, the problem used to be "too little information", now the problem is "too much information, can't find the thing I actually want".

Perhaps a wiki, where the pages could get flagged as "important now" and "unimportant"? Or maybe, important for a specific context? And by default, when you choose a context, you would only see the important pages, and the rest of that only if you search for a specific keyword or follow a grey link. (Which again would require some work creating and maintaining the contexts. And that work should also be as frictionless as possible.)

May be, I was just wrong that I am not unusual.

Unusual is not a binary, it is a "more or less" thing, but yes, you may be much more unusual in some regards than you expected.

Actually, looking at your bio, it may be a cultural taboo for you to admit that you are exceptional. I grew up in communist Czechoslovakia, and the thought "I may be different from most other people, in a good way" went completely against all my conditioning. That's not what you are supposed to ever say, unless you want to get in deep trouble.

It's not just about intelligence, although high intelligence may be a prerequisite. Most people, even the intelligent ones, simply don't give a fuck about many things that we value at LW, such as having a correct model of the world as opposed to merely winning verbal battles., or preferring to know the truth even if it makes you sad as opposed to just enjoying happy thoughts no matter how unrelated to reality they are. Most people just don't click with this, because... I guess they don't see a reason why. Why do things, if they don't make you happy? (Yeah, in theory, looking reality in its face could save your life or something, but in practice, it's not like rationalists are famous in the outside world for actually winning at life, so maybe this all is just our version of wishful thinking.)

So, yeah, actual rationalists are very rare. I couldn't find ten of them in my country. (And I am not familiar with the Bay Area community, but sometimes I suspect that many people are there simply for the vibes. Some people enjoy hanging out with rationalists, even if they don't share the fundamentals. It's just another social activity for them.)

Then there is also the fact that people are busy. Not everyone who has the potential to become a rationalist also has time to spend on LW website. Such people usually have a lot of work they could do instead.

I might hugely underestimate my own ability to contribute something significant

Maybe, I guess you won't know until you try.

Probably there are just not enough people for all of that?

Social media will make you overestimate a lot. When I share a LW post on Facebook, it gets 10 likes. When I invite those people to a local LW meetup, no one comes. Clicks are cheap; even people who don't like rationality are happy to click if the article seems interesting.

This is the kind of thing I don't trust LLMs about, too much hallucinating.

As an example, once an LLM insisted that "infinity plus one" is a finite number... because the sequence (infinity, infinity plus one) has a length two, and of course a finite sequence cannot have an infinite limit. I guess it's because it read many statements like this, but missed the detail that those were finite sequences of finite numbers, not finite sequences already starting at infinity.

So, in my opinions, the current LLMs are unable to do math, because math requires precision, and they are just putting together words that statistically go well together... but there is a huge difference between being 99% correct and 100% correct; one bad step invalidates the entire chain.

I suspect that something like this (words that sound correct, but are taken out of context which makes them completely invalid mathematically) is likely to be in the text you posted, too.

By the way, I am not saying that LLMs couldn't do this in principle, but their context probably needs to be much larger that it is typically today. Basically, a math textbook can define a thing in chapter 1, and then refer to it in chapter 15, by which point a typical LLM already forgot which definition was used in this specific book (did "finite sequence" refer to a sequence of finite length, or a sequence containing finite numbers, or both?).

I think the only reasonable way to do math with LLMs is to have them generate proofs that you afterwards check using some other system, such as Lean. Perhaps someone will build an AI that will do this automatically, i.e. will check its own reasoning in Lean before it tells you the conclusion.

The rich people may keep a few human slaves as a status thing, or maybe because they enjoy having power over humans. I agree that economically human slaves won't be valuable.

Top performers in fields like chess, music and athletics almost universally receive coaching.

I wonder how much of that is actually based on science, and how much is just superstition / scams.

Do you know whether these coaches are somehow trained / certified themselves? Like, are there some scientific studies that a wannabe coach needs to learn and take an exam? Or is it more like some random person decides "I feel smart, I am going to be a coach", and the rest depends only on their charisma and marketing?

If I somehow happen to be a top athlete, is there some organization where I can go and tell them "give me a list of coaches you recommend", or do I have to search online and make a guess about what is a scam and what is not?

One reason I am asking is that if there is a coach certifying body and a list of scientific literature, it might be interesting for someone to look at the literature and maybe write some summary on LW.

I would expand your suggestion; I think it would be interesting to have something like "coaching for intellectuals" in general, not just for AI alignment researchers. Sleep, sport, nutrition, meditation, writing, that applies to many professions. Well, the way I said it, I guess it applies to all humans, but let's say that the coaching for intellectuals would also specifically address things like written communication or risks of sedentary lifestyle, and it could give you articles to read.

The cheapest version could consist of a website with articles on different topics; a coach that would meet you once in a few months to talk to, who would give you a high-level summary and links to more details; and maybe some hand-holding such as "you recommended me to get a blood test, so... what specifically should I tell the doctor I want... ok, now I got this result, how do I interpret it?". And the more expensive versions would include more time spent with the coach, any maybe some other services, like buying the healthy food / supplements / home gym equipment / sleeping masks / whatever. Maybe with some group discounts, if people at the same place subscribe, so the couch can meet them as a group.

What is specific, from this perspective, for AI alignment researchers? Maybe the feeling of great responsibility, higher chance of burnout and nightmares?

This perspective challenges common assumptions about meaning, morality, and subjective experience.

I don't think those assumptions are common on LessWrong.

Specific things -- especially people -- feel more real than abstract threats?

Also, maybe instead of actual damage the intuitions reflect the perceived intention / integrity of the actor? Like, it is more plausible that there is a honest misunderstanding / difference of opinions on AI safety, than about protecting a corrupt mayor. Such intuition may make sense in general (even if not specifically in the case of AI safety), because misunderstandings can possibly be addressed by a dialog, but it doesn't make much sense to talk to people participating in corruption -- they are perfectly aware of what they are doing.

To clarify, I don't see evangelism as a problem per se, but I see it as a problem when the community needs evangelism to survive -- e.g. because the existing members get burned out and are discarded.

A difference between a symbiont and a predator, kind of.

Probably an evolutionary adaptation that (usually) prevents them from killing their 3 years old children.

We wanted computers to be more like humans; didn't realize it would make them suck at math.

If the problem is that nice people are generalizing from their internal experiences, then why is it that even self-declared psychopaths I meet seem ~basically correctly calibrated about how likely others are to mess with them?

I suspect that the psychopath's theory of mind is not "other people are generally nicer than me", but "other people are generally stupid, or too weak to risk fighting with me".

Motivation is invisible, skills are visible. So it is easy to be in denial about differences in motivation, and attribute behavior to the differences in skills.

Mr. Portman probably believed that some children forgot to pay for the chocolate bars, because he was aware that different people have different memory skills.

At the same time, empowering only the user and making the assistant play along with almost every kind of legal NSFW roleplaying content (if that’s what OpenAI ends up shipping) seems very undesirable in the long term.

Why? Do dildos sometimes refuse consent? Would it be better for humanity if they did? Should erotic e-books refuse to be read on certain days? Should pornography be disabled on screens if the user is not sufficiently respectful? What about pornography generated by AIs? When is it proper to worry about objectifying objects?

'Poor' people no longer starve in winter when their farm's food storage runs out.

Homeless people sometimes starve, and also freeze in winter.

(But I agree that the fraction of the starving poor was much larger in the past.)

When you start a new chat, you reset the memory, if I understand it correctly. Maybe you should do that once in a while. Then you may need to explain stuff again, but maybe it gives you a new perspective? Or you could write the background story in a text file, and copy-paste it to each new chat.

Could the LLM accidentally reinforce negative thought patterns or make unhelpful suggestions?

I am not an expert, but I think that LLMs are prone to agreeing with the user, so if you keep posting negative thought patterns, there is a risk that LLM will reflect them back to you.

What if the LLM gives advice that contradicts what my therapist says? How would I know what to do?

Trust the therapist, I guess? And maybe bring it up in the next session, kinda "I got an advice to do X, what is your opinion on that?".

What is the risk of becoming too dependent on the LLM, and how can I check for that?

No idea. People are different; things that are harmless for 99 may hurt 1. I don't know you.

Are there specific prompts or ways of talking to the LLM that would make it safer or more helpful for this kind of use?

Just guessing here, but maybe specify its role (not just "you are a therapist", but specifically e.g. "a Rogerian therapist"), and specify the goal you want to achieve ("your purpose is to help me get better in my everyday life" or something).

Maybe at the end ask LLM for a summary of things that you should tell your human therapist? Something like "now I need to switch to another therapist, prepare notes for him so that he can continue the therapy".

One possible line to draw between "religions" and "cults" is how much they depend on recruiting new people / how much they burn out the existing ones. Whether they can live with a stable population -- of course, many religions would be happy to take more converts, but what happens when they can't, and they need to spend a few decades with the existing ones (and their children) only -- or whether people are so damaged by being in the group that recruiting new ones and discarding the old ones is necessary for the group to function.

For example, you can have stable Catholic or Protestant populations. But Scientology depends on new people giving all their money to the group, then working hard for a few years until they get burned out, and when they become a burden on the group and their performance statistics become bad and no amount of punishment can fix that, they get kicked out. So an isolated population of Scientologists on some island would soon run out of money, and then gradually also run out of people.

I think the early Mormons had a lot of dynamic like "the old high-status guys take many young wives; the young incel boys get a lot of abuse in hope that they will rebel which will give the group a pretext for kicking them out". Monogamy stabilized this a lot, but the question is what exactly caused the old high-status guys to change the rules. Did the young guys who stayed in the group long enough remember how bad it was, and instead of enjoying that it's finally their turn, they decided to change the rule? Or was it something else?

(Also, from this perspective, Zizians without new recruits would run out of members in a decade.)

If there are 10 sellers selling the same thing for the same price, I wouldn't be in any trouble if one of them stopped existing.

we can tell from the externally observed effect - the crazy stuff they got up to - that the technique had a bad effect.

This is probably only convincing from outside. From inside, there is probably a perfectly good explanation, and what seems to us as craziness would seem to them as advanced rationality.

I think Ziz believes in some form of quantum suicide, so from that perspective, even getting killed is not necessarily a bad outcome, because if you apply your timeless algorithm across all Everett branches, of course you are going to win some and lose some, so this just happens to be a losing branch.

Mixing two metaphors together, we get a quantum suicide by a cop, where you boldly keep escalating to achieve your goals, and you either achieve them, or you are no longer in that specific Everett branch. Also, killing the people you consider bad is justified, because they survive in the parallel branches where they pissed you off less, so you just made them less bad on average (they probably should have thanked you).

...which is why it felt so important to me to make it clear that the things about hemispheres are plainly wrong in all Everett branches. Just in case someone who is tempted by their ideology is reading this.

I wonder if the greatest benefit of LDS missionary service isn't recruiting new members, but instead dirtying their children with outside memespace to prevent an autoimmune reaction when they mature.

From what age do the Mormons do this?

It sounds plausible that a small child would not be able to evaluate new arguments correctly, so it will just ask an elder and receive some bullshit excuse which sounds okay. And at later age, it will not even listen to the arguments, because "I have heard it all before many times".

EDIT:

There is a traditional atheist way of bringing up children to faithlessness, where you first read them about the Greek gods, and later some Bible for children. Both in context of "stories that people believed in the past". So when they encounter the meme in real life, they have some antidotes.

Compare to various Chick tracts, where the story often ends with the good guy asking the villain "have you ever heard about Jesus?" and he's like "never, who's that?", "well, let me give you this book"... and soon the villain is begging to get baptized. I don't know how much that is wishful thinking, and how much that happens in real life, but... maybe there is a reason why this was considered plausible by his audience.

Something about Beeminder always rubbed me the wrong way, especially those days when it was very popular on Less Wrong. But I didn't have a better alternative. Doing violence to myself felt wrong. Yet, the work needs to be done, and things that seem important in short term sometimes feel like a waste of time the next day.

I thought that a better inner harmony could be achieved by some kind of peaceful self-talk, generating an inner consensus. Like, if I know I should be doing X, but I feel an urge to do Y, instead I should just lie down on a couch, and do neither X nor Y, but contemplate the reasons why I wanted to do X in first place. (Perhaps use some technique for communicating with unconsciousness, such as visualizing the outcome.) If I am right, at some moment I should start feeling an actual urge to do X. And that is the right moment to actually start doing X.

Problem is, this takes some time (though maybe not more than the time wasted doing Y instead), and more importantly, it looks very bad to external observers.

For example, even if you are going to exercise regularly, it's not clear that a home gym is a good investment: a gym membership may be a better idea.

Dunno; gym membership also feels like a form of blackmail (although preferable to the alternative forms of blackmail), while home gym reduces the inconvenience of exercising.

I think I've heard that "people buying gym membership, and then not going there" is a significant source of income for gym. They are willing to sell a year-long membership quite cheaply compared to the one-time visit, because they know that a majority of people will go there maybe three times at all. Which suggests that buying the membership is probably not a good idea.

Generally, it seems like the problem is signaling. You buy the gym membership to signal your strong commitment to yourself. Then you feel good about sending a strong signal. And then the next day you feel just as lazy as previously, and the fact that you already paid for the membership probably feels bad. So it's like taking a financial debt in order to give yourself an undeserved psychological reward -- you reward yourself for buying the membership, not for actual exercising.

(Yeah, buying the home gym is also a kind of financial blackmail. You should start exercising with your own weight, or maybe just buy a few tools, like $100 max, and see where it goes.)

I wasn't there, so who knows how I would have reacted, it probably looks different in hindsight, but it seems like there were already red flags, some people noticed them, and others ignored them:

Salamon told Open Vallejo that LaSota attended three CFAR events between 2014 and 2018. Concerned by their “weird” behavior and interactions with other CFAR attendees, Salamon tried to convince a joint admissions committee between the Machine Intelligence Learning Institute and CFAR to not admit LaSota into their month-long summer fellowship in 2018. Salamon, however, was overruled.

“When LaSota attended the final program in summer 2018, I was physically afraid in a way I’ve never been with anyone else,” Salamon said in an email to Open Vallejo.

Salamon was concerned with some of the ideas that LaSota talked about during the workshops and with her in private. They included theories on “hemispheric sleep,” in which LaSota claimed that humans can split their consciousness between two sides of the brain, allowing one side to sleep while the other is awake, she said. In addition, these two sides of the brain may be “good,” “evil,” or both.

-- ‘Zizian’ namesake who faked death in 2022 is wanted in two states

I don't see a reason why we should trust Altman's words on this topic more than his previous words on making OpenAI a non-profit.

Before Singularity, I think it just means that OpenAI would like to have everyone as a customer, not just the rich (although the rich will get higher quality), which makes perfect sense economically. Even if governments paid you billions, it would still make sense to also collect $20 from each person on the planet individually.

After Singularity... this just doesn't make much sense, for the reasons you wrote.

I was trying to steelman the plan -- I think the nearest possible option that would work is having one superintelligence that keeps everyone safe and tries to keep the world "normal" as much as people in general want it to have; and to give every human an individual assistant which will do exactly as much as the human wants it to do.

But even this doesn't make much sense, because people interact with other e.g. on the market, so the ones who choose to do it slowly will be hopelessly outcompeted by the ones who choose to do it fast, so there won't be much of a choice.

I imagine we could fix this by e.g. splitting the planet into "zones" with different levels of AI assistants allowed (but the superintelligence making sure all zones are safe), and people could choose which zone they want to live in, and would only compete with other people within the same zone. But these are just my fantasies inspired by reading Yudkowsky, and have little to do with Altman's statements, and shouldn't be projected into them.

Also, what makes Ziz believe that there are always "two [cores] per organism" (source)?

What about using the name that is on the legal documents first, then all other important names, separated by slashes. "Maximilian Snyder / Audere", "Jack LaSota / Ziz", etc. (With only the legal name hyperlinked.) Not everywhere, but at least in the main list of criminals & suspects.

Also, it should be a list rather than a table. Now it seems like there is a distinction between the left and right columns of the names, which I believe wasn't the intention.

Just guessing, but maybe admitting the danger is strategically useful, because it may result in regulations that will hurt the potential competitors more. The regulations often impose fixed costs (such as paying a specialized team which produces paperwork on environmental impacts), which are okay when you are already making millions.

I imagine, someone might figure out a way to make the AI much cheaper, maybe by sacrificing the generality. For example, this probably doesn't make sense, but would it be possible to train an LLM only based on Python code (as opposed to the entire internet) and produce an AI that is only a Python code autocomplete? If it could be 1000x cheaper, you could make a startup without having to build a new power plant for you. Imagine that you add some special sauce to the algorithm (for example the AI will always internally write unit tests, which will visibly increase the correctness of the generated code; or it will be some combination of the ancient "expert system" approach with the new LLM approach, for example the LLM will train the expert system and then the expert system will provide feedback for the LLM), so you would be able to sell your narrow AI even when more general AIs are available. And once you start selling it, you get an income, which means you can expand the functionality.

It is better to have a consensus that such things are too dangerous to leave in hands of startups that can't already lobby the government.

Hey, I am happy that the CEOs admit that the dangers exist. But if they are only doing it to secure their profits, it will probably warp their interpretations of what exactly the risks are, and what is a good way to reduce them.

Upvoting for the footnote, btw.

Yes, but mere persistence does not imply reproduction. Also does not imply improvement, because the improvement in evolution is "make copies, make random changes, most will be worse but some may be better", and if you don't have reproduction, then a random change most likely makes things worse.

Using the government example, I think that the Swiss political system is amazing, but... because it does not reproduce, it will remain an isolated example. (And disappear at some random moment in history.)

Here is my collection of news article, although now it no longer feels like "more is better", because most of the information is repeated many times.

2019-11-18 Mystery in Sonoma County after arrests of protesters in Guy Fawkes masks and robes
https://www.sfchronicle.com/bayarea/article/Mystery-in-Sonoma-County-after-kidnap-arrests-of-14844155.php

2021-11-16 Protesters arrested in 2019 suing Sonoma County, Westminster Woods
https://www.pressdemocrat.com/article/news/protesters-arrested-in-2019-suing-county-westminster-woods/

2022-09-07 Jack LaSota Obituary
https://www.legacy.com/us/obituaries/newsminer/name/jack-lasota-obituary?id=36432272

2022-11-13 Woman shot to death, man reportedly stabbed with sword at Vallejo homeless camp
https://www.sfchronicle.com/bayarea/article/Woman-shot-to-death-man-reportedly-stabbed-with-17582424.php

2022-11-17 Eviction battle leads to sword attack and shooting in Vallejo
https://www.sfchronicle.com/bayarea/article/Attack-in-Vallejo-leaves-one-dead-two-critically-17590160.php

2023-01-03 Bodies Found In Delaware County Home: State Police
https://patch.com/pennsylvania/media/bodies-found-delaware-county-home-state-police

2023-01-05 Homicide Investigation Underway After Bodies Found In Delco Home
https://patch.com/pennsylvania/media/homicide-investigation-underway-after-bodies-found-delco-home

2023-01-05 Chester Heights deaths ruled homicides by medical examiner; victims identified
https://www.delcotimes.com/2023/01/05/chester-heights-deaths-ruled-homicides-by-medical-examiner-victims-identified/

2023-01-06 Police canvass Aston Twp. neighborhood after couple found dead in Chester Heights home
https://6abc.com/delaware-county-murders-richard-zajko-rita-couple-found-dead/12666110/

2023-01-31 Investigation into the murders of Zajko couple continues with new evidence uncovered, state police say
https://www.delcotimes.com/2023/01/31/investigation-murders-zajko-continues-state-police/

2023-02-01 Pa. State Police: Murder of Delaware County couple not believed to be random act of violence
https://www.fox29.com/news/pa-state-police-investigate-homicide-death-of-delaware-county-husband-and-wife

2023-02-01 Double Murder In Delco Not Random Act Of Violence, State Police Say
https://patch.com/pennsylvania/media/double-murder-delco-not-random-act-violence-state-police-say

2023-02-01 New Evidence Shows Chester Heights Double Murder Not ‘Random Act of Violence'
https://www.nbcphiladelphia.com/news/local/new-evidence-shows-chester-heights-double-murder-not-random-act-of-violence/3487640/

2025-01-20 2 dead, including US Border Patrol agent, following shooting in northern Vermont
https://vtdigger.org/2025/01/20/police-respond-to-shooting-involving-border-patrol-agent-in-northern-vermont/

2025-01-21 Authorities say a US border patrol agent was fatally shot in Vermont
https://apnews.com/article/vermont-border-patrol-shooting-0c9dfc32ac99d0527c008dfee4a34ac4

2025-01-21 Officials identify US Border Patrol agent killed Monday in Coventry, disclose more details of shooting
https://vtdigger.org/2025/01/21/vermont-officials-identify-u-s-border-patrol-agent-killed-monday-in-coventry/

2025-01-23 US border patrol agent was fatally shot in Vermont near Canada
https://apnews.com/article/vermont-border-patrol-shooting-d40ffa36bad16cfce6174daa3cd3fb96

2025-01-23 Law enforcement honors fallen border patrol agent as investigators provide few new details of Coventry shooting
https://vtdigger.org/2025/01/23/law-enforcement-honors-fallen-border-patrol-agent-as-investigators-provide-few-new-details-of-coventry-shooting/

2025-01-24 Federal prosecutors file charges in probe of fatal shooting of border patrol agent in Vermont
https://vtdigger.org/2025/01/24/federal-prosecutors-file-charges-in-probe-of-fatal-shooting-of-border-patrol-agent-in-vermont/

2025-01-24 Woman facing charges in deadly shooting of Border Patrol agent in Vermont
https://www.wcax.com/2025/01/24/woman-facing-charges-deadly-shooting-border-patrol-agent-vermont/

2025-01-25 Washington state woman is charged in the fatal shooting of a Border Patrol agent in Vermont
https://apnews.com/article/vermont-border-agent-killed-d99eb6157af85255fa23eea4b2658869

2025-01-27 Suspects in killings of Vallejo witness, Vermont border patrol agent connected by marriage license, extreme ideology
https://openvallejo.org/2025/01/27/suspects-in-killings-of-vallejo-witness-vermont-border-patrol-agent-connected-by-marriage-license-extreme-ideology/

2025-01-27 Man killed in Vallejo was main witness in upcoming murder trial
https://openvallejo.org/2025/01/27/man-killed-in-vallejo-was-main-witness-in-upcoming-murder-trial/

2025-01-27 Prosecutor says woman charged in fatal border agent shooting had ties to people of interest in other homicides
https://vtdigger.org/2025/01/27/prosecutor-says-woman-charged-in-fatal-border-agent-shooting-had-ties-to-people-of-interest-in-other-homicides/

2025-01-28 Federal officials link 3rd ‘person of interest’ to Vermont border patrol agent’s shooting
https://vtdigger.org/2025/01/28/federal-officials-link-third-person-of-interest-to-vermont-border-patrol-agents-shooting/

2025-01-29 String of recent killings linked to Bay Area 'death cult'
https://www.sfgate.com/bayarea/article/bay-area-death-cult-zizian-murders-20064333.php

2025-01-29 Federal officials link 3rd ‘person of interest’ to Vermont border patrol agent’s shooting
https://openvallejo.org/2025/01/29/federal-officials-link-3rd-person-of-interest-to-vermont-border-patrol-agents-shooting/

2025-01-30 More bizarre details uncovered about leader of alleged Bay Area 'death cult'
https://www.sfgate.com/bayarea/article/leader-alleged-bay-area-death-cult-faked-death-sf-20066610.php

2025-01-30 The case of the radical ‘Zizian’ vegan trans cult and the shooting death of a Border Patrol agent
https://nypost.com/2025/01/30/us-news/killing-of-border-patrol-agent-appears-linked-to-zizian-radical-leftist-trans-cult/

2025-02-02 Key figure in ‘Zizian’ group tied to Vermont border patrol shooting faked death in 2022 and is wanted in 2 states
https://vtdigger.org/2025/02/02/key-figure-in-zizian-group-tied-to-vermont-border-patrol-shooting-faked-death-in-2022-and-is-wanted-in-2-states/

2025-02-06 Before the shooting of border patrol officer, visitors took interest in a remote Northeast Kingdom property
https://vtdigger.org/2025/02/06/before-the-shooting-of-border-patrol-officer-visitors-took-interest-in-a-remote-northeast-kingdom-property/

On one hand, thank you, I thought it might be a good idea to have a collaboratively edited document on Zizians. On the other hand, now that I look at the pages, it seems excessive. Do we really need a separate page on each Zizian? Also, are we supposed to just summarize what was in the news and legal documents, or provide our own explanations? But providing our own explanations about living people exposes us to potential libel charges. Etc.

So... I don't know. I wished we had something like this, but now that I see it, I am not so sure anymore.

I am thankful that the glossary exists, because it makes it easier to decode various Zizian writings, and makes it more difficult to sanewash Ziz.

For example, now I have a convenient proof that Ziz literally believes that there are two persons in each human. Not as a vague metaphor for "people are complicated". Literally two. Literally in everyone. Literally persons, in a way that it makes sense to describe them individually as male or female, good or "nongood". Literally believing that you can talk with the individual persons, make them argue against each other, make one murder another.

Which is convenient, because currently I am working on an article explaining how the popular "left brain, right brain" theory is complete bullshit from the scientific perspective. Which means, the Zizian model is bullshit, because it builds on the popular misconception. -- Without the glossary, if I succeed to write the article and it turned out to be convincing, fans of Ziz could simply say "but of course Ziz didn't mean it that way, stop strawmanning her". But now we have written evidence that yes, Ziz meant it literally that way, therefore all the supposed insights people gained from talking to their individual hemispheres should be attributed to some form of dissociative identity disorder, rather then each hemisphere being a person.

The title "infohazardous glossary" sounds pretty insane. The contents of that webpage also strike me as pretty insane.

That's the way Zizians speak. Everything must be said in the most hysterical way possible. Everything they don't like is slavery or something. Every disagreement is addressed by a death threat (though they usually do not act on them). First it seemed like they were just hysterical idiots. Then they actually killed some people. Now it's more like: murderous hysterical idiots.

Sleep deprivation is a traditional mind-control technique in cults; makes it difficult to disbelieve.

Of course you can't just tell your recruits "I need you to be sleep-deprived so that you will find my teaching more credible". Instead, there is so little time and so much work to do. Also, waking up early is healthy (but somehow we forget that going to bed early is healthy, too).

Using sleep deprivation as a way to "know yourself" is an interesting new take. You don't even have to organize the work and the early meditations/prayers, your recruits will voluntarily keep themselves sleep-deprived even when there is absolutely nothing to do. Amazing!

See the Zizian "Infohazardous Glossary":

Left-Female

Of a human, having a female left hemisphere. Abbreviated “lf”.

Right-Female

Of a human, having a female right hemisphere. Abbreviated “rf”.

Left-Male

Of a human, having a male left hemisphere. Abbreviated “lm”.

Right-Male

Of a human, having a male right hemisphere. Abbreviated “rm”.

lmrf

Left-male, right-female.

lfrm

Left-female, right-male.

Double-Female

Both hemispheres female. Cis women or binary trans women. Abbreviated “df”.

Double-Male

Both hemispheres male. Cis men or binary trans men. Abbreviated “dm”.

Seems quite literal.

I often try, but (1) it costs some time, and (2) sometimes it is quite difficult, especially to comment in a way that I think would be understood by the author. Sometimes the author is just so far away in the mental space that I don't believe that a short message could reach him... and I don't have a time to write a long one. (I could write a short message that would predictably fail, just to signal that I am a nice person, but that wouldn't help anyone.)

It's just this: the things that survive will have characteristics that are best for helping it survive.

With some assumptions, for example that the characteristics are permanent (-ish), and preferably heritable if the thing reproduces.

See "No Evolutions for Corporations or Nanodevices"

I don't know. (Which is a convenient way to end this thread.)

Some information is at https://zizians.info/ but it is far from complete, when it comes to the technical details of Zizianism.

My visual metaphor is the angel and the devil sitting on your shoulders, each whispering in one of your ears. Except, they live inside your respective brain hemispheres, because obviously literal angels and devils are unscientific, but left and right brain are the Science™.

That makes Ziz like Jesus, born without sin. Explained by having two angels, conveniently.

(Also, both the angels and the devils can be male or female, which provides a theological Rationalist foundation for explaining trans-sexuality. Makes it easier to recruit among trans-sexual rationalists. Know yourself, by listening to the only person who has the knowledge.)

So the Zizian technology, which involves sleep deprivation and then having one eye closed and the other eye open (as a way to make one personality sleep), seems completely unsupported by what we know about human biology.

It's just creating a split personality, in a way that has nothing to do with the hemispheres. But if you believe that your personalities are already there, waiting in the hemispheres until you find them, it probably helps with the process of creating them (which then feels like a confirmation of the theory).

In one of those two cases, the entire "quote" was one sentence, completely made up (and crazy).

The other case was a mixed bag, where most statements were quoted correctly (not literally, but in a way that didn't change their meaning, and I am perfectly okay with that), but there was one specific thing where the journalist clearly wanted me to say something, tried various ways of "but wouldn't it be possible that..." and "you can't be 100% sure that it isn't the case that...", and after I stubbornly resisted, she just made up a quote that agreed with her, and that I would obviously never have said.

But outside of that one thing, the rest of the interview was okay.

I also know two journalists who make interviews in a style "let's talk for half an hour in front of a camera, then publish it online with minimum editing". One of them started doing it on YouTube, later he got employed in a mainstream newspaper. The other already started as a journalist, first doing paper interviews, later also video interviews. (Both of them non-English.) These two I would trust in a video interview, and probably also in a paper interview. But I've never interacted with either of them.

I think there is a conflation of two different things:

1. Human brain has two hemispheres which communicate through a relatively lower-bandwidth channel, which means they process a lot of things independently.
2. There is the dissociative identity disorder / alter ego / tulpa phenomenon, where a human can produce two or more identities. This is probably something that exists on a spectrum, where the extreme forms are full different personalities with dissociative amnesia; imaginary friends and brainwashing are somewhere in the middle; and the everyday forms are role-playing or different moods.

If I understand it correctly, Ziz assumes that these two are the same thing. Which is pseudoscientific, and in my opinion clearly wrong.

First, because there can be more than two identities (but no one has more than two brain hemispheres, I suppose). Yes, two is the most famous number, but that's simply because two is the smallest integer that is greater than one, and more personalities are less frequent.

Second, even if there are exactly two identities, there is no evidence mapping them to two hemispheres (as opposed to each of them using both hemispheres), and a lot of obvious evidence against that, for example the fact that each personality can use both hands etc.

However the idea of "left brain, right brain" is quite popular in our culture. And there were a few experiments showing that the hemispheres can be separated, and then weird things happen. Which means that Ziz's theory may sound plausible to many people, even in the rationalist community.

Ziz assumes that there are (1) exactly two (2) permanent "cores" in every human. The number two and the permanence are the crucial parts of her ideology; in my opinion this is incompatible with any Buddhist doctrine, which would actually put the emphasis on their impermanence.

The permanence of the "cores", and the Manichean perspective that each of them is either perfectly good or perfectly evil, is the basis of social control that Ziz has over her followers. You can't meaningfully disagree with Ziz, because she is 100% good, and you are 50% good and 50% evil, which means that any disagreement must obviously originate in your evil half, and therefore you should mobilize your good half to fight against it (or kill yourself, if you cannot win). The only moral choice is to believe and obey Ziz unconditionally.

.

I kinda assume that multiple personalities are "just" a stronger form of what people normally do, and that different personalities can present as different genders (including agender etc.).

I reject the "exactly two" and "it maps to hemispheres" parts, the permanence of the personalities, and the Manichean ethics.