Comment by kaj_sotala on Rest Days vs Recovery Days · 2019-03-20T18:38:45.890Z · score: 7 (3 votes) · LW · GW

I think that "video games" is too broad of a category here; I think there are both games which feel Resty and games which feel more Recovery-y.

Comment by kaj_sotala on Ask LW: Have you read Yudkowsky's AI to Zombie book? · 2019-03-17T16:38:29.352Z · score: 6 (3 votes) · LW · GW

I read most of the Sequences when they came out, but haven't tried re-reading them in the book.

Comment by kaj_sotala on Motivational Meeting Place · 2019-03-17T16:28:27.763Z · score: 16 (5 votes) · LW · GW

The Less Wrong Study Hall at least used to be something like this (1 2 3). I don't know how active it currently is.

Comment by kaj_sotala on Active Curiosity vs Open Curiosity · 2019-03-15T13:29:14.134Z · score: 7 (4 votes) · LW · GW

Do you mean to say that open curiosity is diffuse curiosity? If so, then that feels a little off, since you can have open curiosity about e.g. one particular emotion that happens to be in your mind.

Comment by kaj_sotala on Active Curiosity vs Open Curiosity · 2019-03-15T08:46:57.498Z · score: 13 (5 votes) · LW · GW

Open curiosity very much sounds to me like a familiar aspect of the state that IFS calls "being in Self": you're curious about the nature of your parts and those of others, but you don't have any particular agenda that you would want to achieve, and are open both to the possibility of learning more about your parts as well as to the possibility that they won't tell you anything more on this particular session.

When facilitating someone else's IFS session while in Self, you are also not driven by any particular goal (including wanting to "fix" them), but are also just open to exploring things together and seeing what you will find. It's a familiar and pleasant state to me.

In my IFS training, I explicitly heard Self being described both as a curious state and as a state where you don't have an agenda, so I don't think that this is just my interpretation either. Also watching other people who got into a strong state of Self, definitely seemed like they were very much in a state of open curiosity.

Meditation has also once brought me into an even stronger state of Self / open curiosity than IFS work has managed to: one where I've been totally open to all mental and physical experience, without even the slightest need to experience particular thoughts and emotions, and have just been deeply at peace with whatever I might experience.

Comment by kaj_sotala on Understanding information cascades · 2019-03-14T04:58:02.510Z · score: 12 (5 votes) · LW · GW

Separately, there's a part of me that finds it viscerally annoying to have multiple questions around the same theme posted around the same time. It feels like it incentivizes people with a pet topic to promote that topic by asking a lot of questions about it so that other topics get temporarily drowned out. Even if the topic is sometimes important enough to be worth it, it still feels like the kind of thing to discourage.

Comment by kaj_sotala on Understanding information cascades · 2019-03-14T04:50:12.182Z · score: 11 (4 votes) · LW · GW

Is it necessarily a good idea to break up the topic into so many separate questions before having a general discussion post about it first? I would imagine that people might have comments which were related to several of the different questions, but now the discussion is going to get fragmented over many places.

E.g. if someone knows about a historical info cascade in academia and how people failed to deal with that, then that example falls under two different questions. So then the answer with that example either has to be be split into two or to be posted in an essentially similar form on both pages, neither of which is good for keeping the entire context of the discussion in one place.

Comment by kaj_sotala on Subagents, introspective awareness, and blending · 2019-03-13T08:06:39.007Z · score: 5 (2 votes) · LW · GW

Thanks! That's very strong praise. :)

As an aside, from your article:

IFS is a psychoanalytic model that, to my reading, originates from thinking of the Freudian id, ego, and superego like three homunculi competing to control the mind, but replaces the id, ego, and superego with groups of subagents serving exile, firefighter, and manager roles, respectively. These roles differer in both subtle and obvious ways from their inspirational forms,

FWIW, at least Richard Schwartz's published account of the origins of IFS contradicts this reading. Rather than basing the model on any previous theoretical framework, he reports that the observations which led to the creation of IFS actively contradicted many of his previous theoretical assumptions and that he was forced to basically throw them away and construct something new based on what he actually encountered in clincal work. E.g. a few excerpts from Internal Family Systems Theory:

Much of what is contained in this book was taught to me by my clients. Not only did they teach the contents of this approach; they also taught me to listen to them and trust their reports of what happened inside them, rather than to fit their reports into my preconceptions. This lesson was several years in the learning. I had to be corrected repeatedly. Many times clients forced me to discard ideas that had become mainstays of the theory. I also wrestled perpetually with my skeptical parts, which could not believe what I was hearing, and worried about how these reports would be received by academic colleagues. The reason why this model has immediate intuitive appeal for many people is that ultimately I was sometimes able to suspend my preconceptions, tune down my inner skeptics, and genuinely listen. [...]

Some therapists/theorists listened to their clients and consequently walked segments of this intriguing inner road before me. They provided lanterns and maps for some of the territory, while for other stretches I have had only my clients as guides. For the first few years after entering this intrapsychic realm, I assiduously avoided reading the reports of others, for fear that my observations would be biased by preconceptions of what existed and was possible. Gradually I felt secure enough to compare my observations with others who had directly interacted with inner entities. I was astounded by the similarity of many observations, yet intrigued by some of the differences. [...]

Born in part as a reaction against the acontextual excesses of the psychoanalytic movement, the systems-based models of family therapy traditionally eschewed issues of intrapsychic process. It was assumed that the family level is the most important system level to change, and that changes at this level trickle down to each family member’s inner life. Although this taboo against intrapsychic considerations delayed the emergence of more comprehensive systemic models, the taboo had the beneficial effect of allowing theorists to concentrate on one level of human system until useful adaptations of systems thinking could evolve.

I was one of these “external-only” family therapists for the first 8 years of my professional life. I obtained a PhD in marital and family therapy, so as to become totally immersed in the systems thinking that I found so fascinating and valuable. I was drawn particularly to the structural school of family therapy (Minuchin, 1974; Minuchin & Fishman, 1981), largely because of its optimistic philosophy. Salvador Minuchin asserted that people are basically competent but that this competence is constrained by their family structure; to release the competence, change the structure. The IFS model still holds this basic philosophy, but suggests that it is not just the external family structure that constrains and can change. [...]

This position—that it is useful to conceive of inner entities as autonomous personalities, as inner people—is contrary to the common-sense notion of ourselves. [...] It was only after several years of working with people’s parts that I could consider thinking of them in this multidimensional way, so I understand the difficulty this may create for many readers. [...]

When my colleagues and I began our treatment project in 1981, bulimia was a new and exotic syndrome for which there were no family systems explanations or maps. In our anxiety, we seized the frames and techniques developed by structural family therapists for other problems and tried them on each new bulimia case. In several instances this approach worked well, so we leaped to the conclusion that the essential mechanism behind bulimia was the triangulation of the client with the parents. In that sense, we became essentialists; we thought we had found the essence, so we could stop exploring and could use the same formula with each new case. Data that contradicted these assumptions were interpreted as the result of faulty observations or imperfect therapy.

Fortunately, we were involved in a study that required close attention to both the process and outcome of our therapy. As our study progressed, the strain of trying to fit contradictory observations into our narrow model became too great, too Procrustean. We were forced to leave the security and simplicity of our original model and face the anomie that accompanies change. We were also forced to listen carefully to clients when they talked about their experiences and about what was helpful to them. We were forced to discard our expert’s mind, full of preconceptions and authority, and adopt what the Buddhists call beginner’s mind, which is an open, collaborative state. “In the beginner’s mind there are many possibilities; in the expert’s mind there are few” (Suzuki, 1970, p. 21). In this sense, our clients have helped us to change as much as or more than we have helped them.

The IFS model was developed in this open, collaborative spirit. It encourages the beginner’s mind because, although therapists have some general preconceptions regarding the multiplicity of the mind, clients are the experts on their experiences. The clients describe their subpersonalities and the relationships of these to one another and to family members. Rather than imposing solutions through interpretations or directives, therapists collaborate with clients, respecting their expertise and resources.
Comment by kaj_sotala on Subagents, introspective awareness, and blending · 2019-03-13T07:46:12.970Z · score: 3 (1 votes) · LW · GW

Huh! That's very interesting. Has your mind always been like that? As I'll discuss further in the next post, there are meditation techniques which are intended to get people closer to that level of introspective awareness, but I don't know whether anyone has reached the kind of a level that you're describing through practice.

I'm curious - to what extent do you experience things like putting off things which you feel you should do now, or doing things which you know that you'll regret later (like staying up too late)? (My prediction would be that you basically don't.)

Comment by kaj_sotala on Karma-Change Notifications · 2019-03-06T22:41:20.115Z · score: 17 (6 votes) · LW · GW

I think I might have posted significantly less on the original Less Wrong if it had made downvotes this salient at a time when I wasn't yet confident in the quality of my contributions, and if it didn't offer a "don't show downvotes" option.

Comment by kaj_sotala on Karma-Change Notifications · 2019-03-05T12:59:51.570Z · score: 27 (5 votes) · LW · GW
We had the feature activated for mods only in the last three weeks, and I did notice it generally increasing my satisfaction with LessWrong by a pretty significant amount.

What's the current thinking on the way it displays downvotes? My today's karma change showed a red -7, and while I've got enough karma that I don't care that much about karma changes in general and could just shrug it off, it did feel more distinctly more unpleasant than much larger karma increases in the previous days have felt pleasant. I'm a little worried that this might make new users take downvotes even more harshly than they otherwise would.

Sites like FB tend to basically only display positive information, probably for this reason - e.g. they tell you when someone accepts your friend request or likes your comment, but they don't tell you when someone unfriends you or (AFAIK) when someone deletes your comment. I'm not sure if that's quite the right answer here - you did mention wanting to make users also more responsive to downvotes - but having a user come back to the site and run into a red negative number as one of the first things they see feels a little harsh. Maybe have the immediately-visible-number only display the sum of upvotes, but still show the downvotes when clicked on?

Comment by kaj_sotala on "AlphaStar: Mastering the Real-Time Strategy Game StarCraft II", DeepMind [won 10 of 11 games against human pros] · 2019-03-05T09:51:50.175Z · score: 11 (2 votes) · LW · GW

Oh, I thought it was already plainly obvious to everyone that the victories were in part because of unfair AI advantages. We don't need to discuss the APM cap for that, that much was already clear from the fact that the version of AlphaStar which had stricter vision limitations lost to MaNa.

That just seems like a relatively uninteresting point to me, since this looks like AlphaStar's equivalent of the Fan Hui game. That is, it's obvious that AlphaStar is still below the level of top human pros and wouldn't yet beat them without unfair advantages, but if the history with AlphaGo is any guide, it's only a matter of some additional tweaking and throwing more compute at it before it's at the point where it will beat even the top players while having much stricter limitations in place.

Saying that its victories were *unrelated* to what we'd intuitively consider thinking seems too strong, though. I'm not terribly familiar with SC2, but a lot of the discussion that I've seen seems to tend towards AlphaStar's macro being roughly on par with the pro level, and its superior micro being what ultimately carried the day. E.g. people focus a lot on that one game that MaNa arguably should have won but only lost due to AlphaStar having superhuman micro of several different groups of Stalkers, but I haven't seen it suggested that all of its victories would have been attributable to that alone: I didn't get that kind of a vibe from MaNa's own post-game analysis of those matches, for instance. Nor from TLO's equivalent analysis (lost the link, sorry) of his matches, where he IIRC only said something like "maybe they should look at the APM limits a bit [for future matches]".

So it seems to me that even though its capability at what we'd intuitively consider thinking wouldn't have been enough for winning all the matches, it would still have been good enough for winning several of the ones where people aren't pointing to the superhuman micro as the sole reason of the victory.

Comment by kaj_sotala on Rule Thinkers In, Not Out · 2019-03-03T19:35:36.354Z · score: 15 (5 votes) · LW · GW

(After seeing this article many times, I only now realized that the title is supposed to be interpreted "thinkers should be ruled in, not out" rather than "there's this class of rule thinkers who are in, not out")

Comment by kaj_sotala on Subagents, introspective awareness, and blending · 2019-03-03T15:29:02.060Z · score: 3 (1 votes) · LW · GW

Thanks, glad to hear you liked it! I didn't have a chance to look at your linked stuff yet, but will. :)

Comment by kaj_sotala on Book Summary: Consciousness and the Brain · 2019-03-03T15:24:27.301Z · score: 4 (2 votes) · LW · GW

That's a good question. Dehaene explicitly talks about the "objects" corresponding to chunks, so that one of the chunks would be consciousness at a time. There's also a finding that when people are asked to maintain a number of words or digits in memory, the amount of items that they can maintain depends on how many syllables those items have. And since e.g. Chinese has shorter words for various digits than English does, native Chinese speakers can maintain more digits in their working memory than native English speakers.

One standard interpretation has been that "working memory" is composed of a number of different storage systems, each of which is capable of rehearsing a memory trace for a limited time. It would be something like a submodule connected to the workspace, which can take items from the workspace and then "play them back", but its memory decays quickly and it has to keep refreshing that memory by putting object that it has stored back into the workspace in order to then re-store them. So consciousness could still only hold one item, but it was augmented by "caches" which allowed it to rapidly circle through a number of items.

The thing about seeing whole pictures confuses me a bit too, though change blindness experiments would suggest that seeing all of it at once is indeed an illusion to at least some extent. One of the things that people tend to notice when learning to draw is also that they actually don't really see the world as it is, and have to separately learn this.

If we're willing to move away from psychological experiments and also incorporate stuff from the theory of meditation, The Mind Illuminated also has the "only one item object at a time" thing, but distinguishes between objects of attention and objects of awareness:

... any moment of consciousness—whether it’s a moment of seeing, hearing, thinking, etc.—takes the form of either a moment of attention, or a moment of peripheral awareness. Consider a moment of seeing. It could be either a moment of seeing as part of attention, or a moment of seeing as part of peripheral awareness. These are the two options. If it’s a moment of awareness, it will be broad, inclusive, and holistic—regardless of which of the seven categories it belongs to. A moment of attention, on the other hand, will isolate one particular aspect of experience to focus on.
If we examine moments of attention and moments of awareness a bit closer, we see two major differences. First, moments of awareness can contain many objects, while moments of attention contain only a few. Second, the content of moments of awareness undergoes relatively little mental processing, while the content of moments of attention is subject to much more in-depth processing. [...]
Consider the first difference, many objects versus only a few, in terms of hearing. Our ears take in everything audible from our environment. Then our brain processes that information and puts it together in two different ways. First, it creates an auditory background that includes more or less all the different sounds our ears have detected. When that’s projected into consciousness, it becomes a moment of auditory peripheral awareness. The other way the brain processes that information is to pick out just some part—say, one person’s voice—from the total sound in our awareness. When projected into consciousness, that isolated sound becomes the content of a moment of auditory attention. So, the brain has two modes of information processing: one creates moments of awareness with many objects, while the other creates moments of attention with just a few.
These two modes apply to every kind of sensory information, not just hearing. For example, say you’re sitting on a cabin deck in the mountains, gazing out at the view. Each moment of visual awareness will include a variety of objects—mountains, trees, birds, and sky—all at the same time. Auditory moments of awareness will include all the various sounds that make up the audible background—birdsong, wind in the trees, a babbling brook, and so forth—again, all at the same time. On the other hand, moments of visual attention might be restricted just to the bird you’re watching on a nearby branch. Auditory attention might include only the sounds the birds are making. Even when your attention is divided among several things at once—perhaps you’re knitting or whittling a piece of wood while you sit—moments of attention are still limited to a small number of objects. Finally, binding moments of attention and binding moments of awareness take the content from the preceding sensory moments and combine them into a whole: “Sitting on the deck, looking out at the mountain, while carving a piece of wood.”
Comment by kaj_sotala on Subagents, introspective awareness, and blending · 2019-03-03T11:15:23.465Z · score: 5 (2 votes) · LW · GW

The qualia interpretation was what I had in mind when writing this, though of course Dehaene's work is based on conscious access in the sense of information being reportable to others.

It is completely possible to have subjective experiences without thinking anything about them or remembering them. When a person asks "Am I consciousness?" and concludes that she was not consciousness until that question, it doesn't mean she was a phylozombie the whole day.

Agreed, and the post was intended (in part) as an explanation of why this is the case.

Subagents, introspective awareness, and blending

2019-03-02T12:53:47.282Z · score: 58 (18 votes)
Comment by kaj_sotala on Policy-Based vs Willpower-Based Intentions · 2019-03-01T08:19:25.178Z · score: 3 (1 votes) · LW · GW

Thanks, that clarifies it considerably. That tipping example does sound like the kind of a process that I might also go through.

Comment by kaj_sotala on Policy-Based vs Willpower-Based Intentions · 2019-02-28T21:52:03.143Z · score: 8 (4 votes) · LW · GW

We don't really do tipping or paying-for-the-other here, so those don't come up. I do have a general policy of not giving money, but the way by which I "installed" it doesn't resemble your description: it was something like, I ran into people asking for money on several occasions, then on each occasion made a decision about what to do, until I got sufficiently frustrated with feeling manipulated after giving money that the emotion drove me into making a decision which I'd remember in the future.

I guess the thing that sounded the most unusual to me was this part:

It basically costs no willpower to implement the policy. I’m not having to nudge myself, “Now remember I decided I’d do X in these situations.” I’m not having to consciously hold the intention in my mind. It’s more like I changed the underlying code—the old, default behavior—and now it just runs the new script automatically. 

It's not that I don't have policies, it's that this description sounds like you can just... decide to change a policy, and then have that happen automatically. And sure, sometimes it's a small enough thing that I can do this. But like with the key thing, if I suddenly decided to put my keys somewhere else than I used to, I bet that I'd keep putting them in the wrong place until I gradually learned the new policy.

Or maybe your description just makes the concept sound more exotic than you intended and I'm misinterpreting. It sounded to me like you have some magic method of automatically instantiating policy changes that I would need to spend willpower on, without you needing to spend willpower on them. E.g. the Lyft thing sounded complicated to memorize and I would probably need to consciously think about it on several times when I was actually doing the tipping before I had it committed into memory. But you said that you can't use this on everything, so maybe the policies that I would need willpower to install just happen to be different from the policies that you would need willpower to install.

I guess one thing that confuses me is that you say that it costs no willpower to implement a policy, and then you contrast it with willpower-based intentions, which do cost willpower. That makes it sound like costing willpower, is a property of how you've made the decision? But to me it feels like costing willpower is more of a property of the choice. In the other comment you gave the example of having a policy of how you arrange your kitchen, and this sounds to me like the kind of thing that everyone would have as a policy-that-cost-no-willpower, because you can just decide it and there's no need to use willpower on it. Whereas (to again use your example) something like drinking water every day is more complicated, so you can't just make it into a policy without using willpower.

So my model is that everyone already does everything as a policy-based intention, unless there's some property of the act which forces them to use willpower, in which case they can't do it as a policy and have to use some willpower instead. But your post sounded like people have an actual choice in which way around they do it, as opposed to having it decided for them by the nature of the task?

Comment by kaj_sotala on Policy-Based vs Willpower-Based Intentions · 2019-02-28T20:41:51.499Z · score: 15 (4 votes) · LW · GW

Could you go into more detail on how exactly one sets these policy-based intentions? I don't think I know how, nor did I ever even suspect that such a possibility exists.

It does slightly remind me of the KonMari thing where you go through all of your belongings to get a felt sense of where in the apartment they belong, and then automatically put things in their right places because any other place feels wrong.

I've used that for a few things like "where should I put my keys", and phenomenologically it does feel like it just kinda... rewrites my brain to make that feel like the only right place to put them, and then it requires no willpower after that. But that procedure feels more like discovering the right place rather than consciously deciding it; if there's a more general thing that I can do in this category, I'd be curious about hearing more.

Comment by kaj_sotala on Humans Who Are Not Concentrating Are Not General Intelligences · 2019-02-26T21:11:33.507Z · score: 11 (8 votes) · LW · GW
When humans skim semi contradictory text, they produce a more consistent world model that doesn't quite match up with what is said.

I felt like something like this happened to me when I was reading some of the "nonsensical" examples in the post, rather than deeming the text outright nonsensical and non-human I just interpreted it as the writer being sloppy.

Comment by kaj_sotala on Avoiding Jargon Confusion · 2019-02-19T16:49:33.838Z · score: 5 (2 votes) · LW · GW
However, it would never be apparent to me from hearing the phrase

Seconded; this interpretation didn't ever occur to me before reading Raemon's comment just now.

Comment by kaj_sotala on Avoiding Jargon Confusion · 2019-02-19T16:33:48.436Z · score: 14 (3 votes) · LW · GW

Eliezer also mentioned this in his old article on writing advice:

9. Don't invent new words.
Yes, I frequently violate this myself, but at least I've been trying to keep it down.
If you do violate the rule, then make the new word as self-explanatory as possible. "Seed AI", "Friendly AI", and "neurohacking" are good. "External reference semantics" is bad.
Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-02-17T16:41:02.498Z · score: 6 (3 votes) · LW · GW

Glad to hear it's been of use!

Which agent should the sympathetic listener be talking to? The manager, the exile, or both?

First with any of the managers which might be protecting the exiles. Eventually they might give access to the exile, but it's important to not try to rush through them. You only go to the exile after the managers have agreed to give you access to it: bypassing them risks causing damage because the managers had concerns which weren't taken into account. (Self-Therapy has detailed instructions on this.) You might e.g. end up exposing an exile in a situation where you don't have the resources to handle it, and then instead of healing the exile, you end up worsening the original trauma. That will also have the added effect of making your managers less likely to trust you with access to the exile again.

Though sometimes I've had exiles pop up pretty spontaneously, without needing to negotiate with managers. In those situations I've just assumed that all managers are fine with this, since there's no sense of a resistance to contacting the exile. If that happens then it's probably okay, but if it feels like any managers are getting in the way, then address their concerns as much as possible. (As the instructor said in an IFS training I did: "to go fast, you need to go slow".)

IFS also recommends checking back with the managers after healing the exile, so that they can see that the exile is actually healed now and that they can behave differently in the future. Also, you may want to keep checking back with the exile for a while afterwards, to ensure that it's really been healed.

Assuming that one correctly identifies which thoughts (and ultimately, which situations) a manager deems dangerous, and that one successfully does cognitive defusion, to what extent is it feasible, in your opinion, to have the manager (the exile) update by just talking to them vs by experiencing the dangerous situation again but positively?

Depends. I think that either are possible, but I don't have a hard and fast rule: usually I've just gone with whatever felt more right. But I'd guess that in the situations where you can get parts to update just by talking to them, it's in situations where you've already accumulated plenty of evidence about how things are, and the relevant parts just need to become aware of them. E.g. if you had some challenge which was very specifically about your childhood environment, then it shouldn't be too hard to let your parts know that you're no longer in that environment.

On the other hand, for some issues (e.g. social anxiety), the parts might have kept you from ever testing the safety of most situations. For instance, if you're scared of talking to strangers, then you generally won't be talking to strangers. And when you do, you will have parts screaming at you to get out of that situation, which makes it intrinsically unpleasant and won't let you experience it as safe. In that case, you won't actually have collected the evidence needed for making the update, so you need to first persuade the parts to agree that collecting it is sufficiently safe. Then you can go out and get it.

Comment by kaj_sotala on Why didn't Agoric Computing become popular? · 2019-02-17T10:40:10.530Z · score: 4 (2 votes) · LW · GW

It seems like market forces could even actively damage existing cooperation. While I'm not terribly familiar with the details, I've heard complaints of this happening at one university that I know of. There's an internal market where departments need to pay for using spaces within the university building. As a result, rooms that would otherwise be used will sit empty because the benefit of paying the rent isn't worth it.

Possibly this is still overall worth it - the system increasing the amount of spare capacity means that there are more spaces available for when a department really does need a space - but people do seem to complain about it anyway.

Comment by kaj_sotala on Why do you reject negative utilitarianism? · 2019-02-13T13:23:10.544Z · score: 33 (11 votes) · LW · GW

I used to consider myself NU, but have since then rejected it.

Part of my rejection was that, on a psychological level, it simply didn't work for me. The notion that everything only has value to the extent that it reduces suffering meant that most of the things which I cared about, were pointless and meaningless except for their instrumental value in reducing my suffering or making me more effective at reducing suffering. Doing things which I enjoyed, but constantly having a nagging sensation of "if I could just learn to no longer need this, then it would be better for everyone" basically meant that it was very hard to ever enjoy anything. It was basically setting my mind up to be a battlefield, dominated by an NU faction trying to suppress any desires which did not directly contribute to reducing suffering, and opposed by an anti-NU faction which couldn't do much but could at least prevent me from getting any effective NU work done, either.

Eventually it became obvious that even from an NU perspective, it would be better for me to stop endorsing NU, since that way I might end up actually accomplishing more suffering reduction than if I continued to endorse NU. And I think that this decision was basically correct.

A related reason is that I also rejected the need for a unified theory of value. I still think that if you wanted to reduce human values into a unified framework, then something like NU would be one of the simplest and least paradoxical answers. But eventually I concluded that any simple unified theory of value is likely to be wrong, and also not particularly useful for guiding practical decision-making. I've written more about this here.

Finally, and as a more recent development, I notice that NU neglects to take into account non-suffering-based preferences. My current model of minds and suffering is that minds are composed of many different subagents with differing goals; suffering is the result of the result of different subagents being in conflict (e.g. if one subagent wants to push through a particular global belief update, which another subagent does not wish to accept).

This means that I could imagine an advanced version of myself who had gotten rid of all personal suffering, but was still motivated by pursue other goals. Suppose for the sake of argument that I only had subagents which cared about 1) seeing friends 2) making art. Now if my subagents reached agreement of spending 30% of their time making art and 70% of their time seeing friends, then this could in principle eliminate my suffering by removing subagent conflict, but it would still be driving me to do things for reasons other than reducing suffering. Thus the argument that suffering is the only source of value fails; the version of me which had eliminated all personal suffering might be more driven to do things than the current one! (since subagent conflict was no longer blocking action in any situation)

As a practical matter, I still think that reducing suffering is one of the most urgent EA priorities: as long as death and extreme suffering exist in the world, anything that would be called "altruism" should focus its efforts on reducing that. But this is a form of prioritarianism, not NU. I do not endorse NU's prescription that an entirely dead world would be equally good or better as a world with lots of happy entities, simply because there are subagents within me who would prefer to exist and continue to do stuff, and also for other people to continue to exist and do stuff if they so prefer. I want us to liberate people's minds from involuntary suffering, and then to let people do whatever they still want to do when suffering is a thing that people experience only voluntarily.

Comment by kaj_sotala on Minimize Use of Standard Internet Food Delivery · 2019-02-12T08:02:54.219Z · score: 16 (8 votes) · LW · GW

If it is not profitable, then it morally shouldn't ought to exist; the market is indicating that the business is a waste of society's limited resources.

This seems much too strong: it e.g. suggests that no non-profits should exist. Profitability and overall benefit to society are two very different things.

Comment by kaj_sotala on Conclusion to the sequence on value learning · 2019-02-07T20:26:57.810Z · score: 4 (2 votes) · LW · GW

(did you mean to ask goal-directed?)

Comment by kaj_sotala on What are some of bizarre theories based on anthropic reasoning? · 2019-02-05T10:21:35.988Z · score: 3 (1 votes) · LW · GW
We are not currently in the situation of s-risks, so it is not typical state of affairs.

Wouldn't this apply to almost anything? If we are currently not in the situation of X, then X is not a typical state of affairs.

Comment by kaj_sotala on Conclusion to the sequence on value learning · 2019-02-04T13:06:03.706Z · score: 14 (6 votes) · LW · GW
However, this risk is significantly different. If you believed that superintelligent AI must be goal-directed because of math, then your only recourse for safety would be to make sure that the goal is good, which is what motivated us to study ambitious value learning. But if the argument is actually that AI will be goal-directed because humans will make it that way, you could try to build AI that is not goal-directed that can do the things that goal-directed AI can do, and have humans build that instead.

I'm curious about the extent to which people have felt like "superintelligent AI must be goal-directed" has been the primary problem? Now that I see it expressed in this form, I realize that there have for a long time been lots of papers and comments which seem to suggest that this might be people's primary concern. But I always kind of looked at it from the perspective of "yeah this is one concern, but even assuming that we could make a non-goal-directed AI, that doesn't solve the problem of other people having an incentive to make goal-directed-AI (and that's the much more pressing problem)". So since we seemed to agree on goal-directed superintelligence being a big problem, maybe I overestimated the extent of my agreement with other people concerned about goal-directed superintelligence.

Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-01-31T22:07:56.228Z · score: 4 (2 votes) · LW · GW

Wow. I didn't expect to see a therapy approach based on morphic fields.

Comment by kaj_sotala on "AlphaStar: Mastering the Real-Time Strategy Game StarCraft II", DeepMind [won 10 of 11 games against human pros] · 2019-01-31T13:57:44.927Z · score: 3 (3 votes) · LW · GW

It seems very unlikely to me that they would have gotten any less publicity if they'd reported the APM restrictions any differently. (After all, they didn't get any less publicity for reporting the system's other limitations either, like it only being able to play Protoss v. Protoss on a single map, or 10/11 of the agents having whole-camera vision.)

Comment by kaj_sotala on "AlphaStar: Mastering the Real-Time Strategy Game StarCraft II", DeepMind [won 10 of 11 games against human pros] · 2019-01-31T13:55:05.077Z · score: 4 (4 votes) · LW · GW

I was using "fair" to mean something like "still made for an interesting test of the system's capabilities". Under that definition, the explanations seem entirely compatible - they thought that it was an interesting benchmark to try, and also got excited about the results and wanted to share them because they had run a test which showed that the system had passed an interesting benchmark.

Comment by kaj_sotala on Sam Harris and the Is–Ought Gap · 2019-01-31T13:42:48.339Z · score: 38 (15 votes) · LW · GW

I curated this post because "why do people disagree and how can we come to agreement" feels like one of the most important questions of the whole rationality project. In my experience, when two smart people keep disagreeing while having a hard time understanding how the other could fail to understand something so obvious, it's because they are operating under different frameworks without realizing it. Having analyzed examples of this helps understand and recognize it, and then hopefully learn to avoid it.

Comment by kaj_sotala on Can there be an indescribable hellworld? · 2019-01-30T20:40:08.055Z · score: 5 (2 votes) · LW · GW

This seems like it would mainly affect instrumental values rather than terminal ones.

Comment by kaj_sotala on Can there be an indescribable hellworld? · 2019-01-30T20:39:04.834Z · score: 3 (1 votes) · LW · GW

True, not disputing that. Only saying that it seems like an easier problem than solving human values first, and then checking whether those values are satisfied.

Comment by kaj_sotala on Attacking enlightenment · 2019-01-29T21:32:02.847Z · score: 3 (1 votes) · LW · GW

It's been on my to-read list for a while, I bounced off the first time I tried reading it (it seemed to be taking its time to get to the point, started with the personal narrative of the authors and how they got into meditation research etc.) but expect to get around it eventually.

Comment by kaj_sotala on "AlphaStar: Mastering the Real-Time Strategy Game StarCraft II", DeepMind [won 10 of 11 games against human pros] · 2019-01-29T18:00:48.291Z · score: 16 (5 votes) · LW · GW

Apparently as a result of this analysis, DeepMind has edited the caption in the graph:

The distribution of AlphaStar’s APMs in its matches against MaNa and TLO and the total delay between observations and actions. CLARIFICATION (29/01/19): TLO’s APM appears higher than both AlphaStar and MaNa because of his use of rapid-fire hot-keys and use of the “remove and add to control group” key bindings. Also note that AlphaStar's effective APM bursts are sometimes higher than both players.
Comment by kaj_sotala on "AlphaStar: Mastering the Real-Time Strategy Game StarCraft II", DeepMind [won 10 of 11 games against human pros] · 2019-01-29T17:43:49.387Z · score: 7 (7 votes) · LW · GW

Well, apparently that's exactly what happened with TLO and MaNa, and then the DeepMind guys were (at least going by their own accounts) excited about the progress they were making and wanted to share it, since being able to beat human pros at all felt like a major achievement. Like they could just have tested it in private and continued working on it in secret, but why not give a cool event to the community while also letting them know what the current state of the art is.

E.g. some comments from here:

I am an administrator in the SC2 AI discord and that we've been running SC2 bot vs bot leagues for many years now. Last season we had over 50 different bots/teams with prizes exceeding thousands of dollars in value, so we've seen what's possible in the AI space.
I think the comments made in this sub-reddit especially with regards to the micro part left a bit of a sour taste in my mouth, since there seems to be the ubiquitous notion that "a computer can always out-micro an opponent". That simply isn't true. We have multiple examples for that in our own bot ladder, with bots achieving 70k APM or higher, and them still losing to superior decision making. We have a bot that performs god-like reaper micro, and you can still win against it. And those bots are made by researchers, excellent developers and people acquainted in that field. It's very difficult to code proper micro, since it doesn't only pertain to shooting and retreating on cooldown, but also to know when to engage, disengage, when to group your units, what to focus on, which angle to come from, which retreat options you have, etc. Those decisions are not APM based. In fact, those are challenges that haven't been solved in 10 years since the Broodwar API came out - and last Thursday marks the first time that an AI got close to achieving that! For that alone the results are an incredible achievement.
And all that aside - even with inhuman APM - the results are astonishing. I agree that the presentation could have been a bit less "sensationalist", since it created the feeling of "we cracked SC2" and many people got defensive about that (understandably, because it's far from cracked). However, you should know that the whole show was put together in less than a week and they almost decided on not doing it at all. I for one am very happy that they went through with it.

And the top comment from that thread:

Thank you for saying this. A decent sized community of hobbyists and researchers have been working on this for YEARS, and the conversation has really never been about whether or not bots can beat humans "fairly". In the little documentary segment, they show a scene where TLO says (summarized) "This is my off race, but i'm still a top player. If they're able to beat me, i'll be really surprised."
That isn't him being pompous, that's completely reasonable. AI has never even come CLOSE to this level for playing starcraft. The performance of AlphaStar in game 3 against MaNa left both Artosis AND MaNa basically speechless. It's incredible that they've come this far in such a short amount of time. We've literally gone from "Can an AI play SC2 at a high level AT ALL" to "Can an AI win 'fairly'". That's a non-trivial change in discourse that's being completely brushed over IMO.
Comment by kaj_sotala on The 3 Books Technique for Learning a New Skilll · 2019-01-29T17:13:38.981Z · score: 8 (6 votes) · LW · GW

Maybe the "why" book for a language would be something like reading manga for someone wanting to learn Japanese - something that makes the language and culture seem cool and something that you want to learn. :)

Given that learning a language also includes a fair chunk of learning the culture (e.g. knowing which forms of address are appropriate at which times), reading literature from that culture is probably actually useful for accomplishing the "Why" book's goal of explaining the mindset and intuitions behind the skill.

Comment by kaj_sotala on "AlphaStar: Mastering the Real-Time Strategy Game StarCraft II", DeepMind [won 10 of 11 games against human pros] · 2019-01-29T17:09:49.441Z · score: -2 (5 votes) · LW · GW

As noted in e.g. the conversation between Wei Dai and me elsewhere in this thread, it's quite plausible that people thought beforehand that the current APM limits were fair (DeepMind apparently consulted pro players on them). Maybe AlphaStar needed to actually play a game against a human pro before it became obvious that it could be so overwhelmingly powerful with the current limits.

Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-01-29T16:50:18.191Z · score: 31 (7 votes) · LW · GW
things that had been too scary for me to think about became thinkable (e.g. regrettable dynamics in my romantic relationships), and I think this is a crucial observation for the rationality project. When you have exile-manager-firefighter dynamics going on and you don't know how to unblend from them, you cannot think clearly about anything that triggers the exile, and trying to make yourself do it anyway will generate tremendous internal resistance in one form or another (getting angry, getting bored, getting sleepy, getting confused, all sorts of crap), first from managers trying to block the thoughts and then from firefighters trying to distract you from the thoughts. Top priority is noticing that this is happening and then attending to the underlying emotional dynamics.

Yes!

Valentine has also written some good stuff on this, in e.g. The Art of Grieving Well:

I think the first three so-called “stages of grief” — denial, anger, and bargaining — are avoidance behaviors. They’re attempts to distract oneself from the painful emotional update. Denial is like trying to focus on anything other than the hurt foot, anger is like clutching and yelling and getting mad at the situation, and bargaining is like trying to rush around and bandage the foot and clean up the blood. In each case, there’s an attempt to keep the mind preoccupied so that it can’t start the process of tracing the pain and letting the agonizing-but-true world come to feel true. It’s as though there’s a part of the psyche that believes it can prevent the horror from being real by avoiding coming to feel as though it’s real. [...]
In every case, the part of the psyche driving the behavior seems to think that it can hold the horror at bay by preventing the emotional update that the horror is real. The problem is, success requires severely distorting your ability to see what is real, and also your desire to see what’s real. This is a cognitive black hole — what I sometimes call a “metacognitive blindspot” — from which it is enormously difficult to return.
This means that if we want to see reality clearly, we have to develop some kind of skill that lets us grieve well — without resistance, without flinching, without screaming to the sky with declarations of war as a distraction from our pain.
We have to be willing to look directly and unwaveringly at horror.

and also in Looking into the Abyss:

It would be bad if pain weren’t automatically aversive and we had to consciously remember to avoid things that cause it. Instead, we have a really clever automatic system that notices when something is bad or dangerous, grabs our conscious attention to make us change our behavior, and often has us avoiding the problem unconsciously thereafter.
But because pain is an interpretation rather than a sensation, avoiding it acts as an approximation of avoiding things that are actually bad for us.
This can result in some really quirky behavior on beyond things like dangerously bending at the waist. For instance, moving or touching ourselves seems to distract us from painful sensations. So if the goal is to decrease conscious experience of pain, we might find ourselves automatically clutching or rubbing hurt body parts, rocking, or pounding our feet or fists in response to pain. Especially the latter actions probably don’t help much with the injury, but they push some of the pain out of mind, so many of us end up doing this kind of behavior without really knowing why.
Writhing in agony strikes me as a particularly loud example: if some touch and movement can block pain, then maybe more touch and movement can block more pain. So if you’re in extreme pain and the goal is to get away from it, large whole-body movements seem to make sense. (Although I think there might be other reasons we do this too.)
To me, this looks like a Red Queen race, with the two “competitors” being the pain system and the “distract from pain” reflex. First the pain system tries to get our attention and change our behavior (protect a body part, get help, etc.). This is unpleasant, so the look-away reflex grabs onto the nearest available way to stop the experience of pain, and muddles some of the sensation that’s getting labeled as pain. The pain system still perceives a threat, though, so it turns up the volume so to speak. And then the look-away reflex encourages us to look even more wildly for a way out, which causes pain’s volume to go up even more….

The bit about a Red Queen race sounds to me exactly like the description of an exile/firefighter dynamic, though of course there's a deeper bit there about some things being so painful as to trigger a firefighter response even if one didn't exist previously. Probably everyone has some "generic" firefighters built right into the psyche which are our default response to anything sufficiently uncomfortable - similar to the part in my robot design which mentioned that

If a certain threshold level of “distress” is reached, the current situation is designated as catastrophic. All other priorities are suspended and the robot will prioritize getting out of the situation.

even before I started talking about specialized firefighters dedicated to keeping some specific exiles actually exiled. And in the context of something like physical pain or fear of a predator, just having a firefighter response that's seeking to minimize the amount of experienced distress signal makes sense. The presence of the distress signal is directly correlated with the extent of danger or potential threat, so just having "minimize the presence of this signal" works as an optimization criteria which is in turn directly correlated with optimizing survival.

But when we get to things like "thinking about romantic success" or "thinking about existential risk", it's no longer neatly the case that simply not experiencing the stress of thinking about those things is useful for avoiding them...

Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-01-29T16:25:29.947Z · score: 4 (2 votes) · LW · GW

Whoa, glad you found it that useful! Thank you for letting me know. :)

I do recommend reading at least Self-Therapy too, it mentions a number of details which I left out of this explanation, and which might be useful to know about when addressing future issues.

Comment by kaj_sotala on Can there be an indescribable hellworld? · 2019-01-29T15:50:58.251Z · score: 5 (2 votes) · LW · GW
A hellworld is ultimately a world that is against our values. However, our values are underdefined and changeable. So to have any chance of saying what these values are, we need to either extract key invariant values, synthesise our contradictory values into some complete whole, or use some extrapolation procedure (eg CEV). In any case, there is a procedure for establishing our values (or else the very concept of "hellworld" makes no sense).

It feels worth distinguishing between two cases of "hellworld":

1. A world which is not aligned with the values of that world's inhabitants themselves. One could argue that in order to merit the designation "hellworld", the world has to be out of alignment with the values of its inhabitants in such a way as to cause suffering. Assuming that we can come up with a reasonable definition of suffering, then detecting these kinds of worlds seems relatively straightforward: we can check whether they contain immense amounts of suffering.

2. A world whose inhabitants do not suffer, but which we might consider hellish according to our values. For example, something like a Brave New World scenario, where people generally consider themselves happy but where that happiness comes at the cost of suppressing individuality and promoting superficial pleasures.

It's for detecting an instance of the second case that we need to understand our values better. But it's not clear to me that such a world should qualify as a "hellworld", which to me sounds like a world with negative value. While I don't find the notion of being the inhabitant of a Brave New World particularly appealing, a world where most people are happy but only in a superficial way sounds more like "overall low positive value" than "negative value" to me. Assuming that you've internalized its values and norms, existing in a BNW doesn't seem like a fate worse than death, it just sounds like a future that could have gone better.

Of course, there is an argument that even if a BNW would be okay to its inhabitants once we got there, getting there might cause a lot of suffering: for instance, if there were lots of people who were forced against their will to adapt to the system. Since many of us might find the BNW to be a fate worse than death, then conditional on us surviving to live in the BNW, it's a hellworld (at least to us). But again this doesn't seem like it requires a thorough understanding of our values to detect: it just requires detecting the fact that if we survive to live in the BNW, we will experience a lot of suffering due to being in a world which is contrary to our values.

Comment by kaj_sotala on Why is this utilitarian calculus wrong? Or is it? · 2019-01-28T14:24:24.880Z · score: 15 (12 votes) · LW · GW
the extra benefits flow to you, and that is generally not seen as a moral plus

This is correct, but I'm not sure that it should be: there's no intrinsic reason for why your well-being wouldn't be just as important morally as everyone else's. Empirically, people thinking that their own well-being doesn't matter and only other people's well-being does, seems to be a big factor in do-gooders burning out.

Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-01-28T11:23:08.046Z · score: 7 (4 votes) · LW · GW
I really enjoyed this post and starting with the plausible robot design was really helpful for me accessing the IFS model. I also enjoyed reflecting on your previous objections as a structure for the second part.

Thanks, that's very nice and specific feedback. :)

The part with repeated unblending sounds reminiscent of the "Clearing a space" stage of Focusing, in which one acknowledges and sets slightly to the side the problems in one's life.

Yeah, these feel basically like the same kind of thing. I find that Focusing and IFS have basically blended into some hybrid technique for me, with it being hard to tell the difference anymore.

you mention you think that IFS would benefit most people. What do you mean by 'benefit' in this case? That it would increase their wellbeing? Their personal efficacy? Or perhaps that it will increase at least one of their wellbeing and personal efficacy but not necessarily both for any given person?

Possibly combined with other related practices, such as Focusing: Elimination of internal conflicts, increased well-being due to improved access to Self, better ability to do things which feel like worth doing. The personal examples in my other comment may give a better idea.

Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-01-28T11:18:44.634Z · score: 17 (5 votes) · LW · GW

So I find IFS, Focusing, IDC, and some aspects of TMI-style meditation to basically have blended together into one big hybrid technique for me; they all feel like different aspects of what's essentially the same skill of "listening to what your subagents want and bringing their desires into alignment with each other"; IFS has been the thing that gave me the biggest recent boost, but it's not clear to me that I'm always doing "entirely pure IFS", even though I think there's nearly always a substantial IFS component. (Probably most important has been the part about getting into Self, which wasn't a concept I explicitly had before this.)

That said, a few examples. I already mentioned a few in an earlier post:

My experience is that usually if I have an unpleasant emotion, I will try to do one of two things: either reject it entirely and push it out of my mind, or buy into the story that it’s telling and act accordingly. Once I learned the techniques for getting into Self, I got the ability to sort of… just hang out with the emotion, neither believing it to be absolutely true nor needing to show it to be false. And then if I e.g. had feelings of social anxiety, I could keep those feelings around and go into a social situation anyway, making a kind of mental move that I might describe as “yes, it’s possible that these people all secretly hate me; I’m going to accept that as a possibility without trying to add any caveats, but also without doing anything else than accepting its possibility”.
The consequence has been that this seems to make the parts of my mind with beliefs like “doing this perfectly innocuous thing will make other people upset” actually update their beliefs. I do the thing, the parts with this belief get to hang around and observe what happens, notice that nobody seems upset at me, and then they are somewhat less likely to bring up similar concerns in the future.
In terms of global workspace theory, my model here is that there’s a part of the mind that’s bringing up a concern that should be taken into account in decision-making. The concern may or may not be justified, so the correct thing to do is to consider its possibility, but not necessarily give it too much weight. Going into Self and letting the message stay in consciousness this way seems to make it available for decision-making, and often the module that’s bringing it up is happy to just have its message received and evaluated; you don’t have to do anything more than that, if it’s just holding it up as a tentative consideration to be evaluated.

If I had to name one single biggest object-level benefit from IFS, it would be this one: a gradual reduction of my remaining unfounded social anxieties, which is still ongoing but seems to be pretty well on track to eliminating all of them.

This ties into the more meta-level thing that there's less and less of a feeling that negative emotions are something that I need to avoid, or that I would need to fight against my own mind. Now I don't claim to be entirely Zen at all times, and there's still stuff like stress or exhaustion that can make me feel miserable, but at least assuming relatively "normal" conditions... there's increasingly the feeling that if I find myself experiencing procrastination, or feeling bad about something, then that involves some subagents not being in agreement about what to do, and I can just fix that. (Again, this is not to say that this process would cause me to only feel positive emotions at all times: sometimes feeling a negative emotion is the mind-system's endorsed response to a situation. But then when the system as a whole agrees with it, it doesn't feel bad in the same way.)

There are a bunch of examples of minor fixes along the lines of the example from the same post:

E.g. a while back I was having a sense of loneliness as I laid down for a nap. I stepped into the part’s perspective to experience it for a while, then unblended; now I felt it as a black ice hockey puck levitating around my lower back. I didn’t really do anything other than let it be there, and maintained a connection with it. Gradually it started generating a pleasant warmth, and then the visualization transformed into a happy napping cartoon fox, curled up inside a fireball that it was using as its blanket. And then I was no longer feeling lonely.

This has gotten to the slightly annoying point that I often find myself "no longer being able" to say things like "I have a mental block/emotional aversion against doing X" or "I feel bad because Y", because if I have a good enough handle on the situation to be able to describe it in such detail, then I can often just fix it right away, without needing to talk about it to someone else. Recent fixes in this category include:

  • Recognizing that I should get more exercise and getting a subscription to the nearby gym, after living within a five minute walk of it for almost a year and never getting around visiting it before.
  • Managing to actually write my previous post in this sequence, which felt like a relatively boring thing to do since I was just summarizing someone else's work; several blocks came up in the process of doing that, which I then dealt with one at a time, until I could just finish it relatively painlessly.
  • Emotional issues relating to things like being too sensitive to the pain of others, to the point of being frequently upset about various specific things in the world which are horrible, and having difficulties setting my own boundaries if it felt like I could sacrifice some of my well-being in order to make someone else better off.

Some exceptions to the "I can just fix it when I'm feeling bad" thing include:

  • if the issue is actually caused by someone else, e.g. someone else is acting in a way which is preventing me from achieving my needs
  • the problem is caused by a physical issue that I have, such as being hungry, low on sleep, or having such a low level of physical arousal that I get stuck on low-activation energy behaviors
  • there's something else in the external environment that causes an actual concrete problem that I don't have e.g. the skills to deal with myself, so can't just institute an internal fix

Also, I used to think that I'd lost out because when I had the chance to experience some things, I failed to realize that chance and didn't get them and now it's too late. For instance, a chance to focus on my studies free of stress, or experiencing a happy and lasting relationship when young and growing up together with a close partner.

But after doing some IFS and TMI work around those things, I've sometimes been spontaneously experiencing the same kinds of Self-like emotional sensations ("felt senses", to use the Focusing term) that I previously thought that I would only have had if I'd gotten those things.

So I suspect that my "I had the chance to experience X, but lost it because of life circumstance Y" better translates to "I previously had access to a certain aspect of being in Self, which frequently happened in the context of X, but had that access blocked after Y". Examples:

1) A chance to focus on my studies free of stress. When I graduated high school, I was really into learning and studying, and excited about the possibility of spending several years in university doing just that. And for a while it was like that and I really enjoyed it. But then I got a burnout and the rest of it was just desperately trying to catch up on my studies and there was a lot of stress, and I have never again had that opportunity to just focus on nothing but studying and being free to think about nothing else.

Except about a month ago I started reading a textbook, with that study time being squarely sandwiched between a dozen other things I should be doing, and... that felt sense of being able to just focus on studies and nothing else, was there again. Apparently it didn't require the freedom to spend a years at a time just studying, just being able to time-box a few hours from a day was enough. But of course, I hadn't previously re-gotten that feeling from just a few hours. Now it felt more like just enjoying learning, in a way which I hadn't remembered for a long time.

So apparently there was something like, previously being able to just focus on the pleasure of learning had been one way to get myself into Self, but afterwards there had been a priority override which had been left active and blocked that access. After I did things to address that override, I could get into Self that way again, and it turned out that feeling this way wasn't a unique opportunity specific to one part of my life which I had now forever lost.

2) The relationship thing is harder to explain, but there's something analogous to the study thing in that... I recalled experiencing a feeling of openness and optimism towards another person, specifically in the context of my first crushes and with my first girlfriend, which I had never quite experienced the same way afterwards. And the way I interpreted that was something like, that was the experience you get when you consider potential or actual partners with the unique openness of being young, when I was still quite naive about things but also not particularly cynical or jaded.

And there was an implicit notion of... I didn't dissect this so explicitly until recently, but I think that a part of me was making the assumption that if I'd ended up in a lasting relationship with someone back then, then that relationship would somehow have preserved that felt sense of openness, which I didn't experience as surviving into my later relationships. Of course, I didn't explicitly think that it would have preserved that felt sense. Rather it was more that the memory of that felt sense was associated with my memory of how I experienced romance back then, and the combination of those memories was associated with a sense of loss of what could have been.

Until about a month ago, when that felt sense of openness and optimism towards a person suddenly popped when I was talking with 1) my housemate about random stuff for 15 minutes and 2) an old acquaintance in the bus for 5 minutes. And also lingering generally around in a milder form when I wasn't even in anyone's company, just doing stuff by myself.

So I think that, my mind had recalled that there was a really nice felt sense associated with my teenage crushes, and made the assumption that if I'd had managed to get into a lasting relationship back then, that would have preserved the felt sense in question. But actually 1) the relationship itself wasn't the point, the nice felt sense was 2) the felt sense wasn't solely about romantic relationships in the first place, it was about having a particular quality of Self which had since then gotten blocked due to some ongoing override.

(I still haven't permanently addressed this override; it seems like it came back since then, and those specific sensations of Self have again been missing. But I expect to eventually be able to figure out how to integrate the specific managers and exiles which are behind those sensations being blocked.)

A somewhat different framing of this would be in terms of emotional unclogging. Something like: as a teenager there were some aspects of me that were less clogged, though I still needed the context of a romantic relationship to unclog them enough to access those aspects. Afterwards access to those aspects of me got more clogged, so that I couldn't access them even in the context of a relationship anymore, so I thought that I'd lost my chance of ever experiencing those feelings again. And then I did some more unclogging work with IFS and related techniques, and suddenly I started having access to those feelings even when talking with somewhat random people.

Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-01-28T09:53:28.734Z · score: 4 (2 votes) · LW · GW

I looked at the beginning of it a bit before writing this post, but at least the beginning of it gave the impression that its subagents were very low-level (IIRC, it started with an example of building a tower of blocks, or taking some similar physical action, using many different subagents) and overall it had a strong vibe of 80's AI, so then it didn't feel like the most useful thing to be reading.

Building up to an Internal Family Systems model

2019-01-26T12:25:11.162Z · score: 138 (43 votes)
Comment by kaj_sotala on "AlphaStar: Mastering the Real-Time Strategy Game StarCraft II", DeepMind [won 10 of 11 games against human pros] · 2019-01-26T09:37:16.364Z · score: 11 (9 votes) · LW · GW

Is there a reason to assume that DeepMind is being intentionally deceptive rather than making a good-intentioned mistake? After all, they claim that they consulted with TLO on the APM rate, which is something that they seem unlikely to lie about, since it would be easy for TLO to dispute that if it was untrue. So presumably TLO felt like the caps that they instituted were fair, and it would have been reasonable for DeepMind to trust a top player's judgment on that, none of them (AFAIK) being top players themselves. And even with the existing caps, people felt like the version which played TLO would likely have lost to MaNa, and the camera-limited version did actually lose to him. So it seems reasonable for someone to think that a good enough player would still beat versions of AlphaStar even with the existing limits, and that they didn't thus give it an unfair advantage.

On the other hand, while I can think of plenty of reasons for why DeepMind might have honestly thought that this setup was fair, I can't think of any good reason for them to decide to be intentionally deceptive in this way. They've been open about their agent currently only playing Protoss vs. Protoss on a single map and of earlier versions seeing the whole map, and five of their games were played against a non-Protoss player. If they honestly felt that they only won because of the higher APM rate, then I don't see why they wouldn't just admit that the same way they're forthcoming about all of the system's other limitations.

Comment by kaj_sotala on Book Summary: Consciousness and the Brain · 2019-01-23T14:12:31.505Z · score: 5 (2 votes) · LW · GW
When you look at a chair, this pattern recognition module immediately classifies it, and then brings online another module, which makes available all the relevant physical affordances, linguistic and logical implications of a chair being present in your environment. Recognizing something as a chair feels identical to recognizing something as a thing-in-which-I-can-sit. Similarly, you don't have to puzzle out the implications of a tiger walking into the room right now. The fear response will coincide with the recognition of the tiger.

Yeah, this is similar to how I think of it. When I see something, the thoughts which are relevant for the context become available: usually naming the thing isn't particularly necessary, so I don't happen to consciously think of its name.

Doesn't this imply that some submodules of your brain are thinking abstractly and logically about The Matrix completely outside of your conscious awareness? If so, then this either implies that the subconscious processing of individual submodules can be very complex and abstract without needing to share information with other submodules, or that information sharing between submodules can occur without you being consciously aware of it.

Well, we already know from the unconscious priming experiments that information-sharing between submodules can occur without conscious awareness. It could be something like, if you hadn't been conscious of watching The Matrix, the submodules would never have gotten a strong enough signal about its contents to process it; but once the movie was once consciously processed, there's enough of a common reference for several related submodules to "know what the other is talking about".

Or maybe it's all in one submodule; the fact that that submodule feels a need to make its final conclusion conscious, suggests that it can't communicate the entirety of its thinking purely unconsciously.

Comment by kaj_sotala on Life can be better than you think · 2019-01-23T13:55:56.851Z · score: 11 (3 votes) · LW · GW
But the writing is from the perspective of the insight, which tends to be dissimilar from the material that *triggered* the insight.

Related: the Burrito Tutorial Fallacy.

While working on an article for the Monad.Reader, I’ve had the opportunity to think about how people learn and gain intuition for abstraction, and the implications for pedagogy. The heart of the matter is that people begin with the concrete, and move to the abstract. Humans are very good at pattern recognition, so this is a natural progression. By examining concrete objects in detail, one begins to notice similarities and patterns, until one comes to understand on a more abstract, intuitive level. This is why it’s such good pedagogical practice to demonstrate examples of concepts you are trying to teach. It’s particularly important to note that this process doesn’t change even when one is presented with the abstraction up front! For example, when presented with a mathematical definition for the first time, most people (me included) don’t “get it” immediately: it is only after examining some specific instances of the definition, and working through the implications of the definition in detail, that one begins to appreciate the definition and gain an understanding of what it “really says.”
Unfortunately, there is a whole cottage industry of monad tutorials that get this wrong. To see what I mean, imagine the following scenario: Joe Haskeller is trying to learn about monads. After struggling to understand them for a week, looking at examples, writing code, reading things other people have written, he finally has an “aha!” moment: everything is suddenly clear, and Joe Understands Monads! What has really happened, of course, is that Joe’s brain has fit all the details together into a higher-level abstraction, a metaphor which Joe can use to get an intuitive grasp of monads; let us suppose that Joe’s metaphor is that Monads are Like Burritos. Here is where Joe badly misinterprets his own thought process: “Of course!” Joe thinks. “It’s all so simple now. The key to understanding monads is that they are Like Burritos. If only I had thought of this before!” The problem, of course, is that if Joe HAD thought of this before, it wouldn’t have helped: the week of struggling through details was a necessary and integral part of forming Joe’s Burrito intuition, not a sad consequence of his failure to hit upon the idea sooner.
But now Joe goes and writes a monad tutorial called “Monads are Burritos,” under the well-intentioned but mistaken assumption that if other people read his magical insight, learning about monads will be a snap for them. “Monads are easy,” Joe writes. “Think of them as burritos.” Joe hides all the actual details about types and such because those are scary, and people will learn better if they can avoid all that difficult and confusing stuff. Of course, exactly the opposite is true, and all Joe has done is make it harder for people to learn about monads, because now they have to spend a week thinking that monads are burritos and getting utterly confused, and then a week trying to forget about the burrito analogy, before they can actually get down to the business of learning about monads. (Of course, certainly not all monad tutorials are like this, and I don’t even have any particular ones in mind, just a general impression left over from reading many of them, but if the shoe fits…)
What I term the “monad tutorial fallacy,” then, consists in failing to recognize the critical role that struggling through fundamental details plays in the building of intuition.

Book Summary: Consciousness and the Brain

2019-01-16T14:43:59.202Z · score: 86 (26 votes)

Sequence introduction: non-agent and multiagent models of mind

2019-01-07T14:12:30.297Z · score: 79 (29 votes)

18-month follow-up on my self-concept work

2018-12-18T17:40:03.941Z · score: 54 (14 votes)

Tentatively considering emotional stories (IFS and “getting into Self”)

2018-11-30T07:40:02.710Z · score: 38 (10 votes)

Incorrect hypotheses point to correct observations

2018-11-20T21:10:02.867Z · score: 73 (29 votes)

Mark Eichenlaub: How to develop scientific intuition

2018-10-23T13:30:03.252Z · score: 68 (28 votes)

On insecurity as a friend

2018-10-09T18:30:03.782Z · score: 36 (18 votes)

Tradition is Smarter Than You Are

2018-09-19T17:54:32.519Z · score: 68 (24 votes)

nostalgebraist - bayes: a kinda-sorta masterpost

2018-09-04T11:08:44.170Z · score: 16 (7 votes)

New paper: Long-Term Trajectories of Human Civilization

2018-08-12T09:10:01.962Z · score: 27 (13 votes)

Finland Museum Tour 1/??: Tampere Art Museum

2018-08-03T15:00:05.749Z · score: 20 (6 votes)

What are your plans for the evening of the apocalypse?

2018-08-02T08:30:05.174Z · score: 24 (11 votes)

Anti-tribalism and positive mental health as high-value cause areas

2018-08-02T08:30:04.961Z · score: 26 (10 votes)

Fixing science via a basic income

2018-08-02T08:30:04.380Z · score: 30 (14 votes)

Study on what makes people approve or condemn mind upload technology; references LW

2018-07-10T17:14:51.753Z · score: 21 (11 votes)

Shaping economic incentives for collaborative AGI

2018-06-29T16:26:32.213Z · score: 47 (13 votes)

Against accusing people of motte and bailey

2018-06-03T21:31:24.591Z · score: 83 (27 votes)

AGI Safety Literature Review (Everitt, Lea & Hutter 2018)

2018-05-04T08:56:26.719Z · score: 37 (10 votes)

Kaj's shortform feed

2018-03-31T13:02:47.793Z · score: 12 (2 votes)

Helsinki SSC March meetup

2018-03-26T19:27:17.850Z · score: 12 (2 votes)

Is the Star Trek Federation really incapable of building AI?

2018-03-18T10:30:03.320Z · score: 29 (8 votes)

My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms

2018-03-08T07:37:54.532Z · score: 271 (97 votes)

Some conceptual highlights from “Disjunctive Scenarios of Catastrophic AI Risk”

2018-02-12T12:30:04.401Z · score: 63 (18 votes)

On not getting swept away by mental content

2018-01-25T20:30:03.750Z · score: 23 (7 votes)

Papers for 2017

2018-01-04T13:30:01.406Z · score: 32 (8 votes)

Paper: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering

2018-01-03T14:39:18.024Z · score: 1 (1 votes)

Paper: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering

2018-01-03T13:57:55.979Z · score: 16 (6 votes)

Fixing science via a basic income

2017-12-08T14:20:04.623Z · score: 38 (11 votes)

Book review: The Upside of Your Dark Side: Why Being Your Whole Self–Not Just Your “Good” Self–Drives Success and Fulfillment

2017-12-04T13:10:06.995Z · score: 27 (8 votes)

Meditation and mental space

2017-11-06T13:10:03.612Z · score: 26 (7 votes)

siderea: What New Atheism says

2017-10-29T10:19:57.863Z · score: 12 (3 votes)

Postmodernism for rationalists

2017-10-17T12:20:36.139Z · score: 24 (1 votes)

Anti-tribalism and positive mental health as high-value cause areas

2017-10-17T10:20:03.359Z · score: 30 (10 votes)

You can never be universally inclusive

2017-10-14T11:30:04.250Z · score: 34 (10 votes)

Meaningfulness and the scope of experience

2017-10-05T11:30:03.863Z · score: 36 (13 votes)

Social Choice Ethics in Artificial Intelligence (paper challenging CEV-like approaches to choosing an AI's values)

2017-10-03T17:39:00.683Z · score: 8 (3 votes)

LW2.0 now in public beta (you'll need to reset your password to log in)

2017-09-23T12:00:50.345Z · score: 2 (2 votes)

Nobody does the thing that they are supposedly doing

2017-09-23T10:40:06.155Z · score: 54 (29 votes)

Debiasing by rationalizing your own motives

2017-09-03T12:20:10.405Z · score: 1 (1 votes)

Multiverse-wide Cooperation via Correlated Decision Making

2017-08-20T12:01:36.212Z · score: 4 (4 votes)

How I [Kaj] found & fixed the root problem behind my depression and anxiety after 20+ years

2017-07-26T12:56:47.127Z · score: 16 (16 votes)

Tenenbaum et al. (2017) on the computational mechanisms of learning a commonsense moral theory

2017-07-25T13:36:34.648Z · score: 1 (1 votes)

The Internet as an existential threat

2017-07-09T11:40:50.195Z · score: 4 (4 votes)

S-risks: Why they are the worst existential risks, and how to prevent them

2017-06-20T12:34:32.608Z · score: 20 (20 votes)

Cognitive Science/Psychology As a Neglected Approach to AI Safety

2017-06-05T13:55:40.261Z · score: 5 (5 votes)

Cognitive Core Systems explaining intuitions behind belief in souls, free will, and creation myths

2017-05-06T12:13:13.761Z · score: 8 (7 votes)

Brief update on the consequences of my "Two arguments for not thinking about ethics" (2014) article

2017-04-05T11:25:39.618Z · score: 14 (15 votes)

Moral Philosophers as Ethical Engineers: Limits of Moral Philosophy and a Pragmatist Alternative

2017-02-23T13:02:47.305Z · score: 5 (4 votes)