Posts

Subagents, trauma and rationality 2019-08-14T13:14:46.838Z · score: 54 (29 votes)
Subagents, neural Turing machines, thought selection, and blindspots 2019-08-06T21:15:24.400Z · score: 56 (17 votes)
On pointless waiting 2019-06-10T08:58:56.018Z · score: 43 (22 votes)
Integrating disagreeing subagents 2019-05-14T14:06:55.632Z · score: 84 (21 votes)
Subagents, akrasia, and coherence in humans 2019-03-25T14:24:18.095Z · score: 87 (25 votes)
Subagents, introspective awareness, and blending 2019-03-02T12:53:47.282Z · score: 60 (20 votes)
Building up to an Internal Family Systems model 2019-01-26T12:25:11.162Z · score: 152 (54 votes)
Book Summary: Consciousness and the Brain 2019-01-16T14:43:59.202Z · score: 93 (30 votes)
Sequence introduction: non-agent and multiagent models of mind 2019-01-07T14:12:30.297Z · score: 86 (31 votes)
18-month follow-up on my self-concept work 2018-12-18T17:40:03.941Z · score: 55 (15 votes)
Tentatively considering emotional stories (IFS and “getting into Self”) 2018-11-30T07:40:02.710Z · score: 39 (11 votes)
Incorrect hypotheses point to correct observations 2018-11-20T21:10:02.867Z · score: 73 (29 votes)
Mark Eichenlaub: How to develop scientific intuition 2018-10-23T13:30:03.252Z · score: 68 (28 votes)
On insecurity as a friend 2018-10-09T18:30:03.782Z · score: 36 (18 votes)
Tradition is Smarter Than You Are 2018-09-19T17:54:32.519Z · score: 68 (24 votes)
nostalgebraist - bayes: a kinda-sorta masterpost 2018-09-04T11:08:44.170Z · score: 16 (7 votes)
New paper: Long-Term Trajectories of Human Civilization 2018-08-12T09:10:01.962Z · score: 27 (13 votes)
Finland Museum Tour 1/??: Tampere Art Museum 2018-08-03T15:00:05.749Z · score: 20 (6 votes)
What are your plans for the evening of the apocalypse? 2018-08-02T08:30:05.174Z · score: 24 (11 votes)
Anti-tribalism and positive mental health as high-value cause areas 2018-08-02T08:30:04.961Z · score: 26 (10 votes)
Fixing science via a basic income 2018-08-02T08:30:04.380Z · score: 30 (14 votes)
Study on what makes people approve or condemn mind upload technology; references LW 2018-07-10T17:14:51.753Z · score: 21 (11 votes)
Shaping economic incentives for collaborative AGI 2018-06-29T16:26:32.213Z · score: 47 (13 votes)
Against accusing people of motte and bailey 2018-06-03T21:31:24.591Z · score: 83 (27 votes)
AGI Safety Literature Review (Everitt, Lea & Hutter 2018) 2018-05-04T08:56:26.719Z · score: 37 (10 votes)
Kaj's shortform feed 2018-03-31T13:02:47.793Z · score: 12 (2 votes)
Helsinki SSC March meetup 2018-03-26T19:27:17.850Z · score: 12 (2 votes)
Is the Star Trek Federation really incapable of building AI? 2018-03-18T10:30:03.320Z · score: 29 (8 votes)
My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms 2018-03-08T07:37:54.532Z · score: 279 (103 votes)
Some conceptual highlights from “Disjunctive Scenarios of Catastrophic AI Risk” 2018-02-12T12:30:04.401Z · score: 63 (18 votes)
On not getting swept away by mental content 2018-01-25T20:30:03.750Z · score: 23 (7 votes)
Papers for 2017 2018-01-04T13:30:01.406Z · score: 32 (8 votes)
Paper: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering 2018-01-03T14:39:18.024Z · score: 1 (1 votes)
Paper: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering 2018-01-03T13:57:55.979Z · score: 16 (6 votes)
Fixing science via a basic income 2017-12-08T14:20:04.623Z · score: 38 (11 votes)
Book review: The Upside of Your Dark Side: Why Being Your Whole Self–Not Just Your “Good” Self–Drives Success and Fulfillment 2017-12-04T13:10:06.995Z · score: 27 (8 votes)
Meditation and mental space 2017-11-06T13:10:03.612Z · score: 26 (7 votes)
siderea: What New Atheism says 2017-10-29T10:19:57.863Z · score: 12 (3 votes)
Postmodernism for rationalists 2017-10-17T12:20:36.139Z · score: 24 (1 votes)
Anti-tribalism and positive mental health as high-value cause areas 2017-10-17T10:20:03.359Z · score: 30 (10 votes)
You can never be universally inclusive 2017-10-14T11:30:04.250Z · score: 34 (10 votes)
Meaningfulness and the scope of experience 2017-10-05T11:30:03.863Z · score: 35 (14 votes)
Social Choice Ethics in Artificial Intelligence (paper challenging CEV-like approaches to choosing an AI's values) 2017-10-03T17:39:00.683Z · score: 8 (3 votes)
LW2.0 now in public beta (you'll need to reset your password to log in) 2017-09-23T12:00:50.345Z · score: 2 (2 votes)
Nobody does the thing that they are supposedly doing 2017-09-23T10:40:06.155Z · score: 68 (33 votes)
Debiasing by rationalizing your own motives 2017-09-03T12:20:10.405Z · score: 1 (1 votes)
Multiverse-wide Cooperation via Correlated Decision Making 2017-08-20T12:01:36.212Z · score: 4 (4 votes)
How I [Kaj] found & fixed the root problem behind my depression and anxiety after 20+ years 2017-07-26T12:56:47.127Z · score: 16 (16 votes)
Tenenbaum et al. (2017) on the computational mechanisms of learning a commonsense moral theory 2017-07-25T13:36:34.648Z · score: 1 (1 votes)
The Internet as an existential threat 2017-07-09T11:40:50.195Z · score: 4 (4 votes)

Comments

Comment by kaj_sotala on Ruby's Public Drafts & Working Notes · 2019-09-14T20:51:57.522Z · score: 5 (2 votes) · LW · GW

Unlocking the Emotional Brain is basically about this.

Comment by kaj_sotala on Kaj's shortform feed · 2019-09-07T11:25:12.298Z · score: 5 (2 votes) · LW · GW

Janina Fisher's book "Healing the Fragmented Selves of Trauma Survivors" has an interesting take on Internal Family Systems. She conceptualizes trauma-related parts (subagents) as being primarily associated with the defensive systems of Fight/Flight/Freeze/Submit/Attach.

Here's how she briefly characterizes the various systems and related behaviors:

  • Fight: Vigilance. Angry, judgmental, mistrustful, self-destructive, controlling, suicidal, needs to control.
  • Flight: Escape. Distancer, ambivalent, cannot commit, addictive behavior or being disorganized.
  • Freeze: Fear. Frozen, terrified, wary, phobic of being seen, agoraphobic, reports panic attacks.
  • Submit: Shame. Depressed, ashamed, filled with self-hatred, passive, "good girl," caretaker, self-sacrificing.
  • Attach: Needy. Desperate, craves rescue & connection, sweet, innocent, wants someone to depend on.

Here's how she describes a child-like part connected to an "attach" system coming to existence:

... research has demonstrated the propensity of the brain to develop neural networks holding related neural pathways that consistently “fire” together, and these neural systems often encode complex systems of traits or systems (Schore, 2001) that represent aspects of our personalities or ways of being. For example, if neural pathways activating the proximity drive fire consistently in the presence of the attachment figure, along with neural pathway holding feelings of loneliness and yearning for comfort and a neural network holding the tendency to believe that “she loves me—she would never hurt me,” the result might be a neural system representing a young child part of the personality with a toddler’s yearning for comfort and closeness along with the magical thinking that the attachment figure will be safe and loving, yet also the uneasy feeling that something is not right. Such neural systems can be complex with a subjective sense of identity or can be a simpler collection of traits associated with different roles played by the individual.

Here are how she relates various trauma symptoms to these systems:

The paradoxical quality of these symptoms is rarely captured by traditional diagnostic models. Clients report symptoms of major depression (the submit part), anxiety disorders (freeze), substance abuse and eating disorders (flight), anger management or self-harm issues (fight), and they alternately cling to others or push them away (the characteristic symptoms of disorganized or traumatic attachment).

And here's how she describes something that in traditional IFS terms would be described as polarized parts:

Aaron described the reasons for which he had come: “I start out by getting attached to women very quickly—I immediately think they’re the ‘one.’ I’m all over them, can’t see them enough … until they start to get serious or there’s a commitment. Then I suddenly start to see everything I didn’t see before, everything that’s wrong with them. I start feeling trapped with someone who’s not right for me—I want to leave, but I feel guilty—or afraid they’ll leave me. I’m stuck. I can’t relax and be happy, but I can’t get out of it either.”

Aaron was describing an internal struggle between parts: between an attachment-seeking part that quickly connected to any attractive woman who treated him warmly and a hypervigilant, hypercritical fight part that reacted to every less-than-optimal quality she possessed as a sign of trouble. His flight part, triggered by the alarms of the fight part, then would start to feel trapped with what felt like the “wrong person,” generating impulses to get out—an action that his submit and cry for help parts couldn’t allow. Guilt and shame for the commitment he’d promised (the submit part’s contribution) and fear of loss (the input from his traumatically attached part) kept him in relationships that his fight and flight parts resisted with equal intensity. Without a language to differentiate each part and bring it to his awareness, he ruminated constantly: should he leave? Or should he stay? Was she enough? Or should he get out now? Often, suicide seemed to him the most logical solution to this painful dilemma, yet at the same time “he” dreamed of having a family with children and a loving and lovely wife. “He” didn’t approve of his wandering eye, yet “he” couldn’t stop trolling for prospective partners. Who was “he”? The suicidal part’s threat to end it all was in direct conflict with his wish for a wife and family; the “trolling for women” part was at odds with the person he wanted to be and believed he should and could be.

Comment by kaj_sotala on Kaj's shortform feed · 2019-08-28T14:41:03.500Z · score: 10 (5 votes) · LW · GW

I was thinking of a friend and recalled some pleasant memories with them, and it occurred to me that I have quite a few good memories about them, but I don't really recall them very systematically. I just sometimes remember them at random. So I thought, what if I wrote down all the pleasant memories of my friend that I could recall?

Not only could I then occasionally re-read that list to get a nice set of pleasant memories, that would also reinforce associations between them, making it more likely that recalling one - or just being reminded of my friend in general - would also bring to mind all the others.

(This was in part inspired by Steve Andreas's notion of building a self-concept. There you build self-esteem by taking memories of yourself where you exhibited some positive quality, and intentionally associate them together under some heading such as "lovable" or "intelligent", so that they become interconnected exemplars of a quality that you have rather than being isolated instances.)

So I did, and that usual thing happened where I started out with just three or so particularly salient memories, but then in the process of writing them down my mind generated a few more, until I had quite a long list. It felt really good; now I want to write similar lists about all my close friends.

Interestingly I noticed that the majority of the memories on my list were ones where I'd helped my friend and they'd been happy as a result, rather than the other way around. This does say something about me finding it easier to help people than to ask for help, but might also be related to the finding that I've heard quoted, that giving a gift makes people happier than receiving one.

Comment by kaj_sotala on Actually updating · 2019-08-25T15:08:41.798Z · score: 8 (4 votes) · LW · GW

A related example is my "On insecurity as a friend": I'd on some level bought into messages saying that confidence is always good and insecurity is always bad, so when I had feelings of insecurity, I treated them as a sign of irrationality to override.

What I didn't realize was that I had those feelings because of having screwed up socially in the past, and that they were now correctly warning me about things which might again have bad consequences. Just trying to walk over them meant that I was ignoring important warnings, sometimes causing things to blow up in my face. What made the feelings easier to deal with was when I started actually taking them seriously as hypotheses to consider. After that, the relevant part could tone down the intensity of the social anxieties, as shouting so loudly as to force me to withdraw socially wasn't the only way that it could make itself heard.

Comment by kaj_sotala on What are the reasons to *not* consider reducing AI-Xrisk the highest priority cause? · 2019-08-23T10:26:48.593Z · score: 3 (2 votes) · LW · GW

That's certainly true. To be clear, my argument was not "these types of work are entirely overlapping", but rather just that "taking s-risk seriously doesn't necessarily mean no overlap with x-risk prevention".

Comment by kaj_sotala on Towards an Intentional Research Agenda · 2019-08-23T10:19:03.860Z · score: 12 (5 votes) · LW · GW
In Buddhism, intention is considered synonymous with consciousness, while in the west this is considered a contentious claim.

Since some readers are probably unaware of this, worth noting explicitly that the sense of "intention" used in philosophy is different from the common meaning of the term. From the linked article:

Where the term “intentionality” is concerned, we also face confusing and contentious usage. But here the problem lies partly in the fact that the relevant use is definitely not that found in common speech employing cognate terms (as when we speak of doing something intentionally). [...]

One way philosophers have often explained what they mean by “intentionality” is this: it is that aspect of mental states or events that consists in their being of or about things, as pertains to the questions, “What are you thinking of?” and “What are you thinking about?” Intentionality is the aboutness or directedness or reference of mind (or states of mind) to things, objects, states of affairs, events. So if you are thinking about San Francisco, or about the cost of living there, or about your meeting someone at Union Square—your mind, your thinking, is directed toward San Francisco, or the cost of living, or the meeting in Union Square.
Comment by kaj_sotala on What are the reasons to *not* consider reducing AI-Xrisk the highest priority cause? · 2019-08-22T12:48:08.769Z · score: 3 (1 votes) · LW · GW

That seems like a reason to work on AI alignment and figure out ways to avoid that particular failure mode, e.g. hyperexistential separation.

Comment by kaj_sotala on G Gordon Worley III's Shortform · 2019-08-21T18:38:08.438Z · score: 21 (8 votes) · LW · GW
So when we talk about the dharma or justify our actions on it, it's worth noting that it is not really trying to provide consistent episteme. [...] Thus it's a strange inversion to ask the dharma for episteme-based proofs. It can't give them, nor does it try, because its episteme is not consistent and cannot be because it chooses completeness instead.

In my view, this seems like a clear failing. The fact that the dharma comes from a tradition where this has usually been the case is not an excuse for not trying to fix it.

Yes, the method requires temporarily suspending episteme-based reasoning and engaging with less conceptual forms of seeing. But it can still be justified and explained using episteme-based models; if it could not, there would be little reason to expect that it would be worth engaging with.

This is not just a question of "the dharma has to be able to justify itself"; it's also a question of leaving out the episteme component leaves the system impoverished, as noted e.g. here:

Recurrent training to attend to the sensate experience moment-by-moment can undermine the capacity to make meaning of experience. (The psychoanalyst Wilfred Bion described this as an ‘attack on linking’, that is, on the meaning-making function of the mind.) When I ask these patients how they are feeling, or what they are thinking, or what’s on their mind, they tend to answer in terms of their sensate experience, which makes it difficult for them to engage in a transformative process of psychological self-understanding.

and here:

In important ways, it is not possible to encounter our unconscious – at least in the sense implied by this perspective – through moment-to-moment awareness of our sensate experience. Yes, in meditation we can have the experience of our thoughts bubbling just beneath the surface – what Shinzen Young calls the brain’s pre-processing – but this is not the unconscious that I’m referring to, it, or at least not all of it.
Let me give an example. Suppose that I have just learned that a close friend has died. I’m deeply saddened by this news. Moments later, I spill a cup of coffee on my new pants and become quite angry. Let’s further suppose that, throughout my life, I’ve had difficulty feeling sadness. For reasons related to my personal history, sadness frightens me. In my moment of anger, if I adopt the perspective of awareness of sensate experience, moment-by-moment, then I will have no access to the fact that I am sad. On the contrary, my sensate experience seems to reflect the fact that I am angry. But given what I know about myself, it’s quite reasonable to posit that my anger is a defense against the feeling of sadness, a feeling of which I am unconscious as I am caught up in my anger.
Comment by kaj_sotala on Davis_Kingsley's Shortform · 2019-08-21T17:58:56.337Z · score: 5 (2 votes) · LW · GW

I don't know much about what's going on in the Bay Area etc. communities, as I don't live there. But I know plenty of non-rationalist poly people. AFAICT, their numbers seem to have been gradually increasing as some people try it out, find that it works for them, which encourages a few more people to try, etc. (Of course, for some people it isn't appealing, and they don't try.) Overall my impression is that the thing described by Scott in Polyamory is Boring keeps gradually happening to various people:

It just started seeming normal. [...] It just seemed like once the entire culture was no longer uniting to tell me polyamory was something bizarre and different and special, it wasn’t. And then it started to look like a slightly better idea to take part in it than to not take part in it. So I did.

Assuming that both of our experiences are correct, maybe there's something about rationalists in particular that makes poly work worse for them than for non-rationalists? That would seem strange, though. Or maybe you're just seeing a community that has been particularly unlucky, or vice versa.

But in particular, this bit seems like the opposite to what I've been observing:

However, most people are not actually looking back at the evidence IMO -- it's just become an installed and unexamined norm.

The story that I've often heard is more along the lines of "I was in monogamous relationships because that was the prevailing norm, had bad experiences with them, did some soul-searching, decided to try poly instead, and have now been happier as a result".


Comment by kaj_sotala on What are the reasons to *not* consider reducing AI-Xrisk the highest priority cause? · 2019-08-21T16:16:09.807Z · score: 7 (5 votes) · LW · GW

S-risks

Not necessarily a reason to deprioritize AI x-risk work, given that unaligned AI could be bad from an s-risk perspective as well:

Pain seems to have evolved because it has a functional purpose in guiding behavior: evolution having found it suggests that pain might be the simplest solution for achieving its purpose. A superintelligence which was building subagents, such as worker robots or disembodied cognitive agents, might then also construct them in such a way that they were capable of feeling pain - and thus possibly suffering (Metzinger 2015) - if that was the most efficient way of making them behave in a way that achieved the superintelligence’s goals.

Humans have also evolved to experience empathy towards each other, but the evolutionary reasons which cause humans to have empathy (Singer 1981) may not be relevant for a superintelligent singleton which had no game-theoretical reason to empathize with others. In such a case, a superintelligence which had no disincentive to create suffering but did have an incentive to create whatever furthered its goals, could create vast populations of agents which sometimes suffered while carrying out the superintelligence’s goals. Because of the ruling superintelligence’s indifference towards suffering, the amount of suffering experienced by this population could be vastly higher than it would be in e.g. an advanced human civilization, where humans had an interest in helping out their fellow humans. [...]

If attempts to align the superintelligence with human values failed, it might not put any intrinsic value on avoiding suffering, so it may create large numbers of suffering subroutines.

Comment by kaj_sotala on Eli's shortform feed · 2019-08-20T06:37:34.898Z · score: 5 (2 votes) · LW · GW

Related: The red paperclip theory of status describes status as a form of optimization power, specifically one that can be used to influence a group.

The name of the game is to convert the temporary power gained from (say) a dominance behaviour into something further, bringing you closer to something you desire: reproduction, money, a particular social position...

Comment by kaj_sotala on Subagents, akrasia, and coherence in humans · 2019-08-14T15:03:41.891Z · score: 5 (2 votes) · LW · GW

I haven't gotten around watching that particular documentary, but I now briefly discuss DID (as well as quoting what you said about subagents and trauma elsewhere) in subagents, trauma, and rationality.

Comment by kaj_sotala on Machine Learning Analogy for Meditation (illustrated) · 2019-08-13T16:35:07.739Z · score: 3 (1 votes) · LW · GW

(Agreed, didn't mean to imply otherwise.)

Comment by kaj_sotala on Machine Learning Analogy for Meditation (illustrated) · 2019-08-13T15:39:09.368Z · score: 7 (3 votes) · LW · GW

I don't know whether this is the true cause, but re-reading your original comment, the word "just" in this sentence gives me a very slight sense of triggeredness:

They're just like... incidental post-hoc things.

I have a feeling which has the rough shape of something like... "people here [me included] are likely to value thoughts a lot and think of them as important in shaping behavior, and may be put on the defensive by wording which seems to be dismissive towards the importance of thoughts".

Your second comment ("Basically, as far as I know, System 1 is more or less directly responsible for all actions") feels to me like it might trigger a bit of the same, as it can be read to imply something like... "all of the stuff in the Sequences about figuring out biases and correcting for them on a System 2 level is basically useless, since System 1 drives people's actions".

I also feel that neither of these explanations is exactly right, and it's actually something more subtle than that. Maybe something like "thoughts being the cause of actions is related to a central strategy of many people around here".

It's also weird that, like Raemon says, the feeling-of-the-conversation-being-off is subtle; it doesn't feel like anybody is being explicitly aggressive and one could in principle interpret the conversation as everyone just sharing their models and confusion. Yet it feels like there is that argumentative vibe.

Comment by kaj_sotala on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T10:49:58.510Z · score: 5 (2 votes) · LW · GW

Moreover, your time and energy are presumably exchangeable for money (though not indefinitely so)?

I guess to some extent, but it feels like the world is already full of things that I could be getting money for if I had the time and energy to pursue them on top of other things that I am already doing. I feel like monetary rewards for answering messages that I wouldn't have answered otherwise would have to be relatively significant to have an impact, like upwards from 20€ or something. But then in that case I would start feeling concerned that people who didn't want to spend that much money messaging me would feel discouraged from doing so.

Also, if I try to simulate in my head the experience of getting a paid email, it feels like a signal that the message isn't worth responding to? Like, some part of my mind has the model "if the sender knew that this was an important message, then they could count on me responding to it, so if they feel the need to add a monetary incentive they must know that this isn't very important". Or something - I'm not sure if I endorse that reasoning on an intellectual level, but it seems to trigger some kind of an emotional association to things like using money to buy status when you don't have other qualities that would earn you status. (In contexts like romantic relationships or an author paying money to a vanity press to self-publish when they can't get a real publisher to agree to publish their work and pay them.)

Re: this part in your post -

This could be avoided if people who genuinely believed their stuff was important could pay some money as a costly signal of this fact.

As I understand it, the point of a costly signal is that it's supposed to be relatively more affordable if you actually have that quality. If you have lots of health, then you can avoid to burn health on things which aren't directly useful, more than people with little health can. But the amount of money that you have, seems independent of how important your stuff is? Your could be a millionaire who wanted my opinion on something totally unimportant. You say that actual crackpots might be less likely to pay, but I would expect that if anything they would be even more likely to pay.

Comment by kaj_sotala on Machine Learning Analogy for Meditation (illustrated) · 2019-08-12T07:45:35.493Z · score: 5 (2 votes) · LW · GW

Do you define thoughts as something relatively specific - that is, does the post make any more sense if you substitute "mental contents" for "thoughts"?

Comment by kaj_sotala on Could we solve this email mess if we all moved to paid emails? · 2019-08-12T06:55:23.172Z · score: 11 (5 votes) · LW · GW

To the extent that I've experienced these kinds of problems, their core cause has been that I haven't had the time or energy to answer my messages, not that there would have been particularly many of them or because of any information asymmetry. So I wouldn't use this service because I don't recognize the problem that it's describing from my own experience.

Comment by kaj_sotala on What is the state of the ego depletion field? · 2019-08-11T11:05:06.427Z · score: 3 (1 votes) · LW · GW

Papers that make a somewhat related argument:

Why self-control seems (but may not be) limited (2014):

Self-control refers to the mental processes that allow people to override thoughts and emotions, thus enabling behavior to vary adaptively from moment to moment. Dominating contemporary research on this topic is the viewpoint that self-control relies upon a limited resource, such that engaging in acts of restraint depletes this inner capacity and undermines subsequent attempts at control (i.e., ego depletion). Noting theoretical and empirical problems with this view, here we advance a competing model that develops a nonresource-based account of self-control. We suggest that apparent regulatory failures reflect the motivated switching of task priorities as people strive to strike an optimal balance between engaging cognitive labor to pursue ‘have-to’ goals versus preferring cognitive leisure in the pursuit of ‘want-to’ goals.


Proximate and Ultimate Causes of Ego Depletion (2016):

The shifting-priorities process model describes ego depletion as a type of mental fatigue that occurs after engaging in any effortful, unrewarding task. Both humans and nonhuman animals are evolutionarily incentivized to properly balance between their immediate and long-term needs; excessive attention to future preparations can hurt immediate survival, but sometimes delaying gratification is advantageous. The balance between seeking immediate rewards and seeking resources for the future can be seen in foraging decisions (exploitation vs. exploration) as well as trade-offs between labor and leisure, and between have-to and want-to goals. The effortfulness of a task is determined not only by the use of executive functions, but also by the degree of immediate enjoyment produced by the task, and this effortfulness drives the feeling of fatigue. After experiencing an activity as unrewarding and effortful, people's attention and motivation shift away from continuing effortful future-oriented tasks and toward gratifying stimuli; these changes are seen both behaviorally and neurologically.
Comment by kaj_sotala on AI Alignment Open Thread August 2019 · 2019-08-09T10:23:56.215Z · score: 5 (2 votes) · LW · GW

Wasn't hardware overhang the argument that if AGI is more bottlenecked by software than hardware, then conceptual insights on the software side could cause a discontinuity as people suddenly figured out how to use that hardware effectively? I'm not sure how your counterargument really works there, since the AI that arrives "a bit earlier" either precedes or follows that conceptual breakthrough. If it precedes the breakthrough, then it doesn't benefit from that conceptual insight so won't be powerful enough to take advantage of the overhang, and if it follows it, then it has a discontinuous advantage over previous systems and can take advantage of hardware overhang.

---

Separately, your comment also feels related to my argument that focusing on just superintelligence is a useful simplifying assumption, since a superintelligence is almost by definition capable of taking over the world. But it simplifies things a little too much, because if we focus too much on just the superintelligence case, we might miss the emergence of a “dumb” AGI which nevertheless had the "crucial capabilities" necessary for a world takeover.

In those terms, "having sufficient offensive cybersecurity capability that a hacking attempt can snowball into a world takeover" would be one such crucial capability that allowed for a discontinuity.

Comment by kaj_sotala on Benito's Shortform Feed · 2019-08-08T19:47:02.470Z · score: 3 (1 votes) · LW · GW
(b) the more well-understood a problem is, the easier it is to implement a solution.

This might be true, but it doesn't sound like it contradicts the premise of "how will we implement it"? Namely, just because understanding a problem makes it easier to implement, doesn't mean that understanding alone makes it anywhere near easy to implement, and one may still need significant political clout in addition to having the solution. E.g. the whole infant nutrition thing.

Comment by kaj_sotala on AI Alignment Open Thread August 2019 · 2019-08-08T18:31:13.272Z · score: 7 (3 votes) · LW · GW

There's previously been the "an AI could achieve a discontinuous takeoff by exploiting a security vulnerability to copy itself into lots of other computers" argument in at least Sotala 2012 (sect 4.1.) and Sotala & Yampolskiy 2015 (footnote 15), though those don't explicitly mention the "use the additional capabilities to break into even more systems" part. (It seems reasonably implicitly there to me, but that might just be illusion of transparency speaking.)

Comment by kaj_sotala on In defense of Oracle ("Tool") AI research · 2019-08-08T09:29:33.423Z · score: 6 (2 votes) · LW · GW
Are you in the camp of "we should make a benevolent dictator AI implementing CEV", or "we can make task-limited-AGI-agents and coordinate to never make long-term-planning-AGI-agents", or something else?

No idea. :-)

My general feeling is that having an opinion on the best course of approach would require knowing what AGI and the state of the world will be like when it is developed, but we currently don't know either.

Lots of historical predictions about coming problems have been rendered completely irrelevant because something totally unexpected happened. And the other way around; it would have been hard for people to predict the issue of computer viruses before electricity had been invented, and harder yet to think about how to prepare for it. That might be a bit of exaggeration - our state of understanding about AGI is probably better than the understanding that pre-electric people would have had of computer viruses - but it still feels impossible to effectively reason about at the moment.

My preferred approach is to just have people pursue many different kinds of basic research on AI safety, understanding human values, etc., while also engaging with near-term AI issues so that they get influence in the kinds of organizations which will eventually make decisions about AI. And then hope that we figure out something once the picture becomes clearer.


Comment by kaj_sotala on In defense of Oracle ("Tool") AI research · 2019-08-07T16:19:30.183Z · score: 6 (3 votes) · LW · GW
At least as far as I can tell from the public discourse, there seems to be a growing consensus that humans should always and forever be in the loop of AGIs.

Maybe; but there also seems to be a general consensus that humans should be kept in the loop when doing any important decisions in general; yet there are also powerful incentives pushing various actors to automate their modern-day autonomous systems. In particular, there are cases where not having a human in the loop is an advantage by itself, because it e.g. buys you a faster reaction time (see high-frequency trading).

From "Disjunctive Scenarios of Catastrophic AI Risk":



The historical trend has been to automate everything that can be automated, both to reduce costs and because machines can do things better than humans can. Any kind of a business could potentially run better if it were run by a mind that had been custom-built for running the business—up to and including the replacement of all the workers with one or more with such minds. An AI can think faster and smarter, deal with more information at once, and work for a unified purpose rather than have its efficiency weakened by the kinds of office politics that plague any large organization. Some estimates already suggest that half of the tasks that people are paid to do are susceptible to being automated using techniques from modern-day machine learning and robotics, even without postulating AIs with general intelligence (Frey & Osborne 2013, Manyika et al. 2017).

The trend toward automation has been going on throughout history, doesn’t show any signs of stopping, and inherently involves giving the AI systems whatever agency they need in order to run the company better. There is a risk that AI systems that were initially simple and of limited intelligence would gradually gain increasing power and responsibilities as they learned and were upgraded, until large parts of society were under AI control. [...]

[Deploying autonomous AI systems] can happen in two forms: either by expanding the amount of control that already-existing systems have, or alternatively by upgrading existing systems or adding new ones with previously-unseen capabilities. These two forms can blend into each other. If humans previously carried out some functions which are then given over to an upgraded AI which has become recently capable of doing them, this can increase the AI’s autonomy both by making it more powerful and by reducing the amount of humans that were previously in the loop.

As a partial example, the U.S. military is seeking to eventually transition to a state where the human operators of robot weapons are “on the loop” rather than “in the loop” (Wallach & Allen 2013). In other words, whereas a human was previously required to explicitly give the order before a robot was allowed to initiate possibly lethal activity, in the future humans are meant to merely supervise the robot’s actions and interfere if something goes wrong. While this would allow the system to react faster, it would also limit the window that the human operators have for overriding any mistakes that the system makes. For a number of military systems, such as automatic weapons defense systems designed to shoot down incoming missiles and rockets, the extent of human oversight is already limited to accepting or overriding a computer’s plan of actions in a matter of seconds, which may be too little to make a meaningful decision in practice (Human Rights Watch 2012).

Sparrow (2016) reviews three major reasons which incentivize major governments to move toward autonomous weapon systems and reduce human control:

1. Currently existing remotely piloted military “drones,” such as the U.S. Predator and Reaper, require a high amount of communications bandwidth. This limits the amount of drones that can be fielded at once, and makes them dependent on communications satellites which not every nation has, and which can be jammed or targeted by enemies. A need to be in constant communication with remote operators also makes it impossible to create drone submarines, which need to maintain a communications blackout before and during combat. Making the drones autonomous and capable of acting without human supervision would avoid all of these problems.

2. Particularly in air-to-air combat, victory may depend on making very quick decisions. Current air combat is already pushing against the limits of what the human nervous system can handle: further progress may be dependent on removing humans from the loop entirely.

3. Much of the routine operation of drones is very monotonous and boring, which is a major contributor to accidents. The training expenses, salaries, and other benefits of the drone operators are also major expenses for the militaries employing them.

Sparrow’s arguments are specific to the military domain, but they demonstrate the argument that “any broad domain involving high stakes, adversarial decision making, and a need to act rapidly is likely to become increasingly dominated by autonomous systems” (Sotala & Yampolskiy 2015, p. 18). Similar arguments can be made in the business domain: eliminating human employees to reduce costs from mistakes and salaries is something that companies would also be incentivized to do, and making a profit in the field of high-frequency trading already depends on outperforming other traders by fractions of a second. While the currently existing AI systems are not powerful enough to cause global catastrophe, incentives such as these might drive an upgrading of their capabilities that eventually brought them to that point.

In the absence of sufficient regulation, there could be a “race to the bottom of human control” where state or business actors competed to reduce human control and increased the autonomy of their AI systems to obtain an edge over their competitors (see also Armstrong et al. 2016 for a simplified “race to the precipice” scenario). This would be analogous to the “race to the bottom” in current politics, where government actors compete to deregulate or to lower taxes in order to retain or attract businesses.

Suppose that you have a powerful government or corporate actor which has been spending a long time upgrading its AI systems to be increasingly powerful, and achieved better and better gains that way. Now someone shows up and says that they shouldn't make [some set of additional upgrades], because that would push it to the level of a general intelligence, and having autonomous AGIs is bad. I would expect them to do everything in power to argue that no, actually this is still narrow AI, doing these upgrades and keeping the system in control of their operations are fine - especially if they know that failing to do so is likely to confer an advantage to one of their competitors.

The problem is related to one discussed by Goertzel & Pitt (2012): it seems unlikely that governments would ban narrow AI or restrict its development, but there's no clear dividing line between narrow AI and AGI, meaning that if you don't restrict narrow AI then you can't restrict AGI either.

To make the point more directly, the prospect of any modern government seeking to put a damper on current real-world narrow-AI technology seems remote and absurd. It’s hard to imagine the US government forcing a roll-back from modern search engines like Google and Bing to more simplistic search engines like 1997 AltaVista, on the basis that the former embody natural language processing technology that represents a step along the path to powerful AGI.
Wall Street firms (that currently have powerful economic influence on the US government) will not wish to give up their AI-based trading systems, at least not while their counterparts in other countries are using such systems to compete with them on the international currency futures market. Assuming the government did somehow ban AI-based trading systems, how would this be enforced? Would a programmer at a hedge fund be stopped from inserting some more-effective machine learning code in place of the government-sanctioned linear regression code? The US military will not give up their AI-based planning and scheduling systems, as otherwise they would be unable to utilize their military resources effectively. The idea of the government placing an IQ limit on the AI characters in video games, out of fear that these characters might one day become too smart, also seems absurd. Even if the government did so, hackers worldwide would still be drawn to release “mods” for their own smart AIs inserted illicitly into games; and one might see a subculture of pirate games with illegally smart AI.
“Okay, but all these examples are narrow AI, not AGI!” you may argue. “Banning AI that occurs embedded inside practical products is one thing; banning autonomous AGI systems with their own motivations and self-autonomy and the ability to take over the world and kill all humans is quite another!” Note though that the professional AI community does not yet draw a clear border between narrow AI and AGI. While we do believe there is a clear qualitative conceptual distinction, we would find it hard to embody this distinction in a rigorous test for distinguishing narrow AI systems from “proto-AGI systems” representing dramatic partial progress toward human-level AGI. At precisely what level of intelligence would you propose to ban a conversational natural language search interface, an automated call center chatbot, or a house-cleaning robot? How would you distinguish rigorously, across all areas of application, a competent non-threatening narrow-AI system from something with sufficient general intelligence to count as part of the path to dangerous AGI?
A recent workshop of a dozen AGI experts, oriented largely toward originating such tests, failed to come to any definitive conclusions (Adams et al. 2010), recommending instead that a looser mode of evaluation be adopted, involving qualitative synthesis of multiple rigorous evaluations obtained in multiple distinct scenarios. A previous workshop with a similar theme, funded by the US Naval Research Office, came to even less distinct conclusions (Laird et al. 2009). The OpenCog system is explicitly focused on AGI rather than narrow AI, but its various learning modules are also applicable as narrow AI systems, and some of them have largely been developed in this context. In short, there’s no rule for distinguishing narrow AI work from proto-AGI work that is sufficiently clear to be enshrined in government policy, and the banning of narrow AI work seems infeasible as the latter is economically and humanistically valuable, tightly interwoven with nearly all aspects of the economy, and nearly always non-threatening in nature. Even in the military context, the biggest use of AI is in relatively harmless-sounding contexts such as back-end logistics systems, not in frightening applications like killer robots.
Surveying history, one struggles to find good examples of advanced, developed economies slowing down development of any technology with a nebulous definition, obvious wide-ranging short to medium term economic benefits, and rich penetration into multiple industry sectors, due to reasons of speculative perceived long-term risks. Nuclear power research is an example where government policy has slowed things down, but here the perceived economic benefit is relatively modest, the technology is restricted to one sector, the definition of what’s being banned is very clear, and the risks are immediate rather than speculative. More worryingly, nuclear weapons research and development continued unabated for years, despite the clear threat it posed.
Comment by kaj_sotala on In defense of Oracle ("Tool") AI research · 2019-08-07T16:00:49.023Z · score: 14 (6 votes) · LW · GW
Even more importantly, take the rule:
"AGIs are not allowed to do anything except output pixels onto a screen."
This is a nice, simple, bright-line rule,

It is a bright line in one sense, but it has the problem that humans remaining technically in the loop may not make much of a difference in practice. From "Disjunctive Scenarios of Catastrophic AI Risk":

Even if humans were technically kept in the loop, they might not have the time, opportunity, motivation, intelligence, or confidence to verify the advice given by an AI. This would particularly be the case after the AI had functioned for a while, and established a reputation as trustworthy. It may become common practice to act automatically on the AI’s recommendations, and it may become increasingly difficult to challenge the “authority” of the recommendations. Eventually, the AI may in effect begin to dictate decisions (Friedman & Kahn 1992).

Likewise, Bostrom and Yudkowsky (2014) point out that modern bureaucrats often follow established procedures to the letter, rather than exercising their own judgment and allowing themselves to be blamed for any mistakes that follow. Dutifully following all the recommendations of an AI system would be another way of avoiding blame.

O’Neil (2016) documents a number of situations in which modern-day machine learning is used to make substantive decisions, even though the exact models behind those decisions may be trade secrets or otherwise hidden from outside critique. Among other examples, such models have been used to fire school teachers that the systems classified as underperforming and give harsher sentences to criminals that a model predicted to have a high risk of reoffending. In some cases, people have been skeptical of the results of the systems, and even identified plausible reasons why their results might be wrong, but still went along with their authority as long as it could not be definitely shown that the models were erroneous.

In the military domain, Wallach & Allen (2013) note the existence of robots which attempt to automatically detect the locations of hostile snipers and to point them out to soldiers. To the extent that these soldiers have come to trust the robots, they could be seen as carrying out the robots’ orders. Eventually, equipping the robot with its own weapons would merely dispense with the formality of needing to have a human to pull the trigger.
Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-08-07T12:36:46.965Z · score: 3 (1 votes) · LW · GW

(I now talk about exiled protectors a bit in "Subagents, neural Turing machines, thought selection, and blindspots"; quite relevant for the topic of hunting one's shadow, if I may say so myself)

Comment by kaj_sotala on Why Subagents? · 2019-08-07T10:06:43.142Z · score: 7 (4 votes) · LW · GW
If it chooses pepperoni, but refuses to swap mushroom for pepperoni then its decisions depend on how the situation is framed. How close does it have to get to the mushroom before they "have" mushroom and refuse to swap?

Relevant example.

I previously suggested that revealing the two options as equivalent will bring the two subagents into a standstill, requiring some third factor to help decide. Which seems close to what happens if I introspect on what happens if I'm offered a choice between two foods that I think are equally good - I just decide at random or go by e.g. some force of habit that provides a slight starting point bias.

Comment by kaj_sotala on Building up to an Internal Family Systems model · 2019-08-05T18:03:32.553Z · score: 6 (3 votes) · LW · GW

It doesn't really fit nicely into the simplified version of IFS that I presented in this post, but in the context of Hunting the Shadow, it's worth noting that some protector parts can get exiled too.

Comment by kaj_sotala on How to Ignore Your Emotions (while also thinking you're awesome at emotions) · 2019-08-03T20:10:02.014Z · score: 3 (1 votes) · LW · GW

Right, I agree that having an explicit intention to go looking for traumatic memories is likely to be counterproductive.

Comment by kaj_sotala on How to Ignore Your Emotions (while also thinking you're awesome at emotions) · 2019-08-03T18:27:35.137Z · score: 3 (1 votes) · LW · GW

What Kaj is saying, I think, is that the possibility of being wrong is not justification for closing ones eyes and not looking. [...] It's not that "recovering memories" is especially error prone, it's that everything is error prone and people often fail to appreciate how unreliable memory can be because they don't actually get how it works.

I endorse this summary.

Comment by kaj_sotala on How to Ignore Your Emotions (while also thinking you're awesome at emotions) · 2019-08-03T07:23:05.324Z · score: 5 (2 votes) · LW · GW

I don't think it's a problem to just gently ask and then be open to the possibility of something coming up. That's different from the kinds of leading questions that typically create false memories. Especially since Focusing/IFS/etc. style techniques seem to cause memories to come up spontaneously in any case, it's just slightly nudging the process forward.

It also doesn't necessarily matter whether the memories are true or not, as long as it helps the healing process along. We all have plenty of false or misleading memories in our heads anyway.

Comment by kaj_sotala on How to Ignore Your Emotions (while also thinking you're awesome at emotions) · 2019-07-31T22:05:08.038Z · score: 5 (3 votes) · LW · GW
he's slightly more open to lowering the weight when I acknowledge his venting a bit more first.

My suggestion is to continue with this route. Receive his venting, seek to genuinely empathize with it, try to understand and acknowledge his position as well as you can. Remember that understanding his position doesn't mean that you would need to act according to all of his wishes: you can validate his experience and perspective without making a commitment to go along to everything. Just seek to understand as well as possible, without trying to argue with him.

(If you ever find it difficult not to argue or empathize, try treating that desire to argue or empathize as another part of your psyche, one which can be asked if it would be willing to move aside in order to let you help notme better.)

That said, he might not be willing to tell you everything until he trusts you enough. And if he is willing to negotiate with you in exchange for a concession, that can be a useful way to build mutual trust as well.

In all likelihood, you are talking with a traumatized part of your psyche [1, 2, 3]. He has witnessed experiences which make him have extreme beliefs, so that normal IDC is a poor fit and is likely to stall, the way you've seen it stall. The part is only going to relax once you've witnessed the original memories which make him take on that extreme role, understood why he feels that way, and been able to give him the comfort and adult support that he would originally have needed in that situation.

Just keep listening and building trust, until he's ready to show you those original memories. Questions like "what are you afraid would happen if I didn't do what you wanted" or "what would be bad about that" may be useful, as is actively validating whatever he says and offering him comfort. So might "Do you feel like I fully understand you now". "How old are you" and "how old do you think I am" may also provide interesting results.

Like you said, he did the best one could expect of someone that age. But he's probably still partially stuck in those experiences. It's time to help him heal, and to help him see that you've got the resources to handle things on your own now. Once that happens, he's free to take a new role inside your psyche, one which is likely to feel much easier for him.

Comment by kaj_sotala on How to Ignore Your Emotions (while also thinking you're awesome at emotions) · 2019-07-31T19:00:48.192Z · score: 4 (2 votes) · LW · GW
My internal double crux always gets down to this point and then notme says prove it and I don't know how which leads me to...

What happens if, instead of trying to prove to notme that it won't happen, you ask notme to show you (in a way which won't overwhelm you, in case the belief emerges from some particularly nasty memory) why it thinks it will happen?

Comment by kaj_sotala on Drive-By Low-Effort Criticism · 2019-07-31T15:56:47.079Z · score: 60 (20 votes) · LW · GW

As someone who's written high-effort posts, I appreciate the thought.

That said, while I can't speak for others, personally I'd rather have low-effort criticism on my high-effort posts than no engagement at all. Even low-effort criticism can prompt useful discussion, it communicates to me what kinds of immediate reactions my post provoked, the post having more comments increases the odds of more people reading it, etc.

The main situation in which I would be annoyed by low-effort criticism, would be if it was raising an objection which I had already addressed in the post and if the person seemed unaware of me having done so. That doesn't seem to be the case with either of your examples, though.

I also feel that as an author, I should just accept it that if I write long posts, then people are more likely to skim them or only read them in part, and comment based on that. That's a cost that I accept when I decide to write a long post; because me writing a long post means that I'm imposing a cost of time on my readers, and I can't expect every reader to find that cost worth paying. If people don't read my posts in full, then that means that I as a writer have failed to provide them with enough value to make the post worth their time. That's a failing on my part, and I don't consider people impolite for providing me with low value when I've also failed to provide them with high value.

Comment by kaj_sotala on How to Ignore Your Emotions (while also thinking you're awesome at emotions) · 2019-07-31T15:04:13.405Z · score: 38 (24 votes) · LW · GW
The biggest barrier to rational thinking is organizing your mind such that it's safe to think

Related: besides being slapped down, another thing which may discourage subagents from speaking up is if one is too willing to share their reports with other people:

For many years, I thought privacy was a fake virtue and only valuable for self-defense. I understood that some people would be unfairly persecuted for their minority sexuality, say, or stigmatized disease status, but I always saw that more as a flaw in society and not a point in favor of privacy. I thought privacy was an important right, but that the ideal was not to need it.
I’m coming back around to privacy for a few reasons, first of which was my several year experiment with radical transparency. For a lot of that time, it seemed to be working. Secrets didn’t pile up and incubate shame, and white lies were no longer at my fingertips. I felt less embarrassed and ashamed over the kind of things everyone has but no one talks about. Not all of it was unhealthy sharing, but I knew I frequently met the definition of oversharing– I just didn’t understand what was wrong with that.
I noticed after several years of this behavior that I wasn’t as in touch with my true feelings. At first I thought my total honesty policy had purged me of a lot of the messy and conflicted feelings I used to have. But there was something suspiciously shallow about these more presentable feelings. I now believe that, because I scrupulously reported almost anything to anyone who asked (or didn’t ask), I conveniently stopped being aware of a lot of my most personal and tender feelings. [...]
I now think privacy is important for maximizing self-awareness and self-transparency. The primary function of privacy is not to hide things society finds unacceptable, but to create an environment in which your own mind feels safe to tell you things. If you’re not allowing these unshareworthy thoughts and feelings a space to come out, they still affect your feelings and behavior– you just don’t know how or why. And all the while your conscious self-image is growing more alienated from the processes that actually drive you. Privacy creates the necessary conditions for self-honesty, which is a necessary prerequisite to honesty with anyone else. When you only know a cleaned-up version of yourself, you’ll only be giving others a version of your truth.
Here’s an image that’s been occurring to me. Privacy creates a space in which unexpected or unsightly things can be expressed. It’s like a cocoon for thoughts and feelings. A lot of ugly transformational work can take place there that simply couldn’t occur in an open environment (the bug literally dissolves!). The gnarly thoughts and feelings need to do their work undisturbed by any self-consciousness or fear of judgment, just like caterpillars need a tight encasement where the wind won’t scatter their components as they reassemble into butterflies.
Comment by kaj_sotala on Arguments for the existence of qualia · 2019-07-31T13:28:51.902Z · score: 3 (1 votes) · LW · GW

Well, to be fair, there is this bit in Consciousness Explained (which I happened to be reading just now):

Philosophers have adopted various names for the things in the beholder (or properties of the beholder) that have been supposed to provide a safe home for the colors and the rest of the properties that have been banished from the “external” world by the triumphs of physics: “raw feels,” “sensa,” “phenomenal qualities,” “intrinsic properties of conscious experiences,” “the qualitative content of mental states,” and, of course, “qualia,” the term I will use. There are subtle differences in how these terms have been defined, but I’m going to ride roughshod over them. In the previous chapter I seemed to be denying that there are any such properties, and for once what seems so is so. I am denying that there are any such properties. But (here comes that theme again) I agree wholeheartedly that there seem to be qualia.

Comment by kaj_sotala on Arguments for the existence of qualia · 2019-07-30T19:20:34.138Z · score: 3 (1 votes) · LW · GW

That wasn't what I meant.

One could certainly debate the existence of say points, without disputing that they are a primitive notion. For instance, one could argue that points are a contradictory concept since they have an area of zero, but that each point that we can physically draw always has some area. Someone could then present a counterargument to that. Neither of those arguments would dispute points being a primitive notion.

Rather my argument is that if you are discussing the existence of a primitive notion, you have to explain what it would mean for it to not exist. Otherwise it is hard to understand what the debate is about, since naively, points/qualia seem to self-evidently exist.

Comment by kaj_sotala on What woo to read? · 2019-07-29T15:45:07.178Z · score: 3 (1 votes) · LW · GW

Looks like we posted at the same time. :-)

Comment by kaj_sotala on What woo to read? · 2019-07-29T15:43:26.833Z · score: 27 (9 votes) · LW · GW

My favorite LW explanation is Ms. Blue, meet Mr. Green. The short explanation is that while some woo is just outright crazy, much of it is valid in the sense that it is describing ways of manipulating your own consciousness and subjective experiences. This can be used for beneficial psychological effects.

Terms like "energy" should be interpreted to refer to, not energy in the physical sense, but rather sensations which feel like energy and which can be manipulated to achieve various internal effects. From the post:

In general, if you translate all mystical statements to be talking about the internal experiences, they’ll make a lot more sense. Let’s take a few common mystical concepts and see how we can translate them.

Energy -- there are sensations that form a feeling of something liquid (or gaseous) that moves within or outside of your body. When unpacked, it’s likely to be made up of sensations of light (visual), warmth, tingling, and tension (physical). “Channeling energy” is adjusting your state of mind so as to cause these sensations to “move” in a certain way, to appear or disappear.

Chakras -- points in your body where it’s particularly easy to feel certain types of energies. It’s also particularly easy to visualize / feel the energy moving into or out of those places. “Aligning chakras” is about adjusting your state of mind so as to cause the energy to flow evenly through all chakras. (Chakras are a great example of a model that pays rent. You can read about chakras, see what predictions people are making about your internal experience when you explore chakras, and then you can go and explore them within yourself to see if the predictions are accurate.) [...]

Auras -- when you look at a person and they are scowling, you might conclude that they are angry. They have a “danger” aura. If I blurred their face, and you were particularly perceptive, you might feel the aura even then, based on their body language or other cues. Some people can pick up on pretty surprising things, like a person being sick or struggling with an internal dilemma, all the way to being able to “see” things about their future. Auras are subconscious visual representations of emotions (how a person is making you feel / presenting themselves) that you become conscious of. Some people actually have the sensation of seeing the aura, where it’s as real to them as the candle light.

Another nice resource is Eric S. Raymond's Dancing with the Gods, where the author discusses mysticism and religion in similar terms - not as a set of beliefs to uncritically accept, but as a set of practices which can be used to get yourself into useful psychological states.

Comment by kaj_sotala on Arguments for the existence of qualia · 2019-07-29T11:55:30.872Z · score: 22 (6 votes) · LW · GW

To me, a primitive notion is something that doesn't need further defining - you can just point to an example and people will know from that example what you mean. If people don't know what you mean from an example, then it doesn't seem to work as a primitive notion.

There's something like that to qualia, in that you can give examples of subjective experience, and people will know what you mean. But your post was arguing against people who were saying that qualia don't exist. In that context, using the primitive sense of qualia seems insufficient, since you are taking something whose existence seems self-evident from our experience, and start talking about whether or not it exists. That makes me think that you must mean something else than the primitive notion, since I don't understand how there could be a dispute about the existence of the primitive notion.

To use the analogy to points, suppose that someone had written a post saying "there are people who argue that points in geometry do not really exist, but I will now present arguments that they do exist". The existence of points as a primitive notion seems self-evident to me: after all, I can draw a point, do geometry using the primitive notion of points, etc. So I assume that the post must be talking about something else than the primitive notion or using some more technical definition of "exist" or something; otherwise there would be no need to argue for the existence of points, nor could their existence to be disputed.

To discuss the existence of points as something which is up to debate, seems to already presuppose that they are not a primitive notion. Likewise, if you say that "by qualia, I mean qualia as the primitive notion", that doesn't seem useful in clarifying what your post is talking about, since it already seems self-evident to me that qualia as a primitive notion exist. So it feels like any dispute about their existence has to define them as something else than the primitive notion.

Comment by kaj_sotala on Arguments for the existence of qualia · 2019-07-28T14:15:04.584Z · score: 11 (5 votes) · LW · GW

I'm not sure of what it would mean for qualia to exist or not exist. Could you give an explanation/definition of what exactly is the position that you are defending? (possibly it might also be helpful to explain the position that you are arguing against)

Comment by kaj_sotala on Book Review: The Secret Of Our Success · 2019-07-23T03:06:13.309Z · score: 11 (5 votes) · LW · GW

While this is a good post, I objected to curating it as the later "Highlights from the Comments on Cultural Evolution" had lots of comments suggesting that various quoted excerpts were either misleading or false. This made me pretty skeptical about all of the book's other claims too.

Curating this review feels to me at odds with the stated goal of maintaining a repository of the highlights of ongoing conversations, since we have a later post which to me basically said that the book is unreliable enough to not be worth reading.

Comment by kaj_sotala on Do you fear the rock or the hard place? · 2019-07-21T18:31:20.582Z · score: 12 (3 votes) · LW · GW
Acknowledge their fear. Acknowledge that something they think is very important is at stake in your discussion.
Step 1 is simply to acknowledge the fear. The following steps depend on your own beliefs.
You might say “I too am afraid of R”, but:

Important to highlight that "acknowledge" here doesn't just mean "yeah I get it" before moving on to the "but". It will probably take quite a bit of validating their fear before they'll feel heard and calm down enough to listen to you: be prepared to spend a reasonable amount of time doing it. And make sure that you'll actually listen to them rather than just treating the acknowledgement as a token gesture that you need to do before you can present your point - if you're not actually taking people's concerns seriously, they will tend to notice it.

Sadly, this doesn't mean that the converse is true - sometimes they will feel that you're just setting up strawmen of their fears when you are just honestly trying to understand and making a genuine effort to verbalize your best guess of their fears. You'll need to be patient, while also being prepared for the possibility that this will fail despite your best efforts.

Comment by kaj_sotala on Dialogue on Appeals to Consequences · 2019-07-19T11:54:09.447Z · score: 18 (6 votes) · LW · GW

The post would be a lot clearer if it had a motivating example that really did have bad consequences, ask things considered.

The extreme case would be a scientific discovery which enabled anyone to destroy the world, such as the supernova thing in Three Worlds Collide or the thought experiment that Bostrom discusses in The Vulnerable World Hypothesis:

So let us consider a counterfactual history in which Szilard invents nuclear fission and realizes that a nuclear bomb could be made with a piece of glass, a metal object, and a battery arranged in a particular configuration. What happens next? Szilard becomes gravely concerned. He sees that his discovery must be kept secret at all costs. But how. His insight is bound to occur to others. He could talk to a few of his physicist friends, the ones most likely to stumble upon the idea, and try to persuade them not to publish anything on nuclear chain reactions or on any of the reasoning steps leading up to the dangerous discovery. (That is what Szilard did in actual history.)

[...] Soon, figuring out how to initiate a nuclear chain reaction with pieces of metal, glass, and electricity will no longer take genius but will be within reach of any STEM student with an inventive mindset.

Comment by kaj_sotala on 1st Athena Rationality Workshop - Retrospective · 2019-07-17T21:21:56.456Z · score: 3 (1 votes) · LW · GW

Apparently it's been a month since the workshop. Any longer-term impressions of how useful the material turned out to be?

Comment by kaj_sotala on Prereq: Cognitive Fusion · 2019-07-17T19:55:18.718Z · score: 11 (3 votes) · LW · GW

When I focus on the object of attachment (assumption) I'm mostly thinking about just how stupid it is and how I can't believe that anyone would be dumb enough to fall for this, and I most certainly don't believe anything that stupid....

And the irony here is that you are fused with the thought of fusion being stupid. :-)

Comment by kaj_sotala on Open Thread July 2019 · 2019-07-17T07:30:20.801Z · score: 16 (4 votes) · LW · GW

I think that your past criticisms have been useful, and I've explicitly tried to take them into account in the sequence. E.g. the way I defined subagents in the first post of the sequence, was IIRC in part copy-pasted from an earlier response to you, and it was your previous comment that helped/forced me to clarify what exactly I meant. I'd in fact been hoping to see more comments from you on the posts, and expect them to be useful regardless of the tone.

Comment by kaj_sotala on A Key Power of the President is to Coordinate the Execution of Existing Concrete Plans · 2019-07-16T22:08:52.956Z · score: 27 (8 votes) · LW · GW
My new model is that the President's interaction with science is largely to take concrete ideas floating around in the environment that are ready for their time, and push them over the edge into actually being built by the US private sector, or into actually substantially informing government policy. This is similar to the notion that scientific ideas come about when the environment is ready for them (Newton and Leibniz both discovering calculus at the same time).

Reminded me of this:

“Only a crisis - actual or perceived - produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around. That, I believe, is our basic function: to develop alternatives to existing policies, to keep them alive and available until the politically impossible becomes the politically inevitable." (Milton Friedman)
Comment by kaj_sotala on Open Thread July 2019 · 2019-07-14T19:56:25.209Z · score: 11 (6 votes) · LW · GW

Constructivist learning theory is a relevant keyword; its premise is pretty much directly the same as in your quote (my emphasis added):

An important restriction of education is that teachers cannot simply transmit knowledge to students, but students need to actively construct knowledge in their own minds. That is, they discover and transform information, check new information against old, and revise rules when they do not longer apply. This constructivist view of learning considers the learner as an active agent in the process of knowledge acquisition. Constructivist conceptions of learning have their historical roots in the work of Dewey (192 9), Bruner (1961), Vygotsky (1962), and Piaget (1980). Bednar, Cunningham, Duffy, and Perry (1992) and von Glasersfeld (1995) have proposed several implications of constructivist theory for instructional developers stressing that learning outcomes should focus on the knowledge construction process and that learning goals should be determined from authentic tasks with specific objectives. Similarly, von Glasersfeld (1995) states that learning is not a stimulus-response phenomenon, but a process that requires self-regulation and the development of conceptual structures through reflection and abstraction. It is important to note, in this respect, that constructivism is embodied in numerous ways and that these different views share important overlaps, but also c ontain major differences.

Constructivism is an approach to teaching and learning based on the premise that cognition (learning) is the result of "mental construction." In other words, students learn by fitting new information together with what they already know.

It's a big topic in educational research, so there's a lot of stuff about it out there. E.g. "How Learning Works: Seven Research-Based Principles for Smart Teaching" summarizes some research on it:

Students connect what they learn to what they already know, interpreting incoming information, and even sensory perception, through the lens of their existing knowledge, beliefs, and assumptions (Vygotsky, 1978 ; National Research Council, 2000 ). In fact, there is widespread agreement among researchers that students must connect new knowledge to previous knowledge in order to learn (Bransford & Johnson, 1972 ; Resnick, 1983 ). However, the extent to which students are able to draw on prior knowledge to effectively construct new knowledge depends on the nature of their prior knowledge, as well as the instructor's ability to harness it. In the following sections, we discuss research that investigates the effects of various kinds of prior knowledge on student learning and explore its implications for teaching.

One related example of it that I particularly like is in the paper Building Islands of Expertise in Everyday Family Activity. It discusses how a boy who's interested in trains initially learns stuff about trains, which then helps him learn more about other stuff as well:

By the time the boy turns 3-years old, he has developed an island of expertise around trains. His vocabulary, declarative knowledge, conceptual knowledge, schemas, and personal memories related to trains are numerous, well-organized, and flexible. Perhaps more importantly, the boy and his parents have developed a relatively sophisticated conversational space for trains. Their shared knowledge and experience allow their talk to move to deeper levels than is typically possible in a domain where the boy is a relative novice. For example, as the mother is making tea one afternoon, the boy notices the steam rushing out of the kettle and says: “That’s just like a train!” The mother might laugh and then unpack the similarity to hammer the point home: “Yes it is like a train! When you boil water it turns into steam. That’s why they have boilers in locomotives. They heat up the water, turn it into steam, and then use the steam to push the drive wheels. Remember? We saw that at the museum.”

In contrast, when the family was watching football—a domain the boy does not yet know much about—he asked “Why did they knock that guy down?” The mother’s answer was short, simple, stripped of domain-specific vocabulary, and sketchy with respect to causal mechanisms—“Because that’s what you do when you play football.” Parents have a fairly good sense of what their children know and, often, they gear their answers to an appropriate level. When talking about one of the child’s islands of expertise, parents can draw on their shared knowledge base to construct a more elaborate, accurate, and meaningful explanations. This is a common characteristic of conversation in general: When we share domain-relevant experience with our audience we can use accurate terminology, construct better analogies, and rely on mutually held domain- appropriate schema as a template through which we can scribe new causal connections.

As this chapter is being written, the boy in this story is now well on his way to 4- years old. Although he still likes trains and still knows a lot about them, he is developing other islands of expertise as well. As his interests expand, the boy may engage less and less often in activities and conversations centered around trains and some of his current domain-specific knowledge will atrophy and eventually be lost. But as that occurs, the domain-general knowledge that connected the train domain to broader principles, mechanisms, and schemas will probably remain. For example, when responding to the boy’s comment about the tea kettle, the mother used the train domain as a platform to talk about the more general phenomenon of steam.

Trains were platforms for other concepts as well, in science and in other domains. Conversations about mechanisms of locomotion have served as a platform for a more general understanding of mechanical causality. Conversations about the motivation of characters in the Thomas the Tank Engine stories have served as platforms for learning about interpersonal relationships and, for that matter, about the structure of narratives. Conversations about the time when downtown Pittsburgh was threaded with train tracks and heavy-duty railroad bridges served as a platform for learning about historical time and historical change. These broader themes emerged for the boy for the first time in the context of train conversations with his parents. Even as the boy loses interest in trains and moves on to other things, these broader themes remain and expand outward to connect with other domains he encounters as he moves through his everyday life.

Expertise research is also relevant; it talks about how people build up increasingly detailed mental representations of a domain they are learning, and which guide them when they decide what actions to take. The representations start out a coarse, but get increasingly detailed over time. This is an excerpt from the book Peak: Secrets from the New Science of Expertise, which notes a "chicken and egg" problem that requires grounding any skill in some pre-existing knowledge first:

As we’ve just seen from several studies, musicians rely on mental representations to improve both the physical and cognitive aspects of their specialties. And mental representations are essential to activities we see as almost purely physical. Indeed, any expert in any field can be rightly seen as a high-achieving intellectual where that field is concerned. This applies to pretty much any activity in which the positioning and movement of a person’s body is evaluated for artistic expression by human judges. Think of gymnastics, diving, figure skating, or dancing. Performers in these areas must develop clear mental representations of how their bodies are supposed to move to generate the artistic appearance of their performance routines. But even in areas where artistic form is not explicitly judged, it is still important to train the body to move in particularly efficient ways. Swimmers learn to perform their strokes in ways that maximize thrust and minimize drag. Runners learn to stride in ways that maximize speed and endurance while conserving energy. Pole-vaulters, tennis players, martial artists, golfers, hitters in baseball, three-point shooters in basketball, weightlifters, skeet shooters, and downhill skiers—for all of these athletes proper form is key to good performance, and the performers with the best mental representations will have an advantage over the rest. 
In these areas too, the virtuous circle rules: honing the skill improves mental representation, and mental representation helps hone the skill. There is a bit of a chicken-and-egg component to this. Take figure skating: it’s hard to have a clear mental representation of what a double axel feels like until you’ve done it, and, likewise, it is difficult to do a clean double axel without a good mental representation of one. That sounds paradoxical, but it isn’t really. You work up to a double axel bit by bit, assembling the mental representations as you go. 
It’s like a staircase that you climb as you build it. Each step of your ascent puts you in a position to build the next step. Then you build that step, and you’re in a position to build the next one. And so on. Your existing mental representations guide your performance and allow you to both monitor and judge that performance. As you push yourself to do something new—to develop a new skill or sharpen an old one—you are also expanding and sharpening your mental representations, which will in turn make it possible for you to do more than you could before.

Finally, while it only touches on this topic occasionally, Josh Waitzkin had some nice "from the inside" descriptions of this gradual construction of an increasing level of understanding in his book The Art of Learning:

I practiced the Tai Chi meditative form diligently, many hours a day. At times I repeated segments of the form over and over, honing certain techniques while refining my body mechanics and deepening my sense of relaxation. I focused on small movements, sometimes spending hours moving my hand out a few inches, then releasing it back, energizing outwards, connecting my feet to my fingertips with less and less obstruction. Practicing in this manner, I was able to sharpen my feeling for Tai Chi. When through painstaking refinement of a small movement I had the improved feeling, I could translate it onto other parts of the form, and suddenly everything would start flowing at a higher level. The key was to recognize that the principles making one simple technique tick were the same fundamentals that fueled the whole expansive system of Tai Chi Chuan.
This method is similar to my early study of chess, where I explored endgame positions of reduced complexity—for example king and pawn against king, only three pieces on the board—in order to touch high-level principles such as the power of empty space, zugzwang (where any move of the opponent will destroy his position), tempo, or structural planning. Once I experienced these principles, I could apply them to complex positions because they were in my mental framework. However, if you study complicated chess openings and middlegames right off the bat, it is difficult to think in an abstract axiomatic language because all your energies are preoccupied with not blundering. It would be absurd to try to teach a new figure skater the principle of relaxation on the ice by launching straight into triple axels. She should begin with the fundamentals of gliding along the ice, turning, and skating backwards with deepening relaxation. Then, step by step, more and more complicated maneuvers can be absorbed, while she maintains the sense of ease that was initially experienced within the simplest skill set.

Comment by kaj_sotala on How much background technical knowledge do LW readers have? · 2019-07-11T18:30:47.673Z · score: 13 (7 votes) · LW · GW

Nice survey. The gap between "one course or played around a bit" and "undergrad major" in some questions felt pretty big - an "undergrad minor" option would have been nice.

(I assumed that these corresponded to roughly the amounts that I'm used to, with an undergrad major being something like 60 ECTS credits worth of courses specifically in that major, and a minor 25.)

Comment by kaj_sotala on Self-consciousness wants to make everything about itself · 2019-07-11T09:55:10.394Z · score: 5 (2 votes) · LW · GW

To rephrase:

  • I suspect that typically, when someone thinks "I shouldn't do X because that would make me evil and I shouldn't be evil", what their brain is actually computing is something like "I shouldn't do X because people who I care about would then consider me evil, and that would be bad for me", even if that's not how it appears subjectively.
    • Note that this is different from thinking that "I shouldn't do X because that's wrong". I'm not claiming that a fear of punishment would be the only thing that disinclined us from doing bad things. You can consider X to be wrong and refrain from doing it, without having a subconscious fear of punishment. The difference is that if you are afraid of being punished, then any suggestion of having done wrong is likely to trigger a defensive reaction and a desire to show that you did no wrong. Whereas if you don't feel the fear, you might just think "oh, the thing that I did was wrong", feel healthy regret, and act to fix it.
  • Sometimes you might feel like you are evil (that is, likely to be punished) because you have done, or repeatedly do, something bad. In some of these cases, it can help to redefine your concepts. For instance, if you did bad things when you were young, you can think "I did bad things when I was young and didn't know any better, but I'm a grownup now". Or "these things are evil, but nobody is actually purely good, so doing them doesn't make me any worse than other people".
  • Now, if "I shouldn't be evil" actually means "I shouldn't do things that make people punish me", then attempting these kinds of redefinitions will also trigger a subconscious evaluation process which has to be passed for the redefinition to succeed.
    • If you think that "I did bad things when I was young and didn't know any better, but I'm a grownup now", then this corresponds to something like "I did bad things when I was young, but nobody is going to hold those things against me anymore". This redefinition will only succeed if the latter sentence feels true to your brain.
    • If you think that "these things are evil, but nobody is actually purely good, so doing them doesn't make me any worse than other people", then this corresponds to something like "these things are blameworthy, but everyone does them, so doing them won't get me judged any more harshly than anyone else". Again, this redefinition will only succeed if the latter sentence feels true to your brain.
  • This is assuming that you are trying to use those redefinitions to heal the original bad feeling. One can also use these kinds of thoughts to suppress the original bad feeling (create a protector which seeks to extinguish the bad feeling using counter-arguments). In that case, the part of your mind which was originally convinced of you being in danger, doesn't need to be convinced otherwise. But this will set up an internal conflict as that part will continue to try to make itself heard, and may sometimes overwhelm whatever blocks have been put in place to keep in silent.

Related: Scott Alexander's Guilt: Another Gift Nobody Wants.