Comment by kaj_sotala on Open Thread July 2019 · 2019-07-17T07:30:20.801Z · score: 5 (2 votes) · LW · GW

I think that your past criticisms have been useful, and I've explicitly tried to take them into account in the sequence. E.g. the way I defined subagents in the first post of the sequence, was IIRC in part copy-pasted from an earlier response to you, and it was your previous comment that helped/forced me to clarify what exactly I meant. I'd in fact been hoping to see more comments from you on the posts, and expect them to be useful regardless of the tone.

Comment by kaj_sotala on A Key Power of the President is to Coordinate the Execution of Existing Concrete Plans · 2019-07-16T22:08:52.956Z · score: 23 (6 votes) · LW · GW
My new model is that the President's interaction with science is largely to take concrete ideas floating around in the environment that are ready for their time, and push them over the edge into actually being built by the US private sector, or into actually substantially informing government policy. This is similar to the notion that scientific ideas come about when the environment is ready for them (Newton and Leibniz both discovering calculus at the same time).

Reminded me of this:

“Only a crisis - actual or perceived - produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around. That, I believe, is our basic function: to develop alternatives to existing policies, to keep them alive and available until the politically impossible becomes the politically inevitable." (Milton Friedman)
Comment by kaj_sotala on Open Thread July 2019 · 2019-07-14T19:56:25.209Z · score: 8 (4 votes) · LW · GW

Constructivist learning theory is a relevant keyword; its premise is pretty much directly the same as in your quote (my emphasis added):

An important restriction of education is that teachers cannot simply transmit knowledge to students, but students need to actively construct knowledge in their own minds. That is, they discover and transform information, check new information against old, and revise rules when they do not longer apply. This constructivist view of learning considers the learner as an active agent in the process of knowledge acquisition. Constructivist conceptions of learning have their historical roots in the work of Dewey (192 9), Bruner (1961), Vygotsky (1962), and Piaget (1980). Bednar, Cunningham, Duffy, and Perry (1992) and von Glasersfeld (1995) have proposed several implications of constructivist theory for instructional developers stressing that learning outcomes should focus on the knowledge construction process and that learning goals should be determined from authentic tasks with specific objectives. Similarly, von Glasersfeld (1995) states that learning is not a stimulus-response phenomenon, but a process that requires self-regulation and the development of conceptual structures through reflection and abstraction. It is important to note, in this respect, that constructivism is embodied in numerous ways and that these different views share important overlaps, but also c ontain major differences.

Constructivism is an approach to teaching and learning based on the premise that cognition (learning) is the result of "mental construction." In other words, students learn by fitting new information together with what they already know.

It's a big topic in educational research, so there's a lot of stuff about it out there. E.g. "How Learning Works: Seven Research-Based Principles for Smart Teaching" summarizes some research on it:

Students connect what they learn to what they already know, interpreting incoming information, and even sensory perception, through the lens of their existing knowledge, beliefs, and assumptions (Vygotsky, 1978 ; National Research Council, 2000 ). In fact, there is widespread agreement among researchers that students must connect new knowledge to previous knowledge in order to learn (Bransford & Johnson, 1972 ; Resnick, 1983 ). However, the extent to which students are able to draw on prior knowledge to effectively construct new knowledge depends on the nature of their prior knowledge, as well as the instructor's ability to harness it. In the following sections, we discuss research that investigates the effects of various kinds of prior knowledge on student learning and explore its implications for teaching.

One related example of it that I particularly like is in the paper Building Islands of Expertise in Everyday Family Activity. It discusses how a boy who's interested in trains initially learns stuff about trains, which then helps him learn more about other stuff as well:

By the time the boy turns 3-years old, he has developed an island of expertise around trains. His vocabulary, declarative knowledge, conceptual knowledge, schemas, and personal memories related to trains are numerous, well-organized, and flexible. Perhaps more importantly, the boy and his parents have developed a relatively sophisticated conversational space for trains. Their shared knowledge and experience allow their talk to move to deeper levels than is typically possible in a domain where the boy is a relative novice. For example, as the mother is making tea one afternoon, the boy notices the steam rushing out of the kettle and says: “That’s just like a train!” The mother might laugh and then unpack the similarity to hammer the point home: “Yes it is like a train! When you boil water it turns into steam. That’s why they have boilers in locomotives. They heat up the water, turn it into steam, and then use the steam to push the drive wheels. Remember? We saw that at the museum.”

In contrast, when the family was watching football—a domain the boy does not yet know much about—he asked “Why did they knock that guy down?” The mother’s answer was short, simple, stripped of domain-specific vocabulary, and sketchy with respect to causal mechanisms—“Because that’s what you do when you play football.” Parents have a fairly good sense of what their children know and, often, they gear their answers to an appropriate level. When talking about one of the child’s islands of expertise, parents can draw on their shared knowledge base to construct a more elaborate, accurate, and meaningful explanations. This is a common characteristic of conversation in general: When we share domain-relevant experience with our audience we can use accurate terminology, construct better analogies, and rely on mutually held domain- appropriate schema as a template through which we can scribe new causal connections.

As this chapter is being written, the boy in this story is now well on his way to 4- years old. Although he still likes trains and still knows a lot about them, he is developing other islands of expertise as well. As his interests expand, the boy may engage less and less often in activities and conversations centered around trains and some of his current domain-specific knowledge will atrophy and eventually be lost. But as that occurs, the domain-general knowledge that connected the train domain to broader principles, mechanisms, and schemas will probably remain. For example, when responding to the boy’s comment about the tea kettle, the mother used the train domain as a platform to talk about the more general phenomenon of steam.

Trains were platforms for other concepts as well, in science and in other domains. Conversations about mechanisms of locomotion have served as a platform for a more general understanding of mechanical causality. Conversations about the motivation of characters in the Thomas the Tank Engine stories have served as platforms for learning about interpersonal relationships and, for that matter, about the structure of narratives. Conversations about the time when downtown Pittsburgh was threaded with train tracks and heavy-duty railroad bridges served as a platform for learning about historical time and historical change. These broader themes emerged for the boy for the first time in the context of train conversations with his parents. Even as the boy loses interest in trains and moves on to other things, these broader themes remain and expand outward to connect with other domains he encounters as he moves through his everyday life.

Expertise research is also relevant; it talks about how people build up increasingly detailed mental representations of a domain they are learning, and which guide them when they decide what actions to take. The representations start out a coarse, but get increasingly detailed over time. This is an excerpt from the book Peak: Secrets from the New Science of Expertise, which notes a "chicken and egg" problem that requires grounding any skill in some pre-existing knowledge first:

As we’ve just seen from several studies, musicians rely on mental representations to improve both the physical and cognitive aspects of their specialties. And mental representations are essential to activities we see as almost purely physical. Indeed, any expert in any field can be rightly seen as a high-achieving intellectual where that field is concerned. This applies to pretty much any activity in which the positioning and movement of a person’s body is evaluated for artistic expression by human judges. Think of gymnastics, diving, figure skating, or dancing. Performers in these areas must develop clear mental representations of how their bodies are supposed to move to generate the artistic appearance of their performance routines. But even in areas where artistic form is not explicitly judged, it is still important to train the body to move in particularly efficient ways. Swimmers learn to perform their strokes in ways that maximize thrust and minimize drag. Runners learn to stride in ways that maximize speed and endurance while conserving energy. Pole-vaulters, tennis players, martial artists, golfers, hitters in baseball, three-point shooters in basketball, weightlifters, skeet shooters, and downhill skiers—for all of these athletes proper form is key to good performance, and the performers with the best mental representations will have an advantage over the rest. 
In these areas too, the virtuous circle rules: honing the skill improves mental representation, and mental representation helps hone the skill. There is a bit of a chicken-and-egg component to this. Take figure skating: it’s hard to have a clear mental representation of what a double axel feels like until you’ve done it, and, likewise, it is difficult to do a clean double axel without a good mental representation of one. That sounds paradoxical, but it isn’t really. You work up to a double axel bit by bit, assembling the mental representations as you go. 
It’s like a staircase that you climb as you build it. Each step of your ascent puts you in a position to build the next step. Then you build that step, and you’re in a position to build the next one. And so on. Your existing mental representations guide your performance and allow you to both monitor and judge that performance. As you push yourself to do something new—to develop a new skill or sharpen an old one—you are also expanding and sharpening your mental representations, which will in turn make it possible for you to do more than you could before.

Finally, while it only touches on this topic occasionally, Josh Waitzkin had some nice "from the inside" descriptions of this gradual construction of an increasing level of understanding in his book The Art of Learning:

I practiced the Tai Chi meditative form diligently, many hours a day. At times I repeated segments of the form over and over, honing certain techniques while refining my body mechanics and deepening my sense of relaxation. I focused on small movements, sometimes spending hours moving my hand out a few inches, then releasing it back, energizing outwards, connecting my feet to my fingertips with less and less obstruction. Practicing in this manner, I was able to sharpen my feeling for Tai Chi. When through painstaking refinement of a small movement I had the improved feeling, I could translate it onto other parts of the form, and suddenly everything would start flowing at a higher level. The key was to recognize that the principles making one simple technique tick were the same fundamentals that fueled the whole expansive system of Tai Chi Chuan.
This method is similar to my early study of chess, where I explored endgame positions of reduced complexity—for example king and pawn against king, only three pieces on the board—in order to touch high-level principles such as the power of empty space, zugzwang (where any move of the opponent will destroy his position), tempo, or structural planning. Once I experienced these principles, I could apply them to complex positions because they were in my mental framework. However, if you study complicated chess openings and middlegames right off the bat, it is difficult to think in an abstract axiomatic language because all your energies are preoccupied with not blundering. It would be absurd to try to teach a new figure skater the principle of relaxation on the ice by launching straight into triple axels. She should begin with the fundamentals of gliding along the ice, turning, and skating backwards with deepening relaxation. Then, step by step, more and more complicated maneuvers can be absorbed, while she maintains the sense of ease that was initially experienced within the simplest skill set.

Comment by kaj_sotala on How much background technical knowledge do LW readers have? · 2019-07-11T18:30:47.673Z · score: 13 (7 votes) · LW · GW

Nice survey. The gap between "one course or played around a bit" and "undergrad major" in some questions felt pretty big - an "undergrad minor" option would have been nice.

(I assumed that these corresponded to roughly the amounts that I'm used to, with an undergrad major being something like 60 ECTS credits worth of courses specifically in that major, and a minor 25.)

Comment by kaj_sotala on Self-consciousness wants to make everything about itself · 2019-07-11T09:55:10.394Z · score: 5 (2 votes) · LW · GW

To rephrase:

  • I suspect that typically, when someone thinks "I shouldn't do X because that would make me evil and I shouldn't be evil", what their brain is actually computing is something like "I shouldn't do X because people who I care about would then consider me evil, and that would be bad for me", even if that's not how it appears subjectively.
    • Note that this is different from thinking that "I shouldn't do X because that's wrong". I'm not claiming that a fear of punishment would be the only thing that disinclined us from doing bad things. You can consider X to be wrong and refrain from doing it, without having a subconscious fear of punishment. The difference is that if you are afraid of being punished, then any suggestion of having done wrong is likely to trigger a defensive reaction and a desire to show that you did no wrong. Whereas if you don't feel the fear, you might just think "oh, the thing that I did was wrong", feel healthy regret, and act to fix it.
  • Sometimes you might feel like you are evil (that is, likely to be punished) because you have done, or repeatedly do, something bad. In some of these cases, it can help to redefine your concepts. For instance, if you did bad things when you were young, you can think "I did bad things when I was young and didn't know any better, but I'm a grownup now". Or "these things are evil, but nobody is actually purely good, so doing them doesn't make me any worse than other people".
  • Now, if "I shouldn't be evil" actually means "I shouldn't do things that make people punish me", then attempting these kinds of redefinitions will also trigger a subconscious evaluation process which has to be passed for the redefinition to succeed.
    • If you think that "I did bad things when I was young and didn't know any better, but I'm a grownup now", then this corresponds to something like "I did bad things when I was young, but nobody is going to hold those things against me anymore". This redefinition will only succeed if the latter sentence feels true to your brain.
    • If you think that "these things are evil, but nobody is actually purely good, so doing them doesn't make me any worse than other people", then this corresponds to something like "these things are blameworthy, but everyone does them, so doing them won't get me judged any more harshly than anyone else". Again, this redefinition will only succeed if the latter sentence feels true to your brain.
  • This is assuming that you are trying to use those redefinitions to heal the original bad feeling. One can also use these kinds of thoughts to suppress the original bad feeling (create a protector which seeks to extinguish the bad feeling using counter-arguments). In that case, the part of your mind which was originally convinced of you being in danger, doesn't need to be convinced otherwise. But this will set up an internal conflict as that part will continue to try to make itself heard, and may sometimes overwhelm whatever blocks have been put in place to keep in silent.

Related: Scott Alexander's Guilt: Another Gift Nobody Wants.

Comment by kaj_sotala on The AI Timelines Scam · 2019-07-11T09:20:40.787Z · score: 20 (7 votes) · LW · GW
I also read OP as claiming that Yann LeCun is defending the field against critiques that AGI isn’t near.

Same. In particular, I read the "How does the AI field treat its critics" section as saying that "the AI field used to criticize Dreyfus for saying that AGI isn't near, just as it now seems to criticize Marcus for saying that AGI isn't near". But in the Dreyfus case, he was the subject of criticism because the AI field thought that he was wrong and AGI was close. Whereas Marcus seems to be the subject of criticism because the AI field thinks he's being dishonest in claiming that anyone seriously thinks AGI to be close.

Comment by kaj_sotala on Are we certain that gpt-2 and similar algorithms are not self-aware? · 2019-07-11T09:01:59.258Z · score: 15 (9 votes) · LW · GW

Yes, we are.

Wikipedia defines self-awareness as "the capacity for introspection and the ability to recognize oneself as an individual separate from the environment and other individuals". At a minimum, this would require GPT-2 to have a model of the world which included a representation of itself. To be similar to our intuitive understanding of self-awareness, that representation would also need to guide its decision-making and thought in some significant way.

Here is an intuitive explanation of the Transformer architecture that GPT-2 is based on. You can see from the explanation that it's only modeling language; there's no self-representation involved.

Technically, I guess you could say that, if a Transformer architecture was trained on texts which talked about Transformer architectures, it would get a model which did include a representation of itself. But that would be just another data token, which the system gave no special significance to, and which wouldn't guide its behavior any more than any other piece of data.

Comment by kaj_sotala on Jimrandomh's Shortform · 2019-07-09T12:50:32.584Z · score: 12 (3 votes) · LW · GW
Building more highway lanes will cause more people to drive (induced demand), so building more lanes won't fix traffic.

Is this really fallacious? I'm asking because while I don't know the topic personally, I have some friends who are really into city planning. They've said that this is something which is pretty much unambiguously accepted in the literature, now that we've had the time to observe lots and lots of failed attempts to fix traffic by building more road capacity.

A quick Googling seemed to support this, bringing up e.g. this article which mentions that:

In this paper from the Victoria Transport Policy Institute, author Todd Litman looks at multiple studies showing a range of induced demand effects. Over the long term (three years or more), induced traffic fills all or nearly all of the new capacity. Litman also modeled the costs and benefits for a $25 million line-widening project on a hypothetical 10-kilometer stretch of highway over time. The initial benefits from congestion relief fade within a decade.
Comment by kaj_sotala on Instead of "I'm anxious," try "I feel threatened" · 2019-07-08T06:50:06.072Z · score: 3 (1 votes) · LW · GW

You may be interested in my article about IFS, as well as this recent talk by its developer which I think gives a good overview of the model.

Comment by kaj_sotala on Self-consciousness wants to make everything about itself · 2019-07-07T21:41:01.263Z · score: 11 (5 votes) · LW · GW

I tend to (or at least try to remember to) draw on NVC a lot in conversation about emotionally fraught topics, but it's hard for me to recall the specifics of my wording afterwards, and they often include a lot of private context. But here's an old example which I picked because it happened in a public thread and because I remember being explicitly guided by NVC principles while writing my response.

As context, I had shared a link to an SSC post talking about EA on my Facebook page, and some people were vocally critical of its implicit premises (e.g. utilitarianism). The conversation got a somewhat heated tone and veered into, among other things, the effectiveness of non-EA/non-utilitarian-motivated volunteer work. I ended making some rather critical comments in return, which led to the following exchange:

The other person: Kaj, I also find you to be rather condescending (and probably, privileged) when talking about volunteer work. Perhaps you just don't realize how large a part volunteers play in doing the work that supposedly belongs to the state, but isn't being taken care of. And your suggestion that by volunteering you mainly get to feel better about yourself, and then "maybe help some people in the process", is already heading towards offensive. I have personally seen/been involved in/know of situations where a volunteer literally saved someone's life, as in, if they weren't there and didn't have the training they had, the person would most likely have died. So, what was your personal, verified lives saved per dollar efficiency ratio, again, that allows you to make such dismissive comments?
Me: I'm sorry. I was indeed condescending, and that was wrong of me. I was just getting frustrated because I felt that Scott's post had a great message about what people could accomplish if they just worked together, which was getting completely lost squabbling over a part of it that was beside the actual point. When I saw you say call lives saved per dollar an absolutely terrible metric, I felt hurt because I felt that the hard and thoughtful work the EA scene has put into evaluating different interventions was being dismissed without being given a fair evaluation. I felt a need to defend them - any myself, since I'm a part of that community - at that point.
I do respect anyone doing volunteer work, especially the kind of volunteer work you're referring to. When I was writing my comment, I was still thinking of activism primarily in the context of Scott's original post (and the post of his preceding it), which had been talking about campaigns to (re)post things on Twitter, Tumblr, etc. These are obviously very different campaigns from the kind of work that actually saves lives.
Even then, you are absolutely correct about the fact that my remarks have often unfairly devalued the value of volunteer work. So to set matters straight: anyone who voluntarily works to make other's lives better has my utmost respect. Society and the people in it would be a lot worse off without such work, and I respect anyone who actually does important life-saving work much more than I respect, say, someone who hangs around in the EA community and talks about pretty things without actually doing much themselves. (I will also admit that I've kinda fallen to the latter category every now and then.)
I also admit that the people in the field have a lot of experience which I do not have, and that academic analyses about the value of some intervention have lots of limitations that may cause them to miss out important things. I do still think that systematic, peer-reviewed studies are the way to go if one wishes to find the most effective ways of making a difference, but I do acknowledge that the current level of evidence is not yet strong and that there are also lots of individual gut feelings and value judgement involved in interpreting the data.
I don't want there to be any us versus them thing going on, with EAs on the one side and traditional volunteers on the other. I would rather have us all be on the same side, where we may disagree on what is the best way to act, but support each other regardless and agree that we have a shared goal in making the world a better place.

(The conversation didn't proceed anywhere from that, but the other person did "like" my comment.)

Comment by kaj_sotala on Self-consciousness wants to make everything about itself · 2019-07-07T07:43:18.106Z · score: 14 (4 votes) · LW · GW

I think NVC has the thing where, if it's used well, it's subtle enough that you don't necessarily recognize it as NVC.

Comment by kaj_sotala on Self-consciousness wants to make everything about itself · 2019-07-03T08:32:36.229Z · score: 16 (8 votes) · LW · GW

"Feeling guilty/ashamed, then lashing out in order to feel better" is a good example of an exile/protector dynamic. It may be possible to reduce the strength of such reactions through an intellectual argument, as you seem to suggest yourself to have done. But as Ruby suggested, the underlying insecurity causing the protector response is likely to be related to some fear around social rejection, in which case working on the level of intellectual concepts may be insufficient for changing the responses.

I suspect that the "nobody is good" move, when it works, acts on the level of self-concept editing, changing your concepts so as to redefine the "evil" act to no longer be evidence about you being evil. But based on my own experience, this move only works to heal the underlying insecurity when your brain considers the update to be true - which is to say, when it perceives the "evil act" to be something which your social environment will no longer judge you harshly for, allowing the behavior to update. If this seems false, or the underlying insecurity is based on a too strong of a trauma, then one may need to do more emotional kind of work in order to address it.

That said, one may adjust the problematic response pattern even without healing the original insecurity: e.g. sufficiently strong external pressure saying that the response is bad, may set up new mechanisms which suppress the-responses-which-have-been-judged-as-bad. But this may involve building a higher tower of protectors, meaning that healing the original insecurity would allow for more a flexible range of behavior.

Incidentally, I don't think that tone arguments are always made in bad faith. E.g. within EA, there have been debates over "should we make EA arguments which are guilt/obligation-based", which is basically an argument over the kind of a tone to use. Now, I agree that tone arguments often are themselves a protector reaction, aimed at just eliminating the unpleasant tone. But there also exists a genuine disagreement over what kind of a tone is more effective, and there exist lots of people who are genuinely worried that groups using too harsh of a tone to create social change are shooting themselves in the foot.

My guess is that people arguing for a "nice" tone are doing so because of an intuitive understanding that a harsh tone is more likely to trigger protector responses and thus be counterproductive. People may even be trying to communicate that it would be easier for them personally to listen if their protector responses weren't being triggered. On the other hand, people arguing for a harsh tone are doing so because of an intuitive understanding that the "nice" tone is often too easily ignored, and that making people feel guilty over the defensive reaction in the first place is also a strategy which can be made to work. This seems to be an area where different people will react differently to the same strategies, causing people to be inclined to over-generalize from their own experience.

Comment by kaj_sotala on Everybody Knows · 2019-07-02T21:23:39.471Z · score: 13 (8 votes) · LW · GW

Saying that everybody knows is almost never a mistake. The statement isn’t sloppy reasoning. It’s a strategy that aims to cut off discussion or objection, to justify fraud and deception, and to establish truth without evidence.

This isn't my experience; I usually find people to use the expression "everyone X" out of frustration, in situations when they really do feel that everyone X, and can't believe that the other person thinks that not-X is normal or reasonable.

Comment by kaj_sotala on How to deal with a misleading conference talk about AI risk? · 2019-06-29T05:29:06.870Z · score: 6 (3 votes) · LW · GW

Thanks, those specifics are great!

Comment by kaj_sotala on How to deal with a misleading conference talk about AI risk? · 2019-06-28T09:45:57.361Z · score: 10 (4 votes) · LW · GW
But I mostly wouldn't expect both concerns to be operative in the same world—in the possible world where Sussman feels bad about being named and singled out, that means he's taking "us" seriously enough for our curt dismissal to hurt, but in the possible world where we're written off as cranks, then being named and singled out doesn't hurt.

The world can change as a result of one of the concerns. At first you're taking someone seriously (or might at least be open to taking them seriously), then they say something hurtful, then you write them off to make it hurt less. Sour grapes.

Also, the reactions of people who are not being directly criticized but who respect the person being criticized are also important. Even if the target of the criticism never saw it, other people in the target's peer group may also feel disrespected and react in a similar way. (This is not speculation - I've seen various computer scientists have this reaction to writings on LW, many times.)

Comment by kaj_sotala on How to deal with a misleading conference talk about AI risk? · 2019-06-27T21:48:37.126Z · score: 3 (1 votes) · LW · GW

Hi rmoehn,

I didn't look at the contents of that talk yet, but I felt uncomfortable about a specific speaker/talk being named and singled out for the target of rather hard-to-respond-to criticism (consider how you might take it if you came across a forum discussion calling your talk misleading and not well-reasoned, without going into any specifics), so I edited out those details for now.

I feel that the AI risk community should do its best to build friendly rather than hostile relationships with mainstream computer science researchers. In particular, there have been cases before where researchers looked at how their work was being discussed on LW, picked up a condescending tone, and decided that LW/AI risk people were not worth engaging with. Writing a response outlining one's disagreement to the talk (in the style of e.g. "Response to Cegłowski on superintelligence") wouldn't be a problem as it communicates engagement with the talk. But if we are referencing people's work in a manner which communicates a curt dismissal, I think we should be careful about naming specific people.

The question in general is fine, though. :)

Comment by kaj_sotala on Causal Reality vs Social Reality · 2019-06-25T06:45:51.656Z · score: 21 (4 votes) · LW · GW

Previously on this topic: Of Two Minds, A Tale of Two Tradeoffs.

Comment by kaj_sotala on Recommendation Features on LessWrong · 2019-06-23T18:24:44.314Z · score: 3 (1 votes) · LW · GW

The home page now has a From the Archives section, which displays three posts randomly selected from the entire history of LessWrong. Currently, a post can appear in this section if:

(1) you've never read it while logged in (including on old-LessWrong),

This seems bugged; I'm frequently recommended posts that I've already read, and in some cases even voted on (I was just recommended https://www.lesswrong.com/posts/6hfGNLf4Hg5DXqJCF/a-fable-of-science-and-politics , which I know I've read multiple times, and which showed an upvote right when I opened the page).

Comment by kaj_sotala on Integrating disagreeing subagents · 2019-06-12T05:46:45.231Z · score: 4 (2 votes) · LW · GW
do you think it is clear there is something more like Turing-complete webs of strategy within subagents vs merely pseudostrategy ?

I don't know. As suggested by this post, I move pretty freely between the subagent framing and the "associative belief structure" framing as seems appropriate to the situation. To me agentness doesn't necessarily require the agents to be particularly strategic. (A thermostat is technically an agent, but not a very strategic one.)

IFS calls subagents just "parts", which I prefer in some contexts; it has fewer connotations of being particularly strategic.

Comment by kaj_sotala on On pointless waiting · 2019-06-12T05:39:18.845Z · score: 3 (1 votes) · LW · GW

Yeah, I've been thinking a lot about that post. Having read it might have been a part of what helped me notice myself doing this.

Comment by kaj_sotala on On pointless waiting · 2019-06-12T05:38:24.602Z · score: 3 (1 votes) · LW · GW

Depends on how motivated you are to do your job, I guess. If you're only doing your job to get paid, then it's as you describe.

On pointless waiting

2019-06-10T08:58:56.018Z · score: 44 (21 votes)
Comment by kaj_sotala on For the past, in some ways only, we are moral degenerates · 2019-06-08T17:55:54.456Z · score: 14 (4 votes) · LW · GW
If we project out in the future, the first scenario posits continuing increased moral improvements (as the "improvement trend" continues) and the second posits moral degeneration (as the values drift away from our own).

Someone might consider this semantics, but doesn't the "values drift around" model imply that there is neither progress nor degeneration for the values in question, since it's just random drift?

In other words, if there is the possibility of progress in some values, then that implies that some values are better or worse than ours; and if others just show random drift and there's no meaningful "progress" to speak of, then those values drifting away from ours doesn't make them any worse than ours, it just makes them different.

I realize that you did explicitly define "degeneration = moving away from ours" for the drifting values, but it feels weird to then also define "progress = moving away from ours in a good way" for the progress-y values; it feels like you are trying the operation defined for the domain of progress-y values in a domain which it isn't applicable for.

Comment by kaj_sotala on Habryka's Shortform Feed · 2019-06-02T15:45:02.713Z · score: 3 (1 votes) · LW · GW

Thirded.

Comment by kaj_sotala on FB/Discord Style Reacts · 2019-06-02T15:42:49.615Z · score: 9 (5 votes) · LW · GW

"That clarifies it to my satisfaction, thanks for explaining."

(I often see something like 👌 or ✔️being used for this on Discord/Slack, though those are more about acknowledging the explanation rather than the expression of gratitude.)

Comment by kaj_sotala on FB/Discord Style Reacts · 2019-06-02T15:34:35.253Z · score: 8 (5 votes) · LW · GW

My experience from seeing emoticons used on Slack/Discord is that they help combat the muted signal problem of online communication, and thus actually reduce the toxicity of discussion.

People want to feel respected, loved, appreciated, etc. When we interact physically, you can easily experience subtle forms of these feelings. For instance, even if you just hang out in the same physical space with a bunch of other people and don’t really interact with them, you often get some positive feelings regardless. Just the fact that other people are comfortable having you around, is a subtle signal that you belong and are accepted.
Similarly, if you’re physically in the same space with someone, there are a lot of subtle nonverbal things that people can do to signal interest and respect. Meeting each other’s gaze, nodding or making small encouraging noises when somebody is talking, generally giving people your attention. This kind of thing tends to happen automatically when we are in each other’s physical presence.
Online, most of these messages are gone: a thousand people might read your message, but if nobody reacts to it, then you don’t get any signal indicating that you were seen. Even getting a hundred likes and a bunch of comments on a status, can feel more abstract and less emotionally salient than just a single person nodding at you and giving you an approving look when you’re talking.

My hypothesis is that the lack of such subtle "body language" signals is often the cause of toxicity, because generic likes feel more abstract and then people ramp up their outrage in order to elicit reactions and feel like they have been noticed. Whereas, in my own experience at least, the kind of somewhat personalized (in the sense of having been selected from a vast set) reacts on Slack/Discord do have some of the "body language felt sense" that makes me feel genuinely noticed in a way that a mere upvote doesn't. (And the point of the signals being subtle is that they're not major enough that you would normally bother expressing them in a comment.)

Comment by kaj_sotala on What is required to run a psychology study? · 2019-05-30T21:42:36.916Z · score: 5 (2 votes) · LW · GW

If you want to publish it formally, the journal may impose its own requirements. E.g. back when Facebook did a formal study on their users, they appealed to the users having consented to A/B testing when they accepted Facebook's TOS. Afterwards, several researchers argued that this broke the rules for informed consent, with one paragraph in the linked article suggesting that the paper might end up retracted by the publisher:

When asked whether the study had had an ethical review before being approved for publication, the US National Academy of Sciences, which published the controversial paper in its Proceedings of the National Academy of Sciences (PNAS), told the Guardian that it was investigating the issue.

(I don't recall hearing what the results of that investigation were, but I don't think it was ever retracted.)

Comment by kaj_sotala on mAIry's room: AI reasoning to solve philosophical problems · 2019-05-24T10:14:21.628Z · score: 5 (4 votes) · LW · GW
The valuable contribution here is not the formalisation, but the generator behind the formalisation.

I disagree; the "core idea" I'd already thought of before seeing the post, but the valuable contribution to me was seeing why the core idea has to be true and how it works mechanically, rather than being just a plausible-seeming sentence. Technical explanation vs. verbal explanation.

Comment by kaj_sotala on Open Thread May 2019 · 2019-05-24T08:42:07.141Z · score: 11 (2 votes) · LW · GW

My impression is that Chapman is objecting to a different kind of rationality, which he defines in a more specific and narrow way. At least, on several times in conversation when I've objected to him that LW-rationality already takes into account postrationality, he has responded with something like "LW-rationality has elements of what I'm criticizing, but this book is not specifically about LW-rationality".

Comment by kaj_sotala on Open Thread May 2019 · 2019-05-24T08:31:41.921Z · score: 13 (4 votes) · LW · GW

I'm generally sympathetic to the postrationalists on several points, but I agree with this. Coming up with the whole postrationality label was a bad idea in the first place, as it tends to serve tribal purposes and distracts people from the ways in which postrationality is just a straightforward extension of standard rationality.

(Ironically, one could argue that I already fit the criteria of "postrationalist", making it slightly weird for me to say that I'm sympathetic to the postrationalists, rather than being one of them. But to some extent I don't want to identify as post-rat, exactly because I don't think that post-rat is a good term and that good rat is already post-rat.)

Comment by kaj_sotala on Beware Social Coping Strategies · 2019-05-21T20:57:11.839Z · score: 3 (1 votes) · LW · GW

Seconded.

Comment by kaj_sotala on Integrating disagreeing subagents · 2019-05-16T15:38:19.838Z · score: 3 (1 votes) · LW · GW

Interestingly, was just reading a paper from DeepMind which talked about deep reinforcement learning systems learning better if they are supplemented with an episodic memory store which maintains a store of all previous situations. Upon re-encountering a similar situation as one that was encountered in the past, the neural net is restored to a similar state as it was in before:

In episodic meta-RL, meta-learning occurs within a recurrent neural network, as described in the previous section and Box 3. However, superimposed on this is an episodic memory system, the role of which is to reinstate patterns of activity in the recurrent network. As in episodic deep RL, the episodic memory catalogues a set of past events, which can be queried based on the current context. However, rather than linking contexts with value estimates, episodic meta-RL links them with stored activity patterns from the recurrent network's internal or hidden units. These patterns are important because, through meta-RL, they come to summarize what the agent has learned from interacting with individual tasks (see Box 3 for details). In episodic meta- RL, when the agent encounters a situation that appears similar to one encountered in the past, it reinstates the hidden activations from the previous encounter, allowing previously learned information to immediately influence the current policy. In effect, episodic memory allows the system to recognize previously encountered tasks, retrieving stored solutions.
Through simulation work in bandit and navigation tasks, Ritter et al. [39] showed that episodic meta-RL, just like ‘vanilla’ meta-RL, learns strong inductive biases that enable it to rapidly solve novel tasks. More importantly, when presented with a previously encountered task, episodic meta-RL immediately retrieves and reinstates the solution it previously discovered, avoiding the need to re-explore. On the first encounter with a new task, the system benefits from the rapidity of meta-RL; on the second and later encounters, it benefits from the one-shot learning ability conferred by episodic control. [...]
Equally direct links connect episodic meta-RL with psychology and neuroscience. Indeed, the reinstatement mechanism involved in episodic meta-RL was directly inspired by neuroscience data indicating that episodic memory circuits can serve to reinstate patterns of activation in cerebral cortex, including areas supporting working memory (see [40]). Ritter and colleagues [39] (S. Ritter, PhD Thesis, Princeton University, 2019) show how such a function could itself be configured through RL, giving rise to a system that can strategically reinstate information about tasks encountered earlier (see also 50, 51, 52).

This would fit together with the thing about memory reconsolidation being key to adjusting all subagents (if a subagent is something like a memory pattern coding for a specific situation), as well otherwise fitting with a lot of data about memory change being key to this kind of thing.

Then again, H.M. could learn new skills despite being unable to learn new episodic memories...

Comment by kaj_sotala on Integrating disagreeing subagents · 2019-05-16T10:54:19.555Z · score: 10 (6 votes) · LW · GW

I'm not sure if I'd treat "different heuristics in different domains" as an example of non-integration. At least it feels different from the inside. If someone points out to me that I'm not applying a programming heuristic when dealing with humans, I'm likely to react by "well that's because I'm dealing with humans not code", rather than noticing something that feels like a contradiction.

A contradiction feels more like having the heuristics (when X, do A) and (when Y, do not-A), and it then being pointed out to me that actually in this situation, X and Y and both apply.

I'm reminded of this excerpt from a recent paper, Holistic Reinforcement Learning: The Role of Structure and Attention (Trends in Cognitive Sciences, Angela Radulescu, Yael Niv & Ian Ballard 2019; Sci-hub version):

Bayesian nonparametric models group perceptual observations into unobserved ‘latent causes’ (or clusters) [52–55]. For example, consider a serial reversal learning task in which the identity of the high-reward option sporadically alternates. In such tasks, animals initially learn slowly but eventually learn to respond rapidly to contingency changes [56]. Bayesian nonparametric models learn this task by grouping reward outcomes into two latent causes: one in which the first option is better and one in which the second option is better. Once this structure is learned, the model displays one-shot reversals after contingency changes because it infers that the latent cause has changed. This inference about latent causes in the environment has also shed light on several puzzling conditioning effects. When presented with a neutral stimulus such as a tone followed by a shock, animals eventually display a fear response to the tone. The learned fear response gradually diminishes when the tone is later presented by itself (i.e., in extinction) but often returns after some time has passed. This phenomenon is known as spontaneous recovery. Bayesian nonparametric models attribute spontaneous recovery to the inference that extinction signals a new environmental state. This prevents old associations from being updated [57]. Bayesian nonparametric models also predict that gradual extinction will prevent spontaneous recovery, a finding borne out by empirical data [57]. In gradual extinction, the model infers a single latent state and gradually weakens the association between that state and aversive outcome, thereby abolishing the fear memory.
Comment by kaj_sotala on The Relationship Between the Village and the Mission · 2019-05-15T14:09:47.823Z · score: 11 (5 votes) · LW · GW
My actual best guess is that the village should be oriented around truthseeking and the mission oriented around [truthseeking and] impact.

John Tooby has suggested that whatever becomes the orienting thing of a community, becomes automatically the subject of mind-killing impulses:

Coalition-mindedness makes everyone, including scientists, far stupider in coalitional collectivities than as individuals. Paradoxically, a political party united by supernatural beliefs can revise its beliefs about economics or climate without revisers being bad coalition members. But people whose coalitional membership is constituted by their shared adherence to “rational,” scientific propositions have a problem when—as is generally the case—new information arises which requires belief revision. To question or disagree with coalitional precepts, even for rational reasons, makes one a bad and immoral coalition member—at risk of losing job offers, one's friends, and one's cherished group identity. This freezes belief revision.  
Forming coalitions around scientific or factual questions is disastrous, because it pits our urge for scientific truth-seeking against the nearly insuperable human appetite to be a good coalition member. Once scientific propositions are moralized, the scientific process is wounded, often fatally.  No one is behaving either ethically or scientifically who does not make the best case possible for rival theories with which one disagrees. 

I wouldn't go so far as to say that this makes truthseeking a bad idea to orient around, since there does seem to be a way to orient around it in a way which avoids this failure mode, but at least one should be very cautious about how exactly.

If I think of the communities which I've seen that seem to have successfully oriented around truthseeking to some extent, the difference seems to be something like a process vs. content distinction. People aren't going around explicitly swearing allegiance to rationality, but they are constantly signaling a truthseeking orientation through their behavior, such as by actively looking for other people's cruxes in conversation and indicating their own.

Comment by kaj_sotala on Integrating disagreeing subagents · 2019-05-15T13:58:55.861Z · score: 3 (1 votes) · LW · GW

I should finally get around reading that book, thanks for continuing to remind me about it. :-)

I can see how memory reconsolidation would apply for some of the processes. But how would it be involved when you encounter a novel decision, like Eliezer's save-a-life dilemma, where there is presumably no previous memory to reconsolidate?

Comment by kaj_sotala on mAIry's room: AI reasoning to solve philosophical problems · 2019-05-14T16:43:04.339Z · score: 8 (4 votes) · LW · GW

Curated. After reading this post, I've find my thoughts frequently reusing its concepts, and feel like it has significantly clarified my understanding of qualia as a physicalist concept. Mary's room feels pretty strongly dissolved in my mind, and in a very elegant and concise form.

While I referenced this only relatively briefly in my post on integrating subagents, the way this post considerably weakened the distinction between receiving sense data and modifying your values on a conceptual level felt like an important additional element in my of understanding why people/subagents might want to resist belief updates.

Integrating disagreeing subagents

2019-05-14T14:06:55.632Z · score: 82 (19 votes)
Comment by kaj_sotala on Complex Behavior from Simple (Sub)Agents · 2019-05-11T08:05:31.789Z · score: 14 (5 votes) · LW · GW

Really cool! This reminds me of Braitenberg vehicles.

I had a notion here that I could stochastically introduce a new goal that would minimize total suffering over an agent's life-history. I tried this, and the most stable solution turned out to be thus: introduce an overwhelmingly aversive goal that causes the agent to run far away from all of its other goals screaming. Fleeing in perpetual terror, it will be too far away from its attractor-goals to feel much expected valence towards them, and thus won't feel too much regret about running away from them. And it is in a sense satisfied that it is always getting further and further away from the object of its dread.

Interestingly, this seems somewhat similar to the reactions of severely traumatized people, whose senses partially shut down to make them stop feeling or wanting anything. And then there's also suicide for when the "avoid suffering" goal grows too strong relative to the other ones. For humans there's a counterbalancing goal of avoiding death, but your agents didn't have an equivalent balancing desire to stay "alive" (or within reach of their other goals).

Comment by kaj_sotala on Open Thread March 2019 · 2019-04-26T10:56:30.155Z · score: 3 (1 votes) · LW · GW

IME, the need to make a sustainable living is a big reason for why people can't fix those individualist problems. At least in my social circles, there are plenty of people who have all kinds of communal projects and would want to spend time doing meaningful things with people who are important to them... but none of those things bring a living, so instead they have to burn most of their time and energy earning money and have much less left that they could use to build a more communal society.

It's no wonder that people have few friends outside work when work and family combined leave little time for anything else. NYT on why it's hard to make friends after 30:

As external conditions change, it becomes tougher to meet the three conditions that sociologists since the 1950s have considered crucial to making close friends: proximity; repeated, unplanned interactions; and a setting that encourages people to let their guard down and confide in each other, said Rebecca G. Adams, a professor of sociology and gerontology at the University of North Carolina at Greensboro. This is why so many people meet their lifelong friends in college, she added.

In the professional world, “proximity” is hard to maintain, as work colleagues are reassigned or move on to new jobs. Last year, Erica Rivinoja, a writer on the NBC series “Up All Night,” became close with a woman, Jen, when they worked together on a pilot. Almost instantly, they knew each other’s exercise schedules and food preferences. Jen could sense when Ms. Rivinoja needed a jolt of caffeine, and without asking would be there with an iced tea.

“But as soon as the pilot was over, it was hard to be as close without that constant day-to-day interaction,” said Ms. Rivinoja, 35. They can occasionally carve out time for a quick gin and tonic, she said, but “there aren’t those long afternoons which bleed into evenings hanging out at the beach and then heading to a bar.”

The workplace can crackle with competition, so people learn to hide vulnerabilities and quirks from colleagues, Dr. Adams said. Work friendships often take on a transactional feel; it is difficult to say where networking ends and real friendship begins.

Also Qiaochu Yuan:

Here is another world I’ve been seeing the possibility of increasingly clearly lately. The most important feature of this world is that you have a tribe to whom you’re securely attached. You love and support each other. You touch each other. You sing and dance together. And sometimes, some of you explore romantic / sexual connection with each other. And if that gets rocky – when someone gets anxious or avoidant or some other kind of triggered – the attachment that the people involved have with everyone else in the tribe acts as a stabilizing and calming force. If your attachment to your tribe is secure enough, the prospect of a partner leaving you maybe feels less like the end of the world.

(And sometimes, some of you have children, and those children are raised by a tribe of people who are lovingly stabilizing and calming each other, instead of being at the mercy of a fragile little tribe of two…)

It hurts to think about this world, and how far away from it most people are. There are so many forces pushing against it: high school friends going to different colleges, college friends taking jobs in different cities, friends moving into their own apartments, couples living by themselves, the crushing burdens of late-stage capitalism… and, among so many other things, some sense that it’s a little weird to allow your friends to matter to you as much as or more than your partners.

A basic income wouldn't fix all of this, but if it would at least allow people to refuse taking on the kinds of meaningless bullshit jobs that suck your energy dry, then that would help a little. Not being forced to prioritize a job over community would be a great start.

Comment by kaj_sotala on Strategic implications of AIs' ability to coordinate at low cost, for example by merging · 2019-04-25T06:31:34.918Z · score: 14 (5 votes) · LW · GW

Humans tend to highly value their own personal experiences - getting to do things that feel fun, acquiring personal status, putting intrinsic value on their own survival, etc. This limits the extent to which they'll co-operate with a group, since their own interests and the interests of the group are only partially the same. AIs with less personal interests would be better incentivized to coordinate - if you only care about the amount of paperclips in the universe, you will be able to better further that goal with others than if each AI was instead optimizing for the amount of paperclips that they personally got to produce.

Some academics argue that religion etc. evolved for the purpose of suppressing personal interests and giving them a common impersonal goal, partially getting around this problem. I discussed this and its connection to these matters a bit in my old post Intelligence Explosion vs. Co-operative Explosion.

Comment by kaj_sotala on AI Alignment Problem: “Human Values” don’t Actually Exist · 2019-04-22T20:33:48.842Z · score: 8 (4 votes) · LW · GW

could you explain how we can resolve conflicts between two terminal values, if terminal means irreducible?

Suppose the following mind architecture:

  • When in a normal state, the mind desires games.
  • When the body reports low blood sugar levels, the mind desires food.
  • When in danger, the mind desires running away.
  • When in danger AND with low blood sugar levels, the mind desires freezing up.

Something like this has a system of resolving conflicts between terminal values: different terminal values are swapped in as the situation warrants. But although there is an evolutionary logic to them - their relative weights are drawn from the kind of a distribution which was useful for survival on average - the conflict-resolution system is not explicitly optimizing for any common currency, not even survival. There just happens to be a hodgepodge of situational variables and processes which end up resolving different conflicts in different ways.

I presented a more complex model of something like this in "Subagents, akrasia and coherence in humans" - there I did say that the subagents are optimizing for an implicit utility function, but the values for that utility function come from cultural and evolution-historical weights so it still doesn't have any consistent "common currency".

Often minds seem to end up at states where something like a particular set of goals or subagents ends up dominating, because those are the ones which have managed to accumulate the most power within the mind-system. This does not look like some of them became the most powerful through something like an appeal to shared values, but rather through just the details of how that person's life-history, their personal neurobiological makeup, etc. happen to be set up and which kinds of neurological processes those details have happened to favor.

Similarly, governments repeat the same pattern at the intrapersonal level - value conflicts are not resolved through being weighted in terms of some higher-level value. Rather they are determined through a complex process where a lot of contingent details, such as a country's parliamentary procedures, cultural traditions, voting systems etc. having a big influence on shaping which way the chips happen to fall WRT any given decision.

Comment by kaj_sotala on Scrying for outcomes where the problem of deepfakes has been solved · 2019-04-15T15:33:45.777Z · score: 18 (9 votes) · LW · GW

It's unclear to me whether deepfakes is going to be that big of an issue in the first place. Written text can already be "faked" - that is, anyone can claim to have witnessed anything and write a report saying so. Photographs can likewise already be altered, staged, or edited in a misleading way.

Society solves the problem by trusting different claims depending on how reliable the source is considered to be. If an established newspaper publishes an investigative report citing anonymous sources, then they are generally trusted because they have staked their reputation on it, even though the whole report could have been made up. But if your crazy neighbor makes the same claim, they are much less likely to be widely believed.

It seems to me that at worst, deepfakes will only take us to a point where photos are about as trustworthy as the written word. But we seem to mostly already operate fine in a world where the written word is trivial to "fake". I'm sure that photography and video being harder to fake makes some contribution to it being easier to trust claims, but my intuition is that most trustworthiness still comes from the sources themselves being considered trustworthy.

Comment by kaj_sotala on Why is multi worlds not a good explanation for abiogenesis · 2019-04-13T19:59:49.561Z · score: 21 (6 votes) · LW · GW

I know almost no physics so this might be a stupid question, but aren't "makes no testable predictions" and "contradicts general relativity" contradictory? Wouldn't contradicting another theory imply some kind of a prediction?

Comment by kaj_sotala on The Hard Work of Translation (Buddhism) · 2019-04-13T06:58:03.597Z · score: 4 (2 votes) · LW · GW

if help the community was your goal, you'd go about nudging norms to encourage "meditate more, read less".

How do you suggest I do that? I honestly don't think I know of a better way than what I'm currently doing.

Comment by kaj_sotala on The Hard Work of Translation (Buddhism) · 2019-04-12T10:23:22.024Z · score: 3 (1 votes) · LW · GW

I interpreted J-'s comments to be criticism of this post in particular (and some other meditation discussion on LW in general), in that this post isn't giving much in the way of correct instruction; it's giving a very general model of what's happening, but it's not saying what one actually needs to do.

I definitely agree that it's better to have a good theoretical model combined with good concrete instructions of what to do; that's why I recommend The Mind Illuminated so widely. But I didn't read J- to be disputing that; in fact, they seemed to agree. Rather I thought J- to feel that "learning to actually become better at meditation" wasn't the motive for why people post meditation stuff on LW, and that people were actually optimizing for something like "seeming smart and getting to philosophize around an interesting topic", which doesn't get anyone to actually practice.

If everyone is just doing intellectual analysis all the time and never practicing, then shutting up about it for a while and going to do some practice is in fact the thing to do; but this is compatible with also reading up on how and why you should do it, if you haven't already done that reading.

Comment by kaj_sotala on The Hard Work of Translation (Buddhism) · 2019-04-12T08:08:49.294Z · score: 6 (3 votes) · LW · GW

But, for fuck's sakes, philosophizing serves the role of masturbation. This is an endemic problem for LW adjacent people, because you all enable each other!

I agree that this is a problem here.

I don't agree that the solution to it is to shut up and go practice more. Or rather, it may help the individual who makes that choice, but it doesn’t help the community in general. It just means that in the absence of that one person, the other people will shift their philosophizing to other topics.

But if the topic of e.g. meditation keeps getting consistently brought up, and its benefits analyzed in a way that makes it seem understandable and valuable to the community, then that might eventually cause people to give it a shot.

This was in fact what convinced me to originally start meditating. I read Ken Wilber and was convinced by some of his arguments on an intellectual level, but also recognized that it was only an intellectual understanding, and that I would probably need to meditate to turn it into a more experiental understanding. Then I also heard a bunch of stuff about the more conventional psychological benefits of meditation, as well as some neuroscience papers about the proposed mechanisms. Those together convinced me to actually start practicing.

I'm pretty convinced that a lot of people here would also be willing to give it a shot, if they were given a sensible explanation of why it might have benefits and what the mechanism for that would be.

And yes, there will also be some people who read the arguments, find them intellectually plausible, and then never try to practice and instead just go back to philosophizing. But at least some people will have found a better direction, while the pure philosophizers would have kept doing their thing anyway.

Comment by kaj_sotala on The Hard Work of Translation (Buddhism) · 2019-04-12T06:13:36.972Z · score: 7 (4 votes) · LW · GW

This seems hard to engage with, given that you've said little about the mystical truth in question, and in fact stated that it can't be expressed in conventional language. How can I evaluate the claim that M is true, if I don't know what M is?

Comment by kaj_sotala on The Hard Work of Translation (Buddhism) · 2019-04-11T13:02:46.667Z · score: 20 (5 votes) · LW · GW
Trying to be post-metaphysical is often about not thinking much about metaphysis and thus in this case staying with the metaphysics of concentration, equanimity, tranquility, mindfulness and suffering without thinking about whether those are the best concepts to use.

Huh? The article's very much saying that we should think about whether the traditional concepts are useful, and then it has an extended case study where it dismantles and reconstructs the four noble truths into a form that's rather different from the common one but which it argues to support practice better. Whether its proposed new version is actually better is a question I don't have a strong opinion on, but it's certainly at least trying; the "mindlessness trainer" criticism seems off.

Comment by kaj_sotala on The Hard Work of Translation (Buddhism) · 2019-04-11T10:28:27.032Z · score: 24 (7 votes) · LW · GW

I like the way that Stephen Batchelor put it:

In the parable of the raft, the Buddha describes “a man in the course of a journey” who arrives at a body of water that he has to cross. Since there are no boats or bridges available, his only option is to assemble a raft out of the “grass, twigs, branches, leaves” and whatever other materials are to hand. Having bound them together, and “making an effort with his hands and feet” he manages to get across to the opposite shore. Despite its evident usefulness, he realises that there is no point in carrying the raft any further once it has accomplished its purpose. So he leaves it by the shore and continues on his way. Likewise, the Buddha concludes, “I have shown you how the dharma is similar to a raft, being for the purpose of crossing over, not for the purpose of grasping” [M. 22]. This story shows how the dharma is an expedient, a means to achieve an urgent task at hand, not an end in itself that is to be preserved at all cost. It emphasises how one needs to draw upon whatever resources are available at a given time in order to accomplish what you have to do. It does not matter whether these resources are “what the Buddha truly taught” or not. The only thing that matters is whether such a configuration of disparate elements is of any help in getting you across the river. [...] In the light of this parable, it makes little sense to ask: “Is this really Buddhism?” The only relevant question is: “Does it float?”

Comment by kaj_sotala on The Hard Work of Translation (Buddhism) · 2019-04-11T10:24:03.165Z · score: 6 (4 votes) · LW · GW

There are a fuckton of writings on meditation, and the benefit you get from reading them is less than sitting down and practicing.

Which isn't to say that additional instructions would be useless. That basic instruction is useful in getting started, but it is also easy to stall and stop making progress when those instructions prove inadequate (this happened to me until I found better, and much more detailed, instructions).

Additionally, it helps to have pointers to what you should be looking for. IME, the work on the couch is what gets my mind sharp and provides some useful insights, but the real benefit comes from applying that sharpness and insights off the couch. But then there are a lot of different things that you can investigate off the couch - and investigating different things may also bring different results, so having theory to act as a guide is useful (not to mention motivational).

So I'd say that practice & reading beats practice alone and practice alone beats reading alone.

Comment by kaj_sotala on Subagents, akrasia, and coherence in humans · 2019-04-11T07:07:44.092Z · score: 8 (3 votes) · LW · GW

I haven't thought that much about it, but "re-process memories" feels like... it sort of requires language, and orientation around narratives.

Hmm. I'm not sure to what extent, if any, I'm using language when I'm re-processing memories? Except when I'm explicitly thinking about what I want to say to someone, or what I might want to write, I generally don't feel like I think in a language: I feel like I think in mental images and felt senses.

"Narratives", I think, are basically impressions of cause and effect or simple mental models, and any animals that could be described as "intelligent" in any reasonable sense do need to have those. "Memory re-processing", would then just be an update to the mental model that you interpreted the memory in terms of.

I feel like this excerpt from "Don't Shoot the Dog" could be an example of very short-term memory reprocessing:

I once videotaped a beautiful Arabian mare who was being clicker-trained to prick her ears on command, so as to look alert in the show ring. She clearly knew that a click meant a handful of grain. She clearly knew her actions made her trainer click. And she knew it had something to do with her ears. But what? Holding her head erect, she rotated her ears individually: one forward, one back; then the reverse; then she flopped both ears to the sides like a rabbit, something I didn't know a horse could do on purpose. Finally, both ears went forward at once. Click! Aha! She had it straight from then on. It was charming, but it was also sad: We don't usually ask horses to think or to be inventive, and they seem to like to do it.

This (and other similar anecdotes in the book) doesn't look to me like it's just simple reinforcement learning: rather, it looks to me more like the horse has a mental model of the trainer wanting something, and is then systematically exploring what that something might be, until it hits on the right alternative. And when it does, there's a rapid re-interpretation of the memory just a moment ago: from "in this situation, my trainer wants me to do something that I don't know what", to "in this situation, my trainer wants me to prick my ears".

Comment by kaj_sotala on Subagents, akrasia, and coherence in humans · 2019-04-10T06:53:57.085Z · score: 3 (1 votes) · LW · GW

Partly because it seems to implicitly claim there are exactly 2 strategies, or that they evolved in a specific order.

Oh, I didn't mean to imply either of those. (those are the two strategies that I know of, but there could obviously be others as well)

It seems like nonhuman animals need to deal with similar kinds of spaghetti code, but I'd be somewhat surprised if the way they experienced that made most sense to classify as "re-processing memories."

How come?

Subagents, akrasia, and coherence in humans

2019-03-25T14:24:18.095Z · score: 85 (23 votes)

Subagents, introspective awareness, and blending

2019-03-02T12:53:47.282Z · score: 58 (18 votes)

Building up to an Internal Family Systems model

2019-01-26T12:25:11.162Z · score: 148 (48 votes)

Book Summary: Consciousness and the Brain

2019-01-16T14:43:59.202Z · score: 92 (29 votes)

Sequence introduction: non-agent and multiagent models of mind

2019-01-07T14:12:30.297Z · score: 85 (30 votes)

18-month follow-up on my self-concept work

2018-12-18T17:40:03.941Z · score: 55 (15 votes)

Tentatively considering emotional stories (IFS and “getting into Self”)

2018-11-30T07:40:02.710Z · score: 39 (11 votes)

Incorrect hypotheses point to correct observations

2018-11-20T21:10:02.867Z · score: 73 (29 votes)

Mark Eichenlaub: How to develop scientific intuition

2018-10-23T13:30:03.252Z · score: 68 (28 votes)

On insecurity as a friend

2018-10-09T18:30:03.782Z · score: 36 (18 votes)

Tradition is Smarter Than You Are

2018-09-19T17:54:32.519Z · score: 68 (24 votes)

nostalgebraist - bayes: a kinda-sorta masterpost

2018-09-04T11:08:44.170Z · score: 16 (7 votes)

New paper: Long-Term Trajectories of Human Civilization

2018-08-12T09:10:01.962Z · score: 27 (13 votes)

Finland Museum Tour 1/??: Tampere Art Museum

2018-08-03T15:00:05.749Z · score: 20 (6 votes)

What are your plans for the evening of the apocalypse?

2018-08-02T08:30:05.174Z · score: 24 (11 votes)

Anti-tribalism and positive mental health as high-value cause areas

2018-08-02T08:30:04.961Z · score: 26 (10 votes)

Fixing science via a basic income

2018-08-02T08:30:04.380Z · score: 30 (14 votes)

Study on what makes people approve or condemn mind upload technology; references LW

2018-07-10T17:14:51.753Z · score: 21 (11 votes)

Shaping economic incentives for collaborative AGI

2018-06-29T16:26:32.213Z · score: 47 (13 votes)

Against accusing people of motte and bailey

2018-06-03T21:31:24.591Z · score: 83 (27 votes)

AGI Safety Literature Review (Everitt, Lea & Hutter 2018)

2018-05-04T08:56:26.719Z · score: 37 (10 votes)

Kaj's shortform feed

2018-03-31T13:02:47.793Z · score: 12 (2 votes)

Helsinki SSC March meetup

2018-03-26T19:27:17.850Z · score: 12 (2 votes)

Is the Star Trek Federation really incapable of building AI?

2018-03-18T10:30:03.320Z · score: 29 (8 votes)

My attempt to explain Looking, insight meditation, and enlightenment in non-mysterious terms

2018-03-08T07:37:54.532Z · score: 274 (99 votes)

Some conceptual highlights from “Disjunctive Scenarios of Catastrophic AI Risk”

2018-02-12T12:30:04.401Z · score: 63 (18 votes)

On not getting swept away by mental content

2018-01-25T20:30:03.750Z · score: 23 (7 votes)

Papers for 2017

2018-01-04T13:30:01.406Z · score: 32 (8 votes)

Paper: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering

2018-01-03T14:39:18.024Z · score: 1 (1 votes)

Paper: Superintelligence as a Cause or Cure for Risks of Astronomical Suffering

2018-01-03T13:57:55.979Z · score: 16 (6 votes)

Fixing science via a basic income

2017-12-08T14:20:04.623Z · score: 38 (11 votes)

Book review: The Upside of Your Dark Side: Why Being Your Whole Self–Not Just Your “Good” Self–Drives Success and Fulfillment

2017-12-04T13:10:06.995Z · score: 27 (8 votes)

Meditation and mental space

2017-11-06T13:10:03.612Z · score: 26 (7 votes)

siderea: What New Atheism says

2017-10-29T10:19:57.863Z · score: 12 (3 votes)

Postmodernism for rationalists

2017-10-17T12:20:36.139Z · score: 24 (1 votes)

Anti-tribalism and positive mental health as high-value cause areas

2017-10-17T10:20:03.359Z · score: 30 (10 votes)

You can never be universally inclusive

2017-10-14T11:30:04.250Z · score: 34 (10 votes)

Meaningfulness and the scope of experience

2017-10-05T11:30:03.863Z · score: 35 (14 votes)

Social Choice Ethics in Artificial Intelligence (paper challenging CEV-like approaches to choosing an AI's values)

2017-10-03T17:39:00.683Z · score: 8 (3 votes)

LW2.0 now in public beta (you'll need to reset your password to log in)

2017-09-23T12:00:50.345Z · score: 2 (2 votes)

Nobody does the thing that they are supposedly doing

2017-09-23T10:40:06.155Z · score: 61 (29 votes)

Debiasing by rationalizing your own motives

2017-09-03T12:20:10.405Z · score: 1 (1 votes)

Multiverse-wide Cooperation via Correlated Decision Making

2017-08-20T12:01:36.212Z · score: 4 (4 votes)

How I [Kaj] found & fixed the root problem behind my depression and anxiety after 20+ years

2017-07-26T12:56:47.127Z · score: 16 (16 votes)

Tenenbaum et al. (2017) on the computational mechanisms of learning a commonsense moral theory

2017-07-25T13:36:34.648Z · score: 1 (1 votes)

The Internet as an existential threat

2017-07-09T11:40:50.195Z · score: 4 (4 votes)

S-risks: Why they are the worst existential risks, and how to prevent them

2017-06-20T12:34:32.608Z · score: 20 (20 votes)

Cognitive Science/Psychology As a Neglected Approach to AI Safety

2017-06-05T13:55:40.261Z · score: 5 (5 votes)